Configurable data migration method, apparatus, device, and medium

Abstract

The disclosure provides a configurable data migration method, device, equipment and medium, relating to the technical field of computer and the technical field of data processing. The method comprises: obtaining configuration information of to-be-migrated data by calling an interface of a rule configuration device, wherein the configuration information comprises data storage location information, data verification information and data loading information; generating a data migration workflow based on the configuration information of the to-be-migrated data by using a control scheduling device, wherein the data migration workflow at least comprises definitions of a plurality of migration steps and scheduling sequences between the plurality of migration steps; and migrating the to-be-migrated data from a source system to a target system according to the data migration workflow.

Description

Technical Field

[0001] This disclosure relates to the fields of computer technology and data processing technology, and more specifically, to a configurable data migration method, apparatus, device, and medium. Background Technology

[0002] As cloud platform technology matures and is widely adopted, data migration between different systems is sometimes necessary. For example, the financial industry is increasingly providing Software-as-a-Service (SaaS) and customized products via the internet. Some financial companies with strong technological capabilities are gradually starting to offer a complete set of financial solutions related to SaaS financial products and customized products to small and medium-sized financial enterprises. This presents the challenge of migrating data from the existing systems of these small and medium-sized financial enterprises to the SaaS financial product and customized product systems.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technologies: the process of data migration between heterogeneous systems suffers from high R&D costs and low execution efficiency. Summary of the Invention

[0004] In view of this, the present disclosure provides a configurable data migration method, apparatus, device, and medium.

[0005] The first aspect of this disclosure provides a configurable data migration method, comprising:

[0006] The configuration information of the data to be migrated is obtained by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information and data loading information.

[0007] A data migration workflow is generated using a control and scheduling device based on the configuration information of the data to be migrated. The data migration workflow includes at least the definition of multiple migration steps and the scheduling order between the multiple migration steps.

[0008] Based on the data migration workflow described above, the data to be migrated will be migrated from the source system to the target system.

[0009] According to embodiments of this disclosure, the rule configuration device includes a configuration storage unit and a configuration maintenance unit, and the configuration information of the data to be migrated obtained by calling the interface of the rule configuration device includes:

[0010] Obtain the configuration information corresponding to the data to be migrated stored in the above configuration storage unit;

[0011] The configuration storage unit is a distributed cache, and the configuration maintenance unit is used by the development team to maintain the configuration information stored in the configuration storage unit.

[0012] According to embodiments of this disclosure, the data storage location information in the above configuration information is configured as follows:

[0013] Determine the processing location of the target business when the business uses the aforementioned source system to process the target business, wherein the aforementioned target business is associated with the aforementioned data to be migrated;

[0014] Determine the order in which the various sub-data of the data to be migrated are generated during the process of handling the aforementioned target business;

[0015] Configure the data storage location information according to the processing location and the order in which the various sub-data of the data to be migrated are generated.

[0016] According to embodiments of this disclosure, the control and scheduling device includes a scheduling configuration unit, a scheduling execution unit, and a scheduling control unit. The generation of a data migration workflow using the control and scheduling device based on the configuration information of the data to be migrated includes:

[0017] The above data migration workflow is generated and saved using the aforementioned scheduling configuration unit.

[0018] The aforementioned scheduling and execution unit coordinates multiple migration devices to migrate the aforementioned data to be migrated based on the aforementioned data migration workflow.

[0019] The aforementioned scheduling and control unit provides an interface for monitoring the execution status of the data migration workflow and an operation interface for controlling the running status of the data migration workflow.

[0020] According to embodiments of this disclosure, migrating the data to be migrated from the source system to the target system according to the data migration workflow includes:

[0021] Based on the configuration information of the data to be migrated, first migration data is obtained from the source system. The first migration data represents the data of the data to be migrated that has been verified to meet the format requirements of the source system.

[0022] The first migration data is converted into second migration data that is consistent with the data format of the target system.

[0023] The second migration data is validated to obtain the third migration data, wherein the third migration data represents the data in the second migration data that has passed the validation.

[0024] The aforementioned third migration data will be migrated to the aforementioned target system.

[0025] According to embodiments of this disclosure, obtaining the first migration data from the source system based on the configuration information of the data to be migrated includes:

[0026] The data to be migrated is obtained from the aforementioned source system using a data acquisition device;

[0027] The data to be migrated is verified using a data inspection device to obtain the first migration data.

[0028] According to embodiments of this disclosure, converting the first migration data into second migration data consistent with the data format of the target system includes:

[0029] The first migration data is cached using a first buffer device according to the data topic corresponding to the first migration data.

[0030] The consumer device obtains the first migration data from the first buffer device according to the data topic, and sends the first migration data to the second buffer device.

[0031] The second buffer device is used to perform data transformation on the first migration data according to the data topic to obtain the second migration data, and the second migration data is stored.

[0032] According to embodiments of this disclosure, the third migration data includes fifth migration data and sixth migration data, and the third migration data is obtained by performing data verification on the second migration data as follows:

[0033] The second migration data is verified using a data verification device according to the data format of the target system stored in the data verification information, to obtain the fourth migration data and the first overload range.

[0034] Based on the first migration data mentioned above, the fourth migration data mentioned above is summarized and verified to obtain the fifth migration data and the second overload range;

[0035] The second migration data within the first and second overload ranges is verified in detail to obtain the sixth migration data.

[0036] According to embodiments of this disclosure, the above-described configurable data migration method further includes:

[0037] A detailed verification of the second migration data within the first and second overload ranges is performed to obtain the third overload range.

[0038] The test data corresponding to the third overload range mentioned above is re-migrated.

[0039] A second aspect of this disclosure provides a configurable data migration apparatus, comprising:

[0040] The configuration information acquisition module is used to obtain the configuration information of the data to be migrated by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information and data loading information.

[0041] The workflow generation module is used to generate a data migration workflow based on the configuration information of the data to be migrated using the control and scheduling device. The data migration workflow includes at least the definition of multiple migration steps and the scheduling order between the multiple migration steps.

[0042] The data migration module is used to migrate the data to be migrated from the source system to the target system according to the data migration workflow described above.

[0043] A third aspect of this disclosure provides an electronic device comprising:

[0044] One or more processors;

[0045] Memory, used to store one or more instructions.

[0046] When one or more of the above instructions are executed by one or more processors, the one or more processors implement the method described above.

[0047] A fourth aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0048] A fifth aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0049] According to embodiments of this disclosure, configuration information of the data to be migrated is obtained by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information, and data loading information. This enables the migration range and migration order information of the data to be migrated to be obtained based on the data storage location information, the verification rule information for verifying the data to be migrated to be obtained based on the data verification information, and the cache cluster information and data topic information for caching the data to be migrated to be migrated to be obtained based on the data loading information. Then, the control and scheduling device generates a data migration workflow based on the configuration information of the data to be migrated to be migrated. According to the data migration workflow, the data to be migrated to be migrated is migrated from the source system to the target system. This enables the migration and verification of the data to be migrated within the migration range according to the pre-configured migration order, thereby improving the efficiency of data migration while ensuring the correctness of the data to be migrated. Attached Figure Description

[0050] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0051] Figure 1 This diagram illustrates an application scenario of a configurable data migration method and apparatus according to embodiments of the present disclosure.

[0052] Figure 2 A flowchart illustrating a configurable data migration method according to an embodiment of the present disclosure is shown schematically.

[0053] Figure 3 A flowchart illustrating a configurable data migration method according to another embodiment of this disclosure is shown schematically;

[0054] Figure 4 A flowchart illustrating a configurable data migration method according to yet another embodiment of the present disclosure is shown.

[0055] Figure 5 This diagram illustrates how data migration is performed using a data migration method when the workflow is paused as configured in the data verification information.

[0056] Figure 6 This diagram illustrates how data migration is performed using a data migration method without interrupting the workflow, as configured in the data verification information.

[0057] Figure 7 A block diagram schematically illustrates a configurable data migration apparatus according to embodiments of the present disclosure; and

[0058] Figure 8 A block diagram of a computer system suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Detailed Implementation

[0059] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0060] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0061] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0062] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0063] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0064] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0065] Data migration between heterogeneous systems suffers from high R&D costs and low execution efficiency. Therefore, this disclosure provides a configurable data migration method, apparatus, device, and medium.

[0066] Embodiments of this disclosure provide a configurable data migration method, the method comprising: obtaining configuration information of data to be migrated by calling the interface of a rule configuration device, wherein the configuration information includes data storage location information, data verification information, and data loading information; generating a data migration workflow based on the configuration information of the data to be migrated using a control scheduling device, wherein the data migration workflow includes at least the definition of multiple migration steps and the scheduling order between the multiple migration steps; and migrating the data to be migrated from the source system to the target system according to the data migration workflow.

[0067] According to embodiments of this disclosure, configuration information of the data to be migrated is obtained by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information, and data loading information. This enables the migration range and migration order information of the data to be migrated to be obtained based on the data storage location information, the verification rule information for verifying the data to be migrated to be obtained based on the data verification information, and the cache cluster information and data topic information for caching the data to be migrated to be migrated to be obtained based on the data loading information. Then, the control and scheduling device generates a data migration workflow based on the configuration information of the data to be migrated to be migrated. According to the data migration workflow, the data to be migrated to be migrated is migrated from the source system to the target system. This enables the migration and verification of the data to be migrated within the migration range according to the pre-configured migration order, thereby improving the efficiency of data migration while ensuring the correctness of the data to be migrated.

[0068] Figure 1 The illustration shows an application scenario of a configurable data migration method and apparatus according to embodiments of the present disclosure.

[0069] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101 and 102, network 103, and service clusters 104, 105, and 106. Network 103 serves as a medium for providing communication links between terminal devices 101 and 102 and service clusters 104, 105, and 106. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0070] Users can use terminal devices 101 and 102 to interact with service clusters 104, 105, and 106 via network 104 to receive or send messages, etc. Various applications that communicate with service clusters 104, 105, and 106 can be installed on terminal devices 101 and 102, such as shopping applications, search applications, instant messaging tools, email clients, etc. (for example only).

[0071] Terminal devices 101 and 102 can be various electronic devices with displays and web browsing capabilities, including but not limited to tablets, laptops, and desktop computers.

[0072] Service clusters 104, 105, and 106 each consist of multiple servers. They can analyze and process received user requests and other data, and then feed the processing results back to the terminal devices. For example, business data from the source system can be stored in server cluster 104, which can analyze the received user requests and perform access operations on the business data in the source system. Business data from the target system can be stored in server cluster 105, which can also analyze the received user requests and perform access operations on the business data in the target system. Data to be migrated during the data migration process can be stored in server cluster 106, which can analyze the received user requests and perform access, verification, and analysis operations on the data to be migrated during the data migration process. Finally, service clusters 104, 105, and 106 feed the processing results back to the terminal devices.

[0073] It should be noted that the configurable data migration method provided in this disclosure embodiment can generally be executed by server groups 104, 105, and 106. Correspondingly, the configurable data migration apparatus provided in this disclosure embodiment can generally be located in server groups 104, 105, and 106. The configurable data migration method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server groups 104, 105, and 106 and is capable of communicating with terminal devices 101, 102, and / or server groups 104, 105, and 106. Correspondingly, the configurable data migration apparatus provided in this disclosure embodiment can also be located in a server or server cluster that is different from server groups 104, 105, and 106 and is capable of communicating with terminal devices 101, 102, and / or server groups 104, 105, and 106.

[0074] It should be understood that Figure 1 The number of terminal devices, networks, servers, and server groups shown is merely illustrative. Any number of terminal devices, networks, servers, and server groups can be included depending on implementation needs.

[0075] Figure 2 A flowchart illustrating a configurable data migration method according to an embodiment of the present disclosure is shown.

[0076] like Figure 2 As shown, the method includes operations S201 to S203.

[0077] In operation S201, the configuration information of the data to be migrated is obtained by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information and data loading information.

[0078] According to embodiments of this disclosure, the data to be migrated represents data stored in the source system that needs to be migrated to the target system.

[0079] According to embodiments of this disclosure, the data storage location information includes the storage location information of the data to be migrated in the source system and the storage location information in the target system. According to embodiments of this disclosure, the data to be migrated may include multiple sub-data items. Therefore, the data storage location information may include the storage location information of multiple sub-data items in the source system and the storage location information in the target system. Based on the storage order of the multiple sub-data items in the data storage location information, the migration order of the multiple sub-data items can be determined. Therefore, by obtaining the data storage location information, the migration range and migration order of the data to be migrated can be determined.

[0080] According to embodiments of this disclosure, data verification information characterizes rules for verifying the correctness of data to be migrated. The data verification information includes rules for verifying the completeness and conformity of the data to be migrated after its collection, and for verifying its completeness, correctness, and conformity to the rules of the target system after converting the data format of the data to be migrated from the source system format to the target system format.

[0081] According to embodiments of this disclosure, data loading information represents storage information for caching the data to be migrated during the migration process. The data loading information includes storage location information of the buffer cluster caching the data to be migrated and data topic information corresponding to the data to be migrated.

[0082] According to the embodiments of this disclosure, before migrating the data to be migrated, it is necessary to configure the configuration information of the data to be migrated, and clarify the data storage location information, data verification information and data loading information of the data to be migrated, that is, clarify the migration scope, migration order information, verification rule information and cache cluster information and data topic information of the data to be migrated during the migration process.

[0083] In operation S202, the control and scheduling device generates a data migration workflow based on the configuration information of the data to be migrated. The data migration workflow includes at least the definition of multiple migration steps and the scheduling order between the multiple migration steps.

[0084] According to embodiments of this disclosure, the migration steps may include data acquisition, data verification, data caching, data conversion, etc., and multiple steps may be executed multiple times according to the scheduling order generated by the control and scheduling device.

[0085] According to embodiments of this disclosure, for example, the control and scheduling device can schedule the data acquisition device to collect the data to be migrated from the source system based on the data storage location information in the configuration information. Then, based on the data verification information, the control and scheduling device can schedule the data verification device to verify the correctness of the data to be migrated. After verifying that the data to be migrated is complete and conforms to the format of the source system, the control and scheduling device can schedule the first data buffer device to store the data to be migrated based on the data loading information. Then, the control and scheduling device can send the first migration data to the second buffer device. Based on the data verification information, the control and scheduling device can schedule the second buffer device to convert and verify the data to be migrated to obtain the data to be migrated that conforms to the data format of the target system. Finally, the data to be migrated that conforms to the data format of the target system is stored in the target system.

[0086] According to embodiments of this disclosure, if the data to be migrated is found to be incomplete, does not conform to the format of the source system, or does not conform to the format of the target system, the control and scheduling device can decide whether to continue the workflow of migrating the data to be migrated or to re-migrate the data based on the data verification information. Re-migration means correcting the data to be migrated, reconfiguring the configuration information, or correcting the data to be migrated and reconfiguring the configuration information before re-migrating the data to be migrated.

[0087] In operation S203, the data to be migrated is migrated from the source system to the target system according to the data migration workflow.

[0088] According to embodiments of this disclosure, there can be multiple source systems and target systems. Configuration information of the data to be migrated can be obtained by calling the interface of the rule configuration device. The control and scheduling device generates a data migration workflow based on the configuration information of the data to be migrated, thereby realizing the migration of the data to be migrated from multiple source systems to multiple target systems according to the data migration workflow.

[0089] According to embodiments of this disclosure, configuration information of the data to be migrated is obtained by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information, and data loading information. This enables the migration range and migration order information of the data to be migrated to be obtained based on the data storage location information, the verification rule information for verifying the data to be migrated to be obtained based on the data verification information, and the cache cluster information and data topic information for caching the data to be migrated to be migrated to be obtained based on the data loading information. Then, the control and scheduling device generates a data migration workflow based on the configuration information of the data to be migrated to be migrated. According to the data migration workflow, the data to be migrated to be migrated is migrated from the source system to the target system. This enables the migration and verification of the data to be migrated within the migration range according to the pre-configured migration order, thereby improving the efficiency of data migration while ensuring the correctness of the data to be migrated.

[0090] According to embodiments of this disclosure, the rule configuration apparatus includes a configuration storage unit and a configuration maintenance unit, for example, Figure 2 As shown in S201, obtaining the configuration information of the data to be migrated by calling the interface of the rule configuration device can include the following operations:

[0091] Retrieve the configuration information corresponding to the data to be migrated, stored in the configuration storage unit;

[0092] The configuration storage unit is a distributed cache, and the configuration maintenance unit is used by the development team to maintain the configuration information stored in the configuration storage unit.

[0093] According to embodiments of this disclosure, the storage unit can be a distributed cache such as Redis or a distributed cache such as Memcached. Embodiments of this disclosure do not limit the specific distributed cache and can be selected according to the actual situation.

[0094] According to embodiments of this disclosure, the configuration storage unit can be, for example, a distributed cache called Redis. Since Redis is an in-memory database, and its data resides in memory, reading and writing data to Redis is extremely fast. Therefore, setting the configuration storage unit as a distributed cache like Redis can accelerate the acquisition of configuration information for the data to be migrated.

[0095] According to embodiments of this disclosure, when the configuration storage unit is a distributed cache Redis, the configuration information corresponding to the data to be migrated stored in the configuration storage unit can be persistently stored. That is, during the migration of the data to be migrated, if the migration program is closed or the memory is reclaimed for other reasons, the configuration information stored in memory is saved to the disk. After the migration program is restored, the configuration information is reloaded from the disk into memory, avoiding the loss of configuration information, reducing the number of maintenance times and time required for the development side to maintain the configuration information stored in the configuration storage unit, and improving the efficiency of migrating the data to be migrated.

[0096] According to embodiments of this disclosure, the data storage location information in the configuration information is configured as follows:

[0097] Determine the processing location of the target business when the business uses the source system to process the target business, where the target business is associated with the data to be migrated;

[0098] Determine the order in which the various sub-data of the data to be migrated are generated during the process of handling the target business;

[0099] Configure data storage location information according to the processing location and the order of the various sub-data that generate the data to be migrated.

[0100] According to embodiments of this disclosure, the source system may include multiple services, and any one of the multiple services included in the source system may be used as the target service.

[0101] According to embodiments of this disclosure, for example, the source system may include three businesses: loan business, deposit business, and investment and wealth management business. In this case, any one of the loan business, deposit business, and investment and wealth management business can be used as the target business.

[0102] According to embodiments of this disclosure, for example, the processing location of the business terminal when using the source system to process the target business can be determined based on the provincial name, municipal name, district name, county name, or local name. Embodiments of this disclosure do not limit the specific method of dividing the processing location, and can be selected according to the actual situation.

[0103] According to embodiments of this disclosure, for example, when a business uses the source system to process a loan transaction, the processing location can be province A, province B, or province C. In this case, the data corresponding to the loan transaction can be divided into three parts according to province A, province B, and province C, forming three data to be migrated. The storage location information of the source system and the target system corresponding to the three data to be migrated is configured into the data storage location information.

[0104] According to the embodiments of this disclosure, after determining the processing location of the business terminal when using the source system to process the target business, the data under the target business can be divided into multiple data ranges based on the processing location, forming multiple data to be migrated. Then, the storage location information of the source system and the target system corresponding to the multiple data to be migrated is configured into the data storage location information. Subsequently, the multiple data to be migrated can be migrated from the source system to the target system step by step according to the data storage location information, thereby realizing data isolation of multiple data to be migrated under the same target business. If a problem occurs in the migration process of one of the multiple data to be migrated, it will not affect the migration process of other data to be migrated, so that the migration process of other data to be migrated can still run normally, improving the efficiency of migrating data to be migrated under the same target business.

[0105] According to embodiments of this disclosure, the scope of the data to be migrated can be all data corresponding to the target business.

[0106] According to embodiments of this disclosure, the data to be migrated may include multiple sub-data, and the scope of the sub-data may be the same type of data related to the target business.

[0107] According to embodiments of this disclosure, for example, the target business can be a loan business. During the loan processing process, borrower information data is generated first, followed by loan credit data, and then loan amount data. Therefore, during the loan processing process, borrower information data, loan credit data, and loan amount data are generated sequentially.

[0108] According to the embodiments of this disclosure, the data of loan personnel under the target business can be used as the first type of data to form the first sub-data, the loan credit data under the target business can be used as the second type of data to form the second sub-data, and the loan amount data under the target business can be used as the third type of data to form the third sub-data, thus forming three sub-data related to the loan business. The three sub-data can be used as one data to be migrated.

[0109] According to embodiments of this disclosure, multiple sub-data under some target services are related, that is, changes in the first generated sub-data will affect the changes in the later generated sub-data. Therefore, in the process of migrating data to be migrated, if multiple sub-data are migrated as a whole, if the first generated sub-data is wrong, the later generated sub-data will also be wrong. This requires checking the data in multiple sub-data to confirm the range of data that is wrong, increasing the scope of the check and the scope of the data to be migrated, and reducing the efficiency of data migration.

[0110] According to embodiments of this disclosure, data storage location information can be configured according to the order of each sub-data of the data to be migrated generated during the processing of the target business. Subsequently, each sub-data can be migrated sequentially according to the order of each sub-data configured in the storage location information, thereby isolating each sub-data. That is, during the migration process, if the first generated sub-data has an error, the first generated sub-data is checked and re-migrated. If the first generated sub-data is successfully migrated, then the later generated sub-data is migrated. This ensures that the later generated sub-data is not affected by the first generated sub-data, reduces the scope of data that has errors during the data migration process and the scope of data that needs to be re-migrated, and improves migration efficiency.

[0111] According to embodiments of this disclosure, data storage location information can be configured according to the processing location, or according to the order of the various sub-data that generate the data to be migrated, or the data storage location information can be configured jointly according to the processing location and the order of the various sub-data that generate the data to be migrated. The specific configuration method can be selected according to the actual situation.

[0112] According to embodiments of this disclosure, for example, when there is no correlation between multiple sub-data in the data to be migrated and data problems will not affect each other, data storage location information can be configured only by selecting the processing location. When there is a correlation between multiple sub-data and data problems will affect each other, data storage location information can be configured according to the order in which the sub-data to be migrated is generated, or data storage location information can be configured jointly according to the processing location and the order in which the sub-data to be migrated is generated.

[0113] According to embodiments of this disclosure, data storage location information can also be configured according to the type of customer handling the target business, so as to further divide the data to be migrated into multiple sub-data. The specific information used to configure the data storage location information can be selected according to the actual situation.

[0114] According to embodiments of this disclosure, customer types can be, for example, large enterprises, medium-sized enterprises, small enterprises, and micro-enterprises.

[0115] According to embodiments of this disclosure, by determining the processing location of the business terminal when processing the target business using the source system, the order of each sub-data of the data to be migrated generated during the processing of the target business is determined. Then, based on the processing location and the order of each sub-data of the data to be migrated, data storage location information is configured. This allows the migration order of each sub-data to be recorded using the data storage location information. Subsequently, during the migration of each sub-data from the source system to the target system based on the order of each sub-data recorded in the storage location information, each sub-data is isolated, the scope of erroneous data to be checked and the scope of re-migrating data during the data migration process are reduced, and the migration efficiency is improved.

[0116] According to embodiments of this disclosure, the control and scheduling device includes a scheduling configuration unit, a scheduling execution unit, and a scheduling control unit, for example... Figure 2 As shown in S202, the control and scheduling device generates a data migration workflow based on the configuration information of the data to be migrated, which may include the following operations:

[0117] Generate and save the data migration workflow using the scheduling configuration unit;

[0118] The scheduling execution unit coordinates multiple migration devices to migrate data based on the data migration workflow;

[0119] The scheduling and control unit provides an interface for the execution status of the data migration workflow and an operation interface for controlling the running status of the data migration workflow.

[0120] According to embodiments of the present disclosure, multiple migration devices may include a data acquisition device, a data inspection device, a first buffer device, a consumption device, a second buffer device, and a data verification device.

[0121] According to embodiments of this disclosure, coordinating the migration of data to be migrated by multiple migration devices based on a data migration workflow using a scheduling execution unit may include: coordinating data transmission between the acquisition device and the data inspection device, data transmission between the data inspection device and the first buffer device, data transmission between the first buffer device and the consumption device, data transmission between the consumption device and the second buffer device, and data transmission between the first buffer device, the second buffer device and the data verification device.

[0122] According to embodiments of this disclosure, for example, the scheduling execution unit can be used to obtain data loading information configured by the first buffer device, and then the data loading information can be provided to the consumption device, the first buffer device, and the data verification device for use.

[0123] According to embodiments of this disclosure, a data migration workflow is generated and saved using a scheduling configuration unit. A scheduling execution unit coordinates multiple migration devices to migrate data based on the data migration workflow, thereby increasing the speed of data migration by coordinating multiple migration devices based on the data migration workflow. This allows multiple migration devices to migrate data in an orderly manner. The scheduling control unit provides a data migration workflow execution status interface and an operation interface for controlling the running status of the data migration workflow. This allows maintenance personnel to view the data migration status at any time using the data migration workflow execution status interface and control the migration workflow based on the data migration status using the operation interface for controlling the running status of the data migration workflow. This enables maintenance personnel to promptly handle unexpected errors during the data migration process, reducing the overall impact of unexpected errors on the data migration.

[0124] According to embodiments of this disclosure, for example, Figure 2 As shown in S203, migrating the data to be migrated from the source system to the target system according to the data migration workflow may include the following operations:

[0125] Based on the configuration information of the data to be migrated, first migration data is obtained from the source system. The first migration data represents the data to be migrated that has been verified to meet the format requirements of the source system.

[0126] Convert the first migration data into second migration data that is consistent with the data format of the target system;

[0127] The second migration data is validated to obtain the third migration data, whereby the third migration data represents the validated data in the second migration data.

[0128] Migrate the third migration data to the target system.

[0129] According to embodiments of this disclosure, based on the configuration information of the data to be migrated, first migration data is obtained from the source system. The first migration data represents data that has been verified to conform to the format requirements of the source system. The first migration data is converted into second migration data that conforms to the data format of the target system. The second migration data is then verified to obtain third migration data. The third migration data represents the verified data in the second migration data. The third migration data is then migrated to the target system. This achieves the migration of data that conforms to both the data format of the source system and the data format of the target system from the target system to the target system, ensuring the correctness of the data to be migrated to the target system.

[0130] According to embodiments of this disclosure, obtaining first migration data from the source system based on the configuration information of the data to be migrated includes:

[0131] Data to be migrated is acquired from the source system using a data acquisition device;

[0132] The data to be migrated is verified using a data inspection device to obtain the first migration data.

[0133] According to embodiments of this disclosure, the data acquisition device may include a data acquisition unit and a data transmission unit.

[0134] According to embodiments of this disclosure, acquiring data to be migrated from a source system using a data acquisition device may include: acquiring data storage location information and data verification information from a control and scheduling device using a data acquisition unit; then acquiring the storage location and migration order of the data to be migrated based on the data storage location information; acquiring the storage format of the data to be migrated in the source system based on the data verification information; and then reading the data to be migrated from the storage location of the data to be migrated in the source system according to the storage format of the data to be migrated, based on the migration order of the data to be migrated.

[0135] According to embodiments of this disclosure, acquiring data to be migrated from a source system using a data acquisition device may further include: sending the data to be migrated to a data inspection device using a data transmission unit.

[0136] According to embodiments of this disclosure, the data inspection apparatus may include a data inspection unit and a data transmission unit.

[0137] According to embodiments of this disclosure, using a data inspection device to verify the data to be migrated and obtain the first migration data may include: the data inspection unit obtaining data verification information from the control and scheduling device, and then verifying the format of the data to be migrated based on the data verification information to obtain the first migration data.

[0138] According to embodiments of this disclosure, the format of the data to be migrated can be validated based on the data validation information, for example, by performing data non-empty checks, data enumeration checks, data range checks, data association checks, data total score checks, and data interception checks. The specific form of validation can be selected according to the actual situation, and embodiments of this disclosure do not limit the specific form of data validation.

[0139] According to embodiments of this disclosure, a non-empty data check characterizes whether data is missing from locations where data should be present.

[0140] According to embodiments of this disclosure, the enumeration check characterizes whether the data to be migrated conforms to the data types of the source system when the source system has a fixed number of data types. For example, if the source system has two data types, A and B, and type C appears during the verification of the data to be migrated, then the data to be migrated does not conform to the data types of the source system.

[0141] According to embodiments of this disclosure, a data range check characterizes whether the numerical value of the checked data conforms to the data range of the source system. For example, if the upper limit of loan amount data is 10 million, during the migration of loan amount data, it is necessary to verify whether the loan amount data is less than or equal to 10 million.

[0142] According to embodiments of this disclosure, the total data score check characterizes the checking of the total number of data or the total number of values ​​of the data to be migrated.

[0143] According to embodiments of this disclosure, a data association check characterizes a check for missing data on related data that must be migrated together. For example, if data A and B must be migrated together, it is necessary to check whether the data to be migrated includes both data A and B.

[0144] According to embodiments of this disclosure, the data interception check represents the rules for whether to intercept data to be migrated if problems are detected. For data inspection rules requiring interception, if problems are found after inspection, the data to be migrated is corrected and its configuration information reconfigured before being migrated again, until the data passes the inspection and is then migrated to the target system. For data inspection rules not requiring interception, if problems are found after inspection, the inspection result, i.e., the scope of the data problems, is recorded. After the data migration is completed, the data to be migrated is corrected and its configuration information reconfigured before being migrated again, until the data passes the inspection and is then migrated to the target system.

[0145] According to embodiments of this disclosure, the data inspection rules that need to be intercepted are applicable to data in the data to be migrated where multiple sub-data will affect each other, that is, data where the sub-data migrated first has a problem and, if not corrected, will affect the sub-data migrated later.

[0146] According to embodiments of this disclosure, the data inspection rules that do not require interception are applicable to data in the data to be migrated that do not affect each other.

[0147] According to embodiments of this disclosure, after the data to be migrated has been verified, a data compression algorithm can be used to compress the data to be migrated, and the compressed data to be migrated can be used as the first data to be migrated, which can improve the speed of subsequent transmission of the data to be migrated, thereby improving the data migration efficiency.

[0148] According to embodiments of this disclosure, the data compression algorithm can be, for example, several file compression programs (GNU zip, gzip), the lossless compression algorithm lz4, and the algorithm provided by the data compression library snappy. Embodiments of this disclosure do not limit the data compression algorithm, and the algorithm can be selected according to the actual situation.

[0149] According to embodiments of this disclosure, the process of using a data inspection device to verify the data to be migrated and obtain the first migration data further includes: after obtaining the first migration data, sending the first migration data to a data buffer device through a data sending unit.

[0150] According to embodiments of this disclosure, data to be migrated is obtained from the source system using a data acquisition device. This allows the data acquisition device to acquire data within the migration range according to a pre-configured migration order based on the configuration information of the data to be migrated. Then, a data inspection device is used to verify the data to be migrated to obtain the first migration data. This process verifies the data to be migrated according to the data format of the source system, ensuring that the data to be migrated conforms to the data format of the source system and guaranteeing the correctness of the data to be migrated to the target system.

[0151] Figure 3 A flowchart illustrating a configurable data migration method according to another embodiment of the present disclosure is shown.

[0152] like Figure 3 As shown, the configurable data migration method of this embodiment may further include operations S301 to S303.

[0153] In operation S301, the first buffer device is used to cache the first migration data according to the data topic corresponding to the first migration data;

[0154] In operation S302, the consumer device obtains first migration data from the first buffer device according to the data topic, and sends the first migration data to the second buffer device.

[0155] In operation S303, the second buffer device is used to perform data transformation on the first migration data according to the data topic to obtain the second migration data, and the second migration data is stored.

[0156] According to embodiments of this disclosure, a data topic represents a common feature name of the data in the first migration data, and the data topic needs to be configured in the configuration information before migrating the data to be migrated.

[0157] According to embodiments of this disclosure, for example, the business name corresponding to the data to be migrated, the data name generated during the business process (e.g., the loan amount under the loan business), the data cluster name storing the data to be migrated, or the data cluster address storing the data to be migrated can be used as the data subject. Embodiments of this disclosure do not limit the specific method for determining the data subject, and the data subject name can be selected according to the actual situation.

[0158] According to embodiments of this disclosure, for example, when the first migration data consists entirely of loan amounts generated during the loan processing process, "loan amount" can be used as the data subject of the first migration data. When the first migration data consists entirely of a portion of the loan amounts generated during the loan processing process, the loan amount and the storage location information (e.g., storage address) of these loan amounts stored in the source system can be used as the data subject. Alternatively, the storage location information of these loan amounts can be used directly as the data subject.

[0159] According to embodiments of this disclosure, first migration data is converted according to a data topic, which means that the format of the first migration data is converted into the target system format according to the data topic.

[0160] According to embodiments of this disclosure, the first buffer device includes a data buffer unit and a configuration information unit.

[0161] According to an embodiment of this disclosure, caching the first migration data using a first buffer device based on a data topic corresponding to the first migration data includes: obtaining data loading information from a control and scheduling device using a configuration information unit, obtaining a data topic corresponding to the first migration data based on the data loading information, caching the first migration data using a data buffer unit based on the data topic, storing the cache address in the data loading information, and sending the data loading information to the control and scheduling device.

[0162] According to embodiments of this disclosure, the data buffer unit may include multiple servers.

[0163] According to embodiments of this disclosure, message bus Kafka software can also be deployed in the first buffer device to improve the speed of accessing the first migration data.

[0164] According to embodiments of this disclosure, the consumer device may include a data parsing unit, a data routing unit, and a consumer configuration unit.

[0165] According to embodiments of this disclosure, using a consumer device to obtain first migration data from a first buffer device based on a data topic and sending the first migration data to a second buffer device may include: obtaining data loading information from a control and scheduling device using a data parsing unit, obtaining a cache address for caching the first migration data in the first buffer device based on the data loading information, and obtaining the first migration data from a data buffer unit based on the data topic and the cache address.

[0166] According to embodiments of this disclosure, if the first migration data stored in the first cache device is compressed data, the first migration data can also be decompressed using a data parsing unit.

[0167] According to embodiments of this disclosure, the data parsing unit can also be used to parse the first migration data into a specific format, such as a lightweight data exchange format (JavaScript Object Notation, JSON), to improve the speed of parsing and generating data, thereby improving the efficiency of data transmission.

[0168] According to embodiments of this disclosure, the process of using a consumer device to obtain first migration data from a first buffer device based on a data topic and sending the first migration data to a second buffer device may further include: using a data routing unit to send the first migration data to the cluster in the second buffer device corresponding to the data topic, based on the data topic in the configuration information. The consumer configuration unit periodically calls the update notification interface of the control and scheduling device to parse the notification update time. If the notification update time is later than the update time of the data loading information sent by the data routing unit, the control and scheduling device's configuration update interface is called to obtain the latest data loading information to replace the current data loading information, and the routing sending unit is called to send the second migration data according to the latest data loading information.

[0169] According to an embodiment of this disclosure, if the notification update time is earlier than the update time of the data loading information sent by the data routing unit, after the routing sending unit sends the second migration data according to the latest data loading information, the consumer device continues to obtain the first migration data from the first buffer device according to the data topic, and sends the first migration data to the second buffer device.

[0170] According to an embodiment of this disclosure, when the first buffer device notifies the consumer device to retrieve the first data through the control scheduling device, the consumer device then retrieves the first migration data from the first buffer device according to the data topic.

[0171] According to embodiments of this disclosure, the notification update time represents the deadline for updating data in the source system. The deadline for updating data in the source system can be, for example, 10 PM, 11 PM, or midnight. Embodiments of this disclosure do not limit the deadline for updating data in the source system; it can be selected according to actual circumstances.

[0172] According to embodiments of this disclosure, when the amount of data to be migrated is large, a deadline for updating the data in the source system can be set. The migration of the data to be migrated can begin before the deadline, thereby migrating the data to be migrated from the source system to the target system earlier.

[0173] According to embodiments of this disclosure, the second buffer device includes: a data processing unit, a data storage unit, and a processing configuration unit.

[0174] Table 1 schematically illustrates the relationship between the data subject and the storage location corresponding to the first and second buffer devices.

[0175] As shown in Table 1, the first group represents the cluster corresponding to the first buffer device used to store the first migration data, and the second group represents the cluster corresponding to the second buffer device used to store the second migration data.

[0176] Table 1

[0177] Data topic Group 1 Second Group Loan customer information Loan Customer First Cluster Loan Customer Second Cluster Loan Credit Loan Credit First Cluster Loan Credit Second Cluster Loan Agreement Loan Contracts First Cluster Loan Contracts Second Cluster

[0178] As shown in Table 1, when the data subject is loan customer information, the first migration data corresponding to the loan customer information can be cached in the first loan customer cluster. After converting the first migration data into second migration data based on the data subject of loan customer information, the second migration data can be stored in the second loan customer cluster. Similarly, the first migration data corresponding to loan credit can be cached in the first loan credit cluster, the second migration data corresponding to loan credit can be stored in the second loan credit cluster, the first migration data corresponding to loan contracts can be cached in the first loan contract cluster, and the second migration data corresponding to loan contracts can be stored in the second loan contract cluster.

[0179] According to embodiments of this disclosure, the process of using a second buffer device to perform data conversion on the first migration data according to the data topic to obtain second migration data and storing the second migration data includes: using a data processing unit to perform data conversion on the first migration data according to data loading information and data verification information to obtain the second migration data; using a data storage unit to store the second migration data; using a processing configuration unit to periodically call the update notification interface of the control and scheduling device, parse the notification update time, and if the notification update time is later than the update time of the data loading information sent by the data routing unit, then the latest data loading information is obtained from the configuration update interface of the control and scheduling device, and the current data loading information is replaced with the latest data loading information, and the data processing unit is called to process the data according to the latest data loading information.

[0180] According to an embodiment of this disclosure, if the notification update time is earlier than the update time when the data routing unit sends the data loading information, after the data processing unit processes the data according to the latest data loading information, the second buffer device continues to use the data conversion of the first migration data according to the data topic to obtain the second migration data, and the second migration data is stored.

[0181] According to embodiments of this disclosure, the data processing unit performs data transformation on the first migration data in three ways: supporting direct writing, custom processing, and field mapping. The specific data transformation method can be selected according to the actual situation.

[0182] According to embodiments of this disclosure, when the data structure of the first migration data in the source system is completely consistent with the data structure in the target system, the first migration data can be stored directly in the data storage unit without data conversion.

[0183] According to embodiments of this disclosure, when the field names or storage locations of the first migration data change in the source system and the target system, the first migration data is stored in a data storage unit using a field mapping method.

[0184] According to embodiments of this disclosure, field mapping can be, for example, enumeration value conversion, data splitting, or data merging. For instance, the first migration data may be enumeration values ​​A and C in the source system and corresponding to B and D in the target system; data from the same data table in the source system may be split into multiple data tables in the target system; or data from multiple data tables in the source system may be merged into the same data table in the target system.

[0185] According to embodiments of this disclosure, when the format conversion between the source system and the target system is complex, custom processing can be achieved by calling some external program interfaces.

[0186] Table 2 schematically illustrates how the second buffer device performs data transformation on the first migration data according to the data subject when the data subjects of the first migration data are loan contracts, loan credit, and loan customer information.

[0187] Table 2

[0188]

[0189] As shown in Table 2, when the data subject is loan customer information, the data transformation method is field mapping. Specifically, the loan customer information is converted from 4 bytes in the source system to 8 bytes in the target system for storage.

[0190] When the data subject is loan credit, the data transformation method is custom processing. Specifically, the operation involves calling the external program interface Pack credit.credit extension to convert the loan credit into data in the target system's format.

[0191] When the data subject is a loan contract, the data transformation method is direct writing. Specifically, the relevant migration program is called to directly write the data corresponding to the loan contract from Table 1 of the source system to a table with the same format as Table 2 of the target system.

[0192] According to embodiments of this disclosure, a first buffer device caches the first migration data according to a data topic corresponding to the first migration data. A consumption device obtains the first migration data from the first buffer device according to the data topic and sends the first migration data to a second buffer device. The second buffer device performs data conversion on the first migration data according to the data topic to obtain second migration data, and stores the second migration data. This achieves the coordination of the data processing speed of the data acquisition device and the data verification device using the first buffer device, the consumption device, and the second buffer device. Furthermore, it maps the storage location of the data to be migrated in the source system, the storage location in the first buffer device, the storage location in the second buffer device, and the storage location in the target system according to the data topic, making the management of the scope and format of the data to be migrated more convenient and faster.

[0193] Figure 4 A flowchart illustrating a configurable data migration method according to yet another embodiment of the present disclosure is shown.

[0194] like Figure 4 As shown, the configurable data migration method of this embodiment may further include operations S401 to S403.

[0195] In operation S401, the data verification device verifies the second migration data according to the data format of the target system stored in the data verification information, and obtains the fourth migration data and the first overload range.

[0196] In operation S402, based on the first migration data, the fourth migration data is summarized and verified to obtain the fifth migration data and the second overload range;

[0197] In operation S403, detailed verification is performed on the second migration data within the first overload range and the second overload range to obtain the sixth migration data.

[0198] According to embodiments of this disclosure, the fourth migration data represents the data in the second migration data that conforms to the data format requirements of the target system, and the first overload range represents the range of data in the second migration data that does not conform to the data format requirements of the target system.

[0199] According to embodiments of this disclosure, the data verification apparatus includes a summary verification unit, a detailed verification unit, and a verification configuration unit.

[0200] According to embodiments of this disclosure, using a data verification device to verify the second migration data according to the data format of the target system stored in the data verification information to obtain the fourth migration data and the first overload range includes: using a summary verification unit to obtain data verification information and data loading information from a control scheduling device; obtaining the second migration data from the data storage unit of the data loading device according to the data topic and storage address in the data loading information; and verifying the second migration data according to the data format of the target system stored in the data verification information to obtain the fourth migration data and the first overload range.

[0201] According to embodiments of this disclosure, the data verification device verifies the second migration data based on the data format of the target system stored in the data verification information. For example, it may perform data non-empty checks, data enumeration checks, data range checks, data association checks, data total score checks, and data interception checks on the second migration data. The specific form of verification can be selected according to the actual situation. The embodiments of this disclosure do not limit the specific verification form.

[0202] According to embodiments of this disclosure, the fifth migration data represents the data in the fourth migration data that has passed the summary verification, and the second overload range represents the range of data in the fourth migration data that has not passed the summary verification.

[0203] According to embodiments of this disclosure, the process of summarizing and verifying the fourth migration data based on the first migration data to obtain the fifth migration data and the second overload range may include: using a summary verification unit to obtain the first migration data from the data storage unit of the first cache device according to the data topic and storage address in the data loading information, and summarizing and verifying the fourth migration data according to the verification rules stored in the data verification information to obtain the fifth migration data and the second overload range.

[0204] According to an embodiment of this disclosure, the summary verification rule stored in the data verification information can be: when the fourth migration data is text data, the number of the first migration data corresponding to the fourth migration data is calculated to obtain a first total number, the number of the fourth migration data is calculated to obtain a second total number, and the size of the first total number and the second total number are compared to obtain the fifth migration data and the second overload range.

[0205] According to an embodiment of this disclosure, the summary verification rule stored in the data verification information may also be: when the fourth migration data is numerical data, the values ​​of the first migration data corresponding to the fourth migration data are summed to obtain a third total; the values ​​of the fourth migration data are summed to obtain a fourth total; the size of the third total and the fourth total are compared to obtain the fifth migration data and the second overload range.

[0206] Table 3 illustrates the rules for summarizing and validating the fourth migration data corresponding to loan contracts, loan credit, and loan customer information using the summary validation unit, when the data subjects are loan contracts, loan credit, and loan customer information, respectively.

[0207] Table 3

[0208] Data topic Summary of Validation Rules Summary Dimensions Loan customer information Total number of customers Institutional + Client Type Dimension Loan Credit Total credit line Loan Purpose Dimension Loan Agreement Total Loan Amount Institutional Dimension

[0209] As shown in Table 3, the "Institution" indicates the region where the loan customer is located, such as Province A, Province B, Province C, etc. The "Customer Type" indicates the size of the customer's business, such as large enterprise, medium-sized enterprise, small enterprise, micro-enterprise. The "Loan Purpose" can be, for example, working capital loan, project loan, or real estate loan.

[0210] As shown in Table 3, when the data subject is loan customer information, the total number of customers in the first migration data corresponding to the loan customer information can be summarized according to the dimensions of institution and customer type to obtain the third total number. The total number of customers in the fourth migration data corresponding to the loan customer information can be summarized to obtain the fourth total number. By comparing whether the third total number and the fourth total number are equal, the fifth migration data and the second overload range can be obtained.

[0211] When the data subject is loan credit, the total credit limit in the first migration data corresponding to loan credit can be aggregated according to the loan purpose dimension to obtain the third total. Similarly, the total credit limit in the fourth migration data corresponding to loan credit can be aggregated to obtain the fourth total. By comparing the third and fourth totals, the fifth migration data and the second overload range can be obtained. Likewise, based on the data subject loan contract and the aggregation verification rules, the fifth migration data and the second overload range corresponding to the data subject loan contract can be obtained.

[0212] According to embodiments of this disclosure, performing detailed verification on the second migration data within the first overload range and the second overload range to obtain the sixth migration data includes: using a detailed verification unit to perform detailed verification on the second migration data within the first overload range and the second overload range according to detailed verification rules stored in the data verification information to obtain the sixth migration data.

[0213] According to embodiments of this disclosure, the sixth migration data represents the data that has passed detailed verification in the second migration data within the first overload range and the second overload range.

[0214] According to embodiments of this disclosure, detailed verification characterization compares and verifies key information in the second migration data within the first and second heavy load ranges and the first migration data corresponding to the second migration data within the first and second heavy load ranges.

[0215] Table 4 illustrates the rules for detailed verification of the fourth migration data corresponding to loan contracts, loan credit, and loan customer information using the summary verification unit, when the data subjects are loan contracts, loan credit, and loan customer information, respectively.

[0216] Table 4

[0217]

[0218] As shown in Table 4, when the data subject is loan contracts, the first migration data and the second migration data corresponding to the loan customer information can be compared in detail based on the primary key field, customer creation date and institution dimension.

[0219] When the data subject is loan credit, the first and second migration data corresponding to loan credit can be compared in detail based on the primary key field, credit limit, transaction date, loan purpose and institution dimension.

[0220] When the data subject is a loan contract, a detailed comparison can be made between the first and second migration data corresponding to the loan contract based on the primary key field, loan amount, transaction date, loan purpose + institution, and in-table flag dimensions.

[0221] According to embodiments of this disclosure, after detailed verification of the second migration data, the detailed verification results can be visualized using the data visualization analysis tool DataEase.

[0222] According to embodiments of this disclosure, a data verification device verifies the second migration data based on the data format of the target system stored in the data verification information to obtain the fourth migration data and the first overload range. Based on the first migration data, the fourth migration data is summarized and verified to obtain the fifth migration data and the second overload range. The second migration data within the first overload range and the second overload range is verified in detail to obtain the sixth migration data. This achieves multi-level verification of the sixth migration data, confirming that the data format of the sixth migration data conforms to the format of the target system, the total number of the sixth migration data is the same as the data in the source system, and the key information of the sixth migration data is the same as the data in the source system, thus ensuring the correctness of the sixth migration data.

[0223] According to the embodiments disclosed herein, the above-described configurable data migration method further includes:

[0224] Detailed verification of the second migration data within the first and second overload ranges is performed to obtain the third overload range.

[0225] The test data corresponding to the third overload range is re-migrated.

[0226] According to embodiments of this disclosure, the third overload range represents the data in the first overload range and the second overload range that did not pass the detailed verification.

[0227] According to embodiments of this disclosure, re-migrating the test data corresponding to the third overload range includes: reconfiguring the configuration information according to the third overload range and re-migrating the test data corresponding to the third overload range.

[0228] According to embodiments of this disclosure, by reconfiguring configuration information based on the third overload range, the test data corresponding to the third overload range is re-migrated, thereby achieving re-migrating only the difference data within the third overload range and improving data migration efficiency.

[0229] According to embodiments of this disclosure, during the detailed verification of second migration data within the first and second overload ranges, if an anomaly is found in the second migration data, the workflow is paused based on the rules configured in the data verification information.

[0230] If the workflow is paused, it will be paused and manually taken over. After correcting the second migration data and reconfiguring the configuration information according to the third overload range, the second migration data within the third overload range will be migrated again.

[0231] Figure 5 This diagram illustrates how data migration is performed using a data migration method when the workflow is configured to pause in the data verification information.

[0232] like Figure 5 As shown, the data to be migrated includes three sub-data: customer data, rating data, and credit data. The migration order of the three sub-data is customer data, rating data, and credit data. The workflows corresponding to the three sub-data are: workflow 501 for migrating customer data, workflow 502 for migrating rating data, and workflow 503 for migrating credit data.

[0233] like Figure 5 As shown, rating and credit data are migrated only if the customer data is verified to be normal. If the customer data is verified to be abnormal, all workflows are paused and the customer data, rating data, and credit data are migrated again by reconfiguring the configuration information.

[0234] Without pausing the workflow, the third overload range is recorded, and the subsequent workflows are executed.

[0235] Figure 6 This diagram illustrates how data migration is performed using a data migration method when the data verification information is configured to not interrupt the workflow.

[0236] like Figure 6 As shown, the data to be migrated includes three sub-data: customer data, rating data, and credit data. The migration order of the three sub-data is customer data, rating data, and credit data. The workflows corresponding to the three sub-data are: workflow 601 for migrating customer data, workflow 602 for migrating rating data, and workflow 603 for migrating credit data.

[0237] like Figure 6 As shown, following the migration order of the three sub-data, customer data, rating data, and credit data are migrated sequentially according to workflow 601 for migrating customer data, workflow 602 for migrating rating data, and workflow 603 for migrating credit data. After all three sub-data are executed, the data verification results are checked. If there are any anomalies in the three sub-data, a third overload range corresponding to the anomaly data will be obtained. The anomaly data in the third overload range can be repaired, and then the customer data, rating data, and credit data in the third overload range can be migrated again by reconfiguring the configuration information.

[0238] According to embodiments of this disclosure, the data verification results can be checked after all data has been processed, and the configuration information of the overload workflow can be set according to the third overload range to achieve overload migration only for the different data, thereby improving the data migration efficiency.

[0239] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0240] Figure 7 A block diagram of a configurable data migration apparatus according to embodiments of the present disclosure is illustrated schematically.

[0241] like Figure 7 As shown, the configurable data migration device 700 includes a configuration information acquisition module 710, a workflow generation module 720, and a data migration module 730.

[0242] The configuration information acquisition module 710 is used to obtain the configuration information of the data to be migrated by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information and data loading information.

[0243] The workflow generation module 720 is used to generate a data migration workflow based on the configuration information of the data to be migrated using the control and scheduling device. The data migration workflow includes at least the definition of multiple migration steps and the scheduling order between the multiple migration steps.

[0244] The data migration module 730 is used to migrate data from the source system to the target system according to the data migration workflow.

[0245] According to embodiments of this disclosure, the rule configuration device includes a configuration storage unit and a configuration maintenance unit, and the configuration information acquisition module includes a configuration information acquisition submodule.

[0246] The configuration information acquisition submodule is used to acquire the configuration information corresponding to the data to be migrated, which is stored in the configuration storage unit.

[0247] The configuration storage unit is a distributed cache, and the configuration maintenance unit is used by the development team to maintain the configuration information stored in the configuration storage unit.

[0248] According to embodiments of this disclosure, the data storage location information in the configuration information is configured as follows:

[0249] Determine the processing location of the target business when the business uses the source system to process the target business, where the target business is associated with the data to be migrated;

[0250] Determine the order in which the various sub-data of the data to be migrated are generated during the process of handling the target business;

[0251] Configure data storage location information according to the processing location and the order of the various sub-data that generate the data to be migrated.

[0252] According to embodiments of this disclosure, the control and scheduling device includes a scheduling configuration unit, a scheduling execution unit, and a scheduling control unit, and the workflow generation module includes a workflow generation submodule, a workflow coordination submodule, and an interface provision submodule.

[0253] The workflow generation submodule is used to generate and save data migration workflows using the scheduling configuration unit.

[0254] The workflow coordination submodule is used to coordinate multiple migration devices to migrate data based on the data migration workflow using the scheduling execution unit.

[0255] The interface provides sub-modules for providing an interface for the execution status of the data migration workflow and an operation interface for controlling the running status of the data migration workflow using the scheduling and control unit.

[0256] According to embodiments of this disclosure, the data migration module includes a first migration data acquisition submodule, a second migration data conversion submodule, a third migration data acquisition submodule, and a target system migration submodule.

[0257] The first migration data acquisition submodule is used to acquire first migration data from the source system based on the configuration information of the data to be migrated. The first migration data represents the data to be migrated that has been verified to meet the format requirements of the source system.

[0258] The second migration data conversion submodule is used to convert the first migration data into second migration data that is consistent with the data format of the target system.

[0259] The third migration data acquisition submodule is used to perform data verification on the second migration data to obtain the third migration data, wherein the third migration data represents the data in the second migration data that has passed the verification.

[0260] The target system migration submodule is used to migrate third-party migration data to the target system.

[0261] According to embodiments of this disclosure, the first migration data acquisition submodule includes a data to be migrated acquisition unit and a first migration data obtaining unit.

[0262] The data acquisition unit is used to acquire the data to be migrated from the source system using a data acquisition device.

[0263] The first migration data obtaining unit is used to verify the data to be migrated using a data checking device to obtain the first migration data.

[0264] According to embodiments of this disclosure, the second migration data conversion submodule includes a first migration data caching unit, a first migration data sending unit, and a second migration data obtaining unit.

[0265] The first migration data caching unit is used to cache the first migration data according to the data topic corresponding to the first migration data using the first buffer device.

[0266] The first migration data sending unit is used to obtain first migration data from the first buffer device according to the data topic using the consumption device, and send the first migration data to the second buffer device.

[0267] The second migration data obtaining unit is used to perform data transformation on the first migration data according to the data topic using the second buffer device to obtain the second migration data and store the second migration data.

[0268] According to embodiments of this disclosure, the third migration data includes the fifth migration data and the sixth migration data, and the third migration data obtaining submodule includes the fourth data and the first range obtaining unit, the fifth data and the second range obtaining unit, and the sixth data obtaining unit.

[0269] The fourth data and first range obtaining unit is used to verify the second migration data using a data verification device according to the data format of the target system stored in the data verification information, and obtain the fourth migration data and the first overload range.

[0270] The fifth data and second range acquisition unit is used to summarize and verify the fourth migration data based on the first migration data to obtain the fifth migration data and the second overload range.

[0271] The sixth data acquisition unit is used to perform detailed verification of the second migration data within the first heavy load range and the second heavy load range to obtain the sixth migration data.

[0272] According to embodiments of this disclosure, the configurable data migration apparatus further includes a third overload range acquisition module and an overload module.

[0273] The third overload range acquisition module is used to perform detailed verification of the second migration data within the first and second overload ranges to obtain the third overload range.

[0274] The overload module is used to re-migrate the test data corresponding to the third overload range.

[0275] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0276] For example, any plurality of the configuration information acquisition module 710, workflow generation module 720, and data migration module 730 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the configuration information acquisition module 710, workflow generation module 720, and data migration module 730 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the configuration information acquisition module 710, workflow generation module 720, and data migration module 730 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0277] It should be noted that the configurable data migration device in the embodiments of this disclosure corresponds to the configurable data migration method in the embodiments of this disclosure. For a detailed description of the configurable data migration device, please refer to the configurable data migration method, which will not be repeated here.

[0278] Figure 8A block diagram of a computer system suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 8 The computer system shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0279] like Figure 8 As shown, a computer system 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0280] RAM 803 stores various programs and data required for the operation of system 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0281] According to embodiments of this disclosure, system 800 may further include an input / output (I / O) interface 805, which is also connected to bus 804. System 800 may also include one or more of the following components connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. Communication section 809 performs communication processing via a network such as the Internet. Drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 810 as needed so that computer programs read from there are installed into storage section 808 as needed.

[0282] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0283] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0284] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0285] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0286] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code enables the electronic device to implement the configurable data migration method provided in the embodiments of this disclosure.

[0287] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0288] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0289] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0290] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0291] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A configurable data migration method, comprising: The configuration information of the data to be migrated is obtained by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information and data loading information. A data migration workflow is generated using a control and scheduling device based on the configuration information of the data to be migrated. The data migration workflow includes at least the definition of multiple migration steps and the scheduling order between the multiple migration steps. Migrating the data to be migrated from the source system to the target system according to the data migration workflow includes: acquiring the data to be migrated from the source system using a data acquisition device; The data verification unit in the data acquisition device obtains data verification information from the control and scheduling device. Based on the data verification information, the format of the data to be migrated is verified to obtain the first migration data. The format verification of the data to be migrated based on the data verification information includes: data non-empty check, data enumeration check, data range check, data association check, data total score check, and data interception check. The first migration data represents the data to be migrated that has been verified to meet the format requirements of the source system. The first migration data is converted into second migration data that is consistent with the data format of the target system; The second migration data is validated to obtain the third migration data, which includes the fifth migration data and the sixth migration data. The fifth migration data represents the data in the fourth migration data that has passed the summary validation. The fourth migration data represents the data in the second migration data that meets the data format requirements of the target system. The sixth migration data represents the data in the second migration data that has passed the detailed validation within the first overload range and the second overload range. The first overload range represents the range of data in the second migration data that does not meet the data format requirements of the target system. The second overload range represents the range of data in the fourth migration data that has not passed the summary validation. The third migration data is migrated to the target system.

2. The method according to claim 1, wherein, The rule configuration device includes a configuration storage unit and a configuration maintenance unit. Obtaining the configuration information of the data to be migrated by calling the interface of the rule configuration device includes: Obtain the configuration information corresponding to the data to be migrated stored in the configuration storage unit; The configuration storage unit is a distributed cache, and the configuration maintenance unit is used for the development side to maintain the configuration information stored in the configuration storage unit.

3. The method according to claim 1 or 2, wherein, The data storage location information in the configuration information is obtained by configuring it in the following way: Determine the processing location of the business terminal when it uses the source system to process the target business, wherein the target business is associated with the data to be migrated; Determine the order in which the various sub-data of the data to be migrated are generated during the process of handling the target business; Configure the data storage location information according to the processing location and the order in which the various sub-data of the data to be migrated are generated.

4. The method according to claim 1 or 2, wherein, The control and scheduling device includes a scheduling configuration unit, a scheduling execution unit, and a scheduling control unit. The step of generating a data migration workflow using the control and scheduling device based on the configuration information of the data to be migrated includes: The data migration workflow is generated and saved using the scheduling configuration unit. The scheduling execution unit utilizes the data migration workflow to coordinate multiple migration devices to migrate the data to be migrated. The scheduling and control unit provides an interface for monitoring the execution status of the data migration workflow and an operation interface for controlling the running status of the data migration workflow.

5. The method according to claim 1, wherein, The step of converting the first migration data into second migration data that is consistent with the data format of the target system includes: The first migration data is cached using a first buffer device according to the data topic corresponding to the first migration data; The consumer device retrieves the first migration data from the first buffer device according to the data topic, and sends the first migration data to the second buffer device; The second buffer device is used to perform data transformation on the first migration data according to the data topic to obtain the second migration data, and the second migration data is stored.

6. The method according to claim 1, wherein, The step of verifying the second migration data to obtain the third migration data includes: The second migration data is verified using a data verification device according to the data format of the target system stored in the data verification information, to obtain the fourth migration data and the first overload range; Based on the first migration data, the fourth migration data is summarized and verified to obtain the fifth migration data and the second overload range; Detailed verification is performed on the second migration data within the first overload range and the second overload range to obtain the sixth migration data.

7. The method according to claim 6, further comprising: A detailed verification is performed on the second migration data within the first and second overload ranges to obtain the third overload range; The test data corresponding to the third overload range is re-migrated.

8. A configurable data migration apparatus, comprising: The configuration information acquisition module is used to obtain the configuration information of the data to be migrated by calling the interface of the rule configuration device. The configuration information includes data storage location information, data verification information and data loading information. A workflow generation module is used to generate a data migration workflow based on the configuration information of the data to be migrated using a control and scheduling device. The data migration workflow includes at least the definition of multiple migration steps and the scheduling order between the multiple migration steps. The data migration module is used to migrate the data to be migrated from the source system to the target system according to the data migration workflow. The data migration module includes a first migration data acquisition submodule, a second migration data conversion submodule, a third migration data acquisition submodule, and a target system migration submodule; The first migration data acquisition submodule is used to acquire the first migration data from the source system based on the configuration information of the data to be migrated. The first migration data represents the data of the data to be migrated that has been verified to meet the format requirements of the source system. The second migration data conversion submodule is used to convert the first migration data into second migration data that is consistent with the data format of the target system. The third migration data acquisition submodule is used to perform data verification on the second migration data to obtain the third migration data. The third migration data includes the fifth migration data and the sixth migration data. The fifth migration data represents the data in the fourth migration data that has passed the summary verification. The fourth migration data represents the data in the second migration data that meets the data format requirements of the target system. The sixth migration data represents the data in the second migration data that has passed the detailed verification within the first overload range and the second overload range. The first overload range represents the range of data in the second migration data that does not meet the data format requirements of the target system. The second overload range represents the range of data in the fourth migration data that has not passed the summary verification. The target system migration submodule is used to migrate third-party migration data to the target system; The first migration data acquisition submodule includes a data acquisition unit to be migrated and a first migration data obtaining unit; The data acquisition unit is used to acquire data to be migrated from the source system using a data acquisition device. The first migration data obtaining unit is used to verify the migration data using a data checking device. Obtaining the first migration data includes: obtaining data verification information from the control and scheduling device using the data checking unit in the data acquisition device, verifying the format of the migration data according to the data verification information, and obtaining the first migration data. The verification of the format of the migration data according to the data verification information includes: performing data non-empty checks, data enumeration checks, data range checks, data association checks, data total score checks, and data interception checks on the migration data.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more instructions. When the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 7.

11. A computer program product comprising computer-executable instructions, which, when executed, are used to implement the method of any one of claims 1 to 7.

Images