Business data migration method, device, equipment and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a service data migration method, device, equipment and storage medium, the method comprising the following steps: when receiving a service migration requirement of a user, generating to-be-migrated service data according to the service migration requirement; performing migration allocation on the to-be-migrated service data according to system overhead conditions, obtaining a migration detail file corresponding to the to-be-migrated service data and a to-be-migrated temporary file; migrating the to-be-migrated temporary file to a target system, so that the target system writes the to-be-migrated temporary file into a database file when a check result of the to-be-migrated temporary file meets data uniqueness. According to the application, the migration detail file and the to-be-migrated temporary file are allocated, data is automatically uploaded and partition processing is performed, the to-be-migrated temporary file is written into the database file only after data uniqueness is verified when the target system receives the to-be-migrated temporary file, data safety and integrity are ensured, migration data in a shutdown state is avoided, and the data migration efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a business data migration method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of cloud computing and the internet, enterprise digital transformation has become widespread. Many companies have customized their existing systems to varying degrees, leading to a demand for system upgrades. However, given the inherent complexity of systems, upgrades are not easy. Furthermore, with the rapid development of big data technology, data has become the most valuable asset for customers, and product upgrades inevitably create a need for data migration.

[0003] During system data migration, the routine data migration server assignment and business personnel verification functions require downtime of several hours or days. Due to a lack of tools and the inability to conduct migration verification tests, data loss and migration failures often occur. Business data migration is characterized by low migration efficiency and long business interruption times. How to achieve cross-platform, secure, efficient, and stable migration while minimizing business interruption time has become an unresolved issue for enterprise production data migration.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a business data migration method, apparatus, device, and storage medium, which aims to solve the technical problem of low migration efficiency caused by downtime and porting verification testing during traditional system data migration processes.

[0006] To achieve the above objectives, the present invention provides a business data migration method, the method comprising the following steps:

[0007] Upon receiving a user's business migration request, generate business data to be migrated based on the business migration request;

[0008] Based on the system overhead, the business data to be migrated is allocated for migration, and the migration detail file and temporary file to be migrated corresponding to the business data to be migrated are obtained.

[0009] The temporary file to be migrated is migrated to the target system according to the migration details file, so that the target system writes the temporary file to be migrated into the database file when the verification result of the temporary file to be migrated meets the data uniqueness requirement.

[0010] Optionally, the system overhead includes data priority and storage.

[0011] The step of allocating the migration data to be migrated based on system overhead, and obtaining the migration detail file and temporary file to be migrated corresponding to the migration data, includes:

[0012] Based on the storage status of the system overhead, the business data to be migrated is allocated and migrated to obtain the temporary file to be migrated corresponding to the business data to be migrated;

[0013] Based on the data priority of the system overhead and preset business conditions, the temporary files to be migrated are extracted to obtain the migration detail files corresponding to the business data to be migrated.

[0014] Optionally, after generating the business data to be migrated based on the user's business migration request upon receiving the request, the process further includes:

[0015] Based on the business migration requirements, the parameters of the business data to be migrated are configured to obtain the blacklist and whitelist corresponding to the migration scope of the business data to be migrated.

[0016] The system status of the receiving systems in the blacklist and whitelist is checked, and it is determined whether the check result meets the preset health status.

[0017] When the inspection results meet the preset health status, rules are formulated for the business data to be migrated according to the business migration requirements to obtain migration data verification rules.

[0018] Accordingly, after extracting the temporary file to be migrated based on the data priority and preset business conditions according to the system overhead, and obtaining the migration detail file corresponding to the business data to be migrated, the process further includes:

[0019] The temporary file to be migrated is verified according to the migration data verification rules, and the verification result is determined to be consistent with the data consistency.

[0020] When the verification result meets the data consistency requirement, the step of migrating the temporary file to be migrated to the target system according to the migration details file is executed.

[0021] Optionally, when the inspection result meets the preset health status, the step of formulating rules for the business data to be migrated according to the business migration requirements to obtain migration data verification rules includes:

[0022] When the inspection results meet the preset health status, the data size of the business data to be transplanted is extracted;

[0023] Based on the business migration requirements and the data scale, the preset migration strategy library is traversed to obtain the migration strategy corresponding to the business data to be migrated.

[0024] Configure SQL statements on the business data to be migrated according to the business migration requirements to obtain migration data verification rules;

[0025] Accordingly, the step of allocating the ported business data according to the storage situation based on the system overhead, and obtaining the corresponding temporary porting file for the ported business data, includes:

[0026] Based on the storage situation and the migration strategy, the business data to be migrated is allocated and migrated to obtain the temporary file to be migrated corresponding to the business data to be migrated.

[0027] Optionally, before migrating the temporary file to be migrated to the target system according to the migration details file, the method further includes:

[0028] Based on the system status, determine the current resource utilization rate of the current system and the receiving system, and determine whether the current resource utilization rate is lower than a preset threshold;

[0029] Accordingly, when the verification result meets the data consistency requirement, the step of migrating the temporary file to be migrated to the target system according to the migration details file includes:

[0030] When the verification result meets the data consistency requirement and the current resource utilization rate is lower than the preset threshold, the step of migrating the temporary file to be migrated to the target system according to the migration details file is executed.

[0031] Optionally, the target system writes the temporary file to be ported into a database file when the verification result of the temporary file to be ported meets the data uniqueness requirement, including:

[0032] The target system reads the number of entries to be transmitted from the migration details file, compares the number of entries to be transmitted with the number of entries already transmitted from the temporary file to be migrated, and determines whether the comparison results are consistent.

[0033] When the comparison results are consistent, the target system converts the temporary file to be ported into data to be stored in the database.

[0034] The target system performs data verification on the data to be added to the database and determines whether the verification result meets the requirement of data uniqueness.

[0035] When the verification result meets the requirement of data uniqueness, the target system writes the data to be stored into the database file.

[0036] Optionally, after migrating the temporary file to be migrated to the target system according to the migration details file, the method further includes:

[0037] Upon receiving the data to be imported from the target system, a protocol comparison is performed on the data to be imported through the DAP temporary table, and the comparison result is used to determine whether the data to be imported is consistent with the temporary file to be ported.

[0038] When the data to be imported is inconsistent with the temporary file to be ported, the temporary file to be ported is updated using a data processing tool to obtain the updated temporary file to be ported.

[0039] The updated temporary file to be migrated is migrated to the target system until the data to be imported is consistent with the updated temporary file to be migrated, at which point the step of migrating the temporary file to be migrated to the target system according to the migration details file is stopped.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a business data migration apparatus, the apparatus comprising:

[0041] The data generation module is used to generate business data to be migrated based on the business migration requirements when a user's business migration request is received.

[0042] The data allocation module is used to allocate the business data to be migrated according to the system overhead, and to obtain the migration detail file and the temporary file to be migrated corresponding to the business data to be migrated.

[0043] The data migration module is used to migrate the temporary file to be migrated to the target system according to the migration details file, so that the target system writes the temporary file to be migrated into the database file when the verification result of the temporary file to be migrated meets the data uniqueness requirement.

[0044] Furthermore, to achieve the above objectives, the present invention also proposes a business data migration device, the device comprising: a memory, a processor, and a business data migration program stored in the memory and executable on the processor, the business data migration program being configured to implement the steps of the business data migration method described above.

[0045] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a business data migration program, which, when executed by a processor, implements the steps of the business data migration method described above.

[0046] Upon receiving a user's business migration request, this invention generates business data to be migrated based on the request. Then, it allocates the migration of this business data according to system overhead, obtaining a migration detail file and a temporary migration file corresponding to the data. Finally, it migrates the temporary migration file to the target system based on the migration detail file, ensuring that the target system writes the temporary migration file to its database file only after verifying its uniqueness. Because this invention automatically uploads data and performs partitioning by allocating the migration detail file and temporary migration file, and only writes data to the database file after verifying its uniqueness upon receipt by the target system, it ensures data security and integrity while avoiding downtime during data migration, thus improving data migration efficiency. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the structure of the business data migration device in the hardware operating environment involved in the embodiments of the present invention;

[0048] Figure 2 This is a flowchart illustrating the first embodiment of the business data migration method of the present invention;

[0049] Figure 3 This is a flowchart illustrating the second embodiment of the business data migration method of the present invention;

[0050] Figure 4 This is a schematic diagram illustrating the process of real-time migration of massive amounts of data to cloud service data in the second embodiment of the business data migration method of the present invention.

[0051] Figure 5 This is a structural block diagram of the first embodiment of the business data migration device of the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0054] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of the business data migration device in the hardware operating environment involved in the embodiments of the present invention.

[0055] like Figure 1As shown, the service data migration device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the business data migration equipment and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a business data migration program.

[0058] exist Figure 1 In the business data migration device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the business data migration device of the present invention can be set in the business data migration device, and the business data migration device calls the business data migration program stored in the memory 1005 through the processor 1001 and executes the business data migration method provided in the embodiment of the present invention.

[0059] This invention provides a business data migration method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the business data migration method of the present invention.

[0060] In this embodiment, the business data migration method includes the following steps:

[0061] Step S10: Upon receiving a user's business migration request, generate the business data to be migrated based on the business migration request.

[0062] It should be noted that the execution subject of the method in this embodiment can be a computing service device with data allocation, data migration, and data processing capabilities, such as a personal computer or a server. It can also be other electronic devices capable of performing the same or similar functions, such as the aforementioned business data migration device. This embodiment does not limit the scope of the application. Here, the aforementioned business data migration device (hereinafter referred to as the data migration device) will be used to specifically describe this embodiment and the following embodiments.

[0063] Understandably, business migration requirements are based on the needs of specific business scenarios, such as branch numbers, customer IDs, business IDs, and different migration batches. The business data to be migrated is the data extracted from the database according to these migration requirements.

[0064] Step S20: Based on the system overhead, allocate the business data to be migrated, and obtain the migration details file and temporary file to be migrated corresponding to the business data to be migrated.

[0065] It should be noted that system overhead refers to the resource consumption and additional burden involved in performing tasks in a computer system, such as computing resources, memory usage, disk space, network bandwidth, and data priority.

[0066] Understandably, the migration detail file is a file containing specific migration details during the data migration process of the business data to be migrated. For example, the migration process includes details such as: single-batch migration, multi-batch migration, migration time, and the size of each migration file. The temporary migration file is a temporary data file obtained by allocating the business data to be migrated based on system overhead.

[0067] In a specific implementation, when the data migration device receives a user's business migration request, it extracts and generates business data to be migrated from the local database according to the business migration request; then, it allocates the business data to be migrated according to the system overhead, and obtains the temporary file to be migrated and the allocated migration detail file.

[0068] Step S30: Migrate the temporary file to be migrated to the target system according to the migration details file, so that the target system writes the temporary file to be migrated into the database file when the verification result of the temporary file to be migrated meets the data uniqueness requirement.

[0069] It should be noted that data uniqueness refers to whether the data in the temporary file to be migrated is duplicated with the data in the target system, and whether there is any duplicate data migration. If the verification result confirms data uniqueness, the data migration is considered successful, and the target system will write the temporary file to be migrated into its own database.

[0070] Furthermore, during data migration, the data in the database can be formatted and organized into the data format required by the Java Spring framework. This includes data statistics, standardization, removal of redundant data, data classification, and integration, all of which improve the efficiency of data migration. Simultaneously, based on the migration details file, data can be automatically uploaded, indexed and partitioned, and verified before being entered into the database, ensuring both data security and integrity while guaranteeing efficient data entry.

[0071] In practical implementation, after obtaining the temporary file to be migrated and the migration details file, the data migration device can migrate the temporary file to the target system via TCP protocol according to the migration details file. If the file transfer fails, the data file transfer steps can be repeated. When the target system receives the temporary file to be migrated, it can perform data verification on the temporary file, such as checking the uniqueness of the data, whether there are data conflicts, and whether there are logical errors. The target system then writes the temporary file to be migrated into its own database.

[0072] In this embodiment, when the data migration device receives a user's business migration request, it extracts and generates the business data to be migrated from the local database according to the request. Then, based on system overhead, it allocates the business data to be migrated, obtaining a temporary file to be migrated and an allocated migration detail file. Next, the temporary file to be migrated is migrated to the target system via TCP according to the migration detail file. If file transfer fails, the data file transfer steps can be repeated. When the target system receives the temporary file to be migrated, it can perform data verification, such as checking data uniqueness, data conflicts, and logical errors. The target system then writes the temporary file to be migrated into its own database. Because this embodiment automatically uploads data and performs partitioning by allocating the migration detail file and the temporary file to be migrated, and verifies data uniqueness before writing it to the database file upon receipt by the target system, it ensures data security and integrity while avoiding downtime during data migration, thus improving the efficiency of data migration.

[0073] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the business data migration method of the present invention.

[0074] Based on the first embodiment described above, in this embodiment, considering data partitioning, the system overhead includes data priority and storage conditions, and step S20 includes:

[0075] Step S21: Based on the storage status of the system overhead, the business data to be migrated is allocated for migration, and the temporary file to be migrated corresponding to the business data to be migrated is obtained.

[0076] It should be noted that data priority is based on the migration priority corresponding to the previous relationships between data. Storage status refers to the storage situation involved when a task is executed in a computer system, such as memory usage and disk space.

[0077] Step S22: Extract the temporary file to be migrated according to the data priority of the system overhead and the preset business conditions to obtain the migration detail file corresponding to the business data to be migrated.

[0078] It should be noted that the preset business conditions are the conditions prepared for migrating business data according to the needs of the business scenario, such as the business scenario and business data requirements.

[0079] In its implementation, the data migration device allocates data to the business data to be migrated based on system overhead and storage availability. For example, it creates temporary files to be migrated in batches when memory usage is low. Then, based on data priority according to system overhead and preset business conditions (such as business data requirements), it extracts the temporary files to be migrated and determines the migration details (such as migration process and order). Thus, during the automatic data migration process, it performs operations such as indexing, partitioning, and data verification, ensuring data security and integrity while guaranteeing efficient data entry.

[0080] Furthermore, considering the security of the data migration process, this embodiment further includes the following steps after step S10: configuring parameters for the business data to be migrated according to the business migration requirements to obtain a blacklist or whitelist corresponding to the migration scope of the business data to be migrated; checking the system status of the receiving systems in the blacklist or whitelist and determining whether the check result meets the preset health status; when the check result meets the preset health status, formulating rules for the business data to be migrated according to the business migration requirements to obtain migration data verification rules.

[0081] Accordingly, after step S22, the method further includes: verifying the temporary file to be migrated according to the migration data verification rules, and determining whether the verification result meets the data consistency requirements; when the verification result meets the data consistency requirements, performing the step of migrating the temporary file to be migrated to the target system according to the migration details file.

[0082] It should be noted that blacklists and whitelists are lists of objects to be migrated. For example, they can be configured through parameters to control and maintain data migration at dimensions such as the entire bank, branches, customer numbers, and business numbers.

[0083] Understandably, the preset health status refers to the working status of each system during the data migration process, such as whether the network connectivity, CPU utilization, and storage utilization of the requesting and receiving systems are normal. Data migration preparation can proceed when the receiving system is in the preset health status.

[0084] It should be noted that the migration data verification rules are the rules for the data migration process, such as communication or data encoding formats.

[0085] In its implementation, the data migration device first configures parameters for the business data to be migrated according to the business migration requirements, obtaining a blacklist and whitelist corresponding to the data migration scope. Then, it checks the system status of the receiving systems on the blacklist and whitelist. If the receiving system's system status is in a preset healthy state, data migration preparation can begin. Next, rules are formulated for the business data to be migrated according to the business migration requirements, obtaining migration data verification rules. Correspondingly, after obtaining the migration detail file corresponding to the business data to be migrated, the temporary file to be migrated can also be verified according to the migration data verification rules. If the verification result meets the data consistency requirements, the temporary file to be migrated can be migrated to the target system. Thus, through multi-layered security measures and fault recovery mechanisms, the security and reliability of data transmission and data entry processes are ensured.

[0086] Furthermore, in this embodiment, when the inspection result meets the preset health status, the step of formulating rules for the business data to be migrated according to the business migration requirements to obtain migration data verification rules includes: when the inspection result meets the preset health status, extracting the data size of the business data to be migrated; traversing the preset migration strategy library according to the business migration requirements and the data size to obtain the migration strategy corresponding to the business data to be migrated; and configuring SQL statements for the business data to be migrated according to the business migration requirements to obtain migration data verification rules.

[0087] Accordingly, step S21 includes: allocating the business data to be migrated according to the storage situation and the migration strategy, and obtaining the temporary file to be migrated corresponding to the business data to be migrated.

[0088] It should be noted that the data scale refers to the scale of the business data to be migrated, such as the size of the data to be transferred, the number of transactions, and whether it is a single batch or multiple batches. The preset migration strategy library consists of migration strategies pre-configured in the data migration equipment. Based on the business migration requirements and data scale, a suitable migration strategy can be directly queried from the preset migration strategy library, improving the efficiency of data migration.

[0089] In the specific implementation, when the receiving system meets the preset health status, the data size of the business data to be migrated is extracted. Then, based on the business migration requirements and data size, the preset migration strategy library is traversed to determine the migration strategy corresponding to the business data to be migrated. The migration data verification rules can be directly configured with SQL statements according to the business migration requirements, standardizing the management and operation of relational databases. Correspondingly, based on the storage situation and the aforementioned migration strategy, data allocation can be performed on the business data to be migrated to obtain corresponding temporary files for migration. This allows for direct querying of suitable migration strategies from the preset migration strategy library, improving the efficiency of data migration.

[0090] Furthermore, considering the system's working state during data migration, this embodiment further includes, before step S30: determining the current resource utilization rate of the current system and the receiving system based on the system state, and determining whether the current resource utilization rate is lower than a preset threshold.

[0091] Accordingly, the step of migrating the temporary file to be migrated to the target system according to the migration details file when the verification result meets the data consistency requirement includes: when the verification result meets the data consistency requirement and the current resource utilization rate is lower than the preset threshold, the step of migrating the temporary file to be migrated to the target system according to the migration details file is executed.

[0092] It should be noted that the preset threshold is set based on the system's resource utilization rate. For example, the system's resource utilization rate is low at night, and performing data migration at this time can improve the smoothness and efficiency of data migration.

[0093] In its implementation, before data migration, the data migration device can determine whether the current resource utilization rates of the current system and the receiving system are lower than a preset threshold based on the system status. If the data meets the data consistency requirements and the current resource utilization rate is lower than the preset threshold, the temporary files to be migrated can be migrated to the target system according to the migration details file. This ensures data security and integrity while avoiding downtime during data migration when resource utilization is low, thus improving the smoothness and efficiency of data migration.

[0094] Furthermore, in order to...bring forth the beneficial effects or reasons of the right..., in this embodiment, in step S30, the target system writes the temporary file to be ported into the database file when the verification result of the temporary file to be ported meets the data uniqueness requirement. This includes: the target system reading the number of records to be transmitted from the porting details file, comparing the number of records to be transmitted with the number of records already transmitted from the temporary file to be ported, and determining whether the comparison result is consistent; when the comparison result is consistent, the target system converts the temporary file to be ported into data to be imported into the database; the target system performs data verification on the data to be imported into the database and determines whether the verification result meets the data uniqueness requirement; when the verification result meets the data uniqueness requirement, the target system writes the data to be imported into the database file.

[0095] In practical implementation, after receiving the data, the target system can verify the transplanted data. For example, it can read the number of transmitted data entries from the transplantation details file and compare it with the number of data entries in the temporary transplantation file. If the comparison results match, the temporary file to be transplanted is converted into data to be imported into the database, and the uniqueness of the data to be imported into the database is checked against the existing data. If the data uniqueness is satisfied, the data to be imported into the database can be written into the database file. This ensures the security and reliability of data transmission and the import process.

[0096] Furthermore, considering the data format changes during the data migration process, this embodiment further includes the following steps after step S30: upon receiving the data to be imported from the target system, performing a protocol comparison on the data to be imported using a DAP temporary table, and determining whether the data to be imported is consistent with the temporary file to be migrated based on the comparison result; if the data to be imported is inconsistent with the temporary file to be migrated, updating the temporary file to be migrated using a data processing tool to obtain an updated temporary file to be migrated; migrating the updated temporary file to be migrated to the target system until the data to be imported is consistent with the updated temporary file to be migrated, then stopping the step of migrating the temporary file to be migrated to the target system based on the migration details file.

[0097] It's important to note that a DAP (Data Analysis Platform) temporary table is a table structure used to temporarily store data within a data analysis platform. During data analysis, data is typically extracted from different data sources, cleaned, transformed, and calculated, and then the results are stored in the table for subsequent analysis and processing. Due to inconsistent data migration protocols, special symbols or uncommon characters may change, necessitating validation and updates.

[0098] In the specific implementation, upon receiving data to be imported from the target system, a protocol comparison can be performed on the data in the DAP temporary table. If the data to be imported is inconsistent with the temporary file to be migrated, the temporary file to be migrated can be updated using a data processing tool. The updated temporary file to be migrated is then migrated to the target system until the data to be imported is consistent with the updated temporary file to be migrated, at which point the data migration operation is successfully completed. This multi-level fault recovery mechanism prevents data failures and ensures the reliability of data transmission and the import process.

[0099] For ease of understanding, this example illustrates the real-time migration of massive amounts of data to cloud services, but it does not limit the scope of this solution. (Reference) Figure 4 , Figure 4 This is a schematic diagram illustrating the process of real-time migration of massive amounts of data to cloud service data in the second embodiment of the business data migration method of the present invention. Figure 4 As shown, the data migration process is as follows:

[0100] Step 1: Migrate Blacklists and Whitelists. Customize the blacklist and whitelist configuration based on the business migration requirements. Configure them through the parameter configuration module, such as blacklist and whitelist control and maintenance at the dimensions of the entire bank, branch, customer number, and business number.

[0101] Step 2: Check health status. Check the health status of the requesting and receiving systems. For example, check the network connectivity, CPU utilization, and storage utilization of both sides. Run the migration operation when the resource utilization of both systems is low.

[0102] Step 3: Define the migration configuration. Define the migration configuration for single or multiple batches and customize the migration data verification rules. For example, formulate single or multiple batch migration strategies based on the size and number of data entries to be transferred, and configure the number of migration batches and the number of entries migrated per batch through parameters. Simultaneously, configure SQL statements through parameters and configure data verification rules that meet business needs.

[0103] Step 4: Generate the data to be migrated. Generate the business data to be migrated according to business requirements and check the data to be migrated. If this step fails, it can be repeated. For example, based on system overhead, extract data that meets the business conditions and write it into a temporary file to be migrated. Execute the data checking step, execute the data validation statements configured in Step 3, and check the data consistency between the extracted file and the temporary data file.

[0104] Step 5: Send the migration data. This step can be repeated if it fails. Temporary migration files can be transferred via TCP. The number of jobs sent depends on the size of the temporary file. If file transfer fails, the data file transfer step can be repeated.

[0105] Step 6: Verify the data. After receiving the data, the new system verifies the migrated data, checks and compares the data for consistency, identifies the cause of failure, and repeats the process. For example, it reads the number of data entries transferred from the migration details file and compares this number with the number of data entries in the temporary migration file. If the data verification fails, it identifies the cause of the failure and repeats the data checking or data transfer steps.

[0106] Step 7: Import Data into the Database. Import the imported data into the database file of the new system. At this point, read the SQL statement configured in Step 3 to check the uniqueness of the data to be imported against the existing data. If conflicting data exists, analyze the cause of the conflict. For example, deduplicate data is removed, and the deduplicated data is imported into the corresponding database file in the new system.

[0107] Step 8: Validate the business logic. Randomly select business requests for validity verification. Business personnel initiate service requests in both the new and old systems, record the business results, and compare the data results between the new and old systems to see if they are consistent.

[0108] In this embodiment, the data migration device allocates data to the business data to be migrated based on the storage situation and system overhead. For example, it creates temporary files to be migrated in batches when memory usage is low. Then, it extracts the temporary files to be migrated according to the data priority based on system overhead and preset business conditions (e.g., business data requirements), and determines the migration details (e.g., migration process, sequence, etc.) of the temporary files to be migrated. Thus, during the automatic data migration process, it performs operations such as indexing, partitioning, and data verification, ensuring data security and integrity while ensuring efficient data entry. Furthermore, the data migration device can first configure parameters for the business data to be migrated according to the business migration requirements to obtain a blacklist and whitelist corresponding to the data migration scope. Then, it checks the system status of the receiving system in the blacklist and whitelist. If the system status of the receiving system is in a preset healthy state, data migration preparation can be performed. Next, rules are formulated for the business data to be migrated according to the business migration requirements to obtain migration data verification rules. Correspondingly, after obtaining the migration details file corresponding to the business data to be migrated, the temporary files to be migrated can also be verified according to the migration data verification rules. If the verification result meets the data consistency requirements, the temporary files to be migrated can be migrated to the target system. This multi-layered security measures and fault recovery mechanisms ensure the security and reliability of data transmission and database entry. Furthermore, when the receiving system meets a preset health status, the data size of the business data to be migrated is extracted; and based on the business migration requirements and data size, the preset migration strategy library is traversed to determine the migration strategy corresponding to the business data to be migrated. The migration data verification rules can be directly configured with SQL statements according to the business migration requirements, standardizing the management and operation of relational databases. Correspondingly, data allocation can be performed on the business data to be migrated based on storage conditions and the aforementioned migration strategy to obtain corresponding temporary files to be migrated. This allows for direct querying of suitable migration strategies from the preset migration strategy library, improving data migration efficiency. Simultaneously, before data migration, the data migration device can determine whether the current resource utilization rates of the current system and the receiving system are lower than a preset threshold based on the system status. If the data meets the data consistency requirements and the current resource utilization rate is lower than the preset threshold, the temporary files to be migrated can be migrated to the target system according to the migration details file. This ensures data security and integrity while avoiding downtime during data migration when resource utilization is low, improving the smoothness and efficiency of data migration.

[0109] Furthermore, this embodiment of the invention also proposes a storage medium storing a business data migration program, which, when executed by a processor, implements the steps of the business data migration method described above.

[0110] Reference Figure 5 , Figure 5This is a structural block diagram of the first embodiment of the business data migration device of the present invention.

[0111] like Figure 5 As shown, the business data migration device proposed in this embodiment of the invention includes:

[0112] Data generation module 501 is used to generate business data to be migrated according to the business migration requirements when a user's business migration request is received.

[0113] Data allocation module 502 is used to allocate the business data to be migrated according to the system overhead, and obtain the migration detail file and the temporary file to be migrated corresponding to the business data to be migrated.

[0114] The data migration module 503 is used to migrate the temporary file to be migrated to the target system according to the migration details file, so that the target system writes the temporary file to be migrated into the database file when the verification result of the temporary file to be migrated meets the data uniqueness requirement.

[0115] In this embodiment, when the data migration device receives a user's business migration request, it extracts and generates the business data to be migrated from the local database according to the request. Then, based on system overhead, it allocates the business data to be migrated, obtaining a temporary file to be migrated and an allocated migration detail file. Next, the temporary file to be migrated is migrated to the target system via TCP according to the migration detail file. If file transfer fails, the data file transfer steps can be repeated. When the target system receives the temporary file to be migrated, it can perform data verification, such as checking data uniqueness, data conflicts, and logical errors. The target system then writes the temporary file to be migrated into its own database. Because this embodiment automatically uploads data and performs partitioning by allocating the migration detail file and the temporary file to be migrated, and verifies data uniqueness before writing it to the database file upon receipt by the target system, it ensures data security and integrity while avoiding downtime during data migration, thus improving the efficiency of data migration.

[0116] Based on the first embodiment of the business data migration device of the present invention described above, a second embodiment of the business data migration device of the present invention is proposed.

[0117] In this embodiment, the system overhead includes data priority and storage status; the data allocation module 502 is further configured to allocate the business data to be migrated according to the storage status of the system overhead, and obtain the temporary file to be migrated corresponding to the business data to be migrated; and to extract the temporary file to be migrated according to the data priority and preset business conditions of the system overhead, and obtain the migration detail file corresponding to the business data to be migrated.

[0118] Furthermore, the business data migration device also includes a rule-making module 504, which is used to configure parameters of the business data to be migrated according to the business migration requirements, obtain a blacklist and whitelist corresponding to the migration scope of the business data to be migrated; check the system status of the receiving system in the blacklist and whitelist, and determine whether the check result meets the preset health status; when the check result meets the preset health status, formulate rules for the business data to be migrated according to the business migration requirements to obtain migration data verification rules.

[0119] Accordingly, the business data migration device also includes a rule verification module 505, which is used to verify the temporary file to be migrated according to the migration data verification rules and determine whether the verification result meets the data consistency requirements; when the verification result meets the data consistency requirements, the operation of the data migration module 503 is executed.

[0120] Furthermore, the rule formulation module 504 is also used to extract the data size of the business data to be migrated when the inspection result meets the preset health status; to traverse the preset migration strategy library according to the business migration requirements and the data size to obtain the migration strategy corresponding to the business data to be migrated; and to configure the business data to be migrated with SQL statements according to the business migration requirements to obtain the migration data verification rules.

[0121] Correspondingly, the data allocation module 502 is also used to allocate the business data to be migrated according to the storage situation and the migration strategy, and obtain the temporary file to be migrated corresponding to the business data to be migrated.

[0122] Furthermore, the business data migration device also includes a threshold judgment module 506, which judges the current resource utilization rate of the current system and the receiving system based on the system status, and judges whether the current resource utilization rate is lower than a preset threshold.

[0123] Accordingly, the rule verification module 505 is also used to execute the operation of the data migration module 503 when the verification result meets the data consistency and the current resource utilization rate is lower than the preset threshold.

[0124] Furthermore, after the data migration module 503, the system further includes: the target system reading the number of records to be transferred from the migration details file, comparing the number of records to be transferred with the number of records already transferred from the temporary file to be migrated, and determining whether the comparison results are consistent; when the comparison results are consistent, the target system converts the temporary file to be migrated into data to be imported into the database; the target system performs data verification on the data to be imported into the database and determines whether the verification result meets the data uniqueness requirement; when the verification result meets the data uniqueness requirement, the target system writes the data to be imported into the database file.

[0125] Furthermore, the business data migration device also includes a data update module 507. The data update module 507 is used to, upon receiving data to be imported from the target system, perform a protocol comparison on the data to be imported using a DAP temporary table, and determine whether the data to be imported is consistent with the temporary file to be migrated based on the comparison result; if the data to be imported is inconsistent with the temporary file to be migrated, update the temporary file to be migrated using a data processing tool to obtain an updated temporary file to be migrated; migrate the updated temporary file to be migrated to the target system until the data to be imported is consistent with the updated temporary file to be migrated, at which point the step of migrating the temporary file to be migrated to the target system based on the migration details file is stopped.

[0126] Other embodiments or specific implementations of the business data migration device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0127] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0128] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0130] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A business data migration method, characterized in that, The business data migration method includes: Upon receiving a user's business migration request, generate business data to be migrated based on the business migration request; Based on system overhead, the business data to be migrated is allocated for migration, and a migration detail file and a temporary file to be migrated are obtained for the business data to be migrated. The migration detail file is a detail of the business data to be migrated during the data migration process, including migration batch, migration time and migration size. The temporary file to be migrated is a temporary file obtained by allocating the business data to be migrated according to data priority and storage conditions. The temporary file to be migrated is migrated to the target system according to the migration details file, so that the target system writes the temporary file to be migrated into the database file when the verification result of the temporary file to be migrated meets the data uniqueness requirement. The data uniqueness is used to characterize the situation where the data in the temporary file to be migrated does not duplicate the data in the target system.

2. The business data migration method as described in claim 1, characterized in that, The system overhead includes data priority and storage; The step of allocating the migration data to be migrated based on system overhead, and obtaining the migration detail file and temporary file to be migrated corresponding to the migration data, includes: Based on the storage status of the system overhead, the business data to be migrated is allocated and migrated to obtain the temporary file to be migrated corresponding to the business data to be migrated; Based on the data priority of the system overhead and preset business conditions, the temporary files to be migrated are extracted to obtain the migration detail files corresponding to the business data to be migrated.

3. The business data migration method as described in claim 2, characterized in that, Upon receiving a user's business migration request, after generating the business data to be migrated based on the business migration request, the process further includes: Based on the business migration requirements, the parameters of the business data to be migrated are configured to obtain the blacklist and whitelist corresponding to the migration scope of the business data to be migrated. The system status of the receiving systems in the blacklist and whitelist is checked, and it is determined whether the check result meets the preset health status. When the inspection results meet the preset health status, rules are formulated for the business data to be migrated according to the business migration requirements to obtain migration data verification rules. Accordingly, after extracting the temporary file to be migrated based on the data priority and preset business conditions according to the system overhead, and obtaining the migration detail file corresponding to the business data to be migrated, the process further includes: The temporary file to be migrated is verified according to the migration data verification rules, and the verification result is determined to be consistent with the data consistency. When the verification result meets the data consistency requirement, the step of migrating the temporary file to be migrated to the target system according to the migration details file is executed.

4. The business data migration method as described in claim 3, characterized in that, When the inspection result meets the preset health status, the process involves formulating rules for the business data to be migrated based on the business migration requirements to obtain migration data verification rules, including: When the inspection results meet the preset health status, the data size of the business data to be transplanted is extracted; Based on the business migration requirements and the data scale, the preset migration strategy library is traversed to obtain the migration strategy corresponding to the business data to be migrated. Configure SQL statements on the business data to be migrated according to the business migration requirements to obtain migration data verification rules; Accordingly, the step of allocating the ported business data according to the storage situation based on the system overhead, and obtaining the corresponding temporary porting file for the ported business data, includes: Based on the storage situation and the migration strategy, the business data to be migrated is allocated and migrated to obtain the temporary file to be migrated corresponding to the business data to be migrated.

5. The business data migration method as described in claim 4, characterized in that, Before migrating the temporary file to be migrated to the target system according to the migration details file, the method further includes: The current resource utilization rate of the current system and the receiving system is determined based on the system status, and it is determined whether the current resource utilization rate is lower than a preset threshold. Accordingly, when the verification result meets the data consistency requirement, the step of migrating the temporary file to be migrated to the target system according to the migration details file includes: When the verification result meets the data consistency requirement and the current resource utilization rate is lower than the preset threshold, the step of migrating the temporary file to be migrated to the target system according to the migration details file is executed.

6. The business data migration method as described in claim 5, characterized in that, The target system writes the temporary file to be ported into a database file when the verification result of the temporary file to be ported meets the data uniqueness requirement, including: The target system reads the number of entries to be transmitted from the migration details file, compares the number of entries to be transmitted with the number of entries already transmitted from the temporary file to be migrated, and determines whether the comparison results are consistent. When the comparison results are consistent, the target system converts the temporary file to be ported into data to be stored in the database. The target system performs data verification on the data to be added to the database and determines whether the verification result meets the requirement of data uniqueness. When the verification result meets the requirement of data uniqueness, the target system writes the data to be stored into the database file.

7. The business data migration method as described in claim 6, characterized in that, After migrating the temporary file to be migrated to the target system according to the migration details file, the process further includes: Upon receiving the data to be imported from the target system, a protocol comparison is performed on the data to be imported through the DAP temporary table, and the comparison result is used to determine whether the data to be imported is consistent with the temporary file to be ported. When the data to be imported is inconsistent with the temporary file to be ported, the temporary file to be ported is updated using a data processing tool to obtain the updated temporary file to be ported. The updated temporary file to be migrated is migrated to the target system until the data to be imported is consistent with the updated temporary file to be migrated, at which point the step of migrating the temporary file to be migrated to the target system according to the migration details file is stopped.

8. A business data migration device, characterized in that, The device includes: The data generation module is used to generate business data to be migrated based on the business migration requirements when a user's business migration request is received. The data allocation module is used to allocate the business data to be migrated according to the system overhead, and obtain the migration detail file and the temporary file to be migrated corresponding to the business data to be migrated. The migration detail file is a detail of the business data to be migrated in the data migration process, including migration batch, migration time and migration size. The temporary file to be migrated is a temporary file obtained by allocating the business data to be migrated according to data priority and storage conditions. The data migration module is used to migrate the temporary file to be migrated to the target system according to the migration details file, so that the target system writes the temporary file to be migrated into the database file when the verification result of the temporary file to be migrated meets the data uniqueness requirement. The data uniqueness is used to characterize the situation where the data in the temporary file to be migrated does not duplicate the data in the target system.

9. A business data migration device, characterized in that, The device includes: a memory, a processor, and a business data migration program stored in the memory and executable on the processor, the business data migration program being configured to implement the steps of the business data migration method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a business data migration program, which, when executed by a processor, implements the steps of the business data migration method as described in any one of claims 1 to 7.

Images