Data migration method and device, storage medium, program product and electronic equipment

Abstract

The invention discloses a data migration method and device, a storage medium, a program product and electronic equipment. Relates to the distributed field. The method comprises the following steps: determining a data table to be migrated from a source database; for each data table, a data migration rule corresponding to the data table is determined based on the service type associated with the data table, and the data migration rule at least records a mapping mode of data in the data table; performing data mapping on the data table based on a data migration rule to obtain a target data table; and storing the target data table into the target database. Through the method and the device, the problem of low data migration flexibility caused by migration of total data in a database based on a unified mode in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of distribution, and in particular, to a data migration method, apparatus, storage medium, program product, and electronic device. Background Technique

[0002] The data migration of a database refers to the process of migrating data from one database system to another. This process may involve data transfer between different types of databases or between databases of the same type. Data migration is usually for purposes such as meeting business requirements, technological upgrades, data integration, or system optimization.

[0003] In traditional data migration scenarios, the full amount of database data is usually migrated based on a unified method, that is, the same data migration strategy and rules are used for all data tables. This approach simplifies the data migration process to a certain extent, making the migration process easier to manage and implement. However, as the scale of enterprise-level databases continues to increase and the complexity of business scenarios becomes increasingly high, this one-size-fits-all data migration method gradually exposes its inherent limitations and deficiencies, and there is a problem of low flexibility in data migration.

[0004] In response to the above problems in the related art, no effective solution has been proposed yet. Summary of the Invention

[0005] The main purpose of this application is to provide a data migration method, apparatus, storage medium, program product, and electronic device to solve the problem of low flexibility in data migration in the related art when migrating the full amount of data in a database based on a unified method.

[0006] To achieve the above objective, according to one aspect of this application, a data migration method is provided. The method includes: determining the data tables to be migrated from the source database; for each data table, determining the data migration rule corresponding to the data table based on the business type associated with the data table, where at least the mapping method of the data in the data table is recorded in the data migration rule; performing data mapping on the data table based on the data migration rule to obtain a target data table; storing the target data table in the target database.

[0007] Furthermore, the data migration method further includes: for each field in the data table, determining the mapping method matched by the field from the data migration rule; performing mapping processing on the field value of the field based on the mapping method matched by the field to obtain an updated field value; determining the target data table based on the updated field value.

[0008] Further, the data migration method further includes: after obtaining the target data table, the method further includes: calculating a hash value corresponding to the target data table based on the target identifier of the target data table; grouping the multiple target data tables associated with the source database according to the hash value to obtain multiple data table sets; storing the target data table into the target database, including: storing the target data tables in the multiple data table sets into the target database in parallel.

[0009] Further, the data migration method further includes: after storing the target data table into the target database, for each target data table in the target database, determining a data verification rule corresponding to the target data table based on the business type associated with the target data table, where the data verification rule is at least used to verify the accuracy of the data mapping result; performing data consistency verification on the target data table based on the data verification rule to obtain a verification result.

[0010] Further, the data migration method further includes: for each field in the target data table, determining the verification method matching the field from the data verification rule; performing data consistency verification on the field value of the field based on the verification method matching the field to obtain a verification sub-result matching the field; determining the verification result based on the verification sub-result.

[0011] Further, the data migration method further includes: determining the number of data in the target data table; determining whether the number of data in the target data table is the same as the number of data in the data table associated with the target data table; in the case where the number of data is the same, determining a data verification rule corresponding to the target data table based on the business type associated with the target data table.

[0012] Further, the data migration method further includes: determining whether there is abnormal target data in the target data table; in the case where there is no abnormal target data in the target data table, determining a data verification rule corresponding to the target data table based on the business type associated with the target data table.

[0013] To achieve the above object, according to another aspect of the present application, there is provided a data migration apparatus. The apparatus includes: a first determination module, configured to determine a data table to be migrated from the source database; a second determination module, configured to, for each data table, determine a data migration rule corresponding to the data table based on the business type associated with the data table, where the data migration rule at least records a mapping method of the data in the data table; a first processing module, configured to perform data mapping on the data table based on the data migration rule to obtain a target data table; and a storage module, configured to store the target data table into the target database.

[0014] Further, the first processing module further includes: a first determination sub-module, configured to determine, for each field in the data table, the mapping method matched by the field from the data migration rules; a first processing sub-module, configured to perform mapping processing on the field value of the field based on the mapping method matched by the field to obtain an updated field value; a second determination sub-module, configured to determine the target data table based on the updated field value.

[0015] Further, the data migration device further includes: a calculation module, configured to calculate, based on the target identifier of the target data table, a hash value corresponding to the target data table; a second processing module, configured to perform grouping processing on multiple target data tables associated with the source database according to the hash value to obtain multiple data table sets; the storage module further includes: a storage sub-module, configured to store the target data tables in the multiple data table sets into the target database in parallel.

[0016] Further, the data migration device further includes: a third determination module, configured to determine, for each target data table in the target database, the data verification rule corresponding to the target data table based on the service type associated with the target data table, where the data verification rule is at least used to verify the accuracy of the data mapping result; a verification module, configured to perform data consistency verification on the target data table based on the data verification rule to obtain a verification result.

[0017] Further, the verification module further includes: a third determination sub-module, configured to determine, for each field in the target data table, the verification method matched by the field from the data verification rule; a verification sub-module, configured to perform data consistency verification on the field value of the field based on the verification method matched by the field to obtain a verification sub-result matched by the field; a fourth determination sub-module, configured to determine the verification result based on the verification sub-result.

[0018] Further, the third determination module further includes: a fifth determination sub-module, configured to determine the number of data entries in the target data table; a judgment sub-module, configured to judge whether the number of data entries in the target data table is the same as the number of data entries in the data table associated with the target data table; a sixth determination sub-module, configured to, when the number of data entries is the same, determine the data verification rule corresponding to the target data table based on the service type associated with the target data table.

[0019] Further, the sixth determination sub-module further includes: a judgment unit, configured to judge whether there is abnormal target data in the target data table; a determination unit, configured to, when there is no abnormal target data in the target data table, determine the data verification rule corresponding to the target data table based on the service type associated with the target data table.

[0020] To achieve the above object, according to another aspect of the present application, there is provided a computer-readable storage medium, which includes a stored executable program. When the executable program runs, it controls the device where the computer-readable storage medium is located to execute the above data migration method.

[0021] To achieve the above object, according to another aspect of the present application, there is provided an electronic device, which includes a memory storing an executable program; and a processor for running the program. When the program runs, it executes the above data migration method.

[0022] To achieve the above object, according to another aspect of the present application, there is provided a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, the steps of the above data migration method are implemented.

[0023] In the embodiments provided by the present application, by determining the data migration rule corresponding to a data table based on the business type associated with the data table in the source database, it is achieved to specifically determine the data migration rule corresponding to the data table based on the business type of the data table. By performing data mapping on the data table according to the data migration rule to obtain a target data table and storing the target data table in the target database, it is achieved to perform data migration using a targeted data migration rule, thereby improving the flexibility of data migration and avoiding the limitations of migrating data based on a unified method. The embodiments provided by the present application achieve the purpose of performing data migration based on the data migration rule corresponding to the business type, and achieve the technical effect of improving the flexibility of data migration, and solve the technical problem of low flexibility in data migration in the related art when migrating all data in the database based on a unified method. Description of the Drawings

[0024] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0025] Figure 1 is the hardware structure block diagram of a computer terminal provided by an embodiment of the present application;

[0026] Figure 2 is the flowchart of the data migration method provided by an embodiment of the present application;

[0027] Figure 3 is the schematic diagram of a data migration device provided by an embodiment of the present application;

[0028] Figure 4 is the structure block diagram of an electronic device according to an embodiment of the present application. Detailed Embodiments

[0029] To enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solution in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0031] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) collected in this application are information and data authorized by the user or fully authorized by all parties. And the processing of relevant data, such as collection, storage, use, processing, transmission, provision, disclosure, and application, all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, there is an interface between this system and relevant users or institutions to provide corresponding operation entrances for users to choose to agree or refuse the automated decision-making results; if the user chooses to refuse, the expert decision-making process will be entered.

[0032] Embodiment 1

[0033] According to the embodiments of this application, an embodiment of a data migration method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described here can be executed in an order different from that here.

[0034] The method embodiment provided in the first embodiment of this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing the data migration method is shown. AsFigure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (illustrated as 102a, 102b, ……, 102n in the figure) (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown in, or have a different configuration from Figure 1 shown.

[0035] It should be noted that the above one or more processors 102 and / or other data processing circuits may generally be referred to as "data processing circuits" herein. The data processing circuit may be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is used for processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0036] The memory 104 may be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data migration method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned data migration method. The memory 104 may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely set relative to the processor 102, and these remote memories may be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0039] Under the above operating environment, the present application provides a data migration method as Figure 2 shown. Figure 2 It is a flowchart of the data migration method according to Embodiment 1 of the present application.

[0040] Step S201, determine the data tables to be migrated from the source database.

[0041] Optionally, devices such as electronic devices, application systems, and servers can be used as the execution subject of the present application. In this embodiment, the target processing system is used as the execution subject to execute the data migration method.

[0042] Optionally, the source database refers to the database that stores the data to be migrated, and the data to be migrated is stored in the data tables of the database. At the beginning stage of the data migration project, the target processing system can identify all the data tables that need to be migrated from the source database. The source database may contain hundreds or thousands of data tables, and each data table carries specific business data. Determining the data tables to be migrated is the basis of the data migration process, which ensures the pertinence and effectiveness of the subsequent steps. Operationally, a list of all data tables can be obtained by means of database query or metadata reading, and at the same time, the data tables to be migrated can be selected based on business requirements or migration priorities.

[0043] Step S202, for each data table, determine the data migration rule corresponding to the data table based on the business type associated with the data table, where at least the mapping method of the data in the data table is recorded in the data migration rule.

[0044] Optionally, the target processing system may determine the business type associated with the data table based on information such as the storage location of the data table or the name of the data table. For example, a data table stored in directory 1 may be determined to be of business type A. Another example is that the business type is recorded in the data table name, and the business type in the data table name is determined as the business type associated with the data table.

[0045] Optionally, different business types correspond to different data migration rules, and the matching relationships between different business types and different data migration rules are stored in the target processing system. After determining the data tables to be migrated, for the specific business type of each data table (such as transaction records, customer information, product catalogs, etc.), the corresponding data migration rules can be matched. The data migration rules contain key information such as data type mapping, function conversion, and configuration adjustment to ensure that the data in the source database can be accurately mapped to the corresponding data types and structures in the target database. For example, float type data in the source database may need to be mapped to double precision type in the target database to ensure numerical accuracy. Both float type and double precision type are data types used to store floating-point numbers, and the difference lies in the numerical accuracy. Here, the target database refers to the database used to store the migrated data. This step emphasizes the customization and refinement of data migration, avoiding data mismatch and business inadaptation problems that may be caused by a "one-size-fits-all" migration strategy.

[0046] For example, in the case where the business type is transaction records, transaction records usually contain a large amount of numerical data, such as amounts and transaction quantities. In the source database, these data may use the float type. Due to the strict requirements for numerical accuracy in the financial field, when migrating to the target database, these types can be mapped to double precision to ensure higher accuracy.

[0047] In the case where the business type is customer information, for sensitive information such as ID numbers and phone numbers, additional encryption processing may be required in the target database. The migration rules should include the selection and implementation details of the encryption algorithm.

[0048] Step S203: Perform data mapping on the data table based on the data migration rules to obtain the target data table.

[0049] Optionally, after determining the corresponding data migration rules, the target processing system can, according to these rules, perform data mapping on each piece of data in each data table in the source database, convert the data in the data table into a form that meets the requirements of the target database format and data type, so as to obtain the target data table, that is, the data table under the target database schema. Data mapping is a key technical step, which directly determines the accuracy and effectiveness of the migrated data. By performing data mapping, it can be ensured that under the new database schema, the representation of data and the business processing logic remain consistent.

[0050] Step S204, store the target data table into the target database.

[0051] In some embodiments, the target database can be a distributed database.

[0052] Optionally, after completing the data mapping, the target data table can be stored into the target database. This involves the physical migration and storage optimization of data, including but not limited to parallel data import, selection of storage format, reconstruction of indexes, etc. The process of storing into the target database needs to ensure the consistency, integrity and security of data. At the same time, considering the distributed architecture, high availability and disaster tolerance characteristics of the target database, reasonably allocate data and storage resources to achieve efficient access and management after data migration.

[0053] Since the data in the data tables in the source database may be updated very quickly, the longer the migration process takes, the more new data will come in. The new data also needs to be migrated again, and there may be more new data coming in during this process. Therefore, it is particularly crucial to reduce the migration time and ensure that incremental data can be synchronized during the migration process. Therefore, in an optional embodiment, different tables can be monitored. When it is monitored that a certain piece of data in a data table has been migrated and there are changes made in the table of the source database, it can be discovered in a timely manner and automatically synchronized to the target database. Thus, the data consistency during the transmission process can be ensured.

[0054] In some embodiments, after storing the target data table into the target database, the target processing system can perform performance testing on the target database to ensure that the query performance and response time meet the business requirements. Performance testing includes but not limited to: (1) Concurrent query: Simulate multiple users accessing the database simultaneously; (2) Large data volume query: Test the query performance of large data tables; (3) Complex query: Test the performance of complex queries.

[0055] In the embodiments provided in this application, by determining the data migration rules corresponding to a data table based on the business type associated with the data table in the source database, it is achieved to specifically determine the data migration rules corresponding to the data table based on the business type of the data table. By performing data mapping on the data table based on the data migration rules to obtain a target data table and storing the target data table in the target database, it is achieved to perform data migration using targeted data migration rules, thereby improving the flexibility of data migration and avoiding the limitations of migrating data based on a unified method. The embodiments provided in this application achieve the purpose of performing data migration based on the data migration rules corresponding to the business type, realize the technical effect of improving the flexibility of data migration, and solve the technical problem of low flexibility in data migration in the related art when migrating all data in the database based on a unified method.

[0056] Optionally, in the data migration method provided in the embodiments of this application, performing data mapping on a data table based on the data migration rules to obtain a target data table includes: for each field in the data table, determining the mapping method matching the field from the data migration rules; performing mapping processing on the field value of the field based on the mapping method matching the field to obtain an updated field value; and determining the target data table based on the updated field value.

[0057] Optionally, the data migration rules include the mapping methods matching each field. For example, for fields of date and time types, the rule may indicate mapping the datetime data type of the field in the source database to the timestamp without time zone data type in the target database, or mapping the timestamp data type of the field in the source database to the timestamp with time zone data type. The mapping methods may also include conversion rules for fields of numerical types, character types, auto-increment columns, etc., to ensure that the migration of each field can meet the technical requirements of the target database. For example, for fields of numerical types, the rule may indicate mapping data types such as tinyint, int, bigint, int, bigint, float, double, etc. of the field in the source database to data types such as smallint, integer, bigint, real, double precision, etc. in the target database. For fields of auto-increment column types, the rule may indicate mapping the AUTO_INCREMENT data type of the field in the source database to data types such as serial or bigserial in the target database. For fields of character types, the rule may indicate mapping data types such as string or longtext of the field in the source database to data types such as text in the target database.

[0058] Optionally, the target processing system can determine the mapping method for field matching based on the field name of the field or the type of the field value. After determining the mapping method for each field, the next step is to perform specific conversion processing on the field value. This usually involves data type conversion, adjustment of numerical precision, conversion of string encoding, etc. For example, for a field of float type in the source database, it may be necessary to convert it to double precision in the target database to avoid precision loss; for a field of YEAR type in the source database, it may be necessary to map it to SMALLINT type in the target database and adjust the data format to conform to the storage rules of the target database. This process may also involve using specific field conversion functions or scripts to process the field value to ensure that the data can be correctly interpreted and stored in the target database after conversion.

[0059] In some embodiments, after completing the mapping process of the field value, the updated field value can be placed into a new table to obtain the target data table, or the fields in the data table can be directly modified to obtain the target data table.

[0060] In some embodiments, after completing the mapping process of the field value, the target data table can also be constructed based on the updated field value and the table structure and index design of the target database. This step may not only include placing the modified field value into a new table, but also involve optimizing and adjusting the structure and configuration of the table according to business requirements and target database characteristics. For example, according to the data volume and access pattern, designing an appropriate partitioning strategy for the table; creating indexes for frequently queried fields to achieve balanced data distribution and improve query response speed. Finally, the target data table should fully comply with the data migration rules and the technical requirements of the target database to ensure business continuity and data integrity after data migration.

[0061] It should be noted that through the above process, precise mapping at the field level is achieved, enabling each field to meet the technical requirements and business scenarios of the target database. This method effectively avoids problems such as data type mismatch, numerical precision loss, and character encoding errors, ensuring high quality and business applicability after data migration.

[0062] Optionally, in the data migration method provided in the embodiments of the present application, after obtaining the target data table, the method further includes: calculating a hash value corresponding to the target data table based on the target identifier of the target data table; performing grouping processing on multiple target data tables associated with the source database according to the hash value to obtain multiple data table sets; storing the target data table into the target database, including: storing the target data tables in multiple data table sets into the target database in parallel.

[0063] Optionally, due to the large number of data tables in the source database and the complex business, there may or may not be associations between various data tables. Therefore, the data tables can be grouped to reduce the performance consumption during the migration process and subsequent table query process, and avoid performance bottlenecks.

[0064] Optionally, during the data migration process, for each obtained target data table, a hash value can be calculated based on its target identifier (usually the table name or other unique identifier). The calculation of this hash value can utilize hash functions such as MD5, SHA-1, etc. to convert the target identifier into a fixed-length numerical value.

[0065] Optionally, by analyzing the hash values of the target data tables, they can be grouped into different sets. The grouping principle is usually based on the distribution characteristics of the hash values. For example, the target data tables with hash values within a certain range can be grouped into one group to form multiple data table sets. Another example is that the target data tables can be grouped based on the modulo result of the hash value to obtain multiple data table sets. After grouping, the number and size of the target data tables in each set are basically balanced, which can avoid unbalanced loads during parallel storage.

[0066] Optionally, the target database can be a distributed database. After grouping, the parallel storage capability of the target database can be utilized to store the target data tables in multiple data table sets into the target database simultaneously. This process usually involves the collaborative work of multiple threads or distributed nodes, and each thread or node is responsible for the storage operation of one or more data table sets. Through parallel storage, the hardware resources can be fully utilized, the total time of data migration can be reduced, and the consistency and integrity of the data can be maintained.

[0067] It should be noted that by introducing the hash value calculation and parallel storage strategy, the efficiency and performance of the data migration method are further optimized. It effectively considers the parallel processing and load balancing during the data migration process, and is especially suitable for handling large-scale database migration scenarios. The calculation of the hash value ensures the uniform distribution of the data, reduces the bottlenecks and delays during the storage process, while parallel storage makes full use of the distributed advantages of the target database and significantly improves the speed of data migration. This method can not only accelerate the completion time of data migration, but also maintain the high quality and consistency of the data during the migration process, providing strong technical support for the rapid migration and seamless docking of large databases. Especially in business environments that require high availability and low latency, its effect is more significant.

[0068] Optionally, in the data migration method provided in the embodiments of the present application, after storing the target data table in the target database, the method further includes: for each target data table in the target database, determining a data verification rule corresponding to the target data table based on the business type associated with the target data table, where the data verification rule is at least used to verify the accuracy of the data mapping result; performing data consistency verification on the target data table based on the data verification rule to obtain a verification result.

[0069] Optionally, different business types match different data verification rules. The matching relationship between different business types and different data verification rules can be stored in the target processing system. The data verification rule is a detailed inspection process and standard used to verify whether the data mapping result is accurate, meets the business requirements and data integrity requirements. The data verification rule at least includes the verification method for each field, and the verification method can be used to verify the type, length, content, etc. of the field. For example, confirming whether the precision of the numerical field is correct, whether the encoding of the string field is consistent, whether the format of the date and time field is correct, whether the digital encryption is accurate, etc. In the financial business, if the field value of the transaction amount field in the source database matches the account balance, this consistency should also be maintained after migration. This may involve complex business logic verification, such as checking the matching of transaction records and account balances, or ensuring that all foreign key relationships are still valid after migration.

[0070] For example, for the transaction record table, the verification rules may include checking whether the amount of each transaction is correctly mapped, whether the format of the date and time field is correct, and whether the transaction status conforms to the business logic. For the customer information table, confirm whether sensitive information (such as ID number, phone number) has been encrypted or desensitized according to the established rules, and check whether string fields such as customer name and address are correctly mapped to the text type and the character sets are compatible.

[0071] Optionally, after migrating the data to the target database, for each target data table in the target database, the corresponding data verification rule can be determined based on its business type, and then data consistency verification is performed on the target data table based on the data verification rule to obtain a verification result. Among them, the verification result is used to indicate whether the target data table meets the data consistency requirements.

[0072] Optionally, for such large-scale data operations, corresponding batch inspection tools can be set according to the business scenarios of different tables, and specific inspection logics can be encapsulated in the batch to improve the efficiency of obtaining the verification result. For the tables that fail the inspection, an attempt can be made to re-perform data migration and synchronization work, and based on the inspection results, determine which field has problems and what problems have occurred. The inspection results should also include the location of the failure to facilitate program error analysis.

[0073] It should be noted that by verifying the target data table based on targeted data verification rules, the accuracy of the verification result can be improved, thereby enhancing the reliability of data migration.

[0074] Optionally, in the data migration method provided in the embodiments of this application, data consistency verification is performed on the target data table based on the data verification rules to obtain a verification result, including: for each field in the target data table, determining the verification method matching the field from the data verification rules; performing data consistency verification on the field value of the field based on the verification method matching the field to obtain the verification sub-result matching the field; and determining the verification result based on the verification sub-result.

[0075] Optionally, the data verification rules include verification methods matching different types of fields. For example, for date and time fields, the rules may indicate using timestamp comparison to verify the accuracy of the data; for numerical fields, it may involve checking the numerical range or verifying the precision; and for character fields, the verification method may focus on the integrity of the content and the correctness of the format.

[0076] Optionally, the target processing system can determine the verification method matching the field according to the field name of the field or the type of the field value. After determining the verification method for each field, data consistency verification can be performed. This involves reading the field value from the target database and then checking according to the preset verification method to obtain the verification sub-result matching the field. For example, for a field using the double precision type, the verification method may include checking whether the numerical value overflows and whether the precision is lost. The verification sub-result is used to indicate whether the field meets the data consistency requirements.

[0077] After determining the verification sub-results of all fields in the target data table, the verification result is determined based on the verification sub-results. For example, if all verification sub-results indicate that the fields meet the data consistency requirements, it is determined that the verification result indicates that the target data table meets the data consistency requirements; if there is a verification sub-result indicating that the field does not meet the data consistency requirements, it is determined that the verification result indicates that the target data table does not meet the data consistency requirements.

[0078] It should be noted that through the above process, targeted verification of each field in the target data table is achieved, thereby improving the coverage and effectiveness of data verification.

[0079] Optionally, in the data migration method provided in the embodiments of the present application, determining the data verification rule corresponding to the target data table based on the service type associated with the target data table includes: determining the number of data records in the target data table; determining whether the number of data records in the target data table is the same as the number of data records in the data table associated with the target data table; and in the case where the number of data records is the same, determining the data verification rule corresponding to the target data table based on the service type associated with the target data table.

[0080] Optionally, before verifying the data using the data verification rule, a basic verification method can be used to perform consistency verification on the data.

[0081] Optionally, the number of data records in the target data table can be obtained by querying in the target database using a database query statement. The data table associated with the target data table refers to the data table used to update the target data table, and the number of data records in this data table can be recorded by the target processing system during the data migration operation on the data table.

[0082] Optionally, the target processing system can determine whether the number of data records in the target data table is the same as the number of data records in the data table associated with the target data table. Thus, in the case where the number of data records is the same, it is determined that the target data table passes the preliminary data consistency verification. In this case, the data verification rule corresponding to the target data table can be determined based on the service type associated with the target data table to perform a more fine-grained consistency verification. Conversely, in the case where the number of data records is different, it is determined that the verification result indicates that the target data table does not meet the data consistency requirement.

[0083] It should be noted that through the above process, the preliminary verification of the migrated data is first realized, and in the case where the preliminary verification passes, a more fine-grained verification is performed. Thus, on the one hand, the verification accuracy can be improved, and on the other hand, the secondary verification of the data that fails the preliminary verification can be avoided, thereby improving the data verification efficiency.

[0084] Optionally, in the data migration method provided in the embodiments of the present application, determining the data verification rule corresponding to the target data table based on the service type associated with the target data table includes: determining whether there is abnormal target data in the target data table; and in the case where there is no abnormal target data in the target data table, determining the data verification rule corresponding to the target data table based on the service type associated with the target data table.

[0085] Optionally, in the case where the number of data records is the same, the target processing system can first perform another preliminary verification, and thus, in the case where the other preliminary verification also passes, the verification is performed based on the data verification rule.

[0086] Optionally, the target data of the exception may refer to the data with null values in the target data table. That is, the target processing system can detect whether there is data containing null values in the target data table. If so, the data is determined as the target data with exceptions. If there is no data containing null values, it is determined that there is no target data with exceptions in the target data table.

[0087] Optionally, in the case where there is no target data with exceptions in the target data table, it is determined that the target data table passes another preliminary verification, and thus the data verification rules corresponding to the target data table are determined based on the business type associated with the target data table.

[0088] Optionally, in the case where there is target data with exceptions in the target data table, it is determined that the target data table fails another preliminary verification. In this case, it is determined that the verification result indicates that the target data table does not meet the data consistency requirement.

[0089] It should be noted that through the above method, on the one hand, the richness of data verification is improved, and the verification accuracy can be further improved. On the other hand, the secondary verification of the data that fails the preliminary verification can be avoided, thus improving the data verification efficiency.

[0090] It can be seen that the embodiments provided in the present application achieve the purpose of data migration based on the data migration rules corresponding to the business type, realize the technical effect of improving the flexibility of data migration, and solve the technical problem of low flexibility of data migration in the related art when migrating all the data in the database based on a unified method.

[0091] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0092] Embodiment 2

[0093] The embodiment of the present application also provides a data migration device. It should be noted that the data migration device in the embodiment of the present application can be used to execute the data migration method provided in the embodiment of the present application. The data migration device provided in the embodiment of the present application is introduced below.

[0094] According to the embodiment of the present application, there is also provided a device for implementing the above data migration method, as Figure 3 shown, the device includes:

[0095] The first determination module 301 is used to determine the data table to be migrated from the source database;

[0096] A second determination module 302, configured to determine, for each data table, a data migration rule corresponding to the data table based on the service type associated with the data table, where at least a mapping method of the data in the data table is recorded in the data migration rule;

[0097] A first processing module 303, configured to perform data mapping on the data table based on the data migration rule to obtain a target data table;

[0098] A storage module 304, configured to store the target data table into a target database.

[0099] In the embodiments provided in the present application, by determining the data migration rule corresponding to the data table based on the service type associated with the data table in the source database, it is realized to specifically determine the data migration rule corresponding to the data table based on the service type of the data table. By performing data mapping on the data table based on the data migration rule to obtain a target data table and storing the target data table into the target database, it is realized to perform data migration using a targeted data migration rule, thereby improving the flexibility of data migration and avoiding the limitations of migrating data based on a unified method. The embodiments provided in the present application achieve the purpose of performing data migration based on the data migration rule corresponding to the service type, realize the technical effect of improving the flexibility of data migration, and solve the technical problem of low flexibility of data migration in the related art when migrating all the data in the database based on a unified method.

[0100] Optionally, in the data migration device provided in the embodiments of the present application, the first processing module further includes: a first determination sub-module, configured to determine, for each field in the data table, the mapping method matched by the field from the data migration rule; a first processing sub-module, configured to perform mapping processing on the field value of the field based on the mapping method matched by the field to obtain an updated field value; a second determination sub-module, configured to determine the target data table based on the updated field value.

[0101] Optionally, in the data migration device provided in the embodiments of the present application, the data migration device further includes: a calculation module, configured to calculate a hash value corresponding to the target data table based on the target identifier of the target data table; a second processing module, configured to perform grouping processing on multiple target data tables associated with the source database according to the hash value to obtain multiple data table sets; the storage module further includes: a storage sub-module, configured to store the target data tables in the multiple data table sets into the target database in parallel.

[0102] Optionally, in the data migration device provided in the embodiments of the present application, the data migration device further includes: a third determination module, configured to determine, for each target data table in the target database, a data verification rule corresponding to the target data table based on the service type associated with the target data table, where the data verification rule is at least used to verify the accuracy of the data mapping result; a verification module, configured to perform data consistency verification on the target data table based on the data verification rule to obtain a verification result.

[0103] Optionally, in the data migration device provided in the embodiments of the present application, the verification module further includes: a third determination sub-module, configured to determine, for each field in the target data table, a verification method matching the field from the data verification rule; a verification sub-module, configured to perform data consistency verification on the field value of the field based on the verification method matching the field to obtain a verification sub-result matching the field; a fourth determination sub-module, configured to determine the verification result based on the verification sub-result.

[0104] Optionally, in the data migration device provided in the embodiments of the present application, the third determination module further includes: a fifth determination sub-module, configured to determine the number of data records in the target data table; a judgment sub-module, configured to judge whether the number of data records in the target data table is the same as the number of data records in the data table associated with the target data table; a sixth determination sub-module, configured to, when the number of data records is the same, determine a data verification rule corresponding to the target data table based on the service type associated with the target data table.

[0105] Optionally, in the data migration device provided in the embodiments of the present application, the sixth determination sub-module further includes: a judgment unit, configured to judge whether there is abnormal target data in the target data table; a determination unit, configured to, when there is no abnormal target data in the target data table, determine a data verification rule corresponding to the target data table based on the service type associated with the target data table.

[0106] It should be noted here that the above first determination module 301, second determination module 302, first processing module 303, and storage module 304 correspond to steps S201 to S204 in Embodiment 1. The functions of the four modules and the corresponding steps are the same in terms of implementation examples and application scenarios, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules or units may be hardware components or software components stored in a memory (for example, memory 104) and processed by one or more processors (for example, processors 102a, 102b,..., 102n). The above modules may also be part of the device and can run in the computer terminal 10 provided in Embodiment 1.

[0107] Embodiment 3

[0108] Embodiments of the present application may provide an electronic device. Figure 4It is a block diagram of a structure of an electronic device according to an embodiment of the present application. As Figure 4 shown, the electronic device may include: one or more ( Figure 4 only one is shown in the figure) processors 1002, a memory 1004, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0109] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, to implement the above-mentioned methods. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely provided relative to the processor, and these remote memories may be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0110] The processor may call the information and application programs stored in the memory through a transmission device to perform the following steps: determine the data table to be migrated from the source database; for each data table, determine the data migration rule corresponding to the data table based on the service type associated with the data table, wherein at least the mapping method of the data in the data table is recorded in the data migration rule; perform data mapping on the data table based on the data migration rule to obtain a target data table; store the target data table in the target database.

[0111] The processor may also call the information and application programs stored in the memory through a transmission device to perform the following steps: for each field in the data table, determine the mapping method matching the field from the data migration rule; perform mapping processing on the field value of the field based on the mapping method matching the field to obtain an updated field value; determine the target data table based on the updated field value.

[0112] The processor may also call the information and application programs stored in the memory through a transmission device to perform the following steps: for each field in the data table, determine the mapping method matching the field from the data migration rule; perform mapping processing on the field value of the field based on the mapping method matching the field to obtain an updated field value; determine the target data table based on the updated field value.

[0113] The processor can also call the information and application programs stored in the memory through the transmission device to execute the following steps: After storing the target data table into the target database, for each target data table in the target database, determine the data verification rule corresponding to the target data table based on the service type associated with the target data table, and the data verification rule is at least used to verify the accuracy of the data mapping result; perform data consistency verification on the target data table based on the data verification rule to obtain a verification result.

[0114] The processor can also call the information and application programs stored in the memory through the transmission device to execute the following steps: For each field in the target data table, determine the verification method matching the field from the data verification rule; perform data consistency verification on the field value of the field based on the verification method matching the field to obtain the verification sub-result matching the field; determine the verification result based on the verification sub-result.

[0115] The processor can also call the information and application programs stored in the memory through the transmission device to execute the following steps: Determine the number of data records in the target data table; determine whether the number of data records in the target data table is the same as the number of data records in the data table associated with the target data table; in the case where the number of data records is the same, determine the data verification rule corresponding to the target data table based on the service type associated with the target data table.

[0116] The processor can also call the information and application programs stored in the memory through the transmission device to execute the following steps: Determine whether there is abnormal target data in the target data table; in the case where there is no abnormal target data in the target data table, determine the data verification rule corresponding to the target data table based on the service type associated with the target data table.

[0117] In the embodiments provided in the present application, by determining the data migration rule corresponding to the data table based on the service type associated with the data table in the source database, it is realized to specifically determine the data migration rule corresponding to the data table based on the service type of the data table. By performing data mapping on the data table based on the data migration rule to obtain the target data table and storing the target data table into the target database, it is realized to perform data migration using a targeted data migration rule, thereby improving the flexibility of data migration and avoiding the limitations of migrating data based on a unified method. The embodiments provided in the present application achieve the purpose of performing data migration based on the data migration rule corresponding to the service type, realize the technical effect of improving the flexibility of data migration, and solve the technical problem of low flexibility of data migration in the related art when migrating all the data in the database based on a unified method.

[0118] Those of ordinary skill in the art can understand, Figure 4The structure shown is only schematic, and the electronic device can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a personal digital assistant, and terminal devices such as Mobile Internet Devices (MID) and PAD. Figure 4 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device may further include more or fewer components (such as a network interface, a display device, etc.) than those shown Figure 4 in the figure, or have a different configuration from that shown Figure 4 in the figure.

[0119] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware of the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disc, etc.

[0120] Embodiment 4

[0121] An embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the above storage medium may be used to store the program code executed by the data migration method provided in the first embodiment above.

[0122] Optionally, in this embodiment, the above storage medium may be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0123] The present application also provides a computer program product, which is adapted to execute a program for the steps of the data migration method when executed on a data processing device.

[0124] The serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments.

[0125] In the above embodiments of the present application, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0126] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0127] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0129] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0130] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A data migration method, characterized in that: include: Determine the data tables to be migrated from the source database; For each data table, determining a data migration rule corresponding to the data table based on a business type associated with the data table, wherein the data migration rule records at least a mapping method of data in the data table; Performing data mapping on the data table based on the data migration rule to obtain a target data table; The target data table is stored in a target database.

2. The method according to claim 1, characterized in that Performing data mapping on the data table based on the data migration rule to obtain a target data table includes: For each field in the data table, determine a mapping method matching the field from the data migration rule; Mapping the field value of the field based on the mapping method of the field match to obtain an updated field value; The target data table is determined based on the updated field value.

3. The method according to claim 1, characterized in that After obtaining the target data table, the method further includes: Based on the target identifier of the target data table, a hash value corresponding to the target data table is calculated; Grouping multiple target data tables associated with the source database according to the hash value to obtain multiple data table sets; Storing the target data table in a target database includes: The target data tables in the plurality of data table sets are stored in parallel in the target database.

4. The method according to claim 1, characterized in that After storing the target data table in the target database, the method further includes: For each target data table in the target database, determining a data verification rule corresponding to the target data table based on a business type associated with the target data table, wherein the data verification rule is at least used to verify the accuracy of a data mapping result; A data consistency check is performed on the target data table based on the data verification rule to obtain a verification result.

5. The method according to claim 4, characterized in that Performing data consistency verification on the target data table based on the data verification rule to obtain a verification result includes: For each field in the target data table, determining a verification method for matching the field from the data verification rule; Performing data consistency verification on the field value of the field based on the verification method of the field match to obtain a verification sub-result of the field match; The check result is determined based on the syndrome result.

6. The method according to claim 4, characterized in that Determining a data verification rule corresponding to the target data table based on the business type associated with the target data table includes: Determine the number of data entries in the target data table; Determine whether the number of data entries in the target data table is the same as the number of data entries in the data table associated with the target data table; When the number of data items is the same, the data verification rule corresponding to the target data table is determined based on the business type associated with the target data table.

7. The method according to claim 6, characterized in that Determining a data verification rule corresponding to the target data table based on the business type associated with the target data table includes: Determine whether there is abnormal target data in the target data table; In a case where there is no abnormal target data in the target data table, a data verification rule corresponding to the target data table is determined based on a business type associated with the target data table.

8. A data migration device, characterized in that: include: A first determination module, used to determine a data table to be migrated from a source database; A second determination module is configured to determine, for each data table, a data migration rule corresponding to the data table based on a business type associated with the data table, wherein the data migration rule records at least a mapping method of data in the data table; A first processing module, configured to perform data mapping on the data table based on the data migration rule to obtain a target data table; A storage module is used to store the target data table in a target database.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the data migration method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the data migration method described in any one of claims 1 to 7 when running.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the data migration method described in any one of claims 1 to 7 are implemented.

Images