Data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a data processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining migration configuration information of a source database; according to the migration configuration information, determining a to-be-migrated data table in the source database, and migrating the to-be-migrated data table to a target database; obtaining a newly added data table corresponding to the data table to be migrated in the target database, and comparing the newly added data table with the data table to be migrated to obtain a comparison result; and under the condition that the comparison result is that there is a difference, performing exception processing on the newly added data table. Dynamic data identification and processing are carried out in the migration process, the flexibility of the migration process is greatly improved, migration comparison of the hotspot data is automatically processed in the migration process, and therefore data backlog pressure increase caused by frequent change of the hotspot data is reduced, the efficiency of the migration process is improved, and the migration efficiency of the hotspot data is improved. And the cost and risk of manual intervention are also reduced.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] With the continuous increase in data volume and organizations attaching great importance to data security and integrity, the data migration and comparison industry is ushering in a golden period of rapid development. In this context, the importance of data migration and comparison tools has become increasingly prominent. They can ensure the accuracy and consistency of the data migration process, significantly reduce the risk of human errors, improve work efficiency, and effectively avoid data risks, thereby ensuring the smooth progress of the data migration process.

[0003] However, the current data migration and comparison industry also faces many challenges and shortcomings. The first problem is that with the surge in data volume and the increasing complexity of data management, data migration and comparison tasks have become increasingly arduous and time-consuming; secondly, given the diversity of data sources and differences in data formats, data migration and comparison tools may encounter compatibility and consistency problems when processing different types of data; in addition, high costs and technical barriers have also become key factors restricting some small and medium-sized enterprises and organizations from using data migration and comparison tools, thereby limiting their development pace.

[0004] Therefore, there is an urgent need to develop a data processing method, device, electronic device and storage medium to solve one or more of the above-mentioned problems. Summary of the invention

[0005] In view of this, in order to solve the above technical problems or part of the technical problems, an embodiment of the present invention provides a data processing method, device, electronic device and storage medium.

[0006] In a first aspect, the present application provides a data processing method, the method comprising:

[0007] Get the migration configuration information of the source database;

[0008] Determine the data table to be migrated in the source database according to the migration configuration information, and migrate the data table to be migrated to the target database;

[0009] Acquire a newly added data table in the target database corresponding to the data table to be migrated, and compare the newly added data table with the data table to be migrated to obtain a comparison result;

[0010] When the comparison result shows that there is a difference, an exception process is performed on the newly added data table, and the newly added data table is updated according to the exception process result.

[0011] In a possible implementation, determining the data table to be migrated in the source database according to the migration configuration information includes: determining a dynamic data table and a hot data table in the source database as the data table to be migrated in the source database according to the migration configuration information.

[0012] In one possible implementation, the migration configuration information includes the priority of each data table in the source database and the historical pulling information of each data table; determining the hot data table in the source database based on the migration configuration information includes: determining the data table with the highest priority in the source database as the hot data table based on the priority of each data table in the source database included in the migration configuration information; judging whether there is a pulling anomaly in the historical pulling information of the hot data table; and in the event of a pulling anomaly, adjusting the priority of the hot data table to re-determine the hot data table in the source database based on the adjusted priority.

[0013] In a possible implementation, comparing the newly added data table with the data table to be migrated to obtain a comparison result includes: obtaining a first number of data stored in the newly added data table, and a second number of data stored in the data table to be migrated; determining a first comparison result based on the first number of data and the second number of data; obtaining first data information of the data stored in the newly added data table, and second data information of the data stored in the data table to be migrated; and determining a second comparison result based on the first data information and the second data information.

[0014] In a possible implementation, when the comparison result is different, the newly added data table is subjected to exception processing, including: when both the first comparison result and the second comparison result are different, re-executing the migration of the data table to be migrated; when the first comparison result is no difference and the second comparison result is different, obtaining the difference data in the newly added data table and deleting the difference data; obtaining the first data corresponding to the difference data in the data table to be migrated; and re-migrating the first data to the target database.

[0015] In a possible implementation, after migrating the data table to be migrated to the target database, the method further includes: obtaining the migration status of the hot data table in the data table to be migrated; and adjusting the priority of the hot data table when the migration status is that all migrations are successful.

[0016] In one possible implementation, the method further includes: determining whether the dynamic data table and the hot data table are whitelist data tables of the source database; if so, executing the dynamic data table and the hot data table as data tables to be migrated in the source database, otherwise, re-determining the dynamic data table and the hot data table from the source database.

[0017] In a second aspect, the present application provides a data processing device, the device comprising:

[0018] The acquisition module is used to obtain the migration configuration information of the source database;

[0019] A determination module, configured to determine a data table to be migrated in the source database according to the migration configuration information, and migrate the data table to be migrated to the target database;

[0020] A comparison module, used for obtaining a newly added data table in the target database corresponding to the data table to be migrated, and comparing the newly added data table with the data table to be migrated to obtain a comparison result;

[0021] The exception handling module is used to perform exception handling on the newly added data table when the comparison result shows that there is a difference, and update the newly added data table according to the exception handling result.

[0022] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method described in any one of the embodiments of the first aspect when executing the computer program.

[0023] In a fourth aspect, the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data processing method described in any embodiment of the first aspect are implemented.

[0024] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages over the prior art: the data processing method provided by the embodiment of the present application performs dynamic data identification and processing during the migration process, and can automatically exclude hotspot data or other dynamic data tables once they are found to cause the migration comparison task to fail, thereby greatly improving the flexibility and robustness of the migration process; and, the migration comparison of hotspot data is automatically processed during the migration process, thereby reducing the increased data backlog pressure caused by the frequent changes of hotspot data, which not only improves the efficiency of the migration process, but also reduces the cost and risk of manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0028] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0029] Figure 2 A flowchart of another data processing method provided in an embodiment of the present application;

[0030] Figure 3 A hotspot data identification flow chart provided for an embodiment of the present application;

[0031] Figure 4 A dynamic table recognition flow chart provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of the steps of a data processing method provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0034] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0036] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0037] In order to solve the problem that data migration comparison becomes more difficult and time-consuming with the increase of data volume and the complexity of data management in the data migration process in the prior art, due to the diversity of data sources and different data formats, data migration comparison tools may encounter some compatibility and consistency problems when processing different types of data. The present application provides a data processing method, device, electronic device and storage medium, which are committed to solving the problem of inefficiency caused by frequent changes in the amount and type of data on the source side in data migration comparison, and dynamically solve the comparison efficiency of phased hotspot table data, and meet the needs of new migration comparison tasks, and can intelligently identify the data on the source side, determine the hotspot table data and non-hotspot table data, balance the weights, and ensure the focused comparison of the hotspot table data; and in the process, it can intelligently identify the hotspot table and non-hotspot table data, dynamically update the hotspot data storage structure, and ensure that the comparison can balance the comparison weight when different hotspot tables on the source side are frequently changed; in addition, for the problem that the source data table structure does not match the target end table structure, or the migration whitelist configuration is missing or wrong, the method effectively solves the problem through a dynamic table migration comparison mechanism, further improving the ease of use and practicality of the system.

[0038] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method specifically includes:

[0039] S101, obtaining migration configuration information of the source database;

[0040] In this embodiment, metadata information of the migration comparison task is obtained by obtaining the types of both ends of the database, the configuration of the blacklist and whitelist, and other special configurations to help parse the details of the migration comparison task.

[0041] The advantage of this acquisition method is that it provides a comprehensive understanding of the database, including its structure and configuration, so as to better understand the requirements and characteristics of the data migration and comparison task. This detailed information can help optimize the comparison algorithm and improve the accuracy and efficiency of the comparison. In addition, the acquisition of metadata information can also help identify potential risks and problems, such as data type mismatches or configuration errors, so that timely adjustments and processing can be made during the migration and comparison process; and the use of a unified metadata processor can reduce parsing time without considering the consumption of additional time and resources, and the encryption of sensitive information also ensures the security of the data.

[0042] S102: Determine the data table to be migrated in the source database according to the migration configuration information, and migrate the data table to be migrated to the target database;

[0043] Migration configuration information is the basis of database migration and contains all the guidance information required during the migration process; such guidance information includes but is not limited to detailed information of the source and target databases, data table selection criteria, migration strategy and plan information;

[0044] Among them, detailed information of the source database and the target database includes database type (such as MySQL, Oracle, SQLServer, etc.), version, IP address, port number, user name and password, etc. This information is the basis for establishing database connection; data table selection criteria are used to clarify which data tables need to be migrated, usually based on business needs, data importance, access frequency, compliance requirements and other factors. For example, for an e-commerce platform, core tables such as user information, order data, and product information are undoubtedly the focus of migration; migration strategy and plan information, including migration time window, downtime budget, data synchronization method (full migration, incremental migration), error handling mechanism, etc. A reasonable migration strategy can minimize the impact on the business.

[0045] After obtaining the migration configuration information, identify the data tables to be migrated in the source database, use database management tools or scripts to analyze the metadata of the source database, determine the structure, size, index status, dependencies, etc. of each data table, evaluate the migration difficulty based on the determined information, execute the migration plan, and then migrate the data tables to be migrated to the target database according to the migration plan.

[0046] It should be noted that during the migration of the data table to be migrated, the following three steps may be performed but are not limited to:

[0047] Step 1: Export and import data. Select appropriate data export tools (such as mysqldump, expdp, etc.) and import tools (such as LOAD DATA INFILE, impdp, etc.) based on the migration strategy, export the data tables in the source database as files, and import them into the target database.

[0048] Step 2: Incremental synchronization and verification. For scenarios that require incremental migration, you should set up regular data synchronization tasks to ensure that newly generated data can be synchronized to the target database in a timely manner. At the same time, use data verification tools (such as checksum, MD5 verification, etc.) to verify the migrated data to ensure data integrity and consistency.

[0049] Step 3: Performance tuning: After the migration is completed, the target database is tuned for performance, including adjusting index strategies, optimizing query statements, configuring appropriate cache strategies, etc. This helps improve the access speed and responsiveness of the database.

[0050] S103, obtaining a newly added data table in the target database corresponding to the data table to be migrated, and comparing the newly added data table with the data table to be migrated to obtain a comparison result;

[0051] In this embodiment, after the newly added data tables are obtained, the next step is to compare the data table structures and data contents of the newly added data tables with the data tables to be migrated to obtain comparison results.

[0052] Specifically, the data table structure comparison process includes: first, comparing the structural differences between the newly added data table and the data table to be migrated, including table name, field name, field type, primary key, foreign key, etc. Through structural comparison, you can preliminarily determine the similarities and differences between the two data tables.

[0053] The data content comparison process includes: based on the structure comparison, further comparing the data content in the two data tables, which can be achieved by writing SQL query statements and using SQL operators such as JOIN and EXCEPT.

[0054] S104: When the comparison result shows that there is a difference, perform exception processing on the newly added data table, and update the newly added data table according to the exception processing result.

[0055] In this embodiment, after identifying significant differences, an in-depth analysis is performed on the abnormal data in the newly added data table; including checking the specific values, distribution characteristics, causes, etc. of the abnormal data, and then formulating appropriate processing strategies for the identified abnormal data based on the viewing results; specifically including data correction, data deletion, data interpolation and other methods.

[0056] For example, obvious data entry errors are corrected directly; missing data due to equipment failure is repaired based on the trend of adjacent data; and abnormal data whose cause cannot be determined or repaired is deleted and re-migrated to avoid affecting the analysis results.

[0057] The data processing method provided in the embodiment of the present application performs dynamic data identification and processing during the migration process. Once hotspot data or other dynamic data tables are found to cause the migration comparison task to fail, they can be automatically excluded, greatly improving the flexibility and robustness of the migration process; and, the migration comparison of hotspot data is automatically processed during the migration process, thereby reducing the increased data backlog pressure caused by the frequent changes of hotspot data, which not only improves the efficiency of the migration process, but also reduces the cost and risk of manual intervention; and in the event that the source data table structure is inconsistent with the target table structure, or the migration whitelist is under-configured or misconfigured, the problem is solved through dynamic table migration comparison to ensure data quality.

[0058] In an optional solution of the embodiment of the present invention, determining the data table to be migrated in the source database according to the migration configuration information includes:

[0059] A dynamic data table and a hot data table in the source database are determined according to the migration configuration information as data tables to be migrated in the source database.

[0060] In this embodiment, after the migration data information is parsed, data pulling begins. A normal data puller only pulls data from a big data component, performs basic data analysis, assembly and distribution on the pulled data, and does not process the data. This solution performs two major optimization processes on the pulled data, namely dynamic table identification and hot data identification optimization. Priority is given to migrating dynamic data tables and hot data tables to ensure that key data can be migrated and services can be restored in the shortest time.

[0061] The migration configuration information includes the priority of each data table in the source database and the historical pull information of each data table;

[0062] The historical pull information of the data table records the migration status of the data table during the previous data migration process, including but not limited to key indicators such as migration time, migration data volume, and migration success rate, providing reference value for subsequent data migration work; by analyzing the historical pull information, possible problems and bottlenecks in the migration process can be discovered in a timely manner, so as to take targeted optimization measures;

[0063] For example, if a data table has experienced a large amount of data loss during the previous migration process, then you need to pay special attention to the migration strategy of the data table during this migration to ensure the integrity and accuracy of the data.

[0064] In addition, in order to better understand and apply migration configuration information, some empirical and statistical data can be introduced to support it.

[0065] For example, during the data migration process, reasonably setting the data table priority can significantly improve the migration efficiency and reduce the resource consumption and time cost during the migration process; at the same time, through in-depth analysis of historical pull information, some potential rules in the migration process can be discovered, such as the generally low success rate of data migration in certain time periods, which may be related to factors such as network conditions and system load. Appropriate preventive measures should be taken, such as adjusting the migration time, optimizing the system configuration, etc., to improve the success rate and stability of the migration.

[0066] Figure 2 A flow chart of another data processing method provided in an embodiment of the present application is as follows: Figure 2 As shown, the method specifically includes:

[0067] S201, according to the priority of each data table in the source database included in the migration configuration information, determining the data table with the highest priority in the source database as a hot data table;

[0068] Hot data identification optimization is an algorithm optimization for data pulling. It analyzes the data pulling situation and forms an internal hot data identification storage queue for periodic data after analysis, giving priority to ensuring the pulling weight of periodic hot data.

[0069] In this embodiment, after the priority of the data table is obtained, the data table with the highest priority in the source database is used as the hot data table.

[0070] It should be noted that the hot data on the source side changes frequently, so in this case, the internal optimization algorithm cannot remain unchanged and must be guaranteed to change according to the changes in the hot data on the source side.

[0071] like Figure 3 As shown in the figure, at time 1, the hotspot table on the source side is table 1, and the non-hotspot tables are table 2 and table 3. At this time, the migration comparison task will identify table 1 as a hotspot table. The count value score of each table is initially set to 0. After each successful data pull, the score corresponding to the table is increased by 1. Tables table 2 and table 3 are identified as relatively non-hotspot tables. After the data is successfully pulled, the score is increased by 1. If the data of a table is not successfully pulled at a certain moment, it is a non-hotspot table, and the score is set to the initial value 0. If the data currently played back has been caught up, it means that there is no more data for this table to pull at this moment, and the score is set to the initial value 0.

[0072] S202, determining whether there is a pull anomaly in the historical pull information of the hot data table;

[0073] In this embodiment, after determining the hot data table, it is necessary to determine whether the data table is added to the blacklist due to query failure caused by external table structure changes or environmental reasons during the historical data migration process.

[0074] In addition, you need to check the dependencies between the data table and other systems or modules to ensure that the migration process will not affect the normal operation of other components due to data inconsistencies or access permission issues. For data tables with complex dependencies, you need to develop a detailed migration plan to deal with possible unexpected situations.

[0075] S203: In the case of a pull exception, the priority of the hot data table is adjusted to re-determine the hot data table in the source database based on the adjusted priority.

[0076] In this embodiment, when the hot data table is pulled abnormally, the score of the hot data table is set to an initial value of 0 to adjust the priority of the hot data table, and then the hot data table is re-determined from the source database using the adjusted priority.

[0077] like Figure 3 As shown in the figure, if the network of the server where the current component is located is disconnected from the outside world or the server is interrupted, the table will no longer pull new data. The data that has been pulled will be compared or blocked normally, and there will be no omission or error. At this time, the score value does not change, and the original hot table state is still maintained, thereby ensuring that the pull weight of table1 at time 1 is absolute; at time 2, the source hot table is changed to table2, and the non-hot table is changed to table1 and table3. However, at time 1, table1 is the hot table with the highest score value. At time 2, table1 data will be pulled first, but the table has been pulled or added to the blacklist. At this time, the migration comparison task will set the score of the hot table at time 1 to 0, reducing it to a non-hot table. At the same time, after the continuous pulling is successful, the score of table2 is +1. When the score count value is the largest value among the three tables, it means that table2 has become the hot table at the current moment, and table1 and table3 are identified as relatively non-hot tables, thereby ensuring that the pull weight of table2 at time 2 is absolute.

[0078] It should be noted that if table3 is a non-hotspot table and data is pulled at time1, and table3 is also a non-hotspot table and data is pulled at time2, then the points of table1 are reset to zero at time2. Table3 is the relative hotspot table for table1, and its pull weight is greater than that of table1. In the subsequent time period, non-hotspot tables will also be re-sorted according to their own pull situations to ensure the pull weight of the phased tables.

[0079] It should be noted that each data table corresponds to the structure it pulls, and each piece of data also has a unique identifier. The unique identifier set after the data migration and comparison is completed can be used to confirm whether the data migration and comparison are completed. If a high-availability scenario occurs, this unique identifier can be used to perform abnormal switching or breakpoint resumption to prevent repeated data migration and comparison, which may lead to conflicts between target data and waste of resources.

[0080] like Figure 4 As shown in the figure, dynamic identification optimization is mainly carried out when some unavoidable errors are found after the migration comparison task has started, causing the migration comparison task of this table or multiple tables to fail, but the main task does not want to be affected; the migration comparison task will remove such tables after receiving the instruction, and will not participate in subsequent migration comparison tasks. After manually solving external problems of such tables (for example, inconsistent table structure, table name change, etc.), such tables will be dynamically added back; the migration comparison task will dynamically identify the addition of such tables, and perform a new round of migration comparison on the data of such tables. After the migration comparison of historical data is completed, such tables are no different from other tables; during the migration comparison process, you can dynamically choose whether certain tables will participate in this migration comparison.

[0081] Specifically, the initial table A participates in the migration comparison, but does not participate in the migration comparison in stage 1; the initial table A does not participate in the migration comparison, but participates in the migration comparison in stage 2; the initial table A participates in the migration comparison, but does not participate in the migration comparison in stages 1 and 3, but participates in the migration comparison in stages 2 and 4, etc. The object of this migration comparison task can be dynamically selected.

[0082] The data processing method provided in the embodiment of the present application prioritizes the migration of hot data tables during data migration.

[0083] The data processing method of the embodiment of the present application not only prioritizes the migration of hot data tables, but also takes into account the efficiency and stability of data migration. It optimizes the resource allocation during the migration process, dynamically adjusts the priority and resource quota of the migration task, ensures that key data tables can be processed first, and avoids migration delays or failures due to insufficient resources.

[0084] In an optional solution of the embodiment of the present invention, comparing the newly added data table with the to-be-migrated data table to obtain a comparison result includes:

[0085] Obtaining a first number of data items stored in the newly added data table and a second number of data items stored in the to-be-migrated data table;

[0086] Determine a first comparison result according to the first number of items and the second number of items;

[0087] Acquire first data information of data stored in the newly added data table and second data information of data stored in the to-be-migrated data table;

[0088] A second comparison result is determined according to the first data information and the second data information.

[0089] In this embodiment, the query comparison of the pulled data is performed by using a query comparator, and the query is performed in the form of SQL statements, and the key-value pairs are stored and compared to ensure the normal query comparison of the pulled data; there is a corresponding unique identifier for each migrated data, and it can be determined whether the corresponding data participates in the comparison logic. During the comparison process, the specific information of the compared data will be recorded, and the number of difference data and the number of comparison data in the corresponding table will be counted, and the accuracy of the compared data can be confirmed by external comparison. At the same time, there is a function of adjusting the number of numbers, which can be used to adjust the number of differentiated data obtained by comparing the migrated data.

[0090] Specifically, first, extract and obtain the number of records of the first data from the newly added data table. This is usually achieved by executing the `COUNT(*)` function in the SQL query statement to count the specific filter conditions (if any); at the same time, extract the number of records of the second data from the data table to be migrated and perform the counting operation in the same way;

[0091] After obtaining the first and second numbers, a preliminary comparison is performed to determine whether the two data tables are consistent in terms of data magnitude. If the two numbers are equal, there may be no problem in the data migration process at the quantity level; if the numbers are not equal, further investigation is required to find possible data loss or duplication problems; extract detailed information of the first data (i.e., first data information) from the newly added data table, including but not limited to data fields, data types, data values, etc. At the same time, extract detailed information of the second data (i.e., second data information) from the data table to be migrated and perform the same process.

[0092] In an optional solution of the embodiment of the present invention, when the comparison result is different, performing exception processing on the newly added data table includes:

[0093] When both the first comparison result and the second comparison result are different, re-execute the migration of the data table to be migrated;

[0094] When the first comparison result is no difference and the second comparison result is a difference, obtaining difference data in the newly added data table and deleting the difference data;

[0095] Acquire first data corresponding to the difference data in the to-be-migrated data table;

[0096] The first data is re-migrated to the target database.

[0097] In this embodiment, when there are differences between the first comparison result and the second comparison result, it indicates that there are serious problems in the migration process. In this case, the migration process should be suspended, and the specific content and source of the difference data should be analyzed to find possible causes of the error through the three steps of checking the migration script, comparing the data format, and analyzing the data integrity.

[0098] If available, check the migration script to verify whether the migration script has completed the migration task completely and accurately to ensure that there are no omissions or erroneous operations; compare data formats to check whether the data formats between the source database and the target database are consistent, such as character set, time format, etc., to ensure that there are no differences in data due to format issues during the migration process; analyze data integrity to check whether there is data loss or tampering in the source database and the target database to ensure the integrity of the data before and after migration.

[0099] If the first comparison result shows no difference and the second comparison result shows a difference, it can be considered that the source database has not changed during the migration process, and the difference data mainly comes from the newly added data table. In this case, three steps should be taken to process the difference data: obtain the difference data, delete the difference data, and re-migrate the first data:

[0100] Specifically, obtaining differential data includes accurately obtaining differential data from the newly added data table. This can usually be achieved by comparing the contents of the corresponding data tables in the source database and the target database; deleting differential data, including after confirming that the differential data does exist in the newly added data table and does not belong to the scope to be migrated, you can choose to delete these differential data. However, before deleting, be sure to back up just in case; obtaining the corresponding first data requires obtaining the first data corresponding to the differential data from the data table to be migrated. This usually means that it is necessary to find and locate these data based on some kind of association relationship (such as primary key, foreign key, etc.); re-migrate the first data to re-migrate the acquired first data to the target database; in this process, ensure the accuracy and completeness of the migration to avoid the occurrence of differential data again.

[0101] It should be noted that when dealing with differential data during data migration, you also need to pay attention to data consistency, migration efficiency, and risk management. Ensuring the consistency of data before and after migration is the key. While pursuing data accuracy, take a variety of measures to verify the accuracy and integrity of the data during the migration process, such as using tools such as checksums and hash values ​​to compare data differences. You also need to pay attention to the efficiency of migration, and improve the migration speed and reduce the time cost during the migration process by optimizing migration scripts and adjusting database performance parameters. During the migration process, a detailed risk management plan should be formulated, including data backup, recovery strategies, emergency response measures, etc., to deal with various problems and challenges that may arise.

[0102] In an optional solution of the embodiment of the present invention, after migrating the data table to be migrated to the target database, the method further includes:

[0103] Obtain the migration status of the hot data table in the data table to be migrated;

[0104] When the migration status is that all migrations are successful, the priority of the hotspot data table is adjusted.

[0105] In this embodiment, after the data table is migrated to the target database, the migration status of the data table is monitored, wherein the migration status includes multiple aspects such as data integrity, migration speed, and error rate. By real-time monitoring of the migration process, problems are discovered and resolved in a timely manner to ensure smooth data migration.

[0106] Specifically, the migration status can be evaluated through log analysis, performance monitoring, etc. For example, you can regularly check the migration log to see if there is any data loss or damage; at the same time, you can also monitor the performance indicators of the target database, such as query response time, CPU and memory usage, to evaluate the impact of migration on database performance.

[0107] After confirming that the migration status of hot data tables is fully migrated successfully, adjust the priority of hot data tables, optimize database performance and resource utilization, and ensure that core business data can be better processed.

[0108] Specifically, database resources can be reallocated based on the access frequency and importance of data. For example, for hot data tables with high access frequency and high importance, more CPU and memory resources can be allocated to improve their query response speed and processing capacity. Secondly, database performance can be further improved by optimizing database indexes, adjusting query statements, etc.

[0109] In an optional solution of the embodiment of the present invention, the method further includes:

[0110] Determine whether the dynamic data table and the hot data table are whitelist data tables of the source database;

[0111] If so, the dynamic data table and the hot data table are used as the data tables to be migrated in the source database; otherwise, the dynamic data table and the hot data table are re-determined from the source database.

[0112] In this embodiment, in the process of determining the data table to be migrated, it is also necessary to determine whether the dynamic data table and the hot data table belong to the white list data table of the source database, and formulate a reasonable migration strategy based on this; after confirming that the dynamic data table and the hot data table are the white list data tables of the source database, execute the corresponding migration strategy.

[0113] Figure 5 A schematic diagram of the steps of a data processing method provided in an embodiment of the present application is as follows: Figure 5 As shown in the figure, the relationship between each link in the data comparison process, there are special processing strategies for the pulled data. The following is a detailed explanation of the role of each link and processing strategy:

[0114] Metadata Manager:

[0115] The first task in processing data is to obtain metadata information for the migration and comparison task. This is done by obtaining the types of both ends of the database, the configuration of the blacklist and whitelist, and other special configurations to help parse the details of the migration and comparison task. The advantage of this approach is that it provides a comprehensive understanding of the database, including its structure and configuration, so that the requirements and characteristics of the data migration and comparison task can be better understood. This detailed information can help optimize the comparison algorithm and improve the accuracy and efficiency of the comparison. In addition, the acquisition of metadata information can also help identify potential risks and problems, such as data type mismatches or configuration errors, so that timely adjustments and processing can be made during the migration and comparison process. In addition, the use of a unified metadata processor can reduce parsing time without worrying about consuming extra time and resources, and the encryption of sensitive information also ensures data security.

[0116] Data Puller:

[0117] After the metadata information is parsed, data is pulled. Normal data pullers only pull data from big data components, perform basic data parsing, assembly and distribution on the pulled data, and do not process the data. However, this invention performs two major optimization processes on the pulled data, namely dynamic table recognition and hot data recognition optimization.

[0118] Query Comparator:

[0119] The query comparator is mainly used for query comparison of pulled data. It ensures normal query comparison of pulled data by using SQL statements to query, store and compare in the form of key-value pairs.

[0120] Difference Data Processor:

[0121] The difference in the structure of the pulled data or the comparison difference caused by the failure in the migration process will be processed in the difference data processor. There may also be some false positive data, which will be eliminated at the end of the migration comparison task.

[0122] This data processing method allows dynamic data identification and processing during the migration process; once it is found that hot data or other dynamic data tables cause the migration comparison task to fail, the system can automatically exclude them and dynamically add them back in the subsequent migration; this adaptability greatly improves the flexibility and robustness of the migration process. Secondly, this method focuses on accuracy and efficiency. The integrity and consistency of the data are ensured by focusing on and checking the accuracy of hot data during the migration process. At the same time, it provides the option of manually selecting the migration comparison of dynamic data tables, so that users can adjust and optimize according to specific circumstances; most importantly, this data processing method can automatically process the migration comparison of hot data, thereby reducing the increased data backlog pressure caused by frequent changes in hot data on the source side; this automated processing not only improves the efficiency of the migration process, but also reduces the cost and risk of manual intervention.

[0123] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device specifically includes:

[0124] The acquisition module 601 is used to acquire the migration configuration information of the source database;

[0125] A determination module 602 is used to determine the data table to be migrated in the source database according to the migration configuration information, and migrate the data table to be migrated to the target database;

[0126] A comparison module 603 is used to obtain a newly added data table in the target database corresponding to the data table to be migrated, and compare the newly added data table with the data table to be migrated to obtain a comparison result;

[0127] The exception handling module 604 is used to perform exception handling on the newly added data table when the comparison result shows that there is a difference, and update the newly added data table according to the exception handling result.

[0128] In a possible implementation manner, the determination module 602 is further configured to determine, according to the migration configuration information, a dynamic data table and a hot data table in the source database as the data table to be migrated in the source database.

[0129] In one possible implementation, the determination module 602 is also used to determine the data table with the highest priority in the source database as the hot data table based on the priority of each data table in the source database included in the migration configuration information; determine whether there is a pull anomaly in the historical pull information of the hot data table; and if there is a pull anomaly, adjust the priority of the hot data table to re-determine the hot data table in the source database based on the adjusted priority.

[0130] In one possible implementation, the comparison module 603 is also used to obtain a first number of data stored in the newly added data table and a second number of data stored in the data table to be migrated; determine a first comparison result based on the first number and the second number; obtain first data information of the data stored in the newly added data table and second data information of the data stored in the data table to be migrated; and determine a second comparison result based on the first data information and the second data information.

[0131] In one possible implementation, the exception handling module 604 is also used to re-execute the migration of the data table to be migrated when there are differences between the first comparison result and the second comparison result; when there is no difference in the first comparison result and there is a difference in the second comparison result, obtain the difference data in the newly added data table and delete the difference data; obtain the first data corresponding to the difference data in the data table to be migrated; and re-migrate the first data to the target database.

[0132] In a possible implementation, the determination module 602 is further configured to obtain a migration status of a hotspot data table in the data table to be migrated; and when the migration status indicates that all migrations are successful, adjust the priority of the hotspot data table.

[0133] In one possible implementation, the determination module 602 is also used to determine whether the dynamic data table and the hot data table are whitelist data tables of the source database; if so, the dynamic data table and the hot data table are executed as data tables to be migrated in the source database, otherwise, the dynamic data table and the hot data table are re-determined from the source database.

[0134] The data processing device provided in this embodiment may be as follows Figure 6 The data processing device shown in can execute the following Figure 1-5 All steps of data processing in Figure 1-5For details on the technical effects of the data processing shown, please refer to Figure 1-5 For the sake of brevity, the relevant description is not repeated here.

[0135] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, an embodiment of the present application provides an electronic device, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704; the memory 703 is used to store computer programs; the processor 701 is used to execute the program stored in the memory 703, and implement the steps of data processing provided by any of the above method embodiments:

[0137] Obtain migration configuration information of a source database; determine a data table to be migrated in the source database based on the migration configuration information, and migrate the data table to be migrated to a target database; obtain a newly added data table in the target database corresponding to the data table to be migrated, and compare the newly added data table with the data table to be migrated to obtain a comparison result; if the comparison result is different, perform exception processing on the newly added data table, and update the newly added data table according to the exception processing result.

[0138] In a possible implementation, a dynamic data table and a hot data table in the source database are determined according to the migration configuration information as the data tables to be migrated in the source database.

[0139] In one possible implementation, based on the priority of each data table in the source database included in the migration configuration information, determine the data table with the highest priority in the source database as the hot data table; determine whether there is a pull anomaly in the historical pull information of the hot data table; if there is a pull anomaly, adjust the priority of the hot data table to redetermine the hot data table in the source database based on the adjusted priority.

[0140] In one possible implementation, a first number of data items stored in the newly added data table and a second number of data items stored in the data table to be migrated are obtained; a first comparison result is determined based on the first number and the second number; first data information of the data stored in the newly added data table and second data information of the data stored in the data table to be migrated are obtained; a second comparison result is determined based on the first data information and the second data information.

[0141] In one possible implementation, when there are differences between the first comparison result and the second comparison result, migration of the data table to be migrated is re-executed; when there is no difference in the first comparison result and there is a difference in the second comparison result, difference data in the newly added data table is obtained and the difference data is deleted; first data corresponding to the difference data in the data table to be migrated is obtained; and the first data is re-migrated to the target database.

[0142] In a possible implementation, the migration status of the hot data table in the to-be-migrated data table is obtained; when the migration status indicates that all migrations are successful, the priority of the hot data table is adjusted.

[0143] In one possible implementation, determine whether the dynamic data table and the hot data table are whitelist data tables of the source database; if so, execute the dynamic data table and the hot data table as the data tables to be migrated in the source database, otherwise, re-determine the dynamic data table and the hot data table from the source database.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0145] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0146] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that: include: Get the migration configuration information of the source database; Determine the data table to be migrated in the source database according to the migration configuration information, and migrate the data table to be migrated to the target database; Acquire a newly added data table in the target database corresponding to the data table to be migrated, and compare the newly added data table with the data table to be migrated to obtain a comparison result; When the comparison result shows that there is a difference, an exception process is performed on the newly added data table, and the newly added data table is updated according to the exception process result.

2. The method according to claim 1, characterized in that The step of determining the data table to be migrated in the source database according to the migration configuration information includes: A dynamic data table and a hot data table in the source database are determined according to the migration configuration information as data tables to be migrated in the source database.

3. The method according to claim 2, characterized in that The migration configuration information includes the priority of each data table in the source database and the historical pull information of each data table; Determining a hot data table in the source database according to the migration configuration information includes: According to the priorities of the data tables in the source database included in the migration configuration information, determining the data table with the highest priority in the source database as the hot data table; Determine whether there is a pull anomaly in the historical pull information of the hot data table; In the case of a pull exception, the priority of the hot data table is adjusted to re-determine the hot data table in the source database based on the adjusted priority.

4. The method according to claim 1, characterized in that The comparing the newly added data table with the to-be-migrated data table to obtain a comparison result includes: Obtaining a first number of data items stored in the newly added data table and a second number of data items stored in the to-be-migrated data table; Determine a first comparison result according to the first number of items and the second number of items; Acquire first data information of data stored in the newly added data table, and second data information of data stored in the to-be-migrated data table; A second comparison result is determined according to the first data information and the second data information.

5. The method according to claim 4, characterized in that When the comparison result is different, performing exception processing on the newly added data table includes: When both the first comparison result and the second comparison result are different, re-execute the migration of the data table to be migrated; When the first comparison result is no difference and the second comparison result is a difference, obtaining difference data in the newly added data table and deleting the difference data; Acquire first data corresponding to the difference data in the to-be-migrated data table; The first data is re-migrated to the target database.

6. The method according to claim 1, characterized in that After migrating the data table to be migrated to the target database, the method further includes: Obtain the migration status of the hot data table in the data table to be migrated; When the migration status is that all migrations are successful, the priority of the hotspot data table is adjusted.

7. The method according to claim 2, further comprising: Determine whether the dynamic data table and the hot data table are whitelist data tables of the source database; If so, the dynamic data table and the hot data table are used as the data tables to be migrated in the source database; otherwise, the dynamic data table and the hot data table are re-determined from the source database.

8. A data processing device, characterized in that: include: The acquisition module is used to obtain the migration configuration information of the source database; A determination module, configured to determine a data table to be migrated in the source database according to the migration configuration information, and migrate the data table to be migrated to the target database; A comparison module, used for obtaining a newly added data table in the target database corresponding to the data table to be migrated, and comparing the newly added data table with the data table to be migrated to obtain a comparison result; The exception handling module is used to perform exception handling on the newly added data table when the comparison result shows that there is a difference, and update the newly added data table according to the exception handling result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images