Data migration verification method and system for credential and credential platform

Abstract

The invention provides a credential platform data migration verification method and system, and relates to the technical field of computers.The method comprises the steps that all to-be-migrated source data in a credential platform database are read, basic information of the source data is obtained, and a data label is set for each piece of data; reading fields and contents in the migration data, and constructing a data similarity verification model; selecting a migration path and a target database according to the migration demand, and dynamically evaluating the similarity between the source data and the target data through a data similarity verification model; verifying the migrated source data based on the target database, and performing incremental verification on data of different fields in the table; the data table of the to-be-migrated source data is verified in real time among different migration nodes, data changes on each node are verified step by step through an increment verification mechanism, if data missing or verification inconsistency occurs in the verification process, an alarm mechanism is triggered, and the problem of data missing or inconsistency can be automatically recognized and corrected.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data migration verification method and system for an information innovation platform. Background Art

[0002] As enterprises grow, the performance and storage capacity of databases often cannot meet the growing business needs. In order to adapt to the needs of business expansion, enterprises usually migrate databases to transfer data from the original database to the new database. During the migration process, the original database and the target database may be located in different geographical locations or computer rooms, and the business system usually runs in real time. Data migration needs to be carried out without interrupting the business.

[0003] In the data verification phase, existing database migration technologies mainly avoid data loss by verifying and backing up the migration path nodes step by step. However, data migration is not only a data transmission problem, but also involves verification of multiple dimensions such as data labels, migration tables, data source consistency, and similarity analysis. These traditional methods often ignore the relationship between data of different dimensions, upstream and downstream dependencies, and dynamic data changes during the migration process, resulting in the inability to discover data integrity and consistency issues after migration in a timely manner, and often the problems are only exposed after the migration is completed.

[0004] Therefore, how to monitor and accurately identify potential data problems in real time through multi-dimensional verification methods during the migration process has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a data migration verification method and system for a trusted innovation platform, which can automatically identify and correct data missing or inconsistent problems during real-time data migration by verifying and dynamically monitoring multi-dimensional features such as labels, basic information, and similarity analysis of migrated data during data migration. This method ensures high reliability and consistency during data migration by introducing similarity matching algorithms, dynamic data table verification, and target source data comparison.

[0006] To achieve the above object, the present invention provides the following technical solutions: Based on the above objectives, in a first aspect, the present invention provides a data migration verification method for an information innovation platform, comprising the following steps: Read all source data to be migrated in the Xinchuang platform database, obtain basic information about the source data, and set data labels for each piece of data; Process the source data, read the fields and content in the migrated data, and build a data similarity verification model; Select the migration path and target database based on the migration requirements, and dynamically evaluate the similarity between the source data and the target data through the data similarity verification model; When the data table reaches each node along the migration path, the migrated source data is verified based on the target database, and the data of different fields in the table are incrementally verified; During the data migration process, the data tables of the source data to be migrated are verified in real time between different migration nodes. The data changes on each node are gradually verified through the incremental verification mechanism, so that the incremental data of each field in the target database is consistent with the source data; If data is missing or verification is inconsistent during the verification process, the alarm mechanism will be triggered.

[0007] As a further solution of the present invention, a database driver is selected according to the database type used by the trusted innovation platform, a connection is established with the database using the database driver and a database connection string containing the database address and verification information, an SQL query statement is executed to read the source data to be migrated, and the query results are read and stored in the data structure in the program.

[0008] As a further solution of the present invention, obtaining basic information of source data includes querying database metadata to obtain table structure information, and the table structure information includes field name, data type, and constraints; when setting a data label for each piece of data, the data label elements include a unique identifier field in the source data table, a data source identifier, the type of data, and a data creation timestamp, and a data object that is combined into a string through the data label elements forms a unique data label.

[0009] As a further solution of the present invention, processing the source data and reading the fields and contents in the migration data includes the following steps: Read the field name and data type of the target data table; Read the field name and data type in the table structure information of the source data; Obtain the contents of the target data table from the target database through SQL query statements, and extract and convert the source data and target data according to the field mapping relationship; Use Java to map source data fields to target data fields when field names or formats do not match.

[0010] As a further solution of the present invention, when constructing a data similarity verification model, the following steps are included: Determine the string similarity between the source data and the target data according to the data type, calculate the minimum edit distance between the two strings, and calculate the similarity of the text data; Calculate the difference between the source data and target data dates based on the data creation timestamp, and use the absolute value of the date difference to calculate the similarity; The numerical similarities and date differences of all fields are weighted according to the fields to obtain the comprehensive similarity between the source data field and the target data field.

[0011] As a further solution of the present invention, selecting a migration path and a target database according to the migration requirements includes the following steps: Get the amount of data to be migrated, the migration speed, and the type of target database; Determine a compatible target database based on the source data structure and the table structure and data type of the target database; Determine the data migration path based on the migration requirements and the target database selection; among which: When the source database and target database structures are fully compatible, determine the path for direct data migration; If there are structural differences between the source database and the target database, first migrate the data to the middle layer, clean and convert the data, and then migrate the data to the target database; If the source data changes, the incremental data is migrated through the incremental migration path to update the target database.

[0012] As a further solution of the present invention, the similarity between the source data and the target data is dynamically evaluated through a data similarity verification model, which also includes setting a similarity threshold. If the similarity is lower than the similarity threshold, an alarm is triggered and the data is marked as mismatched, wherein the set similarity threshold is a global threshold.

[0013] As a further solution of the present invention, dynamically evaluating the similarity between source data and target data through a data similarity verification model includes the following steps: Initialize the field mapping relationship between source data and target data according to the table structure information and field mapping rules; Based on the field types of the source and target data, the similarity of each field is calculated using the minimum edit distance and date difference; The similarities of all fields are weighted and synthesized according to the field weights to obtain the comprehensive similarity of the entire data record; Determine whether the data migration is successful based on the preset similarity threshold; if the comprehensive similarity is lower than the set threshold, re-evaluate the data mapping or perform manual intervention.

[0014] As a further solution of the present invention, when the data table reaches each node along the migration path, a migration path is designed, including the first, second, and third migration nodes, each of which is used to verify and back up the data and establish a connection between nodes; wherein the migration path and nodes are defined as: The first migration node is used to receive source data, perform preliminary verification, and perform data backup; Second migration node: This node is further verified, focusing on the accuracy and consistency of incremental data; The third migration node: the final node, responsible for migrating data to the target database and performing the final verification to ensure data consistency; Among them, when establishing the connection between nodes, the first migration node on the migration path is connected to ensure that the source data is verified at the first migration node; the second migration node is connected on the path to ensure that the migration data is checked at the second node; the third migration node is connected on the path to ensure that the final data is verified at the target database.

[0015] As a further solution of the present invention, during the data migration process, the data table of the source data to be migrated is verified in real time between different migration nodes, including the following steps: Verification when data arrives at the first migration node: At the first migration node, the source data is backed up and field-level verification is performed. Data integrity is checked for each field in the data table, including: verifying whether the field type and length in the source data table conform to the field specifications of the target data table; verifying whether the data conforms to the non-null constraint of the field, and if the field should be non-null, verifying whether there is a null value; verifying the consistency of the field data type, and whether numbers and dates can be successfully converted to the format supported by the target database; if data is found to be missing or non-compliant at the first migration node, reselect the migration path; Verification when data arrives at the second migration node: At the second migration node, perform incremental verification to check whether the data added or modified since the first verification is consistent with the target database; verify the incremental data based on the timestamp, log record or ID sequence number of the incremental mark, and compare the differences between the source data and the target data; Verification when data arrives at the third migration node: When data arrives at the third migration node, a final verification is performed to verify whether the field data of the data table in the target database is consistent with the source database; the data similarity verification model is used for verification. If data inconsistency or missing is found at the third migration node, the data recovery mechanism is triggered to repair the data, re-migrate the missing data, and repair the data with incorrect format.

[0016] As a further solution of the present invention, the data recovery mechanism includes: Automatic rollback: If the incremental data verification fails, roll back to the previous migration node and reselect another path for data migration; Automatic retransmission: Re-migrate missing incremental data through incremental identifiers; Backup migration path: If one path fails, automatically switch to the backup path.

[0017] In a second aspect, the present invention also provides a data migration verification system for an information innovation platform, comprising the following components: Data collection module: used to extract the data to be migrated from the source database of the trusted computing platform, including reading the basic information of the source data, table structure information and data field content, and setting a unique data label for each piece of data.

[0018] Data processing module: used to process and convert source data, read the table structure information of the target database, perform data field mapping, and process the matching of field types and formats according to preset rules to ensure that the source data can be correctly migrated to the target database.

[0019] Data similarity verification module: Calculates the similarity between source data and target data based on information such as data type, field content, and data creation timestamp, and dynamically evaluates whether the data is consistent. If the similarity is lower than the set threshold, an alarm mechanism is triggered and the data is marked as mismatched. This module also supports determining whether data migration is successful based on similarity calculation and preset rules.

[0020] Migration path selection module: Automatically select the best migration path based on factors such as migration requirements, target database compatibility, and data volume. This module can determine whether to migrate directly, or whether to clean and convert data through an intermediate layer, or whether to use an incremental migration path to update data.

[0021] Node verification module: responsible for verifying data at each migration node. The first node performs preliminary verification and backs up data; the second node performs incremental verification to ensure that the newly added or modified data is consistent with the target database; the third node performs final verification to ensure the consistency of data in the target database. Each node can perform data integrity checks and ensure data accuracy and consistency through field-level verification.

[0022] Alarm and recovery module: During the verification process, if data is missing or the verification is inconsistent, the alarm mechanism is triggered, a notification is sent to relevant personnel, and corresponding measures are taken according to the configured recovery mechanism. The recovery mechanism includes automatic rollback, incremental data retransmission, and backup migration path switching.

[0023] Data backup and recovery module: used to back up data during the migration process and perform recovery operations in case of data loss or verification inconsistency, to ensure that data will not be lost or damaged during the migration process.

[0024] Data migration control module: used to control and schedule the entire data migration process, ensure the smooth migration of data from the source database to the target database, monitor the verification results of each node, dynamically adjust the migration path, and ensure the real-time and integrity of data migration.

[0025] Logging and tracking module: records various operations and verification results during the data migration process, including the verification status of each migration node, data repair and rollback records, etc., in order to trace problems and ensure the transparency of data migration.

[0026] The data migration verification system of the trusted innovation platform of the present invention ensures that the data migration process of the trusted innovation platform can be completed accurately and efficiently through the cooperation of the above-mentioned modules, while providing multi-level data verification, repair and alarm mechanisms to avoid data loss or errors during the migration process and ensure the consistency and reliability of the data after migration.

[0027] Compared with the prior art, the data migration verification method and system of the information innovation platform proposed in the present invention have the following beneficial effects: 1. The present invention dynamically evaluates the similarity between source data and target data through a data similarity verification model, and combines an incremental verification mechanism to ensure accurate matching and consistency between source data and target data during data migration. The incremental data of each field is verified in detail. If there is inconsistency or missing, an alarm mechanism will be triggered to identify and handle potential problems in a timely manner. Through a three-stage verification path (preliminary verification, incremental verification, and final verification), the present invention minimizes errors and data loss during data migration.

[0028] 2. The present invention intelligently selects the migration path according to factors such as migration requirements, data volume, and the type of target database. When the source database and the target database are structurally compatible, data can be directly migrated; if there are structural differences, data cleaning and conversion are performed through the middle layer before migration. The introduction of incremental migration paths ensures flexibility in the data migration process, especially when the source data changes, the target database can be effectively updated. This mechanism ensures dynamic adjustment of the migration path and avoids data inconsistency problems that occur during the migration process.

[0029] 3. The present invention adopts a field-level verification method based on data tags. Each piece of data is uniquely identified by means of data tags, and the field mapping relationship between source data and target data is used to ensure the accurate matching of each field during the migration process. When the field name or format does not match, the system can automatically convert and adjust the data, reducing the need for manual intervention and improving the automation of the migration process.

[0030] 4. During the data migration process, a real-time verification mechanism is used to ensure that each migration node can be fully verified. If data is missing or verification is inconsistent during the migration process, the system will immediately trigger an alarm mechanism and mark the relevant data as mismatched. In addition, the system also has a data recovery mechanism that can automatically roll back to the previous migration node when an exception occurs and reselect the migration path to ensure high reliability of data migration.

[0031] 5. The present invention provides a variety of data recovery methods, including automatic rollback, incremental data retransmission, and backup migration path switching, etc., to ensure that when a failure occurs during the migration process, timely measures can be taken to repair it, thereby minimizing the risk of data loss and migration failure. These mechanisms provide powerful fault tolerance for data migration. The migration efficiency is improved through the optimization of data structure and intelligent data processing flow. The source data undergoes comprehensive preprocessing and cleaning before migration, including field mapping, data type conversion, and timestamp verification, etc., to ensure that the source data can be quickly adapted to the target database. At the same time, the system can gradually verify the data table, avoiding excessive consumption of performance during full migration, and improving the migration speed and resource utilization efficiency.

[0032] In summary, the data migration verification method and system of the Xinchuang platform provided by the present invention greatly reduces the need for manual intervention through a highly automated verification and data processing mechanism. Especially when the data format does not match or the incremental data changes, the system can automatically perform operations such as field mapping and data conversion, and judge the consistency of the data through a similarity verification model to ensure high accuracy and high efficiency of data migration; through a multi-level and multi-dimensional verification mechanism, intelligent migration path selection, real-time verification and alarm, powerful recovery capabilities, and efficient data processing and optimization, the success rate, accuracy and efficiency of data migration are greatly improved, while reducing the cost of human intervention, improving the automation and security of the migration process, and having important practical value.

[0033] These and other aspects of the present application will be more concise and understandable in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the exemplary embodiments or related technical descriptions. The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0035] Figure 1This is a flow chart of a method for verifying data migration on an information and communication platform according to an embodiment of the present invention.

[0036] Figure 2 The present invention is a flowchart of reading fields and contents in migration data in a method for verifying data migration on a trusted computing platform according to an embodiment of the present invention.

[0037] Figure 3 The present invention provides a flowchart for constructing a data similarity verification model in a method for verifying data migration on an information innovation platform according to an embodiment of the present invention.

[0038] Figure 4 This is a flowchart of selecting a migration path and a target database according to migration requirements in a method for verifying data migration on a trusted computing platform according to an embodiment of the present invention.

[0039] Figure 5 This is a flowchart of selecting a migration path and a target database according to migration requirements in a method for verifying data migration on a trusted computing platform according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are intended to distinguish two non-identical entities or non-identical parameters with the same name. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, other steps or units inherent to a process, method, system, product or device that includes a series of steps or units.

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0045] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0046] The existing database migration is not only a problem of data transmission itself, but also involves verification of multiple dimensions such as data labels, migration tables, data source consistency, similarity analysis, etc. These traditional methods often ignore the relationship between data of different dimensions, upstream and downstream dependencies, and dynamic data changes during the migration process, resulting in the inability to timely discover data integrity and consistency issues after migration, and often the problems are only exposed after the migration is completed. The present invention proposes a data migration verification method and system for an information innovation platform, which can automatically identify and correct missing or inconsistent data during real-time data migration by verifying and dynamically monitoring multi-dimensional features such as labels, basic information, and similarity analysis of migrated data during data migration. The method ensures high reliability and consistency during the data migration process by introducing similarity matching algorithms, dynamic data table verification, and target source data comparison.

[0047] See also Figure 1 As shown, an embodiment of the present invention provides a data migration verification method for an information innovation platform, the method comprising the following steps: Step S10: read all source data to be migrated in the trusted innovation platform database, obtain basic information of the source data, and set data labels for each piece of data.

[0048] Step S20: Process the source data, read the fields and content in the migration data, and build a data similarity verification model.

[0049] Step S30: select a migration path and a target database according to the migration requirements, and dynamically evaluate the similarity between the source data and the target data through a data similarity verification model.

[0050] Step S40: When the data table reaches each node along the migration path, the migrated source data is verified based on the target database, and the data of different fields in the table are incrementally verified.

[0051] Step S50: During the data migration process, the data table of the source data to be migrated is verified in real time between different migration nodes, and the data changes on each node are gradually verified through the incremental verification mechanism, so that the incremental data of each field in the target database is consistent with the source data.

[0052] Step S60: If data is missing or verification is inconsistent during the verification process, an alarm mechanism is triggered.

[0053] In this embodiment, a database driver is selected according to the database type used by the xinchuang platform, a connection is established with the database using the database driver and a database connection string containing the database address and verification information, an SQL query statement is executed to read the source data to be migrated, and the query results are read and stored in the data structure in the program. Exemplarily, the database type used by the xinchuang platform can be MySQL, PostgreSQL, Oracle, SQL Server, and the corresponding database driver is selected. For MySQL, MySQL Connector / J can be used; for PostgreSQL, pgjdbc can be used; for Oracle, Oracle JDBC Driver can be used; for SQL Server, MicrosoftJDBC Driver is used.

[0054] Among them, obtaining the basic information of the source data includes querying the database metadata to obtain the table structure information, and the table structure information includes the field name, data type, and constraints; when setting a data label for each piece of data, the data label element includes the unique identifier field in the source data table, the data source identifier, the data type, and the data creation timestamp, and the data object is combined into a string through the data label elements to form a unique data label. Exemplarily, the label format can be set in the format of ID-timestamp-source-status, such as 12345-2024-12-18T10:15:00-sourceSystem-to be migrated.

[0055] In this embodiment, see Figure 2 As shown, in step S20, the source data is processed to read the fields and contents in the migration data, including the following steps: Step S201: Read the field names and data types of the target data table.

[0056] In this step, during the migration process, you first need to read the structure information of the target data table in the target database, including field names, data types, etc. This operation is usually implemented through database metadata query, and the field names and data types of the target table can be obtained through SQL query statements.

[0057] Step S202: Read the field names and data types in the table structure information of the source data.

[0058] In this step, the structure information of the source data table in the source database is read to ensure that the type of the source data field can be correctly understood. SQL query statements are also used to obtain the field information of the source table.

[0059] Step S203: Obtain the content of the target data table from the target database through an SQL query statement, and extract and convert the source data and the target data according to the field mapping relationship.

[0060] In this step, SQL query is used to extract the specific content of the target data table from the target database for subsequent comparison with the source data. For example, all the data in the target table is obtained through SQL statements and saved in the form of data frame, dictionary, etc.

[0061] Step S204: When the field names or formats do not match, use Java to map the source data fields to the target data fields.

[0062] During the migration process of this step, if the field names or data types between the source data and the target data table do not match, field mapping is performed to ensure that the source data can be correctly migrated to the target table. Field mapping can be implemented in Java code. For example, a mapping table or program logic is used to map the fields of the source data table to the fields of the target data table one by one.

[0063] In this embodiment, see Figure 3 As shown, in step S20, when constructing a data similarity verification model, the following steps are included: Step S211: determine the string similarity between the source data and the target data according to the data type, calculate the minimum edit distance between the two strings, and calculate the similarity of the text data.

[0064] In this step, for text fields, the string similarity between the source data and the target data can be calculated to determine their matching degree. The minimum Levenshtein distance is used for calculation.

[0065] Step S212, calculating the difference between the source data and the target data dates according to the data creation timestamp, and calculating the similarity using the absolute value of the date difference; Step S213: weight the numerical similarities and date differences of all fields according to the fields to obtain the comprehensive similarity between the source data field and the target data field.

[0066] The similarity values ​​of each field (including string similarity and date difference) are weighted according to a certain weight to finally get the comprehensive similarity. According to the preset weight (for example, the weight of the text field is 0.7, and the weight of the date field is 0.3), the similarity values ​​of each field are weighted averaged to get the overall similarity.

[0067] In this embodiment, see Figure 4As shown, in step S30, selecting a migration path and a target database according to the migration requirements includes the following steps: Step S301, obtaining the amount of data to be migrated, the migration speed, and the type of the target database; Step S302: Determine a compatible target database according to the source data structure and the table structure and data type of the target database; Step S303: Determine the data migration path according to the migration requirements and the target database selection; if the source data is compatible with the target database structure, it can be directly migrated. If there are structural differences, it can be migrated to the middle layer for cleaning and conversion before being migrated to the target database. If the source data will change, an incremental migration path can be adopted.

[0068] Step S304: When the source database and the target database are fully compatible in structure, a path for direct data migration is determined; Step S305: If there are structural differences between the source database and the target database, the data is first migrated to the middle layer, cleaned and converted, and then migrated to the target database.

[0069] When there are structural differences between the source data and the target database, the data is first migrated to the middle layer for data cleaning and conversion, and then migrated to the target database. The data is processed in the middle layer: converting data types, processing missing fields, renaming fields, etc., and finally the cleaned data is migrated to the target database.

[0070] Step S306: If the source data is changed, the incremental data is migrated through the incremental migration path to update the target database.

[0071] If the source database data has changed (such as added or modified), the changed data needs to be migrated through the incremental migration path. According to the timestamp or incremental identifier (such as log file, ID sequence number, etc.), get the new or modified records and migrate these incremental data to the target database. Incremental migration is usually performed through scheduled tasks or triggers.

[0072] The present invention intelligently selects the migration path according to factors such as migration requirements, data volume, and the type of target database. When the source database and the target database are structurally compatible, data can be directly migrated; if there are structural differences, data cleaning and conversion are performed through the middle layer before migration. The introduction of incremental migration paths ensures flexibility in the data migration process, especially when the source data changes, the target database can be effectively updated. This mechanism ensures dynamic adjustment of the migration path and avoids data inconsistency problems that occur during the migration process.

[0073] Among them, the similarity between the source data and the target data is dynamically evaluated through the data similarity verification model, and also includes setting a similarity threshold. If the similarity is lower than the similarity threshold, an alarm is triggered and the data is marked as mismatched. Among them, the set similarity threshold is a global threshold.

[0074] In this embodiment, see Figure 5 As shown, the similarity between the source data and the target data is dynamically evaluated through the data similarity verification model, including the following steps: Step S311, initializing the field mapping relationship between the source data and the target data according to the table structure information and the field mapping rules; Step S312: Calculate the similarity of each field using the minimum edit distance and date difference according to the field types of the source data and the target data.

[0075] In this step, for each field, the similarity between the source data and the target data is calculated. The edit distance is used for string type fields, and the time difference is used for date type fields. The Levenshtein distance algorithm is used to calculate string similarity, and the date difference is used to calculate the similarity of date type fields.

[0076] Step S313: The similarities of all fields are weighted and synthesized according to the field weights to obtain the comprehensive similarity of the entire data record.

[0077] Among them, the similarity values ​​are weighted according to the importance of different fields to calculate the comprehensive similarity of the entire data record. By defining the field weight, the similarity values ​​of each field are weighted and synthesized to obtain the comprehensive similarity.

[0078] Step S314: determine whether the data migration is successful based on a preset similarity threshold; if the comprehensive similarity is lower than the set threshold, re-evaluate the data mapping or perform manual intervention.

[0079] The success of data migration is determined based on the comprehensive similarity and the set threshold. If the similarity is lower than the set threshold, the migration is considered to have failed and manual intervention or field remapping is required.

[0080] The present invention adopts a field-level verification method based on data tags. Each piece of data is uniquely identified by means of data tags, and the field mapping relationship between source data and target data is used to ensure the accurate matching of each field in the migration process. When the field name or format does not match, the system can automatically convert and adjust the data, reducing the need for manual intervention and improving the automation of the migration process.

[0081] When the data table reaches each node along the migration path, a migration path is designed, including the first, second, and third migration nodes. Each node is used to verify and back up the data and establish a connection between nodes. The migration path and nodes are defined as follows: The first migration node is used to receive source data, perform preliminary verification, and perform data backup; Second migration node: This node is further verified, focusing on the accuracy and consistency of incremental data; The third migration node: the final node, responsible for migrating data to the target database and performing the final verification to ensure data consistency; Among them, when establishing the connection between nodes, the first migration node on the migration path is connected to ensure that the source data is verified at the first migration node; the second migration node is connected on the path to ensure that the migration data is checked at the second node; the third migration node is connected on the path to ensure that the final data is verified at the target database.

[0082] In this embodiment, during the data migration process, the data table of the source data to be migrated is verified in real time between different migration nodes, including the following steps: Verification when data arrives at the first migration node: At the first migration node, the source data is backed up and field-level verification is performed. Data integrity is checked for each field in the data table, including: verifying whether the field type and length in the source data table conform to the field specifications of the target data table; verifying whether the data conforms to the non-null constraint of the field, and if the field should be non-null, verifying whether there is a null value; verifying the consistency of the field data type, and whether numbers and dates can be successfully converted to the format supported by the target database; if data is found to be missing or non-compliant at the first migration node, reselect the migration path; Verification when data arrives at the second migration node: At the second migration node, perform incremental verification to check whether the data added or modified since the first verification is consistent with the target database; verify the incremental data based on the timestamp, log record or ID sequence number of the incremental mark, and compare the differences between the source data and the target data; Verification when data arrives at the third migration node: When data arrives at the third migration node, a final verification is performed to verify whether the field data of the data table in the target database is consistent with the source database; the data similarity verification model is used for verification. If data inconsistency or missing is found at the third migration node, the data recovery mechanism is triggered to repair the data, re-migrate the missing data, and repair the data with incorrect format.

[0083] During the data migration process, a real-time verification mechanism is used to ensure that each migration node can be fully verified. If data is missing or verification is inconsistent during the migration process, the system will immediately trigger an alarm mechanism and mark the relevant data as mismatched. In addition, the system also has a data recovery mechanism that can automatically roll back to the previous migration node when an exception occurs and reselect the migration path to ensure high reliability of data migration.

[0084] Wherein, the data recovery mechanism includes: Automatic rollback: If the incremental data verification fails, roll back to the previous migration node and reselect another path for data migration; Automatic retransmission: Re-migrate missing incremental data through incremental identifiers; Backup migration path: If one path fails, automatically switch to the backup path.

[0085] The present invention provides a variety of data recovery methods, including automatic rollback, incremental data retransmission, and backup migration path switching, etc., to ensure that when a failure occurs in the migration process, timely measures can be taken to repair it, thereby minimizing the risk of data loss and migration failure. These mechanisms provide powerful fault tolerance for data migration. The migration efficiency is improved through the optimization of data structure and intelligent data processing flow. The source data undergoes comprehensive preprocessing and cleaning before migration, including field mapping, data type conversion, and timestamp verification, etc., to ensure that the source data can be quickly adapted to the target database. At the same time, the system can gradually verify the data table, avoiding excessive consumption of performance during full migration, and improving the migration speed and resource utilization efficiency.

[0086] The present invention dynamically evaluates the similarity between source data and target data through a data similarity verification model, and combines an incremental verification mechanism to ensure accurate matching and consistency between source data and target data during data migration. The incremental data of each field is verified in detail. If there is inconsistency or missing, an alarm mechanism will be triggered to identify and handle potential problems in a timely manner. Through a three-stage verification path (preliminary verification, incremental verification, and final verification), the present invention minimizes errors and data loss during data migration.

[0087] In summary, the data migration verification method and system of the Xinchuang platform provided by the present invention greatly reduces the need for manual intervention through a highly automated verification and data processing mechanism. Especially when the data format does not match or the incremental data changes, the system can automatically perform operations such as field mapping and data conversion, and judge the consistency of the data through a similarity verification model to ensure high accuracy and high efficiency of data migration; through a multi-level and multi-dimensional verification mechanism, intelligent migration path selection, real-time verification and alarm, powerful recovery capabilities, and efficient data processing and optimization, the success rate, accuracy and efficiency of data migration are greatly improved, while reducing the cost of human intervention, improving the automation and security of the migration process, and having important practical value.

[0088] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0089] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0090] In a second aspect of the embodiment of the present invention, the present invention further provides a data migration verification system for an information innovation platform, comprising: Data collection module: used to extract the data to be migrated from the source database of the trusted computing platform, including reading the basic information of the source data, table structure information and data field content, and setting a unique data label for each piece of data.

[0091] Data processing module: used to process and convert source data, read the table structure information of the target database, perform data field mapping, and process the matching of field types and formats according to preset rules to ensure that the source data can be correctly migrated to the target database.

[0092] Data similarity verification module: Calculates the similarity between source data and target data based on information such as data type, field content, and data creation timestamp, and dynamically evaluates whether the data is consistent. If the similarity is lower than the set threshold, an alarm mechanism is triggered and the data is marked as mismatched. This module also supports determining whether data migration is successful based on similarity calculation and preset rules.

[0093] Migration path selection module: Automatically select the best migration path based on factors such as migration requirements, target database compatibility, and data volume. This module can determine whether to migrate directly, or whether to clean and convert data through an intermediate layer, or whether to use an incremental migration path to update data.

[0094] Node verification module: responsible for verifying data at each migration node. The first node performs preliminary verification and backs up data; the second node performs incremental verification to ensure that the newly added or modified data is consistent with the target database; the third node performs final verification to ensure the consistency of data in the target database. Each node can perform data integrity checks and ensure data accuracy and consistency through field-level verification.

[0095] Alarm and recovery module: During the verification process, if data is missing or the verification is inconsistent, the alarm mechanism is triggered, a notification is sent to relevant personnel, and corresponding measures are taken according to the configured recovery mechanism. The recovery mechanism includes automatic rollback, incremental data retransmission, and backup migration path switching.

[0096] Data backup and recovery module: used to back up data during the migration process and perform recovery operations in case of data loss or verification inconsistency, to ensure that data will not be lost or damaged during the migration process.

[0097] Data migration control module: used to control and schedule the entire data migration process, ensure the smooth migration of data from the source database to the target database, monitor the verification results of each node, dynamically adjust the migration path, and ensure the real-time and integrity of data migration.

[0098] Logging and tracking module: records various operations and verification results during the data migration process, including the verification status of each migration node, data repair and rollback records, etc., in order to trace problems and ensure the transparency of data migration.

[0099] The data migration verification system of the trusted innovation platform of the present invention ensures that the data migration process of the trusted innovation platform can be completed accurately and efficiently through the cooperation of the above-mentioned modules, while providing multi-level data verification, repair and alarm mechanisms to avoid data loss or errors during the migration process and ensure the consistency and reliability of the data after migration.

[0100] Through the above detailed steps, the trusted innovation platform data migration verification system of the present invention is used to execute the steps of the trusted innovation platform data migration verification method in the above embodiment, which will not be repeated here.

[0101] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0102] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0103] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A data migration verification method for an information innovation platform, characterized in that: The method comprises the following steps: Read all source data to be migrated in the Xinchuang platform database, obtain basic information about the source data, and set data labels for each piece of data; Process the source data, read the fields and content in the migrated data, and build a data similarity verification model; Select the migration path and target database based on the migration requirements, and dynamically evaluate the similarity between the source data and the target data through the data similarity verification model; When the data table reaches each node along the migration path, the migrated source data is verified based on the target database, and the data of different fields in the table are incrementally verified; During the data migration process, the data tables of the source data to be migrated are verified in real time between different migration nodes. The data changes on each node are gradually verified through the incremental verification mechanism, so that the incremental data of each field in the target database is consistent with the source data; If data is missing or verification is inconsistent during the verification process, the alarm mechanism will be triggered.

2. The data migration verification method of the Xinchuang platform as described in claim 1 is characterized in that: Select a database driver according to the database type used by the trusted computing platform, use the database driver and the database connection string containing the database address and verification information to establish a connection with the database, execute SQL query statements to read the source data to be migrated, and read and store the query results into the data structure in the program.

3. The data migration verification method of the Xinchuang platform as described in claim 2 is characterized in that: Obtaining basic information about source data includes querying database metadata to obtain table structure information, which includes field names, data types, and constraints. When setting data labels for each piece of data, data label elements include a unique identifier field in the source data table, data source identifier, data type, and data creation timestamp. Data objects that are combined into strings through data label elements form unique data labels.

4. The data migration verification method of the Xinchuang platform as described in claim 3 is characterized in that: Process the source data and read the fields and contents in the migration data, including the following steps: Read the field name and data type of the target data table; Read the field name and data type in the table structure information of the source data; Obtain the contents of the target data table from the target database through SQL query statements, and extract and convert the source data and target data according to the field mapping relationship; Use Java to map source data fields to target data fields when field names or formats do not match.

5. The data migration verification method of the Xinchuang platform as described in claim 4 is characterized in that: When building a data similarity verification model, the following steps are included: Determine the string similarity between the source data and the target data according to the data type, calculate the minimum edit distance between the two strings, and calculate the similarity of the text data; Calculate the difference between the source data and target data dates based on the data creation timestamp, and use the absolute value of the date difference to calculate the similarity; The numerical similarities and date differences of all fields are weighted according to the fields to obtain the comprehensive similarity between the source data field and the target data field.

6. The data migration verification method of the credential creation platform as described in claim 3 is characterized in that: Selecting the migration path and target database based on migration requirements includes the following steps: Get the amount of data to be migrated, the migration speed, and the type of target database; Determine a compatible target database based on the source data structure and the table structure and data type of the target database; Determine the data migration path based on the migration requirements and the target database selection; among which: When the source database and target database structures are fully compatible, determine the path for direct data migration; If there are structural differences between the source database and the target database, first migrate the data to the middle layer, clean and convert the data, and then migrate the data to the target database; If the source data changes, the incremental data is migrated through the incremental migration path to update the target database.

7. The data migration verification method of the Xinchuang platform as described in claim 6 is characterized in that: The similarity between the source data and the target data is dynamically evaluated through the data similarity verification model, which also includes setting a similarity threshold. If the similarity is lower than the similarity threshold, an alarm is triggered and the data is marked as mismatched, wherein the set similarity threshold is a global threshold.

8. The data migration verification method of the Xinchuang platform as described in claim 7 is characterized in that: Based on co-location, the similarity between source data and target data is dynamically evaluated through the data similarity verification model, including the following steps: Initialize the field mapping relationship between source data and target data according to the table structure information and field mapping rules; Based on the field types of the source and target data, the similarity of each field is calculated using the minimum edit distance and date difference; The similarities of all fields are weighted and synthesized according to the field weights to obtain the comprehensive similarity of the entire data record; Determine whether the data migration is successful based on the preset similarity threshold; if the comprehensive similarity is lower than the set threshold, re-evaluate the data mapping or perform manual intervention.

9. The data migration verification method of the credential creation platform as described in claim 8 is characterized in that: When the data table reaches each node along the migration path, a migration path is designed, including the first, second, and third migration nodes. Each node is used to verify and back up the data and establish a connection between nodes. The migration path and nodes are defined as follows: The first migration node is used to receive source data, perform preliminary verification, and perform data backup; Second migration node: This node is further verified, focusing on the accuracy and consistency of incremental data; The third migration node: the final node, responsible for migrating data to the target database and performing the final verification to ensure data consistency; Among them, when establishing the connection between nodes, the first migration node on the migration path is connected to ensure that the source data is verified at the first migration node; the second migration node is connected on the path to ensure that the migration data is checked at the second node; the third migration node is connected on the path to ensure that the final data is verified at the target database.

10. A data migration verification system for an information innovation platform, characterized in that: Used to execute the data migration verification method of the Xinchuang platform as claimed in claim 9, the system includes: Data collection module: used to extract the data to be migrated from the source database of the Xinchuang platform, including reading the basic information, table structure information and data field content of the source data, and setting a unique data label for each piece of data; Data processing module: used to process and convert source data, read the table structure information of the target database, perform data field mapping, and process the matching of field types and formats according to preset rules; Data similarity verification module: Calculates the similarity between source data and target data based on data type, field content, and data creation timestamp information, and dynamically evaluates whether the data is consistent; if the similarity is lower than the set threshold, an alarm mechanism is triggered and the data is marked as mismatched; Migration path selection module: automatically selects the optimal migration path based on factors such as migration requirements, compatibility of the target database and data volume; Node verification module: responsible for verifying data at each migration node; Alarm and recovery module: During the verification process, if data is missing or the verification is inconsistent, the alarm mechanism will be triggered.

Images