Data consistency verification method, device, equipment, medium and program product

Abstract

The present application provides a data consistency verification method, apparatus, device, medium and program product, the method comprising: an electronic device obtains a plurality of pre-migration fields of each row from a pre-migration data table, and obtains a plurality of post-migration fields of each row from a post-migration data table; for the pre-migration data table, concatenates and encrypts non-numeric fields in the plurality of pre-migration fields of each row to obtain a plurality of pre-migration character strings, and stores the plurality of pre-migration character strings in a first data file in the order of preset parameters; for the post-migration data table, concatenates and encrypts non-numeric fields in the plurality of post-migration fields of each row to obtain a plurality of post-migration character strings, and stores the plurality of post-migration character strings in the order of preset parameters to a second data file; verifies the first data file and the second data file in the storage order to obtain a first verification result, thereby reducing verification time and improving verification efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data consistency verification method, device, equipment, medium and program product. Background Art

[0002] A database is a computer software system that stores and manages data according to data structures. It is divided into many different types, such as relational databases and non-relational databases. It is very common to migrate data tables between different types of databases. However, if the data tables are not properly handled during the migration process, there is a high probability that there will be errors such as missed migration, wrong migration, and data loss during network transmission, resulting in inconsistency between the migrated data tables and the pre-migration data tables. Therefore, it is very necessary to verify the consistency of the data tables before and after the migration.

[0003] At present, the consistency check of the data table before and after migration mainly involves grouping the data in the data table before and after migration, using the message digest algorithm (MD5) to obtain multiple groups of encrypted strings before and after migration, and then comparing the two. If the encrypted strings before and after migration are the same, it means that the group of data before and after migration is consistent; if the encrypted strings before and after migration are different, it means that the group of data before and after migration is inconsistent.

[0004] However, when the amount of data is large, it is necessary to determine the group of pre-migration data corresponding to each group of migrated data, and compare and verify the two groups, which is time-consuming and has low verification efficiency. Summary of the invention

[0005] The present application provides a data consistency verification method, device, equipment, medium and program product to solve the problem that when the amount of data is large, it is necessary to determine the group of pre-migration data corresponding to each group of migrated data and compare and verify the two groups, which is time-consuming and has low verification efficiency.

[0006] In a first aspect, an embodiment of the present application provides a data consistency verification method, including:

[0007] Acquire multiple pre-migration fields of each row from a data table before migration, and acquire multiple post-migration fields of each row from a data table after migration, wherein the data table before migration is stored in a first database, and the data table after migration is stored in a second database;

[0008] For the data table before migration, concatenate and encrypt non-numeric fields in the multiple pre-migration fields of each row to obtain multiple pre-migration character strings, and store the multiple pre-migration character strings in the first data file according to the order of preset parameters;

[0009] For the migrated data table, concatenate and encrypt non-numeric fields in the multiple migrated fields of each row to obtain multiple migrated character strings, and store the multiple migrated character strings in the second data file according to the order of the preset parameters;

[0010] Verifying the first data file and the second data file in a storage order to obtain a first verification result, where the first verification result is used to indicate whether the data table before the migration is consistent with the data table after the migration;

[0011] The preset parameters include the primary key or index of the data table.

[0012] In a possible design of the first aspect, the method further includes:

[0013] Sum the numerical fields in the data table before migration to obtain the numerical values ​​before migration;

[0014] Sum the numerical fields in the migrated data table to obtain the migrated numerical values;

[0015] The pre-migration value is compared and verified with the post-migration value to obtain a second verification result, where the second verification result is used to indicate whether the data table before the migration is consistent with the data table after the migration.

[0016] In another possible design of the first aspect, before encrypting the string concatenated from the multiple pre-migration fields, the method further includes:

[0017] Convert the string of multiple pre-migration fields into a preset format;

[0018] Accordingly, before encrypting the string concatenated from the multiple migrated fields, the method further includes:

[0019] The string formed by concatenating the multiple migrated fields is converted into the preset format.

[0020] In another possible design of the first aspect, before concatenating and encrypting the non-numeric fields in the multiple pre-migration fields of each row of the pre-migration data table, the method further includes:

[0021] If the non-numeric field in the pre-migration field exceeds the preset length, the non-numeric field in the pre-migration field is truncated according to the preset length;

[0022] Accordingly, before concatenating and encrypting the non-numeric fields in the multiple migrated fields of each row of the migrated data table, the method further includes:

[0023] If the non-numeric field in the migrated field exceeds the preset length, the non-numeric field in the migrated field is truncated according to the preset length.

[0024] In another possible design of the first aspect, before concatenating and encrypting the non-numeric fields in the multiple pre-migration fields of each row of the pre-migration data table, the method further includes:

[0025] If the non-numeric field in the pre-migration field is a null value, convert the non-numeric field in the pre-migration field into a space;

[0026] Accordingly, before concatenating and encrypting the non-numeric fields in the multiple migrated fields of each row of the migrated data table, the method further includes:

[0027] If the non-numeric field in the migrated field is a null value, the non-numeric field in the migrated field is converted into a space.

[0028] In yet another possible design of the first aspect, before acquiring multiple pre-migration fields of each row from the data table before migration and acquiring multiple post-migration fields of each row from the data table after migration, the method further includes:

[0029] The data table before the migration in the first database and the data table after the migration in the second database are obtained.

[0030] In a second aspect, an embodiment of the present application provides a data consistency verification device, including:

[0031] an acquisition module, used to acquire multiple pre-migration fields of each row from a data table before migration, and acquire multiple post-migration fields of each row from a data table after migration, wherein the data table before migration is stored in a first database, and the data table after migration is stored in a second database;

[0032] a processing module, configured to concatenate and encrypt non-numeric fields in the plurality of pre-migration fields of each row of the pre-migration data table to obtain a plurality of pre-migration character strings, and store the plurality of pre-migration character strings in a first data file according to an order of preset parameters;

[0033] The processing module is further used to concatenate and encrypt non-numeric fields in the multiple migrated fields of each row of the migrated data table to obtain multiple migrated character strings, and store the multiple migrated character strings in the second data file according to the order of the preset parameters;

[0034] The processing module is further used to verify the first data file and the second data file in a storage order to obtain a first verification result, where the first verification result is used to indicate whether the data table before migration is consistent with the data table after migration;

[0035] The preset parameters include the primary key or index of the data table.

[0036] In a possible design of the second aspect, the processing module is further used to:

[0037] Sum the numerical fields in the data table before migration to obtain the numerical values ​​before migration;

[0038] Sum the numerical fields in the migrated data table to obtain the migrated numerical values;

[0039] The pre-migration value is compared and verified with the post-migration value to obtain a second verification result, where the second verification result is used to indicate whether the data table before the migration is consistent with the data table after the migration.

[0040] In another possible design of the second aspect, the processing module is further used to:

[0041] Convert the string of multiple pre-migration fields into a preset format;

[0042] Accordingly, the processing module is also used for:

[0043] The string formed by concatenating the multiple migrated fields is converted into the preset format.

[0044] In yet another possible design of the second aspect, the processing module is further configured to:

[0045] If the non-numeric field in the pre-migration field exceeds the preset length, the non-numeric field in the pre-migration field is truncated according to the preset length;

[0046] Accordingly, the processing module is also used for:

[0047] If the non-numeric field in the migrated field exceeds the preset length, the non-numeric field in the migrated field is truncated according to the preset length.

[0048] In yet another possible design of the second aspect, the processing module is further configured to:

[0049] If the non-numeric field in the pre-migration field is a null value, convert the non-numeric field in the pre-migration field into a space;

[0050] Accordingly, the processing module is also used for:

[0051] If the non-numeric field in the migrated field is a null value, the non-numeric field in the migrated field is converted into a space.

[0052] In yet another possible design of the second aspect, the acquisition module is further used to:

[0053] The data table before the migration in the first database and the data table after the migration in the second database are obtained.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor, wherein the processor is used to implement the first aspect and the methods provided by various possible designs when executing the computer program instructions.

[0055] In a fourth aspect, an embodiment of the present application may provide a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect and the methods provided by various possible designs.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the first aspect and the methods provided by various possible designs.

[0057] The data consistency verification method, device, equipment, medium and program product provided by the embodiments of the present application include: the electronic device obtains multiple pre-migration fields of each row from the pre-migration data table, and obtains multiple post-migration fields of each row from the post-migration data table, then, for the pre-migration data table, concatenates and encrypts the non-numeric fields in the multiple pre-migration fields of each row to obtain multiple pre-migration strings, and stores the multiple pre-migration strings in a first data file in the order of preset parameters, then, for the post-migration data table, concatenates and encrypts the non-numeric fields in the multiple post-migration fields of each row to obtain multiple post-migration strings, and stores the multiple post-migration strings in a second data file in the order of preset parameters, and finally verifies the first data file and the second data file in the storage order to obtain a first verification result. The electronic device sorts the pre-migration character string and the post-migration character string, and stores the sorted pre-migration character string and the sorted post-migration character string in a first data file and a second data file respectively, verifies the first data file and the second data file in the storage order, and utilizes the index, primary key and other technologies of the database itself to sort the pre-migration character string and the post-migration character string more efficiently, so that inconsistent pre-migration character string and post-migration character string can be quickly located during the verification process, effectively reducing the verification time and improving the verification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of an application scenario of the data consistency verification method provided in an embodiment of the present application;

[0059] Figure 2 A flowchart of a first embodiment of a method for verifying data consistency provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of a flow chart of a second embodiment of a method for verifying data consistency provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the structure of a data consistency verification device provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0063] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] Before introducing the embodiments of the present application, the application scenarios of the embodiments of the present application are first explained:

[0066] The database stores data in a data table as an organizational unit. Each data table has at least one row of data, each row of data has at least one field, and each field has a corresponding data type, such as a numeric field, a date field, a time field, etc. The database can receive data information sent by other electronic devices and store it in a corresponding data table according to the data type of the data information. In order to transmit data between different databases, it is necessary to migrate data tables between different databases so that different databases can share the data information in the data table. However, if the data table is not properly handled during the migration process, it is very likely that errors such as missing migration, mis-migration, and network transmission data loss will occur, making the migrated data table inconsistent with the data table before migration. If the migrated data table is inconsistent with the data table before migration, the data information obtained from the same data table between different databases will be different, so that the processing results obtained when processing the data information are inconsistent, and the accuracy of the processing results is poor. Therefore, it is very necessary to verify the consistency of the data table before and after migration.

[0067] Currently, the consistency check of the data table before and after migration mainly involves setting a grouping factor set consisting of grouping factors of different sizes, dividing the data table before migration into different groups according to the grouping factor set, and dividing the pre-migration data of each group into a fixed length and a remaining part, and labeling each group. Then, MD5 encryption is performed on the pre-migration data of each group, the pre-migration encrypted string is obtained, and the pre-migration encrypted string is saved in the pre-migration log file.

[0068] Among them, MD5 is an information digest algorithm, which can generate a special string from a string or a file according to certain rules, and because the MD5 summary of a file is fixed, it is often used to verify whether the data has been tampered with.

[0069] After that, the migrated data table is divided into different groups according to the grouping factor set, and the migrated data of each group is divided into a fixed length and a remaining part, and each group is labeled. After that, the migrated data of each group is encrypted with MD5, the encrypted string after migration is obtained, and the encrypted string after migration is saved in the post-migration log file.

[0070] Finally, compare the log file before migration with the log file after migration. If the encrypted strings before and after migration are the same, it means that the data set before and after migration is consistent; if the encrypted strings before and after migration are different, it means that the data set before and after migration is inconsistent, thus obtaining a comparison result report. According to the fixed-length part and label of the group reported as failed (English: False), locate the corresponding position in the table after migration, and then correct or retransmit the data at the corresponding position.

[0071] However, when the amount of data is large, it is necessary to determine the group of pre-migration data corresponding to each group of migrated data, and compare and verify the two groups, which is time-consuming and has low verification efficiency.

[0072] In view of the above problems, the invention of the present application is as follows: At present, it is necessary to determine the label of the corresponding group after migration according to the label of each group before migration, so as to compare and verify the corresponding encrypted string before migration and the encrypted string after migration, which takes a long time. Based on this, the inventor found that if the encrypted string before migration can be sorted in a certain order, and the encrypted string after migration can be sorted in the same order, the sorted encrypted string before migration and the encrypted string after migration can be directly compared and verified one by one, which can solve the problem of long verification time in the prior art, thereby improving the efficiency of verification.

[0073] Exemplarily, the data consistency verification method provided in the embodiment of the present application can be applied to Figure 1 In a schematic diagram of an application scenario shown in FIG. Figure 1 A schematic diagram of an application scenario of the data consistency verification method provided in the embodiment of the present application is used to solve the above technical problems. Figure 1 As shown, the application scenario may include: a terminal device, a first database, and a second database.

[0074] For example, in Figure 1 In the application scenario shown, the terminal device can obtain the data table before migration from the first database, and can also obtain the data table after migration from the second database, so as to facilitate subsequent consistency analysis of the data table before migration and the data table after migration.

[0075] In this embodiment, the terminal device can execute the program code of the data consistency verification method based on the acquired data table before migration and the data table after migration, determine a first verification result indicating whether the data table before migration is consistent with the data table after migration, and can also determine a second verification result indicating whether the data table before migration is consistent with the data table after migration.

[0076] It should be noted that the attached Figure 1 This is only a schematic diagram of an application scenario provided by the embodiment of the present application. Figure 1 does not limit the equipment included in Figure 1 The positional relationship between the devices in the Figure 1 In the embodiment, the first database and the second database may be external memories relative to the terminal device. In other cases, the first database and / or the second database may also be placed in the terminal device.

[0077] In practical applications, since the server is also a processing device with data processing capabilities, the above Figure 1 The terminal device in the application scenario shown can also be implemented by a server. In the embodiment of the present application, the terminal device and the server for data processing can be collectively referred to as an electronic device.

[0078] The technical solution of the present application is described in detail below through specific embodiments.

[0079] It should be noted that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0080] Figure 2 The following is a flow chart of a first embodiment of a method for verifying data consistency provided in the present application. Figure 2 As shown, the data consistency verification method may include the following steps:

[0081] S101: Acquire multiple pre-migration fields of each row from a data table before migration, and acquire multiple post-migration fields of each row from a data table after migration.

[0082] In this step, when the data table is migrated from the first database to the second database, the data table in the first database is the data table before migration, and the data table in the second database is the data table after migration. When the data table contains multiple rows of data, the electronic device needs to obtain multiple fields in each row of data so as to process each row of data one by one later.

[0083] The data table before migration is stored in the first database, and the data table after migration is stored in the second database.

[0084] In a specific implementation, multiple pre-migration fields of each row can be obtained from the pre-migration data table in the following manner. The electronic device can use the database structured query language (SQL) to establish a pre-migration field information temporary table, obtain the field information of each field in the pre-migration data table, and store each field and the field information of each field in the pre-migration field information temporary table. Afterwards, the electronic device obtains multiple pre-migration fields of each row from the pre-migration field information temporary table row by row.

[0085] Among them, SQL is a special-purpose programming language, a database query and program language used to access data and query, update and manage relational database systems.

[0086] Optionally, the field information includes: the data type of the field, whether the field is a primary key, and whether the field can store null values.

[0087] Optionally, the data type of the field is divided into numeric and non-numeric types. Non-numeric fields also include date fields, time fields, etc.

[0088] Under this implementation, if there are multiple pre-migration data tables in the first database that need to be verified at this time, the electronic device can also query the pre-migration system information table in the first database, find the table names of all pre-migration data tables from the pre-migration system information table, and store the table names of the pre-migration data tables in the pre-migration field information temporary table.

[0089] Furthermore, the electronic device also needs to store each field and field information of each field in each pre-migration data table in a position corresponding to the table name of the pre-migration data table in the pre-migration field information temporary table.

[0090] Optionally, the step of the electronic device acquiring multiple post-migration fields of each row from the post-migration data table is the same as the step of acquiring multiple pre-migration fields of each row from the pre-migration data table, and the above process may be referred to and will not be described in detail here.

[0091] S102: For the data table before migration, concatenate and encrypt non-numeric fields in multiple pre-migration fields of each row to obtain multiple pre-migration character strings, and store the multiple pre-migration character strings in a first data file according to a sequence of preset parameters.

[0092] In this step, since each row of data may contain multiple fields, it will take a lot of time to directly compare the data of each row in the data table before migration with the data of each row in the data table after migration. The electronic device can process multiple pre-migration fields in each row of the data table before migration to obtain multiple pre-migration strings of the same length, which can effectively improve the efficiency of comparison and verification.

[0093] Among them, the data type of the field is divided into numeric type and non-numeric type. Non-numeric fields also include date fields, time fields, etc.

[0094] Optionally, if the non-numeric field in the field before migration exceeds a preset length, the non-numeric field in the field before migration is truncated according to the preset length so that the length of the non-numeric field after truncation is consistent with the preset length.

[0095] In one feasible manner, since the first database and the second database may have different storage formats for date type fields and time type fields, the electronic device can use a database type conversion function to convert the time type fields and date type fields in the pre-migration fields into time fields and date fields in a unified format.

[0096] Exemplarily, the unified time field format may be QQWWEE, the unified date field format may be YYYYMMDD, and the time field and date field may also include other unified formats, which may be set according to actual needs. The embodiments of the present application do not impose specific limitations on this.

[0097] Optionally, since the first database and the second database may have different storage formats for null values, if the non-numeric field in the pre-migration field is a null value, the electronic device may convert the non-numeric field in the pre-migration field with a null value into a space.

[0098] Furthermore, the electronic device may connect the non-numeric fields in the multiple pre-migration fields in each row using a unified connector.

[0099] Optionally, since the encoding types of the first database and the second database may be different, the same encoding method needs to be selected to process the data. The electronic device may convert the string of multiple pre-migration fields into a preset format, such as UTF8 format.

[0100] On this basis, the electronic device needs to call the MD5 function to encrypt the string of multiple pre-migration fields converted into a preset format, so as to obtain multiple pre-migration strings.

[0101] Optionally, the length of each string before migration is 32 bits.

[0102] Furthermore, after the electronic device obtains the multiple pre-migration character strings, it needs to store the multiple pre-migration character strings into the first data file in the order of preset parameters.

[0103] The preset parameters include the primary key or index of the data table.

[0104] In one achievable manner, the electronic device may sort the multiple pre-migration strings according to the ascending order of the primary key fields in the pre-migration data table, and store the sorted pre-migration strings in the first data file. The electronic device may also sort the multiple pre-migration strings according to the order of the indexes in the pre-migration data table.

[0105] Optionally, when both a primary key and an index exist in the data table before migration, multiple pre-migration strings are sorted in ascending order of the primary key field.

[0106] In one feasible manner, if there is neither a primary key nor an index in the data table before the migration, the strings before the migration can be sorted according to a preset rule.

[0107] Exemplarily, the preset rule may be to sort multiple pre-migration strings according to the ascending order of the first column of fields in the data table before migration, or to sort multiple pre-migration strings according to the descending order of the first column of fields in the data table before migration, or to sort multiple pre-migration strings according to the descending / ascending order of the last column of fields in the data table before migration. The preset rules may be formulated according to actual conditions, and the embodiments of the present application do not impose specific restrictions on this.

[0108] In this achievable manner, the electronic device may further add a self-increasing sequence column identifier to the sorted pre-migration character string, wherein the self-increasing sequence column identifier is a unique identifier of the pre-migration character string.

[0109] S103: For the migrated data table, concatenate and encrypt non-numeric fields in the multiple migrated fields of each row to obtain multiple migrated character strings, and store the multiple migrated character strings in a second data file according to a sequence of preset parameters.

[0110] In this step, the implementation process of this step is consistent with the implementation process of S102. You can refer to the specific implementation process of S102, which will not be repeated here.

[0111] If the non-numeric field in the migrated field exceeds the preset length, the electronic device will truncate the non-numeric field in the migrated field according to the preset length.

[0112] If the non-numeric field in the migrated field is a null value, the electronic device converts the non-numeric field in the migrated field into a space.

[0113] The electronic device may concatenate non-numeric fields in a plurality of pre-migration fields in each row, and convert a string formed by concatenating a plurality of post-migration fields into a preset format.

[0114] It should be understood that there is no requirement for the execution order of step S102 and step S103, that is, step S102 can be executed first and then step S103, or step S103 can be executed first and then step S102. The execution order can be selected according to actual conditions, and the embodiments of the present application do not impose specific restrictions on this.

[0115] S104: Verify the first data file and the second data file in the order in which they are stored, and obtain a first verification result.

[0116] In a specific implementation, the electronic device can use a diff function to verify the first data file and the second data file line by line in the order of storage. If the pre-migration string is inconsistent with the post-migration string, the inconsistent pre-migration string and post-migration string are obtained, and a first verification result is generated, wherein the first verification result is a comparison inconsistency; if the pre-migration string is consistent with the post-migration string, a first verification result is generated, wherein the first verification result is a comparison consistency.

[0117] The first verification result is used to indicate whether the data table before migration is consistent with the data table after migration.

[0118] Optionally, the electronic device may also verify the first data file and the second data file according to the self-increment sequence column identifier.

[0119] The data consistency verification method provided in the embodiment of the present application is that the electronic device obtains multiple pre-migration fields of each row from the data table before migration, and obtains multiple post-migration fields of each row from the data table after migration, and then, for the data table before migration, concatenates and encrypts the non-numeric fields in the multiple pre-migration fields of each row to obtain multiple pre-migration strings, and stores the multiple pre-migration strings in a first data file in the order of preset parameters, and then, for the data table after migration, concatenates and encrypts the non-numeric fields in the multiple post-migration fields of each row to obtain multiple post-migration strings, and stores the multiple post-migration strings in the order of preset parameters to a second data file, and finally verifies the first data file and the second data file in the storage order to obtain a first verification result. The electronic device sorts the pre-migration character string and the post-migration character string, and stores the sorted pre-migration character string and the sorted post-migration character string in a first data file and a second data file respectively, verifies the first data file and the second data file in the storage order, and utilizes the index, primary key and other technologies of the database itself to sort the pre-migration character string and the post-migration character string more efficiently, so that inconsistent pre-migration character string and post-migration character string can be accurately and quickly located during the verification process, effectively reducing the verification time and improving the verification efficiency.

[0120] Based on any of the above embodiments, Figure 3 The following is a flow chart of the second embodiment of the data consistency verification method provided in the present application. Figure 3 As shown, the data consistency verification method may include the following steps:

[0121] S201: summing up the numeric fields in the data table before migration to obtain the values ​​before migration.

[0122] In this step, the electronic device processes the non-numeric fields in the pre-migration fields in the data table before migration and the non-numeric fields in the post-migration fields in the data table after migration in Example 2. Since the data table may also include numeric fields, in order to further improve the verification efficiency, the electronic device can also process the numeric fields separately to facilitate subsequent comparison and verification.

[0123] In a specific implementation, the electronic device can use the database SUM function to sum the numerical fields in the data table before migration to obtain the pre-migration value. At the same time, the electronic device can set a pre-migration variable and store the obtained pre-migration value in the pre-migration variable.

[0124] S202: summing the numeric fields in the migrated data table to obtain the migrated numeric values.

[0125] In this step, the implementation process of this step is consistent with the implementation process of S201. You can refer to the specific implementation process of S201 and will not repeat it here.

[0126] S203: Compare and verify the value before migration with the value after migration to obtain a second verification result.

[0127] In this step, after the electronic device obtains the pre-migration value and the post-migration value, it can compare whether the two are equal. If the two are equal, it means that the data table before migration is consistent with the data table after migration. If the two are not equal, it means that the data table before migration is inconsistent with the data table after migration, and a second verification result is generated.

[0128] The second verification result is used to indicate whether the data table before migration is consistent with the data table after migration.

[0129] In a possible implementation, since there is a difference in the precision of the floating-point value type between the first database and the second database, the electronic device may preset an error range. When the electronic device compares the value before migration with the value after migration, if the difference between the two is within the preset error range, it is considered that the data table before migration and the data table after migration are consistent. Similarly, if the difference between the two is not within the preset error range, it is considered that the data table before migration and the data table after migration are inconsistent.

[0130] The data consistency verification method provided in the embodiment of the present application is that the electronic device sums the numeric fields in the data table before migration to obtain the pre-migration value, then sums the numeric fields in the data table after migration to obtain the post-migration value, and finally compares and verifies the pre-migration value with the post-migration value to obtain a second verification result. In this technical solution, the electronic device shortens the time for verifying the data table before migration and the data table after migration by summing the numeric fields in the data table after migration and the pre-migration value and the post-migration value respectively, thereby improving the verification efficiency.

[0131] Based on any of the above embodiments, before S101, the data consistency verification method may further include the following steps:

[0132] The data table before migration in the first database and the data table after migration in the second database are obtained.

[0133] Furthermore, since the amount of data in different data tables varies greatly, some data tables only include a few hundred rows of data, while some data tables may include tens of millions of rows of data. If the data consistency verification method in the above embodiment is executed on all data tables, a large amount of resources may be consumed, and the verification efficiency is low. Therefore, after obtaining the data table before migration and the data table after migration, the electronic device can also pre-process the data table before migration and the data table after migration to further improve the efficiency of subsequent verification.

[0134] Among them, the amount of data in the data table can be compared with the preset data amount. If the amount of data in the data table is less than or equal to the preset data amount, the data table is not preprocessed; if the amount of data in the data table is greater than the preset data amount, the data table is preprocessed as follows.

[0135] In one possible implementation, a reservoir algorithm can be used to randomly sample each row of data in the data table before migration. The electronic device can establish a temporary data table as a reservoir, and store the data of a preset number of rows (such as k rows) in the data table before migration in the temporary data table. Afterwards, the electronic device uses a cursor starting from the k+1th row of data in the data table before migration, and determines whether the row of data is replaced in the temporary data table with a probability of k / n, where k is the preset number of rows and n is the number of rows in the data table before migration. If the row of data needs to be replaced, a row of data is randomly selected from the temporary data table to be replaced by the row of data. Traverse each row of data in the data table before migration, and after processing all the rows of data in the data table before migration, the temporary data table is the pre-processed data table before migration.

[0136] Among them, since the number of fields in each row in the data table before migration is large, in order to improve the preprocessing efficiency, when using the reservoir algorithm for random sampling, the electronic device can use the primary key or unique index in the data table before migration for sampling. When the row of data is selected, the complete data of the row can be obtained according to the primary key or index of the row.

[0137] Furthermore, after the electronic device obtains the pre-processed data table before migration, it can obtain corresponding data in the migrated data table according to each row of data in the pre-processed data table before migration, and generate the pre-processed migrated data table according to the obtained data.

[0138] In one possible implementation, after the electronic device obtains the pre-processed data table before migration, it can obtain the field corresponding to the field corresponding to the primary key in the migrated data table based on the field corresponding to the primary key of the pre-processed data table before migration, and obtain the complete data of the row where the field is located based on the obtained field.

[0139] It should be understood that after the electronic device obtains the pre-processed pre-migration data table and the pre-processed post-migration data table, it can use the pre-processed pre-migration data table and the pre-processed post-migration data table as the pre-migration data table and the post-migration data table to perform steps S101 to S104 in the embodiment.

[0140] The data consistency verification method provided in the present application can effectively reduce the amount of data for subsequent verification by pre-processing the data table before and after migration using a reservoir algorithm, saving computing resources, reducing verification time, and improving verification efficiency while ensuring verification accuracy.

[0141] Based on any of the above embodiments, the data consistency verification method may further include the following steps:

[0142] The electronic device can also generate a dedicated SQL statement for the data table before migration based on the table structure of the data table before migration, and use the statement to complete the splicing, encryption and sorting of the data table before migration. The electronic device can generate a query header part, a field part, a query footer part and a sorting part based on the table structure of the data table before migration, and then splice these four parts into a dedicated SQL statement for the data table before migration, so as to facilitate the subsequent execution of the dedicated SQL statement, thereby completing the processing of the fields in the data table before migration and the data table after migration.

[0143] Optionally, the query header includes the first keyword of the database (such as SELECT) and the name of the MD5 function used. The first keyword indicates that the exclusive SQL statement starts from here. The query header can be used to encrypt the string of field concatenation before migration.

[0144] Optionally, the field part includes the fields of the data table before migration and the data types of the fields, which can be used to concatenate non-numeric fields in multiple migrated fields of each row.

[0145] Optionally, the query tail includes the name of the data table before migration.

[0146] Optionally, the sorting part includes a second keyword of the database (such as ORDER) and a primary key or index field of the table. The second keyword represents that the exclusive SQL statement ends here. The sorting part can be used to implement step S103.

[0147] The electronic device may also generate an exclusive SQL statement for the data table after the migration. The specific process is the same as the process of generating an exclusive SQL statement for the data table before the migration, and will not be described in detail here.

[0148] The data consistency verification method provided in the present application processes the data table before migration by using the exclusive SQL statement of the data table before migration, and processes the data table after migration by using the exclusive SQL statement of the data table after migration, so as to obtain the first data file and the second data file, laying a foundation for the subsequent verification of the first data file and the second data file.

[0149] In summary, the data consistency verification method provided by the embodiment of the present application has beneficial effects that can be summarized as follows: it utilizes the index, primary key and other technologies of the database itself to more efficiently sort the data before encryption, so that the inconsistent pre-migration character strings and post-migration character strings can be accurately and quickly located during the verification process, effectively reducing the verification time and effectively improving the verification efficiency. The numerical fields in the migrated data table and the pre-migration values ​​and the post-migration values ​​in the migrated data table are summed separately, thereby shortening the time for verifying the pre-migration data table and the post-migration data table and improving the verification efficiency. Furthermore, by using the reservoir algorithm to pre-process the pre-migration data table and the post-migration data table, the amount of data for subsequent verification can be effectively reduced, which saves computing resources, reduces the verification time and improves the verification efficiency while ensuring the accuracy of the verification.

[0150] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0151] Figure 4 This is a schematic diagram of the structure of the data consistency verification device provided in the embodiment of the present application. Figure 4 As shown, the data consistency verification device includes:

[0152] An acquisition module 41 is used to acquire multiple pre-migration fields of each row from a data table before migration, and acquire multiple post-migration fields of each row from a data table after migration, wherein the data table before migration is stored in a first database, and the data table after migration is stored in a second database;

[0153] The processing module 42 is used for concatenating and encrypting non-numeric fields in a plurality of pre-migration fields of each row of the data table before migration to obtain a plurality of pre-migration strings, and storing the plurality of pre-migration strings in a first data file according to an order of preset parameters;

[0154] The processing module 42 is further used to concatenate and encrypt non-numeric fields in a plurality of migrated fields in each row of the migrated data table to obtain a plurality of migrated character strings, and store the plurality of migrated character strings in a second data file according to an order of preset parameters;

[0155] The processing module 42 is further used to verify the first data file and the second data file in the storage order to obtain a first verification result, where the first verification result is used to indicate whether the data table before migration is consistent with the data table after migration;

[0156] The preset parameters include the primary key or index of the data table.

[0157] In a possible design of the embodiment of the present application, the processing module 42 is further configured to:

[0158] Sum the numeric fields in the data table before migration to get the value before migration;

[0159] Sum the numeric fields in the migrated data table to get the migrated value;

[0160] The values ​​before migration are compared and verified with the values ​​after migration to obtain a second verification result, where the second verification result is used to indicate whether the data table before migration is consistent with the data table after migration.

[0161] In another possible design of the embodiment of the present application, the processing module 42 is further configured to:

[0162] Convert the string of multiple pre-migration fields into a preset format;

[0163] Accordingly, the processing module 42 is further used for:

[0164] Convert the string of multiple migrated fields into a preset format.

[0165] In another possible design of the embodiment of the present application, the processing module 42 is further configured to:

[0166] If the non-numeric field in the field before migration exceeds the preset length, the non-numeric field in the field before migration is truncated according to the preset length;

[0167] Accordingly, the processing module 42 is further used for:

[0168] If the non-numeric field in the migrated field exceeds the preset length, the non-numeric field in the migrated field is truncated according to the preset length.

[0169] In another possible design of the embodiment of the present application, the processing module 42 is further configured to:

[0170] If the non-numeric field in the field before migration is empty, the non-numeric field in the field before migration will be converted to spaces;

[0171] Accordingly, the processing module 42 is further used for:

[0172] If the non-numeric field in the migrated field is an empty value, the non-numeric field in the migrated field will be converted to a space.

[0173] In another possible design of the embodiment of the present application, the acquisition module 41 is further used to:

[0174] The data table before migration in the first database and the data table after migration in the second database are obtained.

[0175] The data consistency verification device provided in the embodiment of the present application can be used to execute the data consistency verification method in any of the above embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0176] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0177] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device may include: a processor 51, a memory 52, and computer program instructions stored in the memory 52 and executable on the processor 51. When the processor 51 executes the computer program instructions, the data consistency verification method provided in any of the aforementioned embodiments is implemented.

[0178] Optionally, the electronic device may further include an interface for interacting with other devices.

[0179] Optionally, the above-mentioned components of the electronic device may be connected via a system bus.

[0180] The memory 52 may be a separate storage unit or a storage unit integrated in the processor. The number of processors may be one or more.

[0181] It should be understood that the processor 51 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0182] The system bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0183] All or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape, floppy disk, optical disc and any combination thereof.

[0184] The electronic device provided in the embodiment of the present application can be implemented in a terminal device and a server, and can be used to execute the data consistency verification method provided in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0185] An embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the above-mentioned data consistency verification method.

[0186] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0187] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0188] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the above-mentioned data consistency verification method can be implemented.

[0189] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for verifying data consistency, characterized in that: include: Acquire multiple pre-migration fields of each row from a data table before migration, and acquire multiple post-migration fields of each row from a data table after migration, wherein the data table before migration is stored in a first database, and the data table after migration is stored in a second database; For the data table before migration, multiple pre-migration fields of each row in the data table before migration are processed to obtain multiple pre-migration strings with the same length, non-numeric fields in the multiple pre-migration fields with the same length of each row are concatenated and encrypted to obtain multiple pre-migration strings, and the multiple pre-migration strings are stored in the first data file according to the order of preset parameters; For the migrated data table, concatenate and encrypt non-numeric fields in the multiple migrated fields of each row to obtain multiple migrated character strings, and store the multiple migrated character strings in the second data file according to the order of the preset parameters; Verifying the first data file and the second data file in a storage order to obtain a first verification result, where the first verification result is used to indicate whether the data table before the migration is consistent with the data table after the migration; The preset parameters include a primary key or an index of a data table; if neither a primary key nor an index exists in the data table before migration, the multiple pre-migration strings are sorted according to the descending order of the first column of fields in the data table before migration, or the multiple pre-migration strings are sorted according to the descending / ascending order of the last column of fields in the data table before migration; Add an auto-increment sequence column identifier to the sorted string before migration, where the auto-increment sequence column identifier is the unique identifier of the string before migration; Sum the numerical fields in the data table before migration to obtain the numerical values ​​before migration; Sum the numerical fields in the migrated data table to obtain the migrated numerical values; Compare and verify the pre-migration value with the post-migration value to obtain a second verification result, where the second verification result is used to indicate whether the data table before the migration is consistent with the data table after the migration; The method further includes: obtaining a data table before migration in the first database and a data table after migration in the second database; Compare the amount of data in the data table with the preset amount of data. If the amount of data in the data table is less than or equal to the preset amount of data, the data table is not preprocessed; if the amount of data in the data table is greater than the preset amount of data, the data table is preprocessed; the preprocessing includes: establishing a temporary data table as a reservoir, storing the preset number of rows of data in the data table before migration in the temporary data table; using a cursor starting from the k+1th row of data in the data table before migration, and determining whether the row of data is replaced in the temporary data table with a probability of k / n, where k is the preset number of rows and n is the number of rows in the data table before migration; if the row of data needs to be replaced, a row of data is randomly selected from the temporary data table to be replaced by the row of data; traverse each row of data in the data table before migration, and after processing all the rows of data in the data table before migration, the temporary data table is the preprocessed data table before migration; After obtaining the pre-processed data table before migration, obtaining corresponding data in the post-migration data table according to the data of each row in the pre-processed data table before migration, and generating the pre-processed post-migration data table according to the obtained data; Before concatenating and encrypting the non-numeric fields in the plurality of pre-migration fields in each row of the pre-migration data table, the method further includes: If the non-numeric field in the pre-migration field exceeds the preset length, the non-numeric field in the pre-migration field is truncated according to the preset length; Accordingly, before concatenating and encrypting the non-numeric fields in the multiple migrated fields of each row of the migrated data table, the method further includes: If the non-numeric field in the migrated field exceeds the preset length, the non-numeric field in the migrated field is truncated according to the preset length; Before concatenating and encrypting the non-numeric fields in the plurality of pre-migration fields in each row of the pre-migration data table, the method further includes: If the non-numeric field in the pre-migration field is a null value, convert the non-numeric field in the pre-migration field into a space; Accordingly, before concatenating and encrypting the non-numeric fields in the multiple migrated fields of each row of the migrated data table, the method further includes: If the non-numeric field in the migrated field is a null value, the non-numeric field in the migrated field is converted into a space.

2. The method according to claim 1, characterized in that: Before encrypting the string concatenated from the multiple pre-migration fields, the method further includes: Convert the string of multiple pre-migration fields into a preset format; Accordingly, before encrypting the string concatenated from the multiple migrated fields, the method further includes: The string formed by concatenating the multiple migrated fields is converted into the preset format.

3. The method according to claim 1, characterized in that Before acquiring multiple pre-migration fields of each row from the data table before migration and acquiring multiple post-migration fields of each row from the data table after migration, the method further includes: The data table before the migration in the first database and the data table after the migration in the second database are obtained.

4. A data consistency verification device, characterized in that: include: an acquisition module, used to acquire multiple pre-migration fields of each row from a data table before migration, and acquire multiple post-migration fields of each row from a data table after migration, wherein the data table before migration is stored in a first database, and the data table after migration is stored in a second database; a processing module, for processing, with respect to the pre-migration data table, a plurality of pre-migration fields in each row of the pre-migration data table to obtain a plurality of pre-migration character strings of the same length, concatenating and encrypting non-numeric fields in the plurality of pre-migration fields of the same length in each row to obtain a plurality of pre-migration character strings, and storing the plurality of pre-migration character strings in a first data file in an order of preset parameters; The processing module is further used to concatenate and encrypt non-numeric fields in the multiple migrated fields of each row of the migrated data table to obtain multiple migrated character strings, and store the multiple migrated character strings in the second data file according to the order of the preset parameters; The processing module is further used to verify the first data file and the second data file in a storage order to obtain a first verification result, where the first verification result is used to indicate whether the data table before migration is consistent with the data table after migration; The preset parameters include a primary key or an index of a data table; if neither a primary key nor an index exists in the data table before migration, the multiple pre-migration strings are sorted according to the descending order of the first column of fields in the data table before migration, or the multiple pre-migration strings are sorted according to the descending / ascending order of the last column of fields in the data table before migration; Add an auto-increment sequence column identifier to the sorted string before migration, where the auto-increment sequence column identifier is the unique identifier of the string before migration; The processing module is further used to sum the numerical fields in the data table before migration to obtain the numerical values ​​before migration; Sum the numerical fields in the migrated data table to obtain the migrated numerical values; Compare and verify the pre-migration value with the post-migration value to obtain a second verification result, where the second verification result is used to indicate whether the data table before the migration is consistent with the data table after the migration; The processing module is further used to obtain the data table before migration in the first database and the data table after migration in the second database; Compare the amount of data in the data table with the preset amount of data. If the amount of data in the data table is less than or equal to the preset amount of data, the data table is not preprocessed; if the amount of data in the data table is greater than the preset amount of data, the data table is preprocessed; the preprocessing includes: establishing a temporary data table as a reservoir, storing the preset number of rows of data in the data table before migration in the temporary data table; using a cursor starting from the k+1th row of data in the data table before migration, and determining whether the row of data is replaced in the temporary data table with a probability of k / n, where k is the preset number of rows and n is the number of rows in the data table before migration; if the row of data needs to be replaced, a row of data is randomly selected from the temporary data table to be replaced by the row of data; traverse each row of data in the data table before migration, and after processing all the rows of data in the data table before migration, the temporary data table is the preprocessed data table before migration; After obtaining the pre-processed data table before migration, obtaining corresponding data in the post-migration data table according to the data of each row in the pre-processed data table before migration, and generating the pre-processed post-migration data table according to the obtained data; The processing module is further configured to, if the non-numeric field in the pre-migration field exceeds a preset length, perform a truncation process on the non-numeric field in the pre-migration field according to the preset length; If the non-numeric field in the migrated field exceeds the preset length, the non-numeric field in the migrated field is truncated according to the preset length; The processing module is further configured to convert the non-numeric field in the pre-migration field into a space if the non-numeric field in the pre-migration field is a null value; If the non-numeric field in the migrated field is a null value, the non-numeric field in the migrated field is converted into a space.

5. An electronic device comprising: A processor, a memory, and computer program instructions stored in the memory and executable on the processor, wherein the processor is used to implement the data consistency verification method as described in any one of claims 1 to 3 when executing the computer program instructions.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the data consistency verification method according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it is used to implement the data consistency verification method according to any one of claims 1 to 3.

Images