Incremental data synchronization test method and device, storage medium and electronic equipment

Abstract

The invention provides an incremental data synchronization test method and device, a storage medium and electronic equipment, and is applied to the technical field of data synchronization test. According to the method, the connection information of the source database and the target database is automatically obtained, the mapping relation between the source data table and the target data table is established, SQL statements used for data operation can be efficiently generated, and the SQL statements comprise insertion, updating and deletion operations executed in the source table; selection operations performed in the source table and the target table are also covered. By executing the SQL statements, the field data of the source data table can be converted into the format of the target data table according to the preset field mapping rule, and the consistency of the converted data and the actual data in the target data table is automatically compared. And finally, the success or failure information of incremental data synchronization is output, so that the efficiency and the accuracy of the data synchronization test are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization testing, and in particular to an incremental data synchronization testing method, device, storage medium and electronic equipment. Background Art

[0002] In the financial industry, especially in the digital transformation of banking systems, the importance of data synchronization testing is particularly prominent. Banking systems contain a large amount of user information and financial data, and it is crucial to ensure that this data is backed up and synchronized between multiple databases. However, the data tables in banking systems are complex and heterogeneous, and it is usually necessary to synchronize data from multiple tables into one target table, or to concatenate multiple fields into one field, which increases the difficulty of data synchronization testing. Manual comparison not only requires high professional skills from testers, but also consumes a lot of manpower, and accuracy is difficult to guarantee.

[0003] The existing technology mainly relies on table creation commands input by users to automatically generate data synchronization test cases, which has the following shortcomings: it needs to be based on user table creation commands, which requires certain SQL knowledge of testers, and cannot automatically execute test cases. It only adds expected result fields to the test cases and lacks complete automated testing functions. It is limited to test case generation for isomorphic tables and cannot parse and test data synchronization of heterogeneous tables.

[0004] Therefore, how to perform data synchronization testing efficiently and accurately has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of the above problems, the present invention provides an incremental data synchronization test method, device, storage medium and electronic device that overcome the above problems or at least partially solve the above problems. The technical solution is as follows:

[0006] An incremental data synchronization test method, comprising:

[0007] Obtaining connection information of a source database and a target database, and establishing a mapping relationship between the source data table and the target data table;

[0008] Based on the mapping relationship, automatically generate SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table;

[0009] Execute the SQL statement, convert the field data of the source data table into the format of the target data table according to the field mapping rule, and compare the consistency of the converted field data with the actual data in the target data table to obtain a comparison result;

[0010] The success or failure information of the incremental data synchronization is output according to the comparison result.

[0011] Optionally, the automatically generating, based on the mapping relationship, SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table, includes:

[0012] Based on the mapping relationship, primary key and non-primary key fields are automatically identified according to the source data table name and field characteristics selected by the user to generate SQL statements for performing insert, update and delete operations in the source data table, as well as SQL statements for performing select operations in the source data table and the target data table.

[0013] Optionally, based on the mapping relationship, according to the source data table name and field characteristics selected by the user, the primary key and non-primary key fields are automatically identified to generate SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table, including:

[0014] Based on the mapping relationship, according to the source data table name selected by the user and its field characteristics, the primary key and non-primary key fields are automatically identified to generate an INSERT statement for random data insertion according to the structure of the source data table, generate a random UPDATE statement for updating non-primary key fields, generate a DELETE statement for deleting records, and generate a SELECT statement for data comparison.

[0015] Optionally, comparing the converted field data with the actual data in the target data table for consistency includes:

[0016] Use multithreading technology to set up independent threads for each field comparison.

[0017] Optionally, the field mapping rule includes ignoring newly added fields in the target data table that are not related to the source data table when the data table is a heterogeneous table.

[0018] Optionally, outputting success or failure information of incremental data synchronization according to the comparison result includes:

[0019] If the synchronization is successful and the data is consistent, the result is recorded in the success list;

[0020] In case of synchronization failure, the failed source data table name and primary key information are recorded;

[0021] In case of data inconsistency, the inconsistent fields and their data are recorded.

[0022] Optionally, the connection information includes a database address, a port number, a user name, and a password.

[0023] An incremental data synchronization test device comprises: a mapping relationship establishment unit, an SQL statement generation unit, a comparison result acquisition unit and an incremental data information synchronization information acquisition unit.

[0024] The mapping relationship establishing unit is used to obtain connection information of the source database and the target database, and establish a mapping relationship between the source data table and the target data table;

[0025] The SQL statement generating unit is used to automatically generate SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table based on the mapping relationship;

[0026] The comparison result obtaining unit is used to execute the SQL statement, convert the field data of the source data table into the format of the target data table according to the field mapping rule, and compare the consistency of the converted field data with the actual data in the target data table to obtain the comparison result;

[0027] The incremental data information synchronization information obtaining unit is used to output success or failure information of incremental data synchronization according to the comparison result.

[0028] A computer-readable storage medium stores a program, which, when executed by a processor, implements any of the above-mentioned incremental data synchronization test methods.

[0029] An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute any of the incremental data synchronization test methods described above.

[0030] By means of the above technical scheme, the present invention provides an incremental data synchronization test method, device, storage medium and electronic device, obtains the connection information of the source database and the target database, establishes the mapping relationship between the source data table and the target data table; based on the mapping relationship, automatically generates SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table; executes the SQL statements, converts the field data of the source data table into the format of the target data table according to the field mapping rules, and compares the consistency of the converted field data with the actual data in the target data table to obtain the comparison result; outputs the success or failure information of the incremental data synchronization according to the comparison result. The present invention can efficiently generate SQL statements for data operations by automatically obtaining the connection information of the source database and the target database, and establishing the mapping relationship between the source data table and the target data table, which not only includes the insert, update and delete operations performed in the source table, but also covers the select operations performed in the source table and the target table. By executing these SQL statements, the field data of the source data table can be converted into the format of the target data table according to the preset field mapping rules, and the consistency of the converted data with the actual data in the target data table can be automatically compared. Finally, the success or failure information of incremental data synchronization is output, which significantly improves the efficiency and accuracy of data synchronization testing. The present invention eliminates the tedious steps of manually writing SQL and comparing data, allowing testers to focus more on verifying business logic, saving time and labor costs, while ensuring the reliability of test results.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0033] Figure 1 A schematic diagram showing a flow chart of an implementation of an incremental data synchronization test method provided by an embodiment of the present invention;

[0034] Figure 2 A logical block diagram of a field mapping and comparison process provided by an embodiment of the present invention is shown;

[0035] Figure 3A logical block diagram of SQL statement generation provided by an embodiment of the present invention is shown;

[0036] Figure 4 A program framework diagram of an incremental data synchronization test system provided by an embodiment of the present invention is shown;

[0037] Figure 5 A schematic diagram of the structure of an incremental data synchronization test device provided by an embodiment of the present invention is shown;

[0038] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0039] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0040] In modern information systems, as business needs continue to change, developers often need to synchronize data between multiple independent databases to ensure data consistency and integrity. The mainstream data synchronization method is usually implemented in a publish / subscribe mode, where Kafka is a widely used distributed message queue that achieves data synchronization through data transmission between producers and consumers.

[0041] Traditional data synchronization testing methods require testers to design corresponding test scenarios based on specific scenarios and data synchronization strategies. This usually involves executing SQL statements in the source database and then checking the quality of the synchronization results after the data synchronization is completed. However, in actual applications, testers face many challenges: the need to handle a large number of data tables to be synchronized, complex field splicing and joint table queries, and insufficient tester experience. These factors make it difficult to ensure the correctness and consistency of data synchronization. Therefore, automated data comparison tools have become an urgent need.

[0042] In response to this demand, the present invention proposes an automated method that automatically generates SQL statements for testing add, delete, and modify operations by analyzing the table structure and field types of the source and target databases. This method is based on a field mapping rule library to achieve automatic comparison before and after data synchronization, so that testers do not need to have extensive database knowledge and can complete data synchronization testing by simply selecting relevant table names and fields, significantly reducing manpower and time costs.

[0043] In an embodiment of the present invention, an incremental data synchronization test method is provided, which can verify the correctness and consistency of incremental data synchronization. First, determine the name of the source data table that the user needs to test, and obtain its fields and data types. Automatically generate SQL statements for inserting, updating non-primary key fields and deleting data. Generate SQL statements for SELECT operations based on the mapping relationship between the source table and the target table. Execute the SQL statements of INSERT source table, SELECT source table and target table, UPDATE source table, SELECT source table and target table, DELETE source table, and SELECT source table and target table in sequence. Extract the corresponding data based on the primary key information, and compare the data according to the set field mapping rules. Finally, output the fields and data that are inconsistent in the comparison. It can be seen that the embodiment of the present invention generates SQL test cases for addition, deletion and modification operations by automatically analyzing the table structure and field types of the source database and the target database. Use the field mapping rule library to automatically compare the data before and after data synchronization. In this way, the tester does not need to have an in-depth understanding of database knowledge, and only needs to select the table name and field to complete the data synchronization test, thereby greatly saving manpower and time costs.

[0044] like Figure 1 As shown, a flowchart of an implementation of an incremental data synchronization test method provided by an embodiment of the present invention is provided. The method may include:

[0045] S100: Obtain connection information of a source database and a target database, and establish a mapping relationship between a source data table and a target data table.

[0046] The source database refers to the location where the data is originally stored. In the data synchronization process, the source database is the starting point for providing data, from which data is read for transmission to another database.

[0047] The target database is where the data is ultimately stored. During data synchronization, the target database is the end that receives the data, where the data is transferred from the source database and saved.

[0048] The connection information refers to the information that enables the application to successfully connect to and interact with the database.

[0049] Optionally, the connection information includes database address, port number, user name and password.

[0050] The database address refers to the IP address or domain name of the server where the database is located.

[0051] The port number is the network port used to access the database service.

[0052] The user name is the name of the database access account.

[0053] Here, password refers to the access credential paired with username.

[0054] The tables and fields may not be exactly the same between the source and target databases. The mapping relationship defines how to correspond the tables and fields in the source database to the tables and fields in the target database. By establishing this mapping relationship, the system can know how to correctly convert and transfer data from the source table to the target table, thereby maintaining data structure consistency and ensuring accurate data transmission.

[0055] The information of the source database and the target database that need to be compared in the embodiment of the present invention includes the database address, database name, IP address, user name and password, etc. The source data table and the target data table are associated in a one-to-one correspondence through a mapping relationship. If you want to add more database information or table names that need to be compared, you only need to add the corresponding information in the class to complete the configuration.

[0056] S110 . Based on the mapping relationship, automatically generate SQL statements for executing insert, update and delete operations in the source data table, and SQL statements for executing select operations in the source data table and the target data table.

[0057] Among them, the insert operation (INSERT) is the operation of adding new records to the database table. The SQL statement corresponding to the insert operation is usually "INSERT INTO", followed by the table name, the column name to be inserted, and the corresponding value.

[0058] Among them, the update operation (UPDATE) is an operation to modify existing records in a database table. The SQL statement corresponding to the update operation is "UPDATE", which contains the table name, the column name to be updated and its new value, and optional conditions to specify the affected records.

[0059] The delete operation (DELETE) is an operation to remove records from a database table. The SQL statement corresponding to the delete operation is "DELETE FROM", which can be followed by conditions to specify the records to be deleted.

[0060] Among them, the select operation (SELECT) is an operation to retrieve data from a database table. The SQL statement corresponding to the select operation is "SELECT", followed by the column name to be retrieved, and the table from which these columns are selected. Conditions can be included to limit the returned records. When performing data synchronization, the select operation is usually used to obtain current data from the source data table and obtain it from the target data table for comparison and verification.

[0061] The embodiment of the present invention can realize automated data operation and verification process between data tables by automatically generating the above SQL statements.

[0062] S120, executing SQL statements, converting the field data of the source data table into the format of the target data table according to the field mapping rules, and comparing the consistency of the converted field data with the actual data in the target data table to obtain a comparison result.

[0063] Among them, the field mapping rule defines how the fields in the source data table correspond to the fields in the target data table. This rule describes how to convert the data format of the source field to the data format of the target field during the data transfer process. For example, if the field in the source table is in date format and the field in the target table is in string format, the mapping rule will define how to perform this format conversion.

[0064] Field data refers to the specific values ​​stored in each field in a database table. In each record, field data is the specific content of a column in the table. For example, the data of the "Name" field of a record may be "Zhang San".

[0065] The actual data refers to the specific values ​​currently existing in the target data table. During data synchronization, the actual data refers to the data already in the target table, which is used to compare with the converted source data to check consistency.

[0066] The comparison result refers to the conclusion drawn after comparing the converted source data with the actual data in the target data table. The result can indicate whether the two are consistent, or indicate which data items are inconsistent. The comparison result can be used to determine whether data update, insertion, or deletion operations are required.

[0067] The embodiments of the present invention can effectively convert source data and perform consistency checks with target data by executing SQL statements and operating according to these concepts, thereby ensuring the accuracy and integrity of data synchronization.

[0068] Optionally, the field mapping rule includes ignoring newly added fields in the target data table that are not related to the source data table when the data table is a heterogeneous table.

[0069] During data synchronization and integration, there may be structural differences between the source and target data tables, which are called "heterogeneous tables". Field mapping rules need to handle these differences to ensure that data can be converted and transmitted correctly.

[0070] The source and target tables may have different numbers of fields and field names. In particular, the target table may contain some newly added fields that do not exist in the source table. When more fields are found in the target table than in the source table, field mapping rules need to define how to handle these extra fields. In many cases, the newly added fields can be ignored. This means that when synchronizing data, no attempt will be made to fill or update these newly added fields because there is no corresponding data in the source table. When generating SQL statements or transformation logic, field mapping rules will clearly indicate which target fields should be mapped and which should be ignored. In this way, when processing insert or update operations, only data that has a corresponding field in the source table is processed.

[0071] The embodiment of the present invention ignores the newly added fields in the target table that are not related to the source data table. Since there is no need to artificially or unnecessarily create additional fields in the source table to match the target table, the synchronization logic can be simplified, and it is also helpful to maintain data integrity and consistency, ensuring that only the data actually existing in the source data table is synchronized.

[0072] It should be noted that if the newly added fields in the heterogeneous table are spliced ​​based on the fields in the source data table, a comparison is required.

[0073] Specifically, when performing data synchronization testing, the embodiment of the present invention first runs the SQL statement generation module to generate a SELECT statement for the target data table, thereby extracting the synchronized fields and data, and storing them in the list td_row. Next, another SELECT statement is executed to obtain fields and data from the source data table, and the data are stored in the list db_row. For each set of new table data comparison, a new class is created, and the field mapping rules of the source data table and the target data table are configured through the class to achieve data comparison.

[0074] The specific operation flow is as follows: First, td_row contains the field data to be compared in the target database; db_row contains the field data in the source database; according to the field mapping rules, the data in db_row is converted and stored in db_list, that is, the data in db_list is converted from the data in db_row through the field mapping rules, such as data_B_N corresponds to data_A_N. The target database field list td_row (such as data_C_1, data_C_2,..., data_C_N) is used to compare with db_list to check the consistency of field data and types. For homogeneous tables, the source and target database fields and data are completely consistent, which is expressed as td_row=db_row=db_list. In heterogeneous tables, the target database fields may be generated by the combination of source database fields or mapping rules, or the newly added fields may be irrelevant to the source database and can be ignored. In this case, db_row is converted to db_list through the mapping rules to make db_list consistent with td_row. When performing data conversion, the data in db_row is stored in db_list in sequence according to the field mapping rules. If you encounter inconsistent field types or precision (such as converting from JDouble to Decimal format), you need to ensure that the field type in db_list is consistent with td_row. If there are new fields in the target database, add their data to db_list in order. Finally, compare the data in db_list and td_row in order and output the comparison results.

[0075] For ease of understanding, here we combine Figure 2 The following is an example of the logical diagram of the field mapping and comparison process:

[0076] 1. For data tables with the same structure or without new fields:

[0077] Source data table data--db_row=[data_A_1,data_A_2,..data_A_N],

[0078] According to the field mapping rules, db_row is converted into:

[0079] "--db_list=[data_B_1,data_B_2,..data_B_N]",

[0080] Target data table data--td_row=[data_C_1,data_C_2,...,data_C_N].

[0081] 2. For heterogeneous data tables with newly added fields:

[0082] Source data table data--db_row=[data_A_1,data_A_2,..data_A_N],

[0083] According to the field mapping rules, db_row is converted into:

[0084] "--db_list=[data_B_1,data_B_2,..data_B_N]",

[0085] Fill in the new fields in order:

[0086] "db_list--db_list=[data_B_1,data_B_2,new_1,....new_N,data_B_N],

[0087] Target data table data --td_row=[data_C_1,data_C_2,...,data_C_N],

[0088] Finally, compare the db_lsit and td_row data to see if they are consistent.

[0089] Among them, the field mapping rule function:

[0090] “Function_Map(i):

[0091] db_list = []

[0092] If data_C_iis new_data:

[0093] db_list.apend[data_C_i]

[0094] else:

[0095] data_B_i=function_data_A_i(data_A_i)

[0096] db_list.apend[data_B_i]

[0097] return db_list".

[0098] Result comparison function:

[0099]

[0100] S130. Output success or failure information of incremental data synchronization according to the comparison result.

[0101] The success information of incremental data synchronization refers to the confirmation information that each operation (such as insert, update, delete) during the incremental synchronization process is correctly executed in the target database. The content of the success information may include the specific operation type, the number of records involved, the operation timestamp, and other relevant log information. The success information can be used for auditing and monitoring to ensure that the synchronization process is executed as expected and to provide reliable records for subsequent analysis.

[0102] Among them, the failure information of incremental data synchronization refers to the report that some operations failed to complete correctly or errors occurred during the incremental synchronization process. The content of the failure information can include the error type, error description, affected records or fields, the time when the error occurred, and possible causes (such as network interruption, data conflict, permission problem, etc.). Failure information can be used for fault diagnosis and repair, helping technicians identify and solve problems so that synchronization operations can be re-run, and can also help adjust synchronization strategies to avoid future failures.

[0103] After each incremental data synchronization is completed, the embodiment of the present invention generates a report based on the comparison results, summarizing the success and failure of the synchronization operation. Success information and failure information can be output through log files, monitoring dashboards or notification systems to report to relevant personnel (such as database administrators or developers).

[0104] The invention provides an incremental data synchronization test method, which includes: obtaining connection information of a source database and a target database, establishing a mapping relationship between a source data table and a target data table; based on the mapping relationship, automatically generating SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table; executing SQL statements, converting field data of the source data table into the format of the target data table according to field mapping rules, and comparing the consistency of the converted field data with the actual data in the target data table to obtain a comparison result; and outputting success or failure information of incremental data synchronization according to the comparison result. The invention automatically obtains the connection information of the source database and the target database, and establishes a mapping relationship between the source data table and the target data table, so as to efficiently generate SQL statements for data operations, which not only include insert, update and delete operations performed in the source table, but also cover the select operations performed in the source table and the target table. By executing these SQL statements, the field data of the source data table can be converted into the format of the target data table according to the preset field mapping rules, and the consistency of the converted data with the actual data in the target data table can be automatically compared. Finally, the success or failure information of incremental data synchronization is output, which significantly improves the efficiency and accuracy of data synchronization testing. The present invention eliminates the tedious steps of manually writing SQL and comparing data, allowing testers to focus more on verifying business logic, saving time and labor costs, while ensuring the reliability of test results.

[0105] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, based on the mapping relationship, automatically generating SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table, may specifically include:

[0106] Based on the mapping relationship, the primary key and non-primary key fields are automatically identified according to the source data table name and field characteristics selected by the user to generate SQL statements for performing insert, update and delete operations in the source data table, as well as SQL statements for performing select operations in the source data table and the target data table.

[0107] The source data table name refers to the name of a specific table in the database used to store original data.

[0108] Each table consists of multiple fields (columns). Field characteristics refer to the specific properties of each field, such as data type (integer, string, date, etc.), length, whether to allow null, default value, etc. These characteristics determine what type of data the field can store and how to process the data. For example, a field can be defined as an integer type, not allowed to be null, and has a default value.

[0109] Among them, the primary key is a combination of one or more fields that ensures the uniqueness of data by uniquely identifying each record in the table. The value of the primary key field must be unique and cannot be empty. The primary key is used to determine the uniqueness of the record, allowing the database to accurately retrieve, update, or delete specific records. During the data synchronization process, the primary key field is used to match the records in the source and target tables for update or delete operations.

[0110] Among them, non-primary key fields are other fields in the table that are not unique and are only used to store additional information about the data. Non-primary key fields can contain various types of information, such as the user's name, address, email, etc. In data operations, non-primary key fields are often used in update or insert operations to ensure that all relevant information of the record is accurately transmitted.

[0111] Include all fields in the SQL statement for the insert operation, both primary key and non-primary key fields, to ensure that complete new records are added to the target table.

[0112] The key in the SQL statement of the update operation is used to locate the record to be updated, and the non-primary key fields are used to specify which information will be updated.

[0113] Rely on primary key fields in SQL statements for delete operations to uniquely identify and delete specific records.

[0114] In the SQL statement of the select operation, you can select the primary key and / or non-primary key fields as needed to retrieve the required data.

[0115] The embodiment of the present invention automatically identifies the primary key and non-primary key fields of the source data table based on the mapping relationship, and can efficiently generate accurate SQL statements to perform insert, update, delete and select operations. This automation reduces human intervention, reduces the probability of errors, and ensures consistency and integrity during data synchronization. In addition, it can quickly adapt to different data tables and their field characteristics selected by users, and flexibly meet diverse testing needs. This mechanism not only improves the efficiency of data synchronization testing, but also improves its accuracy, making data management more reliable and controllable.

[0116] Optionally, an embodiment of the present invention can automatically identify primary key and non-primary key fields based on a mapping relationship and according to the source data table name and field characteristics selected by the user, so as to generate an INSERT statement for inserting random data according to the structure of the source data table, generate a random UPDATE statement for updating non-primary key fields, generate a DELETE statement for deleting records, and generate a SELECT statement for data comparison.

[0117] like Figure 3 The logical block diagram of SQL statement generation shown in the figure, in order to efficiently generate and execute SQL statements in data synchronization testing, the embodiment of the present invention will automatically identify the primary key and non-primary key fields according to the source data table name and its field characteristics selected by the user, and then perform the following operations according to the structure of the source data table: First, according to the source database table name selected by the user, an insert statement (INSERT) is randomly generated, and the statement is executed to insert a piece of data into the source data table. Then, using the mapping relationship (MAP) between the source data table and the target data table, the primary key field and its data of each table are automatically extracted to generate corresponding SELECT statements. These SELECT statements are used to extract data from the source and target data tables for use by subsequent data comparison modules. Based on the information previously inserted into the source data table, non-primary key fields are automatically identified and extracted, the data of these fields are randomly updated, corresponding update statements (UPDATE) are generated, and these statements are executed. At the same time, the primary key field and its data are extracted again, and a new SELECT statement is generated to compare the data in the source data table and the target data table to ensure the correctness of the update operation. Finally, based on the information previously inserted into the source data table, the primary key field and its data are extracted, and a delete statement (DELETE) is generated and executed to remove the record. Subsequently, a SELECT statement for comparison is generated again, and the comparison module verifies the impact of the delete operation on the source data table and the target data table.

[0118] The embodiment of the present invention fully utilizes the structure and field characteristics of the source data table when generating and executing SQL statements, thereby ensuring the accuracy and effectiveness of the data synchronization test.

[0119] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, comparing the consistency between the converted field data and the actual data in the target data table may specifically include:

[0120] Use multithreading technology to set up independent threads for each field comparison.

[0121] In order to improve the execution efficiency of the test script and meet the business requirements for a large number of field updates, the embodiment of the present invention uses multi-threading technology to allocate an independent thread to each field for data comparison. In this way, the comparison task of each field can be processed in parallel, thereby speeding up the overall comparison process and improving processing efficiency.

[0122] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, outputting success or failure information of incremental data synchronization according to the comparison result may specifically include:

[0123] If the synchronization is successful and the data is consistent, the result is recorded in the success list.

[0124] Specifically, if the synchronization is successful and the data is consistent (ie, db_list is equal to td_row), the result is recorded in the success list suc_list.

[0125] In case of synchronization failure, the failed source data table name and primary key information are recorded.

[0126] Specifically, if the synchronization fails, the name and primary key information of the failed source data table are recorded in the failure list err_table.

[0127] In case of data inconsistency, the inconsistent fields and their data are recorded.

[0128] Specifically, if the synchronization is successful but the data is inconsistent, all inconsistent fields and their data are recorded in diff_row, and these inconsistent information are uniformly output after all fields are compared.

[0129] The embodiment of the present invention helps to improve the efficiency and accuracy of the test by recording different test results in a classified manner. When the synchronization is successful and the data is consistent, the results are recorded in a success list to facilitate quick identification and confirmation of which data has been correctly synchronized, thereby reducing unnecessary repeated verification work. When the synchronization fails, the failed source data table name and primary key information are specifically recorded to make the problem location more accurate and speed up the troubleshooting process. In the case of inconsistent data, the inconsistent fields and their data are recorded in detail to help developers and testers quickly find the root cause of the difference and make targeted repairs. Overall, this systematic result recording method makes data synchronization testing more organized and efficient, and ensures the accuracy of synchronization.

[0130] The embodiment of the present invention also provides an incremental data synchronization test system, whose program framework is as follows Figure 4 As shown, it may include a comparison library module, an SQL statement generation module, an execution comparison module and a result output module.

[0131] The comparison library module is used to input the address information of the source database and the target database in the comparison library according to business needs, and define the corresponding relationship between each source data table and the target data table.

[0132] The SQL statement generation module is used to connect to the corresponding database using the address information of the source database. Based on the primary key analysis of the source data table structure, it automatically generates INSERT, UPDATE, and DELETE SQL statements for comparing data tables and automatically executes these statements.

[0133] The execution comparison module is used to connect to the corresponding database using the address information of the target database. According to the primary key analysis of the target data table structure, the SELECT SQL statement is automatically generated and executed. According to the corresponding relationship of the data table, the field name and data of the source data table are obtained, and then the fields and data of the target data table are generated according to the mapping rules. Using multi-threading technology, a separate thread is set for each field for data comparison.

[0134] The result output module is used to output the test comparison results after synchronization and record all inconsistent fields and their data in a file.

[0135] The incremental data synchronization test system provided by the embodiment of the present invention realizes incremental data synchronization based on Kafka, and automatically analyzes and extracts the structure of the source data table and the target data table, including primary key and non-primary key fields. By automatically generating and executing SQL statements for adding, deleting, modifying and checking, multi-threading technology is used to efficiently execute scripts and output comparison results, thereby verifying the correctness and consistency of data after synchronization. The system is also versatile and suitable for homogeneous and heterogeneous table structures. For newly added synchronization tables, users only need to configure the mapping relationship of the corresponding fields, which simplifies the requirements for business knowledge and SQL operations of testers. In addition, the system is reusable and can be used for multiple and large-scale regression tests to ensure the continuous consistency and accuracy of data tables.

[0136] Although operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0137] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0138] Corresponding to the above method embodiment, the embodiment of the present invention also provides an incremental data synchronization test device, whose structure is as follows: Figure 5As shown, it may include: a mapping relationship establishing unit 10, an SQL statement generating unit 20, a comparison result obtaining unit 30 and an incremental data information synchronization information obtaining unit 40.

[0139] The mapping relationship establishing unit 10 is used to obtain the connection information of the source database and the target database, and establish the mapping relationship between the source data table and the target data table.

[0140] The SQL statement generating unit 20 is used to automatically generate SQL statements for executing insert, update and delete operations in the source data table, and SQL statements for executing select operations in the source data table and the target data table based on the mapping relationship.

[0141] The comparison result obtaining unit 30 is used to execute SQL statements, convert the field data of the source data table into the format of the target data table according to the field mapping rules, and compare the consistency between the converted field data and the actual data in the target data table to obtain the comparison result.

[0142] The incremental data information synchronization information obtaining unit 40 is used to output success or failure information of the incremental data synchronization according to the comparison result.

[0143] Optionally, the SQL statement generation unit 20 can be specifically used to automatically identify primary key and non-primary key fields based on the mapping relationship and in accordance with the source data table name and field characteristics selected by the user, so as to generate SQL statements for performing insert, update and delete operations in the source data table, as well as SQL statements for performing select operations in the source data table and the target data table.

[0144] Optionally, the SQL statement generation unit 20 can be specifically used to automatically identify primary key and non-primary key fields based on a mapping relationship and according to the source data table name and field characteristics selected by the user, so as to generate an INSERT statement for inserting random data according to the structure of the source data table, generate a random UPDATE statement for updating non-primary key fields, generate a DELETE statement for deleting records, and generate a SELECT statement for data comparison.

[0145] Optionally, the comparison result obtaining unit 30 may be specifically configured to use a multi-threading technology to set an independent thread for the comparison of each field.

[0146] Optionally, the field mapping rule includes ignoring newly added fields in the target data table that are not related to the source data table when the data table is a heterogeneous table.

[0147] Optionally, the incremental data information synchronization information acquisition unit 40 can be specifically used to record the results in a success list when the synchronization is successful and the data is consistent; when the synchronization fails, record the failed source data table name and primary key information; when the data is inconsistent, record the inconsistent fields and their data.

[0148] Optionally, the connection information includes database address, port number, user name and password.

[0149] The invention provides an incremental data synchronization test device, which includes: obtaining connection information of a source database and a target database, establishing a mapping relationship between a source data table and a target data table; based on the mapping relationship, automatically generating SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table; executing SQL statements, converting field data of the source data table into the format of the target data table according to field mapping rules, and comparing the consistency of the converted field data with the actual data in the target data table to obtain a comparison result; and outputting success or failure information of incremental data synchronization according to the comparison result. The invention can efficiently generate SQL statements for data operations by automatically obtaining connection information of the source database and the target database, and establishing a mapping relationship between the source data table and the target data table, which not only includes insert, update and delete operations performed in the source table, but also covers select operations performed in the source table and the target table. By executing these SQL statements, the field data of the source data table can be converted into the format of the target data table according to the preset field mapping rules, and the consistency of the converted data with the actual data in the target data table can be automatically compared. Finally, the success or failure information of incremental data synchronization is output, which significantly improves the efficiency and accuracy of data synchronization testing. The present invention eliminates the tedious steps of manually writing SQL and comparing data, allowing testers to focus more on verifying business logic, saving time and labor costs, while ensuring the reliability of test results.

[0150] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0151] The incremental data synchronization testing device includes a processor and a memory. The above-mentioned mapping relationship establishment unit 10, SQL statement generation unit 20, comparison result acquisition unit 30 and incremental data information synchronization information acquisition unit 40 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0152] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set. By adjusting the kernel parameters, the connection information of the source database and the target database can be automatically obtained, and the mapping relationship between the source data table and the target data table can be established. SQL statements for data operations can be efficiently generated, including not only insert, update and delete operations performed in the source table, but also selection operations performed in the source table and the target table. By executing these SQL statements, the field data of the source data table can be converted to the format of the target data table according to the preset field mapping rules, and the consistency of the converted data with the actual data in the target data table can be automatically compared. Finally, the success or failure information of the incremental data synchronization is output, which significantly improves the efficiency and accuracy of the data synchronization test.

[0153] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the incremental data synchronization test method is implemented.

[0154] An embodiment of the present invention provides a processor, which is used to run a program, wherein the incremental data synchronization test method is executed when the program is running.

[0155] like Figure 6 As shown, an embodiment of the present invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003; wherein the processor 1001 and the memory 1002 communicate with each other through the bus 1003; the processor 1001 is used to call the program instructions in the memory 1002 to execute the above-mentioned incremental data synchronization test method. The electronic device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0156] The present invention also provides a computer program product, which, when executed on an electronic device, is suitable for executing a program that initializes the incremental data synchronization test method steps.

[0157] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0158] In a typical configuration, an electronic device includes one or more processors (CPU), a memory, and a bus. The electronic device may also include an input / output interface, a network interface, and the like.

[0159] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0160] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0162] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0163] In the description of the present invention, it should be understood that the terms "up", "down", "front", "back", "left" and "right" etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the positions or elements referred to must have specific directions, be constructed and operate in specific directions. Therefore, they should not be understood as limitations of the present invention.

[0164] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0165] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0166] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for testing incremental data synchronization, characterized in that: include: Obtaining connection information of a source database and a target database, and establishing a mapping relationship between the source data table and the target data table; Based on the mapping relationship, automatically generate SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table; Execute the SQL statement, convert the field data of the source data table into the format of the target data table according to the field mapping rule, and compare the consistency of the converted field data with the actual data in the target data table to obtain a comparison result; The success or failure information of the incremental data synchronization is output according to the comparison result.

2. The method according to claim 1, characterized in that The automatically generating, based on the mapping relationship, SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table, comprises: Based on the mapping relationship, primary key and non-primary key fields are automatically identified according to the source data table name and field characteristics selected by the user to generate SQL statements for performing insert, update and delete operations in the source data table, as well as SQL statements for performing select operations in the source data table and the target data table.

3. The method according to claim 2, characterized in that Based on the mapping relationship, according to the source data table name and field characteristics selected by the user, the primary key and non-primary key fields are automatically identified to generate SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table, including: Based on the mapping relationship, according to the source data table name selected by the user and its field characteristics, the primary key and non-primary key fields are automatically identified to generate an INSERT statement for random data insertion according to the structure of the source data table, generate a random UPDATE statement for updating non-primary key fields, generate a DELETE statement for deleting records, and generate a SELECT statement for data comparison.

4. The method according to claim 1, characterized in that: The comparison of the converted field data and the actual data in the target data table for consistency includes: Use multithreading technology to set up independent threads for each field comparison.

5. The method according to claim 1, characterized in that The field mapping rule includes ignoring newly added fields in the target data table that are not related to the source data table when the data table is a heterogeneous table.

6. The method according to claim 1, characterized in that Outputting success or failure information of incremental data synchronization according to the comparison result includes: If the synchronization is successful and the data is consistent, the result is recorded in the success list; In case of synchronization failure, the failed source data table name and primary key information are recorded; In case of data inconsistency, the inconsistent fields and their data are recorded.

7. The method according to any one of claims 1 to 6, characterized in that The connection information includes database address, port number, user name and password.

8. An incremental data synchronization test device, characterized in that: include: Mapping relationship establishment unit, SQL statement generation unit, comparison result acquisition unit and incremental data information synchronization information acquisition unit, The mapping relationship establishing unit is used to obtain connection information of the source database and the target database, and establish a mapping relationship between the source data table and the target data table; The SQL statement generating unit is used to automatically generate SQL statements for performing insert, update and delete operations in the source data table, and SQL statements for performing select operations in the source data table and the target data table based on the mapping relationship; The comparison result obtaining unit is used to execute the SQL statement, convert the field data of the source data table into the format of the target data table according to the field mapping rule, and compare the consistency of the converted field data with the actual data in the target data table to obtain the comparison result; The incremental data information synchronization information obtaining unit is used to output success or failure information of incremental data synchronization according to the comparison result.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the incremental data synchronization test method according to any one of claims 1 to 7 is implemented.

10. An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein: The processor and the memory communicate with each other via the bus; The processor is used to call the program instructions in the memory to execute the incremental data synchronization test method as described in any one of claims 1 to 7.

Images