A method and apparatus for detecting characteristics of a distributed database

Abstract

This application discloses a method and apparatus for detecting the characteristics of a distributed database, used to detect the ACID characteristics of the database. The method includes: creating a distributed table on the database for a specified characteristic of the distributed database, and concurrently performing data operations corresponding to the specified characteristic on the distributed table; the specified characteristic is a set of target characteristics or persistence; the set of target characteristics includes atomicity, consistency, and isolation; the data operation corresponding to the set of target characteristics is any one of update, delete, and insert operations; the data operation corresponding to persistence is an insert operation; if the specified characteristic is a set of target characteristics, determining whether the database has a problem with the specified characteristic based on the consistency of the data with the initial data on the distributed table during the execution of the data operation; if the specified characteristic is persistence, determining whether the database has a problem with persistence based on the consistency of the result after the execution of the data operation with the log record.

Description

Technical Field

[0001] This application relates to the field of feature detection technology, and in particular to a feature detection method and apparatus for a distributed database. Background Technology

[0002] Distributed databases involve multiple machines and multiple database shards. In order to ensure that transactions and data on multiple database shards can be coordinated and consistent, it is necessary to effectively guarantee the ACID properties of distributed databases, namely Atomicity, Consistency, Isolation, and Durability. Therefore, accurate testing of these four properties of distributed databases is very important.

[0003] Existing detection methods primarily rely on users directly verifying application code. However, due to the high complexity of applications and the potential for logical errors in application code, when the detection process suspects violations of one or more of the principles of atomicity, consistency, isolation, and durability in a distributed database, it's difficult to promptly determine whether the issue stems from logical problems in the application code or from a problem within the distributed database itself. Therefore, existing detection methods are unreliable and relatively inefficient. Summary of the Invention

[0004] In view of the shortcomings of the prior art, this application provides a characteristic detection method and apparatus for distributed databases to solve the problems that the existing detection methods are too cumbersome and cannot guarantee the accuracy of the detection results.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] The first aspect of this application provides a method for detecting the characteristics of a distributed database, including:

[0007] For a specified characteristic of the distributed database, a distributed table is created on the distributed database, and data operations corresponding to the specified characteristic are concurrently executed on the distributed table; wherein, the specified characteristic is a set of target characteristics or persistence; the set of target characteristics includes atomicity, consistency, and isolation; the data operation corresponding to the set of target characteristics is any one of update, delete, and insert operations; the data operation corresponding to persistence is an insert operation;

[0008] If the specified characteristic is the target characteristic set, then based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table in the specified indicator, it is determined whether the distributed database has a problem with the specified characteristic;

[0009] If the specified characteristic is persistence, then based on the consistency between the result of performing the data operation and the log record, it is determined whether the distributed database has a problem with persistence.

[0010] Optionally, in the above method, the specified characteristic is the target characteristic set, and the data operation is an update operation. The step of creating a distributed table on the distributed database for the specified characteristic of the distributed database, and concurrently performing the data operation corresponding to the specified characteristic on the distributed table, includes:

[0011] A first distributed table is created on the distributed database, with the account number as the hash sharding key; wherein, the first distributed table stores multiple accounts, each account stores one record, and the initial amount in each record is the same;

[0012] For the first distributed table, multiple update operations are executed concurrently; wherein, when one update operation is executed, a random amount is first randomly deducted from one of the accounts, and then the random amount is randomly added to another account.

[0013] The step of determining whether the distributed database has a problem with a specified characteristic based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table on a specified indicator includes:

[0014] During the concurrent execution of multiple update operations, information about the data on the first distributed table is obtained multiple times to obtain multiple first output results; wherein, the first output result includes the timestamp when the data is obtained and the total amount recorded on the first distributed table at that time;

[0015] Filter out the first target output results from each of the first output results, and select the first target output results whose total amount is not equal to the total amount recorded on the initial first distributed table;

[0016] If there are multiple first target output results with consecutive timestamps, then it is determined that there is an atomicity problem in the distributed database;

[0017] If there is only one first target output result, or if there are multiple first target output results with discontinuous timestamps, then it is determined that there is a problem with the consistency or isolation of the distributed database.

[0018] Optionally, in the above method, the specified characteristic is the target characteristic set, and the data operation is a deletion operation. The step of creating a distributed table on the distributed database for the specified characteristic of the distributed database, and concurrently performing the data operation corresponding to the specified characteristic on the distributed table, includes:

[0019] A first source table and a first target table are created on the distributed database; wherein the first source table and the first target table are two distributed tables that both have an account field but have different sharding keys; the account field of the first source table stores multiple records and has a unique index; the account field of the first target table does not have a unique index and does not store any records.

[0020] For the first source table and the first target table, multiple deletion operations are performed concurrently; wherein, when a deletion operation is performed, a record in the first source table is randomly read, and after the record is inserted into the first target table, the record is deleted from the first source table;

[0021] The step of determining whether the distributed database has a problem with a specified characteristic based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table on a specified indicator includes:

[0022] During the concurrent execution of multiple deletion operations, information about the data on the first source table and the first target table is continuously obtained multiple times to obtain multiple second output results; wherein, the second output result is the timestamp when the data is obtained, and the sum of the number of records on the first source table and the first target table at that time;

[0023] Filter out the second target output results from the second output results, where the sum of the quantities is not equal to the initial sum of quantities; wherein, the initial sum of the number of records refers to the sum of the number of records in the initial first source table and the first target table;

[0024] If there are multiple second target output results with consecutive timestamps, then the atomicity of the distributed database is determined to be problematic.

[0025] If there is only one second target output result, or if there are multiple second target output results with discontinuous timestamps, then it is determined that there is a problem with the consistency or isolation of the distributed database.

[0026] Optionally, in the above method, the specified characteristic is the target characteristic set, and the data operation is a first insertion operation. The step of creating a distributed table on the distributed database for the specified characteristic of the distributed database, and concurrently performing the data operation corresponding to the specified characteristic on the distributed table, includes:

[0027] A second source table and a second target table are created on the distributed database; wherein the second source table and the second target table are two distributed tables with the same fields; the second source table has an ID field, with each ID corresponding to one record; the sharding key of the second source table and the second target table is ID, and random sharding is performed, but the ID field of the second target table does not have a unique index.

[0028] For the second source table and the second target table, multiple first insertion operations are executed concurrently; wherein, when the first insertion operation is executed, the IDs corresponding to all records in the second source table are incremented by 1, and then all records in the second source table are inserted under the corresponding IDs in the second target table;

[0029] The step of determining whether the distributed database has a problem with a specified characteristic based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table on a specified indicator includes:

[0030] During the concurrent execution of multiple first insertion operations, information about the data on the second target table is continuously obtained multiple times to obtain multiple third output results; wherein, the third output results include the number of records under each ID in the second target table after the execution of the first insertion operation;

[0031] If the number of records under each ID in the second target table contained in any of the third output results is inconsistent, then it is determined that the distributed database has a problem with one or more of the following: atomicity, consistency, and isolation.

[0032] Optionally, in the above method, the specified characteristic is persistence, and the data operation is a second insert operation. The step of creating a distributed table on the distributed database for the specified characteristic of the distributed database, and concurrently performing the data operation corresponding to the specified characteristic on the distributed table, includes:

[0033] A third distributed table is created on the distributed database; wherein the second distributed table uses an auto-incrementing column as the primary key and the thread number as the sharding key;

[0034] Multiple threads are used to insert records into the third distributed table concurrently, and the cumulative number of times each thread inserts records is recorded in the third distributed table.

[0035] The step of determining whether the distributed database has a persistence problem based on the consistency between the results of the data operation and the log records includes:

[0036] If the cumulative number of times a thread inserts a record in the third distributed table differs from the cumulative number of times a thread inserts a record in the log record by more than 1, then the distributed database is determined to have a persistence problem.

[0037] A second aspect of this application provides a characteristic detection device for a distributed database, comprising:

[0038] An operation unit is configured to create distributed tables on the distributed database based on specified characteristics of the distributed database, and concurrently execute data operations corresponding to the specified characteristics on the distributed tables; wherein, the specified characteristics are a set of target characteristics or persistence; the set of target characteristics includes atomicity, consistency, and isolation; the data operation corresponding to the set of target characteristics is any one of update, delete, and insert operations; and the data operation corresponding to persistence is an insert operation.

[0039] The first determining unit is configured to, if the specified characteristic is the target characteristic set, determine whether the distributed database has a problem with the specified characteristic based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table on the specified indicator.

[0040] The second determining unit is used to determine whether the distributed database has a problem with the persistence if the specified characteristic is persistence, based on the consistency between the result of performing the data operation and the log record.

[0041] Optionally, in the above-described apparatus, the specified characteristic is the target characteristic set, and the data operation is an update operation; the operation unit includes:

[0042] The first creation unit is used to create a first distributed table on the distributed database with the account number as the hash sharding key; wherein, the first distributed table stores multiple accounts, each account stores one record, and the initial amount in each record is the same;

[0043] An update operation unit is configured to execute multiple update operations concurrently for the first distributed table; wherein, when one update operation is executed, a random amount is first randomly deducted from one of the accounts, and then the random amount is randomly added to another account;

[0044] The first determining unit includes:

[0045] The first acquisition unit is used to continuously acquire information about the data on the first distributed table multiple times during the concurrent execution of multiple update operations, and obtain multiple first output results; wherein, the first output result includes the timestamp when the data is acquired and the total amount recorded on the first distributed table at that time;

[0046] The first filtering unit is used to filter out a first target output result from each of the first output results, where the total amount is not equal to the total amount recorded on the initial first distributed table.

[0047] The first problem-determining unit is used to determine that the atomicity of the distributed database is problematic if there are multiple first target output results with consecutive timestamps.

[0048] The second problem-determination unit is used to determine that there is a problem with the consistency or isolation of the distributed database if there is only one first target output result, or if there are multiple first target output results with discontinuous timestamps.

[0049] Optionally, in the above-described apparatus, the specified characteristic is the target characteristic set, and the data operation is a deletion operation. The operation unit includes:

[0050] The second creation unit is used to create a first source table and a first target table on the distributed database; wherein the first source table and the first target table are two distributed tables that both have an account field but have different sharding keys; the account field of the first source table stores multiple records and has a unique index; the account field of the first target table does not have a unique index and does not store any records.

[0051] The deletion operation unit is used to perform multiple deletion operations concurrently on the first source table and the first target table; wherein, when a deletion operation is executed, a record is randomly read from the first source table, and after inserting the record into the first target table, the record is deleted from the first source table;

[0052] The first determining unit includes:

[0053] The second acquisition unit is used to continuously acquire information about the data on the first source table and the first target table multiple times during the concurrent execution of multiple deletion operations, and obtain multiple second output results; wherein the second output result is the timestamp when the data is acquired, and the sum of the number of records on the first source table and the first target table at that time;

[0054] The second filtering unit is used to filter out a second target output result from the second output result whose total number is not equal to the initial total number; wherein, the initial total number of records refers to the initial total number of records in the first source table and the first target table;

[0055] The third problem-determining unit is used to determine that the atomicity of the distributed database is problematic if there are multiple second target output results with consecutive timestamps.

[0056] The fourth problem determination unit is used to determine that there is a problem with the consistency or isolation of the distributed database if there is only one second target output result, or if there are multiple second target output results with discontinuous timestamps.

[0057] Optionally, in the above-described apparatus, the specified characteristic is the target characteristic set, and the data operation is a first insertion operation. The operation unit includes:

[0058] The third creation unit is used to create a second source table and a second target table on the distributed database; wherein the second source table and the second target table are two distributed tables with the same fields; the second source table has an ID field, and each ID corresponds to one record; the sharding key of the second source table and the second target table is ID, and random sharding is used, but the ID field of the second target table does not have a unique index.

[0059] The first insertion operation unit is used to execute multiple first insertion operations concurrently for the second source table and the second target table; wherein, when the first insertion operation is executed, the IDs corresponding to all records in the second source table are incremented by 1, and then all records in the second source table are inserted under the corresponding IDs in the second target table;

[0060] The first determining unit includes:

[0061] The third acquisition unit is used to continuously acquire information about the data on the second target table multiple times during the concurrent execution of multiple first insertion operations, and obtain multiple third output results; wherein, the third output results include the number of records under each ID in the second target table after the execution of the first insertion operation;

[0062] The fifth problem determination unit is used to determine that if the number of records under each ID in the second target table contained in any of the third output results is inconsistent, then the distributed database has a problem in one or more of the following: atomicity, consistency, and isolation.

[0063] Optionally, in the above-described apparatus, the specified characteristic is the persistence, and the data operation is a second insertion operation; the operation unit includes:

[0064] The fourth creation unit is used to create a third distributed table on the distributed database; wherein the second distributed table uses an auto-incrementing column as the primary key and the thread number as the sharding key;

[0065] The second insertion operation unit is used to insert records into the third distributed table in a concurrent manner through multiple threads, and to record the cumulative value of the number of times each thread inserts records in the third distributed table;

[0066] The second confirmation unit includes:

[0067] The second confirmation subunit is used to determine that there is a persistence problem in the distributed database if the cumulative value of the number of times the thread inserts records in any one of the records in the third distributed table has an error greater than 1 compared with the cumulative value of the number of times the thread inserts records in the log.

[0068] This application provides a method for detecting the characteristics of a distributed database. For a specified characteristic of the distributed database, a distributed table is created on the distributed database, and data operations corresponding to the specified characteristic are concurrently executed on the distributed table. The specified characteristic is either a set of target characteristics or persistence. The set of target characteristics includes atomicity, consistency, and isolation. The data operation corresponding to the set of target characteristics is any one of update, delete, and insert operations. The data operation corresponding to persistence is an insert operation. Therefore, the detection process only requires concurrent execution of regular database operations. If the specified characteristic is the set of target characteristics, the consistency between the results of the executed data operations and the data on the initial distributed table in the specified metrics determines whether the distributed database has a problem with the specified characteristic. If the specified characteristic is persistence, since the logs record the operations, the consistency between the results of the executed data operations and the log records determines whether the distributed database has a problem with persistence. Thus, by concurrently executing regular database operations and based on the changes in the operation results, a simple and convenient detection method is achieved, and the detection results are not affected by application code, effectively ensuring the accuracy of the detection. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0070] Figure 1 A flowchart illustrating a characteristic detection method for a distributed database provided in an embodiment of this application;

[0071] Figure 2 A flowchart illustrating a method for detecting a set of target features, provided in another embodiment of this application;

[0072] Figure 3 A flowchart illustrating another method for detecting a set of target features provided in another embodiment of this application;

[0073] Figure 4 A flowchart illustrating yet another method for detecting a set of target features provided in another embodiment of this application;

[0074] Figure 5 A flowchart illustrating another method for detecting a set of target features provided in another embodiment of this application;

[0075] Figure 6 A flowchart of a method for detecting persistence provided in another embodiment of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] This application provides a characteristic detection method for distributed databases, used to examine a target characteristic set, wherein, for example... Figure 1 As shown, the specific steps include:

[0079] S101. Based on the specified characteristics of the distributed database, create a distributed table on the distributed database and concurrently perform data operations corresponding to the specified characteristics on the distributed table.

[0080] First, it should be noted that performing a data operation means performing a transaction corresponding to that data operation.

[0081] The specified characteristic can be a set of target characteristics or durability. The set of target characteristics includes atomicity, consistency, and isolation. Therefore, in this application, atomicity, consistency, and isolation are tested together, while durability is tested separately.

[0082] The data operation corresponding to the target characteristic set is any one of the update, delete, and insert operations. Since, when a distributed database is functioning correctly in terms of atomicity, consistency, and isolation, transactions and data should be consistent before and after performing update, delete, and insert operations on tables in the distributed database, the consistency of transactions and data before and after performing these operations can be used to determine whether there are problems with the atomicity, consistency, and isolation of the distributed database.

[0083] In this context, the data operation corresponding to persistence is the insertion operation.

[0084] Because durability requires that changes made to the database by a transaction be permanently saved in the database after the transaction is completed and will not be rolled back, insert operations can be used to verify whether data has been inserted and is persistently saved in the database.

[0085] It should be noted that the insertion operation mainly refers to the insertion of data, but the specific implementation process of data insertion can be different. Optionally, the insertion operation corresponding to the target feature set can be the first insertion operation. The insertion operation corresponding to persistence can be the second insertion operation. It should be noted that there are certain differences in the specific implementation of the three insertion operations.

[0086] S102. Determine whether the specified characteristic is persistent.

[0087] If the specified characteristic is the target characteristic set, then step S103 is executed. If the specified characteristic is persistence, then step S104 is executed.

[0088] S103. Based on the consistency of the information of the data on the distributed table obtained during the execution of data operations with the information of the initial distributed table on specified indicators, determine whether there are problems with the specified characteristics of the distributed database.

[0089] It's important to note that when the atomicity, consistency, and isolation of a distributed database are normal, the data should be consistent with a specified metric before and after data operations, such as the total amount of data or the sum of the values ​​in the data records. Specifically, different metrics can be specified depending on the data operation. By checking whether this specified metric is consistent before and after the data operation, it can be determined whether there are any problems with the specified characteristics of the distributed database.

[0090] Specifically, when the specified characteristic is the target characteristic set, update, delete, and insert operations can be used to detect it.

[0091] Optionally, in another embodiment of this application, the specified characteristic is a target characteristic set, the data operation is an update operation, and one implementation of step S101 is as follows: Figure 2 As shown, it includes:

[0092] S201. Create the first distributed table on the distributed database with the account as the hash sharding key.

[0093] The first distributed table stores multiple accounts, each account contains one record, and the initial amount in each record is the same.

[0094] S202. For the first distributed table, perform multiple update operations concurrently.

[0095] In this embodiment, an update operation involves first randomly deducting a random amount from one account and then randomly adding a random amount to another account. Specifically, an update operation consists of two operations: one is randomly deducting a random amount from account A, and the other is randomly adding that random amount to account B. It should be noted that deducting or adding an amount corresponds to updating the amount in the record corresponding to the account in the first distributed table. If either operation fails, the entire transaction is rolled back, and the records are restored.

[0096] Since the two accounts in an update operation transaction are both random, it is highly probable that the transfer operation will fall on different shards of different databases, thus ensuring that the transaction is a distributed transaction.

[0097] It should be noted that the data operation in this embodiment is an update operation, and the update operation is the data operation corresponding to the target feature set. Therefore, in this embodiment, the specified feature is the target feature set, so step S102 is not required. Correspondingly, the implementation method of step S103 in this embodiment is also referred to... Figure 2 This includes the following steps:

[0098] S203. During the concurrent execution of multiple update operations, information about the data on the first distributed table is obtained multiple times to obtain multiple first output results.

[0099] The first output includes the timestamp when the data acquisition was completed and the total amount recorded on the first distributed table at that time. In other words, the information of the data acquired from the first distributed table is the timestamp of the data update and the total amount of the data record.

[0100] S204. Filter out the first target output results from each first output result whose total amount is not equal to the total amount recorded on the initial first distributed table.

[0101] It should be noted that, under normal circumstances, the amounts in each account will change as the program concurrently performs update operations, and the number of accounts with amounts not equal to the initial amount will gradually increase. However, under the read consistency isolation level, the sum of all account amounts should always equal the initial total amount. Therefore, by filtering out the first output result whose total amount does not equal the total amount recorded in the initial table, we can use it as the first target output result to analyze the problems it presents.

[0102] S205. Determine if the first target output result exists.

[0103] If the first target output result exists, proceed to step S206. If the first target output result does not exist, it is determined that the atomicity, consistency, and isolation of the distributed database are not problematic.

[0104] S206. Determine if there are multiple first target output results containing consecutive timestamps.

[0105] It should be noted that the continuous timestamps mentioned in the embodiments of this application refer to consecutive timestamps among all the acquired timestamps, which can be understood as the acquisition order of the timestamps being consecutive.

[0106] Atomicity means that all operations within a transaction either complete entirely or fail entirely; it's impossible for them to stall at any point in the process. If an error occurs during transaction execution, it will be rolled back to the state before the transaction began, as if the transaction had never been executed. Therefore, when atomicity is compromised, if a transaction stalls or encounters an error, the total amount recorded in the results of subsequent update operations will not equal the initial total amount. Thus, upon determining that there are multiple consecutive first target output results with matching timestamps, step S207 is executed.

[0107] Consistency and isolation can be simply understood as concurrent transactions, which can be executed sequentially like serial transactions. Therefore, when there are problems with consistency and isolation, there will be transactions that are executed at the same time, resulting in the total amount recorded in the output results of some update operations not being equal to the initial total amount. So when there are multiple first target output results with consecutive timestamps, i.e., there is only one first target output result, or when there are multiple first target output results with non-consecutive timestamps, step S208 is executed.

[0108] S207. It was determined that there was a problem with the atomicity of the distributed database.

[0109] S208. It has been determined that there are problems with the consistency or isolation of the distributed database.

[0110] Optionally, in another embodiment of this application, when the specified characteristic is a target characteristic set and the data operation is a deletion operation, one implementation of step S101 is as follows: Figure 3 As shown, it includes:

[0111] S301. Create the first source table and the first target table on the distributed database.

[0112] The first source table and the first target table are two distributed tables, both with an account field but with different sharding keys, to ensure that the transaction for the deletion operation is a concurrent transaction.

[0113] The first source table stores multiple records under the account field and has a unique index. It uses the account field as its primary key and hash sharding key. The first target table uses an auto-incrementing column as its primary key, and all other fields are the same as the first source table to facilitate data insertion from the first source table. However, it does not store any records in the first target table. Furthermore, the account field in the first target table does not have a unique index, which could lead to duplicate records being inserted into the first target table if atomicity cannot be guaranteed in a distributed database, thus exposing atomicity issues.

[0114] S302. For the first source table and the first target table, perform multiple deletion operations concurrently.

[0115] In a delete operation, a record is randomly read from the first source table, inserted into the first target table, and then deleted from the first source table. Thus, a delete operation transaction consists of three operations. Similarly, if any one of the three operations fails during the execution of any delete operation transaction, the entire transaction is rolled back.

[0116] It should be noted that the data operation in this embodiment is a deletion operation, and the deletion operation is a data operation corresponding to the target feature set. Therefore, in this embodiment, the specified feature is the target feature set, so step S102 is not required. Correspondingly, the implementation method of step S103 in this embodiment is also referred to... Figure 3 This includes the following steps:

[0117] S303. During the concurrent execution of multiple deletion operations, information about the data on the first source table and the first target table is continuously obtained multiple times to obtain multiple second output results.

[0118] The second output includes the timestamp when the data acquisition was completed, as well as the total number of records in the first source table and the first target table at that time.

[0119] S304. Filter out the second target output results from the second output results whose total quantity is not equal to the initial total quantity.

[0120] The initial total number of records refers to the initial first source table and the total number of records in the first source table.

[0121] Similarly, assuming all characteristics of the target feature set are normal, the number of records in each table will change as the program executes concurrently. The number of records in the first source table will gradually decrease, while the number of records in the first target table will gradually increase. However, the total number of records in both tables should always equal the initial total number of records.

[0122] S305. Determine if a second target output result exists.

[0123] It should be noted that the principles of steps S305 to S308 are similar to those of steps S205 to S208, so they will not be repeated here.

[0124] If it is determined that there is a second target output result, then step S306 is executed.

[0125] S306. Determine whether there are multiple second target output results containing consecutive timestamps.

[0126] If it is determined that there are multiple second target output results with consecutive timestamps, then step S307 is executed. If there is only one target output result, or if the timestamps of the multiple second target output results are not consecutive, then step S308 is executed.

[0127] S307. It was determined that there was an atomicity problem with the distributed database.

[0128] S308. It has been determined that there are problems with the consistency or isolation of the distributed database.

[0129] Optionally, in another embodiment of this application, the specified characteristic is a target characteristic set, and the data operation is a first insertion operation. One implementation of step S101 is as follows: Figure 4 As shown, it includes:

[0130] S401. Create a second source table and a second target table on a distributed database.

[0131] The second source table and the second target table are two distributed tables with the same fields. The second source table has an ID field, with each ID corresponding to one record. Therefore, the second target table also has an ID field, but does not store any records in it. The sharding key for both the second source table and the second target table is ID, and sharding is random. However, the ID field in the second target table does not have a unique index. This is to allow duplicate data to be inserted if atomicity issues exist, thus demonstrating the atomicity problem through data analysis.

[0132] It should also be noted that the second target table has an additional auto-incrementing column as the primary key compared to the second source table.

[0133] S402. For the second source table and the second target table, perform multiple first insert operations concurrently.

[0134] When the first insertion operation is executed, the IDs corresponding to all records in the second source table are incremented by 1, and then all records in the second source table are inserted under the corresponding IDs in the second target table, that is, the records are inserted under the IDs obtained by incrementing the corresponding IDs by one.

[0135] Optionally, since the first insert operation can be completed by executing only one SQL statement, it can be set to automatic commit mode, thus eliminating the need to wait for the test code to call commit, thereby saving detection time.

[0136] It should be noted that the data operations in this embodiment are data operations corresponding to the target feature set. Therefore, in this embodiment, the specified feature is the target feature set, so step S102 is not required. Correspondingly, the implementation method of step S103 in this embodiment is also referred to... Figure 4 This includes the following steps:

[0137] S403. During the concurrent execution of multiple first insert operations, information about the data on the second target table is continuously obtained multiple times to obtain multiple third output results.

[0138] The third output contains the number of records under each ID in the second target table after the first insert operation is performed.

[0139] S404. Determine if there exists any third output result containing a different number of records under each ID in the second target table.

[0140] It should be noted that when transactions perform insert operations concurrently, the records in the second target table will continuously increase as the transactions are executed concurrently. Since all data is read from the second source table, the number of records added to each ID in the second target table is equal each time. Therefore, the number of records under each different ID in the second target table should be consistent at any given time.

[0141] Therefore, if there is an inconsistency in the number of records under each ID in the second target table contained in a third output result, it indicates that there is a problem with one or more of the atomicity, consistency, and isolation properties, so step S405 is executed at this time.

[0142] S405. Determine if the distributed database has problems with one or more of the following: atomicity, consistency, and isolation.

[0143] It should be noted that further analysis and testing are needed to determine which of the following properties—atomicity, consistency, and isolation—is problematic.

[0144] S104. Based on the consistency between the results of the data operations and the log records, determine whether there are any persistence issues with the distributed database.

[0145] Optionally, in another embodiment of this application, the specified characteristic is persistence, the data operation is a second insertion operation, and one implementation of step S101 is as follows: Figure 5 As shown, it includes:

[0146] S501. Create a third distributed table on the distributed database.

[0147] The second distributed table uses an auto-incrementing column as the primary key and the thread number as the sharding key.

[0148] S502. Insert records into the third distributed table concurrently using multiple threads, and record the cumulative number of times each thread inserts records in the third distributed table.

[0149] Optionally, the start time of insertion can also be recorded, i.e., the start time of detection, to distinguish between different tests.

[0150] It should be noted that the data operation in this embodiment is the second insertion operation corresponding to persistence. Therefore, since the specified characteristic is persistence in this embodiment, step S102 is not required. Correspondingly, the implementation method of step S104 in this embodiment is also described below. Figure 5 This includes the following steps:

[0151] S503. Determine whether there exists a third distributed table where the cumulative value of any input record for a thread has an error greater than 1 compared to the cumulative value of the log record.

[0152] If the cumulative number of times a thread inserts a record in the third distributed table differs from the cumulative number of times that thread inserts a record in the log record by more than 1, then proceed to step S504.

[0153] S504. It was determined that there was a problem with the persistence of the distributed database.

[0154] This application provides a method for detecting the characteristics of a distributed database. For a specified characteristic of the distributed database, a distributed table is created on the distributed database, and data operations corresponding to the specified characteristic are concurrently executed on the distributed table. The specified characteristic is either a set of target characteristics or persistence. The set of target characteristics includes atomicity, consistency, and isolation. The data operation corresponding to the set of target characteristics is any one of update, delete, and insert operations. The data operation corresponding to persistence is an insert operation. Therefore, the detection process only requires concurrent execution of regular database operations. If the specified characteristic is the set of target characteristics, the consistency between the data information during the data operation process and the data on the initial distributed table in the specified metric determines whether the distributed database has a problem with the specified characteristic. If the specified characteristic is persistence, since the log records the operations, the consistency between the result after the data operation and the log record determines whether the distributed database has a problem with persistence. Thus, by concurrently executing regular database operations and based on the changes in the operation results, a simple and convenient detection method is achieved, and the detection results are not affected by application code, effectively ensuring the accuracy of the detection.

[0155] Another embodiment of this application provides a characteristic detection device for a distributed database, such as... Figure 6 As shown, it includes:

[0156] Operation unit 601 is used to create distributed tables on a distributed database for specified characteristics of the distributed database, and concurrently perform data operations corresponding to the specified characteristics on the distributed tables.

[0157] The specified characteristic is either a set of target characteristics or durability. The set of target characteristics includes atomicity, consistency, and isolation. The data operation corresponding to the set of target characteristics is any one of the update, delete, and insert operations. The data operation corresponding to durability is the insert operation.

[0158] The first determining unit 602 is used to determine whether there is a problem with the specified characteristic of the distributed database based on the consistency between the information of the data on the distributed table obtained during the execution of data operation and the information of the initial data on the distributed table on the specified indicator, if the specified characteristic is a target characteristic set.

[0159] The second determining unit 603 is used to determine whether there is a problem with the persistence of the distributed database based on the consistency between the result of the data operation and the log record if the specified characteristic is persistence.

[0160] Optionally, in another embodiment of this application, the specified characteristic is a target characteristic set, and the data operation is an update operation. The characteristic detection device operation unit of the distributed database provided in this embodiment includes:

[0161] The first creation unit is used to create the first distributed table on the distributed database with the account as the hash sharding key.

[0162] The first distributed table stores multiple accounts, each account contains one record, and the initial amount in each record is the same.

[0163] The update operation unit is used to execute multiple update operations concurrently on the first distributed table.

[0164] In one update operation, a random amount is first deducted from one account and then a random amount is added to another account.

[0165] The first determining unit in this embodiment includes:

[0166] The first acquisition unit is used to continuously acquire information about the data on the first distributed table multiple times during the concurrent execution of multiple update operations, and obtain multiple first output results.

[0167] The first output includes the timestamp when the data was retrieved and the total amount recorded on the first distributed table at that time.

[0168] The first filtering unit is used to filter out the first target output results from the various first output results, including those whose total amount is not equal to the total amount recorded on the initial first distributed table.

[0169] The first problem-determining unit is used to determine if there are multiple first target output results with consecutive timestamps, and if so, that there is an atomicity problem with the distributed database.

[0170] The second problem-determination unit is used to determine whether there is a problem with the consistency or isolation of the distributed database if there is only one first target output result, or if there are multiple first target output results with discontinuous timestamps.

[0171] Optionally, in another embodiment of this application, the specified characteristic is a target characteristic set, and the data operation is a deletion operation. The operation unit in this embodiment includes:

[0172] The second creation unit is used to create the first source table and the first target table on the distributed database.

[0173] The first source table and the first target table are two distributed tables, both with an account field but different sharding keys. The account field of the first source table stores multiple records and has a unique index. The account field of the first target table does not have a unique index and does not store any records.

[0174] The delete operation unit is used to perform multiple delete operations concurrently on the first source table and the first target table.

[0175] When the deletion operation is executed, a record is randomly read from the first source table, and after the record is inserted into the first target table, the record is deleted from the first source table.

[0176] The first determining unit in this embodiment includes:

[0177] The second acquisition unit is used to continuously acquire information about the data on the first source table and the first target table multiple times during the concurrent execution of multiple deletion operations, and obtain multiple second output results.

[0178] The second output result includes the timestamp when the data was acquired, and the total number of records in the first source table and the first target table at that time.

[0179] The second filtering unit is used to filter out the second target output results from the second output results whose total quantity is not equal to the initial total quantity.

[0180] The initial total number of records refers to the total number of records in the initial source table and the initial target table.

[0181] The third problem-determining unit is used to determine if there are multiple second target output results with consecutive timestamps, and if so, that there is an atomicity problem with the distributed database.

[0182] The fourth problem-determination unit is used to determine whether there is a problem with the consistency or isolation of the distributed database if there is only one second target output result, or if there are multiple second target output results with discontinuous timestamps.

[0183] Optionally, in another embodiment of this application, the specified characteristic is a target characteristic set, and the data operation is a first insertion operation. The operation unit in this embodiment includes:

[0184] The third creation unit is used to create a second source table and a second target table on a distributed database.

[0185] The second source table and the second target table are two distributed tables with the same fields. The second source table has an ID field, with each ID corresponding to one record. The sharding key for both the second source table and the second target table is ID, and sharding is random, but the ID field of the second target table does not have a unique index.

[0186] The first insertion operation unit is used to execute multiple first insertion operations concurrently on the second source table and the second target table.

[0187] When the first insert operation is executed, the IDs corresponding to all records in the second source table are incremented by 1, and then all records in the second source table are inserted under the corresponding IDs in the second target table.

[0188] The first determining unit in this embodiment includes:

[0189] The third acquisition unit is used to continuously acquire information about the data on the second target table multiple times during the concurrent execution of multiple first insert operations, and obtain multiple third output results.

[0190] The third output contains the number of records under each ID in the second target table after the first insert operation is performed.

[0191] The fifth problem-determination unit is used to determine that if the number of records under each ID in the second target table contained in any third output result is inconsistent, then the distributed database has a problem in one or more of the following: atomicity, consistency, and isolation.

[0192] Optionally, in another embodiment of this application, the operating unit includes:

[0193] The fourth creation unit is used to create the third distributed table on the distributed database.

[0194] The second distributed table uses an auto-incrementing column as the primary key and the thread number as the sharding key.

[0195] The second insertion operation unit is used to insert records into the third distributed table concurrently through multiple threads, and to record the cumulative value of the number of times each thread inserts records in the third distributed table.

[0196] The second determining unit in this embodiment includes:

[0197] The second determining subunit is used to determine that there is a problem with the persistence of the distributed database if the cumulative value of the number of times a thread inserts a record in the third distributed table and the cumulative value of the number of times a thread inserts a record in the log record have an error greater than 1.

[0198] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the above method embodiments.

[0199] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting characteristics of a distributed database, characterized in that, include: For a specified characteristic of the distributed database, a distributed table is created on the distributed database, and data operations corresponding to the specified characteristic are concurrently executed on the distributed table; wherein, the specified characteristic is a set of target characteristics or persistence; the set of target characteristics includes atomicity, consistency, and isolation; the data operation corresponding to the set of target characteristics is any one of update, delete, and insert operations; the data operation corresponding to persistence is an insert operation; If the specified characteristic is the target characteristic set, then based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table in the specified indicator, it is determined whether the distributed database has a problem with the specified characteristic; If the specified characteristic is persistence, then based on the consistency between the result of performing the data operation and the log record, it is determined whether the distributed database has a problem with persistence; Wherein, the specified characteristic is the target characteristic set, and the data operation is an update operation, the step of creating a distributed table on the distributed database for the specified characteristic of the distributed database, and concurrently performing the data operation corresponding to the specified characteristic on the distributed table, includes: A first distributed table is created on the distributed database, with the account number as the hash sharding key; wherein, the first distributed table stores multiple accounts, each account stores one record, and the initial amount in each record is the same; For the first distributed table, multiple update operations are executed concurrently; wherein, when one update operation is executed, a random amount is first randomly deducted from one account and then the random amount is randomly added to another account. The step of determining whether the distributed database has a problem with a specified characteristic based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table on a specified indicator includes: During the concurrent execution of multiple update operations, information about the data on the first distributed table is obtained multiple times to obtain multiple first output results; wherein, the first output result includes the timestamp when the data is obtained and the total amount recorded on the first distributed table at that time; Filter out the first target output results from each of the first output results, and select the first target output results whose total amount is not equal to the total amount recorded on the initial first distributed table; If there are multiple first target output results with consecutive timestamps, then it is determined that there is an atomicity problem in the distributed database; If there is only one first target output result, or if there are multiple first target output results with discontinuous timestamps, then it is determined that there is a problem with the consistency or isolation of the distributed database.

2. The method according to claim 1, characterized in that, The specified characteristic is the target characteristic set, and the data operation is a deletion operation. The steps of creating a distributed table on the distributed database for the specified characteristic of the distributed database, and concurrently performing the data operation corresponding to the specified characteristic on the distributed table, include: A first source table and a first target table are created on the distributed database; wherein the first source table and the first target table are two distributed tables that both have an account field but have different sharding keys; the account field of the first source table stores multiple records and has a unique index; the account field of the first target table does not have a unique index and does not store any records. For the first source table and the first target table, multiple deletion operations are performed concurrently; wherein, when a deletion operation is performed, a record in the first source table is randomly read, and after the record is inserted into the first target table, the record is deleted from the first source table; The step of determining whether the distributed database has a problem with a specified characteristic based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table on a specified indicator includes: During the concurrent execution of multiple deletion operations, information about the data on the first source table and the first target table is continuously obtained multiple times to obtain multiple second output results; wherein, the second output result is the timestamp when the data is obtained, and the sum of the number of records on the first source table and the first target table at that time; Filter out the second target output results from the second output results, where the sum of the quantities is not equal to the initial sum of quantities; wherein, the initial sum of the number of records refers to the sum of the number of records in the initial first source table and the first target table; If there are multiple second target output results with consecutive timestamps, then the atomicity of the distributed database is determined to be problematic. If there is only one second target output result, or if there are multiple second target output results with discontinuous timestamps, then it is determined that there is a problem with the consistency or isolation of the distributed database.

3. The method according to claim 1, characterized in that, The specified characteristic is the target characteristic set, and the data operation is a first insertion operation. The step of creating a distributed table on the distributed database for the specified characteristic of the distributed database, and concurrently performing the data operation corresponding to the specified characteristic on the distributed table, includes: A second source table and a second target table are created on the distributed database; wherein the second source table and the second target table are two distributed tables with the same fields; the second source table has an ID field, with each ID corresponding to one record; the sharding key of the second source table and the second target table is ID, and random sharding is performed, but the ID field of the second target table does not have a unique index. For the second source table and the second target table, multiple first insertion operations are executed concurrently; wherein, when the first insertion operation is executed, the IDs corresponding to all records in the second source table are incremented by 1, and then all records in the second source table are inserted under the corresponding IDs in the second target table; The step of determining whether the distributed database has a problem with a specified characteristic based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table on a specified indicator includes: During the concurrent execution of multiple first insertion operations, information about the data on the second target table is continuously obtained multiple times to obtain multiple third output results; wherein, the third output results include the number of records under each ID in the second target table after the execution of the first insertion operation; If the number of records under each ID in the second target table contained in any of the third output results is inconsistent, then it is determined that the distributed database has a problem with one or more of the following: atomicity, consistency, and isolation.

4. The method according to claim 1, characterized in that, The specified characteristic is persistence, and the data operation is a second insert operation. The steps of creating a distributed table on the distributed database for the specified characteristic of the distributed database, and concurrently performing the data operation corresponding to the specified characteristic on the distributed table, include: Create a third distributed table on the distributed database; wherein the third distributed table uses an auto-incrementing column as the primary key and the thread number as the sharding key; Multiple threads are used to insert records into the third distributed table concurrently, and the cumulative number of times each thread inserts records is recorded in the third distributed table. The step of determining whether the distributed database has a persistence problem based on the consistency between the results of the data operation and the log records includes: If the cumulative number of times a thread inserts a record in the third distributed table differs from the cumulative number of times a thread inserts a record in the log record by more than 1, then the distributed database is determined to have a persistence problem.

5. A characteristic detection device for a distributed database, characterized in that, include: An operation unit is configured to create distributed tables on the distributed database based on specified characteristics of the distributed database, and concurrently execute data operations corresponding to the specified characteristics on the distributed tables; wherein, the specified characteristics are a set of target characteristics or persistence; the set of target characteristics includes atomicity, consistency, and isolation; the data operation corresponding to the set of target characteristics is any one of update, delete, and insert operations; and the data operation corresponding to persistence is an insert operation. The first determining unit is configured to, if the specified characteristic is the target characteristic set, determine whether the distributed database has a problem with the specified characteristic based on the consistency of the information of the data on the distributed table obtained during the execution of the data operation with the information of the initial data on the distributed table on the specified indicator. The second determining unit is used to determine whether the distributed database has a problem with the persistence if the specified characteristic is persistence, based on the consistency between the result after performing the data operation and the log record. Wherein, the specified characteristic is the target characteristic set, and the data operation is an update operation, the operation unit includes: The first creation unit is used to create a first distributed table on the distributed database with the account number as the hash sharding key; wherein, the first distributed table stores multiple accounts, each account stores one record, and the initial amount in each record is the same; An update operation unit is configured to execute multiple update operations concurrently for the first distributed table; wherein, when one update operation is executed, a random amount is first randomly deducted from one account and then the random amount is randomly added to another account; The first determining unit includes: The first acquisition unit is used to continuously acquire information about the data on the first distributed table multiple times during the concurrent execution of multiple update operations, and obtain multiple first output results; wherein, the first output result includes the timestamp when the data is acquired and the total amount recorded on the first distributed table at that time; The first filtering unit is used to filter out a first target output result from each of the first output results, where the total amount is not equal to the total amount recorded on the initial first distributed table. The first problem-determining unit is used to determine that the atomicity of the distributed database is problematic if there are multiple first target output results with consecutive timestamps. The second problem-determination unit is used to determine that there is a problem with the consistency or isolation of the distributed database if there is only one first target output result, or if there are multiple first target output results with discontinuous timestamps.

6. The apparatus according to claim 5, characterized in that, The specified characteristic is the target characteristic set, and the data operation is a deletion operation. The operation unit includes: The second creation unit is used to create a first source table and a first target table on the distributed database; wherein the first source table and the first target table are two distributed tables that both have an account field but have different sharding keys; the account field of the first source table stores multiple records and has a unique index; the account field of the first target table does not have a unique index and does not store any records. The deletion operation unit is used to perform multiple deletion operations concurrently on the first source table and the first target table; wherein, when a deletion operation is executed, a record is randomly read from the first source table, and after inserting the record into the first target table, the record is deleted from the first source table; The first determining unit includes: The second acquisition unit is used to continuously acquire information about the data on the first source table and the first target table multiple times during the concurrent execution of multiple deletion operations, and obtain multiple second output results; wherein the second output result is the timestamp when the data is acquired, and the sum of the number of records on the first source table and the first target table at that time; The second filtering unit is used to filter out a second target output result from the second output result whose total number is not equal to the initial total number; wherein, the initial total number of records refers to the initial total number of records in the first source table and the first target table; The third problem-determining unit is used to determine that the atomicity of the distributed database is problematic if there are multiple second target output results with consecutive timestamps. The fourth problem determination unit is used to determine that there is a problem with the consistency or isolation of the distributed database if there is only one second target output result, or if there are multiple second target output results with discontinuous timestamps.

7. The apparatus according to claim 5, characterized in that, The specified characteristic is the target characteristic set, and the data operation is a first insertion operation. The operation unit includes: The third creation unit is used to create a second source table and a second target table on the distributed database; wherein the second source table and the second target table are two distributed tables with the same fields; the second source table has an ID field, and each ID corresponds to one record; the sharding key of the second source table and the second target table is ID, and random sharding is used, but the ID field of the second target table does not have a unique index. The first insertion operation unit is used to execute multiple first insertion operations concurrently for the second source table and the second target table; wherein, when the first insertion operation is executed, the IDs corresponding to all records in the second source table are incremented by 1, and then all records in the second source table are inserted under the corresponding IDs in the second target table; The first determining unit includes: The third acquisition unit is used to continuously acquire information about the data on the second target table multiple times during the concurrent execution of multiple first insertion operations, and obtain multiple third output results; wherein, the third output results include the number of records under each ID in the second target table after the execution of the first insertion operation; The fifth problem determination unit is used to determine that if the number of records under each ID in the second target table contained in any of the third output results is inconsistent, then the distributed database has a problem in one or more of the following: atomicity, consistency, and isolation.

8. The apparatus according to claim 5, characterized in that, The specified characteristic is persistence, and the data operation is a second insertion operation. The operation unit includes: The fourth creation unit is used to create a third distributed table on the distributed database; wherein the third distributed table uses an auto-incrementing column as the primary key and the thread number as the sharding key; The second insertion operation unit is used to insert records into the third distributed table in a concurrent manner through multiple threads, and to record the cumulative value of the number of times each thread inserts records in the third distributed table; The second confirmation unit includes: The second confirmation subunit is used to determine that there is a persistence problem in the distributed database if the cumulative value of the number of times the thread inserts records in any one of the records in the third distributed table has an error greater than 1 compared with the cumulative value of the number of times the thread inserts records in the log.

Images