Chaotic testing method for domestic distributed database

Abstract

The application discloses a kind of chaos test methods of domestic distributed database, it is related to database technical field, including: set the clock deviation parameter of target database node in domestic distributed database, make the node and other node clock inconsistent;Business load is applied to domestic distributed database, and the transaction request corresponding to load is handled by node;During load application, the running state data of domestic distributed database transaction processing process is obtained, including business exception information and inter-node synchronization exception information;After load application, according to the comparison result of version identification and time stamp of the data to be verified on different nodes, the consistency check result data between the data to be verified on different nodes is determined;According to running state data and consistency check result data, determine the abnormal condition caused by clock inconsistency.The application can effectively evaluate the abnormal condition of domestic distributed database under clock inconsistency.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a chaos testing method for a domestically produced distributed database. Background Technology

[0002] Domestic distributed databases rely on clock synchronization to ensure transaction consistency and data ordering. However, in actual operation, due to network time protocol service anomalies or hardware drift, clocks on different nodes may become inconsistent. When the clock deviation exceeds the database's tolerance threshold, it can lead to transaction conflicts, data overwriting, or incorrect query results.

[0003] Existing chaos testing solutions primarily focus on common faults such as network partitioning and node failures, failing to adequately cover the scenario of "clock inconsistency" and lacking a systematic testing method for this scenario. Therefore, there is an urgent need for a testing solution that can effectively evaluate the abnormal behavior of domestically produced distributed databases under clock inconsistency. Summary of the Invention

[0004] This invention provides a chaos testing method for domestically produced distributed databases to solve the technical problem that existing technologies cannot effectively assess the abnormal conditions caused by clock inconsistencies in domestically produced distributed databases.

[0005] According to one aspect of the present invention, a chaos testing method for a domestically developed distributed database is provided, the method comprising: A clock skew parameter is set for at least one target database node in a domestically produced distributed database, so that the clock of the target database node is inconsistent with that of at least one other database node; wherein, the clock skew parameter includes clock skew direction and clock skew amount; A business load is applied to the domestically produced distributed database, and the database nodes in the domestically produced distributed database handle the transaction requests corresponding to the business load. During the application of business load, the operational status data of the domestic distributed database during transaction processing is obtained; wherein, the operational status data includes business anomaly information and inter-node synchronization anomaly information; After applying the business load, the consistency verification result data between the data to be verified on different database nodes in the domestic distributed database is determined based on the comparison results of the version identifier and timestamp information of the data to be verified on different database nodes. Based on the operational status data and the consistency verification result data, the abnormal situation caused by the clock inconsistency to the domestic distributed database is determined.

[0006] According to another aspect of the present invention, a chaos testing device for a domestically produced distributed database is provided, the device comprising: A clock skew parameter setting module is used to set clock skew parameters for at least one target database node in a domestic distributed database, so that the clock of the target database node is inconsistent with that of at least one other database node; wherein, the clock skew parameter includes clock skew direction and clock skew amount; A business load application module is used to apply business load to the domestic distributed database, and the database nodes in the domestic distributed database process the transaction requests corresponding to the business load. The runtime status data acquisition module is used to acquire runtime status data of the domestic distributed database during transaction processing when applying business load; wherein, the runtime status data includes business anomaly information and inter-node synchronization anomaly information; The consistency verification result data acquisition module is used to determine the consistency verification result data between the data to be verified on different database nodes in the domestic distributed database based on the comparison results of the version identifier and timestamp information of the data to be verified on different database nodes after applying business load. The abnormal situation determination module is used to determine the abnormal situation caused by the clock inconsistency to the domestic distributed database based on the running status data and the consistency verification result data.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the method described in any one of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method of any one of the present invention.

[0010] The beneficial effects of this invention are as follows: Firstly, by actively setting clock skew parameters to make the clocks of database nodes inconsistent, it is possible to simulate clock failures caused by network time protocol service anomalies or hardware drift, thereby exposing problems such as transaction conflicts and data overwriting caused by clock inconsistency in advance in the test environment, effectively making up for the lack of clock domain failure scenarios in existing chaos testing solutions.

[0011] Secondly, by acquiring business anomaly information and inter-node synchronization anomaly information in real time during the application of business load, and combining the comparison results of version identifiers and timestamps of the data to be verified on different nodes, it is possible to quantitatively assess the abnormal conditions caused by clock inconsistency to domestic distributed databases, accurately identify specific abnormal manifestations such as transaction commit errors, sudden increases in synchronization delays, and inconsistent data versions or timestamps, and provide clear and quantifiable basis for the fault tolerance assessment and defect localization of domestic distributed databases.

[0012] Thirdly, the provided testing methods are systematic and repeatable, providing reliable technical support for the verification of the fault tolerance of domestic distributed databases and the stability assurance of information technology innovation systems, and significantly improving the coverage and effectiveness of clock domain chaos testing.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart of a chaos testing method for a domestically produced distributed database provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the structure of a chaos testing device for a domestically produced distributed database provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the chaos testing method of the domestic distributed database in this embodiment of the invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "target," "benchmark," "abnormal," "first," "second," and "third," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] Example 1 Figure 1 This is a flowchart of a chaos testing method for a domestically produced distributed database according to Embodiment 1 of the present invention. This embodiment is applicable to evaluating the abnormal conditions of a domestically produced distributed database under clock inconsistency faults. The method can be executed by a chaos testing device for a domestically produced distributed database. This device can be implemented in hardware and / or software, for example, as a functional module of a chaos engineering platform or as an independent testing tool. Figure 1 As shown, the method includes: S101. Set a clock skew parameter for at least one target database node in the domestic distributed database so that the clock of the target database node is inconsistent with that of at least one other database node.

[0019] Among them, domestically produced distributed databases refer to distributed database management systems independently developed by Chinese companies and possessing independent intellectual property rights. These databases typically consist of multiple database nodes that work together through a network.

[0020] The target database node refers to the physical server or virtual machine where the database instance selected for injecting clock skew parameters resides in the domestic distributed database. In this embodiment of the invention, clock skew parameters can be set for one or more target database nodes in the domestic distributed database.

[0021] Clock skew parameters are configuration information used to describe the offset characteristics of the target database node clock relative to real time or other database node clocks. Clock skew parameters include clock skew direction and clock skew amount.

[0022] Other database nodes refer to database nodes in the domestic distributed database other than the target database node whose clock skew parameter is currently set. In this embodiment of the invention, at least one other database node is required as a reference object for clock comparison.

[0023] Clock inconsistency refers to the incompatibility or discrepancy between the system times of different database nodes in a domestically produced distributed database. Specifically, the clock of at least one target database node is out of sync with the clock of at least one other database node. Clock inconsistency can manifest as a positive deviation (clock is faster than the reference database node), a negative deviation (clock is slower than the reference database node), or both.

[0024] The direction of clock skew indicates the trend of the target database node's clock offset relative to other database nodes (or real time). A positive skew indicates that the node's clock is faster than the reference clock, while a negative skew indicates that the node's clock is slower than the reference clock.

[0025] Clock skew represents the specific numerical offset of the target database node's clock relative to other database nodes (or real time), such as 5 minutes, 2 minutes, etc. The skew can be zero to indicate no skew, but in embodiments of this invention, a non-zero skew is typically set to simulate a fault.

[0026] In one implementation, the chaos testing platform, in response to user-configured clock offset parameters and a user-selected target database node, determines at least one target database node from the domestic distributed database and obtains the user-set clock offset parameters, including the clock offset direction and the clock offset amount. Based on the clock offset direction, the chaos testing platform adjusts the clock of the target database node according to the clock offset amount via a time modification interface provided by the operating system, causing the clock of the target database node to offset relative to the clock of at least one other database node, thereby creating a clock inconsistency between the target database node and at least one other database node in the domestic distributed database.

[0027] For example, a user selects a target database node (e.g., node A) from a domestic distributed database through the visualization interface of the chaos testing platform, and sets the clock offset parameters: the clock offset direction is positive, and the clock offset amount is 5 minutes. In response to the user's configuration, the chaos testing platform automatically adjusts the system clock of node A forward by 5 minutes, causing node A's clock to shift positively relative to the clocks of other database nodes, thus creating a clock inconsistency between node A and at least one other database node.

[0028] In another implementation, in response to a user-triggered test start command, the testing device reads pre-configured clock offset parameters and pre-selected target database node identifiers, and determines at least one target database node from the domestic distributed database. The clock offset parameters include the clock offset direction and the clock offset amount. The testing device establishes a communication connection with the target database node and sends a time adjustment command to the target database node. The time adjustment command includes the clock offset direction and the clock offset amount, causing the target database node to adjust its system clock according to the time adjustment command. This causes the clock of the target database node to offset relative to the clock of at least one other database node, thereby creating a clock inconsistency between the target database node and at least one other database node in the domestic distributed database.

[0029] For example, the user pre-writes the following in the test configuration file: the target database node is node B, the clock offset direction is negative, and the clock offset amount is 2 minutes. The user then clicks the "Start Test" button. In response to the test start command, the test device reads the parameters in the configuration file, determines node B as the target database node, and sends a time adjustment command to node B, adjusting its system clock backward by 2 minutes. This causes node B's clock to have a negative offset relative to the clocks of other database nodes, thus creating a clock inconsistency between node B and at least one other database node.

[0030] Optionally, a clock skew parameter can be set for at least one target database node in the domestically produced distributed database, including: Determine the first and second database nodes from the domestic distributed database; set the first clock skew parameter for the first database node; set the second clock skew parameter for the second database node.

[0031] The first database node and the second database node are different from each other; the clock deviation direction of the first clock deviation parameter is positive and the clock deviation amount is a preset first deviation amount; the clock deviation direction of the second clock deviation parameter is negative and the clock deviation amount is a preset second deviation amount, and the first deviation amount and the second deviation amount are different.

[0032] In one implementation, the testing device responds to a user's test configuration command by determining a first database node and a second database node from a domestically distributed database, wherein the first database node and the second database node are different from each other.

[0033] The testing device sets a first clock deviation parameter for the first database node. The clock deviation direction of the first clock deviation parameter is positive, and the clock deviation amount is a preset first deviation amount (e.g., 5 minutes). The testing device sets a second clock deviation parameter for the second database node. The clock deviation direction of the second clock deviation parameter is negative, and the clock deviation amount is a preset second deviation amount (e.g., 2 minutes). The first deviation amount is different from the second deviation amount.

[0034] The testing device establishes management channels with the first database node and the second database node respectively. By calling the system time modification interface or executing the time adjustment command, the system clock of the first database node is adjusted to a positive deviation amount, and the system clock of the second database node is adjusted to a negative deviation amount, so that the clocks of the first database node and the second database node are inconsistent with other normal nodes and with each other.

[0035] For example, the testing device determines node A as the first database node and node B as the second database node from the domestic distributed database. The testing device adjusts the clock of node A forward by 5 minutes, adjusts the clock of node B backward by 2 minutes, and keeps the clock of node C unchanged. Thus, a clock inconsistency is formed in the domestic distributed database: node A is 5 minutes fast, node B is 2 minutes slow, and node C is normal.

[0036] The advantages of this optional embodiment are as follows: By setting clock biases with opposite directions and different values ​​for two different database nodes in a domestically produced distributed database, it is possible to effectively simulate complex clock inconsistency scenarios caused by network time protocol service anomalies or hardware drift in the real world (e.g., some nodes are faster, some nodes are slower, and some nodes are normal). This provides a typical fault injection sample for comprehensively testing the data consistency and system stability of domestically produced distributed databases under diverse combinations of clock biases.

[0037] Optionally, a clock skew parameter can be set for at least one target database node in the domestically produced distributed database, including: Set the drift rate; according to the drift rate, continuously adjust the clock skew of the target database node so that the clock skew of the target database node gradually increases over time.

[0038] Drift rate refers to the rate at which the clock deviation of the target database node changes over time. Drift rate characterizes how quickly the clock deviates from its correct value.

[0039] The clock deviation of the target database node increases gradually over time, meaning that it does not jump to the target value all at once, but increases continuously and linearly according to the drift rate. This process simulates the gradual drift of the hardware clock caused by crystal oscillator aging or temperature changes, which is different from a one-time fixed deviation.

[0040] In one implementation, when setting the clock skew parameters, the testing device specifies not only the clock skew direction and initial skew amount, but also an additional drift rate. The testing device initiates a background adjustment process that, at preset time intervals (e.g., triggered once per second), acquires the currently accumulated clock skew amount, calculates the required increase in skew amount based on the drift rate, and then readjusts the clock of the target database node using the system time modification interface. This process is repeated cyclically, ensuring that the clock skew amount of the target database node increases continuously and linearly according to the drift rate, achieving the effect of a gradual increase in clock skew amount over time.

[0041] For example, the test setup sets a positive clock skew for node A, with an initial skew of 0 milliseconds and a drift rate of 10 milliseconds per second. The adjustment process advances node A's clock by 10 milliseconds every second. At the end of the first second, the accumulated skew is 10 milliseconds; at the end of the second second, the accumulated skew is 20 milliseconds; and at the end of the tenth second, the accumulated skew is 100 milliseconds. This continues, with node A's clock skew gradually increasing linearly over time.

[0042] The advantages of this optional embodiment are as follows: By setting the drift rate and continuously adjusting the clock skew of the target database node, a gradual clock inconsistency fault caused by hardware clock drift can be simulated, filling the testing gap that only simulates a one-time fixed deviation. This gradual fault helps test the gradual abnormal performance of domestically produced distributed databases during the slow accumulation of deviations.

[0043] S102. Apply business load to the domestic distributed database, and let the database nodes in the domestic distributed database handle the transaction requests corresponding to the business load.

[0044] The business load refers to a set of database operation tasks that simulate real-world application scenarios, used to stress the domestically produced distributed database and test its processing capacity and stability under specific workloads. In this embodiment of the invention, the business load is preferably an online transaction processing load test, such as the TPC-C benchmark test, which simulates typical transaction operations of an order management system.

[0045] A transaction request is the basic execution unit that constitutes the business workload. Each transaction request corresponds to a complete sequence of database operations that satisfy the characteristics of atomicity, consistency, isolation, and durability, such as creating an order, updating inventory, and querying order status. During the test, the business workload continuously generates multiple transaction requests, which are received and processed collaboratively by the database nodes in the domestic distributed database. The commit result of each transaction request is used to evaluate the transaction processing status of the domestic distributed database.

[0046] In one implementation, the testing device, in response to a user's startup command, reads business load parameters from a preset load profile. These parameters include transaction type ratio, number of concurrent users, and runtime. Based on the business load parameters, the testing device generates multiple transaction requests and sends them to the access address of the domestic distributed database according to a preset concurrency level. Upon receiving the transaction requests, the domestic distributed database collaboratively processes them, including the master database node handling write transactions, the slave database nodes handling read transactions, and synchronizing logs via a consensus protocol. The testing device continuously collects the commit results of each transaction request until the preset runtime is reached.

[0047] For example, a user initiates a TPC-C test using a testing device. The device automatically generates transaction requests such as new orders, payments, and shipments, and continuously sends these requests to the OceanBase domestic distributed database for 10 minutes using 50 concurrent threads. The master node in the domestic distributed database handles write transactions, while the slave nodes share the read transactions and maintain log synchronization.

[0048] In another implementation, the chaos testing platform responds to user-configured business load scenarios by obtaining connection information for the domestic distributed database from its access address. Based on the user-selected workload model (such as an order processing model), the platform dynamically generates a sequence of transaction requests and continuously sends these requests to the domestic distributed database via a load generator. Each database node in the domestic distributed database receives and processes these requests according to its role: the master database node handles transaction writing and commits, while slave database nodes receive logs from the master node and apply them to their local data storage, while also responding to some read transaction requests. The chaos testing platform records real-time metrics such as the response time and commit status of each transaction request.

[0049] S103. During the application of business load, obtain the running status data of the domestic distributed database during the transaction processing process.

[0050] Among them, operational status data refers to a dynamic set of information reflecting the transaction processing capabilities and data synchronization health of the domestic distributed database during the process of handling transaction requests. Operational status data includes business anomaly information and inter-node synchronization anomaly information, used to subsequently determine the abnormal conditions caused by clock inconsistencies to the domestic distributed database.

[0051] Business anomaly information refers to information reflecting transaction execution failures or anomalies when processing transaction requests in a domestically produced distributed database. Specifically, it includes metrics such as the number of failed transaction commits or the transaction commit error rate, used to assess the impact of clock inconsistencies on the success rate of transaction processing at the business level.

[0052] Inter-node synchronization anomaly information refers to information reflecting anomalies or delays in data synchronization between database nodes in domestically developed distributed databases. Specifically, it includes indicators such as sudden increases in consensus log synchronization delays, used to assess the impact of clock inconsistencies on the data replica synchronization process between different database nodes.

[0053] In one implementation, during the application of business load to a domestically developed distributed database, the testing device collects the transaction processing status of the database in real time through a database performance monitoring interface. The testing device sends monitoring query commands to the domestically developed distributed database to obtain the transaction commit success count and transaction commit failure count. Based on the comparison between the two, it generates transaction commit failure event information as business anomaly information. Simultaneously, the testing device obtains the log synchronization latency values ​​between the master database node and each slave database node through the database's replication status view. When an abnormal increase in the latency value is detected, a synchronization latency anomaly event is recorded as inter-node synchronization anomaly information. The testing device stores the collected business anomaly information and inter-node synchronization anomaly information in chronological order to form operational status data.

[0054] In another implementation, during the application of business load to the domestically developed distributed database, the chaos testing platform activates a log monitoring module to receive the operational logs output by the database in real time. The platform parses the transaction commit status identifiers in the logs and counts the number of failed transaction commits as business anomaly information. Simultaneously, by subscribing to consensus protocol events of the domestically developed distributed database, the platform records the occurrence time and the node identifiers involved when it detects that the log application latency reported from the database nodes exceeds the built-in judgment conditions, serving as inter-node synchronization anomaly information. The platform integrates the business anomaly information and the inter-node synchronization anomaly information onto the load timeline to generate operational status data.

[0055] Optionally, obtain the runtime status data of the domestically developed distributed database during transaction processing, including: Collect the number of successfully committed transactions and the number of failed transactions from the domestic distributed database; calculate the transaction commit error rate based on the number of successfully committed transactions and the number of failed transactions as business exception information.

[0056] The number of successfully committed transactions refers to the total number of transactions that, under the applied business load, the domestically developed distributed database successfully completes all operations, is confirmed through the consistency protocol, and is ultimately persisted. Each transaction is counted as one success from the time the request is initiated until the successful commit response is received.

[0057] The number of failed transactions refers to the total number of transactions that were rolled back or aborted due to timestamp conflicts, version number errors, log synchronization failures, or other reasons after the domestic distributed database started the transaction during the application of business load, and were not persisted.

[0058] The transaction commit error rate refers to the proportion of transactions that failed to commit out of all committed transactions (the sum of successful and failed transactions). This ratio is used to quantify the severity of business anomalies and serves as a core indicator for determining whether clock inconsistencies have caused transaction processing errors.

[0059] In one implementation, during the application of business load to a domestically developed distributed database, the testing device reads transaction commit status statistics in real time through the database's performance monitoring interface. Specifically, the testing device periodically (e.g., every second) queries the database's built-in system performance view to obtain the cumulative number of successfully committed transactions and the cumulative number of failed committed transactions since the start of the test. The testing device subtracts the cumulative number obtained from the previous query from the current cumulative number obtained in the current query to calculate the number of successfully committed transactions and the number of failed committed transactions within the current time window. Then, based on the number of successfully committed transactions and the number of failed committed transactions, the testing device calculates the transaction commit error rate, which is the number of failed committed transactions divided by the sum of the number of successfully committed transactions and the number of failed committed transactions. The testing device records and stores the calculated transaction commit error rate as business anomaly information.

[0060] For example, during a 10-minute online transaction processing load, the test device queries the database performance counters every second. At the 30th second, the cumulative number of successfully committed transactions is 15230, and the cumulative number of failed committed transactions is 5; at the 31st second, the cumulative number of successfully committed transactions is 15280, and the cumulative number of failed committed transactions is 10. Therefore, the number of successfully committed transactions in the 31st second is calculated to be 50, the number of failed committed transactions is 5, and the transaction commit error rate is 5 / (50+5) = 9.09%. The test device records this error rate as the current moment's business anomaly information.

[0061] The advantages of this optional embodiment are as follows: It can obtain the transaction commit error rate in real time and quantitatively, transforming abstract business anomalies into quantifiable numerical indicators. This provides an objective and accurate data basis for subsequent determination of whether clock inconsistency has caused abnormal conditions. Moreover, the calculation process is simple, the execution efficiency is high, and it does not affect the real-time performance of the testing process.

[0062] Optionally, obtain the runtime status data of the domestically developed distributed database during transaction processing, including: S1031. Determine the master database node and slave database nodes in the domestic distributed database; obtain in real time the first commit timestamp of the latest consensus log on the master database node, and the second commit timestamp of the last applied consensus log on each slave database node.

[0063] In this context, the master database node refers to the database node in a domestically developed distributed database that is responsible for handling write transaction requests, generating consensus logs, and leading log replication. In consensus protocols (such as Paxos or Raft), the master node is the sole generator of log entries, and other nodes synchronize based on the master node's logs.

[0064] In a domestically developed distributed database database, a slave node is a database node that receives consensus logs from the master database node and applies them to its local data storage. Slave nodes do not actively generate logs; instead, they follow the master node's log sequence to maintain data replication consistency with the master node.

[0065] The latest consensus log refers to the consensus log entry with the largest sequence number that the primary database node has generated and committed at the current point in time. This log entry contains information such as a data modification operation on the database and the corresponding commit timestamp.

[0066] The first commit timestamp refers to the transaction commit time recorded in the latest consensus log entry. This timestamp is generated by the global timestamp generator or hybrid logical clock of the domestic distributed database and is used to identify the logical or physical commit time of the transaction corresponding to this log entry.

[0067] Applying the consensus log refers to the process by which a database node actually writes the data modification operations received from the consensus log entries into its local data storage. After the application is completed, the data replicas on the slave node are consistent with those on the master node for the corresponding log entries.

[0068] The second commit timestamp refers to the transaction commit time recorded in the consensus log entry that was last completed by the database node. This timestamp reflects the latest time boundary of the data that has been synchronized and applied by the slave node.

[0069] In one implementation, during the application of workload to a domestically developed distributed database, the testing device first sends a role query command to the database to determine the current master database node and all slave database nodes. The testing device then obtains the first commit timestamp of the latest consensus log on the master database node in real time through the database's management interface. Simultaneously, the testing device sends a status query request to each slave database node to obtain the second commit timestamp of the last applied consensus log on each node. The testing device aligns the collected first commit timestamps with the second commit timestamps according to time, recording the synchronization status of each slave database node. This collection process is continuously executed in a loop during the application of workload.

[0070] S1032. Calculate the consensus log synchronization delay value of each slave database node based on the difference between the first submission timestamp and the second submission timestamp. When the consensus log synchronization delay value of any slave database node at the current sampling time exceeds the delay threshold and the delay value increment exceeds the increment threshold, it is determined that a consensus log synchronization delay surge event has occurred, which is regarded as an abnormal synchronization information between nodes.

[0071] The consensus log synchronization latency value refers to the time length calculated based on the difference between the first commit timestamp and the second commit timestamp. This latency value represents the amount of time that the data synchronization progress of the database node lags behind that of the master database node, in milliseconds. For example, if the first commit timestamp is T1 and the second commit timestamp is T2, then the latency value = T1 - T2.

[0072] The current sampling time refers to the point in time when the data collection operation is performed. Sampling is performed periodically at preset fixed time intervals (e.g., once per second), and each sampling time corresponds to a set of first submission timestamps and second submission timestamps from each database node.

[0073] The latency threshold is a pre-set critical value used to determine whether the consensus log synchronization latency is abnormal. When the consensus log synchronization latency value exceeds the latency threshold, it indicates that the synchronization latency from the database nodes has reached a level that requires attention.

[0074] The latency increment is the difference between the consensus log synchronization latency at the current sampling time and the consensus log synchronization latency at the previous sampling time; that is, the consensus log synchronization latency at the current sampling time minus the consensus log synchronization latency at the previous sampling time. The latency increment reflects the rate of change of the synchronization latency.

[0075] The increment threshold is a pre-set critical value used to determine whether the increase in delay value is abnormal. When the increase in delay value exceeds the increment threshold, it indicates that the synchronization delay has increased sharply in a short period of time, which meets the sudden characteristics of a "sudden increase".

[0076] A consensus log synchronization latency surge event refers to an event that occurs when the consensus log synchronization latency value of any slave database node exceeds a latency threshold, and the increment of that slave database node's consensus log synchronization latency value exceeds an increment threshold. This event serves as a concrete manifestation of inter-node synchronization anomalies, indicating a sudden abnormality in inter-node synchronization.

[0077] In one implementation, the testing device continuously acquires the first commit timestamp and each of the second commit timestamps, collecting data at a fixed sampling period (e.g., once per second). At each sampling moment, the testing device calculates the consensus log synchronization latency value of a slave database node based on the difference between the first commit timestamp and the second commit timestamp of that slave database node. The testing device then compares this latency value with a preset latency threshold.

[0078] Simultaneously, the testing device calculates the difference between the delay value and the delay value at the previous sampling time, obtains the delay value increment, and compares the increment with a preset increment threshold.

[0079] If the delay value at the current sampling time exceeds the delay threshold, and the delay value increment exceeds the increment threshold, the test device determines that a consensus log synchronization delay surge event has occurred, and records the occurrence time of the event, the identifier of the database node involved, and the delay value and increment value as inter-node synchronization anomaly information.

[0080] The advantages of this optional embodiment are as follows: It can monitor the consensus log synchronization status of each slave database node relative to the master database node in real time and dynamically. It calculates the consensus log synchronization delay value by calculating the difference between the first commit timestamp and the second commit timestamp, and determines whether a consensus log synchronization delay surge event has occurred based on the delay value at the current sampling time and its increment. This allows it to accurately capture sudden synchronization anomalies caused by clock inconsistency and provide key inter-node synchronization anomaly information for subsequent identification of abnormal situations.

[0081] S104. After applying the business load, based on the comparison results of the version identifier and timestamp information of the data to be verified on different database nodes, determine the consistency verification result data between the data to be verified on different database nodes in the domestic distributed database.

[0082] The data to be verified refers to the data objects stored in a domestically produced distributed database used for cross-node consistency comparison. Specifically, it can be records in business data tables within the domestic distributed database, such as data rows in an order table or inventory table. When selecting data to be verified, the primary key of the data row is used as a unique identifier to ensure that the same business data is being compared across different database nodes.

[0083] A version identifier refers to the version information associated with the data to be verified, used to identify the number of times or the order in which the data was updated. A version identifier can be a monotonically increasing version number, a version vector, or other identifier that reflects the order in which the data was modified. By comparing the version identifiers of the same data to be verified on different database nodes, it can be determined whether the update progress between data replicas is consistent.

[0084] Timestamp information refers to the timestamp associated with the commit operation of the data to be verified, used to record the last time the data was modified and committed. Timestamp information is typically generated by a global timestamp generator or a hybrid logical clock in a domestically developed distributed database, reflecting the logical or physical time of the transaction commit. By comparing the timestamp information of the same data to be verified on different database nodes, it is possible to determine whether timestamp offsets or timing discrepancies are caused by clock inconsistencies.

[0085] Consistency verification results refer to the conclusive information obtained by comparing the version identifier and timestamp information of the same data to be verified on different database nodes. The consistency verification results indicate whether there are inconsistencies in version identifiers or timestamps between different database nodes. This result data is used to subsequently determine the abnormal conditions caused by clock inconsistencies in domestically developed distributed databases.

[0086] In one implementation, after the workload is applied, the testing device initiates data sampling requests to multiple database nodes in the domestic distributed database, specifying the data table range (e.g., an order table) and primary key range of the data to be verified. The testing device obtains the version identifier and timestamp information corresponding to the data to be verified from each sampled node. The testing device compares the version identifiers and timestamp information of the same data to be verified pairwise from different nodes. If the version identifiers or timestamp information are different on any pair of nodes, the consistency verification result is determined to be inconsistent; if the version identifiers and timestamp information are the same on all nodes, the consistency verification result is determined to be consistent.

[0087] In another implementation, the chaos testing platform triggers a data consistency verification task after the business load is applied. The platform uses a database export interface to export the data to be verified, along with its accompanying version identifier and timestamp information, from a designated reference node in a domestic distributed database, forming a baseline dataset. The platform then exports data to be verified with the same primary key range, along with its version identifier and timestamp information, from other database nodes in the domestic distributed database, forming a comparison dataset. The platform compares the version identifier of each record in the comparison dataset with the corresponding record in the baseline dataset, and also compares the timestamp information. If a record with a mismatched version identifier or timestamp is found, the consistency verification result indicates inconsistency, and the mismatched data row information is recorded.

[0088] Optionally, based on the comparison results of the version identifiers and timestamp information of the data to be verified on different database nodes, the consistency verification result data between the data to be verified on different database nodes in the domestic distributed database is determined, including: S1041. Determine the benchmark database node from the domestic distributed database; obtain the first version identifier and first timestamp information of the data to be verified on the benchmark database node, and the second version identifier and second timestamp information on at least one comparison database node.

[0089] The benchmark database node refers to the database node selected as the reference standard for consistency verification in a domestically produced distributed database. The version identifier and timestamp information of the data to be verified on this node serve as the benchmark for comparison with the same data to be verified on other nodes.

[0090] In a domestically developed distributed database database, a comparison database node refers to another database node that is used for consistency comparison with the benchmark database node. The comparison database node differs from the benchmark database node and can consist of one or more nodes. The version identifier and timestamp information of the same data to be verified on this node are compared with the corresponding information on the benchmark database node to determine whether they are consistent.

[0091] In one implementation, after applying a workload, the testing device designates a database node from a domestically developed distributed database as a baseline database node. The testing device obtains the first version identifier and first timestamp information of the data to be verified on the baseline database node through a database query interface. Simultaneously, the testing device selects at least one other database node from the domestically developed distributed database as a comparison database node, and obtains the second version identifier and second timestamp information of the same data to be verified on the comparison database node. The testing device associates and stores the obtained first version identifier, first timestamp information, second version identifier, and second timestamp information according to the primary key of the data to be verified for subsequent comparison.

[0092] For example, the testing device designates the master node as the baseline database node, and slave nodes A and B as comparison database nodes. The testing device reads the data row with primary key 1001 from the order table from these three nodes, obtaining the first version identifier V3 and first timestamp T1 on the master node, the second version identifier V3 and second timestamp T1 on slave node A, and the second version identifier V2 and second timestamp T2 on slave node B. The testing device then stores this information in the result set, linked by primary key 1001.

[0093] S1042. Compare the first version identifier with the second version identifier, and compare the first timestamp information with the second timestamp information, and determine the consistency verification result data based on the comparison results.

[0094] In one implementation, after acquiring the first version identifier, the first timestamp information, the second version identifier, and the second timestamp information, the testing device performs the following comparison operations on each piece of data to be verified: It compares the first version identifier with the second version identifier numerically or in terms of size to determine if they are the same; and it compares the first timestamp information with the second timestamp information in terms of chronological order or equality to determine if they are the same. If the first version identifier and the second version identifier are the same, and the first timestamp information and the second timestamp information are the same, then the consistency verification result data is determined to be consistent; otherwise, the consistency verification result data is determined to be inconsistent, and the specific type of inconsistency is recorded.

[0095] For example, for order data with primary key 1001, the first version identifier obtained from the baseline database node is V3, and the first timestamp information is T1; the second version identifier obtained from the comparison database node is V2, and the second timestamp information is T1. After comparison, the testing device found that the first version identifier V3 and the second version identifier V2 are different, while the first timestamp information T1 and the second timestamp information T1 are the same. Therefore, the consistency verification result data is determined to indicate that the version identifiers are inconsistent.

[0096] The advantages of this optional embodiment are as follows: By acquiring the first version identifier and first timestamp information, second version identifier and second timestamp information of the same data to be verified on different nodes, and comparing the version identifier and timestamp information item by item to determine whether they are consistent, the consistency verification result data can be objectively and accurately determined. This result data directly reflects the cross-node data version divergence or timestamp disorder problems that may be caused by clock inconsistency, providing core data consistency evidence for the final determination of the abnormal situation caused by clock inconsistency to the domestic distributed database.

[0097] S105. Based on the running status data and consistency verification results, determine the abnormal situation caused by clock inconsistency to the domestic distributed database.

[0098] Among them, abnormal conditions refer to the overall performance of the domestic distributed database deviating from the expected normal state under clock inconsistency failure. Abnormal conditions are jointly characterized by operational status data and consistency verification result data. When business anomaly information indicates that there is an anomaly in transaction processing, or node synchronization anomaly information indicates that there is an anomaly in node synchronization, or consistency verification result data indicates that there is a version identifier or timestamp inconsistency between the data to be verified on different database nodes, it is determined that clock inconsistency has caused an abnormal condition for the domestic distributed database.

[0099] In one implementation, after acquiring the operational status data and consistency verification result data, the testing device compares the business anomaly information in the operational status data with a preset normal business range, compares the inter-node synchronization anomaly information in the operational status data with a preset normal synchronization range, and compares the consistency verification result data with a preset normal consistency result. If the business anomaly information exceeds the normal business range, or the inter-node synchronization anomaly information exceeds the normal synchronization range, or the consistency verification result data does not match the normal consistency result, the testing device determines that clock inconsistency has caused an abnormal situation for the domestic distributed database; if all of the above are within their respective normal ranges, it determines that no abnormal situation has been caused.

[0100] Optionally, based on the runtime status data and consistency check results, determine the abnormal conditions caused by clock inconsistency to the domestically produced distributed database, including: The transaction commit error rate is compared with the error rate threshold, and the frequency of sudden increases in consensus log synchronization delay events is compared with the frequency threshold. If at least one of the following abnormal conditions exists, it is determined that the domestic distributed database has deviated from steady state, and it is determined that clock inconsistency has caused abnormal conditions in the domestic distributed database.

[0101] The error rate threshold is a pre-defined critical value used to determine whether the transaction commit error rate is within an acceptable range. When the transaction commit error rate exceeds this threshold, it indicates a significant anomaly at the business level. This threshold can be configured according to testing requirements or the database steady-state baseline, for example, 0.01%.

[0102] The frequency threshold is a pre-defined critical value used to determine whether the frequency of sudden increases in consensus log synchronization latency events is abnormally high. When the frequency of sudden events exceeds this threshold, it indicates frequent bursts of anomalies in synchronization between nodes. This threshold can be configured according to test requirements, for example, no more than 5 times per minute.

[0103] Abnormal situations include: First abnormal situation: the transaction commit error rate exceeds the error rate threshold; Second abnormal situation: the frequency of the consensus log synchronization delay surge event exceeds the frequency threshold; Third abnormal situation: the consistency verification result data indicates that the version identifier or timestamp of the data to be verified on at least one set of database nodes is inconsistent.

[0104] Deviation from steady state refers to a situation where key indicators of a domestically developed distributed database, such as transaction processing performance, inter-node synchronization health, and data consistency, no longer meet the predetermined normal range requirements. When at least one of the first, second, or third abnormal conditions occurs, the domestically developed distributed database is determined to have deviated from steady state, indicating that clock inconsistency has had a real impact on the database.

[0105] In one implementation, after acquiring the transaction commit error rate, the frequency of consensus log synchronization delay spikes, and consistency verification results, the testing device compares the transaction commit error rate with a preset error rate threshold, compares the frequency of consensus log synchronization delay spikes with a preset frequency threshold, and checks whether the consistency verification results indicate inconsistencies in the version identifier or timestamp of the data to be verified on at least one set of database nodes. If at least one of the following conditions is detected: the transaction commit error rate exceeds the error rate threshold, the frequency of consensus log synchronization delay spikes exceeds the frequency threshold, or the consistency verification results indicate inconsistencies in the version identifier or timestamp of the data to be verified on at least one set of database nodes, the testing device determines that the domestic distributed database has deviated from a steady state and determines that clock inconsistency has caused an abnormal condition in the domestic distributed database.

[0106] The advantages of this optional embodiment are as follows: By comprehensively comparing the transaction commit error rate with the error rate threshold, the frequency of sudden increases in consensus log synchronization latency events with the frequency threshold, and consistency verification results, it is possible to comprehensively determine whether a domestically produced distributed database deviates from a steady state from three dimensions: business transaction processing, inter-node synchronization health, and data consistency. Once an anomaly occurs in any dimension, it can be determined that clock inconsistency has caused the abnormal situation, thus avoiding underestimating the impact of faults due to a single indicator, and improving the accuracy and robustness of anomaly detection. The beneficial effects of this invention are as follows: Firstly, by actively setting clock skew parameters to make the clocks of domestic distributed database nodes inconsistent, it is possible to simulate clock failures caused by network time protocol service anomalies or hardware drift, thereby exposing problems such as transaction conflicts and data overwriting caused by clock inconsistency in advance in the test environment, effectively making up for the lack of clock domain failure scenarios in existing chaos testing solutions.

[0107] Secondly, by acquiring business anomaly information and inter-node synchronization anomaly information in real time during the application of business load, and combining the comparison results of version identifiers and timestamps of the data to be verified on different nodes, it is possible to quantitatively assess the abnormal conditions caused by clock inconsistency to domestic distributed databases, accurately identify specific abnormal manifestations such as transaction commit errors, sudden increases in synchronization delays, and inconsistent data versions or timestamps, and provide clear and quantifiable basis for the fault tolerance assessment and defect localization of domestic distributed databases.

[0108] Thirdly, the provided testing methods are systematic and highly repeatable, providing reliable technical support for verifying the fault tolerance of distributed databases and ensuring the stability of domestically developed information technology systems, and significantly improving the coverage and effectiveness of clock domain chaos testing.

[0109] Optionally, after determining the abnormal conditions caused by clock inconsistency to the domestically produced distributed database based on the running status data and consistency check results, the following further steps are included: Based on the type of abnormal situation, identify the abnormal database node from the domestic distributed database; extract the error records related to the abnormal database node from the operation log of the domestic distributed database, and identify the database faulty components on the abnormal database node affected by clock inconsistency based on the module information identified in the error records.

[0110] The types of abnormal situations refer to three categories: the first abnormal situation, the second abnormal situation, and the third abnormal situation. Different types correspond to different fault location clues.

[0111] An abnormal database node refers to one or more database nodes in a domestically produced distributed database that are directly associated with anomalies. For example, when a second anomaly occurs, the slave database node experiencing a sudden increase in consensus log synchronization delay is an abnormal database node; when a third anomaly occurs, the node in the node pair containing data with inconsistent version identifiers or timestamps is an abnormal database node.

[0112] Operation logs refer to the event and status information files automatically recorded by each database node in a domestically developed distributed database during operation. The logs contain detailed information such as transaction processing, consensus protocol synchronization, node state changes, and error alarms, used for troubleshooting and analysis.

[0113] Error logs refer to event entries in the runtime log that are marked as errors, warnings, or exceptions. Error logs typically include information such as the error code, the time of occurrence, the name of the module involved, and an error description.

[0114] Module information refers to the name or identifier of the software module that caused the error, as identified in the error log. In distributed databases, module information can indicate the source of the error, such as the consensus log replication module, timestamp generator module, transaction commit manager module, etc.

[0115] Database failure components refer to specific functional modules on abnormal database nodes that malfunction due to clock inconsistencies. Database failure components include, but are not limited to, at least one of the following: consensus log replication component, timestamp generator, or transaction commit manager.

[0116] In one implementation, after identifying an anomaly, the testing device locates the abnormal database node based on the type of the anomaly. If the anomaly is a second anomaly, the testing device identifies the slave database node experiencing the latency surge as the abnormal database node. If the anomaly is a third anomaly, the testing device identifies the node in the node pair involving inconsistent data as the abnormal database node. If the anomaly is a first anomaly, the testing device further locks down the node based on subsequent log analysis. The testing device connects to the abnormal database node and extracts error records from the node's operation log that correspond to the time period of the anomaly. The testing device parses the module information identified in each error record, identifies the names of modules that recur or are related to time synchronization, and thereby determines the faulty database components on the abnormal database node affected by clock inconsistency.

[0117] For example, the testing device determined that a second anomaly had occurred: the frequency of sudden increases in consensus log synchronization latency events on node B exceeded the standard. The testing device identified node B as an abnormal database node and extracted error records from node B's operation log within one minute before and after the anomaly occurred. Multiple "consensus log application timeout" errors appeared in the logs, and the module information pointed to the "consensus log replication module." Based on this, the testing device determined that the consensus log replication component on node B was the affected database failure component.

[0118] The advantages of this optional embodiment are as follows: It can accurately locate abnormal database nodes based on the type of anomaly, and quickly identify database faulty components affected by clock inconsistencies by analyzing error records and module information in the node's operation logs. This process attributes macroscopic anomalies to specific nodes and functional modules, providing clear clues for subsequent problem-solving and improving troubleshooting efficiency.

[0119] Optionally, after determining the abnormal situation caused by the clock inconsistency to the domestic distributed database based on the running status data and the consistency verification result data, the method further includes: While keeping the clock skew parameter unchanged, continue to acquire subsequent running status data and consistency verification result data; if the subsequent transaction commit error rate drops below the error rate threshold, the frequency of subsequent consensus log synchronization delay surge events drops below the frequency threshold, and the subsequent consistency verification result data does not indicate that the version identifier or timestamp of the data to be verified on any set of database nodes is inconsistent, then it is determined that the domestic distributed database has self-healing capability; otherwise, it is determined that the domestic distributed database does not have self-healing capability.

[0120] Self-healing capability refers to the ability of a domestically developed distributed database to automatically adjust its internal state while keeping the clock skew parameter constant. This allows the transaction commit error rate to recover to below the error rate threshold, the frequency of sudden increases in consensus log synchronization delays to decrease to below the frequency threshold, and the consistency verification results to recover to indicate consistency. Having self-healing capability means that a domestically developed distributed database can recover to a normal steady state through its own fault-tolerance mechanism without external intervention.

[0121] In one implementation, after determining an abnormal situation, the testing device maintains the previously set clock skew parameter unchanged and does not cancel any clock skew. The testing device continues to acquire subsequent operational status data and subsequent consistency verification result data according to the original sampling period. The testing device monitors the above three indicators in real time: checking whether the subsequent transaction commit error rate drops below the error rate threshold, checking whether the frequency of subsequent consensus log synchronization delay surge events drops below the frequency threshold, and checking whether the subsequent consistency verification result data does not indicate that the version identifier or timestamp of the data to be verified on any set of database nodes is inconsistent. If all three conditions are met simultaneously, the testing device determines that the domestic distributed database has self-healing capabilities; if any one of the conditions is not met, the testing device determines that the domestic distributed database does not have self-healing capabilities.

[0122] For example, after setting the clock skew parameter, the test device initially measured a transaction commit error rate of 5%, a consensus log synchronization latency spike frequency of 20 times per minute, and inconsistent consistency check results. The test device kept the clock skew parameter unchanged and continued monitoring. After 30 seconds, the transaction commit error rate dropped to 0.008% (below the 0.01% threshold), the latency spike frequency dropped to 2 times per minute (below the 10-times threshold), and the new consistency check results indicated consistency. Based on this, the test device determined that the domestically developed distributed database possesses self-healing capabilities.

[0123] The advantages of this optional embodiment are as follows: By continuously monitoring subsequent operational status data and consistency verification results while keeping the clock skew parameter constant, it is possible to objectively assess whether domestically developed distributed databases possess self-healing capabilities for automatically correcting faults. This provides a quantitative testing method for verifying the robustness and reliability of database systems under extreme clock inconsistency faults, avoiding the obscuring of inherent fault tolerance defects due to external intervention.

[0124] Optionally, after determining the abnormal conditions caused by clock inconsistency to the domestically produced distributed database based on the running status data and consistency check results, the following further steps are included: Calculate the multiple by which the transaction commit error rate exceeds the error rate threshold, and / or the multiple by which the frequency of sudden increases in consensus log synchronization delay exceeds the frequency threshold; determine the severity level of the abnormal situation based on the range of the multiple; if the severity level is mild, record the abnormal information; if the severity level is moderate, trigger an alarm and generate repair suggestion information; if the severity level is severe, trigger an alarm, automatically isolate the abnormal database node, and generate repair suggestion information.

[0125] The severity levels include mild, moderate, and severe abnormalities.

[0126] In one implementation, after determining the abnormal situation, the testing device first calculates the multiple by which the transaction commit error rate exceeds the error rate threshold (i.e., the transaction commit error rate divided by the error rate threshold), and calculates the multiple by which the occurrence frequency of the consensus log synchronization delay surge event exceeds the frequency threshold (i.e., the occurrence frequency divided by the frequency threshold).

[0127] The testing device determines the severity level of the abnormality based on the preset range of the calculated multiple. The severity levels include mild, moderate and severe abnormalities. For example, a multiple between 1 (excluding) and 2 times is considered a mild abnormality, a multiple between 2 and 5 times is considered a moderate abnormality, and a multiple above 5 times is considered a severe abnormality (the specific range can be preset by the tester).

[0128] If the severity level is mild, the testing device only records the anomaly information (including anomaly type, occurrence time, involved nodes, etc.) for later review.

[0129] If the severity level is moderate, the testing device will trigger an alarm (e.g., via email or interface notification) and generate corresponding repair suggestions based on the anomaly type (e.g., "Check Network Time Protocol service configuration").

[0130] If the severity level is severe anomaly, the test device will trigger an alarm and automatically isolate the abnormal database node from the domestic distributed database (e.g., stop it from receiving service requests, remove it from the consensus group, etc.), and generate more detailed repair suggestions (such as "immediately calibrate the clock and check data consistency, etc.").

[0131] For example, if the test device measures a transaction commit error rate of 0.05% and the error rate threshold is 0.01%, then the multiplier is 5. If the preset range is: 1-2 times for mild anomalies, 2-5 times for moderate anomalies, and more than 5 times for severe anomalies, then the severity level is determined to be severe anomaly. The test device triggers an alarm, automatically isolates the master node with the abnormal transaction error rate, prevents it from processing new transaction requests, and generates a repair suggestion: "Clock deviation is too large. Please immediately synchronize the clocks of all nodes and perform data consistency repair."

[0132] The advantages of this optional embodiment are as follows: Based on the multiple by which the transaction commit error rate exceeds the error rate threshold and the multiple by which the frequency of sudden increases in consensus log synchronization latency exceeds the frequency threshold, abnormal situations are quantitatively classified into mild, moderate, and severe levels, and differentiated response measures are implemented for each level. This allows for the rational allocation of operational resources according to the severity of the fault, avoiding overreaction to minor anomalies while ensuring timely and forceful intervention for severe faults, thus improving the intelligence and automation level of the testing system.

[0133] Example 2 Figure 2 This is a schematic diagram of a chaos testing device for a domestically produced distributed database provided in Embodiment 2 of the present invention. It is applicable to assessing abnormal conditions of domestically produced distributed databases under clock inconsistency faults, such as... Figure 2 As shown, the device includes: The clock skew parameter setting module 21 is used to set clock skew parameters for at least one target database node in the domestic distributed database, so that the clock of the target database node is inconsistent with that of at least one other database node; wherein, the clock skew parameter includes clock skew direction and clock skew amount; The business load application module 22 is used to apply business load to the domestic distributed database, and the database nodes in the domestic distributed database process the transaction requests corresponding to the business load. The runtime status data acquisition module 23 is used to acquire runtime status data of the domestic distributed database during transaction processing when applying business load; wherein, the runtime status data includes business anomaly information and inter-node synchronization anomaly information; The consistency verification result data acquisition module 24 is used to determine the consistency verification result data between the data to be verified on different database nodes in the domestic distributed database based on the comparison results of the version identifier and timestamp information of the data to be verified on different database nodes after applying business load. The abnormal situation determination module 25 is used to determine the abnormal situation caused by the clock inconsistency to the domestic distributed database based on the running status data and the consistency verification result data.

[0134] Optionally, the clock offset parameter setting module 21 is specifically used for: A first database node and a second database node are determined from the domestically produced distributed database; wherein the first database node and the second database node are different from each other; A first clock skew parameter is set for the first database node; wherein the clock skew direction of the first clock skew parameter is positive and the clock skew amount is a preset first skew amount; Set a second clock offset parameter for the second database node; wherein the clock offset direction of the second clock offset parameter is negative, the clock offset amount is a preset second offset amount, and the first offset amount is different from the second offset amount.

[0135] Optionally, the consistency verification result data acquisition module 24 is specifically used for: Determine the baseline database node from the domestically produced distributed database; The first version identifier and first timestamp information of the data to be verified on the baseline database node, and the second version identifier and second timestamp information on at least one comparison database node are obtained; wherein the comparison database node is different from the baseline database node. The first version identifier is compared with the second version identifier, and the first timestamp information is compared with the second timestamp information. The consistency verification result data is determined based on the comparison result.

[0136] Optionally, the running status data acquisition module 23 is specifically used for: The number of successfully committed transactions and the number of failed transactions were collected from the domestically produced distributed database. The transaction commit error rate is calculated as the business anomaly information based on the number of successfully committed transactions and the number of failed committed transactions.

[0137] Optionally, the running status data acquisition module 23 is specifically used for: Determine the master database node and slave database nodes in the domestically produced distributed database; The first commit timestamp of the latest consensus log on the master database node and the second commit timestamp of the last applied consensus log on each slave database node are obtained in real time. Based on the difference between the first submission timestamp and the second submission timestamp, calculate the consensus log synchronization delay value for each of the slave database nodes; When the consensus log synchronization delay value of any of the slave database nodes exceeds the delay threshold at the current sampling time, and the delay value increment exceeds the increment threshold, a consensus log synchronization delay surge event is determined to have occurred, which is regarded as an abnormal synchronization information between the nodes; wherein, the delay value increment is the difference between the consensus log synchronization delay value at the current sampling time and the consensus log synchronization delay value at the previous sampling time.

[0138] Optionally, the abnormal condition determination module 25 is specifically used for: The transaction commit error rate is compared with the error rate threshold, and the frequency of the consensus log synchronization delay surge event is compared with the frequency threshold. If at least one of the following abnormal conditions exists, it is determined that the domestic distributed database has deviated from steady state, and it is determined that the clock inconsistency has caused an abnormal condition to the domestic distributed database. The abnormal situations include: First abnormal situation: The transaction commit error rate exceeds the error rate threshold; Second abnormal situation: The frequency of the consensus log synchronization delay surge event exceeds the frequency threshold; The third abnormal situation: The consistency verification result data indicates that the version identifier or timestamp of the data to be verified is inconsistent on at least one set of database nodes.

[0139] Optionally, the device further includes a fault component determination module, specifically used for: Based on the type of the anomaly, the abnormal database node is determined from the domestically produced distributed database; From the operation log of the domestic distributed database, extract the error records related to the abnormal database node, and determine the database fault components on the abnormal database node affected by the clock inconsistency based on the module information identified in the error records.

[0140] Optionally, the device further includes a self-healing ability assessment module, specifically used for: While keeping the clock offset parameter unchanged, continue to acquire the subsequent running status data and the consistency verification result data; If the subsequent transaction commit error rate drops below the error rate threshold, the frequency of subsequent consensus log synchronization delay surge events drops below the frequency threshold, and the subsequent consistency verification result data does not indicate that the version identifier or timestamp of the data to be verified on any set of database nodes is inconsistent, then it is determined that the domestic distributed database has self-healing capability. Otherwise, it is determined that the domestically produced distributed database does not have self-healing capabilities.

[0141] Optionally, the clock offset parameter setting module 21 is specifically used for: Set the drift rate; According to the drift rate, the clock deviation of the target database node is continuously adjusted so that the clock deviation of the target database node gradually increases over time.

[0142] Optionally, the device further includes an anomaly severity level assessment module, specifically used for: Calculate the multiple by which the transaction commit error rate exceeds the error rate threshold, and / or the multiple by which the frequency of the consensus log synchronization delay surge event exceeds the frequency threshold; The severity level of the abnormal condition is determined based on the range of the multiple; wherein the severity level includes mild abnormality, moderate abnormality, and severe abnormality. If the severity level is mild, then record the abnormal information; If the severity level is moderate, an alarm will be triggered and repair suggestion information will be generated; If the severity level is severe anomaly, an alarm will be triggered, the abnormal database node will be automatically isolated, and repair suggestion information will be generated.

[0143] The chaos testing device for domestically produced distributed databases provided in this embodiment of the invention can execute the chaos testing method for domestically produced distributed databases provided in this embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0144] According to embodiments of this disclosure, embodiments of the present invention also provide an electronic device, a readable storage medium, and a computer program product.

[0145] Figure 3 A schematic diagram of an electronic device 30 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0146] like Figure 3 As shown, the electronic device 30 includes at least one processor 31 and a memory, such as a read-only memory (ROM) 32 or a random access memory (RAM) 33, communicatively connected to the at least one processor 31. The memory stores computer programs executable by the at least one processor. The processor 31 can perform various appropriate actions and processes based on the computer program stored in the ROM 32 or loaded from storage unit 38 into the RAM 33. The RAM 33 can also store various programs and data required for the operation of the electronic device 30. The processor 31, ROM 32, and RAM 33 are interconnected via a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.

[0147] Multiple components in electronic device 30 are connected to I / O interface 35, including: input unit 36, such as keyboard, mouse, etc.; output unit 37, such as various types of monitors, speakers, etc.; storage unit 38, such as disk, optical disk, etc.; and communication unit 39, such as network card, modem, wireless transceiver, etc. Communication unit 39 allows electronic device 30 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0148] Processor 31 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 31 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 31 performs the various methods and processes described above, such as the chaos testing method of a domestically developed distributed database.

[0149] In some embodiments, the chaos testing method for a domestically developed distributed database can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 38. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 30 via ROM 32 and / or communication unit 39. When the computer program is loaded into RAM 33 and executed by processor 31, one or more steps of the chaos testing method for a domestically developed distributed database described above can be performed. Alternatively, in other embodiments, processor 31 can be configured to execute the chaos testing method for a domestically developed distributed database by any other suitable means (e.g., by means of firmware).

[0150] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0154] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0155] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0156] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0157] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A chaos testing method for a domestically developed distributed database, characterized in that, The method includes: A clock skew parameter is set for at least one target database node in a domestically produced distributed database, so that the clock of the target database node is inconsistent with that of at least one other database node; wherein, the clock skew parameter includes clock skew direction and clock skew amount; A business load is applied to the domestically produced distributed database, and the database nodes in the domestically produced distributed database handle the transaction requests corresponding to the business load. During the application of business load, the operational status data of the domestic distributed database during transaction processing is obtained; wherein, the operational status data includes business anomaly information and inter-node synchronization anomaly information; After applying the business load, the consistency verification result data between the data to be verified on different database nodes in the domestic distributed database is determined based on the comparison results of the version identifier and timestamp information of the data to be verified on different database nodes. Based on the operational status data and the consistency verification result data, the abnormal situation caused by the clock inconsistency to the domestic distributed database is determined.

2. The method according to claim 1, characterized in that, Setting clock skew parameters for at least one target database node in a domestically produced distributed database includes: A first database node and a second database node are determined from the domestically produced distributed database; wherein the first database node and the second database node are different from each other; A first clock skew parameter is set for the first database node; wherein the clock skew direction of the first clock skew parameter is positive and the clock skew amount is a preset first skew amount; Set a second clock offset parameter for the second database node; wherein the clock offset direction of the second clock offset parameter is negative, the clock offset amount is a preset second offset amount, and the first offset amount is different from the second offset amount.

3. The method according to claim 1, characterized in that, The step of determining the consistency verification result data between the data to be verified on different database nodes in the domestic distributed database based on the comparison results of the version identifier and timestamp information of the data to be verified on different database nodes includes: Determine the baseline database node from the domestically produced distributed database; The first version identifier and first timestamp information of the data to be verified on the baseline database node, and the second version identifier and second timestamp information on at least one comparison database node are obtained; wherein the comparison database node is different from the baseline database node. The first version identifier is compared with the second version identifier, and the first timestamp information is compared with the second timestamp information. The consistency verification result data is determined based on the comparison result.

4. The method according to claim 1, characterized in that, The step of obtaining the operational status data of the domestically produced distributed database during transaction processing includes: The number of successfully committed transactions and the number of failed transactions were collected from the domestically produced distributed database. The transaction commit error rate is calculated as the business anomaly information based on the number of successfully committed transactions and the number of failed committed transactions.

5. The method according to claim 4, characterized in that, The step of obtaining the operational status data of the domestically produced distributed database during transaction processing includes: Determine the master database node and slave database nodes in the domestically produced distributed database; The first commit timestamp of the latest consensus log on the master database node and the second commit timestamp of the last applied consensus log on each slave database node are obtained in real time. Based on the difference between the first submission timestamp and the second submission timestamp, calculate the consensus log synchronization delay value for each of the slave database nodes; When the consensus log synchronization delay value of any of the slave database nodes exceeds the delay threshold at the current sampling time, and the delay value increment exceeds the increment threshold, a consensus log synchronization delay surge event is determined to have occurred, which is regarded as an abnormal synchronization information between the nodes; wherein, the delay value increment is the difference between the consensus log synchronization delay value at the current sampling time and the consensus log synchronization delay value at the previous sampling time.

6. The method according to claim 5, characterized in that, The step of determining the abnormal situation caused by the clock inconsistency to the domestic distributed database based on the running status data and the consistency verification result data includes: The transaction commit error rate is compared with the error rate threshold, and the frequency of the consensus log synchronization delay surge event is compared with the frequency threshold. If at least one of the following abnormal conditions exists, it is determined that the domestic distributed database has deviated from steady state, and it is determined that the clock inconsistency has caused an abnormal condition to the domestic distributed database. The abnormal situations include: First abnormal situation: The transaction commit error rate exceeds the error rate threshold; Second abnormal situation: The frequency of the consensus log synchronization delay surge event exceeds the frequency threshold; The third abnormal situation: The consistency verification result data indicates that the version identifier or timestamp of the data to be verified is inconsistent on at least one set of database nodes.

7. The method according to claim 6, further comprising, after determining the abnormal situation caused by the clock inconsistency to the domestic distributed database based on the running status data and the consistency verification result data: Based on the type of the anomaly, the abnormal database node is determined from the domestically produced distributed database; From the operation log of the domestic distributed database, extract the error records related to the abnormal database node, and determine the database fault components on the abnormal database node affected by the clock inconsistency based on the module information identified in the error records.

8. The method according to claim 6, after determining the abnormal situation caused by the clock inconsistency to the domestic distributed database based on the running status data and the consistency verification result data, further includes: While keeping the clock offset parameter unchanged, continue to acquire the subsequent running status data and the consistency verification result data; If the subsequent transaction commit error rate drops below the error rate threshold, the frequency of subsequent consensus log synchronization delay surge events drops below the frequency threshold, and the subsequent consistency verification result data does not indicate that the version identifier or timestamp of the data to be verified on any set of database nodes is inconsistent, then it is determined that the domestic distributed database has self-healing capability. Otherwise, it is determined that the domestically produced distributed database does not have self-healing capabilities.

9. The method according to claim 1, characterized in that, Set clock skew parameters for at least one target database node in a domestically produced distributed database, including: Set the drift rate; According to the drift rate, the clock deviation of the target database node is continuously adjusted so that the clock deviation of the target database node gradually increases over time.

10. The method according to claim 7, further comprising, after determining the abnormal situation caused by the clock inconsistency to the domestic distributed database based on the running status data and the consistency verification result data: Calculate the multiple by which the transaction commit error rate exceeds the error rate threshold, and / or the multiple by which the frequency of the consensus log synchronization delay surge event exceeds the frequency threshold; The severity level of the abnormal condition is determined based on the range of the multiple; wherein the severity level includes mild abnormality, moderate abnormality, and severe abnormality. If the severity level is mild, then record the abnormal information; If the severity level is moderate, an alarm will be triggered and repair suggestion information will be generated; If the severity level is severe anomaly, an alarm will be triggered, the abnormal database node will be automatically isolated, and repair suggestion information will be generated.

Images