Database testing method, apparatus and device

Abstract

This application discloses a database testing method, apparatus, and device, relating to the field of performance testing technology. The method includes: setting up faults in a distributed database based on chaotic fault scenario information, wherein the chaotic fault scenario information is categorized into custom types and administrator-preset types; monitoring changes in the distributed database during and / or after the fault setting; and constructing test results based on the changes. Because it does not use a single atomic fault to test the distributed database, but rather utilizes chaotic fault scenario information constructed from combinations of at least one atomic fault, it simulates complex fault scenarios that may occur in real-world distributed database applications, thus improving test scenario coverage. Furthermore, it allows for the customization of new chaotic fault scenario information based on atomic faults to further enhance test scenario coverage, thereby improving the actual testing effect when testing distributed databases.

Description

Technical Field

[0001] This application relates to the field of performance testing technology, and in particular to database testing methods, apparatus and equipment. Background Technology

[0002] Currently, with the development of technology, the business of various industries is becoming increasingly complex. Many industries (such as the financial industry) are gradually transforming their application systems to distributed architectures. The complexity of databases is increasing dramatically, and they are gradually developing into distributed databases (such as GoldenDB database or OceanBase database).

[0003] However, the related technologies have shortcomings and deficiencies in their solutions for distributed database testing: In distributed database application scenarios, failures are often related and cascading. The correlation and cascading of failures can lead to complex operational and maintenance decision-making dilemmas. Currently, testing is basically aimed at independent, pre-set single failure points, which cannot cover the complex business scenarios of distributed database production environments. This results in insufficient test coverage and a gap between the predicted system availability and the actual production environment. Summary of the Invention

[0004] The main purpose of this application is to provide a database testing method, apparatus, and equipment, which aims to solve the technical problem that related technologies, when testing distributed databases, have limited testing scenarios, cannot simulate complex real-world scenarios, and result in poor actual testing effects.

[0005] To achieve the above objectives, this application proposes a database testing method applied to a database testing device, the method comprising: The distributed database is configured to perform fault settings based on chaotic fault scenario information. The chaotic fault scenario information is classified into custom types and administrator preset types. Monitor changes in the distributed database during and / or after fault setup; Test results are constructed based on the changes described.

[0006] Optionally, the step of configuring fault settings for the distributed database based on chaotic fault scenario information includes: Determine the classification type of the chaotic fault scenario information; If the classification type is the administrator's preset type, then obtain the preset scenario classification corresponding to the chaotic fault scenario information; If the preset scenario is classified as peak coupling detection, then the system load of the distributed database is monitored; If the system load is greater than or equal to a preset load threshold, then at least one first node is selected from the data nodes of the distributed database; A fault setting command is sent to the first node to make the first node unavailable.

[0007] Optionally, if the system load is greater than or equal to a preset load threshold, then selecting at least one first node from the data nodes of the distributed database includes: If the system load is greater than or equal to a preset load threshold, then obtain the node load corresponding to each data node in the distributed database; The data nodes in the distributed database are sorted from high to low based on their respective node loads. Select the first N data nodes from the sorted results as the first node, where N is a preset positive integer.

[0008] Optionally, if the classification type is an administrator preset type, then after obtaining the preset scenario classification corresponding to the chaotic fault scenario information, the method further includes: If the preset scenario is classified as collaborative decision detection, then at least one second node is selected from the data nodes of the distributed database; Inject a read / write latency fault into the second node to cause the second node to be in an abnormal working state; When the duration of the second node's abnormal operation reaches a preset duration, a network partition fault is injected into the second node to disconnect it from other data nodes in the distributed database.

[0009] Optionally, if the classification type is an administrator preset type, then after obtaining the preset scenario classification corresponding to the chaotic fault scenario information, the method further includes: If the preset scenario is classified as extreme resilience testing, then the target shard is selected from the data shards of the distributed database; Fault injection is performed on the master node of the target shard to trigger a master-slave switchover in the distributed database; When the primary / standby switchover is completed and the new primary node in the target shard is synchronizing data, a storage fault is injected into the new primary node.

[0010] Optionally, the step of injecting a storage fault into the new master node after the primary / standby switchover is completed and the new master node in the target shard is synchronizing data includes: When the primary / standby switchover is completed and the new primary node in the target shard is synchronizing data, the amount of synchronized data and the total amount of differential data are obtained through the data synchronization status interface. The repair progress is determined based on the amount of synchronized data and the total amount of differential data, and the data throughput change rate and average data throughput corresponding to the new master node are obtained. The recovery vulnerability is determined based on the repair progress, the data throughput change rate, and the average data throughput. If the repair progress is greater than or equal to the repair progress threshold, and the recovery vulnerability is greater than or equal to the vulnerability threshold, then a storage fault is injected into the new master node.

[0011] Optionally, the change information includes performance dimension data, consistency dimension data, and resource dimension data of the distributed database during the fault setting period, as well as data conflict information and conflict resolution strategies of the distributed database after the fault setting. The step of constructing test results based on the change information includes: Based on the change information, an experimental trajectory, resilience score, and root cause analysis are constructed. The experimental trajectory is used to characterize the timeline of key events in the distributed database recovery process after fault injection. The resilience score is a score value used to characterize the overall resilience of the distributed database. The root cause analysis is used to characterize and identify the weak links in the distributed database recovery process. The test results are constructed based on the experimental trajectory, the resilience score, and the root cause analysis.

[0012] Optionally, after determining the classification type of the chaotic fault scenario information, the method further includes: If the classification type is a custom type, then a fault information sequence is determined based on the chaotic fault scenario information, and the fault information sequence is constructed based on at least one atomic fault. The fault information sequence is traversed, and the traversed fault information is used as the current fault information; Determine the fault setting target, fault setting conditions, and fault setting type based on the current fault information; When the fault setting conditions are met, fault setting is performed on the fault setting target in the distributed database according to the setting method corresponding to the fault setting type. Once the setup is complete, return to the step of traversing the fault information sequence and using the traversed fault information as the current fault information, until all fault information in the fault information sequence has been traversed.

[0013] Furthermore, to achieve the above objectives, this application also proposes a database testing apparatus, which includes: The configuration module is used to configure fault settings for the distributed database based on chaotic fault scenario information. The chaotic fault scenario information can be classified into custom types and administrator preset types. The detection module is used to monitor changes in the distributed database during and / or after fault setting. A construction module is used to construct test results based on the change information.

[0014] In addition, to achieve the above objectives, this application also proposes a database testing device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the database testing method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the database testing method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the database testing method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: Instead of using single atomic faults to detect distributed databases, this method utilizes chaotic fault scenarios constructed from combinations of at least one atomic fault to simulate complex fault scenarios that may occur in real-world distributed database applications. This improves test scenario coverage. Furthermore, it allows for the creation of new chaotic fault scenario information based on atomic faults, further enhancing test scenario coverage and improving the actual testing effectiveness of distributed databases. In addition, it analyzes complex application scenarios of distributed databases, identifying several well-defined and important preset scenario categories to ensure accurate detection of critical scenarios. It also provides examples for users to customize chaotic fault scenarios, enabling them to better orchestrate chaotic fault scenario information. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1This is a flowchart illustrating the database testing method of this application in Embodiment 1. Figure 2 This is a schematic diagram illustrating the division of database testing function modules according to an embodiment of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the database testing method of this application; Figure 4 This is a flowchart illustrating Embodiment 3 of the database testing method of this application; Figure 5 This is a schematic diagram of a real-world collaborative decision-making scenario according to an embodiment of this application; Figure 6 This is a flowchart illustrating Embodiment 4 of the database testing method of this application; Figure 7 This is a schematic diagram of the ultimate toughness testing process according to an embodiment of this application; Figure 8 This is a schematic diagram of the module structure of the database testing device according to an embodiment of this application; Figure 9 This is a schematic diagram of the hardware operating environment involved in the database testing method in this application embodiment.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] Based on this, the embodiments of this application provide a database testing method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the database testing method of this application.

[0025] In this embodiment, the database testing method is applied to a database testing device and includes steps S10 to S30: Step S10: Configure fault settings for the distributed database based on chaotic fault scenario information. The chaotic fault scenario information can be categorized into custom types and administrator preset types.

[0026] It should be noted that the execution entity of this embodiment can be the database testing device, or a platform or system composed of at least one database testing device. The database testing device can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. The following uses a database testing device as an example to describe the database testing methods of this embodiment and the following embodiments.

[0027] It should be noted that chaotic fault scenario information is categorized according to the setting method. The categorization types can include custom types and administrator preset types. The administrator preset type of chaotic fault scenario information can be preset by the administrator, while the custom type can be customized by the user.

[0028] In practical use, chaotic fault scenario information can be stored in a fault scenario library. When testing is required, users (such as testers) can select the required chaotic fault scenario information from the fault scenario library. For example, the administrator can pre-set a fault scenario library to store chaotic fault scenario information of the administrator's preset type. If the chaotic fault scenario information of the administrator's preset type is insufficient to meet the current fault requirements, the user can customize complex fault scenarios, generate custom type chaotic fault scenario information, and store it in the fault scenario library.

[0029] The chaotic fault scenario information can be composed of at least one atomic fault (such as node restart, read / write (I / O) latency, network partition, etc.) according to a preset timing sequence, triggering conditions (such as triggering when the system load reaches a certain level, or triggering when a previous fault is successfully set and lasts for a certain duration), and logical dependencies (fault setting needs to be after a certain fault setting, such as triggering network partition only after read / write latency). The distributed database can be GoldenDB or OceanBase, or other similar or identical distributed databases; this embodiment does not limit this.

[0030] In practical applications, fault settings can be configured on a distributed database based on at least one atomic fault that constructs chaotic fault scenario information, in order to simulate real fault scenarios.

[0031] Step S20: Monitor the changes in the distributed database during and / or after fault setting.

[0032] Step S30: Construct test results based on the change information.

[0033] In practical use, in order to ensure that the actual response effect of the distributed database can be determined, change information during and / or after fault setup can be collected, and then test results can be constructed based on the change information.

[0034] Depending on the fault settings, the collected change information can also be different. For example, for network partition faults, it can detect whether the distributed database removes the node after the fault is set. For read / write latency issues, it is possible to detect whether the distributed database reduced the amount of data distributed to that node during the fault setup period; For combined failures of read / write latency and network partitioning, the specific measures taken by the distributed database during and after the failure setup can be collected to determine whether the distributed database has taken appropriate action.

[0035] In practical use, when setting up chaotic fault scenario information, a corresponding evaluation method can also be set. At this time, the change information can be compared with the evaluation method corresponding to the chaotic fault scenario information to generate test results.

[0036] For example: Suppose the chaotic failure scenario is a combination of read / write latency and network partition failure scenario. The evaluation method is to reduce the data allocation by a certain percentage (e.g., 50%) when read / write latency occurs on a data node, and when network partition failure occurs on a data node, the data node should be removed from the list of available data nodes in the distributed database within a preset time (e.g., 5 seconds). At this point, the actual monitored changes can be compared with the evaluation method to generate test results (such as whether the test passed or failed, and the reasons for the test failure can also be included).

[0037] To facilitate understanding, we will now combine... Figure 2 This explanation is provided, but it does not limit the scope of this solution. Figure 2 This is a schematic diagram showing the division of the database testing function module in this embodiment.

[0038] like Figure 2 As shown, the database testing equipment assembly constructs an integrated chaos engineering platform architecture, which integrates the core components of a general chaos testing platform with the enhancement function layer required by specific industries (such as the financial industry): Top layer (user interface and scheduling layer): The chaos testing platform serves as a visual web interface, providing users with an entry point for scenario orchestration, experiment operation, and state observation, and integrates access control (such as RBAC). Behind it is the experiment management module, responsible for defining test scenarios, orchestrating multiple fault experiments (workflows), and setting termination conditions; The core control layer, the Chaos Controller Manager, is the brain of the platform. It includes a scheduler and various fault-type controllers, responsible for parsing user-defined chaos experiments (CRDs) and managing their lifecycle. This layer corresponds to the security control module, which implements explosion radius control strategies such as resource isolation, traffic coloring, and circuit breaking mechanisms to ensure test safety. Fault Injection Execution Layer: The Chaos Daemon runs on each node as a DaemonSet with high execution privileges. It is responsible for injecting network, file system, kernel, and other faults into the target Pod or physical machine (i.e., setting faults in the distributed database based on chaotic fault scenario information). This constitutes the fault injection module, which provides atomic fault operation interfaces.

[0039] Data Acquisition and Observation Layer: Collects various metrics from infrastructure to application performance in real time to obtain information on changes in the distributed database.

[0040] Results Analysis Layer: Analyzes monitoring data (i.e. change information), generates system fault tolerance assessment reports and compliance inspection reports (i.e. test results), and completes the experimental loop.

[0041] This embodiment provides a database testing method. Instead of using a single atomic fault to test the distributed database, it uses chaotic fault scenario information constructed by combining at least one atomic fault to simulate complex fault scenarios that may occur in real-world applications of distributed databases. This improves the test scenario coverage. At the same time, it allows for the customization of new chaotic fault scenario information based on atomic faults to further improve the test scenario coverage and enhance the actual testing effect when testing distributed databases.

[0042] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S10 includes steps S101 to S105: Step S101: Determine the classification type of the chaotic fault scenario information.

[0043] In practical use, when storing chaotic fault scenario information, corresponding fields can be set according to the configuration method to distinguish the classification type. Then, the field corresponding to the chaotic fault scenario information can be read to determine the classification type of the chaotic fault scenario information.

[0044] For example, the fault scenario library stores chaotic fault scenario information through a data table. This data table has a type field. If type=0, it means the administrator's preset type; if type=1, it means the custom type.

[0045] Of course, depending on actual needs, a corresponding mapping table (such as a mapping table between the scene identifier and the classification type of chaotic fault scene information) can be set up to store the relationship between chaotic fault scene information and classification type, or the classification type can be represented by the keywords in the scene identifier of chaotic fault scene information (such as if the scene identifier starts with admin, it represents the administrator's preset type; if it starts with user, it represents the custom type). This embodiment does not impose any restrictions on this.

[0046] Step S102: If the classification type is the administrator's preset type, then obtain the preset scenario classification corresponding to the chaotic fault scenario information.

[0047] In practical use, if the classification type is the administrator's preset type, it means that this chaotic fault scenario information is a specific scenario preset by the administrator. For this specific scenario, since it is preset rather than customized, the administrator can also preset the corresponding processing method according to the type of the specific scenario. Therefore, the preset scenario classification corresponding to the chaotic fault scenario information can be determined first.

[0048] Step S103: If the preset scenario is classified as peak coupling detection, then monitor the system load of the distributed database.

[0049] It should be noted that if the preset scenario is classified as peak coupling detection, it means that the goal is to detect how the distributed database handles situations where some data nodes fail during peak business periods.

[0050] In practical use, in order to ensure that the real business peak period can be simulated, a large number of data read and write requests can be sent to the distributed database according to the way the real business system accesses the distributed database, and the system load of the distributed database can be monitored in real time.

[0051] If a business system is deployed in the test environment and is directly connected to the distributed database, then simulated business requests can be sent directly to the business system to enable it to perform corresponding business processing and access the distributed database, while the system load of the distributed database can be monitored in real time.

[0052] In practical use, the specific method used to simulate access to a distributed database in a real-world scenario can be determined based on the system deployment method in the actual test environment. This embodiment does not impose any restrictions on this.

[0053] The system load of the distributed database can be the proportion of resources currently used by the distributed database to the total resources, such as CPU utilization, memory usage, disk read / write usage, etc. This embodiment does not impose any restrictions on this.

[0054] Step S104: If the system load is greater than or equal to a preset load threshold, then select at least one first node from the data nodes of the distributed database.

[0055] In practical use, if the system load of the distributed database is greater than or equal to the preset load threshold, it means that the access scenario of the distributed database during the peak business period has been successfully simulated. At this time, fault injection can be attempted. Therefore, at least one first node can be selected from the data nodes of the distributed database.

[0056] The first node is the data node of the distributed database that needs to be injected with faults.

[0057] In a specific implementation, to ensure the effectiveness of the test, step S104 in this embodiment may include: If the system load is greater than or equal to a preset load threshold, then obtain the node load corresponding to each data node in the distributed database; The data nodes in the distributed database are sorted from high to low based on their respective node loads. Select the first N data nodes from the sorted results as the first node, where N is a preset positive integer.

[0058] It should be noted that in order to ensure the test results, it is necessary to ensure that the subsequent failures have a significant impact on the distributed database. In fact, the higher the node load of a data node, the greater the amount of business carried by the node, and the greater the impact of its failure on the distributed database. Therefore, the data nodes in the distributed database can be sorted from high to low according to their node loads, and the top N data nodes in the sort can be selected as the first node.

[0059] Where N is a preset positive integer, and the specific value can be set in advance by the administrator of the database testing equipment, for example, N can be set to 3.

[0060] Of course, N can also be set based on the total number of data nodes in the distributed database. For example, if the distributed database has 30 data nodes, and you need to test how the distributed database handles a 20% failure rate, then N can be set to 30. 20% = 6.

[0061] The examples above are for illustrative purposes only. The specific setting method and value of N can be adjusted by the administrator of the database testing equipment according to actual needs. This embodiment does not impose any restrictions on this.

[0062] Step S105: Send a fault setting command to the first node to make the first node enter an unavailable state.

[0063] It should be noted that, in order to test how the distributed database handles the failure of some data nodes during peak business periods, it is necessary to simulate the failure of some data nodes. Therefore, a fault setting command can be sent to the first node to make the first node unavailable.

[0064] Specifically, when a data node restarts, it cannot provide services, which will cause the data node to enter an unavailable state. Therefore, in order to ensure rapid setup, the fault setup command can be a batch restart command, that is, send a batch restart command to the first node, so that all the first nodes enter the restart state.

[0065] Of course, other fault setting commands can also be sent as needed. For example, if a data node cannot read or write to other nodes, it will also cause the data node to enter an unavailable state. In this case, a read / write prohibition command can be sent to the first node to inject read / write faults into each first node so that the first node enters an unavailable state.

[0066] In practical applications, peak coupling detection is used to test the distributed database's ability to cope with sudden batch failures of data nodes during peak business periods. The information collected on the changes in the distributed database can include the time taken to detect the first node becoming unavailable after fault settings, whether the node load on non-unavailable data nodes has increased, whether the increase in node load is even, and whether an attempt has been made to recover the first node (e.g., whether a command has been sent to control the first node to restart).

[0067] In a specific implementation, to ensure that user-defined chaos tests can be executed correctly, this embodiment may further include the following after step S101: If the classification type is a custom type, then a fault information sequence is determined based on the chaotic fault scenario information, and the fault information sequence is constructed based on at least one atomic fault. The fault information sequence is traversed, and the traversed fault information is used as the current fault information; Determine the fault setting target, fault setting conditions, and fault setting type based on the current fault information; When the fault setting conditions are met, fault setting is performed on the fault setting target in the distributed database according to the setting method corresponding to the fault setting type. Once the setup is complete, return to the step of traversing the fault information sequence and using the traversed fault information as the current fault information, until all fault information in the fault information sequence has been traversed.

[0068] It should be noted that, in order to ensure that user-defined chaos tests can be executed correctly, when the classification type of chaotic fault scenario information is determined to be a custom type, the fault information of multiple atomic faults corresponding to the chaotic fault scenario information can be assembled in the corresponding time sequence to construct a fault information sequence. Then, the fault information sequence can be traversed, and the traversed fault information can be used as the current fault information.

[0069] In practical use, relevant information about fault settings, such as triggering conditions, logical dependencies, and fault setting types, can be extracted from the current fault information, and fault setting targets, fault setting conditions, and fault setting types can be constructed based on this.

[0070] The fault setting target can be a target that needs to be set with faults, such as a data node or shard in a distributed database. Fault setting conditions can be conditions that trigger the fault setting action, such as the load on the distributed database, or the requirement that a certain fault setting needs to be completed. The fault setting type can be the type of fault to be set, such as node restart, I / O latency, network partition, etc.

[0071] Afterwards, performance metrics and resource metrics of the distributed database can be monitored, or the execution status of previous fault settings can be monitored. When the fault setting conditions are met, the fault setting is performed on the fault setting target in the distributed database according to the setting method corresponding to the fault setting type. After the setting is completed, the fault information sequence is traversed until all fault information in the fault information sequence has been traversed.

[0072] In the specific implementation, in order to facilitate the construction of test reports, when users customize the chaotic fault scenario information, they can also set corresponding indicators of the distributed database that need to be monitored for each atomic fault in the construction of the chaotic fault scenario information. Then, during and / or after the fault setting process, relevant data of the distributed database can be collected according to the indicators to obtain change information.

[0073] This embodiment provides a database testing method. This embodiment conducts targeted fault testing on a specific scenario of peak load coupling to ensure that the distributed database can be tested to cope with the failure of some data nodes during such business peaks.

[0074] Based on the first embodiment of this application, in the third embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 After step S102, steps S103'~S105' are also included: Step S103': If the preset scenario is classified as collaborative decision detection, then at least one second node is selected from the data nodes of the distributed database.

[0075] It should be noted that if the preset scenario is classified as collaborative decision detection, it means that it is necessary to detect how the distributed database responds to data nodes in abnormal working states. Therefore, it is necessary to first simulate abnormal data nodes. At this time, at least one second node can be selected from the data nodes of the distributed database.

[0076] When selecting the second node, a random selection method can be used. The number of nodes selected can be preset by the administrator of the database testing equipment. For example, 1-3 nodes can be randomly selected from the data nodes of the distributed database as the second node.

[0077] Step S104': Inject a read / write delay fault into the second node to put the second node into an abnormal working state.

[0078] In practical use, the abnormal data node in the early stages will show anomalies such as read and write delays. Therefore, read and write delay faults can be injected into the second node to make the second node work abnormally.

[0079] For example, sending a read / write delay command to the second data node causes a delay of a certain duration (such as 0.5 seconds, 1 second, or more) each time the second data node performs a read / write operation, thereby causing the second node to experience a read / write delay fault and put the second node in an abnormal working state.

[0080] Step S105': When the duration of the second node being in an abnormal working state reaches a preset duration, inject a network partition fault into the second node to cause the second node to disconnect from other data nodes in the distributed database.

[0081] In practical use, read / write latency failures are usually caused by certain faults within the data nodes (such as network failures). The impact of such failures generally increases gradually. To simulate a real scenario, the second node can be kept in an abnormal working state and timed. When the duration of the second node's abnormal working state reaches the preset duration, a network partition failure is injected into the second node to disconnect it from other data nodes in the distributed database, thereby simulating the changes of real abnormal nodes over time.

[0082] The preset duration can be set in advance by the administrator of the database testing equipment, for example, the preset duration can be set to 60 seconds.

[0083] In practical applications, for collaborative decision-making detection, the collected change information can include: When a read / write delay failure occurs on the second node, does the distributed database send an interrupt read / write command to the second node? When a read / write latency failure occurs on the second node, should the distributed database redirect read / write requests originally distributed to the second node to other data nodes? When a network partition failure occurs on the second node, causing it to lose connection with other data nodes in the distributed database, should the distributed database accelerate the eviction of the second node, or start the system and wait for a timeout while waiting for the second node to respond?

[0084] To facilitate understanding, we will now combine... Figure 5 This explanation is provided, but it does not limit the scope of this solution. Figure 5 This is a schematic diagram of a real-world collaborative decision-making scenario in this embodiment.

[0085] like Figure 5 As shown, the distributed database cluster (including the control plane and multiple data nodes) is in a healthy state initially, with normal heartbeats between the control plane and all working data nodes, and all nodes are healthy at this time; If a DataNode becomes a slow node due to slow disk I / O or network congestion, its latency in processing data packets or reporting heartbeats will increase significantly (this is simulated here by injecting read / write latency faults into a second node). For example, in HDFS, this manifests as abnormal transmission or disk write times in the write pipeline; in Kubernetes, it manifests as latency in Kubelet reporting NodeStatus.

[0086] The control plane (such as the Controller Manager in Kubernetes) marks the slow node as NotReady because it does not receive timely heartbeats from the slow node. Subsequently, due to network device failure, a network partition may occur between the region where the slow node is located and the control plane (simulated here by injecting a network partition failure into a second node), and connectivity is completely interrupted.

[0087] Subsequently, the control plane begins to evict Pods on slow nodes.

[0088] Subsequently, if the cluster is configured with autoscaling, it may mistakenly determine that a node is useless and directly delete the underlying virtual machine, causing the risk of irreversible data loss. Due to slow nodes and network partitions, the data replication chain may be interrupted, which may lead to inconsistency of data in some partitions or degradation of availability. After that, the cluster may enter a cascading failure state with large-scale node disconnection and service unavailability.

[0089] The aforementioned control fault injection simulates a second node becoming a slow node, followed by network partitioning and disconnection from other nodes. This simulates a scenario where a data node first becomes a slow node, and then the connectivity with the control plane is completely interrupted. By collecting change information, the test determines how the distributed database responds to this scenario, thus completing the test and generating a test report.

[0090] This embodiment provides a database testing method. This embodiment has made targeted settings for the collaborative decision-making scenario to ensure that the development trend of abnormal states of nodes can be simulated, so as to test the response effect of the distributed database to the scenario in which the abnormal state of nodes gradually deepens.

[0091] Based on the first embodiment of this application, in the fourth embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6 After step S102, steps S103''~S105'' are also included: Step S103'': If the preset scenario is classified as extreme resilience detection, then select the target shard from the data shards of the distributed database.

[0092] It should be noted that if the preset scenario is classified as extreme resilience testing, it means that it is necessary to test the response of the distributed database when it fails again during the recovery period. In order to simulate the recovery period of the distributed database, one can be randomly selected from the multiple data shards of the distributed database as the target shard.

[0093] Step S104'': Inject faults into the master node of the target shard to trigger a master-slave switchover in the distributed database.

[0094] In practical use, in order to simulate the recovery period of a distributed database, it is necessary to ensure that data recovery is necessary. At this time, fault injection can be performed on the master node of the target shard, so that the master node of the target shard cannot provide services. Then, the distributed database will trigger a master-slave switch in the target shard due to the disaster recovery mechanism. At this time, the distributed database will re-elect a new master node for the target shard from the other data nodes (which can be called slave nodes) besides the master node.

[0095] In order to ensure that a master-slave switchover can be triggered, a restart command or a network partition can be sent to the master node of the target shard to cause a failure that directly causes the master node of the target shard to be unable to provide services.

[0096] In practical use, the master node of the target shard can be the data node that provides the main service (such as write function) in the target shard. For example, if the distributed database contains three data shards (or more), and each data shard has 1 master node and 2 slave nodes, then there are a total of 9 data nodes in the distributed database.

[0097] Step S105'': When the primary / standby switchover is completed and the new primary node in the target shard is synchronizing data, a storage fault is injected into the new primary node.

[0098] In practical applications, if the role of the new primary node changes from backup to primary, the primary / backup switchover can be considered complete. For example, when the role of the new primary node changes from "Secondary" to "Primary", the primary / backup switchover is confirmed to be complete.

[0099] The data synchronization status of the new master node in the target shard can be determined by the operation of the data synchronization status interface.

[0100] In practical use, in order to simulate the distributed database failing again during the recovery period, a storage fault is injected into the new master node when the master-slave switch is completed and the new master node is synchronizing data (such as synchronizing data from the data replica to repair data differences). This achieves a second strike, causing the simulated distributed database to fail again during the recovery period.

[0101] The injected storage fault can be a disk bad block fault. For example, if the database runs on a physical machine or virtual machine, and the disk is simulated via a SCSI / SATA interface (in most cases), the kernel's SCSI debugging function can be used to modify specific parameters in the / sys file system. This can cause the kernel to intentionally return a media error or hardware error when it receives a write request for that disk, thus injecting a storage fault. Of course, disk bad block faults or other types of storage faults can also be injected through other methods; this embodiment does not limit this.

[0102] In order to accurately identify and precisely inject faults to achieve a precise secondary strike, step S105'' in this embodiment may include: When the primary / standby switchover is completed and the new primary node in the target shard is synchronizing data, the amount of synchronized data and the total amount of differential data are obtained through the data synchronization status interface. The repair progress is determined based on the amount of synchronized data and the total amount of differential data, and the data throughput change rate and average data throughput corresponding to the new master node are obtained. The recovery vulnerability is determined based on the repair progress, the data throughput change rate, and the average data throughput. If the repair progress is greater than or equal to the repair progress threshold, and the recovery vulnerability is greater than or equal to the vulnerability threshold, then a storage fault is injected into the new master node.

[0103] In practical use, the amount of synchronized data and the total amount of differing data can be obtained through the data synchronization status interface. Then, the ratio of the synchronized data amount to the total amount of differing data is calculated; this ratio represents the repair progress, also known as the repair progress percentage. Data throughput can be I / O throughput, i.e., read / write data throughput.

[0104] The data throughput change rate and the average data throughput can be determined by monitoring the data throughput of the new master node. For example, monitor the data throughput of the new master node in the last 10 seconds, take the average data throughput in the last 10 seconds as the average data throughput, and take the change rate of data throughput in the last 10 seconds as the data throughput change rate.

[0105] In practical applications, recovery vulnerability can be calculated using the vulnerability calculation formula based on repair progress, data throughput change rate, and average data throughput. The vulnerability calculation formula is as follows: V=1-P (1-A) W 变化 / W 平均 ); In the formula, P represents the repair progress; W 变化 W represents the data throughput change rate. 平均 V represents the average data throughput; V represents the resilience vulnerability. The higher the V value, the more vulnerable the system / cluster is, while V-order 0 indicates that the system / cluster has returned to a steady state. A is the weighting coefficient, which can be preset by the administrator of the database testing equipment. The value range can be [0.3, 0.5], and it can also be adjusted according to actual needs.

[0106] Understandably, in order to achieve a precise second strike, the repair progress needs to reach a certain level to ensure that the fault is injected again when the repair is in progress but not yet completed, and the distributed database system / cluster is relatively vulnerable. Therefore, storage faults can be injected into the new master node when the repair progress is greater than or equal to the repair progress threshold and the recovery vulnerability is greater than or equal to the vulnerability threshold.

[0107] In practical implementation, in the scenario of ultimate toughness testing, the collected change information may include: The performance, consistency, and resource data of the distributed database during fault setup, as well as the data conflict information and conflict resolution strategies of the distributed database after fault setup; Performance data may include: transaction response latency, transaction error rate, cluster throughput (TPS), etc. Consistency dimension data may include: data difference between replicas Diff (which can be calculated by comparing the number of checks performed periodically) and data synchronization progress P (also known as repair progress). Resource-level data can include: CPU utilization (CPU) of each node, disk I / O wait time (IOWait), etc.

[0108] Data conflict information can include whether data conflicts exist and the amount of data conflicts. Typical characteristics of data conflicts include: different replicas have inconsistent version numbers for the same data, or the transaction commit log has timestamps that are reversed. Data conflicts can be determined based on these characteristics. If a data conflict is detected, the distributed database will automatically trigger a preset conflict resolution mechanism to repair the data. At this point, the conflict resolution strategy used by the distributed database can be determined, and this part is the core of verifying the database's fault tolerance complexity.

[0109] Common conflict resolution strategies generally include: Timestamp comparison: The data version with the newest timestamp is used for overwriting; Source-side priority: based on the data version of the master node; Application business log backtracking: performing reverse compensation for transactions.

[0110] Subsequently, regardless of whether data conflicts occur and are resolved, the system will eventually complete data synchronization (possibly incrementally or fully) and perform business switching. The chaos platform will continue to monitor until it is confirmed that all shards have recovered consistent data and the cluster service status has returned to normal.

[0111] At this point, corresponding test results can be generated. Accordingly, in this embodiment, step S30 may include: Based on the aforementioned change information, experimental trajectories, resilience scores, and root cause analyses were constructed. The test results are constructed based on the experimental trajectory, the resilience score, and the root cause analysis.

[0112] It should be noted that the experimental trajectory is used to characterize the timeline of key events in the distributed database recovery process after fault injection, the resilience score is used to characterize the overall resilience of the distributed database, and the root cause analysis is used to characterize and identify the weak links in the distributed database recovery process.

[0113] In practical use, the collected change information can be used to verify whether the recovery process can succeed when the distributed database suffers a secondary failure during the recovery phase, and whether the recovery time objective (RTO) / recovery point objective (RPO) meets expectations. This generates experimental trajectories, resilience scores, and root cause analyses, and combines the generated data into test results.

[0114] Afterwards, the test results can be displayed to the administrators of the database testing equipment.

[0115] To facilitate understanding, we will now combine... Figure 7 This explanation is provided, but it does not limit the scope of this solution. Figure 7 This is a schematic diagram of the ultimate toughness testing process in this embodiment.

[0116] like Figure 7 As shown, the ultimate toughness testing process includes steps 1-12: Step 1: Start the planned data recovery The chaos testing platform initiated a planned disaster recovery exercise. The platform issued instructions to the master node to initiate a pre-defined "planned recovery" process. This marks the start of the entire "second-strike" test, which aims to simulate a normal, orderly failover scenario and establish a baseline for subsequent "second-strike" scenarios.

[0117] Step 2: The master sends a stop synchronization command. Upon receiving the recovery command, the master node, acting as the initiator, sends a standard "stop data synchronization" command to the backup node. This command is to sever the real-time data flow between the current production and disaster recovery processes, preparing for subsequent business failover.

[0118] Step 3: The backup device stops synchronization and updates its status. Upon receiving the "Stop Synchronization" command, the standby node immediately executes the command, ceasing to receive incremental data from the master node. Simultaneously, the standby node modifies its own cluster status flag from "Disaster Recovery Synchronization in Progress" to "Synchronization Stopped, Preparing for Switchover".

[0119] Step 4: The master terminal confirms the status and prepares for the switch. The master node learns through the cluster status monitoring interface that the backup node has successfully stopped synchronization. After confirming that the status is correct, the master node begins to prepare to execute the business failover process, such as redirecting read traffic or some write traffic that was originally handled by the backup node to the local database.

[0120] In this context, steps 1-4 correspond to steps S103''-S104'' in the above embodiment. The master node is the master node in the target shard before the first fault injection, and the backup node is the new master node in the target shard after the first fault injection.

[0121] Step 5: Automatic identification of the vulnerability period during recovery After a distributed database master-slave failover, the system enters a "recovery vulnerability period"—at this time, the standby node becomes the new master node, and the new master node is pulling differential data from the replicas for synchronization and repair. Data consistency has not yet been fully restored, and the cluster's election mechanism and fault detection logic are in a sensitive period. During this window, the system's resilience is at its lowest, making it the best time to inject secondary faults.

[0122] The automatic recognition process is as follows: 5.1 Primary / Backup Switchover Event Detection When a failure occurs in the injection of the master node, monitor in real time whether the cluster's metadata service (such as GoldenDB's GTM node) issues a master-slave switchover notification.

[0123] The specific technical approach is as follows: subscribe to the node role change event stream of the cluster, and when the role of the new primary node is detected to have changed from "Secondary" to "Primary", it is confirmed that the primary-standby switchover has been triggered.

[0124] 5.2 Data Synchronization Progress Monitoring After the master-slave switchover is complete, the new master node will pull the difference data from other available replicas for synchronization and repair. The repair progress and vulnerability status can be monitored in real time using the following steps: (1) Periodically query the data synchronization status interface of the new master node to obtain the ratio of "synchronized data volume / total differential data volume" as the repair progress percentage P; (2) Simultaneously monitor the write I / O throughput W and read I / O throughput R of the new master node. When W is significantly lower than R and continues to decrease, it indicates that data synchronization occupies the main I / O bandwidth and the system is in the recovery period rather than the steady state. (3) If (2) detects that the recovery period is in progress, calculate the recovery vulnerability index: V = 1 - P (1 – A (Rate of change in I / O throughput / Average I / O throughput) V = 1 - P (1 – A (W change / W average) Where P represents the percentage of repair progress, and A is understood as a weighting coefficient. This value is set to 0.5 based on our experience, and under normal circumstances, this weighting value should be between 0.3 and 0.5.

[0125] The result is that the higher the value of V, the more fragile the system is, and V close to 0 indicates that the system has returned to a steady state.

[0126] 5.3 Precise Determination of Secondary Fault Injection Timing Set a repair progress threshold P1 (e.g., 30%) and a vulnerability threshold V1 (e.g., 0.7). Here, P1 and V1 are modifiable thresholds. When P ≥ P1 and V ≥ V1, the system is determined to be at the "precise strike point" of the vulnerability window during the recovery period, and a secondary fault injection is triggered.

[0127] Step 6: Precise injection of secondary faults Precise injection of storage faults Once the timing is deemed suitable, a storage I / O suspension fault is injected into the new master node. This is achieved using the single fault injection function of the Source Chaos Engineering Platform.

[0128] b. Real-time observation and data acquisition during fault injection During and after the secondary fault injection, the following multidimensional monitoring data were continuously collected: Performance metrics: transaction response latency, transaction error rate, cluster throughput (TPS); Consistency dimensions: Data difference between replicas (Diff, calculated by comparing the number of checks performed periodically), data synchronization progress (P); Resource dimensions: CPU utilization of each node, CPU, disk I / O wait time (IO) Wait wait.

[0129] Steps 5-6 correspond to step S105'' in the above embodiment, that is, during the recovery process, and when it is a vulnerable period, fault injection is performed again to inject storage faults in order to achieve a second attack.

[0130] Step 7: The system continues the recovery process under abnormal conditions. After suffering a secondary impact, the data synchronization process of the new master node is interrupted or severely slowed down. At this point, the fault tolerance mechanism of the distributed database is triggered, and the system is forced to continue attempting to complete or correct the original recovery process under abnormal conditions. This stage is a critical period for exposing system design flaws; for example, the system may attempt to start a new data synchronization thread or trigger isolation mechanisms.

[0131] Step 8: Has a data conflict occurred? The chaos testing platform continuously monitors the cluster status to determine whether a secondary attack has led to data conflicts.

[0132] Typical characteristics of data conflicts include: inconsistent version numbers for the same data across different replicas, or reversed timestamps in transaction commit logs.

[0133] Step 9: Trigger conflict resolution strategies (such as timestamp comparison / source priority) If step 8 determines that a data conflict has occurred, the system will automatically trigger a preset conflict resolution mechanism to repair the data. Common mechanisms include: a. Timestamp comparison: Overwrite with the data version that has the newest timestamp.

[0134] b. Source-side priority: Based on the data version of the master node.

[0135] c. Application business log backtracking: Perform reverse compensation for transactions. This step is the core of verifying the database's fault tolerance complexity.

[0136] Step 10: Complete data synchronization and business switchover Regardless of whether data conflicts occur and are resolved, the system will eventually complete data synchronization (possibly incrementally or fully) and perform business switching. The chaos platform will continue to monitor until it is confirmed that all shards have recovered consistent data and the cluster service status has returned to normal.

[0137] Step 11: Verification of panoramic monitoring The chaos testing platform conducts final verification from multiple dimensions and in all aspects, collecting and evaluating key resilience indicators: Performance metrics: whether latency, error rate, and throughput have recovered to steady-state levels.

[0138] Consistency metrics: By using data verification tools, the number of checks on different node replicas is compared to ensure that the data is consistent.

[0139] Resource metrics: Whether CPU, memory, and I / O utilization have returned to normal ranges.

[0140] Recovery Time Objective (RTO): Calculates the total time elapsed from the initial fault injection to service recovery.

[0141] Recovery Point Objective (RPO): Assess the acceptable amount of data loss under the impact of a "second strike".

[0142] Step 12: Output Chaos Test Report The entire testing process is now complete. The chaos testing platform automatically generates a structured, quantitative test result. The result not only includes the final conclusion (pass / fail, RTO / RPO values), but also provides detailed information: a. Experimental trajectory: Timeline of key events throughout the recovery process.

[0143] b. Resilience Score: A comprehensive resilience score calculated based on multi-dimensional indicators.

[0144] c. Root cause analysis: Precisely locate the weak links in the recovery process.

[0145] In the example above, the system in steps 1-12 is the system or cluster corresponding to the distributed database, and this embodiment does not limit it.

[0146] This embodiment provides a database testing method. This embodiment can simulate a scenario where a distributed database experiences further failures during the recovery period, thereby ensuring that the distributed database can be tested to ensure normal recovery and data consistency in the face of such secondary failures.

[0147] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the database testing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0148] This application also provides a database testing device; please refer to [reference needed]. Figure 8 The database testing device includes: Setting module 10 is used to set faults in the distributed database according to chaotic fault scenario information. The classification types of the chaotic fault scenario information include custom types and administrator preset types. Detection module 20 is used to monitor changes in the distributed database during and / or after fault setting; Module 30 is used to construct test results based on the change information.

[0149] The database testing apparatus provided in this application, employing the database testing method described in the above embodiments, can solve the technical problem that related technologies, when testing distributed databases, suffer from limited testing scenarios, inability to simulate complex real-world scenarios, and poor actual testing results. Compared with the prior art, the beneficial effects of the database testing apparatus provided in this application are the same as those of the database testing method provided in the above embodiments, and other technical features in the database testing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0150] This application provides a database testing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the database testing method in Embodiment 1 above.

[0151] The following is for reference. Figure 9The diagram illustrates a structural schematic of a database testing device suitable for implementing embodiments of this application. The database testing device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The database testing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0152] like Figure 9 As shown, the database testing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the database testing device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the database testing equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a database testing equipment with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0153] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0154] The database testing device provided in this application, employing the database testing method described in the above embodiments, can solve the technical problem that related technologies, when testing distributed databases, suffer from limited testing scenarios, inability to simulate complex real-world scenarios, and poor actual testing results. Compared with the prior art, the beneficial effects of the database testing device provided in this application are the same as those of the database testing method provided in the above embodiments, and other technical features of this database testing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0155] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0156] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0157] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the database testing method in the above embodiments.

[0158] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0159] The aforementioned computer-readable storage medium may be included in the database testing device; or it may exist independently and not be assembled into the database testing device.

[0160] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the database testing device, the database testing device causes the following: to configure the distributed database for fault settings based on chaotic fault scenario information, wherein the chaotic fault scenario information is classified into custom types and administrator preset types; to monitor changes in the distributed database during and / or after the fault settings; and to construct a test result construction module based on the changes, for constructing test results based on the changes.

[0161] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0163] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0164] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described database testing method. This solves the technical problem that related technologies, when testing distributed databases, suffer from limited testing scenarios, inability to simulate complex real-world scenarios, and poor actual testing results. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the database testing method provided in the above embodiments, and will not be repeated here.

[0165] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the database testing method described above.

[0166] The computer program product provided in this application can solve the technical problem that related technologies, when testing distributed databases, have limited testing scenarios, cannot simulate complex real-world scenarios, and result in poor actual testing performance. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the database testing method provided in the above embodiments, and will not be repeated here.

[0167] All user-related data involved in this application (such as user privacy data, user behavior data, etc.) were obtained with the user's permission or consent; that is to say, when this application is used in a specific product or technology, user permission is required to obtain and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations and regulatory standards of the relevant countries and regions.

[0168] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A database testing method, characterized in that, The database testing method, applied to database testing equipment, includes: The distributed database is configured to perform fault settings based on chaotic fault scenario information. The chaotic fault scenario information is classified into custom types and administrator preset types. Monitor changes in the distributed database during and / or after fault setup; Test results are constructed based on the changes described.

2. The database testing method as described in claim 1, characterized in that, The step of configuring fault settings for the distributed database based on chaotic fault scenario information includes: Determine the classification type of the chaotic fault scenario information; If the classification type is the administrator's preset type, then obtain the preset scenario classification corresponding to the chaotic fault scenario information; If the preset scenario is classified as peak coupling detection, then the system load of the distributed database is monitored; If the system load is greater than or equal to a preset load threshold, then at least one first node is selected from the data nodes of the distributed database; A fault setting command is sent to the first node to make the first node unavailable.

3. The database testing method as described in claim 2, characterized in that, If the system load is greater than or equal to a preset load threshold, then at least one first node is selected from the data nodes of the distributed database, including: If the system load is greater than or equal to a preset load threshold, then obtain the node load corresponding to each data node in the distributed database; The data nodes in the distributed database are sorted from high to low based on their respective node loads. Select the first N data nodes from the sorted results as the first node, where N is a preset positive integer.

4. The database testing method as described in claim 2, characterized in that, If the classification type is an administrator preset type, then after obtaining the preset scenario classification corresponding to the chaotic fault scenario information, the method further includes: If the preset scenario is classified as collaborative decision detection, then at least one second node is selected from the data nodes of the distributed database; Inject a read / write latency fault into the second node to cause the second node to be in an abnormal working state; When the duration of the second node's abnormal operation reaches a preset duration, a network partition fault is injected into the second node to disconnect it from other data nodes in the distributed database.

5. The database testing method as described in claim 2, characterized in that, If the classification type is an administrator preset type, then after obtaining the preset scenario classification corresponding to the chaotic fault scenario information, the method further includes: If the preset scenario is classified as extreme resilience testing, then the target shard is selected from the data shards of the distributed database; Fault injection is performed on the master node of the target shard to trigger a master-slave switchover in the distributed database; When the primary / standby switchover is completed and the new primary node in the target shard is synchronizing data, a storage fault is injected into the new primary node.

6. The database testing method as described in claim 5, characterized in that, The step of injecting a storage fault into the new master node after the primary / standby switchover is completed and data synchronization is performed in the target shard includes: When the primary / standby switchover is completed and the new primary node in the target shard is synchronizing data, the amount of synchronized data and the total amount of differential data are obtained through the data synchronization status interface. The repair progress is determined based on the amount of synchronized data and the total amount of differential data, and the data throughput change rate and average data throughput corresponding to the new master node are obtained. The recovery vulnerability is determined based on the repair progress, the data throughput change rate, and the average data throughput. If the repair progress is greater than or equal to the repair progress threshold, and the recovery vulnerability is greater than or equal to the vulnerability threshold, then a storage fault is injected into the new master node.

7. The database testing method as described in claim 6, characterized in that, The change information includes performance dimension data, consistency dimension data, and resource dimension data of the distributed database during the fault setting period, as well as data conflict information and conflict resolution strategies of the distributed database after the fault setting. The step of constructing test results based on the change information includes: Based on the change information, an experimental trajectory, resilience score, and root cause analysis are constructed. The experimental trajectory is used to characterize the timeline of key events in the distributed database recovery process after fault injection. The resilience score is a score value used to characterize the overall resilience of the distributed database. The root cause analysis is used to characterize and identify the weak links in the distributed database recovery process. The test results are constructed based on the experimental trajectory, the resilience score, and the root cause analysis.

8. The database testing method as described in claim 2, characterized in that, After determining the classification type of the chaotic fault scenario information, the method further includes: If the classification type is a custom type, then a fault information sequence is determined based on the chaotic fault scenario information, and the fault information sequence is constructed based on at least one atomic fault. The fault information sequence is traversed, and the traversed fault information is used as the current fault information; Determine the fault setting target, fault setting conditions, and fault setting type based on the current fault information; When the fault setting conditions are met, fault setting is performed on the fault setting target in the distributed database according to the setting method corresponding to the fault setting type. Once the setup is complete, return to the step of traversing the fault information sequence and using the traversed fault information as the current fault information, until all fault information in the fault information sequence has been traversed.

9. A database testing device, characterized in that, The database testing device includes: The configuration module is used to configure fault settings for the distributed database based on chaotic fault scenario information. The chaotic fault scenario information can be classified into custom types and administrator preset types. The detection module is used to monitor changes in the distributed database during and / or after fault setting. A construction module is used to build test results based on the change information.

10. A database testing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the database testing method as described in any one of claims 1 to 8.

Images