A distributed system, and a fault testing method, apparatus and medium thereof

Abstract

The application discloses a kind of distributed systems, and its fault test method, device and medium, it is related to distributed system technical field, for realizing the fault test of distributed system, for the fault superposition problem that traditional fault test scheme is difficult to solve, provide a kind of distributed system fault test method, client / server architecture is adopted, the independent fault test of each node can be realized, whether the fault cause and fault processing meet the expectation are all in-depth to each node, so as to effectively avoid the problem that the fault between multiple nodes is difficult to judge because of superposition. Based on this, the fault test of distributed system is carried out by the method, can improve test efficiency and accuracy, so as to be favorable to improve the software quality of distributed system, reduce maintenance cost, reduce the additional analysis workload of development.

Description

Technical Field

[0001] This invention relates to the field of distributed systems, and in particular to a distributed system, and its fault testing method, apparatus and medium. Background Technology

[0002] A distributed system is a system composed of multiple computers and a communication network. Unlike a centralized system, it has no central control node; each node can independently handle tasks. This architecture offers more efficient performance for storing, processing, and querying large-scale data. As a product, distributed systems typically require various tests to ensure their performance and reliability.

[0003] Currently, fault testing of distributed systems still largely relies on manual methods, using black-box testing to perform targeted fault tests on various functions of the distributed system. However, when using this approach, problems often arise where overlapping faults prevent testers from accurately analyzing the cause of the failure. In distributed systems, because each node can operate independently and there is a certain degree of coupling between them, when several nodes experience different failures, the external manifestation is a composite fault resulting from the superposition of multiple faults. Therefore, it is difficult for testers to determine whether the fault handling meets expectations, requiring development intervention. Alternatively, there may be situations where the fault handling appears to meet expectations, but the actual software logic is not functioning correctly. Both of these problems hinder accurate fault testing of distributed systems, thus affecting their performance and reliability.

[0004] Therefore, those skilled in the art urgently need a distributed system fault testing method to address the problem that traditional fault testing schemes cannot guarantee testing accuracy in scenarios such as distributed systems where faults are prone to overlapping. Summary of the Invention

[0005] The purpose of this invention is to provide a distributed system, and its fault testing method, apparatus and medium, to solve the problem that traditional fault testing schemes are difficult to guarantee testing accuracy in scenarios such as distributed systems where faults are prone to superposition.

[0006] To address the aforementioned technical problems, this invention provides a method for testing faults in a distributed system, applicable to a distributed system; the distributed system includes multiple nodes, wherein one node acts as a client and the other nodes act as servers;

[0007] This method is applied to the client and includes:

[0008] Obtain fault testing strategies; among which, a set of fault testing strategies includes: the node's Internet Protocol address, fault injection command, fault recovery command, status monitoring command, and result judgment configuration information;

[0009] Based on the Internet Protocol address, the fault injection command, fault recovery command, and status monitoring command in the same fault testing strategy group are sent to the corresponding server, so that the server can: execute the fault injection command to perform fault injection and obtain the fault injection result; execute the fault recovery command to perform fault recovery; execute the status monitoring command to monitor the changes of the specified status after fault injection and fault recovery and obtain the status monitoring result; and return the fault injection result and the status monitoring result to the client.

[0010] Receive the fault injection result and the status monitoring result returned by the server;

[0011] Based on the result determination configuration information, the fault test result is determined according to the corresponding fault injection result and the status monitoring result.

[0012] In one possible embodiment, determining the fault test result based on the configuration information determined by the result, and according to the corresponding fault injection result and the status monitoring result, includes:

[0013] Based on the fault injection result, determine whether the fault injection on the server was successful;

[0014] If the injection failure is successful, the configuration information and the status monitoring results are used to determine whether the change in the specified status is in line with expectations.

[0015] If the conditions are met, the fault test result is considered a successful test.

[0016] If the test does not meet the requirements, the test result is deemed a failure.

[0017] If the injection fails, the fault injection command is resent to the corresponding server so that the server can perform fault injection again.

[0018] This method also includes:

[0019] Count the number of times the same fault injection command is sent repeatedly;

[0020] If the number of repetitions exceeds the preset repetition threshold, an alarm will be triggered and a corresponding prompt message will be returned.

[0021] The prompt information includes: the fault injection command and the corresponding Internet Protocol address.

[0022] In one possible embodiment, if multiple sets of the fault test strategies are obtained;

[0023] Then, according to the Internet Protocol address, sending the fault injection command, the fault recovery command, and the status monitoring command from the fault testing strategy in the same group to the corresponding server includes:

[0024] Based on the Internet Protocol address, the fault testing strategies of each group are classified.

[0025] Each set of fault injection commands, fault recovery commands, and status monitoring commands corresponding to a certain server is packaged and sent to the corresponding server.

[0026] In one possible embodiment, before sending the fault injection command, the fault recovery command, and the status monitoring command from the same fault testing strategy to the corresponding server according to the Internet Protocol address, the method further includes:

[0027] Get node configuration information;

[0028] Based on the node configuration information, a communication connection establishment request is sent to each of the servers to establish a communication connection with each of the servers.

[0029] To address the aforementioned technical problems, this invention also provides a method for testing faults in a distributed system, applicable to a distributed system; the distributed system includes multiple nodes, wherein one node acts as a client and the other nodes act as servers;

[0030] This method is applied to the server and includes:

[0031] Receive fault injection commands, fault recovery commands, and status monitoring commands sent by the client;

[0032] Perform fault injection according to the fault injection command and obtain the fault injection result;

[0033] After fault injection, fault recovery is performed according to the fault recovery command;

[0034] The status monitoring command is used to monitor the changes in the specified status after fault injection and fault recovery, and the status monitoring results are obtained.

[0035] The fault injection result and the status monitoring result are returned to the client so that the client can determine the fault test result based on the result determination configuration information that belongs to the same group of fault test strategies as the fault injection command, the fault recovery command and the status monitoring command, according to the corresponding fault injection result and status monitoring result.

[0036] In one possible embodiment, it further includes:

[0037] Create a first process and a second process;

[0038] The first process is configured to: execute the fault injection command and return the fault injection result to the client; and execute the fault recovery command after executing the fault injection command.

[0039] The second process is used to: execute the status monitoring command and return the status monitoring result to the client;

[0040] The first process and the second process are two processes running in parallel.

[0041] In one possible embodiment, returning the fault injection result and the status monitoring result to the client includes:

[0042] The fault injection result and the status monitoring result are analyzed to obtain multiple result items containing time-series features;

[0043] The results items are reordered according to their chronological order and merged into a single result data packet.

[0044] The result data packet is sent to the client.

[0045] In one possible embodiment, before receiving the fault injection command, fault recovery command, and status monitoring command sent by the client, the method further includes:

[0046] Open a socket connection and wait for the client to send a communication connection establishment request;

[0047] After receiving the communication connection establishment request sent by the client, respond to the communication connection establishment request to establish a communication connection with the client.

[0048] To address the aforementioned technical problems, the present invention also provides a distributed system fault testing device, characterized in that it is applied to a distributed system; the distributed system includes: multiple nodes, wherein one node acts as a client and the other nodes act as servers;

[0049] This device is used in the client and includes:

[0050] The strategy acquisition module is used to acquire fault testing strategies; among them, a set of fault testing strategies includes: the node's Internet Protocol address, fault injection command, fault recovery command, status monitoring command, and result judgment configuration information;

[0051] The strategy distribution module is used to send the fault injection command, the fault recovery command, and the status monitoring command in the same group of fault testing strategies to the corresponding server according to the Internet Protocol address, so that the server can: execute the fault injection command to perform fault injection and obtain the fault injection result; execute the fault recovery command to perform fault recovery; execute the status monitoring command to monitor the changes of the specified status after fault injection and fault recovery and obtain the status monitoring result; and return the fault injection result and the status monitoring result to the client.

[0052] The result receiving module is used to receive the fault injection result and the status monitoring result returned by the server;

[0053] The fault determination module is used to determine the fault test result based on the result determination configuration information, according to the corresponding fault injection result and the status monitoring result.

[0054] To address the aforementioned technical problems, the present invention also provides a distributed system fault testing device, applied to a distributed system; the distributed system includes: multiple nodes, wherein one node acts as a client and the other nodes act as servers;

[0055] This device is used on the server side and includes:

[0056] The command receiving module is used to receive fault injection commands, fault recovery commands, and status monitoring commands sent by the client.

[0057] The fault injection module is used to inject faults according to the fault injection command and obtain the fault injection result;

[0058] The fault recovery module is used to perform fault recovery according to the fault recovery command after fault injection;

[0059] The status monitoring module is used to monitor the changes of a specified status after fault injection and fault recovery according to the status monitoring command, and to obtain the status monitoring results.

[0060] The result return module is used to return the fault injection result and the status monitoring result to the client, so that the client can: determine the fault test result based on the result judgment configuration information that belongs to the same group of fault test strategies as the fault injection command, the fault recovery command and the status monitoring command, according to the corresponding fault injection result and status monitoring result.

[0061] Distributed systems applied on the client side include: multiple nodes; one node acts as a client, and the other nodes act as servers;

[0062] in:

[0063] The client is used to implement the steps of the distributed system fault testing method applied to the client side as described above;

[0064] The server is used to implement the steps of the distributed system fault testing method applied to the server side as described above.

[0065] To address the aforementioned technical problems, the present invention also provides a non-volatile storage medium storing a computer program, which, when executed by a processor, implements the steps of the distributed system fault testing method described above.

[0066] This invention provides a distributed system fault testing method. By utilizing existing nodes in the distributed system, it divides them into client and server components, each executing different fault testing logic. Specifically, after obtaining the fault testing strategy for actual fault testing, the client sends corresponding fault injection commands, fault recovery commands, and status monitoring commands to the corresponding server (i.e., the corresponding node) based on the node's Internet Protocol (IP) address. The server then receives and executes these commands to complete the corresponding fault testing process and returns the corresponding results (fault injection results and status monitoring results) to the client. After receiving the test results returned by the server, the client can process the test results according to the result determination configuration information in the fault testing strategy to determine the fault test result of the server, thus completing one fault test case for that node.

[0067] In summary, this method employs a client / server (C / S) architecture. The client can perform independent fault testing on each server within the distributed system, achieving fault detection at the node level and effectively resolving the problem of fault overlap between nodes. Fault testing based on this method enables independent fault testing of each node, delving into the cause of the fault and whether the fault handling meets expectations at each node, thus effectively avoiding the difficulty in diagnosing faults caused by the overlap of multiple nodes. Therefore, using this method for fault testing of distributed systems can improve testing efficiency and accuracy, thereby contributing to improved software quality, reduced maintenance costs, and reduced additional analysis workload during development.

[0068] The distributed system, its fault testing device, and the non-volatile storage medium provided by this invention correspond to the above-mentioned methods and have the same effects. Attached Figure Description

[0069] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A structural diagram of a distributed system provided by the present invention;

[0071] Figure 2 A flowchart illustrating a distributed system fault testing method applied to the client side, provided by the present invention;

[0072] Figure 3 A flowchart of a distributed system fault testing method applied to the server side provided by the present invention;

[0073] Figure 4 The present invention provides a structural diagram of a distributed system fault testing device applied on the client side;

[0074] Figure 5 This invention provides a structural diagram of a distributed system fault testing device applied to the server side. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0076] The core of this invention is to provide a distributed system, and its fault testing method, apparatus and medium.

[0077] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] For devices and systems with high data processing capabilities, such as computers, testing their software is a crucial means of ensuring device and system performance. One of the most common testing scenarios is fault testing. Distributed systems, as systems composed of multiple computers and communication networks, also require fault testing. Currently, the mainstream fault testing approach involves manual black-box testing to specifically test the functions of the distributed system to determine if any faults have occurred.

[0079] However, due to the unique structure of distributed systems, which contain multiple independently running nodes with a certain degree of coupling between them, a particular service may be implemented through the collaborative efforts of multiple nodes. Therefore, during fault testing, a situation of overlapping faults may occur, meaning that multiple nodes may fail, and these faults may affect each other, resulting in a more complex and cumulative fault manifestation.

[0080] Because of this problem of overlapping faults, there are too many combinations of node states and software fault handling functions after a fault. It requires testers who are proficient in fault handling functions and developers of distributed systems to personally intervene and analyze the fault test, and it is also difficult to guarantee the accuracy of the fault test.

[0081] Therefore, to address the fault testing problem in scenarios where faults overlap and are prone to occur in distributed systems, this invention provides a distributed system fault testing method. This method is applied to distributed systems, such as... Figure 1 As shown, the distributed system consists of multiple nodes. One node acts as a client, and the other nodes act as servers.

[0082] It should be noted that this method is implemented on both the client and server sides of the aforementioned distributed system. The following embodiments will provide a detailed explanation of this method from both the client and server perspectives:

[0083] First, it is applied to the client, such as Figure 2 As shown, the method includes:

[0084] S11: Obtain the fault test strategy.

[0085] One set of fault testing strategies includes: the node's Internet Protocol address, fault injection command, fault recovery command, status monitoring command, and result determination configuration information.

[0086] S12: Based on the Internet Protocol address, send the fault injection command, fault recovery command, and status monitoring command in the same group of fault testing strategies to the corresponding server.

[0087] To facilitate the server in: executing fault injection commands to perform fault injection and obtaining fault injection results; executing fault recovery commands to perform fault recovery; executing status monitoring commands to monitor changes in the specified status after fault injection and fault recovery and obtaining status monitoring results; and returning fault injection results and status monitoring results to the client.

[0088] S13: Receive the fault injection results and status monitoring results returned by the server.

[0089] S14: Based on the result determination configuration information, determine the fault test result according to the corresponding fault injection result and status monitoring result.

[0090] For step S11, it's easy to understand that a distributed system generally includes one client and multiple servers. A single fault test performed on a server is achieved through a set of fault test strategies. Accordingly, in practical applications, if this method is used to perform fault tests on multiple nodes (servers), or to perform multiple different fault tests on a single node, the fault test strategies obtained in step S11 can include multiple sets.

[0091] Each set of fault testing strategies corresponds to one fault test case for a single node. Specifically, as described above, a set of fault testing strategies includes, but is not limited to: the node's Internet Protocol (IP) address, fault injection commands, fault recovery commands, status monitoring commands, and result determination configuration information. Among these:

[0092] 1. The IP address of the node, which represents the node (server) corresponding to this fault testing strategy. The client can determine the server applicable to this fault testing strategy by the node's IP address.

[0093] 2. Fault injection commands, which are commands used to simulate fault scenarios. When executed by the corresponding node, this command can be used to inject specific faults, thereby simulating the corresponding fault environment on the node and achieving fault testing.

[0094] In other words, a fault injection command is a command that simulates a specific fault environment; it may be a single command or a collection of multiple commands. For example, if the fault test item is a network sub-health fault on the test node, a possible fault injection command would include:

[0095] Commands for initializing a network in a sub-healthy environment, commands for modifying network sub-healthy configuration parameters, and commands for injecting network latency.

[0096] Since the above commands are common commands in the actual operation of distributed storage systems, this embodiment will not elaborate on the specific command format or implementation.

[0097] In addition, this embodiment also provides another example: if the fault test item is a slow disk access fault on the test node, a possible fault injection command includes:

[0098] Commands for initializing a slow disk access environment and commands for injecting a slow disk failure into a disk.

[0099] As can be seen from the above examples, this method supports any single node fault test case. In the fault injection command stage, it can be obtained by simulating the corresponding fault environment instruction set.

[0100] 3. Fault recovery commands, which are commands that implement recovery methods for corresponding faults. They are used to recover the faults simulated by the fault injection commands mentioned above.

[0101] Similar to the fault injection commands mentioned above, fault recovery commands can be a single command or a collection of multiple commands, depending on the specific fault type. When the fault testing strategy uses existing fault test cases and the fault type is known, the corresponding fault handling method is also known, and the specific fault handling instructions are the fault recovery commands mentioned above.

[0102] 4. Status monitoring commands are instructions used to monitor specific status changes that require attention during fault injection and fault repair.

[0103] It's easy to see that the specific state changes to monitor differ depending on the type of fault. For example, in the network sub-health fault test described above, the focus is on the changes in the node's network interface status (the changes between the network interface's up and down states). Similarly, for the slow disk access fault in another example, the focus is on the OSD process status and the Ceph storage service status.

[0104] Ceph is an open-source distributed storage service, while OSD is the process by which Ceph accesses the disk. It's important to note that if the OSD's status changes to "down," the Ceph status will change from "ok" to "warning."

[0105] 5. Result judgment configuration information, that is, the basis for judging the test results of a specific fault.

[0106] It is easy to understand that, since different fault types focus on different state changes, the corresponding criteria for judging the results are also different.

[0107] For example, for the network sub-health fault test case in the above example, one possible result determination configuration information includes: the network interface status becomes down within a first time range after the fault is injected; and the network interface status returns to up within a second time range after the fault is injected. In practical applications, the first time range can be 20~50s, and the second time range can be 100~150s.

[0108] Furthermore, for the disk access slow failure test case in the above example, one possible result determination configuration information includes: the OSD becomes down within the third time range after the failure injection; and the Ceph status reverts to warning within the fourth time range after the failure injection. In practical applications, the third time range can be 20~50s, and the fourth time range can be 20~60s.

[0109] Therefore, based on existing test cases for possible node failure types, these test cases can be encapsulated into a fault testing strategy using the aforementioned instructions. In practical applications, the fault testing strategy can be encapsulated as a fault testing interface, which can be directly called to implement the corresponding fault test cases. Furthermore, this method also supports the extension of fault test cases. By specifying the IP address of the node under test, converting the fault simulation process, fault recovery process, and monitoring process of the state of interest into instructions executable by the node under test, and encapsulating the corresponding fault result judgment criteria, the fault testing strategy can be extended to implement any type of fault test case.

[0110] Based on this, for step S12, the node IP address in the fault test strategy is mainly used by the client to determine the corresponding server and does not need to be sent to the server; the fault injection command, fault recovery command, and status monitoring command are executed on the server and need to be sent by the client to the specified server according to the IP address after obtaining the fault test strategy in order to complete the fault test of the specified node; the result determination configuration information does not need to be sent to the server. It is the basis for the client to determine the final fault test result based on the results returned by the server (fault injection result and status monitoring result) and does not need to be sent to the server.

[0111] That is, in step S12, for the implementation of any fault test case, after the client obtains the complete fault test strategy, the fault injection command, fault recovery command and status monitoring command in the fault test strategy are selectively sent to the corresponding server, and the rest do not need to be sent to the server.

[0112] Correspondingly, after receiving the above command, the server needs to return the corresponding result to the client to complete the execution of the command. Specifically, the fault injection result corresponding to the fault injection command includes two types: fault injection success and fault injection failure. The status monitoring result is relatively complex and needs to record the changes in specific states according to different actual concerns. It should also be noted that the determination of the above results should also include temporal characteristics, that is, such as obtaining a fault injection success / failure result at a certain moment, or a change in the state of concern at a certain moment.

[0113] Next, step S13 involves obtaining the fault injection result and status monitoring result returned by the corresponding server. It's easy to see that, based on the node IP address returned by the server, a correspondence can be established between the server-returned result and the result judgment configuration information. This allows step S14 to process and judge the fault injection result and status monitoring result using the result judgment configuration information, obtaining the final fault test result, determining whether the fault test was successful or failed, and completing one fault test.

[0114] As described above, this invention provides a method for fault testing in distributed systems, dividing each node in the distributed system into client and server types, thus constructing a client / server (C / S) architecture. The client can independently test the faults of each server within the distributed system, achieving fault detection at the node level and effectively solving the problem of fault overlap between nodes. Fault testing based on this method allows for independent fault testing of each node, delving into the cause of the fault and whether the fault handling meets expectations at each node, effectively avoiding the difficulty in judging faults caused by the overlap of multiple nodes. Therefore, using this method for fault testing of distributed systems can improve testing efficiency and accuracy, thereby contributing to improved software quality, reduced maintenance costs, and reduced additional analysis workload during development.

[0115] Furthermore, based on the distributed system fault testing method provided on the client side described above, this embodiment also provides a possible implementation scheme for determining the fault test result, wherein step S14 specifically includes:

[0116] S141: Based on the fault injection result, determine whether the server successfully injected the fault; if the fault injection is successful, proceed to step S142; if the fault injection fails, proceed to step S145.

[0117] S142: Based on the results of the configuration information and status monitoring, determine whether the change of the specified status meets expectations; if it does, proceed to step S143; if it does not, proceed to step S144.

[0118] S143: The fault test result is determined to be a successful test.

[0119] S144: The fault test result is determined to be a test failure.

[0120] S145: Resend the fault injection command to the corresponding server so that the server can perform fault injection again.

[0121] It should be noted that step S145 in this embodiment can be achieved by re-executing step S12, or by simply resending the fault injection command in a single step. This embodiment does not impose any restrictions on this.

[0122] Accordingly, this method also includes:

[0123] S151: Count the number of times the same fault injection command is sent repeatedly.

[0124] S152: If the number of repetitions exceeds the preset repetition threshold, an alarm will be triggered and the corresponding prompt message will be returned.

[0125] The prompt message includes: the fault injection command and the corresponding Internet Protocol address.

[0126] In other words, in this embodiment, the judgment of any fault test result is based on the premise that the fault scenario is normally simulated. That is, the result of this fault test can only be judged based on the status monitoring result after the fault is successfully injected. If the fault injection fails, the fault test is invalid. If the fault injection is successful, the fault test is valid. The status monitoring result can be judged based on the result judgment configuration information to determine whether it meets expectations. If it does, the fault test result is a successful test; otherwise, the test fails.

[0127] Furthermore, for the aforementioned scenario where fault testing is ineffective, this embodiment provides a possible solution by attempting to resolve the problem of ineffective fault testing through a repetitive writing mechanism. This solution has a good effect on resolving occasional problems such as sudden network failures and connection anomalies, and can solve some of the problems of ineffective fault testing, thereby improving the reliability of this method.

[0128] Furthermore, if repeatedly writing fault injection commands fails to simulate the fault scenario correctly, it indicates a persistent anomaly may have occurred in the distributed system. In this case, this embodiment issues an alarm and returns corresponding prompts to notify operations personnel of the anomaly and locate the node where the anomaly occurred, enabling them to promptly investigate the anomaly and resume subsequent fault testing.

[0129] Furthermore, based on the distributed system fault testing method provided on the client side described above, this embodiment also provides a possible implementation scheme for distributing fault injection commands, fault recovery commands, and status monitoring commands:

[0130] If multiple sets of fault test strategies are obtained in step S11 above, then step S12 specifically includes:

[0131] S121: Classify the fault testing strategies of each group according to Internet Protocol (IP) addresses;

[0132] S122: Package and send the fault injection commands, fault recovery commands, and status monitoring commands corresponding to a certain server to the corresponding server.

[0133] It is readily apparent that distributed systems typically involve more than two nodes, generally including multiple servers. As demonstrated in the above embodiments, this method can execute fault test cases on each node individually. Furthermore, during fault testing, multiple different nodes can be tested simultaneously to improve testing efficiency. Additionally, different fault tests can be performed on the same node. Therefore, in step S11, multiple sets of fault test strategies will be obtained. These strategies may correspond to different servers or the same server, depending on the actual fault testing requirements.

[0134] Furthermore, after the client obtains multiple sets of fault testing strategies, the command distribution in step S12 can be implemented using the same distribution method. That is, as shown in the above scheme of this embodiment, all fault testing strategies are classified according to the node's IP address, and the corresponding fault injection commands, fault recovery commands, and status monitoring commands are packaged and sent to the corresponding server, thus completing the distribution of multiple sets of fault testing strategies to a specific server. Similarly, by distributing the grouped instructions to the corresponding servers, all instructions can be distributed, improving fault testing efficiency in the preparation stage before fault testing.

[0135] Furthermore, based on the distributed system fault testing method provided on the client side described above, this embodiment also provides a possible implementation scheme, wherein the method further includes the following step before step S12:

[0136] S161: Obtain node configuration information;

[0137] S162: Send a communication connection establishment request to each server according to the node configuration information in order to establish a communication connection with each server.

[0138] It is readily apparent that a distributed system contains multiple nodes, which are divided into client and server sides based on the requirements of this method. However, during a fault test, not all nodes in the distributed system need to undergo fault testing. Among the nodes that do not need to be tested, any one node can be selected as the aforementioned client node, while the other nodes that need to be tested become the aforementioned server nodes. The other nodes that do not need to be tested can be excluded from this fault test process.

[0139] It should also be noted that this embodiment does not limit the specific data included in the node configuration information, and can be determined according to the communication needs between nodes in the actual distributed system. For example, in one possible implementation, the above-mentioned node configuration information includes: the node's IP address and port number.

[0140] As described above, the client and server are not static during each fault test, but can change as needed. Furthermore, to avoid impacting irrelevant nodes (i.e., nodes that do not require fault testing), this embodiment provides a targeted communication connection method. Specifically, by pre-entering node configuration information into a specific node, the client is identified (the node whose configuration information is entered is the client); the server is also identified (the node configuration information includes the server's node information). This establishes a communication connection between the client and server, preparing for the aforementioned fault test.

[0141] On the other hand, this embodiment also provides a method for fault testing of distributed systems applied to the server side, such as... Figure 3 As shown, it includes:

[0142] S21: Receive fault injection commands, fault recovery commands, and status monitoring commands sent by the client.

[0143] S22: Perform fault injection according to the fault injection command and obtain the fault injection result.

[0144] S23: After fault injection, perform fault recovery according to the fault recovery command.

[0145] S24: Monitor the changes in the specified state after fault injection and fault recovery according to the status monitoring command, and obtain the status monitoring results.

[0146] S25: Return the fault injection results and status monitoring results to the client.

[0147] To facilitate the client: Based on the result determination configuration information of the fault test strategy belonging to the same group as the fault injection command, fault recovery command and status monitoring command, the fault test result is determined according to the corresponding fault injection result and status monitoring result.

[0148] It is readily apparent that the method applied to the server side provided in this embodiment has the same or corresponding parts as the method applied to the client side in the above embodiments. Some embodiments of the server side method can be obtained by combining the embodiments of the client side method described above. This embodiment will not repeat the details.

[0149] However, it should be noted that steps S22, S23, and S24 can be executed in parallel, such as... Figure 3 As shown, this ensures that the status monitoring can detect complete status changes.

[0150] In particular, the following embodiments mainly describe some implementation schemes that are only applied to the server side or not described in the above client-side method embodiments.

[0151] For example, this embodiment provides a possible implementation on the server side, and the method further includes:

[0152] S201: Create the first process and the second process;

[0153] The first process is used to: execute fault injection commands and return fault injection results to the client; and execute fault recovery commands after executing fault injection commands; the second process is used to: execute status monitoring commands and return status monitoring results to the client; the first process and the second process are two processes running in parallel.

[0154] In other words, in this embodiment, the server implements fault injection, fault recovery, and status monitoring through two pre-created parallel processes.

[0155] As can be seen from the above embodiments, fault recovery occurs after fault injection, but status monitoring is performed concurrently with fault injection and recovery. Status monitoring needs to monitor the entire process of fault injection and recovery to obtain information on changes in the state of interest. Therefore, on the server side, this embodiment creates two parallel processes to implement fault injection and recovery, as well as status monitoring, respectively. This satisfies the requirements of fault testing while avoiding missed monitoring or detection, thus ensuring the accuracy of fault testing.

[0156] Furthermore, this embodiment also provides a possible implementation scheme on the server side for returning the fault injection result and the status monitoring result to the client side. The above-mentioned step S25 specifically includes:

[0157] S251: Analyze the fault injection results and status monitoring results to obtain multiple result items containing timing features.

[0158] S252: Reorder the result items according to their time sequence and merge them into a single result data packet.

[0159] S253: Send the result data packet to the client.

[0160] The fault injection result is the server's response to the fault injection command. In practice, including the time of the response is a common approach and requires no additional configuration. Similarly, the purpose of the status monitoring command is to record changes in the status of interest, which can be achieved through a monitor. Recording the time of status change while recording the status change is also a standard implementation and requires no additional configuration.

[0161] In other words, this embodiment utilizes the timing characteristics attached to the aforementioned instruction responses to rearrange the result items in the fault injection results and status monitoring results based on these timing characteristics, merging them into a single result data packet, which is then sent to the client. On one hand, this establishes a direct physical connection between the fault injection results and status monitoring results of the same fault test case; on the other hand, it rearranges the result items according to timing characteristics, both of which benefit the client's subsequent fault result judgment and processing, reducing the client's load. Furthermore, considering the one-to-many relationship between the client and server, this embodiment's approach of transferring some of the workload from the client to the server improves overall efficiency and prevents the client from becoming a performance bottleneck affecting fault testing efficiency.

[0162] Furthermore, this embodiment also provides a possible implementation scheme for establishing a communication connection between the server and the client on the server side. Before step S21 above, the method further includes:

[0163] S202: Open socket connection and wait for the client to send a communication connection establishment request;

[0164] S203: After receiving a communication connection establishment request from the client, respond to the communication connection establishment request to establish a communication connection with the client.

[0165] In this embodiment, the client and server establish a communication connection via sockets. Before the fault test officially begins, the server opens the socket connection for the client to connect. Once the client connects, the server becomes the node under test in this fault test. It is readily apparent that this embodiment provides only one possible communication connection scheme between the server and client, applicable to the communication connection scheme provided in the client-side embodiment described above, to meet the fault testing requirements of this method.

[0166] In the above embodiments, a method for testing distributed system faults has been described in detail. The present invention also provides an embodiment of a distributed system fault testing device. It should be noted that the distributed system fault testing device provided by the present invention is also applicable to the aforementioned distributed system, which includes multiple nodes, wherein one node acts as a client and the other nodes act as servers.

[0167] Furthermore, this embodiment also describes the device from both sides: one is a device applied to the client side, and the other is a device applied to the server side.

[0168] Applied to the client side, such as Figure 4 As shown, this embodiment provides a distributed system fault testing device, including:

[0169] The strategy acquisition module 11 is used to acquire fault testing strategies; wherein, a set of fault testing strategies includes: the node's Internet Protocol address, fault injection command, fault recovery command, status monitoring command and result judgment configuration information;

[0170] The strategy distribution module 12 is used to send the fault injection command, fault recovery command, and status monitoring command in the same fault test strategy group to the corresponding server according to the Internet Protocol address, so that the server can: execute the fault injection command to perform fault injection and obtain the fault injection result; execute the fault recovery command to perform fault recovery; execute the status monitoring command to monitor the changes of the specified status after fault injection and fault recovery and obtain the status monitoring result; and return the fault injection result and status monitoring result to the client.

[0171] Result receiving module 13 is used to receive the fault injection results and status monitoring results returned by the server;

[0172] The fault determination module 14 is used to determine the fault test result based on the result determination configuration information, according to the corresponding fault injection result and status monitoring result.

[0173] On the other hand, it is applied to the server side, such as Figure 5 As shown, this embodiment provides a distributed system fault testing device, including:

[0174] Command receiving module 21 is used to receive fault injection commands, fault recovery commands and status monitoring commands sent by the client;

[0175] The fault injection module 22 is used to inject faults according to the fault injection command and obtain the fault injection result;

[0176] The fault recovery module 23 is used to perform fault recovery according to the fault recovery command after fault injection;

[0177] The status monitoring module 24 is used to monitor the changes of a specified status after fault injection and fault recovery according to the status monitoring command, and to obtain the status monitoring results.

[0178] The result return module 25 is used to return the fault injection result and the status monitoring result to the client, so that the client can determine the fault test result based on the result judgment configuration information that belongs to the same group of fault test strategies as the fault injection command, fault recovery command and status monitoring command, according to the corresponding fault injection result and status monitoring result.

[0179] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0180] Figure 1 A structural diagram of a distributed system provided for another embodiment of the present invention, such as... Figure 1 As shown, a distributed system includes: multiple nodes; one node acts as a client, and the other nodes act as servers.

[0181] The client is used to implement the steps of the distributed system fault testing method applied to the client side as described above;

[0182] The server is used to implement the steps of the distributed system fault testing method applied to the server side as described above.

[0183] Similarly, since the embodiments of the system part correspond to the embodiments of the method part, the embodiments of the system part can also be found in the description of the embodiments of the method part, and will not be repeated here.

[0184] In addition to the distributed system fault testing method, apparatus, and distributed system embodiment provided in the above embodiments, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instructions, which, when executed by a processor, can implement the steps of the distributed system fault testing method described in any of the above embodiments.

[0185] Since the embodiments of the computer program product portion correspond to the embodiments of the method portion, please refer to the description of the embodiments of the method portion for the embodiments of the computer program product portion, and will not be repeated here.

[0186] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. A computer program is stored on the non-volatile storage medium, and when the computer program is executed by a processor, it implements the steps described in the above method embodiments (which may be a method corresponding to the client side, a method corresponding to the server side, or a method corresponding to both the client side and the server side).

[0187] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] The foregoing has provided a detailed description of a distributed system, its fault testing method, apparatus, and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0189] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for testing faults in a distributed system, characterized in that, Applied to distributed systems; The distributed system includes multiple nodes, where one node acts as a client and the other nodes act as servers. This method is applied to the client and includes: Obtain fault testing strategies; among which, a set of fault testing strategies includes: the node's Internet Protocol address, fault injection command, fault recovery command, status monitoring command, and result judgment configuration information; Based on the Internet Protocol address, the fault injection command, fault recovery command, and status monitoring command from the fault testing strategy in the same group are sent to the corresponding server, so that the server can: create a first process and a second process; wherein the first process and the second process are two parallel processes; the first process executes the fault injection command to perform fault injection and obtains the fault injection result; the first process executes the fault recovery command to perform fault recovery; and the second process executes the status monitoring command to monitor the changes of the specified status after fault injection and fault recovery and obtain the status. The status monitoring results are returned to the client via the first process and the status monitoring results are returned via the second process. If multiple sets of fault testing strategies are obtained, the step of sending the fault injection command, fault recovery command, and status monitoring command within the same set of fault testing strategies to the corresponding server according to the Internet Protocol address includes: classifying each set of fault testing strategies according to the Internet Protocol address; and packaging and sending each set of fault injection commands, fault recovery commands, and status monitoring commands corresponding to a specific server to the corresponding server. Receive the fault injection result and the status monitoring result returned by the server; Based on the result determination configuration information, the fault test result is determined according to the corresponding fault injection result and the status monitoring result.

2. The distributed system fault testing method according to claim 1, characterized in that, Based on the configuration information determined by the results, the fault test results are determined according to the corresponding fault injection results and the status monitoring results, including: Based on the fault injection result, determine whether the fault injection on the server was successful; If the injection failure is successful, the configuration information and the status monitoring results are used to determine whether the change in the specified status is in line with expectations. If the conditions are met, the fault test result is considered a successful test. If the test does not meet the requirements, the test result is deemed a failure. If the injection fails, the fault injection command is resent to the corresponding server so that the server can perform fault injection again. This method also includes: Count the number of times the same fault injection command is sent repeatedly; If the number of repetitions exceeds the preset repetition threshold, an alarm will be triggered and a corresponding prompt message will be returned. The prompt information includes: the fault injection command and the corresponding Internet Protocol address.

3. The distributed system fault testing method according to claim 1 or 2, characterized in that, Before sending the fault injection command, the fault recovery command, and the status monitoring command from the same fault testing strategy to the corresponding server according to the Internet Protocol address, the method further includes: Get node configuration information; Based on the node configuration information, a communication connection establishment request is sent to each of the servers to establish a communication connection with each of the servers.

4. A method for testing faults in a distributed system, characterized in that, Applied to distributed systems; The distributed system includes multiple nodes, where one node acts as a client and the other nodes act as servers. This method is applied to the server and includes: The system receives fault injection commands, fault recovery commands, and status monitoring commands sent by the client. These commands are generated by the client after acquiring a fault testing strategy, and are sent to the corresponding server based on the Internet Protocol (IP) address. A fault testing strategy includes: the node's IP address, fault injection commands, fault recovery commands, status monitoring commands, and result determination configuration information. If multiple fault testing strategies are acquired, sending the fault injection commands, fault recovery commands, and status monitoring commands to the corresponding server based on the IP address includes: classifying each group of fault testing strategies according to the IP address; and packaging and sending each group of fault injection commands, fault recovery commands, and status monitoring commands corresponding to a specific server to the corresponding server. Create a first process and a second process; wherein the first process and the second process are two processes running in parallel; The fault injection is performed by executing the fault injection command through the first process, and the fault injection result is obtained. After fault injection, the fault recovery command is executed by the first process to perform fault recovery; The second process executes the status monitoring command to monitor the changes in the specified status after fault injection and fault recovery, and obtains the status monitoring results. The first process returns the fault injection result, and the second process returns the status monitoring result to the client, so that the client can: determine the fault test result based on the result determination configuration information that belongs to the same group of fault test strategies as the fault injection command, the fault recovery command, and the status monitoring command, according to the corresponding fault injection result and the status monitoring result.

5. The distributed system fault testing method according to claim 4, characterized in that, Returning the fault injection result and the status monitoring result to the client includes: The fault injection result and the status monitoring result are analyzed to obtain multiple result items containing time-series features; The results items are reordered according to their chronological order and merged into a single result data packet. The result data packet is sent to the client.

6. The distributed system fault testing method according to claim 4 or 5, characterized in that, Before receiving the fault injection command, fault recovery command, and status monitoring command sent by the client, the method further includes: Open a socket connection and wait for the client to send a communication connection establishment request; After receiving the communication connection establishment request sent by the client, respond to the communication connection establishment request to establish a communication connection with the client.

7. A distributed system fault testing device, characterized in that, Applied to distributed systems; The distributed system includes multiple nodes, where one node acts as a client and the other nodes act as servers. This device is used in the client and includes: The strategy acquisition module is used to acquire fault testing strategies; among them, a set of fault testing strategies includes: the node's Internet Protocol address, fault injection command, fault recovery command, status monitoring command, and result judgment configuration information; The strategy distribution module is used to send the fault injection command, the fault recovery command, and the status monitoring command from the same group of fault testing strategies to the corresponding server according to the Internet Protocol address, so that the server can: create a first process and a second process; wherein the first process and the second process are two parallel processes; execute the fault injection command through the first process to perform fault injection and obtain the fault injection result; execute the fault recovery command through the first process to perform fault recovery; and execute the status monitoring command through the second process to monitor the changes of the specified status after fault injection and fault recovery. The system obtains status monitoring results; returns the fault injection result through the first process and the status monitoring result through the second process to the client; wherein, if multiple sets of fault testing strategies are obtained, the step of sending the fault injection command, fault recovery command and status monitoring command in the same set of fault testing strategies to the corresponding server according to the Internet Protocol address includes: classifying each set of fault testing strategies according to the Internet Protocol address; and packaging and sending each set of fault injection commands, fault recovery commands and status monitoring commands corresponding to a certain server to the corresponding server. The result receiving module is used to receive the fault injection result and the status monitoring result returned by the server; The fault determination module is used to determine the fault test result based on the result determination configuration information, according to the corresponding fault injection result and the status monitoring result.

8. A distributed system fault testing device, characterized in that, Applied to distributed systems; The distributed system includes multiple nodes, where one node acts as a client and the other nodes act as servers. This device is used on the server side and includes: The command receiving module is used to receive fault injection commands, fault recovery commands, and status monitoring commands sent by the client. The fault injection commands, fault recovery commands, and status monitoring commands are generated by the client after acquiring a fault testing strategy, and are sent to the corresponding server based on the Internet Protocol address. A fault testing strategy includes: the node's Internet Protocol address, fault injection commands, fault recovery commands, status monitoring commands, and result determination configuration information. If multiple sets of fault testing strategies are acquired, sending the fault injection commands, fault recovery commands, and status monitoring commands to the corresponding server based on the Internet Protocol address includes: classifying each set of fault testing strategies according to the Internet Protocol address; and packaging and sending each set of fault injection commands, fault recovery commands, and status monitoring commands corresponding to a specific server to the corresponding server. The fault injection module is used to perform fault injection by executing the fault injection command through the first process and to obtain the fault injection result. The fault recovery module is used to perform fault recovery by executing the fault recovery command through the first process after fault injection. The status monitoring module is used to monitor the changes in a specified status after fault injection and fault recovery by executing the status monitoring command through the second process, and to obtain the status monitoring results; wherein, the first process and the second process are two pre-created processes running in parallel; The result return module is used to return the fault injection result through the first process and the status monitoring result through the second process to the client, so that the client can determine the fault test result based on the result judgment configuration information that belongs to the same group of fault test strategies as the fault injection command, the fault recovery command and the status monitoring command, according to the corresponding fault injection result and the status monitoring result.

9. A distributed system, comprising: Multiple nodes; characterized in that one node acts as a client and the other nodes act as servers; in: The client is used to implement the steps of the distributed system fault testing method as described in any one of claims 1 to 3; The server is used to implement the steps of the distributed system fault testing method as described in any one of claims 4 to 6.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the distributed system fault testing method as described in any one of claims 1 to 6.

Images