Testing method, system, device, electronic equipment and medium for fault-tolerant system

Abstract

The present disclosure provides a method, system, device, electronic device and medium for testing a fault-tolerant system, which relates to the field of computer technology, and in particular to the field of artificial intelligence, software testing and model training technology. The specific implementation scheme is: receiving a test sample, wherein the test sample includes: a target fault type and expected behavior data. Then, based on the preset correspondence between various fault types and preset faults, the target fault corresponding to the target fault type is determined. The target fault is then injected into the training device of the model to be tested. And actual behavior data is obtained from the fault-tolerant system, and the actual behavior data is: data generated by the response behavior executed by the fault-tolerant system to respond to the target fault. Afterwards, the actual behavior data and the expected behavior data are compared to obtain the test results of the fault-tolerant system. The manpower consumption of the testing process is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, software testing, and model training. Background Art

[0002] According to the scaling law, the larger the model, the more tokens the input data can include, and the higher the model's prediction accuracy. Model size can be measured by the number of parameters, the number of network layers, and / or the complexity of the network structure. Furthermore, the larger the model, the more frequent failures occur during training. To ensure the training efficiency of large models, fault-tolerant systems for model training have been developed. These systems can take timely countermeasures when failures occur during model training, ensuring the stability and continuity of the training process. Summary of the Invention

[0003] The present disclosure provides a method, system, device, electronic device, and medium for testing a fault-tolerant system.

[0004] A first aspect of an embodiment of the present disclosure provides a method for testing a fault-tolerant system, comprising:

[0005] receiving a test sample, wherein the test sample includes: a target fault type and expected behavior data;

[0006] Determining a target fault corresponding to the target fault type based on a preset correspondence between various fault types and preset faults;

[0007] Injecting the target fault into the training device of the model to be tested;

[0008] Acquire actual behavior data from the fault-tolerant system, wherein the actual behavior data is data generated by the response behavior performed by the fault-tolerant system in response to the target fault;

[0009] The actual behavior data and the expected behavior data are compared to obtain a test result of the fault-tolerant system.

[0010] A second aspect of the embodiments of the present disclosure provides a testing system for a fault-tolerant system, including:

[0011] A test framework is configured to receive a test sample, determine a target fault corresponding to the target fault type based on a preset correspondence between various fault types and preset faults, and inject the target fault into a training device of a model to be tested, wherein the test sample includes: a target fault type and expected behavior data;

[0012] A fault-tolerant system, configured to execute a response action after the target failure occurs;

[0013] The test framework is also used to obtain actual behavior data from the fault-tolerant system, compare the actual behavior data with the expected behavior data, and obtain test results for the fault-tolerant system. The actual behavior data is: data generated by the response behavior performed by the fault-tolerant system to respond to the target fault.

[0014] A third aspect of the embodiments of the present disclosure provides a testing device for a fault-tolerant system, comprising:

[0015] A receiving module, configured to receive a test sample, wherein the test sample includes: a target fault type and expected behavior data;

[0016] A determination module, configured to determine a target fault corresponding to the target fault type based on a preset correspondence between various fault types and preset faults;

[0017] An injection module, configured to inject the target fault into a training device of a model to be tested;

[0018] An acquisition module is used to acquire actual behavior data from the fault-tolerant system, wherein the actual behavior data is data generated by the response behavior performed by the fault-tolerant system in response to the target fault;

[0019] The comparison module is used to compare the actual behavior data with the expected behavior data to obtain a test result of the fault-tolerant system.

[0020] According to a fourth aspect of the present disclosure, an electronic device is provided, including:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods according to the first aspect.

[0024] According to a fifth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any one of the first aspects.

[0025] According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method according to any one of the first aspects when executed by a processor.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0028] Figure 1 This is a flow chart of a method for testing a fault-tolerant system provided by an embodiment of the present disclosure;

[0029] Figure 2 is an exemplary schematic diagram of a testing process for a fault-tolerant system provided by an embodiment of the present disclosure;

[0030] Figure 3 This is a schematic diagram of the structure of a test system for a fault-tolerant system provided by an embodiment of the present disclosure;

[0031] Figure 4 is an exemplary schematic diagram of another testing process for a fault-tolerant system provided by an embodiment of the present disclosure;

[0032] Figure 5 1 is a schematic structural diagram of a test device for a fault-tolerant system provided by an embodiment of the present disclosure;

[0033] Figure 6 It is a block diagram of an electronic device used to implement the testing method of the fault-tolerant system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] However, faults in fault-tolerant systems can lead to higher failure rates during model training. Faults in fault-tolerant systems are often imperceptible and difficult to self-recover, making testing fault-tolerant systems crucial. Testing fault-tolerant systems can identify potential issues in a timely manner, allowing for proactive countermeasures or timely elimination.

[0036] Currently, there's no universal way to test fault-tolerant systems. Therefore, testing a fault-tolerant system requires manually constructing simple failover scenarios and injecting faults for testing. Failover refers to the process of rapidly activating a redundant or backup device to take over the work of the terminated device when a running device unexpectedly terminates.

[0037] However, this manual testing approach requires manual identification of the faults to be injected, triggering the fault injection, and observing the results. The testing-to-development manpower ratio is often no less than 3:1, and the testing process consumes approximately one person-week (a person-week represents a week's worth of time or workload). This significant labor consumption and low efficiency hinder the overall project progress, including testing.

[0038] To address the aforementioned issues, embodiments of the present disclosure provide a method for testing a fault-tolerant system. This method is applicable to electronic devices, such as servers, desktop computers, or laptop computers, which possess communication and data processing capabilities. The software module within the electronic device that executes the method for testing a fault-tolerant system provided by embodiments of the present disclosure can be referred to as a test framework.

[0039] like Figure 1 As shown, the method for testing a fault-tolerant system provided by an embodiment of the present disclosure includes the following steps:

[0040] S101: Receive a test sample, wherein the test sample includes: a target fault type and expected behavior data.

[0041] The electronic device can display a configuration page that includes multiple parameters for the test case. For example, the configuration page includes a fatal events column and an expectation column. The fatal events column includes the failure type, and the expectation column includes the expected behavior data. This allows the tester to fill in or select the parameter value for each parameter on the configuration page. When the tester clicks submit, the electronic device receives the test case configured by the tester.

[0042] Alternatively, the electronic device may receive test samples sent by other devices. The manner in which the other devices obtain the test samples is the same as the manner in which the electronic device obtains the test samples, and is not described in detail here.

[0043] The target failure type is the failure type configured by the tester. Examples include: training process hang, abnormal exit, single-node disconnection, and remote storage failure. A training process hang indicates that the training process in the training service that trains the model under test has temporarily stopped running. An abnormal exit indicates that the training service exited due to an exception such as a crash or forced shutdown. A single-node disconnection indicates that the single training device where the training service resides cannot communicate with other devices. A remote storage failure indicates that the training device cannot store training data to the remote storage device.

[0044] Expected behavior data refers to the data generated by the behavior that the fault-tolerant system is expected to perform.

[0045] S102: Determine a target fault corresponding to a target fault type based on a preset correspondence between various fault types and preset faults.

[0046] In the embodiment of the present disclosure, the electronic device may pre-store preset correspondences between various fault types and preset faults.

[0047] For example, the preset fault corresponding to the training process Hang is: injecting an infinite loop into the Nth operator running serially on the graphics processing unit (GPU) of the training device, making it impossible to enter the next operator, and thus preventing the training process from continuing.

[0048] For another example, the preset fault corresponding to the abnormal exit is: throwing abnormal data or abnormal operation to the driver of the training device, causing the training service to exit due to the driver abnormality.

[0049] For example, the preset fault corresponding to the loss of connection of a single node is: injecting a fault into the network communication module of the central processing unit (CPU) of a single training device, making it impossible for the training device to communicate with other devices through the network.

[0050] For another example, the preset fault corresponding to the remote storage failure is: intercepting the message sent by the training service to the remote storage device, so that the training service cannot store data in the remote storage device.

[0051] The above correspondence between fault types and preset faults is only an example, and the fault types supported by the embodiments of the present disclosure and the preset faults corresponding to the fault types are not limited thereto.

[0052] S103: Inject target faults into the training device of the model to be tested.

[0053] The model to be tested may be a fixed model used for testing, or the model to be tested may be a model configured by the tester for this test.

[0054] S104. Acquire actual behavior data from the fault-tolerant system.

[0055] The actual behavior data refers to the data generated by the response behavior executed by the fault-tolerant system to respond to the target fault.

[0056] S105: Compare the actual behavior data with the expected behavior data to obtain a test result of the fault-tolerant system.

[0057] Since the actual behavior data is the data generated by the response behavior actually executed by the fault-tolerant system, and the expected behavior data is the data generated by the behavior expected to be executed by the fault-tolerant system, by comparing the actual behavior data with the expected behavior data, it is possible to obtain whether the response behavior actually executed by the fault-tolerant system meets expectations, thereby testing whether the fault-tolerant system has taken correct actions to respond to the target fault.

[0058] In the disclosed embodiments, a testing framework can receive test cases, automatically inject faults into training devices, and determine test results for the behavior executed by the fault-tolerant system in response to the faults. Specifically, testers only need to configure parameters in the test case, such as the fault type and expected behavior data, and the testing framework automatically determines the faults to be injected, performs the fault injection, and observes the test results. This eliminates the need for manual intervention during the testing process, significantly reducing labor consumption and improving testing efficiency.

[0059] The following describes in detail the method for testing a fault-tolerant system provided by an embodiment of the present disclosure:

[0060] In some embodiments of the present disclosure, after receiving the test sample in S101, the test framework may further send a test instruction to the fault-tolerant system, so that the fault-tolerant system responds to the test instruction and notifies the training device to train the model to be tested.

[0061] The testing framework can generate training jobs based on test cases. A training job can include a job identifier (ID), information about the model to be tested, and training device information. The model to be tested information can include the model ID, code representing the model to be tested, or the storage location of the model to be tested. Training device information can include the device ID or device address.

[0062] The test instructions sent by the test framework to the fault-tolerant system include training jobs.

[0063] After receiving the training job, the fault-tolerant system may send the model to be tested represented by the model to be tested information to the training device represented by the training device information, so that the training device trains the model to be tested.

[0064] The training device is configured with a training container, and the training device can create a training service within the training container. The training service trains the model to be tested through a training process. The training service can also send the current training status to the fault-tolerant system in real time during the training of the model to be tested. For example, the training service can periodically send the current training status to the fault-tolerant system, or send the current training status to the fault-tolerant system whenever the training status changes.

[0065] The training status indicates the progress of the training process. For example, the training status indicates the number of iterations of the model to be tested, and the number of iterations is the number of optimization times of the model parameters.

[0066] Through the above method, the disclosed embodiment can control the training device to start model training only when a test example is received, allowing the test framework to perform fault injection in any state within the complete training state. This avoids missing some training states when testing begins, resulting in the inability to perform fault injection in these training states. This improves the controllability of the test process and the coverage of testable training states.

[0067] In some embodiments of the present disclosure, there is at least one target fault type, and the test case also includes a trigger state corresponding to the target fault type. That is, the test case may include one or more target fault types, and each fault type corresponds to a trigger state. The trigger state represents the training state of the model under test during fault injection.

[0068] On this basis, after receiving the test sample in S101 above, the test framework can also monitor the training status of the model to be tested through the fault-tolerant system during the training process of the model to be tested.

[0069] Since the training device can report the training status to the fault-tolerant system in real time during the training process of the model to be tested, the test framework can obtain the current training status of the model to be tested by monitoring the fault-tolerant system.

[0070] Correspondingly, the method in which the above-mentioned S103 test framework injects the target fault into the training device of the model to be tested can be implemented as follows: when the training state is a trigger state, a target fault of the target fault type corresponding to the trigger state is injected into the training device.

[0071] For example, a test case includes a training process Hanging and an abnormal exit. The trigger state for the training process Hanging is step 10, and the trigger state for the abnormal exit is step 12. Step 10 indicates that the training state of the model under test is the 10th iteration, and step 12 indicates that the training state of the model under test is the 12th iteration. In this case, when the testing framework detects the training state is step 10, it injects the target fault corresponding to the training process Hanging. When the testing framework detects the training state is step 12, it injects the target fault corresponding to the abnormal exit.

[0072] The testing framework can send simulated fault instructions to the training device. The training device can respond to the simulated fault instructions by performing fault injection operations on the training service or training device to cause the target fault on the training device. The software module in the training device that responds to the simulated fault instructions is called an execution service, and the execution service and the training service run in the same container on the training device.

[0073] The disclosed embodiments allow users to set trigger states, enabling fault injection under user-specified training conditions, enabling refined control of fault injection. Furthermore, the disclosed embodiments also allow users to configure multiple fault types, enabling testing under simulated mixed fault scenarios. This meets the testing requirements of complex fault scenarios, making the resulting test results more realistic and valuable for reference.

[0074] In addition to the fact that the fault-tolerant system may take inappropriate actions when a training device fails, which may affect the normal operation of the training process, failures in the fault-tolerant system itself may also affect the operation of the training process. Therefore, it is also possible to perform fault testing on the fault-tolerant system itself.

[0075] Therefore, before injecting the target fault into the training device of the model to be tested in the above S103, the test framework may also determine whether the faulty device corresponding to the target fault type is a training device.

[0076] In one implementation, the test framework can be preconfigured with the fault classification for each fault type, such as device-related and system-related faults. When the target fault type is a device-related fault, the faulty device corresponding to the target fault type is determined to be a training device; when the target fault type is a system-related fault, the faulty device corresponding to the target fault type is determined to be a fault-tolerant system.

[0077] For example, a training process hang is a device-related issue, while a scheduling failure is a system-related issue. Scheduling failure means that the fault-tolerant system cannot use a backup device to replace a failed training device in the training cluster.

[0078] In another implementation, the test case may include faulty device information corresponding to each target fault type. This faulty device information may represent a training device or a fault-tolerant system. Therefore, based on the faulty device information corresponding to each target fault type in the test case, the testing framework can determine whether the faulty device corresponding to that target fault type is a training device.

[0079] If so, the test framework may execute the above step S103 to inject the target fault into the training device of the model to be tested.

[0080] If not, the test framework can inject the target fault into the fault-tolerant system. The method of injecting the target fault into the fault-tolerant system is the same as the method of injecting the target fault into the training device. Please refer to the relevant description and will not be repeated here.

[0081] Through the above method, the embodiment of the present disclosure not only supports injecting faults into the training device, but also supports injecting faults into the fault-tolerant system, so that the test coverage is wider and the test results are more comprehensive.

[0082] Exemplary test examples include:

[0083] "case_param = { #Test sample parameters

[0084] "caseName": "test-case", #Test case name

[0085] "fatalEvents": [ # Failure events

[0086] {

[0087] "trainer": 1, ##Device identifier of the target training device

[0088] "step": 12, ##Trigger status

[0089] "reason": "Hang" ##Target failure type

[0090] },

[0091] ],”

[0092] Since the larger the model size, the larger the training cluster size required for training, that is, the training device for the model to be tested can be one or more. The embodiment of the present disclosure supports the user to configure the training device to be injected with faults, that is, the test sample can also include the device information of the target training device. The target training device is at least one training device among the training devices of the model to be tested. The device information includes: the number of devices, device ID and / or device address, etc.

[0093] On this basis, the method of injecting target faults into the training device of the model to be tested by the above S103 test framework can be implemented as follows: based on the device information, a simulated fault instruction is sent to the target training device.

[0094] Among them, the simulated fault instruction is used to instruct the target training device to perform a fault injection operation to cause a target fault to occur in the target training device.

[0095] When the device information includes the number of devices, the test framework can select the number of training devices from the training devices as target training devices. For example, the selection method can be random selection or selection in descending order of pre-set priority.

[0096] Alternatively, when the device information includes a device ID, the test framework may use the training device corresponding to the device ID as the target training device. Alternatively, when the device information includes a device address, the test framework may use the training device corresponding to the device address as the target training device.

[0097] The embodiment of the present disclosure supports users to select the training devices into which faults need to be injected. Compared with directly injecting faults into all training devices or injecting faults into randomly selected devices, the embodiment of the present disclosure has a finer control granularity for fault injection, making the test more flexible and better meeting the needs of actual application scenarios.

[0098] In some embodiments of the present disclosure, the S105 test framework compares actual behavior data with expected behavior data to obtain a test result for the fault-tolerant system, which can be implemented as follows: determining the test result based on the comparison result obtained in at least one of the following ways.

[0099] Method 1: Compare the telephone alarm information included in the actual behavior data with the telephone alarm information included in the expected behavior data to see whether they match, thereby obtaining a first comparison result.

[0100] Optionally, when the actual behavior data includes phone alert information that matches the expected behavior data, the first comparison result can be determined to be a match. Alternatively, when the two types of phone alert information are identical, the first comparison result can be determined to be a match. Alternatively, determining whether the two types of phone alert information match can be performed using other methods, which are not specifically limited in the present embodiment.

[0101] The telephone alarm information refers to the information generated by the fault-tolerant system when it detects a fault and sends an alarm via telephone. The telephone alarm information includes: the alarm number dialed and / or the department to which the alarm number belongs.

[0102] Method 2: comparing the message alarm information included in the actual behavior data with the message alarm information included in the expected behavior data to determine whether they match, thereby obtaining a second comparison result.

[0103] Optionally, when the message alert information included in the actual behavior data matches the message alert information included in the expected behavior data, the second comparison result can be determined to be a match. Alternatively, when the two message alert information are identical, the second comparison result can be determined to be a match. Alternatively, determining whether the two message alert information match can be performed using other methods, which are not specifically limited in the present embodiment.

[0104] Message alert information refers to information generated by the fault-tolerant system when it detects a fault. Messages include, for example, text messages, push messages, and / or instant messaging messages. Message alert information includes the text content of the message, the recipient account, and / or the department to which the recipient account belongs.

[0105] Method 3: Compare the fault recovery method included in the actual behavior data with the fault recovery method included in the expected behavior data to see whether they match, thereby obtaining a third comparison result.

[0106] Optionally, when the fault recovery method included in the actual behavior data matches the fault recovery method included in the expected behavior data, the third comparison result can be determined to be a match. Alternatively, when the two fault recovery methods are the same, the third comparison result can be determined to be a match. Alternatively, other methods can be used to determine whether the two fault recovery methods match, which is not specifically limited in the present embodiment.

[0107] Fault recovery methods refer to the measures taken by a fault-tolerant system to eliminate a fault when it detects one. Fault recovery methods include retrying, restarting, clearing cache, and updating the operating system.

[0108] Mode 4: Compare the post-recovery training data included in the actual behavior data with the post-recovery training data included in the expected behavior data to determine whether they match, thereby obtaining a fourth comparison result.

[0109] The post-recovery training data represents the training data of the model under test obtained after the fault-tolerant system detects a fault and executes a fault recovery method. The post-recovery training data includes: the first training state of the model under test after executing the fault recovery method, the loss value of that training state, and / or the maximum loss difference.

[0110] Optionally, it is possible to determine whether the training state included in the actual behavior data is the same as the training state included in the expected behavior data, and to determine whether the difference between the loss value of the training state after executing the fault recovery method and the loss value of the trigger state is less than the maximum loss difference included in the expected behavior data. If the judgment results are all yes, the fourth comparison result is determined to be a match. Otherwise, the fourth comparison result is determined to be a mismatch. Alternatively, it is possible to determine whether the training state included in the actual behavior data is the same as the training state included in the expected behavior data, and to determine whether the loss value of the training state after executing the fault recovery method is the same as the loss value included in the expected behavior data. If the judgment results are all yes, the fourth comparison result is determined to be a match. Otherwise, the fourth comparison result is determined to be a mismatch. Alternatively, it is possible to determine whether the two types of recovered training data match in other ways, and the embodiments of the present disclosure do not specifically limit this.

[0111] For example, when the actual behavior data includes a training state of step11, the expected behavior data includes a training state of step11, and the actual behavior data includes a loss value of 0.2, the loss value of the trigger state is 0.2, and the maximum loss difference included in the expected behavior data is 0.05, then it is determined that the two types of recovered training data match.

[0112] When determining the test result, in the obtained comparison results, if all comparison results are matched, the test result is determined to have passed the test; otherwise, if any comparison result is mismatched, the test result is determined to have failed the test.

[0113] Alternatively, if more than a preset ratio of the comparison results are mismatches, the test result is determined to have failed; otherwise, the test result is determined to have passed. For example, the preset ratio is 1 / 2. Alternatively, the test result may be determined in other ways, which are not specifically limited in the present embodiment.

[0114] After obtaining the test results, the test framework can also generate a test report. For example, the test report includes test samples and test results.

[0115] Through the above method, the embodiment of the present disclosure can verify from multiple aspects whether the response behavior actually executed by the fault-tolerant system matches the expected behavior, so that the test results obtained thereby are more comprehensive and have greater reference value.

[0116] Exemplary test examples include:

[0117] ""expect": { # expected behavior data

[0118] "message": { ##Alarm information

[0119] "call": "no call / call a / call b / all calls", ## call alarm information

[0120] "duty": ## Message warning information [

[0122] ["[Status]: Hang", "No update for 2 minutes", "About to restart"], ### Content of the first warning message

[0123] ["[Status]: Task started successfully", "Number of start steps: 11", "Continue: True"]### Content of the second alarm message ]

[0125] },

[0126] "recovery": { ## Fault recovery method

[0127] "method": ["rerun"] ### Process level rescheduling

[0128] },

[0129] "metrics": { ## Training data after recovery

[0130] "firstStep": 11, ###Training status after executing the fault recovery method

[0131] "firstLoss": "equal" ###The loss value of the training state after executing the fault recovery method is equal to the loss value of the trigger state, that is, the maximum loss difference is 0

[0132] },

[0133] },”

[0134] After obtaining the test result in S105 , the test framework may further analyze the reasons why the fault-tolerant system failed the test in the following two ways.

[0135] Method 1: If the test result indicates that the test has failed, the comparison result is matched with each preset rule to obtain a target rule that matches the test result, and the failure reason corresponding to the target rule is obtained.

[0136] The test framework is pre-configured with multiple preset rules, each of which corresponds to a reason for failure.

[0137] For example, the preset rule 1 is: the telephone alarm information does not match and the telephone alarm information included in the actual behavior data is empty, and the failure reason corresponding to the preset rule 1 is: dialing failure.

[0138] For another example, the preset rule 2 is: the message alarm information does not match and the message alarm information included in the actual behavior data is empty, and the failure reason corresponding to the preset rule 2 is: network failure.

[0139] For another example, preset rule 3 is: training data does not match after recovery; the failure reason corresponding to preset rule 3 is: training data storage failure.

[0140] The correspondence between the above preset rules and the reasons for failure is only an example, and the preset rules supported by the embodiments of the present disclosure and the reasons for failure corresponding to the preset rules are not limited to this.

[0141] The disclosed embodiments can pre-set preset rules corresponding to different failure reasons. When a test result indicates a failure, the comparison result can be matched with each preset rule to obtain the preset rule that matches the comparison result, thereby obtaining the failure reason corresponding to the preset rule. This improves the efficiency of obtaining failure reasons. Furthermore, the disclosed embodiments support analysis of failure reasons, which can help developers or testers quickly locate problems and take timely countermeasures.

[0142] Method 2: If the test result indicates that the test has failed, the preset analysis model is used to determine the reason for failure based on the comparison results.

[0143] The analysis model is a model with text processing capabilities, such as a transformer, Bidirectional Encoder Representations from Transformers (BERT), or a Large Language Model (LLM).

[0144] The comparison results, or the comparison results and actual behavior data, can be used as input data. The input data is then fed into the analysis model to obtain the failure reasons output by the analysis model. The input data may also include other information, such as brief information about various failure reasons, which is not specifically limited in the present embodiment.

[0145] The disclosed embodiment can use a preset analysis model to analyze the reasons for failure of this test. Thanks to the powerful learning and generalization capabilities of the analysis model, the reasons for failure output by the analysis model are not limited to the various reasons pre-set by the staff. Therefore, the failure reasons obtained in this way are more diverse and more accurate, and have greater reference value. Moreover, the disclosed embodiment supports the analysis of failure reasons, which can help developers or testers quickly locate problems and take countermeasures more promptly.

[0146] In some embodiments of the present disclosure, the above test sample may further include a test end condition.

[0147] For example, the test result conditions include: maximum training duration and / or maximum training status.

[0148] On this basis, after obtaining the test result in the above S105 , when the test end condition is met, if the test result indicates that the test has passed, the test framework may delete the snapshot data (checkpoint) of the model to be tested in the preset storage location.

[0149] The preset storage location may be a designated storage location in the cloud, or a designated storage location on a designated storage device, etc. The training device may store a checkpoint of the model to be tested in the preset storage location during the training of the model to be tested.

[0150] The snapshot data includes: training data such as the parameters of each network layer of the model to be tested and the optimizer status. The network layer parameters include the weights and biases of the network layer.

[0151] If the test result indicates a pass, the snapshot data in the preset storage location can be deleted to prevent the snapshot data from occupying storage space and improve storage resource utilization. Conversely, if the test result indicates a fail, the snapshot data is not deleted, allowing staff to refer to the snapshot data to analyze the problems in the fault-tolerant system and take timely countermeasures.

[0152] In the disclosed embodiment, since the test case may also include a test end condition, the test framework may send a training end instruction to the training device when the test end condition is met, thereby controlling the training device to stop training the model to be tested and obtain actual behavior data from the fault-tolerant system. The test framework may send a behavior data acquisition request to the fault-tolerant system, wherein the behavior data acquisition request carries the identifier of the current training job, so that the fault-tolerant system sends the actual behavior data corresponding to the identifier of the current training job to the test framework.

[0153] Exemplary test examples include:

[0154] “"clean": #Clear

[0155] {

[0156] "forceStop": { ## Test end condition

[0157] "maxStep": 21, ### Maximum training state

[0158] "maxDuration": 10, ### Maximum training duration

[0159] },

[0160] "action": [ ## Action

[0161] "killLongjob", ### End the training job

[0162] "removeCheckpointOnSuccess" ###Delete checkpoint when the test passes ]

[0164] }

[0165] }”

[0166] See also Figure 2 The following describes the overall process of the fault-tolerant system testing method provided by the embodiment of the present disclosure:

[0167] The testing framework receives test cases, generates training jobs based on them, and submits them to the fault-tolerant system. Furthermore, the testing framework parses the test cases to determine the target fault type, trigger status, and device information of the target training device, as well as the expected behavior data. The testing framework then monitors the training status of the model under test through the fault-tolerant system and determines whether the training status is triggered. If so, the testing framework injects the target fault into the training device of the model under test. Otherwise, the testing framework returns to determine whether the training status is triggered.

[0168] After receiving the training job, the fault-tolerant system notifies the training device to train the model to be tested. During the training process of the model to be tested, the training device sends the training status to the fault-tolerant system.

[0169] The testing framework obtains actual behavior data from the fault-tolerant system and compares it with the expected behavior data to generate a test result for the fault-tolerant system. If the test result indicates a failure, the comparison result is matched against the preset rules to obtain the target rule that matches the test result. The failure reason corresponding to the target rule is obtained and a test report is generated. When the test end conditions are met, if the test result indicates a pass, the snapshot data of the model under test is deleted from the preset storage location.

[0170] In the manual testing of fault-tolerant systems, the faults injected into the fault-tolerant system are manually determined to simulate the fault scenario. However, due to the different experience levels of different testers and the lack of unified testing standards, the test results obtained under the same fault scenario may vary greatly, making the reference value of the test results low.

[0171] In the embodiment of the present disclosure, it is only necessary to manually configure information such as the fault type, trigger status, and expected behavior data in the test sample to automatically perform the test, which reduces the experience requirements for the tester, achieves zero threshold, and improves the configuration efficiency of the test sample and the testing efficiency of the fault-tolerant system.

[0172] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of test samples involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0173] It should be noted that the to-be-tested model and the analysis model in this embodiment are not models for a specific user and cannot reflect the personal information of a specific user.

[0174] It should be noted that the test samples in this embodiment may come from a public dataset.

[0175] Based on the same inventive concept, corresponding to the above method embodiment, the embodiment of the present disclosure also provides a test system for a fault-tolerant system, such as Figure 3 As shown, the system includes: a test framework 301, a training device 302 and a fault-tolerant system 303;

[0176] The test framework 301 is configured to receive test samples, determine a target fault corresponding to a target fault type based on a preset correspondence between various fault types and preset faults, and inject the target fault into the training device 302 of the model to be tested. The test sample includes: the target fault type and expected behavior data;

[0177] The fault-tolerant system 303 is used to perform response actions after a target failure occurs;

[0178] The test framework 301 is also used to obtain actual behavior data from the fault-tolerant system 303, compare the actual behavior data with the expected behavior data, and obtain the test results of the fault-tolerant system 303. The actual behavior data is: the data generated by the response behavior performed by the fault-tolerant system 303 to respond to the target fault.

[0179] In the disclosed embodiments, a testing framework can receive test cases, automatically inject faults into training devices, and determine test results for the behavior executed by the fault-tolerant system in response to the faults. Specifically, testers only need to configure parameters in the test case, such as the fault type and expected behavior. The testing framework automatically determines the faults to be injected, performs the fault injection, and observes the test results. This eliminates the need for manual intervention during the testing process, significantly reducing labor consumption and improving testing efficiency.

[0180] In some embodiments of the present disclosure, the test framework is further configured to send a test instruction to the fault-tolerant system after receiving the test sample;

[0181] The fault-tolerant system is further configured to send a training notification to the training device in response to the test instruction;

[0182] The training device is configured to create a training service in a training container in response to a training notification, and train the model to be tested through the training service.

[0183] Through the above method, the disclosed embodiment can control the training device to start model training only when a test example is received, allowing the test framework to perform fault injection in any state within the complete training state. This avoids missing some training states when testing begins, resulting in the inability to perform fault injection in these training states. This improves the controllability of the test process and the coverage of testable training states.

[0184] In some embodiments of the present disclosure, there is at least one target fault type, and the test sample further includes a trigger state corresponding to the target fault type;

[0185] The training device is further used to send the training status of the model to be tested to the fault-tolerant system during the training of the model to be tested;

[0186] The testing framework is also used to monitor the training status of the model to be tested through the fault-tolerant system during the training process of the model to be tested after receiving the test examples;

[0187] The test framework is specifically used to inject a target fault of a target fault type corresponding to the trigger state into the training device when the training state is a trigger state.

[0188] The disclosed embodiments allow users to set trigger states, enabling fault injection under user-specified training conditions, enabling refined control of fault injection. Furthermore, the disclosed embodiments also allow users to configure multiple fault types, enabling testing under simulated mixed fault scenarios. This meets the testing requirements of complex fault scenarios, making the resulting test results more realistic and valuable for reference.

[0189] In some embodiments of the present disclosure, the test framework is further configured to:

[0190] Before injecting the target fault into the training device of the model to be tested, determine whether the faulty device corresponding to the target fault type is a training device;

[0191] If yes, then the step of injecting the target fault into the training device of the model to be tested is performed;

[0192] If not, the target fault is injected into the fault-tolerant system.

[0193] Through the above method, the embodiment of the present disclosure not only supports injecting faults into the training device, but also supports injecting faults into the fault-tolerant system, so that the test coverage is wider and the test results are more comprehensive.

[0194] In some embodiments of the present disclosure, the test sample also includes device information of the target training device;

[0195] The test framework is also used to send simulated fault instructions to the target training device based on the device information;

[0196] The training device is also used to execute services through faults in the training container, and in response to simulated fault instructions, perform fault injection operations on the training service or the training device where it is located, so as to cause a target fault to occur on the training device where it is located.

[0197] The embodiment of the present disclosure supports users to select the training devices into which faults need to be injected. Compared with directly injecting faults into all training devices or injecting faults into randomly selected devices, the embodiment of the present disclosure has a finer control granularity for fault injection, making the test more flexible and better meeting the needs of actual application scenarios.

[0198] In some embodiments of the present disclosure, the test framework is specifically used to:

[0199] Determine the test result based on the comparison results obtained by at least one of the following methods:

[0200] Comparing the telephone alert information included in the actual behavior data with the telephone alert information included in the expected behavior data to determine whether they match;

[0201] Comparing the message alarm information included in the actual behavior data with the message alarm information included in the expected behavior data to determine whether they match;

[0202] Comparing the fault recovery method included in the actual behavior data with the fault recovery method included in the expected behavior data to see whether they match;

[0203] The post-recovery training data included in the actual behavior data is compared with the post-recovery training data included in the expected behavior data to determine whether they match.

[0204] Through the above method, the embodiment of the present disclosure can verify from multiple aspects whether the response behavior actually executed by the fault-tolerant system matches the expected behavior, so that the test results obtained thereby are more comprehensive and have greater reference value.

[0205] In some embodiments of the present disclosure, the test framework is further configured to:

[0206] After comparing the actual behavior data with the expected behavior data to obtain the test results of the fault-tolerant system, if the test results indicate that the test has failed, the comparison results are matched with each preset rule to obtain the target rules that match the test results;

[0207] Get the failure reason corresponding to the target rule.

[0208] The disclosed embodiments can pre-set preset rules corresponding to different failure reasons. When a test result indicates a failure, the comparison result can be matched with each preset rule to obtain the preset rule that matches the comparison result, thereby obtaining the failure reason corresponding to the preset rule. This improves the efficiency of obtaining failure reasons. Furthermore, the disclosed embodiments support analysis of failure reasons, which can help developers or testers quickly locate problems and take timely countermeasures.

[0209] In some embodiments of the present disclosure, the test framework is further configured to:

[0210] After comparing the actual behavior data with the expected behavior data to obtain the test results of the fault-tolerant system, if the test results indicate that the test failed, the preset analysis model is used to determine the reason for the failure based on the comparison results.

[0211] The disclosed embodiment can use a preset analysis model to analyze the reasons for failure of this test. Thanks to the powerful learning and generalization capabilities of the analysis model, the reasons for failure output by the analysis model are not limited to the various reasons pre-set by the staff. Therefore, the failure reasons obtained in this way are more diverse and more accurate, and have greater reference value. Moreover, the disclosed embodiment supports the analysis of failure reasons, which can help developers or testers quickly locate problems and take countermeasures more promptly.

[0212] In some embodiments of the present disclosure, the test case further includes a test end condition and a test framework, and is further used to:

[0213] After comparing the actual behavior data with the expected behavior data to obtain the test results of the fault-tolerant system, when the test end condition is met, if the test result indicates that the test has passed, the snapshot data of the model to be tested in the preset storage location is deleted.

[0214] If the test result indicates a pass, the snapshot data in the preset storage location can be deleted to prevent the snapshot data from occupying storage space and improve storage resource utilization. Conversely, if the test result indicates a fail, the snapshot data is not deleted, allowing staff to analyze the fault-tolerant system based on the snapshot data and take timely countermeasures.

[0215] See also Figure 4 The following describes the interaction between modules during the testing of the fault-tolerant system provided by the embodiment of the present disclosure:

[0216] The software module in the test framework used to receive test samples and perform fault injection is called the Mock Failure Driver; the software module in the test framework used to analyze test results is called the Results Analyzer.

[0217] The software module in the training device that responds to simulated fault instructions is called a MockFailure Executor.

[0218] The Mock Failure Driver receives test samples, generates training jobs based on the test samples, and submits the training jobs to the fault-tolerant system.

[0219] After receiving the training job, the fault-tolerant system sends a training notification to the training device.

[0220] The training device creates a training service in the training container in response to the training notification, and trains the model to be tested through the training process of the training service; and during the process of training the model to be tested, sends the training status of the model to be tested to the fault-tolerant system.

[0221] The Mock Failure Driver monitors the training status of the model to be tested through the fault-tolerant system. When the training status is triggered, it sends a simulated fault instruction to the Mock Failure Executor.

[0222] The Mock Failure Executor responds to the simulated fault instruction and performs fault injection operations on the training service to cause the target fault to occur on the training device where it is located.

[0223] Results Analyzer obtains actual behavior data from the fault-tolerant system, compares the actual behavior data with the expected behavior data, and obtains the test results of the fault-tolerant system.

[0224] Based on the same inventive concept, corresponding to the above method embodiment, the embodiment of the present disclosure also provides a testing device for a fault-tolerant system, such as Figure 5 As shown, the device includes: a receiving module 501, a determining module 502, an injecting module 503, an acquiring module 504 and a comparing module 505;

[0225] The receiving module 501 is configured to receive a test sample, wherein the test sample includes: a target fault type and expected behavior data;

[0226] A determination module 502 is configured to determine a target fault corresponding to a target fault type based on a preset correspondence between various fault types and preset faults;

[0227] An injection module 503 is used to inject a target fault into the training device of the model to be tested;

[0228] An acquisition module 504 is configured to acquire actual behavior data from the fault-tolerant system. The actual behavior data is data generated by the response behavior executed by the fault-tolerant system to respond to the target fault.

[0229] The comparison module 505 is used to compare the actual behavior data with the expected behavior data to obtain the test results of the fault-tolerant system.

[0230] In some embodiments of the present disclosure, the apparatus further comprises:

[0231] The sending module is used to send a test instruction to the fault-tolerant system after receiving the test sample, so that the fault-tolerant system responds to the test instruction and notifies the training device to train the model to be tested.

[0232] In some embodiments of the present disclosure, there is at least one target fault type, and the test sample further includes a trigger state corresponding to the target fault type; the apparatus further includes:

[0233] A monitoring module is used to monitor the training status of the model to be tested through the fault-tolerant system during the training process of the model to be tested after receiving the test sample;

[0234] The injection module 503 is specifically used to:

[0235] When the training state is the trigger state, a target fault of the target fault type corresponding to the trigger state is injected into the training device.

[0236] In some embodiments of the present disclosure, the apparatus further comprises:

[0237] A judgment module is used to judge whether the faulty device corresponding to the target fault type is a training device before injecting the target fault into the training device of the model to be tested;

[0238] A calling module, configured to call the injection module 503 to execute the step of injecting the target fault into the training device of the model to be tested if the judgment result of the judgment module is yes;

[0239] The injection module 503 is further configured to inject a target fault into the fault-tolerant system if the judgment result of the judgment module is negative.

[0240] In some embodiments of the present disclosure, the test sample also includes device information of the target training device, and the injection module 503 is specifically used to:

[0241] Based on the device information, a simulated fault instruction is sent to the target training device, where the simulated fault instruction is used to instruct the target training device to perform a fault injection operation to cause a target fault to occur in the target training device.

[0242] In some embodiments of the present disclosure, the comparison module 505 is specifically configured to:

[0243] Determine the test result based on the comparison results obtained by at least one of the following methods:

[0244] Comparing the telephone alert information included in the actual behavior data with the telephone alert information included in the expected behavior data to determine whether they match;

[0245] Comparing the message alarm information included in the actual behavior data with the message alarm information included in the expected behavior data to determine whether they match;

[0246] Comparing the fault recovery method included in the actual behavior data with the fault recovery method included in the expected behavior data to see whether they match;

[0247] The post-recovery training data included in the actual behavior data is compared with the post-recovery training data included in the expected behavior data to determine whether they match.

[0248] In some embodiments of the present disclosure, the apparatus further comprises:

[0249] A matching module is used to compare the actual behavior data with the expected behavior data to obtain the test results of the fault-tolerant system. If the test result indicates that the system fails the test, the matching module matches the comparison result with each preset rule to obtain the target rule that matches the test result.

[0250] The acquisition module 504 is further configured to acquire the failure reason corresponding to the target rule.

[0251] In some embodiments of the present disclosure, the apparatus further comprises:

[0252] The analysis module is used to compare the actual behavior data with the expected behavior data to obtain the test results of the fault-tolerant system. If the test results indicate that the test has failed, the preset analysis model is used to determine the reason for failure based on the comparison results.

[0253] In some embodiments of the present disclosure, the test sample further includes a test end condition, and the apparatus further includes:

[0254] The deletion module is used to delete the snapshot data of the model to be tested in the preset storage location after comparing the actual behavior data with the expected behavior data to obtain the test results of the fault-tolerant system. When the test end conditions are met, if the test results indicate that the test has passed.

[0255] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0256] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0257] like Figure 6 As shown, electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of electronic device 600. Computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0258] Multiple components in the electronic device 600 are connected to the I / O interface 605, including an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0259] The computing unit 601 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the fault-tolerant system testing method. For example, in some embodiments, the fault-tolerant system testing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the fault-tolerant system testing method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the fault-tolerant system testing method via any other suitable means (e.g., via firmware).

[0260] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0261] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0262] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0263] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0264] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0265] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0266] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0267] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for testing a fault-tolerant system, comprising: receiving a test sample, wherein the test sample includes: a target fault type and expected behavior data; Determining a target fault corresponding to the target fault type based on a preset correspondence between various fault types and preset faults; Injecting the target fault into the training device of the model to be tested; Acquire actual behavior data from the fault-tolerant system, wherein the actual behavior data is data generated by the response behavior performed by the fault-tolerant system in response to the target fault; The actual behavior data and the expected behavior data are compared to obtain a test result of the fault-tolerant system.

2. The method according to claim 1, further comprising, after receiving the test sample: A test instruction is sent to the fault-tolerant system, so that the fault-tolerant system responds to the test instruction and notifies the training device to train the model to be tested.

3. The method according to claim 1, wherein the target fault type is at least one, and the test sample further includes a trigger state corresponding to the target fault type; After receiving the test sample, the method further includes: During the training process of the model to be tested, monitoring the training status of the model to be tested through the fault-tolerant system; Injecting the target fault into the training device of the model to be tested includes: When the training state is the trigger state, a target fault of a target fault type corresponding to the trigger state is injected into the training device.

4. The method according to claim 1, before injecting the target fault into the training device of the model to be tested, further comprising: Determining whether the faulty device corresponding to the target fault type is the training device; If yes, then executing the step of injecting the target fault into the training device of the model to be tested; If not, injecting the target fault into the fault-tolerant system.

5. The method according to claim 4, wherein The test sample also includes device information of a target training device, and injecting the target fault into the training device of the model to be tested includes: Based on the device information, a simulated fault instruction is sent to the target training device, where the simulated fault instruction is used to instruct the target training device to perform a fault injection operation to cause the target fault to occur in the target training device.

6. The method according to any one of claims 1 to 5, wherein: The comparing the actual behavior data with the expected behavior data to obtain a test result of the fault-tolerant system includes: The test result is determined based on a comparison result obtained in at least one of the following ways: comparing the telephone alarm information included in the actual behavior data with the telephone alarm information included in the expected behavior data to determine whether they match; comparing the message warning information included in the actual behavior data with the message warning information included in the expected behavior data to determine whether they match; comparing the fault recovery mode included in the actual behavior data with the fault recovery mode included in the expected behavior data to determine whether they match; The post-recovery training data included in the actual behavior data is compared with the post-recovery training data included in the expected behavior data to determine whether they match.

7. The method according to claim 6, after comparing the actual behavior data with the expected behavior data to obtain a test result of the fault-tolerant system, further comprising: If the test result indicates that the test has failed, the comparison result is matched with each preset rule to obtain a target rule that matches the test result; Get the failure reason corresponding to the target rule.

8. The method according to claim 6, after comparing the actual behavior data with the expected behavior data to obtain a test result of the fault-tolerant system, further comprising: If the test result indicates that the test has failed, the reason for the failure is determined based on the comparison result using a preset analysis model.

9. The method according to any one of claims 1 to 5, wherein the test sample further includes a test end condition, and after the actual behavior data and the expected behavior data are compared to obtain a test result for the fault-tolerant system, the method further includes: When the test end condition is met, if the test result indicates that the test has passed, the snapshot data of the model to be tested in the preset storage location is deleted.

10. A testing system for a fault-tolerant system, comprising: A test framework is configured to receive a test sample, determine a target fault corresponding to a target fault type based on a preset correspondence between various fault types and preset faults, and inject the target fault into a training device of a model to be tested, wherein the test sample includes: a target fault type and expected behavior data; A fault-tolerant system, configured to execute a response action after the target failure occurs; The test framework is also used to obtain actual behavior data from the fault-tolerant system, compare the actual behavior data with the expected behavior data, and obtain test results for the fault-tolerant system. The actual behavior data is: data generated by the response behavior performed by the fault-tolerant system to respond to the target fault.

11. The test system according to claim 10, further comprising: The test framework is further configured to send a test instruction to the fault-tolerant system after receiving the test sample; The fault-tolerant system is further configured to send a training notification to the training device in response to the test instruction; The training device is configured to create a training service in a training container in response to the training notification, and train the model to be tested through the training service.

12. The test system according to claim 11, wherein the target fault type is at least one, and the test sample further includes a trigger state corresponding to the target fault type; The training device is further configured to send the training status of the model to be tested to the fault-tolerant system during the process of training the model to be tested; The testing framework is further configured to monitor the training status of the model to be tested through the fault-tolerant system during the training process of the model to be tested after receiving the test sample; The test framework is specifically used to inject a target fault of a target fault type corresponding to the trigger state into the training device when the training state is the trigger state.

13. The test system according to claim 10, wherein the test framework is further configured to: Before injecting the target fault into the training device of the model to be tested, determining whether the faulty device corresponding to the target fault type is the training device; If yes, then executing the step of injecting the target fault into the training device of the model to be tested; If not, injecting the target fault into the fault-tolerant system.

14. The test system according to claim 13, wherein: The test sample also includes device information of the target training device; The test framework is further configured to send a simulated fault instruction to the target training device based on the device information; The training device is also used to execute the service through the fault in the training container, and in response to the simulated fault instruction, perform a fault injection operation on the training service or the training device where it is located, so as to cause the target fault to occur in the training device where it is located.

15. The test system according to any one of claims 10 to 14, wherein: The test framework is specifically used for: The test result is determined based on a comparison result obtained in at least one of the following ways: comparing the telephone alarm information included in the actual behavior data with the telephone alarm information included in the expected behavior data to determine whether they match; comparing the message warning information included in the actual behavior data with the message warning information included in the expected behavior data to determine whether they match; comparing the fault recovery mode included in the actual behavior data with the fault recovery mode included in the expected behavior data to determine whether they match; The post-recovery training data included in the actual behavior data is compared with the post-recovery training data included in the expected behavior data to determine whether they match.

16. The test system according to claim 15, wherein the test framework is further configured to: After the actual behavior data and the expected behavior data are compared to obtain a test result of the fault-tolerant system, if the test result indicates that the test fails, the comparison result is matched with each preset rule to obtain a target rule that matches the test result; Get the failure reason corresponding to the target rule.

17. The test system according to claim 15, wherein the test framework is further configured to: After the actual behavior data and the expected behavior data are compared to obtain a test result of the fault-tolerant system, if the test result indicates that the test fails, a preset analysis model is used to determine the reason for the failure based on the comparison result.

18. The test system according to any one of claims 10 to 14, wherein the test case further includes a test end condition, and the test framework is further configured to: After comparing the actual behavior data with the expected behavior data to obtain the test result of the fault-tolerant system, when the test end condition is met, if the test result indicates that the test has passed, the snapshot data of the model to be tested in the preset storage location is deleted.

19. A testing device for a fault-tolerant system, comprising: A receiving module, configured to receive a test sample, wherein the test sample includes: a target fault type and expected behavior data; A determination module, configured to determine a target fault corresponding to the target fault type based on a preset correspondence between various fault types and preset faults; An injection module, configured to inject the target fault into a training device of a model to be tested; An acquisition module is used to acquire actual behavior data from the fault-tolerant system, wherein the actual behavior data is data generated by the response behavior performed by the fault-tolerant system in response to the target fault; The comparison module is used to compare the actual behavior data with the expected behavior data to obtain a test result of the fault-tolerant system.

20. The apparatus according to claim 19, further comprising: The sending module is used to send a test instruction to the fault-tolerant system after receiving the test sample, so that the fault-tolerant system responds to the test instruction and notifies the training device to train the model to be tested.

21. The apparatus according to claim 19, wherein the target fault type is at least one, and the test sample further includes a trigger state corresponding to the target fault type; and further includes: A monitoring module, configured to monitor the training status of the model to be tested through the fault-tolerant system during the training process of the model to be tested after the test sample is received; The injection module is specifically used for: When the training state is the trigger state, a target fault of a target fault type corresponding to the trigger state is injected into the training device.

22. The apparatus of claim 19, further comprising: A judgment module, configured to judge whether the faulty device corresponding to the target fault type is the training device before injecting the target fault into the training device of the model to be tested; A calling module, configured to call the injection module to execute the step of injecting the target fault into the training device of the model to be tested if the judgment result of the judgment module is yes; The injection module is further configured to inject the target fault into the fault-tolerant system if the judgment result of the judgment module is no.

23. The device according to claim 22, wherein The test sample also includes device information of the target training device, and the injection module is specifically used to: Based on the device information, a simulated fault instruction is sent to the target training device, where the simulated fault instruction is used to instruct the target training device to perform a fault injection operation to cause the target fault to occur in the target training device.

24. The device according to any one of claims 19 to 23, wherein: The comparison module is specifically used for: The test result is determined based on a comparison result obtained in at least one of the following ways: comparing the telephone alarm information included in the actual behavior data with the telephone alarm information included in the expected behavior data to determine whether they match; comparing the message warning information included in the actual behavior data with the message warning information included in the expected behavior data to determine whether they match; comparing the fault recovery mode included in the actual behavior data with the fault recovery mode included in the expected behavior data to determine whether they match; The post-recovery training data included in the actual behavior data is compared with the post-recovery training data included in the expected behavior data to determine whether they match.

25. The apparatus according to claim 24, further comprising: a matching module configured to, after comparing the actual behavior data with the expected behavior data to obtain a test result of the fault-tolerant system, match the comparison result with each preset rule to obtain a target rule that matches the test result if the test result indicates that the test fails; The acquisition module is further configured to acquire the failure reason corresponding to the target rule.

26. The apparatus of claim 24, further comprising: The analysis module is used to compare the actual behavior data with the expected behavior data to obtain a test result of the fault-tolerant system. If the test result indicates that the test has failed, the analysis module uses a preset analysis model to determine the reason for failure based on the comparison result.

27. The apparatus according to any one of claims 19 to 23, wherein the test sample further includes a test end condition, further comprising: A deletion module is used to delete the snapshot data of the model to be tested in a preset storage location after the test result of the fault-tolerant system is obtained by comparing the actual behavior data with the expected behavior data, and when the test end condition is met, if the test result indicates that the test has passed.

28. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

29. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

30. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images