Reliability test method and system for publishing and subscribing system of smart power grid

Abstract

The invention discloses a method and a system for testing the reliability of a publishing and subscribing system for a smart power grid, and relates to the technical field of software. The method comprises the following steps: collecting fault types and fault instances aiming at network faults faced by a publishing and subscribing system, and constructing a fault library; configuring a test script for the publishing and subscribing system; testing the reliability of the publishing and subscribing system by using the configured test script; monitoring the running state of the test script and the state of the publishing and subscribing system, collecting running log information of the publishing and subscribing system, and if abnormity is detected, recording abnormal information into a file; and carrying out reliability evaluation on the publishing and subscribing system. According to the method, various fault scenes possibly faced by the publishing and subscribing system in an intelligent power grid environment in an actual application scene are simulated, the fault-tolerant mechanism and the recovery capability of the publishing and subscribing system are comprehensively evaluated, defects possibly existing in the system are helped to be explored, and the reliability of the publishing and subscribing system is improved.

Description

Technical Field

[0001] The present invention relates to the field of software technology, and in particular to a reliability testing method and system for a publish-subscribe system for a smart grid. Background Art

[0002] Publish and subscribe is a message delivery mechanism in a distributed environment. It is currently widely used in cloud and edge scenarios, such as sensor status acquisition, system status monitoring and alarm, microservice communication, real-time content notification and distribution, etc. It supports the core business of key fields such as electricity and finance, and has the characteristics of low coupling and strong scalability. The publish-subscribe system mainly includes three roles: publisher, subscriber and message broker. The publisher publishes messages to the message broker. The message broker is responsible for receiving messages from the publisher and routing them to the subscriber. The subscriber subscribes to the message broker for specific messages that meet the conditions. The publisher and the subscriber are independent of each other and do not need to know the existence of each other. In the publish-subscribe system, there are two ways of message filtering, namely, topic-based publish-subscribe and content-based publish-subscribe. Topic-based publish-subscribe means that messages are classified according to topics, and subscribers only subscribe to and receive messages of specific topics. Content-based publish-subscribe means that messages are distributed according to content, and subscribers subscribe according to message content attributes, etc. Only message data that meets specific conditions will be received. In order to ensure the reliability of the publish-subscribe system, the message broker will persist the message data and store the message data on disk or database to avoid message loss.

[0003] In the core scenarios of key areas such as smart grids, the publish-subscribe system is usually deployed in the form of a distributed system with multiple nodes. In a distributed environment, there are many system nodes, and the communication behavior between systems may be complicated and vulnerable to various faults or errors, such as network anomalies, power outages, software errors, etc. For example, once a fault occurs in a smart grid scenario, it will directly affect the reliability of the power system, and may cause serious problems such as data leakage, system failures, and even power supply interruptions, leading to property losses; once there is a problem with the monitoring and alarm system of power equipment, there may be false alarms or omissions of abnormal conditions, leading to production accidents. Therefore, the reliability test and evaluation of the publish-subscribe system is very important. Testing the performance of the publish-subscribe system under various distributed environment faults and abnormal conditions can well reduce the occurrence of problems in production scenarios, improve system reliability, and ensure the stable operation of the publish-subscribe system.

[0004] At present, there are some testing tools that test system reliability by fault injection. Among them, ChaosMesh mainly supports fault injection from the physical host and Kubernetes levels, including disk failure, network failure, process failure and Pod failure; Byte-Monkey is mainly used to simulate fault scenarios in JVM applications, introducing exceptions and delays by adjusting the application bytecode; FIAT supports injecting errors into application code and data, and can inject errors into messages, tasks and timers; FERRARI uses software to simulate hardware failures or errors, supports injecting CPU, memory and bus failures, such as simulating actual data corruption by changing the contents of selected registers or memory locations; NetLoiter modifies the underlying network interface to cause network failures, and can specify the timing of fault injection according to time, number of packets and address information. However, the existing testing tools only target some coarse-grained fault types, and cannot be directly applied to publish-subscribe systems, and lack fault injection methods in edge environments. At present, there is a lack of reliability testing methods and tools specifically for distributed publish-subscribe systems, and it is impossible to evaluate the correctness of message transmission and network reliability of distributed publish-subscribe systems. Summary of the invention

[0005] The purpose of the present invention is to propose a reliability testing method and system for a publish-subscribe system for smart grids. By simulating various failure scenarios that the publish-subscribe system in a smart grid environment may face in actual application scenarios, the fault tolerance mechanism and recovery capability of the publish-subscribe system are comprehensively evaluated, helping to discover possible defects in the system and improve the reliability of the publish-subscribe system.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A reliability testing method for a publish-subscribe system for a smart grid includes the following steps:

[0008] 1) For network failures faced by the publish-subscribe system, collect failure types and failure instances and build a failure database;

[0009] 2) Configure test scripts for the publish-subscribe system;

[0010] 3) Use the configured test script to test the reliability of the publish-subscribe system;

[0011] 4) Monitor the running status of the test script and the status of the publish-subscribe system, collect the running log information of the publish-subscribe system, and if an abnormality is detected, record the abnormal information in a file;

[0012] 5) Conduct reliability assessment on the publish-subscribe system.

[0013] Furthermore, the steps of configuring the test script include:

[0014] Automatically combine fault scenarios among the fault types included in the fault library. Fault types include network transmission faults and network delay faults. Fault scenarios are instances of each fault type.

[0015] Configure test script execution time;

[0016] Configure the address information of the publish-subscribe system to be tested.

[0017] Furthermore, the instance of the fault type is a specific operation configured for the fault type, including a network transmission fault instance and a network delay fault instance. The network transmission fault instance includes:

[0018] The message tampering operation is to flip the bits of data in a randomly selected message according to a preset probability;

[0019] The message loss operation is to delete the data in the randomly selected message;

[0020] Message duplication operation is to copy the data in a randomly selected message;

[0021] The message shuffle operation is to randomly select different data in the message and swap their positions.

[0022] Examples of network latency failures include:

[0023] The node or cluster disconnection operation generates a probability value for each node and disconnects the nodes whose probability value exceeds the threshold.

[0024] Furthermore, the step of automatically combining fault scenarios includes:

[0025] Randomly generate the number of fault combinations m;

[0026] Randomly generate m non-repeating fault types from the fault library;

[0027] Search the above fault type combination from the test records. If this fault combination has been tested, repeat the previous step and reselect a new fault type combination.

[0028] The fault type combinations are stored in the fault list in order and saved in the test record file.

[0029] Further, the steps to test the reliability of the publish-subscribe system include:

[0030] Read a fault scenario in the fault list in order;

[0031] Apply the operations in the failure scenario to the publish-subscribe system at the specified address;

[0032] Repeat the reading until all the fault scenarios in the fault list are actually executed, and the current test ends;

[0033] If the program of the publish-subscribe system has not been executed to completion but the execution time of the program set in the test script has been reached, the current round of testing will be forcibly terminated.

[0034] Furthermore, the step of monitoring the running status of the test script and the status of the publish-subscribe system includes: when executing each fault operation in the fault list, collecting the running log information of the publish-subscribe system, the fault operation information, the message sent by the publisher, the message received by the subscriber and the running status information of the cluster node.

[0035] Furthermore, the steps of evaluating the reliability of the publish-subscribe system include:

[0036] For each fault operation in the fault list, view the test report file;

[0037] If the test result is as expected, it means that the publish-subscribe system can handle the fault operation. Otherwise, it means that the publish-subscribe system cannot handle the fault operation.

[0038] Count the number of expected failure operations f and calculate the reliability index f / m of the publish-subscribe system.

[0039] A publish-subscribe system reliability test system for smart grid, comprising:

[0040] The fault collection module is used to collect fault types and fault instances for network faults faced by the publish-subscribe system, build a fault library, and configure the fault operations required in the test script;

[0041] Test script configuration module, used to configure test scripts for the publish-subscribe system;

[0042] A test script execution module is used to perform reliability testing on the publish-subscribe system according to the fault type configured by the test script configuration module;

[0043] The monitoring module receives monitoring requests from the test script execution module and monitors the performance of the application and the publish-subscribe system in real time;

[0044] Reliability evaluation module, used to evaluate the reliability of the publish-subscribe system;

[0045] The storage module receives storage requests from the test script configuration module, the test script execution module, the monitoring module and the reliability assessment module, and stores the recorded information persistently in the specified file.

[0046] Furthermore, the steps performed by the monitoring module include:

[0047] Monitor the process status of the application of the publish-subscribe system to determine whether the application is in an abnormal state, including abnormal process exit or long-term abnormal suspension;

[0048] Monitor the log information generated by the publish-subscribe system and use keyword searches to determine whether the system has any anomalies, including error messages or error code information;

[0049] Monitor the status of the current node and determine whether the current node can be accessed normally.

[0050] Furthermore, the execution steps of the storage module include: when the monitoring module detects an abnormality, the storage module promptly records current test information, including fault type information, fault operation information, application status information, system status information and node status information.

[0051] The advantages of the present invention compared with the prior art are:

[0052] 1. The present invention solves the problem of reliability test and evaluation of publish-subscribe system, and can help discover potential defects in publish-subscribe system and improve system reliability;

[0053] 2. The present invention deeply analyzes the architecture characteristics and application scenario characteristics of the publish-subscribe system, designs a variety of specific failure types of the publish-subscribe system, and can simulate various specific failure scenarios that the publish-subscribe system may face;

[0054] 3. The present invention designs a test method for systematically combining fault scenarios, supports automatic search for various fault combination scenarios, and systematically tests the reliability of the publish-subscribe system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is an architecture diagram of a publish-subscribe system reliability testing system for smart grids of the present invention. DETAILED DESCRIPTION

[0056] In order to make the various technical features and advantages or technical effects in the above technical solutions of the present invention more obvious and easy to understand, they are described in detail below with reference to the accompanying drawings.

[0057] The embodiment of the present invention provides a publish-subscribe system reliability testing method and system for smart grid. The system executes each step of the method. The architecture diagram of the system is shown in FIG. Figure 1 As shown. This method performs specific fault operations on the publish-subscribe system according to the predefined fault scenarios based on the test script configured by the user, so as to discover the defects in the publish-subscribe system and evaluate the reliability performance of the publish-subscribe system. The specific implementation steps of this method are as follows:

[0058] Step 1: Build a fault library.

[0059] In view of the network failures that publish-subscribe systems often face, the failure collection module collects common failure types and failure instances to form a failure library. Failure types include network transmission failures and network delay failures. Network transmission failures refer to message tampering, message loss, message duplication, message disorder, etc. during the message transmission process of the publish-subscribe system. Network delay failures refer to the situation where nodes or clusters are disconnected during the message transmission process of the publish-subscribe system.

[0060] For example, the fault library is formally represented as F = {Ft, Fl}, where Ft = {Mc, Ml, Mr, Md}, Mc indicates that the publish-subscribe system has message tampering during the message transmission process, Ml indicates that the publish-subscribe system has message loss during the message transmission process, Mr indicates that the publish-subscribe system has message duplication during the message transmission process, and Md indicates that the publish-subscribe system has message disorder during the message transmission process. Fl = {Nd}, Nd indicates that the publish-subscribe system has node or cluster disconnection during the message transmission process.

[0061] Step 2: Configure test scripts for the publish-subscribe system.

[0062] The test script configuration module configures the test script, including the following steps:

[0063] 1) Use the automatic combination of fault scenarios function to generate a fault list by the system. Specifically, the following steps are included:

[0064] (a) Randomly generate a fault combination number m, that is, the number of faults included in the test script, where m does not exceed the number of fault types, for example, m is less than the total number of fault scenarios 5. In this embodiment, m is randomly generated to be 2.

[0065] Fault scenarios are instances of various fault types. Fault type instances are specific operations configured for the fault type, including network transmission fault instances and network delay fault instances. Network transmission fault instances include:

[0066] The message tampering operation is to perform bit flipping on the data in a randomly selected message;

[0067] The message loss operation is to delete the data in the randomly selected message;

[0068] Message duplication operation is to copy the data in a randomly selected message;

[0069] The message shuffle operation is to randomly select different data in the message and swap their positions.

[0070] Examples of network delay failures include: node or cluster disconnection operations, generating a probability value for each node, and disconnecting the node when the probability value exceeds 80%.

[0071] (b) Randomly select m non-repetitive fault scenarios from the fault library, for example, the selected fault scenarios are {Ml,Nd}.

[0072] (c) Search the test records to see if the fault sequence {Ml, Nd} already exists. This process only requires generating fault IDs (unique identifiers of faults) corresponding to m fault types. If the fault sequence has been tested, repeat step (b) and reselect a new fault type combination.

[0073] (d) The selected fault scenario is stored in the fault list, i.e., FList = {Ml, Nd}, and the fault sequence is recorded in a file.

[0074] 2) Configure the test script execution time. For example, set the execution time T = 5 minutes. If the execution time of 5 minutes has been reached and the test has not been completed, the system will force the test process to end.

[0075] 3) Configure the address information of the publish-subscribe system to be tested, including the access address accessAddress of the publish-subscribe system and the test report storage address storeAddress.

[0076] For example, the configuration access address is accessAddress=http: / / 192.168.1.101 / opt / undertest, and the configuration storage address is storeAddress=http: / / 192.168.1.101 / opt / storage.

[0077] Step 3: Use the configured test script to test the reliability of the publish-subscribe system.

[0078] The test script execution module performs reliability testing in the following steps:

[0079] 1) Read the first fault scenario in the fault list in order. For example, first read the first fault scenario Ml in the fault list FList = {Ml, Nd}, that is, the message loss fault. Apply the fault operation to the publish-subscribe system at the specified address. For example, the fault operation is a message loss fault, that is, randomly select a number of messages generated by the publisher according to probability for deletion.

[0080] 2) Continue to read the second fault scenario Nd in the fault list FList = {Ml, Nd}, that is, the node disconnection fault. Specifically, a probability value will be generated for each cluster node, and when the probability value exceeds 80%, the node will be disconnected. For example, the publish-subscribe system cluster has a total of 5 nodes. Among the randomly generated probability values, if two nodes exceed 80%, the two nodes will be disconnected.

[0081] 3) When all the fault scenarios in the fault list are actually executed, the current test ends. If the program of the publish-subscribe system is not executed completely but the execution time corresponding to the program running set in the test script has reached 5 minutes, this round of testing will be forcibly terminated.

[0082] Step 4: Monitor the running status of the test script and the status of the publish-subscribe system, and collect the running log information of the publish-subscribe system.

[0083] During the above reliability test, the monitoring module will monitor the status information of the publish-subscribe system in real time, including the fault list FList = {Ml, Nd}, the message M sent by the publisher, the message M' received by the subscriber, the running status of the cluster node, etc. Specifically, the monitoring module monitors the process status of the application of the publish-subscribe system to determine whether the application is in an abnormal state, such as abnormal suspension, etc.; monitors the log information generated by the publish-subscribe system, and determines whether the system has abnormalities through keyword retrieval, such as error information and error code information; monitors the status of the current node to determine whether the current node can be accessed normally.

[0084] Step 5: Conduct reliability assessment on the publish-subscribe system.

[0085] The reliability assessment module analyzes the reliability of the publish-subscribe system based on the log information collected during the reliability test. Specifically, the reliability assessment includes the following steps:

[0086] 1) For each fault operation M1 and Nd in the fault list, check the corresponding test report file;

[0087] 2) If the test result is in line with expectations, it means that the publish-subscribe system can cope with the fault operation, otherwise it means that the publish-subscribe system cannot cope with the fault operation. For example, for fault operation M1, the subscriber did not find that the message was lost, so it is considered that the publish-subscribe system does not meet the expected behavior; for fault operation Nd, when the two nodes are disconnected, the subscriber can still get the message, so it is considered that the publish-subscribe system meets expectations.

[0088] 3) Count the number of expected failure operations 1 and calculate the reliability index 1 / 2 of the publish-subscribe system.

[0089] Finally, the storage module receives storage requests from the test script configuration module, test script execution module, monitoring module and reliability assessment module, and stores the recorded information persistently in the specified file. When the monitoring module detects an abnormality, the storage module promptly records the current test information, including fault type information, fault operation information, application status information, system status information and node status information.

[0090] A specific example is provided below:

[0091] (1) The distributed publish-subscribe system consists of five nodes (i.e., message brokers) that jointly provide services to the outside world. There are 25 subscribers connected to the publish-subscribe system (five subscribers connected to the same message broker), and five publishers connected to the publish-subscribe system (one publisher connected to one message broker). Messages published by publishers to a specified topic will be synchronized among the five message brokers, and the subscribers who have subscribed to the topic will be notified.

[0092] (2) The user intends to test whether the publishing and subscription system can cope with message tampering and loss failures during the message publishing process. The constructed fault library contains two types of faults: "message tampering" and "message loss". The maximum probability of message tampering is configured as 1 bit flip per MB of data, and the minimum probability of message tampering is configured as 1 bit flip per 100 MB of data. The maximum probability of message loss is 0.5%, and the minimum probability of message loss is 0.5%.

[0093] (3) The reliability test system automatically generates fault combinations. For example, if the fault scenario is "only includes message tampering", "only includes message loss", or "both includes message tampering and message loss", the corresponding fault execution sequence is generated and saved.

[0094] (4) The user configures the test time and the access address and storage address of the publish-subscribe system to be tested, and injects faults through the access address.

[0095] (5) Specific reliability tests are performed. For each fault scenario in the fault list, a specific fault is injected. For example, in this scenario, the publisher sends 5MB of data (split into 1,000 independent requests) to the message broker and pushes it to all 25 subscribers. According to the probability configured in step (2), specific bits are randomly selected for flipping (0 changes to 1, 1 changes to 0), and specific messages are randomly deleted to inject message tampering and message loss faults.

[0096] (6) The monitoring module continuously monitors the publish-subscribe system process, for example, determining whether the response information sent to the subscriber is erroneous, whether data tampering is detected, whether message resending can be triggered, whether message loss is detected, etc.

[0097] (7) Determine whether the publish-subscribe system can cope with specified failure scenarios based on user-defined criteria, such as whether all subscribers eventually receive the correct message.

[0098] (8) Repeat steps (5) to (8), record the expected failure scenarios, and calculate the reliability index of the system.

[0099] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.

Claims

1. A reliability testing method for a publish-subscribe system for smart grids, characterized in that: The following steps are involved: 1) For network failures faced by the publish-subscribe system, collect failure types and failure instances and build a failure database; 2) Configure test scripts for the publish-subscribe system; 3) Use the configured test script to test the reliability of the publish-subscribe system; 4) Monitor the running status of the test script and the status of the publish-subscribe system, collect the running log information of the publish-subscribe system, and if an abnormality is detected, record the abnormal information in a file; 5) Conduct reliability assessment on the publish-subscribe system.

2. The method according to claim 1, characterized in that The steps to configure a test script include: Automatically combine fault scenarios among the fault types included in the fault library. Fault types include network transmission faults and network delay faults. Fault scenarios are instances of each fault type. Configure test script execution time; Configure the address information of the publish-subscribe system to be tested.

3. The method according to claim 2, characterized in that An instance of a fault type is a specific operation configured for the fault type, including a network transmission fault instance and a network delay fault instance. The network transmission fault instance includes: The message tampering operation is to flip the bits of data in a randomly selected message according to a preset probability; The message loss operation is to delete the data in the randomly selected message; Message duplication operation is to copy the data in a randomly selected message; The message shuffle operation is to randomly select different data in the message and swap their positions. Examples of network latency failures include: The node or cluster disconnection operation generates a probability value for each node and disconnects the nodes whose probability value exceeds the threshold.

4. The method according to claim 2, characterized in that The step of automatically combining fault scenarios comprises: Randomly generate the number of fault combinations m; Randomly generate m non-repeating fault types from the fault library; Search the above fault type combination from the test records. If this fault combination has been tested, repeat the previous step and reselect a new fault type combination. The fault type combinations are stored in the fault list in order and saved in the test record file.

5. The method according to claim 1, characterized in that The steps to test the reliability of the publish-subscribe system include: Read a fault scenario in the fault list in order; Apply the operations in the failure scenario to the publish-subscribe system at the specified address; Repeat the reading until all the fault scenarios in the fault list are actually executed, and the current test ends; If the program of the publish-subscribe system has not been executed to completion but the execution time of the program set in the test script has been reached, the current round of testing will be forcibly terminated.

6. The method according to claim 1, characterized in that The steps of monitoring the running status of the test script and the status of the publish-subscribe system include: when executing each fault operation in the fault list, collecting the running log information of the publish-subscribe system, the fault operation information, the message sent by the publisher, the message received by the subscriber and the running status information of the cluster node.

7. The method according to claim 1, characterized in that The steps to evaluate the reliability of a publish-subscribe system include: For each fault operation in the fault list, view the test report file; If the test result is as expected, it means that the publish-subscribe system can handle the fault operation. Otherwise, it means that the publish-subscribe system cannot handle the fault operation. Count the number of expected failure operations f and calculate the reliability index f / m of the publish-subscribe system.

8. A publish-subscribe system reliability test system for smart grid, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The fault collection module is used to collect fault types and fault instances for network faults faced by the publish-subscribe system, build a fault library, and configure the fault operations required in the test script; Test script configuration module, used to configure test scripts for the publish-subscribe system; A test script execution module is used to perform reliability testing on the publish-subscribe system according to the fault type configured by the test script configuration module; The monitoring module receives monitoring requests from the test script execution module and monitors the performance of the application and the publish-subscribe system in real time; Reliability evaluation module, used to evaluate the reliability of the publish-subscribe system; The storage module receives storage requests from the test script configuration module, the test script execution module, the monitoring module and the reliability assessment module, and stores the recorded information persistently in the specified file.

9. The system according to claim 8, characterized in that The steps performed by the monitoring module include: Monitor the process status of the application of the publish-subscribe system to determine whether the application is in an abnormal state, including abnormal process exit or long-term abnormal suspension; Monitor the log information generated by the publish-subscribe system and use keyword searches to determine whether the system has any anomalies, including error messages or error code information; Monitor the status of the current node and determine whether the current node can be accessed normally.

10. The system according to claim 8, characterized in that The execution steps of the storage module include: when the monitoring module detects an abnormality, the storage module promptly records current test information, including fault type information, fault operation information, application status information, system status information and node status information.

Images