Cluster test method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to the technical field of big data testing, and particularly discloses a cluster testing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring resource transaction data and node control modes of a to-be-tested cluster in a plurality of test scenes; for each test scene, determining a cluster fault injection strategy in the test scene according to the cluster initial state of the to-be-tested cluster, the resource transaction data and the node control mode in the test scene under the condition that the online service is started; performing online fault simulation on the to-be-tested cluster based on the cluster fault injection strategy in the test scene and the resource transaction data to obtain cluster performance index data of the to-be-tested cluster in the test scene; obtaining a test result of the test scene according to the cluster performance index data; and obtaining a test result of the to-be-tested cluster based on the test results of the plurality of test scenes. By adopting the method, the high availability of the cluster can be comprehensively tested.

Description

Technical Field

[0001] The present application relates to the field of big data testing technology, and in particular to a cluster testing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of computer technology, clusters have emerged. Clusters are a computer technology that connects multiple computers (called nodes or servers) together so that they work together as a single system. The main purpose of clusters is to improve the availability, reliability, scalability and performance of the system. Clusters are widely used in various fields and can be applied to online resource transaction processing in financial institutions. Before putting a cluster into use, it is usually necessary to evaluate the high availability of the cluster to ensure the stable execution of online resource transaction processing.

[0003] In traditional technology, the cluster is tested by performing read and write operations in the cluster to verify whether the data can be correctly distributed and accessed between different cluster nodes. However, the traditional method is relatively simple and cannot fully test the high availability of the cluster. Summary of the invention

[0004] Based on this, it is necessary to provide a cluster testing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can comprehensively test the high availability of the cluster in response to the above technical problems.

[0005] In a first aspect, the present application provides a cluster testing method, comprising:

[0006] Obtain resource transaction data for cluster testing and node control methods of the cluster under test in multiple test scenarios;

[0007] For each of the test scenarios, when starting the online service, the cluster initial state of the cluster to be tested is obtained, and the cluster fault injection strategy under the test scenario is determined according to the cluster initial state, the resource transaction data and the node control mode under the test scenario;

[0008] Based on the cluster fault injection strategy in the test scenario and the resource transaction data, an online fault simulation is performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario;

[0009] According to the cluster performance indicator data, the cluster to be tested is evaluated to obtain a test result of the test scenario;

[0010] Based on the test results of each of the multiple test scenarios, a test result of the cluster to be tested is obtained.

[0011] In one embodiment, determining a cluster fault injection strategy in the test scenario according to the cluster initial state, the resource transaction data, and the node control mode in the test scenario includes:

[0012] Based on the transaction type to which the resource transaction data belongs, determining a transaction processing node in the cluster to be tested that processes the resource transaction data;

[0013] Determine, according to the cluster initial state and the transaction processing node, a master node in the cluster to be tested that processes the resource transaction data;

[0014] A cluster fault injection strategy in the test scenario is determined according to the master node and the node control mode in the test scenario.

[0015] In one embodiment, based on the cluster fault injection strategy in the test scenario and the resource transaction data, online fault simulation is performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario, including:

[0016] Based on the cluster fault injection strategy in the test scenario, performing fault setting on the cluster to be tested;

[0017] After the fault setting is completed, the online service is continuously run based on the resource transaction data to obtain cluster performance indicator data of the cluster to be tested in the test scenario.

[0018] In one embodiment, after the fault setting is completed, the online service is continuously run based on the resource transaction data to obtain cluster performance indicator data of the cluster to be tested in the test scenario, including:

[0019] After the fault setting is completed, the online service is continuously operated based on the resource transaction data, and during the operation of the online service, cluster status information, transaction online processing results and server resource usage data are obtained;

[0020] Until the online service runs to the end, the cluster performance indicator data of the cluster to be tested in the test scenario is obtained according to the acquired cluster status information, the transaction online processing result and the server resource usage data.

[0021] In one embodiment, the cluster to be tested is evaluated according to the cluster performance indicator data to obtain the test result of the test scenario, including:

[0022] Obtaining cluster performance indicator requirements for the test scenario;

[0023] By comparing the cluster performance indicator requirement and the cluster performance indicator data, the cluster to be tested is evaluated to obtain a test result of the test scenario.

[0024] In one embodiment, the cluster testing method further includes:

[0025] Based on the resource transaction data, an online service request is initiated to an online server connected to the cluster to be tested to start an online service.

[0026] In a second aspect, the present application also provides a cluster testing device, including:

[0027] The data acquisition module is used to obtain resource transaction data for cluster testing and node control methods of the cluster to be tested in multiple test scenarios;

[0028] A strategy formulation module, for each of the test scenarios, when starting the online service, obtains the cluster initial state of the cluster to be tested, and determines the cluster fault injection strategy in the test scenario according to the cluster initial state, the resource transaction data and the node control mode in the test scenario;

[0029] A test module, used to perform online fault simulation on the cluster to be tested based on the cluster fault injection strategy in the test scenario and the resource transaction data, and obtain cluster performance indicator data of the cluster to be tested in the test scenario;

[0030] An evaluation module, used to evaluate the cluster to be tested according to the cluster performance indicator data to obtain a test result of the test scenario;

[0031] The aggregation module is used to obtain the test result of the cluster to be tested based on the test results of each of the multiple test scenarios.

[0032] In one embodiment, the policy making module includes:

[0033] A transaction analysis unit, configured to determine a transaction processing node in the cluster to be tested that processes the resource transaction data based on a transaction type to which the resource transaction data belongs;

[0034] A master node determination unit, configured to determine a master node for processing the resource transaction data in the cluster to be tested according to the cluster initial state and the transaction processing node;

[0035] The strategy generating unit is used to determine the cluster fault injection strategy in the test scenario according to the master node and the node control mode in the test scenario.

[0036] In one embodiment, the test module includes:

[0037] A fault setting unit, configured to perform fault setting on the cluster to be tested based on a cluster fault injection strategy in the test scenario;

[0038] The data collection unit is used to continuously run the online service based on the resource transaction data after the fault setting is completed, and obtain the cluster performance indicator data of the cluster to be tested in the test scenario.

[0039] In one embodiment, the data collection unit is specifically used to:

[0040] After the fault setting is completed, the online service is continuously operated based on the resource transaction data, and during the operation of the online service, cluster status information, transaction online processing results and server resource usage data are obtained;

[0041] Until the online service runs to the end, the cluster performance indicator data of the cluster to be tested in the test scenario is obtained according to the acquired cluster status information, the transaction online processing result and the server resource usage data.

[0042] In one embodiment, the evaluation module includes:

[0043] A requirement acquisition unit, used to acquire the performance indicator requirements of the test scenario;

[0044] The comparison unit is used to evaluate the cluster to be tested by comparing the performance indicator requirement with the cluster performance indicator data to obtain a test result of the test scenario.

[0045] In one embodiment, the cluster testing device further includes an online module for:

[0046] Based on the resource transaction data, an online service request is initiated to an online server connected to the cluster to be tested to start an online service.

[0047] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0048] Obtain resource transaction data for cluster testing and node control methods of the cluster under test in multiple test scenarios;

[0049] For each of the test scenarios, when starting the online service, the cluster initial state of the cluster to be tested is obtained, and the cluster fault injection strategy under the test scenario is determined according to the cluster initial state, the resource transaction data and the node control mode under the test scenario;

[0050] Based on the cluster fault injection strategy in the test scenario and the resource transaction data, an online fault simulation is performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario;

[0051] According to the cluster performance indicator data, the cluster to be tested is evaluated to obtain a test result of the test scenario;

[0052] Based on the test results of each of the multiple test scenarios, a test result of the cluster to be tested is obtained.

[0053] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0054] Obtain resource transaction data for cluster testing and node control methods of the cluster under test in multiple test scenarios;

[0055] For each of the test scenarios, when starting the online service, the cluster initial state of the cluster to be tested is obtained, and the cluster fault injection strategy under the test scenario is determined according to the cluster initial state, the resource transaction data and the node control mode under the test scenario;

[0056] Based on the cluster fault injection strategy in the test scenario and the resource transaction data, an online fault simulation is performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario;

[0057] According to the cluster performance indicator data, the cluster to be tested is evaluated to obtain a test result of the test scenario;

[0058] Based on the test results of each of the multiple test scenarios, a test result of the cluster to be tested is obtained.

[0059] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0060] Obtain resource transaction data for cluster testing and node control methods of the cluster under test in multiple test scenarios;

[0061] For each of the test scenarios, when starting the online service, the cluster initial state of the cluster to be tested is obtained, and the cluster fault injection strategy under the test scenario is determined according to the cluster initial state, the resource transaction data and the node control mode under the test scenario;

[0062] Based on the cluster fault injection strategy in the test scenario and the resource transaction data, an online fault simulation is performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario;

[0063] According to the cluster performance indicator data, the cluster to be tested is evaluated to obtain a test result of the test scenario;

[0064] Based on the test results of each of the multiple test scenarios, a test result of the cluster to be tested is obtained.

[0065] The above-mentioned cluster testing method, apparatus, computer equipment, computer-readable storage medium and computer program product, based on obtaining resource transaction data for cluster testing and node control methods of the cluster to be tested in multiple test scenarios, can obtain the cluster initial state of the cluster to be tested for each test scenario when starting the online service, and can use the cluster initial state, resource transaction data and the node control method in the test scenario to determine the cluster fault injection strategy in the test scenario. Then, based on the cluster fault injection strategy in the test scenario and the resource transaction data, online fault simulation can be performed on the cluster to be tested, and cluster performance indicator data of the cluster to be tested in the test scenario can be obtained. According to the cluster performance indicator data, the cluster to be tested is evaluated to obtain the test results of the test scenario, so that the test results of each of the multiple test scenarios can be summarized to obtain the test results of the cluster to be tested. In the whole process, the cluster fault injection strategy in each test scenario can be used to formulate cluster fault injection strategies in multiple test scenarios by using the cluster initial state, resource transaction data and node control methods, and a comprehensive test of the high availability of the cluster can be achieved by using the cluster fault injection strategy in each test scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 An application environment diagram of a cluster testing method in an embodiment;

[0068] Figure 2 An application environment diagram of a cluster testing method in an embodiment;

[0069] Figure 3 A schematic diagram of a cluster testing method in one embodiment;

[0070] Figure 4A schematic diagram of a process for determining a cluster fault injection strategy in a test scenario in an embodiment;

[0071] Figure 5 A schematic diagram of a cluster testing method according to another embodiment;

[0072] Figure 6 This is a general structural diagram of a Redis cluster test in one embodiment;

[0073] Figure 7 A schematic diagram of a process flow for stopping an application test in an embodiment;

[0074] Figure 8 A schematic diagram of a hang test process in one embodiment;

[0075] Fig. 9 A schematic diagram of a process of shutting down a master node network card in an embodiment;

[0076] Fig.10 is a structural block diagram of a cluster testing device in one embodiment;

[0077] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.

[0080] The cluster testing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the cluster to be tested 104 through the network. The data storage system can store the data that the cluster to be tested 104 needs to process. The data storage system can be integrated on the cluster to be tested 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains resource transaction data for cluster testing and the node control method of the cluster to be tested 104 in multiple test scenarios. For each test scenario, when the online service is started, the cluster initial state of the cluster to be tested 104 is obtained, and the cluster fault injection strategy in the test scenario is determined according to the cluster initial state, resource transaction data and the node control method in the test scenario. Based on the cluster fault injection strategy in the test scenario and the resource transaction data, the cluster to be tested 104 is simulated online for faults, and the cluster performance indicator data of the cluster to be tested 104 in the test scenario is obtained. According to the cluster performance indicator data, the cluster to be tested 104 is evaluated to obtain the test results of the test scenario, and the test results of the cluster to be tested 104 are obtained based on the test results of each of the multiple test scenarios.

[0081] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The cluster to be tested 104 includes multiple servers, which may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide cloud computing services.

[0082] The cluster testing method provided in the embodiment of the present application can be applied to Figure 2 In the application environment shown. Among them, the terminal 202 communicates with the intelligent financial technology component through the network, and the intelligent financial technology component includes an online server 204, a cluster to be tested 206 and a database server 208. The cluster to be tested 206 can communicate with the online server 204 and the database server 208. After receiving the online service request, the online server 204 can process the online service request by communicating with the cluster to be tested 206 or the database server 208. The data storage system can store data that needs to be processed by the cluster to be tested 206. The data storage system can be integrated on the cluster to be tested 206, or it can be placed on the cloud or other network servers.

[0083] The terminal 202 obtains resource transaction data for cluster testing and the node control method of the cluster to be tested 206 in multiple test scenarios. For each test scenario, when an online service request is initiated to the online server 204 based on the resource transaction data and the online service is started, the cluster initial state of the cluster to be tested 206 is obtained. According to the cluster initial state, the resource transaction data and the node control method in the test scenario, the cluster fault injection strategy in the test scenario is determined. Based on the cluster fault injection strategy in the test scenario and the resource transaction data, an online fault simulation is performed on the cluster to be tested 206 to obtain cluster performance indicator data of the cluster to be tested 206 in the test scenario. According to the cluster performance indicator data, the cluster to be tested 206 is evaluated to obtain the test result of the test scenario. Based on the test results of multiple test scenarios, the test result of the cluster to be tested 206 is obtained.

[0084] The terminal 202 may be a JMeter (stress testing tool) simulator, which uses the JMeter simulator as a stress testing tool to initiate an online service request to the smart financial technology component. The cluster to be tested 206 includes multiple servers. The server in this embodiment may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0085] In an exemplary embodiment, Figure 3 As shown, a cluster testing method is provided, which can be applied to a server or a terminal, and can also be implemented through the interaction between the server and the terminal. Figure 1 The terminal in is taken as an example to illustrate, including the following steps 302 to 310. Among them:

[0086] Step 302: Acquire resource transaction data for cluster testing and node control methods of the cluster to be tested in multiple test scenarios.

[0087] Among them, resource transaction data refers to various types of data generated in the process of resource transaction. For example, resource transaction data can specifically refer to various types of data generated in the process of resource transaction under various services provided by financial institutions. For example, resource transaction data can specifically refer to data related to processing the decimal places of the amount according to the currency. For another example, resource transaction data can specifically refer to data related to viewing the basic information of financial instruments.

[0088] The cluster to be tested refers to the cluster to be tested. It is understood that the cluster to be tested includes multiple cluster nodes, which connects multiple cluster nodes together so that they work together as a single system. The test scenario refers to a series of actions and conditions designed to verify the behavior and performance of the test object under specific conditions during the test process. In this embodiment,

[0089] The test scenario refers to a series of actions and conditions designed during the test process to verify the behavior and performance of the cluster under test under specific fault cluster conditions. For example, the test scenario can be stopping applications, crashing network cards, suspending processes, etc. The node control method under the test scenario refers to the method of controlling cluster nodes under the test scenario, which can be stopping the application of the master node, stopping the applications of the master node and slave nodes, crashing the network card of the master node, suspending the operation of the master node, etc.

[0090] Among them, stopping applications means deliberately stopping one or more application instances in the cluster to be tested to simulate application failures or node failures. It is mainly used to verify the fault tolerance, stability and failure recovery mechanism of the cluster when facing application layer failures. Through the "stop application" test, the performance of the cluster when encountering application failures in actual operation, as well as the automatic recovery and load balancing capabilities of the cluster can be evaluated. This kind of test is crucial to ensure the high availability and business continuity of the cluster.

[0091] Among them, the network card failure refers to the situation of simulating the failure of the network interface (network card), which is mainly used to test the fault tolerance and stability of the cluster when facing network failures. Through the "network card failure" test, it can be verified whether the services in the cluster can automatically recover, reroute traffic or failover when a network failure occurs to ensure the high availability of the service. This test is crucial for evaluating the fault tolerance mechanism and fault recovery process of the cluster.

[0092] Among them, process suspension refers to deliberately pausing or suspending one or more running processes to simulate process-level failures or abnormal conditions. It is mainly used to test the cluster's fault tolerance, stability, and failure recovery mechanism when facing process failures. Process suspension testing is an important part of cluster testing. It helps to discover and solve potential stability problems in advance and improve the reliability and robustness of the cluster.

[0093] For example, when the cluster to be tested needs to be tested, the terminal will obtain resource transaction data for cluster testing and node control methods of the cluster to be tested in multiple test scenarios. The resource transaction data can specifically be online transaction data generated by the smart financial technology component in the past. Using real online transaction data to conduct online testing on the cluster to be tested can improve the accuracy of the test results.

[0094] In specific applications, the node control methods under multiple test scenarios can be configured according to the actual application scenarios. The node control method under each test scenario is different, and when configuring the node control method, the basic test scenario can be expanded in combination with the cluster layout information of the cluster to be tested, and the node control method of the cluster to be tested under multiple test scenarios can be obtained. For example, the basic test scenarios include application shutdown, network card failure, process suspension, etc. The terminal can expand the basic test scenarios in combination with the cluster layout information of the cluster to be tested, and obtain node control methods under multiple application shutdown scenarios, node control methods under multiple network card failure scenarios, and node control methods under multiple process suspension scenarios.

[0095] Among them, cluster layout information refers to the collection of key information about the architecture, network configuration, node distribution, resource allocation, etc. of the cluster to be tested when building and deploying the cluster to be tested. This information is crucial for understanding and managing the cluster to be tested. Cluster architecture is used to describe how the cluster to be tested is composed, including the number of nodes, decisions (such as master nodes and slave nodes), and the communication methods between them. Network configuration refers to the network layout of the cluster to be tested, including node networks, container networks, and service networks, as well as how to assign network addresses to these networks and ensure that there are no conflicts between network segments. Node distribution refers to the address location allocation of cluster nodes in the cluster to be tested, which may be distributed in one computer room, multiple computer rooms, or different geographical locations.

[0096] In a specific application, based on the cluster layout information, it is determined that the cluster to be tested includes three application servers, and two nodes A1 and A2 are arranged on each application server, which are a master node and a slave node respectively, and three masters and three slaves are arranged crosswise. For example, the stop application is expanded, and the node control methods obtained in multiple stop application scenarios can be: stop the master node on an application server, stop the two nodes on an application server where the master node is located (both nodes are master nodes), stop the two nodes on an application server where the master node is located (the two nodes are master nodes and slave nodes respectively), stop the corresponding set of master and slave nodes on the two application servers, and stop the three master nodes and three slave nodes of the cluster to be tested. The node control method in the network card down scenario can be specifically to down the master node network card on an application server, and the node control method in the process suspension scenario can be specifically to suspend the master node operation process.

[0097] Step 304, for each test scenario, when starting the online service, obtain the cluster initial state of the cluster to be tested, and determine the cluster fault injection strategy in the test scenario according to the cluster initial state, resource transaction data and the node control mode in the test scenario.

[0098] The cluster initial state refers to the state of the cluster to be tested before the online fault simulation is performed when the online service is started. It is understandable that, because the cluster to be tested will be subjected to online fault simulation in each test scenario, and the online fault simulation may change the roles of the cluster nodes in the cluster to be tested, therefore, for each test scenario, when the online service is started, it is necessary to first obtain the cluster initial state of the cluster to be tested, so as to determine the cluster nodes that serve as master nodes and the cluster nodes that serve as slave nodes in the cluster to be tested according to the cluster initial state. In this embodiment, changing the roles of the cluster nodes in the cluster to be tested may specifically refer to changing the cluster nodes from master nodes to slave nodes, or changing from slave nodes to master nodes.

[0099] The cluster fault injection strategy refers to a method of testing the fault tolerance, stability and reliability of the cluster under test by simulating fault conditions. The purpose of this strategy is to ensure that the cluster under test can maintain normal operation or recover quickly when facing various faults that may occur in the real world. For example, the cluster fault injection strategy can specifically be at least one of application shutdown processing, network card downtime processing, and process suspension processing for at least one cluster node in the cluster under test.

[0100] Exemplarily, for each test scenario, the terminal will first start the online service. When the online service is started, the cluster initial state of the cluster to be tested is obtained, and the transaction processing node that processes resource transaction data in the cluster to be tested is determined. Based on the cluster initial state, transaction processing node, and node control method in the test scenario, the cluster fault injection strategy in the test scenario is determined.

[0101] Step 306 , based on the cluster fault injection strategy in the test scenario and the resource transaction data, an online fault simulation is performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario.

[0102] Among them, online fault simulation refers to simulating the occurrence of faults in the network environment to test the stability, reliability and fault recovery capability of the cluster under test. Through online fault simulation, the performance of the cluster under test in actual operation can be detected. Cluster performance indicator data refers to key data that measures the operating status and performance of the cluster under test. For example, cluster performance indicator data can specifically include average transaction response time, number of transactions per second, transaction processing accuracy, server resource usage data, node switching time, etc.

[0103] The average transaction response time refers to the average speed of processing transactions within a certain period of time, usually in seconds or milliseconds. It measures the total time from the user initiating a request to receiving a server response. The number of transactions per second refers to the number of transactions that can be processed in a unit of time (usually seconds). A transaction refers to the process in which the client sends a request to the server and the server responds. In this embodiment, the number of transactions per second can specifically refer to the number of transactions that can be processed in a unit of time.

[0104] Among them, the transaction processing accuracy rate refers to the probability of successful transaction processing. Server resource usage data is used to describe the resource usage of the server providing online services. The server providing online services here includes the cluster to be tested, the online server connected to the cluster to be tested, the database server, etc. The server resource usage data may specifically include the server's central processing unit usage rate, memory usage rate, etc. Node switching time refers to the time required to switch the slave node corresponding to the master node in the cluster to the master node when a master node in the cluster becomes unavailable for some reason.

[0105] Exemplarily, when determining the cluster fault injection strategy in the test scenario, the terminal will perform online fault simulation on the cluster to be tested based on the cluster fault injection strategy in the test scenario and resource transaction data to obtain cluster performance indicator data of the cluster to be tested in the test scenario.

[0106] In specific applications, the terminal can set a fault for the cluster to be tested based on the cluster fault injection strategy in the test scenario, and then, after the fault setting is completed, continue to run online services based on resource transaction data to obtain cluster performance indicator data of the cluster to be tested in the test scenario.

[0107] Step 308: Evaluate the cluster to be tested based on the cluster performance indicator data to obtain a test result of the test scenario.

[0108] Exemplarily, the terminal evaluates the cluster to be tested according to the cluster performance indicator data to obtain a test result of the test scenario.

[0109] Step 310: obtaining a test result of the cluster to be tested based on the test results of each of the multiple test scenarios.

[0110] Exemplarily, after obtaining the test results of each of the multiple test scenarios, the terminal aggregates the test results of each of the multiple test scenarios to obtain the test result of the cluster to be tested, so as to achieve a comprehensive test of the high availability of the cluster to be tested.

[0111] The above-mentioned cluster testing method, based on obtaining resource transaction data used for cluster testing and the node control method of the cluster to be tested in multiple test scenarios, can obtain the cluster initial state of the cluster to be tested for each test scenario when starting the online service, and can use the cluster initial state, resource transaction data and the node control method in the test scenario to determine the cluster fault injection strategy in the test scenario. Then, based on the cluster fault injection strategy in the test scenario and the resource transaction data, online fault simulation can be performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario. According to the cluster performance indicator data, the cluster to be tested is evaluated to obtain the test result of the test scenario, so that the test results of multiple test scenarios can be summarized to obtain the test result of the cluster to be tested. In the whole process, the cluster fault injection strategy in each test scenario can be used to formulate cluster fault injection strategies in multiple test scenarios by using the cluster initial state, resource transaction data and node control method. A comprehensive test of the high availability of the cluster can be achieved by using the cluster fault injection strategy in each test scenario.

[0112] In an exemplary embodiment, Figure 4 As shown, the steps include determining the cluster fault injection strategy under the test scenario according to the cluster initial state, resource transaction data and the node control mode under the test scenario, including steps 402 to 406. Among them:

[0113] Step 402: Determine a transaction processing node in the cluster to be tested that processes the resource transaction data based on the transaction type to which the resource transaction data belongs.

[0114] Step 404: Determine the master node for processing resource transaction data in the cluster to be tested according to the cluster initial state and the transaction processing node.

[0115] Step 406: Determine a cluster fault injection strategy in the test scenario according to the master node and the node control mode in the test scenario.

[0116] The transaction type to which the resource transaction data belongs refers to the classification of transactions according to the nature, purpose, form or processing method of the transaction in financial, commercial or other transaction scenarios. Different transaction types may involve different processes, rules and legal requirements. In this embodiment, different transaction types involve different cluster nodes, that is, different cluster nodes can be used to process resource transaction data belonging to different transaction types. The transaction processing node is the cluster node that processes the resource transaction data in the cluster to be tested.

[0117] Among them, the master node refers to the node in the cluster that assumes specific management responsibilities and control functions. The specific role and function of the master node depends on the type and purpose of the cluster. In this embodiment, the master node that processes resource transaction data refers to the master node whose function is to process resource transaction data in the cluster to be tested. In the cluster, there are slave nodes corresponding to the master node. The slave node is also called auxiliary node, backup node and working node. It is relative to the master node. It can have the same hardware and software configuration as the master node to ensure that it can quickly take over the responsibilities of the master node when needed.

[0118] Exemplarily, based on the transaction type to which the resource transaction data belongs, the terminal can determine the slot for processing the resource transaction data in the cluster to be tested, and then determine the transaction processing node for processing the resource transaction data according to the slot for processing the resource transaction data. On this basis, the terminal can determine the master node for processing the resource transaction data in the cluster to be tested according to the cluster initial state and the transaction processing node, and then determine the cluster fault injection strategy in the test scenario according to the master node and the node control method in the test scenario.

[0119] In specific applications, the number of transaction processing nodes is usually two, one is the master node and the other is the slave node. Based on the determination of the transaction processing node, the terminal can determine the master-slave relationship between the two transaction processing nodes in combination with the initial state of the cluster, and then determine the master node for processing resource transaction data based on the master-slave relationship. Based on the determination of the master node, the terminal can determine the cluster nodes that need to be controlled in the test scenario and the control method for the cluster nodes based on the master node and the node control method in the test scenario, and generate a cluster fault injection strategy in the test scenario based on the cluster nodes that need to be controlled and the control method for the cluster nodes.

[0120] In this embodiment, based on the transaction type to which the resource transaction data belongs, the transaction processing node in the cluster to be tested that can process the resource transaction data can be determined, and then the master node for processing the resource transaction data in the cluster to be tested can be determined based on the cluster initial state and the transaction processing node, so that the cluster fault injection strategy in the test scenario can be determined by using the master node and the node control method in the test scenario.

[0121] In an exemplary embodiment, based on the cluster fault injection strategy in the test scenario and the resource transaction data, an online fault simulation is performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario, including:

[0122] Based on the cluster fault injection strategy in the test scenario, set up faults for the cluster to be tested;

[0123] After the fault setting is completed, the online service is continuously run based on the resource transaction data to obtain the cluster performance indicator data of the cluster under test in the test scenario.

[0124] For example, based on the cluster fault injection strategy in the test scenario, the terminal can determine the cluster nodes that need to be controlled and the control method for the cluster nodes, and then can set the fault of the cluster to be tested by controlling the cluster nodes that need to be controlled according to the control method for the cluster nodes. After the fault setting is completed, the terminal will continue to run the online service based on the resource transaction data, obtain the cluster performance indicator data of the cluster to be tested in the test scenario, and use the cluster performance indicator data to evaluate the high availability of the cluster to be tested.

[0125] In a specific application, when controlling the cluster nodes to be controlled according to the control method for the cluster nodes, it means setting the cluster nodes to be controlled to a fault state according to the control method for the cluster nodes. The fault state here can specifically be stopping the application of the cluster node to be controlled, shutting down the network card of the cluster node to be controlled, suspending the process of the cluster node to be controlled, etc. In this way, fault setting can be implemented for the cluster to be tested, thereby verifying the recovery capability of the cluster to be tested to faults.

[0126] In a specific application, after the fault setting is completed, the terminal will initiate an online service request to the online server based on the resource transaction data to continuously run the online service, and during the operation of the online service, obtain information related to the cluster to be tested, information related to online transaction processing, and information related to resource usage, so as to obtain cluster performance indicator data of the cluster to be tested in the test scenario based on the information related to the cluster to be tested, information related to online transaction processing, and information related to resource usage.

[0127] In this embodiment, based on the cluster fault injection strategy in the test scenario, a fault setting is performed on the cluster to be tested, which can realize the fault simulation of the cluster to be tested in the test scenario. After the fault setting is completed, the online service is continuously run based on the resource transaction data. By continuously running the online service, the recovery capability of the cluster to be tested to faults can be verified, so that the high availability of the cluster to be tested can be evaluated by obtaining the cluster performance indicator data of the cluster to be tested in the test scenario.

[0128] In an exemplary embodiment, after the fault setting is completed, the online service is continuously run based on the resource transaction data to obtain cluster performance indicator data of the cluster to be tested in the test scenario, including:

[0129] After the fault setting is completed, the online service is continuously operated based on the resource transaction data, and during the operation of the online service, the cluster status information, transaction online processing results and server resource usage data are obtained;

[0130] Until the online service runs to the end, the cluster performance indicator data of the cluster to be tested in the test scenario is obtained based on the obtained cluster status information, transaction online processing results and server resource usage data.

[0131] The cluster status information refers to information describing the status of the cluster to be tested during the online fault simulation. For example, the cluster status information may specifically refer to information describing the status of the cluster nodes in the cluster to be tested during the online fault simulation. For example, the information on the status of the cluster nodes may include the role change of the nodes before and after the online fault simulation, such as the master node becoming a slave node or the slave node becoming a master node. For another example, for example, the information on the status of the cluster nodes may include the running status of the cluster nodes before and after the online fault simulation, such as whether they are online or offline.

[0132] Among them, the transaction online processing result refers to the processing result obtained by processing resource transaction data through online services, including the processing time of resource transaction data, the number of processed transactions, and the processing accuracy rate. Server resource usage data is used to describe the resource usage of the server providing online services. The server providing online services here includes the cluster to be tested, the online server connected to the cluster to be tested, the database server, etc. The server resource usage data can specifically include the server's central processing unit usage rate, memory usage rate, etc.

[0133] Exemplarily, after the fault setting is completed, the terminal will continue to initiate online service requests to the online server based on the resource transaction data to continuously run the online service, and during the operation of the online service, obtain cluster status information, transaction online processing results and server resource usage data until the continuous operation time of the online service reaches the preset operation time, and control the online service to stop running, and calculate the cluster performance indicator data of the cluster to be tested in the test scenario based on the obtained cluster status information, transaction online processing results, server resource usage data and cluster performance indicators in the test scenario.

[0134] In a specific application, the preset running time can be configured according to the actual application scenario. For example, the preset running time can be ten minutes, and the terminal will control the online service to stop running after the online service has been running for ten minutes. In a specific application, the way to control the online service to stop running can be to stop initiating online service requests to the online server, that is, when the terminal no longer initiates online service requests to the online server, the online service stops running.

[0135] In specific applications, cluster performance indicators in different test scenarios are not exactly the same. Therefore, the terminal needs to combine the cluster performance indicators in the test scenario to calculate the cluster performance indicator data of the cluster to be tested in the test scenario. In a specific application, the cluster performance indicators in the test scenario can include the average transaction response time after the fault setting is completed, the number of transactions per second, and the transaction processing accuracy, which can be calculated based on the transaction online processing results. In addition, the cluster performance indicators in the test scenario can also include server resource usage data, node switching time, etc., among which the node switching time can be obtained through the cluster status information.

[0136] In a specific application, after the fault setting is completed, the cluster to be tested will enter the fault period. After entering the fault period, if the cluster to be tested can automatically resume normal operation, then when calculating the cluster performance indicator data, in addition to the transaction processing accuracy and node switching time, the terminal needs to calculate the cluster performance indicator data for the fault period and the fault recovery period respectively. For example, the cluster performance indicator data of the cluster to be tested in the test scenario include: transaction processing accuracy, node switching time, average transaction response time during the fault period, number of transactions per second, server resource usage data, and average transaction response time, number of transactions per second, server resource usage data during the fault recovery period. If the cluster to be tested cannot automatically resume normal operation, such as when the test scenario is to stop all nodes of the cluster to be tested, the terminal only needs to calculate the cluster performance indicator data during the fault period.

[0137] In a specific application, after the control online service stops running, the terminal will release the fault setting for the cluster to be tested, that is, restore the cluster nodes that have been set to a faulty state, and check the cluster status after recovery. It is understandable that since the roles of the cluster nodes in the cluster to be tested may change after the fault setting is performed on the cluster to be tested, the cluster status after recovery may be different from the initial state of the cluster. By comparing the initial state of the cluster, the cluster status information, and the cluster status after recovery, it is possible to judge whether the cluster status in the test scenario meets expectations, and then the judgment result can be used to evaluate the high availability of the cluster to be tested, so as to achieve a comprehensive evaluation of the cluster to be tested.

[0138] In this embodiment, after the fault setting is completed, the online service is continuously run based on the resource transaction data, so that the cluster to be tested can be monitored during the operation of the online service, and cluster status information, transaction online processing results and server resource usage data can be obtained. Therefore, when the online service ends, the cluster performance indicator data of the cluster to be tested in the test scenario can be determined based on the obtained cluster status information, transaction online processing results and server resource usage data.

[0139] In an exemplary embodiment, the cluster to be tested is evaluated according to the cluster performance indicator data to obtain the test result of the test scenario, including:

[0140] Obtain cluster performance indicator requirements for test scenarios;

[0141] By comparing the cluster performance indicator requirements and cluster performance indicator data, the cluster to be tested is evaluated to obtain the test results of the test scenario.

[0142] The cluster performance index requirements refer to a series of performance standards and goals that need to be achieved in the test scenario, and are key parameters for measuring whether the cluster under test is operating normally and whether it meets business requirements. For example, the cluster performance index requirements may include switching effectiveness index requirements, mean failure recovery time index requirements, and failed transaction processing capability recovery level requirements.

[0143] Among them, the switching effectiveness index requirement is used to evaluate the system's ability to recover from a faulty state to a normal operating state when a fault occurs. In this embodiment, it may specifically refer to the master-slave switching effectiveness requirement after the tested cluster fails, the switching effectiveness requirement between the tested cluster and the database, etc. (that is, after all the tested clusters are abnormal, the transaction can be restored to proceed normally). Among them, the master-slave switching effectiveness requirement may specifically refer to the node switching time being less than the first switching time threshold. The switching effectiveness requirement between the tested cluster and the database may specifically refer to the switching time to the database being less than the second switching time threshold.

[0144] The mean failure recovery time indicator is an indicator that measures the average time required to repair a system or device after a failure. The mean failure recovery time indicator requirement can specifically be that there is no error in the processing of resource transaction data. The failure transaction processing capability recovery level refers to the system's ability to recover the ability to process transactions after a failure occurs. In this embodiment, that is, after a failure occurs in the cluster to be tested, the system is able to recover the ability to process resource transaction data.

[0145] Exemplarily, the terminal obtains the cluster performance indicator requirements of the test scenario, evaluates the cluster to be tested by comparing the cluster performance indicator requirements with the cluster performance indicator data, and obtains the test result of the test scenario.

[0146] In a specific application, the cluster performance index requirement includes multiple index requirements. For each index requirement, the terminal obtains the index data associated with the index requirement from the cluster performance index data, compares the index requirement and the index data associated with the index requirement, obtains the test result of the index requirement, and summarizes the test results of the multiple index requirements to obtain the test result of the test scenario. The test result of the index requirement can specifically be that the index requirement is met or not met.

[0147] In a specific application, for example, the test results of the test scenario specifically include: whether the switching effectiveness index requirements are met, whether the mean failure recovery time index requirements are met, and whether the failure transaction processing capability recovery level requirements are met. Among them, whether the mean failure recovery time index requirements and the failure transaction processing capability recovery level requirements are met can be determined by whether there is no error in the processing of resource transaction data. If there is no error in the processing of resource transaction data, the mean failure recovery time index requirements and the failure transaction processing capability recovery level requirements are met.

[0148] In this embodiment, based on obtaining the cluster performance indicator requirements of the test scenario, the cluster to be tested can be accurately evaluated by comparing the cluster performance indicator requirements and the cluster performance indicator data, and the test results of the test scenario can be obtained. The test results of the test scenario can then be used to achieve a comprehensive test of the high availability of the cluster to be tested.

[0149] In an exemplary embodiment, the cluster testing method further includes:

[0150] Based on the resource transaction data, an online service request is initiated to an online server connected to the cluster to be tested to start the online service.

[0151] The online server refers to a server that continuously runs and provides services in a network environment, and can respond to client requests and perform specific tasks or services. In this embodiment, the online server can respond to the online service request of the terminal and forward the online service request to the cluster to be tested for processing.

[0152] Exemplarily, the terminal will initiate an online service request to the online server connected to the cluster to be tested according to the resource transaction data to start the online service. After receiving the online service request, the online server will forward the online service request to the cluster to be tested to start the online test.

[0153] In this embodiment, based on resource transaction data, an online service request is initiated to an online server connected to the cluster to be tested, so that the online service can be started. Then, when the online service is started, the high availability of the cluster to be tested can be fully tested.

[0154] In an exemplary embodiment, Figure 5 As shown, a cluster testing method of the present application is described through a flow chart, and the cluster testing method includes steps 502 to 522. Among them:

[0155] Step 502, obtaining resource transaction data for cluster testing and node control methods of the cluster to be tested in multiple test scenarios;

[0156] Step 504, for each test scenario, based on the resource transaction data, an online service request is initiated to an online server connected to the cluster to be tested, so as to start the online service;

[0157] Step 506, when the online service is started, the cluster initial state of the cluster to be tested is obtained, and based on the transaction type to which the resource transaction data belongs, a transaction processing node for processing the resource transaction data in the cluster to be tested is determined;

[0158] Step 508, determining the master node for processing resource transaction data in the cluster to be tested according to the cluster initial state and the transaction processing node;

[0159] Step 510, determining a cluster fault injection strategy in the test scenario according to the master node and the node control mode in the test scenario;

[0160] Step 512, based on the cluster fault injection strategy in the test scenario, perform fault settings on the cluster to be tested;

[0161] Step 514, after the fault setting is completed, the online service is continuously operated based on the resource transaction data, and during the operation of the online service, cluster status information, transaction online processing results and server resource usage data are obtained;

[0162] Step 516, until the online service operation is completed, the cluster performance indicator data of the cluster to be tested in the test scenario is obtained according to the acquired cluster status information, transaction online processing results and server resource usage data;

[0163] Step 518, obtaining cluster performance indicator requirements for the test scenario;

[0164] Step 520, evaluating the cluster to be tested by comparing the cluster performance indicator requirements with the cluster performance indicator data, and obtaining a test result of the test scenario;

[0165] Step 522: Based on the test results of each of the multiple test scenarios, the test result of the cluster to be tested is obtained.

[0166] In an exemplary embodiment, the cluster testing method of the present application is applied to a Redis cluster, and the Redis cluster to be tested includes three application servers (0.0.0.1 / 0.0.0.2 / 0.0.0.3), and two nodes a1 and a2 are arranged on each application server, which are a master node and a slave node respectively, and three masters and three slaves are arranged crosswise, and the transaction types involved in processing resource transaction data involving the Redis cluster include transaction type A and transaction type B. For example, the tests under multiple test scenarios of the present application are described.

[0167] In specific applications, the overall structure diagram of the Redis cluster test can be as follows Figure 6 As shown in the figure, the intelligent financial technology component has 6 deployment units, including online, high frequency, end of day, revaluation high frequency, database, and Redis cluster. The terminal uses JMeter as a simulation stress tool to initiate online services (run online transactions) to the intelligent financial technology component. During the test, the test monitoring tool is used to obtain the test data generated during the test, including the initial state of the cluster, cluster state information, transaction online processing results, and server resource usage data. Among them, the test monitoring tool can be configured according to the actual application scenario, specifically Grafana (an open source data visualization and monitoring platform), which allows users to monitor and analyze data by creating dynamic dashboards.

[0168] In specific applications, the overall resource configuration of the test environment can be configured according to the actual application scenario. For example, the overall resource configuration of the test environment can be: Test environment: 3* (application server cluster, 4C16G) + 3*P3 (high-frequency batch server cluster, 4C16G) + 3*P1 (high-frequency batch server cluster, 2C8G) + 2 * (DB database cluster, 8C32G) + 3* (end-of-day batch server cluster, 2C8G) + 3* (redis server cluster, 2C8G). The test environment and production environment AP both use X86 virtual machines, and the basic software versions such as RHEL linux and weblogic are consistent. The database uses LINUX virtual machines and Oracle RAC deployment mode.

[0169] In a specific application, before testing, the Redis cluster information can be: 3 master nodes: 0.0.0.2:a1, 0.0.0.3:a2, 0.0.0.3:a1, 3 standby nodes (i.e. slave nodes): 0.0.0.1:a2, 0.0.0.2:a2, 0.0.0.1:a1. Among them, the slots of the Redis cluster corresponding to the online transaction of transaction type A are 01 and 02, and the two nodes of the corresponding Redis cluster are 0.0.0.2:a1 and 0.0.0.1:a2. The slot of the Redis cluster corresponding to the online transaction of transaction type B is 03, and the two nodes of the corresponding Redis cluster are 0.0.0.1:a1 and 0.0.0.3:a2.

[0170] The following are examples of several test scenarios:

[0171] Test scenario 1: Stop the master node application of a Redis server:

[0172] Test description: Take the transaction type A to which resource transaction data belongs, and the need to process resource transaction data as an example. The corresponding Redis server slots are 01 and 02, and the corresponding two Redis servers are 0.0.0.2:a1 (primary) and 0.0.0.1:a2 (secondary). By stopping the application of the primary node alone, the application can be stopped to test and verify.

[0173] The specific application stop test process can be as follows: Figure 7 As shown, it includes steps 702 to 710. Among them:

[0174] Step 702, start the online service and run it for ten minutes;

[0175] Step 704, check the initial state of the Redis cluster before stopping the application;

[0176] Step 706: stop the application of the master node 0.0.0.2:a1 corresponding to the Redis server, and the standby node 0.0.0.1:a2 automatically switches to the master node. The switching takes 17 seconds.

[0177] Step 708: The online service continues to run for ten minutes. During the switching process, online transactions are not affected (the total number of transactions is 54101, and 0 errors are reported);

[0178] Step 710: Restore the 0.0.0.2:a1 application as a standby node without switching back.

[0179] Among them, the checkpoints mainly include the following aspects: First, the initial state of the Redis cluster: 3 master nodes: 0.0.0.2:a1, 0.0.0.3:a2, 0.0.0.3:a1; 3 backup nodes: 0.0.0.1:a2, 0.0.0.2:a2, 0.0.0.1:a1. Second, the application of the master node 0.0.0.2:a1 corresponding to the Redis server was stopped at 10:24:12. Third, the backup node 0.0.0.1:a2 was started as the master node at 10:24:29. Fourth, the cluster status after switching: 3 master machines: 0.0.0.1:a2, 0.0.0.3:a2, 0.0.0.3:a1, 2 standby machines: 0.0.0.2:a2, 0.0.0.1:a1, one abnormal machine: 0.0.0.2:a1, the status is failed, and the connection with the Redis cluster is disconnected. Fifth, restore the application of 0.0.0.2:a1 without switching back. The redis status after recovery: 3 master machines: 0.0.0.1:a2, 0.0.0.3:a2, 0.0.0.3:a1, 3 standby machines: 0.0.0.2:a2, 0.0.0.1:a1; 0.0.0.2:a1. Sixth, transaction status: when the master node is stopped, the overall average response time of the transaction and TPS (Transactions Per Second) fluctuate briefly, and the Redis standby node takes over and becomes the master node, and then returns to normal. The rest of the process is stable. Seventh, resource usage: when Redis stops the master node, the CPU (Central Processing Unit) usage of the online server fluctuates slightly for a short period of time (the Redis switch causes the affected transaction response time to rise briefly), and the memory is stable without any increase. The CPU and memory resources of the Redis server and database server are stable without any increase during the whole process.

[0180] By analyzing the checkpoints, we can obtain the test results: the master node stops, and the backup node takes over as the new master node after 17 seconds, meeting the switching effectiveness index; there are no errors in the transaction, meeting the average failure recovery time index; the failure transaction processing capacity recovery level is 100%.

[0181] It should be noted that the application stopping test scenario can also be specifically: stopping two nodes on an application server where the master node is located (both nodes are master nodes), stopping two nodes on an application server where the master node is located (the two nodes are respectively a master node and a slave node), stopping a corresponding group of master-slave nodes on the two application servers, stopping the three master nodes and three slave nodes of the cluster to be tested, etc. The test process, checkpoints and test results involved are similar to those of test scenario one, and no further explanation is given here in this embodiment.

[0182] Test scenario 2: Process hang:

[0183] Test description: Take the transaction type B to which resource transaction data belongs, and the need to process resource transaction data as an example. The corresponding slot of the Redis server is 03, and the corresponding two Redis servers are 0.0.0.1:a1 (primary) and 0.0.0.3:a2 (secondary). By suspending the operation process on the primary node, the validity of the process suspension is tested and verified.

[0184] The specific process of the suspension test can be as follows: Figure 8 As shown, it includes steps 802 to 810. Among them:

[0185] Step 802, start the online service and run it for ten minutes;

[0186] Step 804, check the initial state of the Redis cluster before the process is suspended;

[0187] Step 806: Suspend the process of the master node 0.0.0.1:a1 corresponding to Redis, and automatically switch the backup node 0.0.0.3:a2 to the master node, which takes 20 seconds.

[0188] Step 808: The online service continues to run for ten minutes. During the switching process, online transactions are not affected (total number of transactions is 46,177, and 0 errors are reported);

[0189] Step 810, restore the process of 0.0.0.1:a1 without switching back.

[0190] Among them, the checkpoints mainly include the following aspects: First, the initial state of the Redis cluster: 3 master nodes: 0.0.0.1:a2, 0.0.0.1:a1, 0.0.0.3:a1, 3 backup nodes: 0.0.0.2:a1, 0.0.0.2:a2, 0.0.0.3:a2. Second, the process of the master node 0.0.0.1:a1 corresponding to Redis is suspended at 15:17:56. Third, the backup node 0.0.0.3:a2 automatically switches to the master node at 15:18:16. Fourth, the cluster status after switching: 3 master machines: 0.0.0.1:a2, 0.0.0.3:a1, 0.0.0.3:a2, 2 standby machines: 0.0.0.2:a1, 0.0.0.2:a2, one abnormal machine: 0.0.0.1:a1, the status is failed, and the connection with the Redis cluster is disconnected. Fifth, restore the process of 0.0.0.1:a1, do not switch back, and check the redis status after recovery: 3 master machines: 0.0.0.1:a2, 0.0.0.3:a1, 0.0.0.3:a2, 3 standby machines: 0.0.0.2:a1, 0.0.0.2:a2, 0.0.0.1:a1. Sixth, transaction status: from the time when the master node process is suspended to the time when the standby node takes over, the overall average response time of the transaction surges and the TPS drops briefly (slightly). After the Redis standby master node takes over, it recovers smoothly, and the overall average response time of the transaction and the rest of the TPS process are stable. Seventh, resource usage: when the Redis master node is suspended, the CPU usage of the online server and the Redis server fluctuates slightly for a short time (the redis switch causes the response time of the affected transactions to surge for a short time), and the memory is stable without increasing; the CPU and memory resources of the database server are stable without increasing throughout the process; in other time periods, the CPU usage of each server is stable, and the memory is stable without increasing.

[0191] By analyzing the checkpoints, we can obtain the test results: the master node stops, and the backup node takes over as the new master node after 20 seconds, meeting the switching effectiveness index; there are no errors in the transaction, meeting the average failure recovery time index; the failure transaction processing capacity recovery level is 100%.

[0192] It should be noted that the process suspension test scenario in this embodiment is only an example. In addition to what is described in this embodiment, there are other test scenarios, and the test process, checkpoints and test results of other test scenarios are similar to those of the test scenario in this embodiment. No further explanation is given here in this embodiment.

[0193] Test scenario three: The master node network card is down:

[0194] Test description: Take the transaction type B to which resource transaction data belongs, and the need to process resource transaction data as an example. The corresponding slot of the Redis server is 03, and the corresponding two Redis servers are 0.0.0.1:a1 (backup) and 0.0.0.3:a2 (master) (follow test scenario 2, at this time the role of the node has changed, the master node in test scenario 2 becomes the backup node, and the backup node becomes the master node). Shut down the network card of the master node to verify the effectiveness of Redis shutting down the network card.

[0195] The specific process of shutting down the master node network card can be as follows: Fig. 9 As shown, it includes steps 902 to 910. Among them:

[0196] Step 902, start the online service and run it for ten minutes;

[0197] Step 904, check the initial state of the Redis cluster before the network card fails;

[0198] Step 906: Switch to the network segment 0.0.0.3 of the redis application where the master node a2 corresponding to the transaction is located. The switching takes 24 seconds.

[0199] Step 908: The online service continues to run for ten minutes. During the process of the main node network card being shut down, the online transactions are not affected (a total of 60,391 transactions are executed, and 0 transactions are reported as errors);

[0200] Step 910, restore the network segment, Redis status is restored, and it takes 4 seconds to restore the network segment.

[0201] Among them, the checkpoints mainly include the following aspects: First, the initial state of the Redis cluster: 3 master nodes: 0.0.0.1:a2, 0.0.0.3:a2, 0.0.0.2:a2, 3 backup nodes: 0.0.0.2:a1, 0.0.0.1:a1, 0.0.0.3:a1. Second, the network segment of the Redis application 0.0.0.3 of the master node corresponding to the transaction was down, at 15:49:39. Third, the corresponding backup node 0.0.0.1:a1 automatically switched to the master node, at 15:50:03. Fourth, the redis status after the network card is down: 3 masters: 0.0.0.1:a2, 0.0.0.2:a2, 0.0.0.1:a1, 1 backup: 0.0.0.2:a1, 2 exceptions: 0.0.0.3:a1 and 0.0.0.3:a2, the status is failed, and the connection with the Redis cluster is disconnected. Fifth, the redis status after restoring the 0.0.0.3 network card and restoring the 0.0.0.3 network segment: 3 masters: 0.0.0.1:a2, 0.0.0.2:a2, 0.0.0.1:a1, 3 backups: 0.0.0.2:a1, 0.0.0.3:a1, 0.0.0.3:a2. Sixth, transaction status: during the process from the main node network failure to the standby node taking over, the overall average response time of the transaction surged and TPS dropped briefly (slightly), and the Redis standby master node recovered smoothly after taking over; the overall average response time of the transaction and TPS were stable at other times during the entire process. Seventh, resource usage: when the Redis main node network failed, the CPU usage of the online server had a short-term small fluctuation (the redis switch caused the affected transaction response time to surge briefly), and the memory was stable without increasing. The CPU of a server affected by Redis restored monitoring data only after the network card failed, and the whole process fluctuated slightly. The CPU and memory resources of the database server were stable and did not increase throughout the process. In other time periods, the CPU usage of each server was stable, and the memory was stable without increasing.

[0202] By analyzing the checkpoints, we can obtain the following test results: after the network card of a group of master nodes fails, the backup node takes over as the new master node after 24 seconds, meeting the switching effectiveness index; within a short period of time, the response time and TPS of transactions whose slots fall on this group of nodes fluctuate; there are no errors in transactions, meeting the average failure recovery time index; the failed transaction processing capacity recovery level is 100%, verifying the effectiveness of fault isolation.

[0203] It should be noted that the network card test scenario of the down master node in this embodiment is only an example. In addition to what is described in this embodiment, there are other test scenarios, and the test process, checkpoints and test results of other test scenarios are similar to those of the test scenario in this embodiment. No further explanation is given here in this embodiment.

[0204] The cluster testing method in this embodiment adopts the mode of the entire component, selects resource transaction data that interacts with or does not interact with Redis, applies pressure with load, verifies transactions with Redis involving different slots through monitoring and testing of full-process business data in actual business scenarios, and respectively stops applications, crashes network cards, suspends processes, and stops applications on the master nodes of different groups / a group of master nodes together, and stops and starts all three Redis applications. The control variable method covers a variety of test scenario combinations, functional and performance tests, and conducts comprehensive and complete testing and verification. The effectiveness of Redis master-slave switching, the effectiveness of one-click start and stop of Redis services, the effectiveness of switching between Redis and the database, and the effectiveness of Redis cache data initialization were verified.

[0205] It should be noted that the cluster testing method of the present application uses resource transaction data as the background transaction pressure. After the pressure is stable, multiple groups of abnormal operations are performed on the Redis cluster in sequence: stopping the application, crashing the network card, suspending the process, and monitoring the transaction status of online transactions that have logical interactions with Redis. The TPS, average response time, server resource usage, and database resource usage of the overall transaction can be used to verify the effectiveness of the master-slave switching after the redis exception, the average failure recovery time meets the index requirements, the recovery level of the failed transaction processing capacity reaches 100%, the effectiveness of the switching between Redis and the database (the recovery transaction can proceed normally after all redis cluster exceptions), and the redis application is stopped or started within the effective time. Cluster performance index requirements such as the requirements can be verified to achieve a comprehensive test of the high availability of the cluster.

[0206] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0207] Based on the same inventive concept, the embodiment of the present application also provides a cluster test device for implementing the cluster test method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more cluster test device embodiments provided below can refer to the above limitations on the cluster test method, and will not be repeated here.

[0208] In an exemplary embodiment, Fig.10 As shown, a cluster testing device is provided, including: a data acquisition module 1002, a strategy formulation module 1004, a testing module 1006, an evaluation module 1008 and a summary module 1010, wherein:

[0209] The data acquisition module 1002 is used to acquire resource transaction data for cluster testing and node control methods of the cluster to be tested in multiple test scenarios;

[0210] The strategy formulation module 1004 is used to obtain the cluster initial state of the cluster to be tested for each test scenario when the online service is started, and determine the cluster fault injection strategy in the test scenario according to the cluster initial state, resource transaction data and the node control mode in the test scenario;

[0211] The test module 1006 is used to perform online fault simulation on the cluster to be tested based on the cluster fault injection strategy in the test scenario and the resource transaction data, and obtain cluster performance indicator data of the cluster to be tested in the test scenario;

[0212] Evaluation module 1008, used to evaluate the cluster to be tested according to the cluster performance indicator data, and obtain the test result of the test scenario;

[0213] The aggregation module 1010 is used to obtain the test result of the cluster to be tested based on the test results of each of the multiple test scenarios.

[0214] The above-mentioned cluster testing device, based on obtaining resource transaction data for cluster testing and node control methods of the cluster to be tested in multiple test scenarios, can obtain the cluster initial state of the cluster to be tested for each test scenario when starting the online service, and can use the cluster initial state, resource transaction data and the node control method in the test scenario to determine the cluster fault injection strategy in the test scenario. Then, based on the cluster fault injection strategy in the test scenario and the resource transaction data, online fault simulation can be performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario. According to the cluster performance indicator data, the cluster to be tested is evaluated to obtain the test results of the test scenario, so that the test results of multiple test scenarios can be summarized to obtain the test results of the cluster to be tested. In the whole process, the cluster fault injection strategy in each test scenario can be used to formulate cluster fault injection strategies in multiple test scenarios by using the cluster initial state, resource transaction data and node control methods. A comprehensive test of the high availability of the cluster can be achieved by using the cluster fault injection strategy in each test scenario.

[0215] In an exemplary embodiment, the policy formulation module includes:

[0216] A transaction analysis unit, used to determine a transaction processing node in the cluster to be tested that processes resource transaction data based on the transaction type to which the resource transaction data belongs;

[0217] A master node determination unit, used to determine the master node for processing resource transaction data in the cluster to be tested according to the cluster initial state and the transaction processing node;

[0218] The strategy generation unit is used to determine the cluster fault injection strategy in the test scenario according to the master node and the node control mode in the test scenario.

[0219] In an exemplary embodiment, the test module includes:

[0220] A fault setting unit is used to set faults for the cluster to be tested based on the cluster fault injection strategy in the test scenario;

[0221] The data collection unit is used to continuously run online services based on resource transaction data after the fault setting is completed, and obtain cluster performance indicator data of the cluster to be tested in the test scenario.

[0222] In an exemplary embodiment, the data collection unit is specifically configured to:

[0223] After the fault setting is completed, the online service is continuously operated based on the resource transaction data, and during the operation of the online service, the cluster status information, transaction online processing results and server resource usage data are obtained;

[0224] Until the online service runs to the end, the cluster performance indicator data of the cluster to be tested in the test scenario is obtained based on the obtained cluster status information, transaction online processing results and server resource usage data.

[0225] In an exemplary embodiment, the evaluation module includes:

[0226] A requirement acquisition unit is used to obtain the performance indicator requirements of the test scenario;

[0227] The comparison unit is used to evaluate the cluster to be tested by comparing the performance indicator requirements with the cluster performance indicator data to obtain the test result of the test scenario.

[0228] In an exemplary embodiment, the cluster testing device further includes an online module, which is used to:

[0229] Based on the resource transaction data, an online service request is initiated to an online server connected to the cluster to be tested to start the online service.

[0230] Each module in the above cluster test device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0231] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.11As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a cluster testing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0232] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0233] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0234] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0235] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0236] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0237] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0238] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A cluster testing method, characterized in that: The method comprises: Obtain resource transaction data for cluster testing and node control methods of the cluster under test in multiple test scenarios; For each of the test scenarios, when starting the online service, the cluster initial state of the cluster to be tested is obtained, and the cluster fault injection strategy under the test scenario is determined according to the cluster initial state, the resource transaction data and the node control mode under the test scenario; Based on the cluster fault injection strategy in the test scenario and the resource transaction data, an online fault simulation is performed on the cluster to be tested to obtain cluster performance indicator data of the cluster to be tested in the test scenario; According to the cluster performance indicator data, the cluster to be tested is evaluated to obtain a test result of the test scenario; Based on the test results of each of the multiple test scenarios, a test result of the cluster to be tested is obtained.

2. The method according to claim 1, characterized in that The determining, according to the cluster initial state, the resource transaction data and the node control mode in the test scenario, a cluster fault injection strategy in the test scenario includes: Based on the transaction type to which the resource transaction data belongs, determining a transaction processing node in the cluster to be tested that processes the resource transaction data; Determine, according to the cluster initial state and the transaction processing node, a master node in the cluster to be tested that processes the resource transaction data; A cluster fault injection strategy in the test scenario is determined according to the master node and the node control mode in the test scenario.

3. The method according to claim 1, characterized in that The cluster fault injection strategy under the test scenario and the resource transaction data are used to simulate the online fault of the cluster to be tested, and the cluster performance indicator data of the cluster to be tested under the test scenario is obtained, including: Based on the cluster fault injection strategy in the test scenario, performing fault setting on the cluster to be tested; After the fault setting is completed, the online service is continuously run based on the resource transaction data to obtain cluster performance indicator data of the cluster to be tested in the test scenario.

4. The method according to claim 3, characterized in that After the fault setting is completed, the online service is continuously run based on the resource transaction data to obtain cluster performance indicator data of the cluster to be tested in the test scenario, including: After the fault setting is completed, the online service is continuously operated based on the resource transaction data, and during the operation of the online service, cluster status information, transaction online processing results and server resource usage data are obtained; Until the online service runs to the end, the cluster performance indicator data of the cluster to be tested in the test scenario is obtained according to the acquired cluster status information, the transaction online processing result and the server resource usage data.

5. The method according to any one of claims 1 to 4, characterized in that: The step of evaluating the cluster to be tested according to the cluster performance indicator data to obtain a test result of the test scenario includes: Obtaining cluster performance indicator requirements for the test scenario; By comparing the cluster performance indicator requirement and the cluster performance indicator data, the cluster to be tested is evaluated to obtain a test result of the test scenario.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Based on the resource transaction data, an online service request is initiated to an online server connected to the cluster to be tested to start an online service.

7. A cluster testing device, characterized in that: The device comprises: A data acquisition module is used to obtain resource transaction data for cluster testing and node control methods of the cluster to be tested in multiple test scenarios; A strategy formulation module, for each of the test scenarios, when starting the online service, obtains the cluster initial state of the cluster to be tested, and determines the cluster fault injection strategy in the test scenario according to the cluster initial state, the resource transaction data and the node control mode in the test scenario; A test module, used to perform online fault simulation on the cluster to be tested based on the cluster fault injection strategy in the test scenario and the resource transaction data, and obtain cluster performance indicator data of the cluster to be tested in the test scenario; An evaluation module, used to evaluate the cluster to be tested according to the cluster performance indicator data to obtain a test result of the test scenario; The aggregation module is used to obtain the test result of the cluster to be tested based on the test results of each of the multiple test scenarios.

8. The device according to claim 7, characterized in that The strategy formulation module includes: A transaction analysis unit, configured to determine a transaction processing node in the cluster to be tested that processes the resource transaction data based on a transaction type to which the resource transaction data belongs; A master node determination unit, configured to determine a master node for processing the resource transaction data in the cluster to be tested according to the cluster initial state and the transaction processing node; The strategy generating unit is used to determine the cluster fault injection strategy in the test scenario according to the master node and the node control mode in the test scenario.

9. The device according to claim 7, characterized in that The test module includes: A fault setting unit, configured to perform fault setting on the cluster to be tested based on a cluster fault injection strategy in the test scenario; The data collection unit is used to continuously run the online service based on the resource transaction data after the fault setting is completed, and obtain the cluster performance indicator data of the cluster to be tested in the test scenario.

10. The device according to claim 9, characterized in that The data collection unit is specifically used for: After the fault setting is completed, the online service is continuously operated based on the resource transaction data, and during the operation of the online service, cluster status information, transaction online processing results and server resource usage data are obtained; Until the online service runs to the end, the cluster performance indicator data of the cluster to be tested in the test scenario is obtained according to the acquired cluster status information, the transaction online processing result and the server resource usage data.

11. The device according to any one of claims 7 to 10, characterized in that The evaluation module includes: A requirement acquisition unit, used to acquire the performance indicator requirements of the test scenario; The comparing unit is used to evaluate the cluster to be tested by comparing the performance indicator requirement with the cluster performance indicator data to obtain a test result of the test scenario.

12. The device according to any one of claims 7 to 10, characterized in that The device also includes an online module, which is used for: Based on the resource transaction data, an online service request is initiated to an online server connected to the cluster to be tested to start an online service.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images