Distributed system fault simulation method, device, equipment and medium

Abstract

The invention relates to the field of distributed technologies, and particularly discloses a distributed system fault simulation method and device, equipment and a medium, and the method comprises the steps: obtaining a target container in a distributed system; applying different loads to the target container in the target load range, setting the target container to be in a simulated fault state, and obtaining a robustness score of the distributed system in the simulated fault state; and generating a probability density function corresponding to the distributed system according to the robustness scores of the distributed system under different loads. According to the above scheme, the simulation system of the distributed system is containerized by using the container technology, the fault simulation environment can be quickly established, configured and managed, and different requirements and scales of fault simulation tests can be met by selecting different containers and different fault types, so that the fault simulation test efficiency is improved. And the flexibility of distributed system fault simulation is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed technology, and specifically discloses a distributed system fault simulation method, device, equipment and medium. Background Art

[0002] Fault simulation is a systematic method for simulating and testing possible fault scenarios. Discover potential problems: By simulating various fault scenarios, potential problems and defects in the system can be discovered early.

[0003] Traditional fault simulation methods face the following problems. First, time and efficiency issues. That is, traditional fault simulation usually requires a lot of time and manpower. Taking distributed systems as an example, building the experimental environment involved in distributed systems, setting up fault scenarios, executing experiments, and analyzing results all require a lot of time and effort. This may extend the product development cycle or delay system maintenance and repair time. Secondly, the simulation problem is single. Faults in the real world are often in complex environments. Traditional fault simulation is difficult to fully simulate multiple problem scenarios, which leads to limitations in simulation results. Therefore, there is an urgent need for a technical solution that can flexibly simulate distributed systems. Summary of the invention

[0004] In view of this, the present application provides a distributed system fault simulation method, device, equipment and medium, which can meet the fault simulation tests of different needs and scales. The technical solution is as follows.

[0005] In a first aspect, a distributed system fault simulation method is provided, the method comprising:

[0006] Obtain a target container in a distributed system; each node of the distributed system is deployed on the multiple containers respectively;

[0007] Apply different loads to the target container within a target load range, set the target container to a simulated fault state, and obtain a robustness score of the distributed system in the simulated fault state;

[0008] According to the robustness scores of the distributed system under different loads, a probability density function corresponding to the distributed system is generated; the probability density function is used to indicate the robustness of the distributed system under different loads.

[0009] In a possible implementation manner, the target container is a container with the largest load among the containers in the distributed system.

[0010] In a possible implementation, obtaining a target container in a distributed system includes:

[0011] Sampling each container in the distributed system multiple times to obtain an operating indicator of each container in the distributed system; the operating indicator includes at least one of CPU usage, memory usage, network bandwidth usage, and disk usage;

[0012] According to the operating indicators of each container in the distributed system, a target container with the largest load is selected in the distributed system.

[0013] In a possible implementation, the simulated fault state includes a node fault state; the robustness score includes a node fault score;

[0014] The step of setting the target container to a simulated fault state includes:

[0015] stopping the target container;

[0016] The method further comprises:

[0017] After the node fault score is obtained, the target container is started.

[0018] In a possible implementation, the simulated fault state includes a network fault state; the robustness score includes a network fault score;

[0019] The step of setting the target container to a simulated fault state includes:

[0020] Obtain the ID and network ID of the target container;

[0021] According to the ID of the target container and the network ID, disconnect the network bound to the target container;

[0022] The method further comprises:

[0023] After the network fault score is obtained, the target container is reconnected to the bound network.

[0024] In a possible implementation, the simulated fault state includes a resource fault state; the robustness score includes a resource fault score;

[0025] The step of setting the target container to a simulated fault state includes:

[0026] Obtain the ID of the target container;

[0027] According to the ID of the target container, obtaining resource configuration of the target container; the resource configuration includes at least one of a CPU parameter and a memory parameter;

[0028] According to the resource configuration of the target container, adjusting the available resource amount of the target container to be less than the target threshold;

[0029] The method further comprises:

[0030] After the resource fault score is obtained, the available resource amount of the target container is restored.

[0031] In a possible implementation, generating a probability density function corresponding to the distributed system according to the robustness score of the distributed system under different loads includes:

[0032] Obtaining a mean and a standard deviation according to the robustness scores of the distributed system under different loads;

[0033] According to the mean and the standard deviation, a normal distribution probability density function of the distributed system under different loads is obtained.

[0034] In a second aspect, a distributed system fault simulation device is provided, the device comprising:

[0035] A container acquisition module, used to acquire a target container in a distributed system; each node of the distributed system is deployed on the multiple containers respectively;

[0036] A score acquisition module, used to apply different loads to the target container within a target load range, and set the target container to a simulated fault state, and obtain a robustness score of the distributed system in the simulated fault state;

[0037] A function generation module is used to generate a probability density function corresponding to the distributed system according to the robustness score of the distributed system under different loads; the probability density function is used to indicate the robustness of the distributed system under different loads.

[0038] In a third aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the above-mentioned distributed system fault simulation method.

[0039] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned distributed system fault simulation method.

[0040] In a fifth aspect, a computer program product or a computer program is provided, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned distributed system fault simulation method.

[0041] The technical solution provided by this application may have the following beneficial effects:

[0042] When it is necessary to simulate a fault in a distributed system, each node of the distributed system can be deployed on multiple containers. At this time, the nodes on multiple containers together constitute a distributed system. At this time, for the distributed system, you can select the target container, apply different loads, and set the target container to a simulated fault state. At this time, according to the performance of the distributed system, you can obtain the robustness score of the distributed system under simulated faults. At this time, according to the robustness score of the distributed system under different loads, you can generate the corresponding probability density function of the distributed system, thereby characterizing the robustness of the distributed system under different loads. The above scheme uses container technology to containerize the simulation system of the distributed system, which can quickly build, configure and manage the fault simulation environment. By selecting different containers and different fault types, it can meet fault simulation tests of different needs and scales, greatly improving the flexibility of distributed system fault simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a schematic diagram of a distributed system built on a container platform involved in an embodiment of the present application.

[0045] Figure 2 The present invention is a method flow chart of a distributed system fault simulation method according to an exemplary embodiment.

[0046] Figure 3 The present invention is a method flow chart of a distributed system fault simulation method according to an exemplary embodiment.

[0047] Figure 4 A distributed node failure logic diagram involved in an embodiment of the present application is shown.

[0048] Figure 5 A distributed network fault logic diagram involved in an embodiment of the present application is shown.

[0049] Figure 6 A distributed resource failure logic diagram involved in an embodiment of the present application is shown.

[0050] Figure 7 A schematic diagram of the structure of a distributed system fault simulation system based on container technology involved in an embodiment of the present application is shown.

[0051] Figure 8 A schematic structural diagram of a short-board sidecar module involved in an embodiment of the present application is shown.

[0052] Fig. 9 A schematic diagram of the structure of a detection and scoring module involved in an embodiment of the present application is shown.

[0053] Fig.10 It is a structural diagram of a distributed system fault simulation device provided in an embodiment of the present application.

[0054] Fig.11 It is a structural schematic diagram of a computer device provided by an optional embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0057] Fault simulation is a systematic method for simulating and testing possible fault scenarios. Discover potential problems: By simulating various fault scenarios, potential problems and defects in the system can be discovered early. This helps prevent and solve potential faults and improve the reliability and stability of the system; Improve system reliability: By performing fault simulation tests on the system, the adaptability and resilience of the system under different fault conditions can be discovered. This helps improve the design and implementation of the system and improve the reliability and fault tolerance of the system. Optimize maintenance and repair: Fault simulation can help determine the best strategy for troubleshooting and repair. By simulating different fault scenarios, different maintenance and repair methods can be evaluated, and the most effective solution can be found to shorten the system recovery time. In summary, fault simulation has a wide range of applications in software development, system operation and maintenance, and industrial control.

[0058] Traditional fault simulation methods face the following problems:

[0059] (1) Time and efficiency issues: Traditional fault simulation usually requires a lot of time and manpower. Building the experimental environment, setting up fault scenarios, executing experiments, and analyzing results all take a lot of time and effort. This may extend the product development cycle or delay system maintenance and repair time.

[0060] (2) Single simulation problem: Faults in the real world are often associated with complex environments. Traditional fault simulation is difficult to fully simulate multiple problem scenarios, which leads to limitations in simulation results.

[0061] (3) Cost issue: Traditional fault simulation usually requires the establishment of a complete simulation environment or laboratory equipment, which requires a lot of capital investment and resource consumption. The cost of purchasing, maintaining and operating equipment is high, which limits the feasibility and scope of implementation of fault simulation.

[0062] Especially in distributed systems, due to the large number of devices designed and the wide deployment range, it is very difficult to deploy simulation environments or laboratory equipment. Even if a virtual machine is used to perform virtual construction of a distributed system, the workload is very large, and different fault simulations may require different environments. Therefore, the flexibility of fault simulation in distributed systems is low and the feasibility is poor. In order to achieve the above purpose, the present invention adopts a distributed system fault simulation method based on container technology, builds a distributed system through a container platform, improves the flexibility of fault simulation, and is suitable for fault simulation test requirements in different environments and scales. Figure 1 Schematic diagram of a distributed system constructed by a container platform involved in an embodiment of the present application. Figure 1 As shown, the container platform includes several containers (for example, container 1, container 2, and container 3). More containers can be deployed on the container platform as needed to meet the requirements of distributed system deployment.

[0063] Optionally, the above-mentioned container can be deployed on different computer devices, or can be deployed in the same computer device, or multiple containers can be deployed as a group on one computer device, and each of the multiple computer devices has a container deployed, and the containers are connected through a wired network or a wireless network to form the above-mentioned container platform.

[0064] Optionally, each node of the distributed system is deployed on each container, and each container corresponds to each node one by one, that is, when the distributed system needs to be deployed, the corresponding container can be run on the computer device to serve as each node of the distributed system.

[0065] Optionally, in the present application, the computer device may be a server.

[0066] Optionally, the above-mentioned server can be a server cluster or a distributed system composed of multiple physical servers, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms and other technical computing services.

[0067] Optionally, the above-mentioned wireless network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any other network, including but not limited to any combination of local area network, metropolitan area network, wide area network, mobile, limited or wireless network, private network or virtual private network. In some embodiments, the technology and / or format including hypertext markup language, extensible markup language, etc. are used to represent the data exchanged through the network. In addition, conventional encryption technologies such as secure socket layer, transport layer security, virtual private network, Internet protocol security, etc. can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology.

[0068] Figure 2 is a method flow chart of a distributed system fault simulation method according to an exemplary embodiment. The method is applied to Figure 1 The distributed system is built on the container platform shown in Figure 1. Figure 2 As shown, the distributed system fault simulation method may include the following steps:

[0069] Step 201, obtaining a target container in a distributed system; each node of the distributed system is deployed on the multiple containers respectively.

[0070] The target container is the container with the largest load among the containers of the distributed system.

[0071] The target container refers to the container with the largest load and the heaviest task. The largest load of the target container means that the target container processes a relatively high number of requests, computing tasks, or data transmissions, and is usually a bottleneck node or key node in a distributed system. Therefore, the failure of the target node usually has the greatest impact on the entire system. Therefore, selecting it as the object of fault simulation can more accurately characterize the performance of the distributed system when a fault occurs under different loads.

[0072] Step 202: applying different loads to the target container within a target load range, setting the target container to a simulated fault state, and obtaining a robustness score of the distributed system in the simulated fault state.

[0073] Optionally, in the embodiment of the present application, different levels of loads may be applied to the target container, that is, the actual stress situation may be simulated by increasing the workload of the container. For example, the actual stress situation may be simulated by simulating high concurrent requests, data storage pressure, computing task load, etc.

[0074] In an embodiment of the present application, the robustness of a distributed system under different loads may be different, so different levels of loads may be applied to the target container and then simulated failures may be performed to more comprehensively measure the robustness of the distributed system under various circumstances.

[0075] Step 203: Generate a probability density function corresponding to the distributed system according to the robustness scores of the distributed system under different loads.

[0076] At this time, the probability density function is used to indicate the robustness of the distributed system under different loads.

[0077] In summary, when it is necessary to simulate a fault in a distributed system, each node of the distributed system can be deployed on multiple containers. At this time, the nodes on multiple containers together constitute a distributed system. At this time, for the distributed system, you can select the target container, apply different loads, and set the target container to a simulated fault state. At this time, according to the performance of the distributed system, you can obtain the robustness score of the distributed system under simulated faults. At this time, according to the robustness score of the distributed system under different loads, you can generate the corresponding probability density function of the distributed system, thereby characterizing the robustness of the distributed system under different loads. The above scheme uses container technology to containerize the simulation system of the distributed system, which can quickly build, configure and manage the fault simulation environment, and through the selection of different containers and different fault types, it can meet the fault simulation tests of different needs and scales, greatly improving the flexibility of distributed system fault simulation.

[0078] Figure 3 is a method flow chart of a distributed system fault simulation method according to an exemplary embodiment. The method is applied to Figure 1 The distributed system is built on the container platform shown in Figure 1. Figure 3 As shown, the distributed system fault simulation method may include the following steps:

[0079] Step 301: perform multiple sampling on each container in the distributed system to obtain an operation index of each container in the distributed system.

[0080] First of all, in order to realize the fault simulation of distributed systems, a container platform is first needed to create basic resources. The platform can be an orchestration platform such as docker-compose and kubernetes.

[0081] Furthermore, it is necessary to define simulation programs and resource lists on the container platform. The simulation program can be a distributed program, such as etcd program, kafka program, etc., and define the corresponding distributed program data writing method, query method, etc.; the resource list can be the number of container replicas, the resource allocation required when the container is running, such as the number of CPUs, the amount of memory, etc.

[0082] Furthermore, after applying the resource list and the simulation program configuration file on the container platform, the container platform will automatically create the container and deploy the distributed system that needs to be tested.

[0083] After the container and distributed program are deployed, the distributed program creation module will notify the main program that rendering can be performed.

[0084] Differential patch module rendering is a method of dividing the configuration into multiple modules, and first generating a set of vector interval values ​​according to the user configuration, and combining the DOM template file to perform fine-grained difference change tracking, only updating the changed parts, thereby improving rendering efficiency and quickly obtaining the simulation interval.

[0085] The DOM template is the core of the distributed fault simulation system, acting as the middle layer for data update and rendering. It includes the deployment scripts, resource configuration scripts, and inspection program scripts of distributed programs.

[0086] If the user configures the initial static load value of module A to be c, then the vector group of module A is A[c,c*w], where w is the data width coefficient. The function cus(i) represents the module rendering operation with parameter i, and dom(n) represents the nth template. In the difference patch algorithm, a difference patch function Diff(cus,dom) is defined, which represents the difference between the current module configuration parameter rendering and the DOM template. The input of the difference function Diff is the defined parameters and the parameters of the pre-DOM template, and the output is the data structure diff representing the difference between the two parameters.

[0087] There are the following expressions for the difference patch rendering process:

[0088] diff=Diff(cus(i),dom(n))

[0089] In distributed resource configuration rendering, diff is usually a string structure.

[0090] For the rendering process of the entire configuration, the following expression can be used to describe the process:

[0091] RenderConfig(cus[c,c*w],dom(n))=Diff(cus[c],dom(1))∪Diff(cus[c],dom(2))∪...∪Diff(cus[c*w],dom(n-1))∪Diff(cus[c*w],dom(n))

[0092] It means that given the data width w and the template, the difference between each module is calculated in turn, and their differences are merged into a whole, so as to obtain a simulation script that contains the simulation range faults.

[0093] Under the default configuration, simulation scripts usually only include node failures, network failures, and resource failures; detection programs only include scripts such as data writing and data query.

[0094] After the distributed resources are rendered, the distributed system can be considered to have been created. At this time, a target still needs to be selected for fault simulation. In this solution, the object is the selected container. In a system, the weakest part will determine the strength and efficiency of the entire system. It is like water in a bucket. If there is a hole at the bottom or somewhere in the bucket, the water will flow out from that place and no longer be maintained as designed. Therefore, in the simulation system, the "worst" container (that is, the container with the largest load) is used as the simulation object.

[0095] Specifically, the distributed system can obtain the target container based on the operation indicators.

[0096] The operation indicator includes at least one of CPU usage, memory usage, network bandwidth usage, and disk usage.

[0097] Specifically, a short-board sidecar module is designed in the simulation system, which can collect and record the operating indicators of the current container in real time through the container platform, such as CPU usage, memory usage, network traffic, disk IO and other indicators. After data collection, comprehensive load calculation will be performed. For example, the first collection data of container A: CPU usage a c1 , memory usage a m1 , network bandwidth usage a n1 , disk usage a d1 ; Second data collection: CPU usage a c2 , memory usage a m2 , network bandwidth usage a n2 , disk usage a d2 ; and so on.

[0098] Step 302: According to the operating indicators of each container in the distributed system, a target container with the largest load is selected in the distributed system.

[0099] The sidecar module collects the container operation indicators three times and multiplies them by the data width w in step 4 to obtain the largest value in the simulated data set.

[0100] A c =max[a c1 *w,a c2 *w,a c3 *w]

[0101] A c Indicates the value of the change in CPU usage of container A within the simulation range. In the same way, we can get A c , A m , A n , A d Then the multiplied comprehensive load index of container A is:

[0102] A s =(A c xω1)+(A m xω2)+(A n xω3)+(A d xω4)

[0103] Where ω represents the weight of each indicator, and satisfies (ω1+ω2+ω3+ω4)=1.

[0104] In this way, we can get the container A multiplied comprehensive load index A s , container B doubles the comprehensive load index B s , container C doubles the comprehensive load index C s The larger the multiplied comprehensive load index value of the container, the higher the load of the container during the fault simulation, and the selected simulation object A=max[A s ,B s ,C s ].

[0105] Step 303: Apply different loads to the target container within the target load range, set the target container to a simulated fault state, and obtain a robustness score of the distributed system in the simulated fault state.

[0106] After the fault simulation is executed, the cluster availability is tested through the test module. The module writes the data set R to the cluster and records the data read from the cluster as D. The first check will be through the complete consistency check function hash():

[0107] hash(R)==hash(D)

[0108] This can quickly determine whether the written data is completely consistent with the data read by the cluster, P = 100%. If not, it will enter the scoring mechanism:

[0109] if C(R,D)is true:

[0110] count=count+1

[0111] Specifically, the total write volume R is obtained through C(R,D) polling check statistics. t , read the number of consistent data D c .So:

[0112] P=(D c / R t )%

[0113] P represents the percentage of consistent data in the total data, and is also used as the robustness score of the distributed system. The larger P is, the more reliable it is.

[0114] In the process of fault simulation, it is necessary to apply different loads to the target container within the target load range for cyclic simulation. Specifically, if the user configures the initial static load value as c and the simulation data width system as w, Magnet learns that the simulated load range is [c, c*w], then in the first simulation, a load of c is applied to container A, and according to the set step size i, the cyclic simulation reaches the load c*w:

[0115] Optionally, the simulated fault state includes a node fault state; in this case, the robustness score includes a node fault score.

[0116] At this time, the distributed system may stop the target container, thereby setting the distributed system to a node failure state; after the node failure score is obtained, the target container is started.

[0117] For details, please refer to Figure 4 , which shows a distributed node fault logic diagram involved in an embodiment of the present application. The selected target container (e.g., container A) can be stopped by the stop container stop command. Since a copy of the distributed program is deployed in the selected target container, when the container stops, it is equivalent to simulating the failure of a distributed node. After the container stops, data operations begin. The client calls the distributed interface to write data and performs multiple write operations. At this time, the above-mentioned full consistency check function is used to obtain the robustness score of the distributed system in the node failure state, that is, the node failure score. The final result will be output to the console through standard output to evaluate the simulation results. After the fault simulation is completed, the start container start command needs to be used to start the previously stopped target container in order to restore the state of the fault simulation.

[0118] Optionally, the simulated fault state includes a network fault state, and in this case the robustness score includes a network fault score.

[0119] That is, at this time, the distributed system can obtain the ID and network ID of the target container;

[0120] According to the ID of the target container and the network ID, the network bound to the target container is disconnected; after the network fault score is obtained, the target container is reconnected to the bound network.

[0121] For details, please refer to Figure 5 , which shows a distributed network fault logic diagram involved in an embodiment of the present application. According to the ID of container A (i.e., the target container), a request is made to the container platform to obtain the network ID (networkidA) to which container A is bound. After obtaining the ID (idA) and network ID (networkidA) of container A, the disconnect command is used to cut off the relationship between container A and the bound network. The disconnection of container A from the network interrupts the communication between other distributed containers and container A because they only communicate through this network. After the network fault simulation is successful, data operations begin. The client calls the distributed interface to perform multiple data write operations. At this time, the above-mentioned full consistency check function is used to obtain the robustness score of the distributed system under the network fault state. The final result will be output to the console through standard output to evaluate the simulation results. After the network fault simulation is completed, the container that is disconnected from the network needs to be restored. At this time, the container and the network can be reconnected by restoring the network connect command to restore the state before the fault simulation.

[0122] Optionally, the simulated fault state includes a resource fault state; and the robustness score includes a resource fault score.

[0123] At this time, the distributed system can obtain the ID of the target container; obtain the resource configuration of the target container according to the ID of the target container; the resource configuration includes at least one of a CPU parameter and a memory parameter; and adjust the available resource amount of the target container to be less than a target threshold according to the resource configuration of the target container.

[0124] Furthermore, after the resource fault score is obtained, the distributed system restores the available resource amount of the target container.

[0125] For details, please refer to Figure 6, which shows a distributed resource failure logic diagram involved in an embodiment of the present application. According to the ID of container A, request the container platform to obtain the resource configuration of the current container, including the number of CPU cores and memory size, and record it. Use the container resource adjustment update command to dynamically adjust the number of CPU cores and memory size of container A, and reduce the resource allocation of the container to 10% of the original (configurable ratio), so as to simulate the scenario of resource exhaustion. After the resource failure simulation is successful, data operations begin. The client will call the distributed interface to perform multiple data write operations. At this time, the above-mentioned full consistency check function is used to obtain the robustness score of the distributed system in the resource failure state. The final result will be output to the console through standard output to evaluate the simulation results. After the resource failure simulation is completed, the resources of the container need to be restored. The script will read the previously recorded configuration information and use the docker update command to restore the container's resources to the original configuration.

[0126] At this point, a fault scenario simulation is completed, and the robustness scores P1, P2, and P3 under the three scenarios can be obtained respectively, and the next load simulation can begin.

[0127] Step 304: Generate a probability density function corresponding to the distributed system according to the robustness scores of the distributed system under different loads; the probability density function is used to indicate the robustness of the distributed system under different loads.

[0128] Furthermore, according to the robustness score of the distributed system under different loads, a mean and a standard deviation are obtained; according to the mean and the standard deviation, a normal distribution probability density function of the distributed system under different loads is obtained.

[0129] Specifically, the distributed simulation will execute the operation (c*wc) / i times to obtain 3*(c*wc) / i robust scores: P1, P2, ..., P 3*(c*w-c) / i .

[0130] First calculate the mean and standard deviation of these data.

[0131] Calculate the mean (μ):

[0132] μ=(P1+P2...、P 3*(c*w-c) / i-1 +P 3*(c*w-c) / i ) / 3*(c*wc) / i

[0133] Calculate the standard deviation (σ):

[0134] σ=√[(P1-μ)^ 2 +(P2-μ)^ 2 +...+(P 3*(c*w-c) / i-1 -μ)^ 2+(P 3*(c*w-c) / i -μ)^ 2 ]

[0135] Now that we have the mean (μ) and standard deviation (σ) of our data, we can use these values ​​to construct a normal distribution probability density function (PDF) for our data.

[0136] For example, the probability density of P1 is:

[0137] P(P1)=(1 / (σ*√(2π)))*exp(-(P1-μ)^ 2 / (2*σ^ 2 ))

[0138] Similarly, the probability density corresponding to the values ​​in the entire simulated data range can be calculated.

[0139] Step 305: Draw a probability density graph according to the probability density function corresponding to the distributed system.

[0140] Finally, the calculated probability density can be used to draw a probability density graph. Drawing a probability density graph of the data can intuitively display the distribution of the data. In the probability density graph, the horizontal axis represents the value range of the robustness data collected in step 7, and the vertical axis represents the probability density at the corresponding value. If the data conforms to the normal distribution, the probability density graph will show the shape of a bell curve, that is, most of the data is concentrated near the mean, and as the data value moves away from the mean, the probability density gradually decreases.

[0141] Therefore, through the normal distribution diagram, we can comprehensively judge the robustness characteristics of distributed storage under certain loads.

[0142] Please refer to Figure 7 , which shows a schematic diagram of the structure of a distributed system fault simulation system based on container technology involved in an embodiment of the present application. Figure 7 As shown, after the system is created, by defining the simulation program and reading the resource list, creating each container on the container platform, and defining the nodes, network and other parameters through the simulation program to generate a distributed system, it can be used as shown in Figure 8 The shortboard sidecar module shown in the figure performs data sampling and determines the target container, making the target container in a faulty state ( Figure 8 The execution principle of the short board sidecar module shown in FIG. 3 can be referred to step 302, which will not be described in detail at this time); at this time, Fig. 9 The detection and scoring module shown detects the distributed system in real time and calculates the robustness score ( Fig. 9The execution principle of the detection and scoring module shown can be referred to step 303, which will not be described here), and then the target container exits from the fault state. After multiple cycles, the detection and scoring module enters the obtained results into the statistical module to generate a probability density function and a probability density map to characterize the robustness of the distributed system under different loads.

[0143] In summary, when it is necessary to simulate a fault in a distributed system, each node of the distributed system can be deployed on multiple containers. At this time, the nodes on multiple containers together constitute a distributed system. At this time, for the distributed system, you can select the target container, apply different loads, and set the target container to a simulated fault state. At this time, according to the performance of the distributed system, you can obtain the robustness score of the distributed system under simulated faults. At this time, according to the robustness score of the distributed system under different loads, you can generate the corresponding probability density function of the distributed system, thereby characterizing the robustness of the distributed system under different loads. The above scheme uses container technology to containerize the simulation system of the distributed system, which can quickly build, configure and manage the fault simulation environment, and through the selection of different containers and different fault types, it can meet the fault simulation tests of different needs and scales, greatly improving the flexibility of distributed system fault simulation.

[0144] In addition, the probability density map mentioned in the present invention is made based on the robustness score, and a large number of fault simulations are required to obtain a certain amount of data and then make a probability density map. Each fault simulation can only obtain one robustness score P under this fault, not the data consistency probability data of the distributed system under the fault. Please note this. Generally, distributed systems are strongly consistent. After the fault is injected into this fault simulation system, the data check module is configured with an acquisition timeout judgment. If the data is not obtained within the specified time or the obtained data is different from most of the data, it is judged as "inconsistent" as the basis for the robustness score.

[0145] The difference patch module rendering involved in the present invention can generate a set of vector data widths according to the static parameters configured by the user. The configurations in this data width range will be the scenes of fault simulation. Many configurations and scripts are divided into multiple modules, and the difference between the current module data and the previous data is compared to find the difference between the two, and only the difference part is updated. Compared with the traditional configuration rendering method, it can avoid completely re-rendering the entire configuration, reducing unnecessary calculations and rendering. Therefore, with a small number of configurations, it is possible to quickly and accurately generate a set of configurations and scripts for fault simulation scenes.

[0146] The shortboard sidecar module in the present invention can calculate the multiplied comprehensive load index of multiple containers based on the operating index of the current container and the data width coefficient of the simulation scenario through a multiplication algorithm, and then select the "worst situation" container in the simulation scenario from multiple containers as the simulation and detection object. This improves the applicability and resource diversity of fault simulation.

[0147] The invention can also perform multiple groups of distributed fault simulations. In the process of detecting the scoring module, the distributed data write and read consistency results will be processed, first performing an overall hash comparison, and then a round-robin hash comparison, so as to obtain the data consistency score as the distributed robustness result. Since multiple simulations will be performed within the data width range, that is, multiple different scores will be obtained, the scoring module will finally combine a large number of data and data widths for normal distribution calculation, thereby obtaining the probability density function of the robustness score of the distribution degree under different configurations. This can help users better and intuitively understand the quantitative scoring concentration trend and discreteness of the robustness of distributed programs.

[0148] In the embodiments of the present application, a distributed system fault simulation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0149] The present application provides a distributed system fault simulation device. Fig.10 : is a schematic diagram of the structure of a distributed system fault simulation device provided in an embodiment of the present application, the device comprising:

[0150] The container acquisition module 1001 is used to acquire a target container in a distributed system; each node of the distributed system is deployed on the multiple containers respectively;

[0151] A score acquisition module 1002 is used to apply different loads to the target container within a target load range, and set the target container to a simulated fault state, and obtain a robustness score of the distributed system in the simulated fault state;

[0152] The function generation module 1003 is used to generate a probability density function corresponding to the distributed system according to the robustness score of the distributed system under different loads; the probability density function is used to indicate the robustness of the distributed system under different loads.

[0153] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0154] The distributed system fault simulation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0155] The embodiment of the present invention also provides a computer device having the above Fig.10 The distributed system failure simulation device shown.

[0156] See also Fig.11 , Fig.11 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.11 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphic information in a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 10 is taken as an example.

[0157] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0158] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0159] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0161] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0162] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0163] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0164] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the present invention.

Claims

1. A distributed system fault simulation method, characterized in that: The method comprises: Obtain a target container in a distributed system; each node of the distributed system is deployed on multiple containers respectively; Apply different loads to the target container within a target load range, set the target container to a simulated fault state, and obtain a robustness score of the distributed system in the simulated fault state; According to the robustness scores of the distributed system under different loads, a probability density function corresponding to the distributed system is generated; the probability density function is used to indicate the robustness of the distributed system under different loads.

2. The method according to claim 1, characterized in that The target container is the container with the largest load among the containers in the distributed system.

3. The method according to claim 2, characterized in that The obtaining of the target container in the distributed system includes: Sampling each container in the distributed system multiple times to obtain an operating indicator of each container in the distributed system; the operating indicator includes at least one of CPU usage, memory usage, network bandwidth usage, and disk usage; According to the operating indicators of each container in the distributed system, a target container with the largest load is selected in the distributed system.

4. The method according to any one of claims 1 to 3, characterized in that: The simulated fault state includes a node fault state; the robustness score includes a node fault score; The step of setting the target container to a simulated fault state comprises: stopping the target container; The method further comprises: After the node fault score is obtained, the target container is started.

5. The method according to any one of claims 1 to 3, characterized in that: The simulated fault state includes a network fault state; the robustness score includes a network fault score; The step of setting the target container to a simulated fault state comprises: Obtain the ID and network ID of the target container; According to the ID of the target container and the network ID, disconnect the network bound to the target container; The method further comprises: After the network fault score is obtained, the target container is reconnected to the bound network.

6. The method according to any one of claims 1 to 3, characterized in that: The simulated fault state includes a resource fault state; the robustness score includes a resource fault score; The step of setting the target container to a simulated fault state includes: Obtain the ID of the target container; According to the ID of the target container, obtaining resource configuration of the target container; the resource configuration includes at least one of a CPU parameter and a memory parameter; According to the resource configuration of the target container, adjusting the available resource amount of the target container to be less than the target threshold; The method further comprises: After the resource fault score is obtained, the available resource amount of the target container is restored.

7. The method according to any one of claims 1 to 3, characterized in that: Generating a probability density function corresponding to the distributed system according to the robustness score of the distributed system under different loads includes: Obtaining a mean and a standard deviation according to the robustness scores of the distributed system under different loads; According to the mean and the standard deviation, a normal distribution probability density function of the distributed system under different loads is obtained.

8. A distributed system fault simulation device, characterized in that: The device comprises: A container acquisition module, used to acquire a target container in a distributed system; each node of the distributed system is deployed on multiple containers respectively; A score acquisition module, used to apply different loads to the target container within a target load range, and set the target container to a simulated fault state, and obtain a robustness score of the distributed system in the simulated fault state; A function generation module is used to generate a probability density function corresponding to the distributed system according to the robustness score of the distributed system under different loads; the probability density function is used to indicate the robustness of the distributed system under different loads.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the distributed system fault simulation method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute any one of the distributed system fault simulation methods in claims 1 to 7.

Images