A chaos training method, device and computer readable medium
By splitting the chaos drill into multiple components and randomly assigning values to generate exceptions, combining traffic segmentation and canary analysis, the problems of insufficient randomness and difficulty in automatic judgment in the existing technology are solved, and efficient Automatic chaos drill effect.
Patent Information
- Application Number
- CN202211390031.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-08
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-11-08
AI Technical Summary
The existing chaos drill technology cannot achieve automatic drilling, the randomness of exception injection is insufficient, and the inability to automatically judge the drill results, resulting in poor results of chaos drilling.
The chaos drill is split into multiple constituent elements. Each constituent element contains at least one random variable. Exceptions are generated by random assignment and exception injection is performed. The drill results are determined using traffic segmentation and canary analysis.
It realizes a high randomness and automation chaos drill, which can be repeated for a long time, automatically judges the results of the drill, and improves the efficiency and accuracy of the drill.
Smart Images

Figure CN115834404B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of chaos engineering technology, and in particular, relates to a chaos training method, device and computer-readable medium. Background Art
[0002] Chaos engineering is an emerging technical discipline that conducts experiments on large distributed systems in a proactive manner to identify risk points where problems may occur. The purpose is to build the system's ability and confidence to resist out-of-control conditions in the production environment, and to fundamentally change the way developers think about dealing with software defects and failures.
[0003] The founder of Chaos Engineering Netflix only proposed 5 basic principles, and there is no implementation plan or suggestion. There is little industry knowledge and practice accumulation, which leads to a series of problems in the current chaos drills, such as the inability to achieve automatic drills, insufficient randomness of exception injection, and inability to automatically judge the results of drills, etc. For example, Alibaba's open source ChaosBlade tool can cause chaos to the machine through the command line or http, but it can only solve the problem of single exception injection, and it cannot achieve automatic drills or automatically judge the results of drills; ChaosIQ's open source ChaosToolkit chaos framework only abstractly splits chaos engineering into abstract concepts such as steady state, action, and rollback. This design lacks randomness and does not have the meaning of long-term repeated execution. Summary of the invention
[0004] In view of this, the present application provides a chaos training method, device and computer-readable medium to solve at least some of the technical problems existing in the current chaos training methods.
[0005] The specific plan is as follows:
[0006] A chaos training method, comprising:
[0007] Obtain multiple components of a chaos exercise; wherein a chaos exercise is pre-divided into multiple components, and each component includes at least one random variable;
[0008] Randomly assign values to random variables included in different component elements to obtain random assignment results of the random variables included in different component elements;
[0009] Generate the anomalies required for the current chaos exercise according to the random assignment results of the random variables contained in different component elements;
[0010] Perform exception injection processing on the generated exceptions to perform chaos drills based on the injected exceptions.
[0011] Optionally, the obtaining of multiple components contained in a chaos exercise includes:
[0012] Obtain at least some of the components of an abnormal template, running time, number of abnormalities, and running mode included in a chaos drill;
[0013] The exception templates under different exception types provide different dynamic attributes accordingly, and the dynamic attributes provided by the exception template serve as random variables included in the exception template.
[0014] Optionally, the running time indicates the duration of an exception in a chaos drill, the number of exceptions indicates the number of exception types to be executed in a chaos drill, and the exception mode indicates the execution mode between different exceptions in a chaos drill;
[0015] The method of generating the anomaly required for the current chaos exercise according to the random assignment results of the random variables contained in the different component elements includes:
[0016] Under the conditional constraints corresponding to the running time, the number of exceptions and the current random assignment result of the exception pattern, exceptions are continuously generated based on the exception template under the corresponding exception type.
[0017] Optionally, performing exception injection processing on the generated exception to perform chaos drill based on the injected exception includes:
[0018] Inject the generated exceptions into the corresponding production environment, and perform chaos drills based on the injected exceptions in the pre-created chaos environment.
[0019] Optionally, before obtaining multiple components included in a chaos exercise, the method further includes:
[0020] Based on the configured scheduled tasks, the daily inspection process is triggered regularly. Based on the daily inspection process, a chaotic environment for chaos drills is created, and exception injection and chaos drills based on the injected exceptions are performed based on the created chaotic environment. When the drill end conditions are met, the drill is ended and the chaotic environment is destroyed.
[0021] Optionally, the creating a chaotic environment for chaos drill based on the daily inspection process includes:
[0022] The task scheduling center sends parameter data of preset creation parameters to the cloud platform based on the daily inspection process, so that the cloud platform creates a chaotic environment based on the parameter data of the preset creation parameters.
[0023] Optionally, the above method further includes:
[0024] Determine the drill results based on traffic segmentation and canary analysis.
[0025] Optionally, the determining of the drill result based on traffic segmentation processing and canary analysis includes:
[0026] Divide the inlet traffic into two groups with the same proportion, which are the traffic of the chaotic group and the traffic of the control group respectively;
[0027] Obtain the service indicator monitoring result corresponding to the chaos drill of the service based on the traffic of the chaos group as the chaos group indicator;
[0028] Obtain the service indicator monitoring results of the service under the operation performance corresponding to the control group traffic as the control group indicator;
[0029] The drill results are determined based on the offset between the chaos group indicators and the control group indicators.
[0030] A chaos training device, comprising:
[0031] An acquisition unit is used to acquire multiple components contained in a chaos exercise; wherein a chaos exercise is pre-divided into multiple components, and each component element corresponds to at least one random variable;
[0032] A random assignment unit, used for randomly assigning values to random variables contained in different component elements, and obtaining random assignment results of the random variables contained in each component element;
[0033] The anomaly generation unit is used to generate the anomaly required for the current chaos exercise according to the random assignment results of the random variables contained in different component elements;
[0034] The exception injection and chaos drill unit is used to perform exception injection processing on the generated exceptions to perform chaos drills based on the injected exceptions.
[0035] A computer readable medium stores a computer program thereon, wherein the computer program comprises a program code for executing any of the methods described above.
[0036] In summary, the chaos drill method, device, and computer-readable medium provided by the present application split a chaos drill into multiple components, each of which corresponds to at least one random variable. When a chaos drill is required, the multiple components included in a chaos drill are obtained, and random values are assigned to the random variables included in each component. According to the random assignment results of the random variables included in each component, the exceptions required for the current chaos drill are generated, and the generated exceptions are injected to perform chaos drills based on the injected exceptions. The present application splits a chaos drill into multiple components, each of which contains at least one random variable. Each drill generates the exceptions required for the drill from these random variables like opening a blind box, so that each chaos drill is almost non-repetitive, ensuring the high randomness of the chaos drill, at least solving the problem of insufficient randomness of chaos drills in the prior art, and having the significance of long-term repeated execution. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0038] Figure 1 It is a flow chart of the chaos drill method provided by this application;
[0039] Figure 2 This is a random walkthrough blind box factory diagram provided by this application;
[0040] Figure 3 It is another flow chart of the chaos drill method provided by this application;
[0041] Figure 4 This is the daily inspection timing diagram provided by this application;
[0042] Figure 5 It is another flow chart of the chaos drill method provided by this application;
[0043] Figure 6 It is a flow chart provided by this application for determining the results of the drill based on traffic segmentation and canary analysis;
[0044] Figure 7 It is a logical diagram of the automatic judgment exercise results provided by this application;
[0045] Figure 8 This is a structural diagram of the chaos training device provided in this application. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] The present application discloses a chaos training method, device and computer-readable medium, which are used to provide corresponding solutions to at least some of the technical problems such as the randomness problem of abnormal injection, the automatic training problem and the automatic determination of the training results.
[0048] See also Figure 1 The chaos training method provided by the present application at least includes the following processing flow:
[0049] Step 101: Obtain multiple components of a chaos exercise.
[0050] The embodiment of the present application pre-splits a chaos exercise into multiple components, each of which contains at least one random variable. A chaos exercise is split into multiple components, including but not limited to abnormal templates, running time, number of abnormalities, part or all of the running mode, each component contains at least one random variable that can be dynamically randomly assigned, and each exercise is like opening a blind box, and these random variables generate the abnormalities required in the exercise process.
[0051] Among them, the exception template will contain different dynamic attributes depending on the exception type / scenario. That is, the exception templates under different exception types provide different dynamic attributes, and the dynamic attributes provided by the exception template serve as random variables contained in the exception template.
[0052] Taking the "packet loss" exception type / scenario as an example, the packet loss exception template contains two dynamic attributes, "packet loss rate" and "target IP". Taking the IP attribute as an example, the IP expression is 10.0.0.1 / 32. If only a B-segment intranet IP starting with 10 is used, the other 3 digits have 255 possibilities, the last digit has 24 possibilities, and the total number of possible IP expressions is 400 million. Accordingly, the IP attribute can be randomly selected from various possible values. Therefore, the dynamic attributes contained in the packet loss exception template, such as "packet loss rate" and "target IP", can be used as its random variables.
[0053] The running time indicates the duration of an abnormality in a chaos drill. Optionally, the embodiment of the present application configures it to a random number of seconds within a set time period. If the time period is set to 10 minutes, there will be 600 possible running times, and a random value can be assigned from the 600 possibilities.
[0054] The number of exceptions refers to how many types of exceptions will be executed in a chaos drill. Specifically, it indicates the number of exception types that will be executed in a chaos drill. For example, it can be set to execute 8 types of exceptions in one drill. In actual applications, random assignment can be performed within the preset upper limit.
[0055] The operation mode refers to the execution method between different exceptions in a chaos drill, which is divided into parallel and serial execution modes, and the execution mode can be randomly assigned.
[0056] According to the above design, it is supported to randomly assign values to random variables corresponding to various components of an exercise, so that each generated chaotic exercise will hardly be repeated.
[0057] When a chaos drill is required for a business service or application, multiple components of a chaos drill may be obtained accordingly, for example, at least some of the components of an exception template, running time, number of exceptions, and running mode included in a chaos drill may be obtained.
[0058] Step 102: Randomly assign values to random variables included in different component elements to obtain random assignment results for the random variables included in different component elements.
[0059] After obtaining the various components contained in a chaos exercise, random values can be further assigned to the random variables corresponding to the various components based on a random algorithm.
[0060] Step 103: Generate the anomalies required for the current chaos exercise according to the random assignment results of the random variables contained in different component elements.
[0061] Afterwards, under the conditional constraints corresponding to the current random assignment result of the running time, the number of exceptions and the exception mode, exceptions are continuously generated based on the exception template under the corresponding exception type.
[0062] See also Figure 2 For example, Figure 2A schematic diagram of a random rehearsal blind box factory is provided. In this example, the blind box factory is specifically used to randomly assign values to random variables and generate exceptions. For example, for a random rehearsal, the running time, number of exceptions, and running mode are randomly assigned to "180s", "3", and "parallel", respectively. Correspondingly, three types of exceptions are randomly selected from the multiple exception types provided, such as excessive CPU utilization, excessive memory utilization, IO contention, insufficient disk remaining space, domain name resolution exceptions, network delays, network packet loss, and unexpected return messages, and the dynamic attributes in the exception template corresponding to each exception type are randomly assigned. Based on the random assignment results, three types of exceptions are continuously generated during the duration of the running time.
[0063] Figure 2 In the example, percentage, read IO, write IO, directory, domain name, delay, offset, packet loss rate, target IP, etc. are dynamic attributes included in the exception template under the corresponding exception type, that is, random variables included in the exception template, which can be randomly assigned within the corresponding value range.
[0064] Step 104: perform exception injection processing on the generated exception to perform chaos drill based on the injected exception.
[0065] Afterwards, the anomalies generated for the current chaos drill can be injected into the production environment, such as injecting the anomalies into the corresponding production environment in parallel or serially, and a chaos drill can be performed on the business service or application based on the injected anomalies in the pre-created chaos environment.
[0066] In summary, the chaos drill method, device, and computer-readable medium provided by the present application split a chaos drill into multiple components, each of which corresponds to at least one random variable. When a chaos drill is required, the multiple components included in a chaos drill are obtained, and random values are assigned to the random variables included in each component. According to the random assignment results of the random variables included in each component, the exceptions required for the current chaos drill are generated, and the generated exceptions are injected to perform chaos drills based on the injected exceptions. The present application splits a chaos drill into multiple components, each of which contains at least one random variable. Each drill generates the exceptions required for the drill from these random variables like opening a blind box, so that each chaos drill is almost non-repetitive, ensuring the high randomness of the chaos drill, at least solving the problem of insufficient randomness of chaos drills in the prior art, and having the significance of long-term repeated execution.
[0067] In one embodiment, see Figure 3 The chaos training method flow chart shown in the figure, the chaos training method provided by the present application, before step 101, i.e., obtaining multiple components included in a chaos training, may also include the following processing:
[0068] Step 301, based on the configured scheduled tasks, the daily inspection process is triggered regularly, and a chaotic environment for chaos drills is created based on the daily inspection process, so as to perform anomaly injection and chaos drills based on the injected anomalies based on the created chaotic environment, and end the drill and destroy the chaotic environment when the drill end conditions are met.
[0069] The embodiment of the present application replaces manual operation with machine computing power, and designs a daily inspection mechanism to achieve fully automatic execution of chaos drills.
[0070] Specifically, a set of scheduled tasks can be configured for each application or business service that needs to perform chaos drills to trigger daily inspections regularly. The inspection is divided into three stages: preparing the environment, injecting and rehearsing exceptions, and destroying the environment. The abnormal injection and rehearsal stage is the core of daily inspections. Its logic is to use the blind box factory to continuously generate exceptions and send them to the machine for rehearsal until the end of the rehearsal is reached.
[0071] The timing diagram of daily inspection is as follows: Figure 4 As shown, the task scheduling center can maintain scheduled tasks through the daemon process, and start the daily inspection process based on the scheduled tasks. After the daily inspection process is started, it first enters the preparation stage. In this stage, the task scheduling center first sends the parameter data of the preset creation parameters to the chaos drill platform such as the cloud platform. The preset creation parameters include but are not limited to the domain name set, traffic range and number of drills, where the domain name set represents the domain name corresponding to the server used to create the chaos environment, and the traffic range represents the traffic information used to participate in the chaos drill, such as the terminal / device identifier used to participate in the chaos drill. After receiving the corresponding parameter data, the cloud platform creates the corresponding chaos environment based on the received parameter data. After that, it enters the abnormal injection and drill stage. In this stage, the task scheduling center uses the blind box factory to continuously generate abnormalities and send them to the machine with the chaos environment created in the cloud platform for drill until the drill ends when the drill ends, such as reaching the threshold of time or number of times or finding that the business service / application has risk points or vulnerabilities. After the drill ends, it enters the destruction environment, which can further destroy the chaos environment.
[0072] This embodiment replaces manual operation with machine computing power and designs a daily inspection mechanism to achieve fully automatic execution of chaos drills, thereby overcoming the drawbacks of the prior art of performing drills manually using command lines and other methods.
[0073] In one embodiment, see Figure 5 The chaos training method flow chart shown in the figure, the chaos training method provided by the present application may also include the following processing:
[0074] Step 501: Determine the drill result based on traffic segmentation processing and canary analysis.
[0075] The embodiment of the present application adopts the "traffic segmentation + canary analysis" method to automatically determine the results of the exercise. Figure 6 , the process can be further implemented as:
[0076] Step 601: Divide the inlet traffic into two groups of traffic with the same proportion, which are used as the traffic of the chaos group and the traffic of the control group respectively.
[0077] Ingress traffic refers to public network traffic. The traffic of Internet users is generally requested through the public network to the service nodes directly facing the public network. The dependency between services usually goes through the intranet. On the one hand, the intranet has better network conditions, and on the other hand, the cost of the intranet is lower.
[0078] Optionally, this embodiment first counts the hash distribution of the unique identifier of the inlet traffic within a unit time period, and divides the inlet traffic into two groups of traffic with the same proportion according to the counted hash distribution. For example, the device IDs corresponding to all traffic within 24 hours are counted, these identifiers are segmented according to the first letter of the ID, and two groups of traffic with the same proportion of hash identifiers are divided, respectively as the chaos group and the control group.
[0079] Step 602: Obtain the service indicator monitoring result corresponding to the chaos drill of the service based on the traffic of the chaos group as the chaos group indicator.
[0080] The chaos group is used to perform chaos drills, and when performing chaos drills based on the chaos group traffic, the service indicators are monitored to obtain chaos group indicators.
[0081] The monitored service indicators include but are not limited to UEEC (user behavior performance indicator), HTTP return code, and application log volume.
[0082] Step 603: Obtain the service indicator monitoring result under the operation performance of the service corresponding to the control group traffic as the control group indicator.
[0083] The control group is used to provide a control function. When monitoring the service indicators corresponding to the chaos group traffic in the chaos drill, see Figure 7 , and at the same time monitor the service indicators under the operating performance corresponding to the control group traffic to obtain the control group indicators.
[0084] Step 604: Determine the drill result based on the offset between the chaos group index and the control group index.
[0085] After that, canary analysis is performed on the indicators of the chaos group and the control group. Specifically, the offset of the corresponding indicators in the two groups of monitoring indicators is determined. If the offset is within the set range, it is considered that the service performance is stable under the chaos exercise and there are no problems or loopholes. Otherwise, if the deviation is large and exceeds the set range, it is considered that there are problems / loopholes in the executed exceptions, that is, there are risk points in the service, and the test needs to be stopped and the environment destroyed.
[0086] Subsequently, the drill case, the business services / applications targeted by the drill, the performance of the user clients participating in the drill, and other information can be used as a basis for analyzing risk location, type, and other information to identify risk points where problems may occur.
[0087] This embodiment realizes automatic determination of drill results by proposing a “traffic segmentation + canary analysis” method, thus overcoming the drawback that the prior art cannot automatically determine drill results.
[0088] Corresponding to the above-mentioned chaos training method, the present application also provides a chaos training device, the composition structure of which is as follows: Figure 8 As shown, including:
[0089] The acquisition unit 10 is used to acquire multiple components included in a chaos exercise; wherein a chaos exercise is pre-divided into multiple components, and each component includes at least one random variable;
[0090] A random assignment unit 20 is used to randomly assign values to random variables contained in different component elements to obtain random assignment results of the random variables contained in each component element;
[0091] The anomaly generating unit 30 is used to generate anomalies required for a current chaos exercise according to the random assignment results of random variables contained in different component elements;
[0092] The exception injection and chaos drill unit 40 is used to perform exception injection processing on the generated exceptions to perform chaos drills based on the injected exceptions.
[0093] In one implementation, the acquisition unit 10 is specifically configured to:
[0094] Obtain at least some of the components of an abnormal template, running time, number of abnormalities, and running mode included in a chaos drill;
[0095] The exception templates under different exception types provide different dynamic attributes accordingly, and the dynamic attributes provided by the exception template serve as random variables included in the exception template.
[0096] In one implementation, the running time indicates the duration of an exception in a chaos exercise, the number of exceptions indicates the number of exception types to be executed in a chaos exercise, and the exception mode indicates the execution method between different exceptions in a chaos exercise.
[0097] The exception generating unit 30 is specifically used for:
[0098] Under the conditional constraints corresponding to the running time, the number of exceptions and the current random assignment result of the exception pattern, exceptions are continuously generated based on the exception template under the corresponding exception type.
[0099] In one embodiment, the anomaly injection and chaos exercise unit 40 is specifically used to: inject the generated anomaly into the corresponding production environment, and perform chaos exercise based on the injected anomaly in the pre-created chaotic environment.
[0100] In one embodiment, the above-mentioned device also includes a patrol unit, which is used to: based on the configured scheduled tasks, regularly trigger the daily patrol process, create a chaotic environment for chaos drills based on the daily patrol process, perform anomaly injection and chaos drills based on the injected anomalies based on the created chaotic environment, and end the drill and destroy the chaotic environment when the drill end conditions are met.
[0101] In one embodiment, when the inspection unit creates a chaotic environment for chaos drills based on the daily inspection process, it is specifically used to:
[0102] The task scheduling center sends parameter data of preset creation parameters to the cloud platform based on the daily inspection process, so that the cloud platform creates a chaotic environment based on the parameter data of the preset creation parameters.
[0103] In one embodiment, the above device further comprises:
[0104] The drill result determination unit is used to determine the drill result based on traffic segmentation processing and canary analysis.
[0105] In one embodiment, the drill result determination unit is specifically used to: divide two groups of flows of the same proportion from the inlet flow, respectively serving as the flow of the chaotic group and the flow of the control group;
[0106] Obtain the service indicator monitoring result corresponding to the chaos drill of the service based on the traffic of the chaos group as the chaos group indicator;
[0107] Obtain the service indicator monitoring results of the service under the operation performance corresponding to the control group traffic as the control group indicator;
[0108] The drill results are determined based on the offset between the chaos group indicators and the control group indicators.
[0109] As for the chaos training device provided in the embodiment of the present application, since it corresponds to the chaos training method provided in the above method embodiment, the description is relatively simple. For the relevant similarities, please refer to the description of the above method embodiment, which will not be described in detail here.
[0110] The present application also provides a computer-readable medium having a computer program stored thereon, wherein the computer program includes a program code for executing the chaos training method provided in the above method embodiment.
[0111] In the context of the present application, a computer readable medium (machine readable medium) may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine readable medium may be a machine readable signal medium or a machine readable storage medium. A machine readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0112] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0113] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.
[0114] In summary, the chaos training method, device, and computer-readable medium provided in the embodiments of the present application have at least the following technical advantages:
[0115] 1) Machines replace manual labor, automatically conduct tests on a daily basis, and automatically determine the results, saving a lot of labor costs;
[0116] 2) The drill command is regenerated every time. If there are loopholes / risks in the system architecture, theoretically, they will be discovered if we persist for a long time.
[0117] 3) High scalability. If you need to add new drills, you can just add exception templates.
[0118] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0119] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0120] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0121] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0122] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A chaos training method, characterized in that: The method comprises: Obtain multiple components of a chaos exercise; wherein the chaos exercise is pre-divided into multiple components, and each component includes at least one random variable; Randomly assign values to random variables included in different component elements to obtain random assignment results of the random variables included in different component elements; Generate the anomaly required for the current chaos exercise according to the random assignment results of the random variables contained in different component elements; Perform exception injection processing on the generated exceptions to perform chaos drills based on the injected exceptions; The multiple components of a chaos exercise include: Obtain at least some of the components of an abnormal template, running time, number of abnormalities, and running mode included in a chaos drill; The exception templates under different exception types provide different dynamic attributes accordingly, and the dynamic attributes provided by the exception template serve as random variables included in the exception template.
2. The method according to claim 1, characterized in that The running time indicates the duration of an exception in a chaos drill, the number of exceptions indicates the number of exception types to be executed in a chaos drill, and the running mode indicates the execution mode between different exceptions in a chaos drill; The method of generating the anomaly required for the current chaos exercise according to the random assignment results of the random variables contained in the different component elements includes: Under the conditional constraints corresponding to the running time, the number of exceptions and the current random assignment result of the running mode, exceptions are continuously generated based on the exception template under the corresponding exception type.
3. The method according to claim 2, characterized in that The step of injecting the generated exception into a chaos drill based on the injected exception includes: Inject the generated exceptions into the corresponding production environment, and perform chaos drills based on the injected exceptions in the pre-created chaos environment.
4. The method according to claim 1, characterized in that: Before obtaining the multiple components of a chaos exercise, it also includes: Based on the configured scheduled tasks, the daily inspection process is triggered regularly. Based on the daily inspection process, a chaotic environment for chaos drills is created, and exception injection and chaos drills based on the injected exceptions are performed based on the created chaotic environment. When the drill end conditions are met, the drill is ended and the chaotic environment is destroyed.
5. The method according to claim 1, characterized in that The chaotic environment for chaos drills is created based on the daily inspection process, including: The task scheduling center sends parameter data of preset creation parameters to the cloud platform based on the daily inspection process, so that the cloud platform creates a chaotic environment based on the parameter data of the preset creation parameters.
6. The method according to claim 1, characterized in that Also includes: Determine the drill results based on traffic segmentation and canary analysis.
7. The method according to claim 6, characterized in that The drill results are determined based on traffic segmentation processing and canary analysis, including: Divide the inlet traffic into two groups with the same proportion, which are the traffic of the chaotic group and the traffic of the control group respectively; Obtain the service indicator monitoring result corresponding to the chaos drill of the service based on the traffic of the chaos group as the chaos group indicator; Obtain the service indicator monitoring results of the service under the operation performance corresponding to the control group traffic as the control group indicator; The drill results are determined based on the offset between the chaos group indicators and the control group indicators.
8. A chaos training device, characterized in that: The device comprises: An acquisition unit is used to acquire multiple components contained in a chaos exercise; wherein a chaos exercise is pre-divided into multiple components, and each component element corresponds to at least one random variable; A random assignment unit is used to randomly assign values to random variables contained in different component elements, and obtain random assignment results of the random variables contained in each component element; The anomaly generation unit is used to generate the anomaly required for the current chaos exercise according to the random assignment results of the random variables contained in different component elements; An exception injection and chaos drill unit, used for performing exception injection processing on the generated exceptions, so as to perform chaos drill based on the injected exceptions; The multiple components of a chaos exercise include: Obtain at least some of the components of an abnormal template, running time, number of abnormalities, and running mode included in a chaos drill; The exception templates under different exception types provide different dynamic attributes accordingly, and the dynamic attributes provided by the exception template serve as random variables included in the exception template.
9. A computer-readable medium, characterized in that A computer program is stored thereon, the computer program comprising a program code for executing the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Blockchain network test method and device, medium and electronic equipment
CN112291119A
Random event correlation analysis method and system in chaos project
CN113221096A
Distributed system testing system and method, storage medium and electronic equipment
CN113238950A
Method for realizing observability for cloud native chaos engineering experiment
CN114791846A
