Fault timing control system and fault timing control method

Abstract

Simulate a failure at the appropriate time and state to reproduce the failure. [Solution] The computer in the test execution environment reads the system state and conditions from the trace information at the time of failure, calculates a waiting time to vary the timing of the simulated failure according to the read state, and simulates a failure based on the time of sending the request to simulate the failure and the calculated waiting time.

Description

Technical Field

[0001] The present invention relates to a failure timing control system and a failure timing control method.

Background Art

[0002] The importance of continuously ensuring the security of IT systems is increasing. To address vulnerabilities in underlying software that are continuously and suddenly discovered, it is necessary to sequentially update the foundation. However, system testing for IT systems with a large amount of man-hours cannot always be handled manually and requires automation.

[0003] For example, in a public cloud environment where IT systems are built for multiple tenants, the influence of other tenants cannot be excluded. As an example, the impact of a change in the transaction volume on the IT system of a certain tenant on the IT system of another tenant can be cited. Therefore, Non-Deterministic Bugs that occur depending on specific timings or system states, where the same result cannot be obtained every time the same operation is performed, and Flaky Tests that rarely fail are more likely to occur than in an on-premises environment. Such non-reproducible failures can only be waited for until they recur, and verification is difficult.

[0004] As a technology for solving such problems, for example, there is Patent Document 1. In Patent Document 1, live migration at multiple time zones (multi-point) and CPU lockstep are combined to consistently reproduce bugs that are difficult to reproduce. In particular, it reproduces bugs that occur in multi-threaded race conditions and containerized systems.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

[0006] While Patent Document 1 enables bug analysis, it does not solve problems caused by the timing of external factors such as failures. For example, in a failure test that involves sending a request to perform a test and injecting a simulated failure into an IT system composed of multiple components such as an application server and a data store, it is not possible to reproduce the problem by causing a simulated failure at the appropriate timing and state for a series of processing processes that take place over a predetermined time width (e.g., on the order of milliseconds to tens of milliseconds).

[0007] Therefore, the present invention aims to provide a technology that can simulate a fault at an appropriate timing and state, and reproduce the fault. [Means for solving the problem]

[0008] The fault timing control system according to the present invention is a fault timing control system having a test execution environment that performs tests to simulate faults occurring in the production environment of a system, which are executed by a computer having a processor and memory, wherein the computer in the test execution environment reads a state indicating the state and conditions of the system from trace information at the time of fault occurrence, calculates a waiting time to change the timing of the simulated fault occurrence according to the read state, and simulates the fault based on the time of sending a request to simulate the fault and the calculated waiting time. [Effects of the Invention]

[0009] According to the present invention, it is possible to simulate a fault at an appropriate timing and state, and to reproduce the fault. [Brief explanation of the drawing]

[0010] [Figure 1]This figure shows an example of the configuration of an application system, which is one embodiment of the present invention. [Figure 2] This is a diagram illustrating a schematic example of a computer. [Figure 3] This figure shows the functional configuration of the test execution system shown in Figure 1. [Figure 4] This figure shows an example of the sequence of events that occur when a failure occurs in a production environment. [Figure 5] This figure shows an example of a typical test sequence performed in a verification environment when the failure shown in Figure 4 occurs. [Figure 6] This figure shows an example of a sequence for performing tests with a delay applied in the verification environment when the failure shown in Figure 4 occurs. [Figure 7A] Figure 6 is a flowchart illustrating an example of the processing steps for the test orchestration shown. [Figure 7B] Figure 6 is a flowchart illustrating an example of the processing steps for the test orchestration shown. [Figure 8] This figure shows an example of a failure history table. [Figure 9] This figure shows an example of a simulated failure history table. [Figure 10] This figure shows an example of trace information. [Figure 11] This figure shows an example of trace information for a single request, which is the basis for the trace information shown in Figure 10. [Figure 12] This figure shows an example of a single request pattern where similar trace combinations have been eliminated at each step count. [Figure 13] This figure shows an example of a statetable. [Figure 14] This figure shows an example of a state history table when a pseudo-fault occurs. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The examples are illustrative for explaining the present invention, and for the sake of clarity of the explanation, appropriate omissions and simplifications have been made. The present invention can also be implemented in various other forms. Unless otherwise limited, each component may be singular or plural. In the drawings, the positions, sizes, shapes, ranges, etc. of the components shown may not represent the actual positions, sizes, shapes, ranges, etc. in order to facilitate the understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.

[0012] As examples of various types of information, explanations may be given in expressions such as "table", "list", "queue", etc., but various types of information may also be represented by data structures other than these. For example, various types of information such as "XX table", "XX list", "XX queue" may also be referred to as "XX information". When explaining identification information, expressions such as "identification information", "identifier", "name", "ID", "number", etc. are used, but these are mutually replaceable.

[0013] When there are a plurality of components having the same or similar functions, they may be described with the same reference numeral and different subscripts. Also, when it is not necessary to distinguish these plurality of components, the subscripts may be omitted in the description.

[0014] In the embodiments, the processing performed by executing a program may be described. Here, a computer executes a program by a processor (e.g., a CPU or a GPU), and performs the processing defined by the program while using a storage resource (e.g., a memory) and an interface device (e.g., a communication port), etc. Therefore, the subject of the processing performed by executing the program may be the processor. Similarly, the subject of the processing performed by executing the program may be a controller, a device, a system, a computer, or a node having a processor. The subject of the processing performed by executing the program may be an arithmetic unit, and may include a dedicated circuit for performing a specific processing. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a CPLD (Complex Programmable Logic Device), etc.

[0015] The program may be installed in a computer from a program source. The program source may be, for example, a program distribution server or a storage medium readable by a computer. When the program source is a program distribution server, the program distribution server includes a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Also, in the embodiments, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] FIG. 1 is a diagram showing an example of the configuration of an application system 1 which is an embodiment of the present invention. The application system 1 has a production environment 1100 which is applied as a test target of a test execution system 100 functioning as a failure timing control system, and a verification environment 1000 for reproducing a failure that has occurred in the production environment 1100, and these are communicably connected via a monitoring manager 200.

[0017] The verification environment 1000 is configured as a system environment for reproducing failures that occurred in the production environment 1100. As shown in Figure 1, the verification environment 1000 includes a test execution system 100, communication equipment 300b, one or more data stores 800b (in this example, data stores 800b-1 and 800b-2), and one or more virtual servers 900b (in this example, virtual servers 900b-1, 900b-2, and 900b-3).

[0018] The test execution system 100 is a system that performs test orchestration to simulate failures at appropriate times and states, and reproduces failures that occur in the production environment 1100. The state refers to the state or conditions of the system at a specific point in time (for example, when a failure occurs). The specific configuration of the test execution system 100 will be described later.

[0019] The communication device 300b is, for example, an L3 router, which controls communication between the data store 800b and the virtual server 900b and the test execution system 100.

[0020] Datastore 800b is, for example, a Blob store or RDB, and the data simulates datastore 800b in the production environment 1100.

[0021] Virtual server 900b is, for example, a server for running VMs, and is a server that simulates virtual server 900a in the production environment 1100.

[0022] Furthermore, virtual server 900b has sub-applications 500b (in this example, sub-applications 500b-1, 500b-2, and 500b-3), monitoring agents 600b (in this example, monitoring agents 600b-1, 600b-2, and 600b-3), and reverse proxies 700b (in this example, reverse proxies 700b-1, 700b-2, and 700b-3).

[0023] Sub-application 500b is, for example, a web application, and is an application that simulates sub-application 500a of virtual server 900a in the production environment 1100.

[0024] Monitoring agent 600b is, for example, a VM container and is an agent that simulates monitoring agent 600a of virtual server 900a in the production environment 1100.

[0025] The reverse proxy 700 is a server that performs load balancing at the layer, and in this embodiment, it plays the role of delaying requests for testing in order to cause failures at the timing and state of a desired condition.

[0026] The production environment 1100 is the system environment applied as the test target for the tests performed by the test execution system 100. As shown in Figure 1, the production environment 1100 includes a user terminal 400, a communication device 300a, one or more data stores 800a (in this example, data stores 800a-1 and 800a-2), and one or more virtual servers 900a (in this example, virtual servers 900a-1, 900a-2, and 900a-3).

[0027] User terminal 400 is, for example, a device such as a PC or smartphone operated by a user utilizing the production environment 1100.

[0028] The communication device 300a is, for example, an L3 router, which controls communication between the data store 800a and the virtual server 900a and the user terminal 400.

[0029] Datastore 800a is, for example, a Blob store or RDB, which holds the data necessary for the services that one or more tenants provide to users.

[0030] A virtual server 900a is, for example, a server for running VMs, and a server for running services that one or more tenants provide to users.

[0031] Furthermore, virtual server 900a has sub-applications 500a (in this example, sub-applications 500a-1, 500a-2, and 500a-3) and monitoring agents 600a (in this example, monitoring agents 600a-1, 600a-2, and 600a-3).

[0032] Sub-application 500a is, for example, a web application, and monitoring agent 600a is, for example, a VM container.

[0033] The monitoring manager 200 is a server that monitors both the production environment 1100 and the testing environment 1000, collecting traces, metrics, and logs for each of these environments and storing them in data stores. In this example, the monitoring manager 200 stores the collected traces, metrics, and logs in the trace data store 210, the metrics data store 220, and the log data store 230, respectively. Traces are information collected, for example, to confirm the flow of processing from processing request to response in the production environment 1100. Metrics are information collected, for example, CPU utilization and disk capacity, to understand the internal state of the system in the production environment 1100. Logs are information collected regarding the occurrence and operation status of events processed by the system in the production environment 1100, including information about failures. In this example, the trace data store 210, metrics data store 220, and log data store 230 are maintained separately, but they may be managed and stored together in a single location.

[0034] The components of the production environment 1100, the verification environment 1000, and the monitoring manager 200, such as servers, data stores, and terminals, may be made up of general-purpose computers. For example, such a computer can be realized by a general-purpose computer 1600 equipped with a CPU 1601, memory 1602, an external storage device 1603 such as an HDD (Hard Disk Drive), a reader 1607 for reading and writing information to a portable storage medium 1608 such as a CD (Compact Disk) or USB memory, an input device 1606 for receiving various types of information such as a keyboard, mouse, reader, and scanner, an output device 1605 such as a display for outputting various types of information that are input and used for processing, a communication device 1604 such as a NIC (Network Interface Card) for connecting to a communication network, and an internal communication line (referred to as a system bus) 1609 such as a system bus that connects these.

[0035] Furthermore, various data stored in the monitoring system 200 or used for processing can be realized by the CPU 1601 reading and using data from memory 1602 or external storage device 1603. In addition, each function of the monitoring system 200 can be realized by the CPU 1601 loading a predetermined program stored in external storage device 1603 into memory 1602 and executing it.

[0036] The aforementioned programs and data may be stored (downloaded) from the storage medium 1608 via the reading device 1607, or from the network via the communication device 1604, into the external storage device 1603, and then loaded onto the memory 1602 and executed by the CPU 1601. Alternatively, they may be loaded directly onto the memory 1602 via the reading device 1607 from the storage medium 1608, or from the network via the communication device 1604, and then executed by the CPU 1601.

[0037] Figure 3 shows the functional configuration of the test execution system 100 shown in Figure 1. As mentioned above, the test execution system 100 may be configured as a general-purpose computer in terms of hardware. As shown in Figure 3, the test execution system 100 includes a state reproduction program 110, a timing control program 115, a request transmission / reception program 120, a delay control program 125, a failure request program 130, a pseudo-failure history table 150, a state table 151, and a state history table 152 for when a pseudo-failure occurs. The processing performed by each program and the data stored in each table will be described later.

[0038] Figure 4 shows an example of the sequence of events when a failure occurs in the production environment. As shown in Figure 4, when a failure occurs in the production environment 1100, the user terminal 400 sends a processing request to the sub-application 500a-1 (Web), which is a web application (S401), and then the sub-application 500a-1 sends a logic processing request to another sub-application 500a-2 (App) that is called from the web application (S402).

[0039] In the other sub-application 500a-2, processing is performed according to the logic processing request described above, and the blob data is stored in the data store 800a-1 which is the storage location for the blob data (S403), and then the metadata is stored in the data store 800a-1 which is the storage location for the metadata (S404). At this time, due to some failure SF occurring in the other sub-application 500a-2, the response to the logic processing request may be sent with a delay (S405), and the response to the processing request may be sent to sub-application 500a-1 (S406). In the production environment 1100, for example, applications from various tenants are running and transactions are changing moment by moment, but if one tries to reproduce the above failure, it is necessary to reproduce the failure that occurred in the few milliseconds to tens of milliseconds between the processing request in S401 and the response in S406 under such circumstances.

[0040] Figure 5 shows an example of a test sequence typically performed in the verification environment when the failure shown in Figure 4 occurs. As shown in Figure 5, in the verification environment 1000, for example, in order to perform a test against some failure SF shown in Figure 4, the test execution system 100 performs test orchestration according to a predetermined test plan (S501). The test execution system 100 sends a processing request to the sub-application 500b-1 (Web), which is a web application, including a request to perform a test according to the above test plan (S502), and the sub-application 500b-1 then sends the processing request to the reverse proxy 700b-1 of the web application (S503).

[0041] The reverse proxy 700b-1 sends a logic processing request to the other sub-application 500b-2 (App) that is called from the web application, via the reverse proxy 700b-2 of the other sub-application 500b-2 that is called from the web application (S504). The reverse proxy 700b-2 sends the logic processing request received in S504 to the other sub-application 500b-2 (S505).

[0042] Other sub-applications 500b-2 perform the processing related to the above test in accordance with the logic processing request above and send the results to the reverse proxy 700b-2 (S506). The reverse proxy 700b-2 stores the blob data in the data store 800b-1, which is the storage destination for the blob data, on behalf of the other application 500b-2 (S507), and then sends the results to the other sub-application 500b-2 (S508).

[0043] Other sub-applications 500b-2 send a request to the reverse proxy 700b-2 to store metadata (S509), and the reverse proxy 700b-2 stores the metadata in the data store 800b-1, which is the destination for storing the metadata (S510). The reverse proxy 700b-2 notifies the other sub-applications 500b-2 that the storage of blob data and metadata is complete (S511), and the other sub-applications 500b-2 send the processing results of the processing performed in accordance with the logic processing request above to the reverse proxy 700b-2 (S512).

[0044] The reverse proxy 700b-2 sends a response to the logic processing request received in S505 to the reverse proxy 700b-1 (S513). The reverse proxy 700b-1 sends the response received from the reverse proxy 700b-2 to the sub-application 500b-1 (S514), and the sub-application 500b-1 sends a response to the processing request to the reverse proxy 700b-1 (S515). The reverse proxy 700b-1 sends the response received from the sub-application 500b-1 to the test execution system 100 (S516). In the verification environment 1000, the processes from S502 to S516 are repeatedly executed according to a test plan that combines various conditions to reproduce failures that occurred within a few milliseconds to tens of milliseconds in the test plan of the test orchestration created in S501.

[0045] However, if the production environment 1100 is, for example, a public cloud environment where IT systems are built for multiple tenants, the influence of other tenants cannot be eliminated, and it cannot be said that failures occurring within a few milliseconds to tens of milliseconds can be reproduced with accuracy. As already explained, changes in the transaction volume for one tenant's IT system may affect the IT systems of other tenants. Therefore, in the failure test shown in Figure 5, in sending test requests and injecting simulated failures, it is not possible to cause simulated failures at the appropriate timing and state for a series of processing processes that take place over a predetermined time width (e.g., on the order of milliseconds to tens of milliseconds) and reproduce the failure. Therefore, in this embodiment, we consider the case where a delay is added to the test in the verification environment 1000.

[0046] Figure 6 shows an example sequence for performing tests with a delay imposed in the verification environment when the failure shown in Figure 4 occurs. In Figure 6, the procedure is the same as in Figure 5, but a predetermined delay time defined in the test plan is introduced when the proxy servers (reverse proxies 700b-1 and 700b-2) and the application server (other sub-applications 500b-2) exchange data. This allows for accurate reproduction of failures that may occur in milliseconds.

[0047] Figure 6 shows an example of the test sequence performed in the verification environment in this embodiment when the failure shown in Figure 4 occurs. In Figure 6, as in Figure 5, when test orchestration is performed (S601), the test execution system 100 sends a simulated failure request to the application server (other sub-application 500b-2) where the failure occurred (S6011). At this time, the test execution system 100 refers to the simulated failure history table 150, which will be described later, and sets the trigger time of a previously simulated failure stored in association with the failure type to be tested and the location of the failure (in this case, the other sub-application 500b-2). The other sub-application 500b-2 obtains logs, traces, and metrics from the trigger time to the failure occurrence time (or response time) via the test execution system 100 (S6012).

[0048] Subsequently, as in the case of Figure 5, the test execution system 100 sends the above processing request to the sub-application 500b-1 (S602), and the sub-application 500b-1 sends the processing request to the reverse proxy 700b-1 of the web application (S603).

[0049] After the reverse proxy 700b-1 counts that a predetermined delay period (for example, on the order of milliseconds to tens of milliseconds) has elapsed (S6031), it sends a logic processing request to the other sub-application 500b-2 to the reverse proxy 700b-2, similar to the case in Figure 5 (S604). The reverse proxy 700b-2 sends the above logic processing request to the other sub-application 500b-2 after the predetermined delay period has elapsed (S6041), similar to S6031 (S605).

[0050] Other sub-applications 500b-2 perform the processing related to the above test in accordance with the logic processing request above and send the result to the reverse proxy 700b-2 (S606). Similar to S6041, after a predetermined delay period has elapsed (S6061), the reverse proxy 700b-2 stores the blob data in the data store 800b-1, which is the storage location for the blob data, on behalf of the other application 500b-2 (S607). Furthermore, similar to S6061, after a predetermined delay period has elapsed (S6071), it sends the result to the other sub-application 500b-2 (S608).

[0051] In the other sub-application 500b-2, as in the case of Figure 5, a request to store metadata is sent to the reverse proxy 700b-2 (S609). The reverse proxy 700b-2, as in S6071, stores the metadata in the data store 800b-1, which is the destination for storing the metadata, after a predetermined delay period has elapsed (S6091) (S610). The reverse proxy 700b-2, as in S6091, notifies the other sub-application 500b-2 that the storage of the blob data and metadata is complete after a predetermined delay period has elapsed (S6101) (S611). Furthermore, the other sub-application 500b-2 sends the processing results of the processing performed in accordance with the above logic processing request to the reverse proxy 700b-2 (S612).

[0052] Similar to S6101, after a predetermined delay period has elapsed (S6121), the reverse proxy 700b-2 sends a response to the logic processing request to the reverse proxy 700b-1, as in the case of Figure 5 (S613). Similar to S6121, after a predetermined delay period has elapsed (S6131), the reverse proxy 700b-1 sends the response received from the reverse proxy 700b-2 to the sub-application 500b-1 (S614), and the sub-application 500b-1 sends a response to the processing request to the reverse proxy 700b-1 (S615). Similar to S6131, after a predetermined delay period has elapsed (S6151), the reverse proxy 700b-1 sends the response received from the sub-application 500b-1 to the test execution system 100 (S616). In this embodiment, for each of the application servers that experienced a failure (another sub-application 500b-2) and the proxy servers (reverse proxies 700b-1 and 700b-2) that exchange data with the said application server, a predetermined delay time is set according to the test plan, and the test is repeatedly executed according to a test plan that combines various conditions to reproduce the failure that occurred between a few milliseconds and tens of milliseconds.

[0053] Figures 7A and 7B are flowcharts showing an example of the processing procedure for the test orchestration shown in Figure 6.

[0054] First, the state reproduction program 110 requests information from the failure history table 5000a in the production environment 1100, which is held by the monitoring manager 200, and selects the failure to reproduce (S3000). The failure history table 5000a is a table that stores failures that have occurred in the production environment 1100.

[0055] Figure 8 shows an example of the failure history table 5000a. The failure history table 5000a is a table that stores information about failures that occurred in the production environment 1100 managed by the monitoring manager 200, and in this system, it is assumed that this information is stored in advance. The items that the failure history table 5000a holds are almost the same as those of the pseudo-failure history table 150, which will be described later, although it does not hold columns related to "trigger time" and "time until occurrence time". The monitoring manager 200 obtains and generates this information from traces, logs, metrics, etc. of the production environment 1100.

[0056] As shown in Figure 8, the failure history table 5000a stores the failure identifier to identify the failure that occurred, the failure type indicating the type of failure that occurred, the failure location indicating the location of the failure, and the failure time indicating the time when the failure occurred. In Figure 8, for example, the failure identified by the failure identifier "F001" is a failure of failure type "Network Issue" that occurred at the failure location "Application Server", and the failure time indicates that the failure occurred at "2024-08-23T12:00:02Z".

[0057] Next, the timing control program 115 refers to the pseudo-fault history table 150 and calculates the reflection time (e.g., mean and deviation) of the pseudo-fault for entries that contain the same fault identifier as the fault identifier selected in step S3000 (S3100). If no identical entries exist, a predefined value is used. The reflection time is the time from the trigger time, which is the time when the pseudo-fault request shown in Figure 6 was sent, to the time when the pseudo-fault occurred. The reflection time may also be the time from sending the pseudo-fault request to the response. The time when the pseudo-fault request was sent may be called the reference time. In this case, the start time of each test defined in the test orchestration becomes the reference time.

[0058] Figure 9 shows an example of a simulated failure history table 150. The simulated failure history table 150 is a table that defines the conditions for reproducing a failure that occurred in the production environment 1100. As shown in Figure 9, in addition to the items held by the failure history table 5000a described above, the simulated failure history table 150 holds columns related to "trigger time," which indicates the time when a simulated failure request was sent as a trigger to simulate a failure at the failure location, and "time until occurrence time," which indicates the time from the trigger time to the failure occurrence time, which is the actual time when the simulated failure occurred.

[0059] Figure 9 shows, for example, that the fault identified by fault identifier "F001" is a fault of type "Network Issue" that occurred at the fault location "Application Server" and occurred at the fault time "2024-08-23T12:00:02Z". It also shows that the reflection time, which is the time from the trigger time "2024-09-23T12:00:00Z" to the fault occurrence time, is "2200" seconds.

[0060] Next, the state reproduction program 110 acquires all trace information where the trace start time is earlier than the fault occurrence time (Figure 8) corresponding to the fault identifier of the "fault to reproduce" selected in step S3000, and the trace end time is later than the fault occurrence time (S3125).

[0061] Figure 10 shows an example of trace information 5000b. Trace information 5000b is a table that stores information regarding the tracing of failures that occurred in the production environment 1100 managed by the monitoring manager 200, and it is assumed that this information is stored in advance. Trace information 5000b is information that stores the trace of requests actually made in the production environment 1100, and is stored in the trace data store 210 shown in Figure 1. As shown in Figure 10, trace information 5000b stores a trace identifier to identify the trace information, a trace start time indicating the time when the trace started, and a trace end time indicating the time when the trace ended, all linked together. In Figure 10, trace information for the production environment 1100 is shown as an example, but the same information is stored for the verification environment 1000.

[0062] Figure 10 shows, for example, that trace information identified by the trace identifier "987fabc3-e123-4567-89ab-cdef12345678" is information traced from the trace start time "2024-09-23T12:00:00Z" to the trace end time "2024-09-23T12:00:03Z". This information can be obtained from trace information (traces managed by monitoring manager 200) that includes the content and results of one or more processes performed sequentially in a single request.

[0063] Figure 11 shows an example of trace information for a single request, which is the source of the trace information shown in Figure 10. As shown in Figure 11, a single request contains trace information for processing in one or more steps. In Figure 11, it can be seen that the trace identifier, trace start time, and trace end time of the trace information shown in Figure 10 are obtained from "trace_id," "start_time," and "end_time," respectively. The trace information shown in Figure 11 consists of trace information for processing performed in each step from step 1 to step 7. In other words, when this request is executed, the state showing the result of processing performed in the seven steps is stored in the trace information. In Figure 11, for example, the processing result is described as state in each item such as "payload," "status," and "errors."

[0064] Returning to Figure 7, the state reproduction program 110 selects all traces from the trace information (Figures 10 and 11) acquired in step S3125 that contain the "fault location" (Figure 8) corresponding to the fault identifier of the "fault to reproduce" selected in step S3000 (S3150).

[0065] The state reproduction program 110 acquires all combinations of traces selected in step S3150 whose number of traces is less than or equal to a predetermined upper limit (S3175). For example, it acquires trace information (Figure 11) that includes the “fault location” (Figure 8) above, with a number of steps of 7 or less. As a result, one or more combinations of trace information are acquired for each number of steps.

[0066] The state reproduction program 110 stores the trace information of combinations obtained in step S3175 as a request pattern, excluding similar trace combinations at each step number (S3180). Similar trace combinations are, for example, those in the traces stored as a request pattern where the "method," "endpoint," and "payload" are the same, and the difference time from the time specified as "start_time" is within a predetermined time.

[0067] Figure 12 shows an example of a request pattern in which similar trace combinations have been eliminated for each step count. In Figure 12, trace information with two steps is stored as one type of request ("type_id" is "r1"), where "method" is "POST", "endpoint" is " / api / upload", and "payload" is "". Similarly, a second request ("type_id" is "r2") is stored. The requests stored in this way become requests to execute the steps of the trace information shown in Figure 11. In Figure 12, as a result of eliminating the similar traces mentioned above from the combination of trace information obtained in step S3175 (trace information with a step count of "2"), two types of traces are included, and these are registered as one request pattern (request pattern identifier (id) is "RP001").

[0068] Returning to Figure 7, the state reproduction program 110 selects an unselected request pattern from the request patterns obtained in step S3180 and issues a request pattern identifier (S3200). In the example in Figure 12, the request pattern obtained from the trace information with step number "2" is registered as request pattern identifier (id) "RP001", so request pattern identifiers are also issued for the other request patterns.

[0069] The state reproduction program 110 obtains the possible states (S3250) for the above request pattern based on the trace combination information held in step S3180. For example, the state reproduction program 110 refers to the original trace information (Figure 11) for each step included in the request pattern identified by each request pattern identifier and reads the state at each step.

[0070] The state reproduction program 110 checks if there is an entry in the state table 151 where the request pattern identifier and step stored there match both the request pattern selected in step S3200 and the state identified in step S3250. If no such entry exists, it records it as a new entry in the state table (S3255).

[0071] Figure 13 shows an example of a state table 151. The state table 151 is generated by the state reproduction program 110. As shown in Figure 13, the state table 151 stores the following in association: a request pattern identifier for identifying a request pattern, the state of the request pattern, the time from the reference time which is the time the simulated failure request was sent, the average duration of the state which indicates the average time the state has persisted, and the standard deviation of the state duration which indicates the average time the state has persisted. The reference time is the start time of each test defined in the test orchestration. The request pattern identifier is information for associating with the request pattern 4100. The state at the time of the request for the request pattern identified by the request pattern identifier is represented by a combination of the states of each step included in the request. For example, in the request pattern shown in Figure 12, since it includes two steps, four different states, which are combinations of the states of each of these steps, are registered as the states of the request pattern.

[0072] Figure 13 shows, for example, that for a request pattern identified by the request pattern identifier "RP001", the time to reach state "r1_step1" is "3.3 sec", the average duration of that state is "130 msec", and the standard deviation of the state duration is "10 msec". In this example, since there are four possible combinations of each step included in the request pattern shown in Figure 12, the "time from the reference time", "average duration of the state", and "standard deviation of the state duration" are stored for each state.

[0073] The state of a request in the request pattern shown in Figure 12 consists of a combination of the request type identifier described in request pattern 4100 and the steps included in the trace associated with that request. In this example, the request pattern contains two requests, but the state of the request is expressed as "r1_step1" based on the "type_id": "r1" in request pattern 4100 and the "step": 1 described in the corresponding request trace information 4000. Similarly, the states of the requests are expressed as r1_step1 and r2_step1 based on the "type_id": "r2" in request pattern 4100 and the "step": 1 described in the corresponding request trace information 4000.

[0074] "Time from reference time" is the time from the "trigger time" in the simulated failure history table 150 shown in Figure 9, and can be obtained, for example, by reading the time (e.g., timestamp) until the state described in the trace information (Figures 10 and 11) is output. "Average state duration" is obtained by calculating the value obtained by subtracting the "state start time" from the "state end time" for each entry with the same request pattern identifier and state in the state history table 152 when a simulated failure occurs (described later), and then calculating the average value. In addition, "standard deviation of state duration" is obtained by calculating the value obtained by subtracting the "state start time" from the "state end time" for each entry with the same request pattern identifier and state in the state history table 152 when a simulated failure occurs, and then calculating the standard deviation.

[0075] Figure 14 shows an example of a state history table when a pseudo-failure occurs. The state history table 152 when a pseudo-failure occurs is a table that stores the history of states when pseudo-failures occurred in the past, and is generated by the state reproduction program 110. As shown in Figure 14, the state history table 152 when a pseudo-failure occurs stores the same request pattern identifier and state as the state table 151 in Figure 13, along with the state of the pseudo-failure that occurred in that request pattern identifier and state, and the start and end times of that state.

[0076] Figure 14 shows, for example, that the pseudo-failure state generated when the request pattern identifier "RP001" is in the "r1_step1" state is "987fabc3-e123-4567-89ab-cdef12345678_step1", and that the start and end times of this state were "2024-09-23T12:00:00Z" and "2024-09-23T12:00:02Z", respectively. The start and end times of the state can be obtained from the trace information shown in Figures 10 and 11. It also shows that the pseudo-failure state "987fabc3-e123-4567-89ab-cdef12345678_step1" is associated with the trace identifier in the trace information (Figures 10 and 11) when the pseudo-failure occurred.

[0077] Returning to Figure 7, the delay control program 125 refers to the state table 151 and selects an unselected state from the possible states (S3300). For example, the delay control program 125 selects the "r1_step1" state for the request pattern identifier "RP001" from the state table 151 shown in Figure 13.

[0078] The delay control program 125 calculates the waiting time to be generated for the state selected in S3300. If the calculated waiting time is within the range of the above reflection time, the delay control program 125 adopts it as a waiting time to artificially alter the timing of generating a failure and sets it in the reverse proxy (S3350). For example, the delay control program 125 reads the time from the reference time "3.3 sec" corresponding to the "r1_step1" state of request pattern identifier "RP001" and the reflection time calculated in S3100, and if the time from the reference time "3.3 sec" is within the range of the above reflection time, it sets that time as a waiting time in the reverse proxy (reverse proxy 700b-1, 700b-2).

[0079] The failure request program 130 requests the occurrence of a simulated failure, and as explained in S6011 in Figure 6, the request sending / receiving program 120 sends the request selected in step S3200 to the application server where the failure occurred (another sub-application 500b-2) (S3400).

[0080] The fault request program 130 requests the monitoring manager 200 to obtain the time of occurrence of the simulated fault requested in step S3400 (S3410). Since the time of occurrence of the simulated fault is recorded in the traces, metrics, and logs of the monitoring manager 200, the above time of occurrence can be obtained by searching this information (for example, traces and logs within a predetermined time period from the time the simulated fault request was sent in step S3400).

[0081] The fault request program 130 stores the fault information corresponding to the occurrence time obtained in step S3410 as a new entry in the pseudo-fault history table 150 shown in Figure 9 (S3425). For example, the fault request program 130 stores the fault identifier, fault type, and fault location items in the pseudo-fault history table 150 shown in Figure 9 as the same fault identifier as the fault identifier selected from the fault history table 5000a in step S3000, along with the corresponding fault type and fault location. The fault request program 130 also stores the time when it sent the pseudo-fault request in step S3400 as the trigger time, and the time obtained in step S3410 as the fault occurrence time. Furthermore, the fault request program 130 stores the value obtained by subtracting the trigger time from the fault occurrence time as the time until occurrence.

[0082] Next, the failure request program 130 obtains trace information of the request sent in step S3400 from the monitoring manager 200 and stores it as a new entry in the simulated failure state history table 152 shown in Figure 14 (S3430).

[0083] The failure request program 130 updates the time from the reference time, the average duration of the state, and the standard deviation of the state duration for entries in the state table 151 where the request pattern identifier and state match both the request pattern identified in step S3200 and the state identified in step S3250 (entries recorded in S3255) (S3440).

[0084] Subsequently, the failure request program 130 determines whether or not the failure has been reproduced (S3450). Whether or not the failure has been reproduced can be determined by whether or not the same logs, traces, and metrics as those of the production environment 1100 have been output.

[0085] If the failure request program 130 determines that the failure has not been reproduced (S3450; No), it determines whether all states have been selected or not (S3460). If the failure request program 130 determines that the failure has been reproduced (S3450; Yes), it terminates processing. If the failure request program 130 determines that not all states have been selected (S3460; No), it returns to S3300, selects another unselected state, and repeats the subsequent processing.

[0086] On the other hand, if the failure request program 130 determines that all states have been selected (S3460; Yes), it determines whether all requests have been selected or not (S3470). If the failure request program 130 determines that not all requests have been selected (S3470; No), it selects another unselected request pattern and repeats the subsequent processing.

[0087] If the failure request program 130 determines that all requests have been selected (S3470; Yes), it proceeds to step S3480. Step S3480 is the case where the failure was not reproduced in step S3450. Therefore, the failure request program 130 determines that the sampling rate monitored by the monitoring manager 200 for the production environment 1100 may not meet the predetermined criteria and is insufficient, and requests the monitoring manager 200 to increase the sampling rate of trace, metrics, or both (S3480). The monitoring manager 200 sets the sampling rate of trace, metrics, or both to be higher than before in accordance with the request. In S3480, the sampling rate of trace, metrics, or both was used as an example, but the sampling rate of metrics may also be set to be higher than before. When the processing in step S3480 is completed, the processing shown in Figures 7A and 7B is completed (S3500).

[0088] As explained above, according to this system, as explained in Figure 6, Figure 77A S3350, etc., in a failure timing control system (e.g., application system 1) having a test execution environment (e.g., verification environment 1000) that performs tests to simulate failures that occur in the system's production environment (e.g., production environment 1100) executed by a computer having a processor and memory, the computer in the test execution environment (e.g., test execution system 100) reads the state indicating the state and conditions of the system from the trace information at the time of failure occurrence (e.g., Figures 10, 11), calculates a waiting time to change the timing of the simulated failure according to the read state, and simulates the failure based on the time of sending the request to simulate the failure and the calculated waiting time. This makes it possible to simulate a failure at the appropriate timing and state and reproduce the failure.

[0089] Furthermore, as explained in the loop S3300-S3460 in Figure 7A, the computer in the test execution environment reproduces the failure that occurred in the production environment by repeatedly generating simulated failures while changing the read state and calculating the waiting time after the state change. This improves the accuracy of failure reproduction.

[0090] Furthermore, as explained in S3125 of Figure 7A, the computer in the test execution environment acquires trace information where the trace start time is earlier than the time the failure occurred and the trace end time is later than the time the failure occurred, and calculates the waiting time according to the state of the acquired trace information. This makes it possible to narrow down the trace information that is to be reproduced and reproduce the failure efficiently.

[0091] Furthermore, as explained in S3150, etc., in Figure 7A, the computer in the test execution environment selects the traces from the acquired trace information that contain the locations of the failures that occurred in the production environment of the system. This ensures that failures that occurred in the production environment can be reliably reproduced.

[0092] Furthermore, as explained in S3350, etc., in Figure 7A, the computer in the test execution environment adopts the above-mentioned waiting time as the waiting time if it falls within the reflection time, which is the time from the time the request is sent until the time the simulated failure occurs. This allows the waiting time to be set from the time the request is sent until the time the simulated failure occurs, and after the waiting time has elapsed, subsequent processing can be executed to continue the test. For example, in the example in Figure 13, there are four states for one request pattern, so a waiting time can be set according to each state and the test can be executed.

[0093] Furthermore, as explained in S3480, etc., in Figure 7B, if the computer in the test execution environment fails to simulate the failure, it increases the sampling rate of the trace information. This increases the probability of simulating a failure.

[0094] The present invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and implemented without departing from the gist of the invention, or the multiple components disclosed in the embodiments can be appropriately combined. [Explanation of Symbols]

[0095] 1. Applicable Systems 100 Test Execution Systems 1100 Production Environment 1000 Verification Environments 200 Monitoring Managers 300a, 300b communication equipment 800a, 800b datastore 900a, 900b virtual servers 110 State Reproduction Program 115 Timing control program 120 Request Sending and Receiving Program 125 Delay Control Program 130 Incident Request Program 150 Pseudo-fault history table 151 State Tables 152 State history table when a pseudo-fault occurs

Claims

1. A fault timing control system having a test execution environment that runs on a computer having a processor and memory, and performs tests that simulate faults occurring in the production environment of the system, The computer in the aforementioned test execution environment is From the trace information at the time of the failure, the system reads the state indicating the state and conditions of the system, and calculates a waiting time to simulate a different timing for the failure to occur according to the read state. Based on the time of sending the request to simulate the failure and the calculated waiting time, the system simulates the failure. A fault timing control system characterized by the following:

2. The computer in the aforementioned test execution environment is By repeatedly generating a simulated failure while changing the read state and calculating the waiting time after the state change, the failure that occurred in the production environment is reproduced. The fault timing control system according to feature 1.

3. The computer in the aforementioned test execution environment is Obtain trace information where the trace start time is earlier than the time the failure occurred, and the trace end time is later than the time the failure occurred. The waiting time is calculated according to the state of the acquired trace information. The fault timing control system according to feature 1.

4. The computer in the aforementioned test execution environment is From the acquired trace information, select the trace that contains the location of the failure that occurred in the production environment of the system. The fault timing control system according to claim 3.

5. The computer in the aforementioned test execution environment is The aforementioned waiting time is adopted if it falls within the range of the reflection time, which is the time from the time the request was sent to the time the pseudo-failure occurred. The fault timing control system according to feature 1.

6. The computer in the aforementioned test execution environment is If the aforementioned failure cannot be simulated, the sampling rate of the trace information will be increased compared to before. The fault timing control system according to feature 1.

7. A failure timing control method having a test execution environment that performs tests to simulate failures that occur in the production environment of a system, which are executed by a computer, From the trace information at the time of the failure, the system reads the state indicating the state and conditions of the system, and calculates a waiting time to simulate a different timing for the failure to occur according to the read state. Based on the time of sending the request to simulate the failure and the calculated waiting time, the system simulates the failure. A fault timing control method characterized by the following.

Images