Detection device, detection method, and detection program

Abstract

To provide a detection apparatus, a detection method, and a detection program for accurately detecting a deficiency of test execution performed on a system in which a plurality of components communicate messages with each other.SOLUTION: A test failure specification apparatus (detection apparatus) includes: a body test case abstraction unit for specifying a test pattern in a first test by abstracting information of a message communicated by a component to be tested in a first test; a test pattern association unit for determining whether or not the specified test pattern can be associated with a message of each component in a second test performed by communicating messages among a plurality of components to be tested, and, if the association is possible, associating the message with the test pattern; and a comprehensive test completeness derivation unit 1501 (detection unit) for detecting a deficiency in the pattern of the message in the first test or the second test based on a result of the association.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a detection device, a detection method, and a detection program. [Background technology]

[0002] In recent years, system development based on microservice architecture has been gaining attention in software development. Microservice architecture (MSA) refers to a system architecture in which system functions are implemented as small services (hereinafter referred to as "microservices") and realized through intercommunication between microservices, either synchronously using REST APIs (Representational State Transfer Application Programming Interfaces) or asynchronously using messaging services. Because individual microservices in MSA are small and loosely coupled, changes and deployment can be made on a microservice-by-microservice basis, enabling rapid, gradual development in response to changing market needs. However, a challenge is that functions are realized by calling multiple microservices via synchronous or asynchronous communication, which leads to complex dependencies between microservices in terms of which microservices each call to realize a particular function.

[0003] The complexity of dependencies between microservices also affects the testing process for systems that use an MSA configuration. The scope of testing for systems that use an MSA configuration can be broadly divided into tests of the classes that make up the microservices, tests that confirm the behavior of individual microservices, and tests of the functionality or entire system when multiple microservices are combined. When testing multiple microservices together, the complexity of the dependencies between microservices makes it difficult to grasp the test coverage within that test scope or to find missed tests.

[0004] On the other hand, in unit testing of microservices, the scope of testing is localized to a single microservice by using a mock of the dependent microservice, which is a microservice that simulates the behavior of the microservice by pre-specifying the response to a request when calling the microservice.However, there is a possibility that imperfections in the mock, such as the response value to a request in the mock setting differing from the behavior of the actual microservice, or the request and response pairs set in the mock being limited compared to the actual microservice, could result in imperfections in the testing of the microservice unit or missed tests.

[0005] To address this issue, the technology described in Patent Document 1 compares the information in the system specifications with data patterns such as the number of digits or data type of data previously input to the system to generate a data pattern that conforms to the current specifications.Then, by reflecting test data that was input in past tests in the generated data pattern, highly comprehensive test data that conforms to the current specifications is derived.

[0006] Furthermore, in the technology described in Patent Document 2, in an information processing system in which a plurality of components operate in cooperation with each other, test cases for other similar components are added based on test cases used in unit tests of a certain component. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-140166 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-253465 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the technology described in Patent Document 1 uses information from specifications, but because the format of specifications varies depending on the project, it is difficult to mechanically collect information from specifications. Furthermore, both Patent Document 1 and Patent Document 2 can only expand test coverage for the same test scope or identify missed tests using information from a single test scope. Unexpected requests and responses to a microservice that could lead to defects in a microservice unit test may not be discovered unless a system similar to the production environment is executed, such as by not using a mock for the called microservice or by using a database for the production system. In such cases, applying the methods described in Patent Document 1 and Patent Document 2 to microservice unit testing does not necessarily result in sufficient detection of test defects.

[0009] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a detection device, a detection method, and a detection program that can accurately detect insufficient test execution for a system in which multiple components communicate messages with each other. [Means for solving the problem]

[0010] One aspect of the present invention for solving the above problem is a detection device having a processor and memory, which detects insufficient test execution of components performed on a system in which multiple components communicate messages with each other, and which includes: an abstraction unit that identifies a test pattern in a first test by abstracting information about messages communicated by the components to be tested in the first test; a test pattern linking unit that determines whether the identified test pattern can be associated with messages of each component in a second test performed by communicating messages between multiple components to be tested, and if so, associates the message with the test pattern; and a detection unit that detects insufficient message patterns in the first test or the second test based on the results of the association. [Effects of the Invention]

[0011] According to the present invention, it is possible to accurately detect insufficient test execution performed on a system in which a plurality of components communicate messages with each other. Configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 illustrates an example of a system under test. [Figure 3] FIG. 2 is a block diagram illustrating an example of functions of a test deficiency identifying device. [Figure 4] FIG. 10 is a diagram showing an example of comprehensive test request and response information. [Figure 5] FIG. 10 is a diagram illustrating an example of unit test request and response information. [Figure 6] FIG. 10 is a diagram illustrating an outline of a process performed by the test deficiency identifying device. [Figure 7] FIG. 10 is a process flow diagram illustrating details of unit test abstraction processing. [Figure 8] FIG. 10 is a processing flow diagram illustrating details of a request abstraction process. [Figure 9] FIG. 10 is a diagram illustrating an example of request pattern information. [Figure 10] FIG. 10 is a process flow diagram illustrating details of a response abstraction process. [Figure 11] FIG. 10 is a diagram illustrating an example of response pattern information. [Figure 12] FIG. 10 is a diagram illustrating an example of unit test pattern information. [Figure 13] FIG. 10 is a process flow diagram illustrating details of a request and response pattern linking process. [Figure 14] FIG. 10 is a diagram illustrating an example of pattern linking result information. [Figure 15]FIG. 10 is a process flow diagram illustrating details of unit test pattern linking processing. [Figure 16] FIG. 10 is a process flow diagram illustrating details of the overall test coverage derivation process. [Figure 17] FIG. 10 is a diagram illustrating an example of comprehensive test coverage information. [Figure 18] FIG. 10 is a diagram illustrating an example of a comprehensive test coverage visualization screen. [Figure 19] FIG. 10 is a diagram showing another example of the comprehensive test coverage visualization screen. [Figure 20] FIG. 10 is a process flow diagram illustrating details of a process for estimating the cause of a unit test not being performed. [Figure 21] FIG. 10 is a process flow diagram illustrating details of a process for estimating the cause of a unit test not being performed. [Figure 22] FIG. 10 is a diagram showing an example of comprehensive test request and response information to be inferred. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The following description and drawings are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings. In the following explanation, various types of information may be described using expressions such as "table," "list," and "queue," but the various types of information may also be expressed using data structures other than these. To indicate that the information is not dependent on the data structure, "XX table," "XX list," etc. may be referred to as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable. When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. However, when there is no need to distinguish between these multiple components, the subscripts may be omitted. Furthermore, in the following description, processing performed by executing a program may be described, but the program is executed by a processor (e.g., a CPU or a GPU) to perform the specified processing while appropriately using storage resources (e.g., a memory) and / or interface devices (e.g., a communication port), and therefore the subject of the processing may be the processor. Similarly, the subject of the processing performed by executing a program may be a controller, device, system, computer, or node having a processor. The subject of the processing performed by executing a program may be any computing unit, and may include a dedicated circuit (e.g., an FPGA or an ASIC) that performs a specific processing. A program may be installed on a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. If the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Also, in the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0014] <System configuration> FIG. 1 is a diagram showing an example of the configuration of an information processing system 1 according to this embodiment. The information processing system 1 includes a test deficiency identification device 10 and a test target system 20. In the information processing system 1, the test deficiency identification device 10 (detection device) This system detects insufficient execution of two types of tests (described later) (for example, coverage (coverage rate, or whether or not coverage is achieved) or whether or not there are any missing tests) performed on the system under test 20. The system under test 20 is an MSA-configured system in which multiple microservices communicate with each other to achieve various functions.

[0015] In this embodiment, of the two types of tests performed on the test target system 20, a test that checks the behavior of individual microservices (hereinafter simply referred to as "services") (messages sent or received by each microservice) is referred to as a "unit test (first test)," and a test that is performed by communicating messages between multiple microservices is referred to as a "comprehensive test (second test)." The comprehensive test is a test that checks the behavior when multiple microservices are linked together.

[0016] Each microservice communicates (sends and receives) messages of various types and content patterns (for example, variable types and values). The first and second tests must test (cover) all of these message patterns, and avoid redundant testing (avoiding any omissions).

[0017] The test shortage identifying device 10 and the test target system 20 are connected via a wired connection such as the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or a dedicated line. Alternatively, they are communicably connected via a wireless communication network 5.

[0018] The test deficiency identification device 10 is an information processing device that detects a lack of tests performed on a test target system 20 .

[0019] The test shortage identifying device 10 is, for example, a CPU (Central Processing Unit) or the like. The system includes a computing device 11, a memory 12 such as a RAM (Random Access Memory) or a ROM (Read Only Memory), an external storage device 13 such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive), an input device 14 such as a keyboard, a mouse, or a touch panel, an output device 15 such as a display or a printer, and a network interface card (NIC), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, or the like. The communication device 16 is configured as follows.

[0020] In the test deficiency identifying device 10, data such as programs used to execute predetermined processes is stored in the external storage device 13, for example, and the arithmetic device 11 reads the data into the RAM of the memory 12 and executes it.

[0021] The test target system 20 is realized by one or more information processing devices. The information processing device that realizes the test target system 20 includes, for example, an arithmetic unit 21 such as a CPU, a memory 22 such as a RAM or a ROM, an external storage device 23 such as an HDD or an SSD, and a communication device 24 configured by a NIC, a wireless communication module, a USB module, a serial communication module, or the like.

[0022] In the information processing device that realizes the system under test 20, data such as programs used to execute predetermined processes is stored in an external storage device 23, for example, and the arithmetic device 21 reads it into the RAM of the memory 22 and executes it.

[0023] <Test target system> FIG. 2 is a diagram illustrating an example of a system under test 20. As shown in FIG. In the illustrated example, the system under test 20 includes an account transfer service 201, a customer information acquisition service 202, an automatic withdrawal setting service 203, and a financial institution information acquisition service 204. Various functions are realized by multiple microservices communicating with each other. Arrows A1 to A4 in the figure represent communication dependencies between microservices. For example, arrow A2 between account transfer service 201 and customer information acquisition service 202 represents a dependency relationship in which account transfer service 201 sends a request to customer information acquisition service 202, and customer information acquisition service 202 returns a response to account transfer service 201. Hereinafter, the service that sends the request will be referred to as the "consumer," and the service that sends the response to the request to the consumer will be referred to as the "provider." In the dependency relationship indicated by arrow A2, account transfer service 201 is the "consumer," and customer information acquisition service 202 is the "provider." Note that the system under test 20 does not necessarily have to have an MSA configuration as long as the functions that make up the system communicate with each other via synchronous communication such as REST API.

[0024] Next, the functional configuration of the insufficient test identifying device 10 will be described.

[0025] <Test shortage identification device> FIG. 3 is a block diagram showing an example of the functions of the insufficient test identifying device 10. As shown in FIG. The insufficient test identification device 10 automatically identifies the test coverage and test omissions of the comprehensive test, the test omissions of the unit test, and the characteristics of the requests and their responses that caused the test omissions of the unit tests from all messages (requests and responses) sent and received in the unit tests and the comprehensive test.

[0026] The insufficient test identification device 10 stores, as programs executed by the calculation device 11, a unit test case abstraction unit 601 (abstraction unit), a request and response pattern linking unit 1201, a unit test pattern linking unit 1301 (test pattern linking unit), a total test coverage derivation unit 1501 (detection unit), a total test coverage visualization unit 1701 (display unit, output unit), and a unit test non-execution factor estimation unit 1901 (identification unit).

[0027] In addition, the test deficiency identification device 10 stores the total request and response accumulated information 401 for the overall test, the total request and response accumulated information 501 for the unit test, the total request pattern accumulated information 901, the total response pattern accumulated information 1001, the unit test pattern information 1101, the pattern linking result information 1401, and the total test coverage information 1601 in the memory 12 or the external storage device 13, respectively.

[0028] The integrated test all request and response accumulated information 401 is information that accumulates all the integrated test request and response information in the integrated test. The integrated test request and response information is request and response information that represents a set of requests and their responses that were actually transmitted and received between each service of the test target system 20 in the integrated test.

[0029] For example, when a comprehensive test using all services is performed on the test target system 20 in Fig. 2, the following information is obtained: a request from the front to the account transfer service 201 and request and response information showing the content of the response; a request from the account transfer service 201 to the customer information acquisition service 202 and request and response information showing the content of the response; a request from the account transfer service 201 to the automatic debit setting service 203 and request and response information showing the content of the response; and a request from the automatic debit setting service 203 to the financial institution information acquisition service 204 and request and response information showing the content of the response. The front is a module, such as a screen program, that cannot itself be the subject of a unit test as a provider.

[0030] (Comprehensive test request and response information) FIG. 4 is a diagram showing an example of the comprehensive test request and response information 411. As shown in FIG. The integrated test request and response information 411 is one of the pieces of information that make up the integrated test all request and response accumulated information 401. The integrated test request and response information 411 includes an integrated test ID 412 that identifies each integrated test request and response information 411, a request 413, and a response 414.

[0031] The first element of the comprehensive test ID 412 indicates the consumer, the second element indicates the provider, and the third element indicates the order of the request and response from the consumer to the provider. In the example shown, the comprehensive test ID 412 has a first element that is "account transfer service," a second element that is "customer information acquisition service," and a third element that is "1," indicating that this is the first request and response information with the account transfer service 201 as the consumer and the customer information acquisition service as the provider. In other words, the comprehensive test request and response information 411 shown in this figure indicates the first request from the account transfer service 201 to the customer information acquisition service 202 in the comprehensive test and the contents of that response.

[0032] A request 413 indicates the contents of the request information sent by the consumer to the provider in the comprehensive test request and response information 411. The request 413 includes data such as a method 415, a resource path 416, and a query body 417. In the illustrated example, the request 413 is a request sent using the GET HTTP (Hyper Text Transfer Protocol) method with the query parameter "client_num=87654321" for the resource path " / client_info." Note that when the POST method is used, the query body 417 stores body information.

[0033] Hereinafter, the names of the variables stored in the request 413 and the response 414 will be referred to as “variable names,” and the values ​​of the variables will be referred to as “variable values.” For example, in the illustrated example, “client_num” 417a stored in the query body 417 is the variable name, and “87654321” 417b is its variable value.

[0034] Response 414 indicates the content of the response information sent from the provider to the consumer in comprehensive test request and response information 411. Response 414 includes data in status 418 and body 419. In the illustrated example, response 414 is a response that returns a status code of 200, a variable value "87654321" for the variable name "client_num," a variable value "Sato Hanako" for the variable name "client_name," and a variable value "00400510123456" for the variable name "account_num."

[0035] In this embodiment, a case will be described in which comprehensive test request and response information 411 exists independently for each piece of inter-service communication information, but the comprehensive test all request and response accumulated information 401 is not limited to this form as long as it includes the contents of all comprehensive test request and response information 411 in the comprehensive test.

[0036] The unit test all request and response accumulated information 501 is information that accumulates all unit test request and response information in the unit test of each service. The unit test request and response information is request and response information that represents a set of requests and responses that each service in the test target system 20 actually sent and received in the unit test of each service.

[0037] (Unit test request and response information) FIG. 5 is a diagram showing an example of unit test request and response information 511. Unit test request and response information 511 is one piece of information that constitutes unit test all request and response accumulated information 501. The unit test request and response information 511 has data such as a comprehensive test ID 512 that identifies each unit test request and response information 511, a request 513, and a response 514.

[0038] The first element of unit test ID 512 indicates the provider, and the second element indicates the number of the request and response to the provider. In other words, the first element of unit test ID 512 is the service being tested by the unit test. Due to the nature of unit tests, which are tests on individual services, the consumer service is not set in unit test ID 512. In the example shown, the first element of unit test ID 512 is "customer information acquisition service," and the second element is "1," indicating that this is the first request and response information with the customer information acquisition service as the provider. In other words, unit test request and response information 511 shown in this figure indicates the contents of the first request and its response in the unit test of the customer information acquisition service.

[0039] The data stored in the request 513 and the response 514 is the same as the data stored in the request 413 and the response 414 of the comprehensive test request and response information 411 described above, and therefore a description thereof will be omitted.

[0040] In this embodiment, we will explain the case where unit test request & response information 511 exists independently for each request to each service and its response, but the unit test all request & response accumulated information 501 is not limited to this form as long as it includes the contents of all unit test request & response information 511 for all unit tests.

[0041] The total request and response accumulated information 401 for comprehensive testing and the total request and response accumulated information 501 for unit testing are information that are input to the insufficient test identification device 10. The insufficient test identification device 10 may acquire the total request and response accumulated information 401 for comprehensive testing and the total request and response accumulated information 501 for unit testing from the inter-service communication logs for unit testing or comprehensive testing stored in the external storage device 23 of the system under test 20 via the communication network 5, or may acquire the inter-service communication logs that have been processed in another system. Furthermore, the information stored in the total request and response accumulated information 501 for unit testing may be a manually created test case as is.

[0042] The unit test case abstraction unit 601 executes a process of identifying a unit test pattern, which is a pattern of messages in a unit test, by abstracting all unit test messages (requests and responses). Specifically, the unit test case abstraction unit 601 first receives the unit test all request and response accumulated information 501 and abstracts the requests 513 and responses 514 of all unit test request and response information 511, respectively, to generate a request pattern and a response pattern. In this embodiment, abstraction refers to converting a message into a more comprehensive expression with a higher level of abstraction. Specifically, the unit test case abstraction unit 601 abstracts the message by performing predetermined processing on the variable values ​​included in the message. Examples of predetermined processing include deleting the variable values ​​or converting the variable values ​​to regular expressions. The unit test case abstraction unit 601 then adds the abstracted request to the all request pattern accumulated information 901. The all request pattern accumulated information 901 is information that accumulates abstracted requests (hereinafter referred to as "request patterns"). Furthermore, the unit test case abstraction unit 601 adds the abstracted response to the all response pattern accumulation information 1001. The turn accumulation information 1001 is information that accumulates abstracted responses (hereinafter referred to as "response patterns").

[0043] Then, the unit test case abstraction unit 601 stores the set of the request pattern and the response pattern as a unit test pattern representing the test pattern of the unit test in unit test pattern information 1101, which will be described later.

[0044] The request & response pattern associating unit 1201 executes processing for associating request patterns and response patterns of unit tests with requests and responses of integrated tests. The request & response pattern associating unit 1201 receives all request pattern accumulated information 901, all response pattern accumulated information 1001, and all integrated test request & response accumulated information 401. The request & response pattern associating unit 1201 then associates requests 413 of integrated test request & response information 411 with request patterns, and associates responses 414 with response patterns, and outputs pattern associating result information 1401, which will be described later.

[0045] The unit test pattern associating unit 1301 executes processing to associate a unit test pattern with the integrated test request and response information 411. The unit test pattern associating unit 1301 receives the pattern associating result information 1401 and the unit test pattern information 1101, updates the pattern associating result information 1401, and outputs the pattern associating result information 1401. Specifically, the unit test pattern associating unit 1301 searches the unit test pattern information 1101 for a unit test pattern that is a combination of a request pattern and a response pattern that corresponds to the integrated test request and response information 411, and associates the integrated test request and response information 411 with the searched unit test pattern.

[0046] As a result, the integrated test request and response information 411 is associated with a certain unit test pattern. If there is no association, it is understood that a unit test has not been performed on the request and response pair of the integrated test request and response information 411.

[0047] The overall test coverage derivation unit 1501 detects a lack of message patterns (test patterns) in the overall test or unit test. For example, the overall test coverage derivation unit 1501 derives the extent to which the sets of requests and responses between services in the overall test cover the unit test patterns (test coverage rate and test omissions). The overall test coverage derivation unit 1501 receives the unit test pattern information 1101 and the pattern linking result information 1401 as input, and outputs overall test coverage information 1601, which will be described later.

[0048] The comprehensive test coverage visualization unit 1701 receives the all request pattern accumulated information 901, the all response pattern accumulated information 1001, the pattern linking result information 1401, and the comprehensive test coverage information 1601 as input, and visualizes the coverage rate and test omissions of the comprehensive test, and the test omissions of the unit tests, using the output device 15. The comprehensive test coverage visualization unit 1701 changes the display content of the output device 15 according to the input content via the input device 14.

[0049] When the unit test unexecuted cause estimation unit 1901 is unable to find a unit test pattern that can be associated with the integrated test request and response information 411, it identifies the characteristics of the request and response information of the test pattern that has not been executed in the unit test based on the integrated test request and response information 411. For example, the unit test unexecuted cause estimation unit 1901 estimates the characteristics of the request and response information of the integrated test request and response information 411 that is not included in the unit test pattern information 1101 but is communicated between services in the integrated test (hereinafter referred to as "integral test request and response information to be estimated"). It is inferred which information is unique in generating the target comprehensive test request and response information.

[0050] Specifically, the unit test non-execution factor estimation unit 1901 identifies the characteristics of the request and response information of a test pattern not executed in a unit test based on the variable combination, variable value, or request method of the integrated test request and response information to be estimated. For example, the unit test non-execution factor estimation unit 1901 identifies a method or variable combination that is not used in all request and response information communicated in unit tests and integrated tests by the service that sent the response for the integrated test request and response information to be estimated. Furthermore, the unit test non-execution factor estimation unit 1901 identifies specifications (e.g., the number of digits, type, or size of variable values) that differ from all normal request and response information communicated in unit tests and integrated tests by the service that sent the response for the integrated test request and response information to be estimated. Normal request and response information is request and response information that indicates that the provider's service has successfully processed the request and is responding normally. For example, in the case of HTTP, request and response information with a response status of 200 corresponds to normal request and response information.

[0051] The unit test non-execution cause estimation unit 1901 receives the unit test all request and response accumulated information 501, the overall test all request and response accumulated information 401, and the pattern linking result information 1401, and outputs information indicating the characteristics of the request and response information not communicated in the unit test identified based on the overall test request and response information to be estimated. This output may be sent to the overall test coverage visualization unit 1701.

[0052] The above-described functions of the test deficiency identification device 10 and the test target system 20 are realized, for example, by their hardware or by their arithmetic units (11, 21) reading and executing programs stored in memories (12, 22) or external storage devices (13, 23). These programs are stored in non-transitory data storage media readable by each information processing device, such as a secondary storage device, non-volatile semiconductor memory, hard disk drive, or SSD, or an IC (Integrated Circuit) card, flash memory card, or optical disk.

[0053] Next, the processing performed in the information processing system 1 will be specifically described with reference to the drawings.

[0054] <Processing Overview> FIG. 6 is a diagram for explaining an outline of the process performed by the test deficiency identifying device 10. As shown in FIG. The insufficient test identifying device 10 executes a unit test abstraction process S1 that abstracts unit test requests and responses as patterns. The unit test abstraction process S1 will be described in detail later.

[0055] Next, the insufficient test identifying device 10 executes a request and response pattern linking process S4 that links the abstracted unit test requests and responses with the comprehensive test requests and responses. The request and response pattern linking process S4 will be described in detail later.

[0056] Next, the insufficient test identifying device 10 executes a unit test pattern linking process S5 for linking the test patterns of the unit tests with the requests and responses of the integrated tests. The unit test pattern linking process S5 will be described in detail later.

[0057] Next, the insufficient test identifying device 10 executes an overall test coverage derivation process S6 to derive the extent to which the sets of requests and responses between services in the overall test cover the test patterns of the unit tests. The details of the overall test coverage derivation process S6 will be described later.

[0058] Furthermore, the insufficient test identifying device 10 executes a process S7 for inferring the cause of non-execution of unit tests, which identifies the characteristics of request and response information that has not been communicated in the unit tests, based on the comprehensive test request and response information to be inferred. The details of the process S7 for inferring the cause of non-execution of unit tests will be described later. Next, each of the processes described above will be described in detail.

[0059] <Unit test abstraction processing> 7 is a process flow diagram illustrating the details of the unit test abstraction process S1. The unit test abstraction process S1 is executed, for example, when the total integrated test request and response accumulated information 401 and the total unit test request and response accumulated information 501 are input to the insufficient test identification device 10, when a predetermined input is made to the insufficient test identification device 10 by the user, or at a predetermined timing (for example, a predetermined time or a predetermined time interval).

[0060] First, the unit test case abstraction unit 601 acquires one unit test request and response information 511 from the unit test all request and response accumulated information 501 (S602). Next, the unit test case abstraction unit 601 executes request abstraction processing S2 on the unit test request and response information 511 acquired in S602.

[0061] <Request Abstraction Processing> FIG. 8 is a process flow diagram illustrating the request abstraction process S2 in detail. In the request abstraction process S2, first, the unit test case abstraction unit 601 deletes the variable values ​​of each variable in the query body of the request for the request and response information to be processed (unit test request and response information 511 or integrated test request and response information 411) and replaces them with a description of only the name of each variable (S702). At this time, the unit test case abstraction unit 601 performs the rewriting process not on the data of the request and response information to be processed (hereinafter referred to as the "request and response information to be processed") itself, but on the data that is a copy of the request and response information to be processed.

[0062] 5, the only information held by the query body 517 is "client_num." The request in the request and response information to be processed that has been rewritten in S702 becomes one of the request patterns.

[0063] Next, the unit test case abstraction unit 601 determines whether or not request pattern information 911 having the same content as the request of the request and response information to be processed that was rewritten in S702 exists in the total request pattern storage information 901 (S703).

[0064] (Request pattern information) 9 is a diagram showing an example of request pattern information 911. The request pattern information 911 is one of a plurality of tables constituting the total request pattern accumulation information 901, and represents a request pattern. The request pattern information 911 has a request pattern ID 912 that identifies each piece of request pattern information 911, and each piece of data for a request 913. The request 913 has a method, a resource path, and a query body 914, just like the request 513 in the unit test request and response information 511, but The query body 914 differs from the request 513 of the unit test request and response information 511 in that it has only variable names and no variable values.

[0065] For example, when the request 513 obtained by rewriting the unit test request and response information 511 in Fig. 5 in S702 is compared with the request 913 in the request pattern information 911 in Fig. 9, the determination in S703 is YES because the contents are identical. On the other hand, if the request pattern information 911 in Fig. 9 does not exist in the all request pattern accumulation information 901, the determination in S703 is NO.

[0066] If the determination in S703 is YES, the unit test case abstraction unit 601 holds the request pattern ID 912 of the request pattern information 911 having the same content identified in S703 (S704). After that, the unit test case abstraction unit 601 ends this request abstraction process S2.

[0067] For example, in the case of the request pattern information 911 in FIG. 9, the unit test case abstraction unit 601 holds information that the request pattern ID 912 is "1".

[0068] On the other hand, if the determination in S703 is NO, the unit test case abstraction unit 601 assigns a request pattern ID equal to the number of request pattern information 911 existing in the total request pattern accumulation information 901 at that time plus one to the request of the request and response information to be processed that was rewritten in S702. Then, the unit test case abstraction unit 601 adds the request to which the request pattern ID has been assigned as new request pattern information 911 to the total request pattern accumulation information 901. Then, the unit test case abstraction unit 601 holds the request pattern ID 912 of the newly generated request pattern information 911 (S705). Thereafter, the unit test case abstraction unit 601 ends this request abstraction process S2.

[0069] For example, if the request pattern information 911 does not exist in the total request pattern storage information 901, the request pattern ID 912 of the request pattern information 911 generated in S705 is "1".

[0070] Following the request abstraction process S2, the unit test case abstraction section 601 executes a response abstraction process S3 on the unit test request and response information 511 acquired in S602.

[0071] <Response abstraction processing> FIG. 10 is a process flow diagram illustrating the response abstraction process S3 in detail. The response abstraction process S3 has many similar processes to the request abstraction process S2, so only a brief description will be given of the processes similar to the request abstraction process S2.

[0072] In the response abstraction process S3, first, the unit test case abstraction unit 601 determines whether an error code is described in the response body of the request and response information to be processed (unit test request and response information 511 or integrated test request and response information 411) (S802). The unit test case abstraction unit 601 may determine whether an error code is present based on the variable name of the body or the variable value. For example, assume that the variable name in which the error code is stored is "error_code." In this case, if the response body has the variable name "error_code," the determination in S802 is YES; otherwise, the determination in S802 is NO. If the determination in S802 is NO, the unit test case abstraction unit 601 proceeds to S804.

[0073] For example, since the response 514 of the unit test request and response information 511 in FIG. 5 does not have "error_code", the determination in S802 becomes NO and the process proceeds to S804.

[0074] If the determination in S802 is YES, the unit test case abstraction unit 601 deletes the variable name and variable value of the error code from the body, adds a new line "Error Code" to the response, and sets the variable value of the error code to the added line "Error Code" (S803). For example, if the body 519 of the unit test request and response information 511 has "error_code:3220", the unit test case abstraction unit 601 adds a new line "Error Code" to the response 514 and sets its value to "3220". At this time, the unit test case abstraction unit 601 deletes the description of "error_code:3220" from the body 519. Note that the unit test case abstraction unit 601 performs the rewriting process not on the data of the request and response information to be processed itself, but on the data that is a copy of the request and response information to be processed.

[0075] Next, the unit test case abstraction unit 601, similar to S702, deletes each variable value from the body of the response of the request and response information to be processed, and replaces it with a description of only each variable name (S804).

[0076] Next, the unit test case abstraction unit 601 determines whether or not the response pattern information 1011 having the same content as the response of the request and response information to be processed that was rewritten in S803 or S804 exists in the total response pattern storage information 1001 (S805).

[0077] (Response pattern information) 11 is a diagram showing an example of response pattern information 1011. The response pattern information 1011 is one of multiple tables constituting the total response pattern accumulation information 1001, and represents a response pattern. The response pattern information 1011 includes a response pattern ID 1012 that identifies each piece of response pattern information 1011, and data for a response 1013. The response 1013 includes a status and a body 1014, similar to the response 514 of the unit test request and response information 511. However, the response 1013 differs from the response 514 of the unit test request and response information 511 in that the body 1014 includes only a variable name, the body 1014 does not include a variable name for an error code, and the response 1013 may include a line for an error code.

[0078] In S805, for example, when response 514 obtained by rewriting unit test request and response information 511 in Fig. 5 in S803 and S804 is compared with response 1013 of response pattern information 1011 in Fig. 11, the contents are identical, so the determination in S805 is YES. On the other hand, if response pattern information 1011 in Fig. 11 does not exist in all response pattern storage information 1001, the determination in S805 is NO.

[0079] If the determination in S805 is YES, the unit test case abstraction unit 601 holds the response pattern ID 1012 of the response pattern information 1011 having the same content identified in S805 (S806), as in S704. Thereafter, the unit test case abstraction unit 601 ends this response abstraction process S3.

[0080] On the other hand, if the determination in S805 is NO, the unit test case abstraction unit 601 assigns a response pattern ID equal to the number of response pattern information 1011 existing in the total response pattern storage information 1001 at that time plus one to the response of the request and response information to be processed that has been rewritten in S803 and S804, as in S704. Then, the unit test case abstraction unit 601 adds the response with the assigned response pattern ID to the total response pattern information 1001 as new response pattern information 1011. Then, the unit test case abstraction unit 601 holds the response pattern ID 1012 of the newly generated response pattern information 1011 (S807). Thereafter, the unit test case abstraction unit 601 ends this response abstraction process S3.

[0081] After the request abstraction process S2 and the response abstraction process S3 are completed, the unit test case abstraction unit 601 determines whether the generated set of request pattern and response pattern exists in the unit test pattern information 1101 (S603).

[0082] (Unit test pattern information) 12 is a diagram showing an example of unit test pattern information 1101. The unit test pattern information is a table having the following data items: unit test pattern ID 1102, in which a unit test pattern ID identifying each unit test pattern is set; request pattern ID 1103, in which a request pattern ID is set; response pattern ID 1104, in which a response pattern ID is set; and unit test ID 1105, in which a unit test ID of unit test request and response information 511 corresponding to the unit test pattern is set.

[0083] In the illustrated example, the unit test pattern ID 1102 "[Customer information acquisition, 3]" indicates that this is the third unit test pattern of the unit test whose provider is the customer information acquisition service 202. Furthermore, the row of unit test pattern ID 1102 "[Customer information acquisition, 3]" indicates that the third unit test pattern of the unit test whose provider is the customer information acquisition service 202 is a combination of request pattern information 911 whose request pattern ID 912 is "3" and response pattern information 1011 whose response pattern ID 1012 is "3". Furthermore, the row with unit test pattern ID 1102 "[Customer information acquisition, 3]" indicates that there are two unit test request and response information 511 corresponding to the third unit test pattern of the unit test with the customer information acquisition service 202 as the provider: unit test request and response information 511 with unit test ID 512 "[Customer information acquisition, 3]" and unit test request and response information 511 with unit test ID 512 "[Customer information acquisition, 4]". In this way, there may be multiple unit test request and response information 511 belonging to the same unit test pattern ID 1102.

[0084] In S603, specifically, the unit test case abstraction unit 601 determines whether a row having the pair of request pattern ID 912 and response pattern ID 1012 that it holds exists in the unit test pattern information 1101, and whether the first element of the unit test pattern ID 1102 of that row is identical to the first element of the unit test ID 512 of the unit test request & response information 511 acquired in S602.

[0085] For example, if unit test request and response information 511 in Figure 5 is obtained in S602 and "1" is stored as request pattern ID 912 and "2" is stored as response pattern ID 1012, then the request pattern ID 1103 of the row in unit test pattern information 1101 in Figure 12 where unit test pattern ID 1102 is "[customer information acquisition, 1]" is "1" and the response pattern ID 1104 is "2", so the determination in S603 will be YES.

[0086] If the determination in S603 is YES, the set of request pattern information 911 and response pattern information 1011 obtained by patterning the request 513 and response 514 of the unit test request and response information 511 acquired in S602 has already been used in the unit test pattern. It is stored as one of the turn information 1101. Therefore, if the determination in S603 is YES, the unit test case abstraction unit 601 adds the unit test ID 512 of the unit test request and response information 511 acquired in S602 to the unit test ID 1105 of the row identified in S603 (S604).

[0087] For example, when the unit test request and response information 511 whose unit test ID 512 is "[Customer information acquisition, 4]" is acquired in S602, and the request pattern ID 1103 and response pattern ID 1104 stored in the previous processing are "3" and "3", the unit test case abstraction unit 601 adds "[Customer information acquisition, 4]" to the unit test ID 1105 of the row in the unit test pattern information 1101 that has the same pair of request pattern ID 1103 and response pattern ID 1104 and whose unit test pattern ID 1102 is "[Customer information acquisition, 3]".

[0088] On the other hand, if the determination in S603 is NO, the unit test case abstraction unit 601 adds the set of request pattern information 911 and response pattern information 1011, which are obtained by abstracting the unit test request and response information 511 acquired in S602, to the unit test pattern information 1101 as a new unit test pattern (S605). For example, if the unit test request and response information 511 of FIG. 5, in which the unit test ID 512 is "[Acquire Customer Information, 1]", is acquired in S602, the request pattern ID 912 will be "1" and the response pattern ID 1012 will be "2" immediately before S605. At this time, it is assumed that the unit test pattern information 1101 does not contain a row in which the first element of the unit test pattern ID 1102 is "Acquire Customer Information". In this case, in S605, the unit test case abstraction unit 601 adds a row to the unit test pattern information 1101 where the unit test pattern ID 1102 is "[Acquire Customer Information, 1]", the request pattern ID 1103 is the same value as request pattern ID 912, "1", the response pattern ID 1104 is the same value as response pattern ID 1012, "2", and the unit test ID 1105 is the same value as unit test ID 512, "[Acquire Customer Information, 1]". This row is the first unit test pattern whose provider is the "Acquire Customer Information" service, the same as the first element of unit test ID 512.

[0089] Thereafter, the unit test case abstraction unit 601 determines whether or not the above-described processes of S602 to S605 have been performed for all unit test request and response information 511 included in the unit test all request and response accumulated information 501 (S606). If the determination in S606 is NO, the unit test case abstraction unit 601 returns to S602 and performs the processes of S602 to S605 for all unit test request and response information 511 included in the unit test all request and response accumulated information 501. On the other hand, if the determination in S606 is YES, the unit test case abstraction unit 601 ends this unit test abstraction process S1.

[0090] The unit test abstraction process S1 abstracts all unit test request and response information 511 contained in the unit test all request and response accumulated information 501 according to the rules of the request abstraction process S2 and the response abstraction process S3. By patterning the test cases of the unit tests in this way, it becomes possible to link the test patterns of the unit tests with the test patterns of the integrated tests in the subsequent process. Note that the contents of the request abstraction process S2 and the response abstraction process S3 are not limited to the above-described flow, as long as multiple request pattern information 911 or multiple response pattern information 1011 are not linked to one integrated test request and response information 411 in the request and response pattern linking process S4 executed by the request and response pattern linking unit 1201 in the subsequent stage. For example, the unit test case abstraction unit 601 may perform the request abstraction process S2 so that the variable values ​​of the query body of the request pattern information 911 conform to regular expressions. may be performed.

[0091] <Request & response pattern linking process> FIG. 13 is a process flow diagram illustrating the request and response pattern linking process S4 in detail. In the request and response pattern linking process S4, first, the request and response pattern linking unit 1201 acquires one piece of integrated test request and response information 411 from the integrated test all request and response accumulated information 401 (S1202).

[0092] First, the request and response pattern associating unit 1201 associates the request communicated in the comprehensive test with the request pattern. The request and response pattern associating unit 1201 acquires one piece of request pattern information 911 from the all request pattern storage information 901 (S1203).

[0093] Next, the request & response pattern associating unit 1201 determines whether the request 413 of the integrated test request & response information 411 acquired in S1202 corresponds to the request pattern represented by the request pattern information 911 acquired in S1203 (S1204). Specifically, the request & response pattern associating unit 1201 determines whether the character string of the method 415 of the request 413 of the integrated test request & response information 411 acquired in S1202 is the same as the character string of the method of the request 913 of the request pattern information 911, the character string of the resource path 416 of the request 413 is the same as the character string of the resource path of the request 913, and the pair of variable names of the query body 417 of the request 413 is the same as the pair of variable names of the query body 914 of the request 913. If the determination in S1204 is NO, the request & response pattern associating unit 1201 proceeds to S1206.

[0094] For example, in the case of request 413 of comprehensive test request and response information 411 in FIG. 4 and request 913 of request pattern information 911 in FIG. 9, the method is the same ("GET"), the resource path is the same (" / client_info"), and the variable name of the query body is the same ("client_num"), so the determination in S1204 is YES.

[0095] If the determination in S1204 is YES, the request & response pattern associating unit 1201 holds the request pattern ID 912 of the request pattern information 911 acquired in S1203 (S1205). For example, in the case of the request pattern information 911 in FIG. 9, the request & response pattern associating unit 1201 holds "1".

[0096] Next, the request and response pattern linking unit 1201 determines whether or not the above-described processes from S1203 to S1205 have been performed for all the request pattern information 911 included in the all request pattern storage information 901 (S1206).

[0097] If the determination in S1206 is NO, the request and response pattern associating unit 1201 returns to S1203. That is, the request and response pattern associating unit 1201 performs the processes from S1203 to S1205 on all the request pattern information 911 included in the all request pattern storage information 901.

[0098] On the other hand, if the determination in S1206 is YES, the request and response pattern associating unit 1201 determines whether or not there is a request pattern ID 912 currently being held (S1207).

[0099] If there is a retained request pattern ID 912, there is request pattern information 911 associated with the request 413 of the comprehensive test request and response information 411 acquired in S1202. Therefore, if the determination in S1207 is YES, the request and response pattern associating unit 1201 proceeds to the processing of S1208.

[0100] On the other hand, if there is no retained request pattern ID 912, there is no request pattern information 911 that can be linked to the request 413 of the comprehensive test request and response information 411 acquired in S1202. Therefore, if the determination in S1207 is NO, the request and response pattern linking unit 1201 performs request abstraction processing S2 on the comprehensive test request and response information 411 acquired in S1202 in order to register the request 413 as a new request pattern, and retains the request pattern ID 912 of the newly created request pattern information 911 (S2).

[0101] Next, the request & response pattern linking unit 1201 sets the currently held request pattern ID 912 to the request pattern ID 1403, and adds a new row in which the overall test ID 412 of the overall test request & response information 411 acquired in S1202 is set to the overall test ID 1402 to the pattern linking result information 1401 described below (S1208).

[0102] (Pattern linking result information) 14 is a diagram showing an example of pattern linking result information 1401. The pattern linking result information 1401 is a table having data items of a comprehensive test ID 1402, a request pattern ID 1403, a response pattern ID 1404, and a unit test pattern ID 1405. For example, in the illustrated data example, the row in which the comprehensive test ID 1402 is "[Account Transfer, Automatic Debit Settings, 1]" indicates that the comprehensive test request and response information 411 in which the comprehensive test ID 412 is "[Account Transfer, Automatic Debit Settings, 1]" is linked to request pattern information 911 in which the request pattern ID 912 is "

[45] ", response pattern information 1011 in which the response pattern ID 1012 is "

[43] ", and a unit test pattern in which the unit test pattern ID 1102 is "[Automatic Debit Settings, 4]".

[0103] For example, if the overall test ID 412 of the overall test request and response information 411 acquired in S1202 is "[Account transfer, Customer information acquisition, 1]", the request and response pattern linking unit 1201 sets "[Account transfer, Customer information acquisition, 1]" as the value of the overall test ID 1402, and adds a new row to the pattern linking result information 1401 in which the request pattern ID 912 of the request pattern information 911 in Figure 9 saved in S1205, "1", is set as the value of the request pattern ID 1403.

[0104] Next, the request and response pattern linking unit 1201 links the response of the comprehensive test with the response pattern. First, the request and response pattern linking unit 1201 acquires one piece of response pattern information 1011 from the total response pattern information storage 1001 (S1209).

[0105] Next, the request and response pattern linking unit 1201 determines whether the response 414 of the integrated test request and response information 411 acquired in S1202 corresponds to the response pattern represented by the response pattern information 1011 acquired in S1209 (S1210). Specifically, if the character string of the status 418 of the response 414 of the integrated test request and response information 411 and the character string of the status of the response 1013 of the response pattern information 1011 are the same, and if the set of variable names other than the variable having the error code in the body 419 of the response 414 and the set of variable names in the body of the response 1013 are the same, If the response 1013 has an error code, the request and response pattern linking unit 1201 determines whether the value of the error code in the body 419 of the response 414 is the same as the value of the error code in the response 1013. If the determination in S1210 is NO, the request and response pattern linking unit 1201 proceeds to S1212.

[0106] For example, in response 414 of comprehensive test request and response information 411 in FIG. 4 and response 1013 of response pattern information 1011 in FIG. 11, the status is the same, "200", the set of variable names in the body is all the same, "client_num", "client_name", and "account_num", and response 1013 does not have an error code, so the determination in S1210 is YES.

[0107] If the determination in S1210 is YES, the request and response pattern associating unit 1201 holds the response pattern ID 1012 of the response pattern information 1011 acquired in S1209 (S1211). For example, in the case of the response pattern information 1011 in FIG. 11, the request and response pattern associating unit 1201 holds "2".

[0108] Next, the request and response pattern linking unit 1201 determines whether or not the above-described processes from S1209 to S1211 have been performed for all the response pattern information 1011 included in the total response pattern storage information 1001 (S1212).

[0109] If the determination in S1212 is NO, the request and response pattern associating unit 1201 returns to S1209. That is, the request and response pattern associating unit 1201 performs the processes from S1209 to S1211 on all the response pattern information 1011 included in the all response pattern storage information 1001.

[0110] On the other hand, if the determination in S1212 is YES, the request and response pattern linking unit 1201 determines whether or not there is a response pattern ID 1012 currently being held (S1213).

[0111] If there is a stored response pattern ID 1012, there is response pattern information 1011 associated with the response 414 of the comprehensive test request and response information 411 acquired in S1202. Therefore, if the determination in S1213 is YES, the request and response pattern associating unit 1201 proceeds to S1214.

[0112] On the other hand, if there is no request pattern ID 1012 held, there is no response pattern information 1011 that can be linked to the response 414 of the comprehensive test request and response information 411 acquired in S1202. Therefore, if the determination in S1213 is NO, the request and response pattern linking unit 1201 performs response abstraction processing S3 on the comprehensive test request and response information 411 acquired in S1202 in order to register the response 414 as a new response pattern, and holds the response pattern ID 1012 of the newly created response pattern information 1011 (S3).

[0113] Next, the request & response pattern linking unit 1201 sets the currently held response pattern ID 1012 to the response pattern ID 1404 of the row added in S1208 to the pattern linking result information 1401 (S1214). For example, if the overall test ID 412 of the overall test request & response information 411 acquired in S1202 is "[Account transfer, Get customer information, 1]", the held response pattern ID 1012 is set to the response pattern ID 1404 of the row in the pattern linking result information 1401 where the overall test ID 1402 is "[Account transfer, Get customer information, 1]".

[0114] Next, the request and response pattern linking unit 1201 determines whether or not the above-described processes from S1202 to S1214 have been performed for all of the integrated test request and response information 411 in the integrated test all request and response accumulated information 401 (S1215).

[0115] If the determination in S1215 is NO, the request & response pattern associating unit 1201 returns to S1202. That is, the request & response pattern associating unit 1201 performs the processes from S1202 to S1214 on all of the integrated test request & response information 411 included in the integrated test all request & response accumulated information 401.

[0116] On the other hand, if the determination in S1215 is YES, the request and response pattern associating unit 1201 ends the request and response pattern associating process S4. At this time, the request and response pattern associating unit 1201 initializes the request pattern ID 912 and response pattern ID 1012 that it holds.

[0117] <Unit test pattern linking process> FIG. 15 is a process flow diagram illustrating the unit test pattern linking process S5 in detail. In the unit test pattern linking process S5, first, the unit test pattern linking unit 1301 extracts one line of data from the pattern linking result information 1401 (S1302).

[0118] Next, the unit test pattern linking unit 1301 extracts one line of data from the unit test pattern information 1101 (S1303).

[0119] Next, the unit test pattern linking unit 1301 determines whether the second element of the comprehensive test ID 1402, the request ID 1403, and the response ID 1404 of the row of the pattern linking result information 1401 extracted in S1302 are the same as the first element of the unit test pattern ID 1102, the request ID 1103, and the response ID 1104 of the row of the unit test pattern information 1101 extracted in S1303 (S1304). If the determination in S1304 is NO, the unit test pattern linking unit 1301 proceeds to S1306.

[0120] 14 in S1302, and the first line of the unit test pattern information 1101 in S12 in S1303 will be specifically described as an example. In the data in the first line of the pattern linking result information 1401, the second element of the integrated test ID 1402 is "acquire customer information," the request ID 1403 is "1," and the response ID 1404 is "2." In addition, in the data in the first line of the unit test pattern information 1101, the first element of the unit test pattern ID 1102 is "acquire customer information," the request ID 1103 is "1," and the response ID 1104 is "2." Therefore, the second element of the integrated test ID 1402 and the first element of the unit test pattern ID 1102, the request ID 1403 and the request ID 1103, and the response ID 1404 and the response ID 1104 are the same, and therefore the determination in S1304 is YES.

[0121] If the determination in S1304 is YES, the integrated test request and response information 411 having the integrated test ID 412 that matches the integrated test ID 1402 of the row in the pattern linking result information 1401 extracted in S1302 is linked to the unit test pattern of the row in the unit test pattern information 1101 extracted in S1303. Therefore, if the determination in S1304 is YES, the unit test pattern linking unit 1301 links the unit test pattern ID 1405 of the row in the pattern linking result information 1401 extracted in S1302 to the unit test pattern ID 110 of the row in the unit test pattern information 1101 extracted in S1303. Set 2 (S1305).

[0122] In the above example, the unit test pattern linking unit 1301 sets the unit test pattern ID 1405 in the first line of the pattern linking result information 1401 in Figure 14 to ``[Customer information acquisition, 1]'', which is the unit test pattern ID 1102 in the first line of the unit test pattern information 1101 in Figure 12.

[0123] Next, the unit test pattern linking unit 1301 determines whether or not the above-described processes from S1303 to S1305 have been performed on all rows of the unit test pattern information 1101 (S1306).

[0124] If the determination in S1306 is NO, the unit test pattern associating unit 1301 returns to S1303. That is, for one line of data in the pattern associating result information 1401, the unit test pattern associating unit 1301 performs the processes from S1303 to S1305 on all lines of the unit test pattern information 1101.

[0125] On the other hand, if the determination in S1306 is YES, the unit test pattern linking unit 1301 determines whether or not the above-described processes from S1302 to S1306 have been performed on all rows of the pattern linking result information 1401 (S1307).

[0126] If the determination in S1307 is NO, the unit test pattern associating unit 1301 returns to S1302. That is, the unit test pattern associating unit 1301 performs the processes from S1302 to S1306 for all rows of the pattern associating result information 1401.

[0127] On the other hand, if the determination in S1307 is YES, the unit test pattern linking unit 1301 ends this unit test pattern linking process S5.

[0128] The request and response pattern linking process S4 described above links the integrated test request and response information 411 to the request pattern information 911 and the response pattern information 1011. Then, the unit test pattern linking process S5 links the integrated test request and response information 411 to a unit test pattern in the unit test pattern information 1101. If there is no linking, a unit test has not been performed on the pair of request pattern information 911 and response pattern information 1011 in the integrated test request and response information 411, and a unit test omission is identified. For example, at the end of the unit test pattern linking process S5, for a pair of request pattern ID 1403 and response pattern ID 1404 in a row where no value exists in unit test pattern ID 1405, it is determined that a unit test pattern having that pair does not exist as a unit test that has at least the second element of integrated test ID 1402 in the same row as the provider, i.e., it is a test pattern that has not been unit tested. Furthermore, by linking the comprehensive test request and response information 411 with the unit test patterns, it becomes possible to determine to what extent the inter-service communications in the comprehensive test cover the unit test patterns, and which unit test patterns have not been executed as inter-service communications in the comprehensive test.

[0129] <Comprehensive test coverage derivation process> FIG. 16 is a process flow diagram illustrating the details of the comprehensive test coverage derivation process S6. In the overall test coverage derivation process S6, first, the overall test coverage derivation unit 1501 extracts one line of data from the pattern linking result information 1401 (S1502).

[0130] Next, the comprehensive test coverage derivation unit 1501 extracts the pattern linkages extracted in S1502. It is determined whether data corresponding to the data in the row of the result information 1401 exists in the comprehensive test coverage information 1601 (to be described later) (S1503).

[0131] (Comprehensive test coverage information) 17 is a diagram showing an example of the integrated test coverage information 1601. The integrated test coverage information 1601 is a table having the following data items: consumer 1602, provider 1603, total pattern count 1604, number of coverage patterns 1605, coverage rate 1606, and unit test coverage pattern ID 1607. The total number of unit test patterns of the provider set in provider 1603 is set in the total pattern count 1604. The number of coverage patterns 1605 is set to the number of unit test patterns implemented in the integrated test. The coverage rate 1606 is set to the coverage rate in the integrated test for all unit test patterns (the value set in the number of coverage patterns 1605 divided by the value set in the total pattern count 1604). The unit test coverage pattern ID 1607 is set to the second element of one or more unit test pattern IDs 1102 implemented in the integrated test.

[0132] For example, the first line of data shown in the figure indicates that the comprehensive test request and response information 411, in which the consumer is the account transfer service 201 and the provider is the customer information acquisition service 202, covers 25 of the 50 unit test patterns in which the customer information acquisition service 202 is the provider, with a coverage rate of 0.50, and the unit test pattern IDs 1102 covered by the comprehensive test are [customer information acquisition, 1], [customer information acquisition, 2], [customer information acquisition, 5], [customer information acquisition, 6], etc.

[0133] In S1503, specifically, the comprehensive test coverage derivation unit 1501 determines whether a row that is the first element of the comprehensive test ID 1402 of a row in the pattern linking result information 1401 extracted by the consumer 1602 in S1502 and the second element of the comprehensive test ID 1402 of a row in the pattern linking result information 1401 extracted by the provider 1603 in S1502 exists in the comprehensive test coverage information 1601. If the determination in S1503 is YES, the request & response pattern linking unit 1201 proceeds to S1505.

[0134] On the other hand, if the determination in S1503 is NO, the combination of consumer and provider in the integrated test ID 1402 of the row extracted in S1502 does not exist in the integrated test coverage information 1601. Therefore, if the determination in S1503 is NO, the integrated test coverage derivation unit 1501 adds a row in which the first element of the integrated test ID 1402 of the row extracted in S1502 is consumer 1602 and the second element is provider 1603 to the integrated test coverage information 1601 (S1504).

[0135] Next, the comprehensive test coverage derivation unit 1501 determines whether, for a row of the comprehensive test coverage information 1601 in which the first element of the comprehensive test ID 1402 of the row extracted in S1502 is consumer 1602 and the second element is provider 1603, the unit test coverage pattern ID 1607 has the same number as the second element of the unit test pattern ID 1405 of the row of the pattern linking result information 1401 extracted in S1502 (S1505).

[0136] For example, in S1502, when the data in the first row of the pattern linking result information 1401 in Fig. 14 is extracted, the second element of the unit test pattern ID 1405 is "1", and the unit test coverage pattern ID 1607 in the first row of the overall test coverage information 1601 in Fig. 17 contains "1", so the determination in S1505 is YES. If the determination in S1505 is YES, the overall test coverage derivation unit 1501 proceeds to S1507.

[0137] On the other hand, if the determination in S1505 is NO, the overall test coverage derivation unit 1501 adds the second element of the unit test pattern ID 1405 of the row in the pattern linking result information 1401 extracted in S1502 to the unit test coverage pattern ID 1607 of the row identified in S1503 of the pattern linking result information 1401 (S1506).

[0138] Next, the comprehensive test coverage deriving unit 1501 determines whether or not the processes from S1502 to S1506 have been performed on all rows of the pattern linking result information 1401 (S1507).

[0139] If the determination in S1507 is NO, the overall test coverage deriving unit 1501 returns to S1502. That is, the overall test coverage deriving unit 1501 performs the processes from S1502 to S1506 for all rows of the pattern linking result information 1401.

[0140] On the other hand, if the determination in S1507 is YES, the comprehensive test coverage deriving unit 1501 extracts one line of data from the unit test pattern information 1101 (S1508).

[0141] Next, the total test coverage derivation unit 1501 adds 1 to the number of total patterns 1604 for the row in the total test coverage information 1601 that is equal to the first element of the unit test pattern ID 1102 of the row extracted by the provider 1603 in S1508 (S1509).

[0142] For example, when the first line of the unit test pattern information 1101 in Fig. 12 is extracted in S1508, the first element of the unit test pattern ID 1102 is "acquire customer information," so the overall test coverage derivation unit 1501 adds 1 to the value of the total number of patterns 1604 in the first line of the overall test coverage information 1601 in Fig. 17. Note that the overall test coverage derivation unit 1501 initializes all of the total number of patterns 1604 in the overall test coverage information 1601 to 0 in advance when starting the overall test coverage derivation process S6.

[0143] Next, the comprehensive test coverage deriving unit 1501 determines whether the processes from S1508 to S1509 have been performed for all rows of the unit test pattern information 1101 (S1510).

[0144] If the determination in S1510 is NO, the overall test coverage deriving unit 1501 returns to S1508. That is, the overall test coverage deriving unit 1501 performs the processes from S1508 to S1509 for all rows of the unit test pattern information 1101.

[0145] On the other hand, if the determination in S1510 is YES, the integrated test coverage deriving unit 1501 extracts one line of data from the integrated test coverage information 1601 (S1511).

[0146] Next, the overall test coverage derivation unit 1501 stores the number of elements in the unit test coverage pattern ID 1607 of the same line as the coverage pattern count 1605 of the line extracted in S1511 (S1512). For example, when the number of elements in the unit test coverage pattern ID 1607 of the first line of the overall test coverage information 1601 is "25", the overall test coverage derivation unit 1501 sets "25" as the coverage pattern count of the first line.

[0147] Next, the overall test coverage derivation unit 1501 sets the value obtained by dividing the number of coverage patterns 1605 of the line extracted in S1511 by the total number of patterns 1604 as the coverage rate 1606 of the same line (S1513). For example, if the total number of patterns 1604 of the first line of the overall test coverage information 1601 is "50" and the number of coverage patterns 1605 is "25", the overall test coverage derivation unit 1501 sets the coverage rate 1606 of the first line as "0.50", which is the value obtained by dividing 25 by 50. The calculation of the coverage rate is not limited to the method shown in this example. For example, the overall test coverage derivation unit 1501 may assign a weight to each unit test pattern and calculate the coverage rate.

[0148] Next, the integrated test coverage deriving unit 1501 determines whether or not the processes from S1511 to S1513 have been performed on all rows of the integrated test coverage information 1601 (S1514).

[0149] If the determination in S1514 is NO, the integrated test coverage derivation unit 1501 returns to S1511. That is, the integrated test coverage derivation unit 1501 performs the processes from S1511 to S1513 for all rows of the integrated test coverage information 1601.

[0150] On the other hand, if the determination in S1514 is YES, the integrated test coverage derivation unit 1501 ends the integrated test coverage derivation process S6.

[0151] By the comprehensive test coverage process S6, coverage of the comprehensive test all request and response accumulated information 401 for the unit test pattern information 1101 in a request from a consumer to a provider and its response can be derived.

[0152] (Comprehensive test coverage visualization screen) 18 is a diagram showing an example of an integrated test coverage visualization screen 1700. The integrated test coverage visualization screen 1700 is displayed by an integrated test coverage visualization unit 1701 on, for example, the output device 15. The integrated test coverage visualization screen 1700 includes, for example, an MSA diagram 1602, a possible pattern table 1705, and request and response pattern details 1710.

[0153] The MSA diagram 1602 uses arrows 1704A-D (shapes connecting components) to represent the communication dependencies (communications between components) between services 1703A-D of the system under test 20 and the coverage of the integrated test. For lines in the integrated test coverage information 1601 whose starting points are consumer 1602 and whose ending points are provider 1603, the thickness of the arrows 1704A-D is proportional to the value of the total number of patterns 1604 for the line in question, and the color intensity of the arrows 1704A-D is proportional to the coverage 1606 for the line in question (i.e., the degree of insufficient testing). In the illustrated example, the thickest arrow 1704B indicates the largest number of patterns 1604 in which the account transfer service 1703A is the consumer and the customer information acquisition service 1703B is the provider. The thinnest arrow 1704D indicates the smallest number of patterns 1604 in which the automatic debit setting service 1703C is the consumer and the financial institution information acquisition service 1703D is the provider. The darkest arrow 1704C indicates the highest coverage rate 1606 in which the account transfer service 1703A is the consumer and the automatic debit setting service 1703C is the provider. The lightest arrow 1704A indicates the lowest coverage rate 1606 in which the front is the consumer and the account transfer service 1703A is the provider. In this way, the MSA diagram 1602 visualizes the comprehensive test coverage for unit test patterns in the communication dependencies between services. This allows the user to easily understand the comprehensive test coverage.

[0154] In this embodiment, the integrated test coverage visualization unit 1701 represents the communication between services with arrows in the MSA diagram 1602, but this is not limiting and the communication between services may be represented in other display modes such as other figures, symbols, or characters. Also, in the MSA diagram 1602, the integrated test coverage visualization unit 1701 represents the integrated test coverage rate 1606 with the color density of the arrow, but this is not limiting and the communication between services may be represented in other display modes such as the color of the arrow. Similarly, the total test coverage visualization unit 1701 indicates the total number of unit test patterns 1604 in the MSA diagram 1602 by the thickness of the arrow, but this is not limited to this and may be indicated in other display modes, such as by a different graphic.

[0155] In addition, in this embodiment, the coverage rate 1606 of the comprehensive test is shown in a figure (arrow) representing communication between services, but this is not limited to this, and test omissions in the comprehensive test may also be shown in a figure (arrow) representing communication between services.

[0156] Possible pattern table 1705 shows an overview of the unit test patterns that can be taken by arrows 1704A-D selected by the user via input device 14, and the coverage (whether or not each unit test pattern is covered) of the integrated test. Possible pattern table 1705 has the following data items: ID 1706, which displays the second element of unit test pattern ID 1102; usability 1707, which displays information indicating whether or not the pattern is covered in the integrated test; request 1708; and response 1709. Usability "◯" indicates that the pattern is covered (implemented) in the integrated test, and usability "×" indicates that the pattern is not covered (not implemented) in the integrated test.

[0157] This figure shows an example of how the possible pattern table 1705 is displayed when the user selects arrow 1704B. For example, the row in the illustrated possible pattern table 1705 where ID 1706 is "1" has unit test pattern ID 1102 of [Acquire customer information, 1], and it can be seen that this unit test pattern has a request method of GET, a resource path of client_info, a response status of 200 with no error code, and is covered by the comprehensive test. The comprehensive test coverage derivation unit 1501 derives this information to be displayed in the possible pattern table 1705 from the total request pattern accumulation information 901, the total response pattern accumulation information 1001, the pattern linking result information 1401, and the comprehensive test coverage information 1601.

[0158] The request and response pattern details 1710 displays the request pattern information 911 and response pattern information 1011 of the unit test pattern information 1101, which has as its first element the service at the end of the arrow 1704A-D selected by the user on the input device 14, and as its second element the ID 1706 of the possible pattern table 1705 selected by the user on the input device 14.

[0159] For example, the request and response pattern details 1710 illustrated in the figure displays request pattern information 911 with request pattern ID 912 of "3" and response pattern information 1011 with response pattern ID 1012 of "3", which are shown in the row in unit test pattern information 1101 in Figure 12 where unit test pattern ID 1102 is [Customer information acquisition, 3].

[0160] The possible pattern table 1705 and the request and response pattern details 1710 allow the user to grasp the details of the unit test patterns that have been covered or not covered in the comprehensive test.

[0161] 19 is a diagram showing another example of the comprehensive test coverage visualization screen 1700. This diagram shows an example of visualization of missing unit tests. The symbol-marked arrow 1704A indicates that a test pattern that has the end-point service as the provider and is not included in the unit test pattern information 1101 exists in the comprehensive test all request and response accumulated information 401. In other words, the symbol-marked arrow 1704A indicates that there is a missing unit test that has the end-point service as the provider.

[0162] In this embodiment, the comprehensive test coverage visualization unit 1701 visualizes the MSA diagram 1602. In this example, a specific symbol is attached to the arrow to indicate a missing unit test, but this is not limited to this, and a missing unit test may be indicated in other ways, such as by using other symbols or characters, or by the color of the arrow.

[0163] A test pattern (a combination of request pattern information 911 and response pattern information 1011) that is not included in the unit test is represented by, for example, "N1" in the ID 1706 of the possible pattern table 1705 and by, for example, a predetermined symbol "!" in the usability 1707. In the request and response pattern details 1710 for such a test pattern, a note 1805 indicates that the test pattern does not exist in the unit test (in the illustrated example, "This is a request and response combination that does not exist in the unit test"). Then, for example, when a user selects the note 1805 using the input device 14, the unit test non-execution cause inference process S7 (described later) is executed, and a non-execution cause candidate 1806, which is the result of identifying the message characteristics of the test pattern, is displayed. The identified message characteristics may be, for example, the cause of the non-execution of the test pattern in the unit test. In the illustrated example of the non-execution cause candidate 1806, the comprehensive test coverage visualization unit 1701 indicates that the setting value (value deviation) of the query body variable "amount" is a message characteristic.

[0164] <Inferring causes of non-execution of unit tests> 20 and 21 are process flow diagrams illustrating the details of the process S7 for estimating the cause of non-execution of unit tests. In the unit test non-execution factor estimation process S7, first, the unit test non-execution factor estimation unit 1901 acquires the integrated test request and response information 411 having a test pattern that identifies the characteristics of a message (hereinafter referred to as the "test pattern to be estimated") (S1902). The test pattern that identifies the characteristics of a message is a test pattern for which the user has selected the remarks 1805 using the input device 14. Specifically, the unit test non-execution factor estimation unit 1901 identifies the integrated test ID 1402 having the test pattern to be estimated from the pattern linking result information 1401 based on the request pattern information 911 and response pattern information 1011 of the test pattern to be estimated, and information on the consumer service and the provider service. Then, the unit test non-execution factor estimation unit 1901 acquires the integrated test request and response information 411 having the identified integrated test ID from the integrated test all request and response accumulated information 401. In this embodiment, the unit test unexecution cause estimation unit 1901 acquires the request pattern information 911, response pattern information 1011, and information on the consumer service and provider service of the test pattern to be estimated from the comprehensive test coverage visualization unit 1701.

[0165] A specific example will be described below in which the request pattern ID 912 of the request pattern information 911 of the test pattern to be inferred is "111," the response pattern ID 1012 of the response pattern information 1011 is "99," the consumer service is "front desk," and the provider service is "account transfer service." In this case, the unit test unexecuted cause inference unit 1901 identifies the overall test ID "[front desk, account transfer, 15]" from the fourth line of the pattern linking result information 1401 in FIG. 14. The unit test unexecuted cause inference unit 1901 then acquires the overall test request and response information 411 of the identified overall test ID "[front desk, account transfer, 15]" from the overall test all request and response accumulated information 401. Hereinafter, the overall test request and response information 411 acquired in S1902 will be referred to as the "total test request and response information to be inferred."

[0166] (Comprehensive test request and response information to be estimated) 22 is a diagram showing an example of the total test request and response information 2101 to be inferred. In this figure, the total test request and response information 2101 to be inferred includes the following: 4 shows an example of comprehensive test request and response information 411 with a comprehensive test ID of "[front desk, account transfer, 15]".

[0167] Next, the unit test unexecuted factor estimation unit 1901 acquires all of the integrated test request & response information 411 and unit test request & response information 511 whose provider and request resource path are the same as those of the integrated test request & response information to be estimated (S1903). Specifically, the unit test unexecuted factor estimation unit 1901 acquires all of the integrated test request & response information 411 whose second element of the integrated test ID 412 and request resource path are the same as those of the integrated test request & response information to be estimated from the integrated test all request & response accumulated information 401. In addition, the unit test unexecuted factor estimation unit 1901 acquires all of the unit test request & response information 511 whose first element of the unit test ID 512 is the same as the second element of the integrated test ID 412 of the integrated test request & response information to be estimated and whose request resource path is the same as those of the integrated test request & response information to be estimated from the unit test all request & response accumulated information 501.

[0168] 22, the unit test unexecuted cause estimation unit 1901 acquires all of the unit test request and response information 411 whose resource path 416 is " / furikae" and whose second element of the unit test ID 412 is "account transfer" from the total unit test request and response accumulated information 401. The unit test unexecuted cause estimation unit 1901 also acquires all of the unit test request and response information 511 whose resource path 516 is " / furikae" and whose first element of the unit test ID 512 is "account transfer" from the total unit test request and response accumulated information 501.

[0169] In the subsequent processing from S1904 to S1908, the unit test non-execution factor estimation unit 1901 determines whether the comprehensive test request and response information to be estimated has a unique value or a unique set of variable names from the perspective of the value of the request method or the set of variable names in the query body of the request, for the multiple request and response information acquired in S1903. Note that the unit test non-execution factor estimation unit 1901 performs this determination, for example, for the set of variable names in the body of the response, the response status, or the value of the response error code, and if the value is unique, it may output a message to that effect in S1905, which will be described later.

[0170] First, the unit test non-execution cause estimation unit 1901 extracts one piece of request and response information from the plurality of pieces of request and response information acquired in S1903 (S1904).

[0171] Next, the unit test non-execution factor estimation unit 1901 determines whether the value of the request method in the request & response information extracted in S1904 is the same as the value of the request method in the comprehensive test request & response information to be estimated (S1905). For example, in the case of the comprehensive test request & response information 2101 to be estimated in Fig. 22, the unit test non-execution factor estimation unit 1901 determines whether the value of the request method in the request & response information extracted in S1904 is "POST". If the determination in S1905 is NO, the unit test non-execution factor estimation unit 1901 proceeds to S1907.

[0172] On the other hand, if the determination in S1905 is YES, the unit test non-execution cause estimation unit 1901 sets a flag flag_m indicating that request and response information having the same HTTP method exists in the multiple request and response information acquired in S1903 (S1906).

[0173] Next, the unit test non-execution factor estimation unit 1901 determines whether the pair of variable names in the query body of the request in the request & response information extracted in S1904 is the same as the pair of variable names in the query body of the request in the comprehensive test request & response information to be estimated (S1907). For example, in the case of the comprehensive test request & response information 2101 to be estimated in FIG. 22, the unit test non-execution factor estimation unit 1901 determines whether the pair of variable names in the query body of the request in the table extracted in S1904 is "client_num," "furikae_num," and "amount." If the determination in S1907 is NO, the unit test non-execution factor estimation unit 1901 proceeds to S1909.

[0174] On the other hand, if the determination in S1907 is YES, the unit test non-execution cause estimation unit 1901 sets a flag flag_v indicating that request and response information having the same pair of query body variable names exists in the multiple request and response information acquired in S1903 (S1908).

[0175] Next, the unit test unexecuted cause estimation unit 1901 determines whether or not the above-mentioned processes from S1904 to S1908 have been performed for all the request and response information acquired in S1903 (S1909).

[0176] If the determination in S1909 is NO, the unit test unexecuted cause estimation unit 1901 returns to S1904. That is, the unit test unexecuted cause estimation unit 1901 performs the processes from S1904 to S1908 for all the request and response information acquired in S1903.

[0177] On the other hand, if the determination in S1909 is YES, the unit test unexecuted factor estimation unit 1901 determines whether the flag flag_m is set (S1910). If the determination in S1910 is YES, the unit test unexecuted factor estimation unit 1901 proceeds to S1912.

[0178] If the flag flag_m is not set, the integrated test request and response information to be inferred contains a method value that is not contained in any of the request and response information acquired in S1903. Therefore, if the determination in S1910 is NO, the unit test non-execution cause inference unit 1901 adds "HTTP method mismatch" to the output content to be output to the non-execution cause candidate 1806 of the integrated test coverage visualization unit 1701 (S1911). Note that the text of the output content is not limited to this example.

[0179] Next, the unit test unexecuted factor estimation unit 1901 determines whether the flag flag_v is set (S1912). If the determination in S1912 is YES, the unit test unexecuted factor estimation unit 1901 proceeds to S1914.

[0180] If the flag_v is not set, the comprehensive test request and response information to be inferred contains a set of variables in the query body that are not included in any of the request and response information acquired in S1903. Therefore, if the determination in S1912 is NO, the unit test non-execution cause inference unit 1901 adds "Mismatch in set of variables in query body" to the output content (S1913). Note that the text of the output content is not limited to this example.

[0181] Next, the unit test non-execution factor estimation unit 1901 determines whether a character string is currently stored in the output content (S1914). If a character string is stored in the output content, it is estimated that the characteristic of the integrated test request and response information being estimated is that it has a unique value in terms of the value of the request method or the pair of variable names in the request query body. Therefore, if the determination in S1914 is YES, the unit test non-execution factor estimation unit 1901 outputs the output content to the integrated test coverage visualization unit 1701 (S1915). After that, the unit test non-execution factor inference unit 1901 ends the integrated test non-execution factor inference process S7. When the integrated test coverage visualization unit 1701 receives the output content from the unit test non-execution factor inference unit 1901, it displays the received output content in the non-execution factor candidate 1806.

[0182] The unit test non-execution factor inference unit 1901 may proceed to S2002 (described later) without terminating the unit test non-execution factor inference unit S7 after S1915. In this case, the comprehensive test request and response information to be inferred may have a unique query body variable name compared with the multiple pieces of request and response information extracted in S1903, and in this case, the variable name will not be included in the targets to be extracted in S2003 (described later).

[0183] On the other hand, if no character string is stored in the output content, it is not possible to infer which information in the request and response is unique in generating the comprehensive test request and response information to be inferred. Therefore, if the determination in S1914 is NO, the unit test unexecution cause inference unit 1901 proceeds to S2002, where it compares the comprehensive test request and response information to be inferred with the multiple request and response information acquired in S1903, and determines whether the variable values ​​in the query body of the request have any characteristics.

[0184] First, the unit test non-execution cause estimation unit 1901 extracts multiple pieces of request and response information with a response status of "200" from the multiple pieces of request and response information acquired in S1903 (S2002). The reason for extracting information with a status of 200 is to extract the specifications of each variable in the query body. Therefore, the status number specified in S2002 does not matter as long as normal request and response information can be extracted.

[0185] Next, the unit test non-execution cause estimation unit 1901 extracts one variable from the query body of the comprehensive test request and response information to be estimated (S2003).

[0186] Next, the unit test non-execution cause estimation unit 1901 determines whether the number of digits or type of the variable value of the variable extracted in S2003 in the query body of all request and response information extracted in S2002 that has the variable extracted in S2003 are the same, and whether the number of digits or type of the variable value of the comprehensive test request and response information to be estimated is different from the number of digits or type of the variable value of the request and response information extracted in S2002 (S2004).

[0187] 22, the variable value of client_num "123456" is six digits, but if the variable value of client_num of all the request and response information extracted in S2002 that has client_num in the query body is seven digits, the determination in S2004 will be YES. Alternatively, for example, the variable value of client_num "123456" is numeric type in the comprehensive test request and response information 2101 that is the target of inference, but if the variable value of client_num of all the request and response information extracted in S2002 that has client_num in the query body is character string type, the determination in S2004 will also be YES.

[0188] If the determination in S2004 is YES, in all the request and response information extracted in S2002, the number of digits or type of the variable value of the variable extracted in S2003 differs from the number of digits or type of the variable value in the comprehensive test request and response information 2101 to be inferred. Therefore, if the determination in S2004 is YES, the unit test unexecution cause inference unit 1901 adds "Number of digits and type mismatch of (variable name of variable extracted in S2003)" to the output content (S2005), and then proceeds to S2008. Note that the text of the output content is not limited to this example.

[0189] The processes in S2004 and S2005 may be divided into processes depending on the number of digits of the variable value of the variable extracted in S2003 and the type of the variable value.

[0190] On the other hand, if the determination in S2004 is NO, the unit test non-execution factor estimation unit 1901 determines whether the values ​​of a predetermined digit of the variable value of the variable in the query body of all request and response information extracted in S2002 having the variable extracted in S2003 are the same, and whether the value of the same digit of the variable value of the comprehensive test request and response information to be estimated is different from the variable value of the request and response information extracted in S2002 (S2006).If the determination in S2006 is NO, the unit test non-execution factor estimation unit 1901 proceeds to S2008.

[0191] For example, in the comprehensive test request and response information 2101 to be inferred in Figure 22, the value of furikae_num is "00400230456789", but if the first four digits of the furikae_num value of all request and response information extracted in S2002 that has furikae_num in the query body are "0010", the determination in S2006 will be YES based on the information in the third and fourth digits from the top.

[0192] If the determination in S2006 is YES, the value of a certain digit in the variable value of the variable extracted in S2003 in all of the request and response information extracted in S2002 differs from the value of the same digit in the variable value of the comprehensive test request and response information to be inferred. Therefore, if the determination in S2006 is YES, the unit test unexecution cause inference unit 1901 adds "Value mismatch (variable name of variable extracted in S2003)" to the output content (S2007). Note that the text of the output content is not limited to this example.

[0193] Next, the unit test non-execution factor estimation unit 1901 determines whether the variable value of the variable extracted in S2003 of the integrated test request and response information to be estimated is a numeric type (S2008). Note that in the flow of the unit test non-execution factor estimation process S7 in this embodiment, it is sufficient to determine whether the variable value is a numeric type only for the integrated test request and response information to be estimated. However, if the process proceeds to S2002 after S1915, it is also necessary to determine whether the variable value of the variable extracted in S2003 is a numeric type for all of the request and response information extracted in S2002. If the determination in S2008 is NO, the unit test non-execution factor estimation unit 1901 proceeds to S2011.

[0194] On the other hand, if the determination in S2008 is YES, the unit test non-execution factor estimation unit 1901 determines whether the variable value of the comprehensive test request and response information to be estimated is the maximum or minimum value when the variable value of the variable extracted in S2003 of the comprehensive test request and response information to be estimated and the variable values ​​of the variable in all the request and response information extracted in S2002 are arranged in ascending order (S2009).If the determination in S2009 is NO, the unit test non-execution factor estimation unit 1901 proceeds to S2011.

[0195] For example, if the variable value "3100000" of the amount in the query body of the comprehensive test request and response information 2101 to be inferred in Fig. 22 is the maximum or minimum value compared with the variable values ​​of amount in all the request and response information extracted in S2002, the determination in S2009 will be YES. This can be inferred to mean that the unit test has not been able to test an event that causes an error when the amount in the internal specifications of the account transfer service 201 or a service that depends on it becomes above or below a certain value.

[0196] If the determination in S2009 is YES, the unit test non-execution cause estimation unit 1901 outputs The content is added with "Deviation of the value of (the variable name of the variable extracted in S2003)" (S2010). Note that the text of the output content is not limited to this example.

[0197] Next, the unit test non-execution cause estimation unit 1901 determines whether or not the processes from S2003 to S2010 have been performed for all variables in the query body of the comprehensive test request and response information to be estimated (S2011).

[0198] If the determination in S2011 is NO, the unit test non-execution factor estimation unit 1901 returns to S2003. That is, the unit test non-execution factor estimation unit 1901 performs the processes from S2003 to S2010 for all variables in the query body of the integrated test request and response information to be estimated.

[0199] On the other hand, if the determination in S2011 is YES, the unit test non-execution factor estimation unit 1901 determines whether or not a character string exists in the output content at this time (S2012).

[0200] If the determination in S2012 is YES, the unit test unexecuted factor inference unit 1901 outputs the output content to the overall test coverage visualization unit 1701 (S2013). Thereafter, the unit test unexecuted factor inference unit 1901 ends this unit test unexecuted factor inference process S7. When the output content is input from the unit test unexecuted factor inference unit 1901, the overall test coverage visualization unit 1701 displays the input output content in the unexecuted factor candidate 1806.

[0201] On the other hand, if no character string is present in the output content, it is not possible to infer which information in the request and response is unique in generating the integrated test request and response information to be inferred. Therefore, if the determination in S2012 is NO, the unit test non-execution factor inference unit 1901 outputs "inference failed" as output content to the integrated test coverage visualization unit 1701 (S2014). Thereafter, the unit test non-execution factor inference unit 1901 ends the unit test non-execution factor inference process S7. When the output content "inference failed" is input from the unit test non-execution factor inference unit 1901, the integrated test coverage visualization unit 1701 displays the input output content "inference failed" in the unexecution factor candidate 1806.

[0202] The process S7 for inferring the cause of non-execution of unit tests makes it possible to infer which information in the request and response is peculiar for the comprehensive test request and response information 411 having test patterns for which unit tests have been omitted, from the perspective of the API specifications and values ​​of the query body.

[0203] In this embodiment, the description is based on the assumption that comprehensive test request and response information for one inference target is acquired in S1902 of the unit test non-execution factor inference process S7, but comprehensive test request and response information for multiple inference targets may be acquired in S1902. In this case, the unit test non-execution factor inference unit 1901 may perform subsequent processing on the comprehensive test request and response information for any one of the inference targets. Alternatively, the unit test non-execution cause estimation unit 1901 may perform S2004 and S2005 when the number of digits and type of the variable of S2003 are the same in the comprehensive test request and response information of all estimation targets acquired in S1902, perform S2006 and S2007 only for a certain digit of the variable of S2003 when the value of that digit is the same in the comprehensive test request and response information 2101 of all estimation targets acquired in S1902, perform S2008 to S2010 when the variable value extracted in S2003 is a numeric type in the comprehensive test request and response information 2101 of all estimation targets acquired in S1902, and determine YES in S2009 when the value of the comprehensive test request and response information of all estimation targets acquired in S1902 is fixed at the minimum value or the maximum value, or the like.

[0204] As described above, the insufficient test identifying device 10 of this embodiment is a device that handles multiple services in a message format. The detection device detects insufficient test execution of services performed on a test target system (20) that communicates messages with each other, and includes: a unit test case abstraction unit (601) that identifies test patterns in the unit tests by abstracting information on messages communicated by the test target services in the unit tests; a unit test pattern linking unit (1301) that determines whether the identified test patterns can be associated with messages of each service in a comprehensive test that is performed by communicating messages between a plurality of test target services, and, if so, associates the messages with the test patterns; and a comprehensive test coverage derivation unit (1501) that detects insufficient message patterns in the unit tests or comprehensive tests based on the association results.

[0205] The insufficient test identification device 10 of this embodiment can improve the accuracy of understanding the correspondence between unit tests and comprehensive tests by correlating test patterns that abstract messages communicated in unit tests that check the behavior of individual services with comprehensive tests (tests that check the behavior between services) that are conducted by communicating messages between multiple services. This makes it possible to accurately detect insufficient test executions conducted on a test target system 20 in which multiple services communicate messages with each other.

[0206] Furthermore, the insufficient test identification device 10 of this embodiment generates request patterns and response patterns that abstract the requests and responses communicated by each service in the unit test, and uses the pairs of request patterns and response patterns as test patterns. In the comprehensive test, the requests and responses communicated by each service are associated with the test patterns that are pairs of request patterns corresponding to the requests and response patterns corresponding to the responses.

[0207] The requests and responses communicated between services are data that originate from different objects (services) and must correspond to each other. Therefore, by abstracting and patterning this data, it becomes easier to find gaps and duplications in tests, thereby increasing test coverage and reducing gaps.

[0208] Furthermore, the insufficient test identifying device 10 of this embodiment abstracts the message by performing a predetermined process on the value of the variable included in the message.

[0209] The content of each test performed using a message varies depending on the type (name) or number of variables in the message. Therefore, by performing a specific processing (e.g., deletion) on the variable values, the combination of variable names can be used as a message pattern. This allows you to efficiently detect insufficient tests based on the names of the variables that define the test content.

[0210] Furthermore, the insufficient test identifying device 10 of this embodiment detects the test coverage rate or test omissions in the comprehensive test, or test omissions in the unit test, based on the result of the association.

[0211] With this configuration, it is possible to more specifically identify a lack of test patterns in the comprehensive test or the unit test.

[0212] Furthermore, the insufficient test identifying device 10 of this embodiment calculates the coverage rate of the comprehensive test based on the number of test patterns executed in the comprehensive test relative to the total number of test patterns.

[0213] By defining coverage in this way, it is possible to objectively determine whether comprehensive testing has been performed sufficiently in relation to unit testing.

[0214] Furthermore, when the insufficient test identification device 10 of this embodiment determines that the identified test pattern cannot be associated with a message communicated in the comprehensive test, it identifies the characteristics of the message of the pattern that has not been implemented in the unit test based on the combination of variables in the message, the variable values, or the request method, and outputs the characteristics of the identified message.

[0215] The actual content of the test varies depending on the combination of variables in the message, the variable values, and the request method. Therefore, by using the combination of variables, the variable values, or the request as a criterion, it is possible to efficiently and accurately identify the characteristics of messages with patterns that are not implemented in unit tests.

[0216] Furthermore, when the insufficient test identification device 10 of this embodiment determines that the identified test pattern cannot be associated with a message communicated in the comprehensive test, it identifies the characteristics of the message of the pattern that has not been implemented in the unit test by comparing the message with other messages communicated by the service that communicated the message in the unit test or the comprehensive test.

[0217] If a message of a pattern not implemented in a unit test contains a combination of variables, variable values, or request methods that are not used by other messages communicated in a unit test or a comprehensive test, the message contains data that substantially determines the content of the test, as described above. Therefore, by comparing such a message with other messages communicated by the service in a unit test or a comprehensive test, it is possible to identify characteristics that differ from other messages. This makes it possible to efficiently identify characteristics of messages of a pattern not implemented in a unit test.

[0218] Furthermore, if the insufficient test identification device 10 of this embodiment determines that the identified test pattern cannot be associated with the request and its response communicated in the comprehensive test, it identifies a combination of methods or variables that the service that sent the response did not use in other requests and their responses communicated in the unit test and comprehensive test.

[0219] There are a large number of combinations of methods or variables that can be used in unit tests. Therefore, it is time-consuming and laborious to manually identify combinations of methods or variables as characteristics of messages with patterns that are not implemented in unit tests. Therefore, combinations of methods or variables that are not used in other requests and their responses communicated by the service in unit tests or comprehensive tests are identified as characteristics of messages with patterns that are not implemented in unit tests. This allows users to easily identify combinations of methods or variables that are not used in unit tests and accurately identify message characteristics.

[0220] Furthermore, when the insufficient test identification device 10 of this embodiment determines that the identified test pattern cannot be associated with the request and its response communicated in the comprehensive test, it identifies specifications that differ from those of the service that sent the response by comparing them with the normal request and its response communicated in the unit test and comprehensive test.

[0221] With this configuration, specifications (e.g., the number of digits, type, or size of a variable) that differ from the normal requests and responses communicated by the service in unit tests or comprehensive tests are identified as characteristics of messages with patterns that have not been implemented in unit tests. This allows the user to identify omissions in test cases when, for example, an abnormal value (e.g., a value outside the range defined in the design) is set for a variable.

[0222] Furthermore, the insufficient test identifying device 10 of this embodiment controls the communication between each service by The degree of insufficiency in test execution for communication between services in unit tests or comprehensive tests is displayed by changing the display mode of the figures.

[0223] By displaying communications between services using a diagram connecting each service, the correspondence between the services in terms of communication becomes easier to understand. Since insufficient test execution is determined based on the service, connecting services makes it easier to understand the insufficient test execution for each service. Furthermore, since insufficient test execution is determined based on the service as described above, the display mode of the diagram connecting services varies depending on the degree of insufficient test execution, making it possible to clearly display the degree of insufficient test execution for each service. This allows the user to easily understand the degree of insufficient test execution for services performed on a system under test 20 in which multiple services communicate messages with each other.

[0224] The present invention is not limited to the above-described embodiments, and can be implemented using any components within the scope of the present invention. The above-described embodiments and modifications are merely examples, and the present invention is not limited to these contents as long as the characteristics of the invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0225] For example, part of the hardware provided in each device of this embodiment may be provided in another device.

[0226] Furthermore, each program of the insufficient test identifying device 10 may be provided in another device, a program may be made up of multiple programs, or multiple programs may be integrated into one program.

[0227] For example, the test deficiency identification device 10 may include an interface for inputting and outputting data using a non-transitory storage medium. Data may be input and output via the interface in the test deficiency identification device 10. As an example of the storage medium, a portable non-transitory data storage medium may be used.

[0228] In addition, in this embodiment, an example has been described in which a unit test of a single service is the first test and a comprehensive test conducted by communication between multiple services is the second test, but this is not limited to this, and the second test may be conducted by communication between the components that are the test targets of the first test.For example, a test of classes that make up a service may be the first test, and a test of a single service may be the second test.

[0229] Furthermore, the information included in the comprehensive test request and response information 411 or the unit test request and response information 511 is not limited to the information exemplified in this embodiment, as long as the abstraction method in the unit test case abstraction unit 601 and the linking method in the request and response pattern linking unit 1201 are determined. For example, the comprehensive test request and response information 411 or the unit test request and response information 511 may have header information. [Explanation of symbols]

[0230] 1 Information processing system, 10 Test shortage identification device, 20 Systems Under Test, 601 Unit Test Case Abstraction Unit, 1201 Request & Response Pattern Linking Section, 1301 Unit test pattern linking part, 1501 Comprehensive test coverage derivation part, 1701 Comprehensive Test Coverage Visualization Department, 1901 Unit test unexecuted factor estimation part, 401 Comprehensive test all request & response accumulation information, 501 Unit test all request & response accumulation information, 901 All request pattern accumulation information, 1001 All response pattern accumulation information, 1101 Unit test pattern information, 1401 Pattern linking result information, 1601 Comprehensive Test Coverage Information

Claims

1. a processor and a memory; 1. A detection device for detecting a deficiency in component test execution performed on a system in which a plurality of components communicate messages with each other, comprising: an abstraction unit that identifies a test pattern for the first test by abstracting information of messages communicated by a component under test in the first test; a test pattern linking unit that determines whether the specified test pattern can be associated with a message of each component in a second test that is performed by communicating messages among the plurality of test target components, and, if the association is possible, associates the message with the test pattern; a detection unit that detects a lack of a message pattern of the first test or the second test based on the result of the association; A detection device comprising:

2. the abstraction unit generates a request pattern and a response pattern by abstracting the requests and responses communicated by each component in the first test, and defines a set of the request pattern and the response pattern as the test pattern; The test pattern associating unit associates a request and a response communicated by each component in the second test with the test pattern of a set of the request pattern corresponding to the request and the response pattern corresponding to the response. The detection device according to claim 1 .

3. The abstraction unit abstracts the message by performing predetermined processing on the value of a variable included in the message. The detection device according to claim 1 .

4. The detection unit detects test coverage or test omissions in the second test, or test omissions in the first test, based on the result of the association. The detection device according to claim 1 .

5. The detection unit calculates a coverage rate of the second test based on the number of the test patterns executed in the second test relative to the total number of the test patterns. The detection device according to claim 4.

6. an identification unit that, when it is determined that the identified test pattern cannot be associated with the message communicated in the second test, identifies characteristics of a message of a pattern that has not been executed in the first test based on a combination of variables, a value of a variable, or a method of a request in the message; an output unit that outputs the identified features of the message; Equipped with The detection device according to claim 1 .

7. When the identification unit determines that the identified test pattern cannot be associated with the message communicated in the second test, the identification unit identifies characteristics of the message of the pattern not executed in the first test by comparing the message with other messages communicated by the component that communicated the message in the first test or the second test. The detection device according to claim 6.

8. When the identifying unit determines that the identified test pattern cannot be associated with the request and its response communicated in the second test, the identifying unit identifies a combination of methods or variables that are not used by the component that sent the response in other requests and their responses communicated in the first test and the second test. The detection device according to claim 7.

9. When it is determined that the identified test pattern cannot be associated with the request and its response communicated in the second test, the identifying unit identifies different specifications by comparing the component that sent the response with the normal request and its response communicated in the first test and the second test. The detection device according to claim 7.

10. a display unit that displays the communication between each component by a graphic connecting the components, and that displays the degree of insufficiency in test execution for the communication between each component in the first test or the second test by changing the display mode of the graphic; The detection device according to claim 1 .

11. An information processing device for detecting a lack of component test execution performed on a system in which a plurality of components communicate messages with each other, an abstraction process for identifying a test pattern for the first test by abstracting information of messages communicated by a component under test in the first test; a test pattern linking process for determining whether the identified test pattern can be associated with a message of each component in a second test that is performed by communicating messages among the plurality of test target components, and, if the association is possible, associating the message with the test pattern; and executing a detection process for detecting a lack of a message pattern of the first test or the second test based on the result of the association. Detection method.

12. An information processing device for detecting a lack of component test execution performed on a system in which a plurality of components communicate messages with each other, an abstraction process for identifying a test pattern for the first test by abstracting information of messages communicated by a component under test in the first test; a test pattern linking process for determining whether the identified test pattern can be associated with a message of each component in a second test that is performed by communicating messages among the plurality of test target components, and, if the association is possible, associating the message with the test pattern; and executing a detection process for detecting a lack of a message pattern of the first test or the second test based on the result of the association. Detection program.

Images