Test support system, test support method, and program

Abstract

To provide a technique which automates a communication test.SOLUTION: A test support system includes: a first apparatus; a simulation device in which a simulator which simulates a second apparatus is mounted; and a test support device which supports a communication test between the first apparatus and the second apparatus. The test support system comprises: an executable file creation unit which creates an executable file, when a source code and a test scenario are registered with a repository; a first script creation unit which creates a test environment construction script in which instructions for constructing a test environment are included, on the basis of the test scenario and the simulator information; a second script creation unit which creates a test execution script in which the instructions for performing the communication test are included, on the basis of the test scenario; a first transmission unit which transmits the test environment construction script and the test execution script to the second apparatus; and a second transmission unit which transmits installation instructions of the executable file to the first apparatus.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present disclosure relates to a test support system, a test support method, and a program. [Background technology]

[0002] A method known as continuous integration (CI) / continuous delivery (CD) is one of the known program development techniques. Programs that implement CI / CD are called CI / CD tools, and have been widely used in program development in recent years. Note that one known prior art related to CI / CD is, for example, a technique for automatically and selectively optimizing a CI / CD pipeline (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2023-527700 Summary of the Invention [Problem to be solved by the invention]

[0004] CI / CD tools can automatically execute builds, tests, deployments, etc., but tests generally involve static analysis or unit testing of individual modules, and communication tests that assume communication with external devices cannot be automated.

[0005] The present disclosure has been made in consideration of the above points, and provides a technology for automating communication tests. [Means for solving the problem]

[0006] A test support system according to one embodiment of the present disclosure includes a first device, a simulation device equipped with a simulator that simulates a second device that communicates with the first device, and a test support device that supports a communication test between the first device and the second device. The test support device includes: an executable file creation unit that, when source code to be the subject of the communication test and a test scenario representing a scenario for the communication test are registered in a repository of a version control system, creates an executable file of the source code in response to the registration; a first script creation unit that creates a test environment creation script including instructions for constructing a test environment for the communication test using the simulator based on the test scenario and simulator information that indicates information about a simulator that simulates the second device; a second script creation unit that creates a test execution script based on the test scenario, the test environment creation script including instructions for the second device to transmit transmission data to the first device and perform the communication test; a first transmission unit that transmits the test environment creation script and the test execution script to the second device; and a second transmission unit that transmits an instruction to install the executable file to the first device. [Effects of the Invention]

[0007] Techniques are provided for automating communications testing. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining a conventional example of a communication test. [Figure 2] 1 is a diagram illustrating an example of the overall configuration of a development support system according to a first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a test support program according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a simulation apparatus according to the first embodiment. [Figure 5]10 is a flowchart illustrating an example of a test support process according to the first embodiment. [Figure 6] FIG. 10 illustrates an example of a test scenario. [Figure 7] FIG. 10 is a diagram illustrating an example of a test environment creation script. [Figure 8] FIG. 10 illustrates an example of a test execution script. [Figure 9] FIG. 2 is a diagram illustrating an example of the configuration of a target embedded device. [Figure 10] 10 is a flowchart illustrating an example of a test execution process according to the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of test result determination. [Figure 12] FIG. 10 is a diagram showing an example of test result information. [Figure 13] FIG. 2 illustrates an example of a hardware configuration of a computer. [Figure 14] FIG. 10 is a diagram illustrating an example of replacement with a mockup. [Figure 15] FIG. 10 is a diagram illustrating an example of a functional configuration of a simulation apparatus according to a second embodiment. [Figure 16] 10 is a flowchart illustrating an example of a test execution process according to the second embodiment. [Figure 17] FIG. 10 is a diagram illustrating an example of a priority order for replacement with a mock. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0010] <Continuous Integration / Continuous Delivery> First, we will explain the outline of continuous integration (CI) and continuous delivery (CD). CI and CD are sometimes collectively referred to as "CI / CD."

[0011] Both CI / CD are program development methods in which the contents of processes (jobs) such as build, test, and deploy are defined as scripts, and the scripts are automatically executed when triggered by registration (commit) to a version control system that manages source code. A series of processes achieved by executing one or more jobs is called a "pipeline" or "pipeline processing." Generally, CI refers to a method for automating builds and tests, while CD refers to a method for automating deployment.

[0012] Programs that achieve CI / CD are called CI tools / CD tools, etc. These tools execute jobs according to the processing content (scripts) and execution order of those processes defined in a pipeline configuration file. For this reason, users need to define the processing content of the jobs they want to automate and the execution order of those processes in the pipeline configuration file.

[0013] <Conventional example of communication test> As an example, the following describes a conventional example of software development for embedded devices, in which a communication test is performed between the embedded device with the software (program) installed and an external measuring device, with reference to Figure 1. Hereinafter, the embedded device that is the target of the communication test will be referred to as the "target embedded device." Examples of embedded devices include various types of industrial equipment such as freezers, showcases, and vending machines, but these are all examples and are not limited to these.

[0014] As shown in Figure 1, (1) when source code is registered in a repository by a developer's device, (2) this registration triggers an instruction to execute a pipeline to the CI / CD tool. As a result, the CI / CD tool, according to the pipeline configuration file, (3) instructs the build tool to build the source code, (4) instructs the test tool to test (unit tests, etc.) the executable file created by this build, and then (5) instructs the deployment tool to deploy the executable file that passed the test. As a result, this executable file is deployed to the distribution server. Note that before issuing the build instruction, the test tool may also be instructed to perform static analysis of the source code, etc.

[0015] The person conducting the communication test (hereinafter referred to as the "tester") (6) operates the target embedded device to download the executable file from the distribution server. The tester also (7) operates the simulation device to start the simulator of the measuring device and sets various parameters for the measuring device (e.g., the connection interface used for communication with the target embedded device, etc.) to create a test environment. Note that the measuring device is a control device (e.g., a programmable controller, etc.) that controls the embedded device. The simulation device is, for example, a PC (personal computer), and can create a test environment that simulates the measuring device using the simulator.

[0016] The tester then (8) operates the simulation device to conduct a communication test between the target embedded device and the measurement device (the measurement device simulated by the simulator on the simulation device). The tester then (9) operates his / her own terminal to obtain the test results from the target embedded device, and (10) sends the test results by email to the developer terminal. The communication test may also be called, for example, a "connection test."

[0017] As described above, in communication tests, measurement equipment is simulated using a simulator installed on a PC or other device, so the test environment must be created manually, including starting the simulator and setting test data. Furthermore, the communication test itself is performed manually by operating the simulation device, which is time-consuming and subject to the risk of errors. Furthermore, test results must be manually sent by email, which is time-consuming.

[0018] Therefore, the following describes a development support system 1 that can automatically build a test environment when deploying source code that is the target of a communication test, and then automatically conduct the communication test, and also automatically acquire and notify the test results. This development support system 1 eliminates the need for the user to manually build a test environment and conduct a communication test, and also eliminates the need to manually notify the test results, thereby reducing the workload required for communication tests. This reduces the number of test steps and also prevents mistakes during testing.

[0019] [First embodiment] First, a development support system 1 according to the first embodiment will be described.

[0020] <Example of overall configuration of development support system 1 according to the first embodiment> 2, the development support system 1 according to the first embodiment includes a version control system 10, a CI / CD system 20, a distribution server 30, a developer terminal 40, a target embedded device 50, and a simulation device 60. The version control system 10 and the CI / CD system 20, the version control system 10 and the developer terminal 40, the version control system 10 and the simulation device 60, the CI / CD system 20 and the distribution server 30, the CI / CD system 20 and the simulation device 60, the distribution server 30 and the target embedded device 50, and the target embedded device 50 and the simulation device 60 are connected to each other so as to be able to communicate with each other via a communication network such as an in-house network or a test environment network.

[0021] The version control system 10 is a computer or computer system such as a general-purpose server that includes a version control program 110 and a repository 120 .

[0022] When the developer terminal 40 registers source code and a test scenario in the repository 120, the version control program 110, triggered by this registration, sends a pipeline execution instruction to the CI / CD system 20 to build, test (unit test), and deploy the source code, as well as to perform a communication test. Here, the test scenario refers to a scenario for a communication test between the target embedded device 50 and a measuring device simulated by a simulator on the simulation device 60. The test scenario includes information required to build a test environment, such as the type (type of connected device) and number (number of connected devices) of embedded devices connected to the measuring device, connection I / F, etc., and information required to set and send transmission data to be sent from the measuring device (measuring device simulated by a simulator on the simulation device 60) to the target embedded device 50.

[0023] Furthermore, when the simulation device 60 registers test result information including the test results of the communication test in the repository 120, the version control program 110 uses this registration as a trigger to send an email to the developer terminal 40 notifying the developer that the test result information has been registered. This notifies the developer of the results of the communication test between the target embedded device 50 and the measuring device. However, sending an email is only one example of a means for notifying the developer of the test results, and the test results may also be notified by means other than email (e.g., a chat application, etc.).

[0024] The repository 120 is a storage area for managing source code, test scenarios and their change history, test result information, various scripts, and the like.

[0025] The CI / CD system 20 is a computer or computer system such as a general-purpose server having a CI / CD tool 210, a build tool 220, a test tool 230, a deployment tool 240, a test support program 250, and a simulator information DB 260.

[0026] The CI / CD tool 210 executes a pipeline upon receiving a pipeline execution instruction from the version control system 10. The CI / CD tool 210 includes a build instruction unit 211, a test instruction unit 212, a deploy instruction unit 213, and a pipeline configuration file 214. Upon receiving a pipeline execution instruction, the CI / CD tool 210 issues a build instruction for the target source code using the build instruction unit 211 in accordance with the instruction and the pipeline configuration file 214, and issues a test instruction for the source code or an executable file of the source code using the test instruction unit 212. After that, the CI / CD tool 210 issues a deploy instruction for the executable file using the deploy instruction unit 213. The CI / CD tool 210 also transmits a test scenario and information indicating a deployment destination of the executable file (hereinafter also referred to as "deployment destination information") to the test support program 250.

[0027] The build tool 220 builds the source code to create an executable file in accordance with the build instructions.

[0028] The test tool 230 tests the source code or executable files (for example, static analysis for source code, unit testing for executable files, etc.) in response to the test instructions.

[0029] The deployment tool 240 deploys the executable file to the distribution server 30 in response to the deployment instruction.

[0030] Any known CI / CD tool can be used as the CI / CD tool 210, such as GitLab. Any known build tool can be used as the build tool 220, such as Maven, Ant, or Gradle. Any known test tool can be used, such as Jtest. Any known deployment tool can be used as the deployment tool 240, such as an SSH (Secure Shell) tool that can use the scp command.

[0031] When the test support program 250 receives a test scenario from the CI / CD tool 210, it creates a test environment construction script based on the test scenario and simulator information stored in the simulator information DB 260. The test support program 250 also creates a test execution script based on the test scenario. The test support program 250 then stores (preserves) the test environment construction script and the test execution script in the repository 120, and transmits the test scenario, the test environment construction script, and the test execution script to the simulation device 60. Here, the test environment construction script is a script (program) for constructing a test environment (a simulator that simulates a measuring device) on the simulation device 60. The test execution script is a script (program) for setting transmission data for a measuring device simulated by a simulator on the simulation device 60 and for causing the transmission data to be transmitted.

[0032] Furthermore, when the test support program 250 receives the deployment destination information from the CI / CD tool 210, it transmits a program update instruction to the target embedded device 50 to download and install the executable file deployed to the deployment destination indicated by the deployment destination information. The test support program 250 will be described in detail later.

[0033] The simulator information DB 260 stores simulator information for constructing a test environment. The simulator information is information for identifying a simulator and its parameters, for example, from the "connection device type" and "connection I / F" included in a test scenario. For each connection device type, the simulator information includes, for example, the name of a simulator that simulates a measuring device capable of controlling an embedded device of that connection device type, and the file path of an executable file for starting the simulator. The simulator information also includes, for each measuring device and connection I / F, parameters (e.g., COM number, eth number, baud rate, communication frame information, etc.) that need to be set in the simulator to simulate communication via the connection I / F with that measuring device.

[0034] The distribution server 30 is a computer or computer system such as a general-purpose server on which executable files of source code that are the subject of communication testing are deployed.

[0035] The developer terminal 40 is a terminal (for example, a PC, a smartphone, a tablet terminal, etc.) used by a person (developer) who develops source code and test scenarios.

[0036] The target embedded device 50 is an embedded device in which an executable file of source code to be subjected to a communication test is installed. In response to a program update instruction transmitted from the CI / CD system 20, the target embedded device 50 acquires and installs the executable file from the distribution server 30.

[0037] The simulation device 60 is a computer or computer system such as a PC equipped with a simulator capable of simulating a measuring device. The simulation device 60 executes a test environment construction script transmitted from the CI / CD system 20, starts a simulator, sets parameters, and so on to construct a test environment. The simulation device 60 also executes a test execution script transmitted from the CI / CD system 20, and executes a communication test by setting and transmitting transmission data. Details of the simulation device 60 will be described later.

[0038] 2 is an example, and the overall configuration of the development support system 1 is not limited to this. For example, two or more systems among the version control system 10, the CI / CD system 20, and the distribution server 30 may be configured as an integrated system. In this case, for example, all or part of the system may be realized by a virtual machine (VM). Furthermore, in the example shown in FIG. 2, only one target embedded device 50 and one simulation device 60 are shown, but there may be multiple of each.

[0039] <Example of functional configuration of test support program 250 according to the first embodiment> As shown in FIG. 3, the test support program 250 according to the first embodiment can realize a script creation unit 251, a scenario / script transmission unit 252, a script storage unit 253, and a program update instruction unit 254 by processing executed by a processor such as a CPU (Central Processing Unit).

[0040] The script creation unit 251 creates a test environment construction script based on the test scenario received from the CI / CD tool 210 and the simulator information stored in the simulator information DB 260. The script creation unit 251 also creates a test execution script based on the test scenario received from the CI / CD tool 210.

[0041] The scenario / script transmission unit 252 transmits the test scenario received from the CI / CD tool 210 and the test environment creation script and test execution script created by the script creation unit 251 to the simulation device 60.

[0042] The script storage unit 253 stores (preserves) the test environment creation script and the test execution script created by the script creation unit 251 in the repository 120.

[0043] Based on the deployment destination information received from the CI / CD tool 210, the program update instruction unit 254 sends a program update instruction to the target embedded device 50 to download and install the executable file deployed to the deployment destination indicated by the deployment destination information.

[0044] <Example of functional configuration of simulation device 60 according to the first embodiment> 4, the simulation device 60 according to the first embodiment includes a script and expected result storage unit 601, an environment construction unit 602, a test execution unit 603, a test result creation unit 604, and a test result storage unit 605. Each of these units is realized, for example, by a processor such as a CPU executing one or more programs including a simulator installed in the simulation device 60. The simulation device 60 according to the first embodiment also includes a script and expected result storage unit 606 and a test result storage unit 607. Each of these storage units is realized, for example, by a storage area such as an auxiliary storage device included in the simulation device 60.

[0045] When the script and expected result storage unit 601 receives a test scenario, a test environment creation script, and a test execution script from the CI / CD system 20, it stores the expected result included in the test scenario, the test environment creation script, and the test execution script in the script and expected result storage unit 606. The expected result is the correct result expected as a result of executing control according to transmission data when that transmission data is sent to the target embedded device 50.

[0046] The environment construction unit 602 executes the test environment construction script stored in the script / expected result storage unit 606 to construct a test environment.

[0047] The test execution unit 603 executes a test execution script stored in the script / expected result storage unit 606 to execute a communication test with the target embedded device 50. That is, the test execution unit 603 sets transmission data in the measuring device according to the test execution script, and transmits the transmission data to the target embedded device 50. As a result, the target embedded device 50 that receives the transmission data executes control according to the transmission data.

[0048] The test result creation unit 604 acquires the execution results of the target embedded device 50, and then determines the test results from the execution results using the expected results stored in the script and expected result storage unit 606. The test result creation unit 604 also creates test result information that includes the test results, and stores the test result information in the test result storage unit 607.

[0049] When the communication test with the target embedded device 50 is completed, the test result saving unit 605 saves (registers) the test result information saved in the test result storage unit 607 in the communication test in the repository 120. This registration triggers the version control program 110 to send an email to the developer terminal 40 notifying that the test result information has been registered.

[0050] The script and expected result storage unit 606 stores the expected results, test environment creation scripts, and test execution scripts saved by the script and expected result saving unit 601 .

[0051] The test result storage unit 607 stores the test result information created and saved by the test result creation unit 604 .

[0052] <Test Support Process According to the First Embodiment> The test support process for automatically executing a communication test on the target embedded device 50 in response to the registration of the source code will be described below with reference to FIG.

[0053] The developer terminal 40 registers the source code to be subjected to the communication test and the test scenario of the communication test in the repository 120 (step S101). This registration triggers a pipeline execution instruction to be sent from the version control program 110 to the CI / CD tool 210.

[0054] When the CI / CD tool 210 receives the pipeline execution instruction, it executes the pipeline to build and test (static analysis, unit test, etc.) the source code, and deploys the executable file of the source code to the distribution server 30 (step S102). At this time, the CI / CD tool 210 also obtains a test scenario from the repository 120, and transmits this test scenario and deployment destination information of the executable file (for example, the folder path of the folder in the distribution server 30 where the executable file is stored, etc.) to the test support program 250.

[0055] The test support program 250 creates a test environment creation script and a test execution script (step S103). Here, as an example, a case where a test environment creation script and a test execution script are created from a test scenario 1000 shown in FIG. 6 will be described below. The test scenario 1000 shown in FIG. 6 includes test environment information 1100 and transmission data information 1200. The test environment information 1100 is information necessary for creating a test environment, and in the example shown in FIG. 6, it includes first test environment information 1110 and second test environment information 1120. The first test environment information 1110 includes connection device type information 1111 indicating that the connection device type is a "refrigerator manufactured by Company A," connection number information 1112 indicating that the number of connected target embedded devices 50 of this connection device type is "two," and connection I / F information 1113 indicating that "RS-485" is used as the connection I / F. Similarly, the second test environment information 1120 includes connected device type information 1121 indicating that the connected device type is a "showcase manufactured by Company B," connected device number information 1122 indicating that the number of connected target embedded devices 50 of this connected device type is "three," and connection I / F information 1123 indicating that "RS-485" is used as the connection I / F. Furthermore, the transmission data information 1200 includes, for each communication count (test case), the transmission data to be transmitted in that test case and the expected results expected as the execution results of control according to the transmission data. In the example shown in FIG. 6, the transmission data information 1200 includes, as the transmission data for test case 1, two pieces of transmission data 1211 and 1212 to be transmitted respectively to the two freezers of the "freezer manufactured by Company A," and three pieces of transmission data 1213 to 1215 to be transmitted respectively to the three showcases of the "showcase manufactured by Company B." In addition, the transmission data information 1200 includes, as expected results for test case 1, expected result 1221 for transmission data 1211, expected result 1222 for transmission data 1212, expected result 1223 for transmission data 1213, expected result 1224 for transmission data 1214, and expected result 1225 for transmission data 1215.

[0056] It goes without saying that the test scenario 1000 shown in FIG. 6 is an example and is not limited to this.

[0057] -Creating a test environment construction script The script creation unit 251 creates a test environment creation script based on the test environment information 1100 included in the test scenario 1000 shown in Fig. 6 and the simulator information stored in the simulator information DB 260. For example, the script creation unit 251 creates a test environment creation script 2000 shown in Fig. 7 based on the test environment information 1100 and the simulator information stored in the simulator information DB 260. The test environment creation script 2000 shown in Fig. 7 is data in a bat file format, and includes a first test environment creation command 2100 and a second test environment creation command 2200.

[0058] The first test environment construction command 2100 includes a company A refrigerator simulator start command 2101, a connection I / F setting command 2102, and a connection number setting command 2103. The company A refrigerator simulator start command 2101 is a command for starting a simulator that simulates a measuring instrument capable of controlling a company A refrigerator. The company A refrigerator simulator start command 2101 is created, for example, based on connection device type information 1111 and a file path of a simulator that simulates a measuring instrument that can control an embedded device of the connection device type indicated by the connection device type information 1111. The connection I / F setting command 2102 is a command for setting a connection I / F of the measuring instrument to be simulated by the simulator. The connection I / F setting command 2102 is created, for example, based on connection I / F information 1113 and parameters that need to be set in the simulator to simulate communication via the connection I / F indicated by the connection I / F information 1113, such as a COM number, an eth number, a baud rate, and communication frame information. The connection number setting command 2103 is a command for setting the number of connected measuring instruments to be simulated by the simulator. The connection number setting command 2103 is created based on the connection number information 1112, for example.

[0059] Similarly, the second test environment construction command 2200 includes a company B showcase simulator startup command 2201, a connection I / F setting command 2202, and a connection number setting command 2203. The company B showcase simulator startup command 2201 is a command for starting a simulator that simulates a measuring device capable of controlling a company B showcase. The company B showcase simulator startup command 2201 is created, for example, based on the connection device type information 1121 and the file path of a simulator that simulates a measuring device capable of controlling an embedded device of the connection device type indicated by the connection device type information 1121. The connection I / F setting command 2202 is a command for setting the connection I / F of the measuring device to be simulated by the simulator. The connection I / F setting command 2202 is created, for example, based on the connection I / F information 1123 and information such as a COM number, an eth number, a baud rate, and a communication frame, which are parameters that need to be set in the simulator to simulate communication via the connection I / F indicated by the connection I / F information 1123. The connection number setting command 2203 is created based on the connection number information 1122, for example.

[0060] 7 assumes that the response data sent by the measuring device when it receives data sent from the target embedded device 50 is default response data set in the simulator. However, this is not limiting, and the test environment creation script may include, for example, a command to set response data sent back by the measuring device simulated by the simulator, depending on the value included in the data sent from the target embedded device 50.

[0061] ·Creating test execution scripts The script creation unit 251 creates a test execution script based on transmission data information 1200 included in the test scenario 1000 shown in Fig. 6. For example, the script creation unit 251 creates a test execution script 3000 shown in Fig. 8 based on the transmission data information 1200. The test execution script 3000 shown in Fig. 8 is data in a bat file format, and includes, for each test case, a transmission data setting and transmission command that describes the setting of transmission data and a transmission command for that test case. In the example shown in Fig. 8, a transmission data setting and transmission command 3100 for test case 1 is included, and the transmission data setting and transmission command 3100 includes transmission data setting commands 3101, 3103, 3105, 3107, and 3109, and transmission data transmission commands 3102, 3104, 3106, 3108, and 3110.

[0062] Note that the transmission data setting command 3101 and the transmission data sending command 3102 are created from the transmission data 1211 included in the test scenario 1000 shown in Fig. 6. Similarly, the transmission data setting command 3103 and the transmission data sending command 3104 are created from the transmission data 1212 included in the test scenario 1000 shown in Fig. 6. Similarly, the transmission data setting command 3105 and the transmission data sending command 3106 are created from the transmission data 1213 included in the test scenario 1000 shown in Fig. 6. Similarly, the transmission data setting command 3107 and the transmission data sending command 3108 are created from the transmission data 1214 included in the test scenario 1000 shown in Fig. 6. Similarly, the transmission data setting command 3109 and the transmission data sending command 3110 are created from the transmission data 1215 included in the test scenario 1000 shown in Fig. 6.

[0063] Returning to the explanation of Fig. 5, the scenario / script transmission unit 252 transmits the test scenario received from the CI / CD tool 210, and the test environment creation script and test execution script created in the above step S103, to the simulation device 60 (step S104).

[0064] Furthermore, the script storage unit 253 stores (saves) the test environment creation script and the test execution script created in the above step S103 in the repository 120 (step S105).

[0065] When the script / expected result storage unit 601 of the simulation device 60 receives the test scenario, the test environment creation script, and the test execution script, it stores the expected results included in the test scenario, the test environment creation script, and the test execution script in the script / expected result storage unit 606 (step S106). At this time, the script / expected result storage unit 601 stores the expected results corresponding to each test case in the script / expected result storage unit 606.

[0066] The environment construction unit 602 of the simulation device 60 executes the test environment construction script stored in the script / expected result storage unit 606 (step S107), which starts up a simulator on the simulation device 60 and constructs a test environment that simulates the desired measuring device.

[0067] Meanwhile, the program update instruction unit 254 transmits a program update instruction to the target embedded device 50 to download and install the executable file deployed to the deployment destination indicated by the deployment destination information received from the CI / CD tool 210 (step S108). As a result, the target embedded device 50 downloads the executable file from the distribution server 30 in accordance with the program update instruction and installs it in the target embedded device 50.

[0068] The test execution unit 603 of the simulation device 60 executes the test execution script stored in the script and expected result storage unit 606 (step S109). As a result, a communication test is performed between the measuring device simulated by the simulator on the simulation device 60 and the target embedded device 50, and test result information is stored in the test result storage unit 607. Details of the test execution process will be described later.

[0069] The test result saving unit 605 of the simulation device 60 saves (registers) the test result information saved in the test result storage unit 607 in the repository 120 (step S110). This registration triggers the version control program 110 to send an email to the developer terminal 40 notifying that the test result information has been registered.

[0070] <Test Execution Process According to the First Embodiment> <Configuration of target embedded device 50> Fig. 9 shows an example of the configuration of the target embedded device 50 after the executable file has been installed in step S108 of Fig. 5. The example shown in Fig. 9 shows a case where the executable file is configured with a communication module 501, a control module 502, an alarm determination module 503, and an operation management module 504. Here, the communication module 501 is a module that executes communication processing with the simulation device 60, the control module 502 is a module that executes control processing of the target embedded device 50, the alarm determination module 503 is a module that executes processing to determine whether an alarm is necessary, and the operation management module 504 is a module that executes processing to collect and store execution results, which are the operation results of each module of the target embedded device 50. The target embedded device 50 also includes a shared memory 505, which is a memory area commonly used by each module, a shared library 506, which is a library commonly used by each module, and an execution result storage unit 507, in which the execution results of each module are stored.

[0071] 9 may have different operation cycles (communication module 501, control module 502, alarm determination module 503, and operation management module 504). In the example shown in Fig. 9, the operation cycle of the communication module 501 is 20 seconds, the operation cycle of the control module 502 is 5 minutes, the operation cycle of the alarm determination module 503 is 1 minute, and the operation cycle of the operation management module 504 is 1 minute.

[0072] In this way, the executable file that is the subject of a communications test is generally made up of multiple modules, each of which may have a different operating cycle. Therefore, when judging the test results, it is necessary to take these operating cycles into consideration.

[0073] <Test execution process flow> The test execution process for executing a communication test with the target embedded device 50 by executing a test execution script and creating and saving test result information including the test results will be described below with reference to FIG.

[0074] The test execution unit 603 initializes a variable n representing the test case number to 1, and also initializes the number of waits to 0 (step S201). The number of waits is the number of times the wait process of step S206, which will be described later, has been executed.

[0075] The test execution unit 603 executes the test of test case n by setting and transmitting the transmission data of test case n included in the test execution script stored in the script / expected result storage unit 606 (step S202). That is, the test execution unit 603 executes the test of test case n by setting the transmission data of test case n included in the test execution script in the measuring device and transmitting the transmission data to the target embedded device 50. As a result, the target embedded device 50 executes control in accordance with the transmission data.

[0076] The test result generation unit 604 acquires the execution result from the execution result storage unit 507 of the target embedded device 50 (step S203).

[0077] The test result creation unit 604 determines whether there is an execution result obtained immediately before the execution result obtained in step S203 above, and whether there has been a change between the execution result obtained in step S203 above and the execution result obtained immediately before that (step S204).

[0078] If the execution result obtained one time previously in step S204 exists and it is determined that there is no change between the current execution result and the previous execution result, the test result creation unit 604 determines whether the number of waits has exceeded a predetermined number of times (step S205).

[0079] If it is determined in step S205 above that the number of waits has not exceeded the predetermined number of times, the test result creation unit 604 waits for a predetermined time (e.g., several seconds to several tens of seconds) and adds 1 to the number of waits (step S206), and then returns to step S203 above, whereby the processes from step S203 onwards are executed again.

[0080] On the other hand, if it is determined in step S204 that a previous execution result exists and that there has been a change between the current execution result and the previous execution result, or if it is determined in step S205 that the number of waits has exceeded a predetermined number, the test result generation unit 604 determines whether the execution result obtained in step S203 satisfies the expected result and generates a test result (step S207). That is, the test result generation unit 604 obtains the expected result corresponding to test case n from the script / expected result storage unit 606, determines whether the execution result satisfies the expected result, and generates the determination result as the test result. For example, if the execution result satisfies the expected result (i.e., if the execution result matches the expected result), "OK" indicating that the expected result is satisfied is generated as the test result. Otherwise, "NG" indicating that the expected result is not satisfied is generated as the test result.

[0081] The test result creation unit 604 creates test result information including the test result created in step S207 above, and saves this test result information in the test result storage unit 607 (step S208). The test result creation unit 604 may create test result information including, for example, the test case number n, the transmission data for this test case n, the expected result corresponding to this test case n, the execution result determined to be whether or not the expected result is met, and the test result.

[0082] The test result generation unit 604 determines whether or not the tests for all test cases have been completed (step S209).

[0083] If it is determined in step S209 above that the testing of all test cases has been completed, the simulation device 60 ends the test execution process.

[0084] On the other hand, if it is not determined in step S209 that testing of all test cases has been completed, the test execution unit 603 adds 1 to the variable n representing the test case number and resets the number of waits to 0 (step S210), and returns to step S202. As a result, the processing from step S202 onwards is executed for the next test case n.

[0085] <<Examples of test result judgments>> 10, it is determined whether the execution result has changed, and if the execution result has not changed, the process waits a predetermined number of times and then repeatedly obtains the execution result again. This is because the operation cycle of each module may differ, and it is necessary to wait until the execution results of all modules are obtained.

[0086] As an example, a case where the test result for the target embedded device 50 shown in Fig. 9 is determined will be described with reference to Fig. 11. Note that the example shown in Fig. 11 shows a case where the waiting time in step S206 in Fig. 10 is "20 seconds."

[0087] (1) First, when the transmission data of test case 1 is transmitted to the target embedded device 50, the execution result is acquired in step S203 of Fig. 10. Note that, since the previous execution result does not exist, step S206 of Fig. 10 is executed and no test result is generated.

[0088] (2) Next, in step S203 of FIG. 10, the execution result is obtained again, but since there is no change from the previous execution result, step S206 of FIG. 10 is executed and no test result is generated.

[0089] (3) Next, the execution result is obtained again in step S203 of FIG. 10, but similarly, since there is no change from the previous execution result, step S206 of FIG. 10 is executed and no test result is generated.

[0090] (4) Next, the execution result is obtained again in step S203 of Fig. 10, and since there is a change from the previous execution result, it is determined whether the execution result satisfies the expected result in step S207 of Fig. 10. In the example shown in Fig. 11, the execution result matches the expected result, so a test result of "OK" is generated.

[0091] (5) Next, when the transmission data of test case 2 is sent to the target embedded device 50, the execution result is obtained in step S203 of Figure 10, but since there is no change from the previous execution result, step S206 of Figure 10 is executed and no test result is created.

[0092] (6) Next, in step S203 of FIG. 10, the execution result is obtained again, but since there is no change from the previous execution result, step S206 of FIG. 10 is executed and no test result is generated.

[0093] (7) Next, the execution result is obtained again in step S203 of Fig. 10, and since there is a change from the previous execution result, it is determined whether the execution result satisfies the expected result in step S207 of Fig. 10. In the example shown in Fig. 11, the execution result matches the expected result, so a test result of "OK" is generated.

[0094] (8) Next, when the transmission data of test case 3 is sent to the target embedded device 50, the execution result is obtained in step S203 of Figure 10, but since there is no change from the previous execution result, step S206 of Figure 10 is executed and no test result is created.

[0095] (9) Next, in step S203 of FIG. 10, the execution result is obtained again, but since there is no change from the previous execution result, step S206 of FIG. 10 is executed and no test result is generated.

[0096] (10) Next, the execution result is obtained again in step S203 of Fig. 10, and since there is a change from the previous execution result, it is determined whether the execution result satisfies the expected result in step S207 of Fig. 10. In the example shown in Fig. 11, the execution result does not match the expected result, so a test result of "NG" is generated.

[0097] In this way, by determining whether the execution result satisfies the expected result only when the execution result has changed, it is possible to appropriately determine the test result even if the operating cycles of each module are different.

[0098] <Examples of test result information> As a result, in step S208 of Fig. 10, it is possible to create and save test result information such as that shown in Fig. 12. In Fig. 12, each row represents test result information for each test case, and each piece of test result information includes the test case number n, the transmitted data for this test case n, the expected result corresponding to this test case n, the execution result determined to have satisfied the expected result, and the test result. This allows, for example, a developer or the like to refer to this test result information and find out what execution result was obtained for which test case and what the test result was.

[0099] <Hardware configuration example> The version control system 10, CI / CD system 20, distribution server 30, developer terminal 40, and simulation device 60 included in the development support system 1 according to the first embodiment can be realized, for example, by the hardware configuration of a computer 900 shown in Fig. 13. The computer 900 shown in Fig. 13 includes an input device 901, a display device 902, an external I / F 903, a communication I / F 904, a RAM (Random Access Memory) 905, a ROM (Read Only Memory) 906, an auxiliary storage device 907, and a processor 908. Each of these pieces of hardware is connected to each other via a bus 909 so as to be able to communicate with each other.

[0100] The input device 901 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 902 is, for example, a display, a display panel, etc. Note that the computer 900 does not necessarily have to have at least one of the input device 901 and the display device 902, for example.

[0101] The external I / F 903 is an interface with an external device such as a recording medium 903a. Examples of the recording medium 903a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0102] The communication I / F 904 is an interface for connecting the computer 900 to a communication network. The RAM 905 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 906 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 907 is a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 908 is an arithmetic device such as a CPU.

[0103] 13 is an example, and is not limited to this hardware configuration of the computer 900. For example, the computer 900 may have multiple auxiliary storage devices 907 or multiple processors 908, may not have some of the hardware shown in the figure, or may have various types of hardware other than the hardware shown in the figure.

[0104] [Second embodiment] Next, a development support system 1 according to a first embodiment will be described. In the first embodiment, if an abnormality occurs in some of the modules installed in the target embedded device 50, it may become difficult to continue testing. In response to this, it is possible to repair the module causing the abnormality, but it is often difficult to identify the cause of the abnormality, and there is a problem that testing of other modules will be suspended until the repair is completed.

[0105] Therefore, in the second embodiment, a case will be described in which, when an abnormality occurs in a test, a certain module is replaced with a mock to ensure the continuity of the test.

[0106] In the second embodiment, differences from the first embodiment will be mainly described, and descriptions of components that are the same as those in the first embodiment will be omitted.

[0107] <Example of replacing with mock> For example, suppose an abnormality occurs in the communication module 501 of the target embedded device 50 during a communication test, as shown in the upper diagram of Fig. 14. In this case, it becomes difficult to continue the test, so the communication module 501 is replaced with its mock communication module 511, as shown in the lower diagram of Fig. 14. This makes it possible to continue testing other modules such as the control module 502, alarm determination module 503, and operation management module 504.

[0108] <Example of functional configuration of simulation device 60 according to second embodiment> 15, the simulation device 60 according to the second embodiment has, in addition to the units described in the first embodiment, an abnormality detection unit 608 and a replacement instruction unit 609. These units are realized, for example, by a processor such as a CPU executing one or more programs including a simulator installed in the simulation device 60.

[0109] The anomaly detection unit 608 detects that an anomaly has occurred in the target embedded device 50 during a communication test with the target embedded device 50. The anomaly detection unit 608 may detect an anomaly in the target embedded device 50 using any anomaly detection method. For example, the anomaly detection unit 608 may detect that an anomaly has occurred in the target embedded device 50 if, after transmitting transmission data to the target embedded device 50, it does not receive response data in response to the transmission data within a predetermined time.

[0110] When the anomaly detection unit 608 detects an anomaly, the replacement instruction unit 609 transmits a replacement instruction to the CI / CD tool 210 to instruct the CI / CD tool 210 to replace at least some of the modules constituting the executable file installed in the target embedded device 50 with mocks. As a result, the CI / CD tool 210 replaces the relevant modules with mocks, and then executes build, test, and deployment to recreate the executable file. Note that this recreated executable file is downloaded and installed in the target embedded device 50, and the communication test continues.

[0111] <Test execution process according to the second embodiment> <Test execution process flow> The test execution process, in which if an abnormality occurs, the relevant module is replaced with a mock and the communication test is then continued, will be described below with reference to Fig. 16. Note that steps S301 to S302 in Fig. 16 are similar to steps S201 to S202 in Fig. 10, respectively, and steps S305 to S312 in Fig. 16 are similar to steps S203 to S210 in Fig. 10, respectively, and therefore descriptions thereof will be omitted.

[0112] The abnormality detection unit 608 determines whether or not an abnormality has occurred in the target embedded device 50 (step S303).

[0113] If it is not determined in step S303 above that an abnormality has been detected, the simulation device 60 executes the process of step S305. On the other hand, if it is determined in step S303 above that an abnormality has been detected, the replacement instruction unit 609 transmits a replacement instruction to the CI / CD tool 210 (step S304). At this time, the replacement instruction unit 609 transmits to the CI / CD tool 210 a replacement instruction including module identification information (e.g., module name, etc.) that identifies the module to be replaced with a mock. As a result, step S102 in FIG. 5 is executed, and after the module is replaced with a mock, build, test, and deployment are executed to recreate the executable file. Furthermore, thereafter, step S108 in FIG. 5 is executed, and the recreated executable file is downloaded and installed in the target embedded device 50.

[0114] <Priority of replacement with mock> If the module identification information of the module in which the abnormality occurred can be identified in step S303 of Fig. 16, the module identification information can be included in the replacement instruction in step S304 of Fig. 16. On the other hand, there are cases where the module in which the abnormality occurred cannot be identified and the module identification information cannot be identified. For this reason, the replacement instruction unit 609 can determine the module identification information of the module to be replaced with a mock in accordance with a predetermined priority order.

[0115] Examples of priorities include the priorities shown in FIG. 17. In the priorities shown in FIG. 17, priority "1" is assigned to the communication module 501, priority "2" is assigned to the control module 502, and priority "3" is assigned to the alarm determination module 503, and the operation management module 504 cannot be replaced with a mock. This indicates that, for example, if none of the modules have been replaced with a mock, the communication module 501 can be targeted for replacement with a mock. Also, for example, if the communication module 501 has already been replaced with a mock, the control module 502 can be targeted for replacement with a mock. Furthermore, if both the communication module 501 and the control module 502 have already been replaced with mocks, the alarm determination module 503 can be targeted for replacement with a mock.

[0116] <<Indication that it has been replaced with a mockup>> When a module is replaced with a mock and an executable file is recreated, it is preferable to clearly indicate that the executable file has a module replaced with a mock. For example, it is preferable to include the name of the module to be replaced in the version information of the executable file. This allows, for example, developers to know which module has been replaced with a mock.

[0117] [summary] As described above, the development support system 1 according to the first embodiment can automatically create a test environment and perform a communication test by registering source code to be subjected to a communication test in the repository 120, and can automatically obtain and notify the test results. This makes it possible to significantly reduce the costs required for creating a communication test environment, performing the test, obtaining the test results, etc. It also makes it possible to prevent mistakes that occur when performing tests manually.

[0118] Furthermore, the development support system 1 according to the second embodiment can ensure the continuity of communication tests by automatically replacing a module in which an abnormality has occurred with a mock. This makes it possible to prevent situations such as an abnormality occurring in a module making it impossible to test other modules, thereby contributing to a reduction in the number of test man-hours.

[0119] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims. [Explanation of symbols]

[0120] 1. Development Support System 10 Version Control Systems 20 CI / CD Systems 30 Distribution Server 40 Developer Terminals 50 Target embedded devices 60 Simulation Device 110 Version Control Programs 120 repositories 210 CI / CD Tools 211 Build Instructions 212 Test Instructions 213 Deploy Instructions 214 Pipeline Configuration File 220 Build Tools 230 Test Tools 240 Deployment Tools 250 Test Support Program 251 Script Creation Department 252 Scenario / Script Transmission Department 253 Script Storage 254 Program update instruction section 260 Simulator Information DB 601 Script and expected result storage section 602 Environment Construction Department 603 Test Execution Department 604 Test Results Creation Department 605 Test Results Storage Unit 606 Script and expected result memory section 607 Test result storage unit 608 Anomaly detection unit 609 Replacement instruction section 900 Computers 901 Input Device 902 Display device 903 External I / F 903a Recording media 904 Communication I / F 905 RAM 906 ROM 907 Auxiliary storage 908 processor 909 Bus

Claims

1. A test support system including a first device, a simulation device equipped with a simulator that simulates a second device that communicates with the first device, and a test support device that supports a communication test between the first device and the second device, The test support device an executable file creation unit that, when a source code to be a target of the communication test and a test scenario representing a scenario of the communication test are registered in a repository of a version control system, creates an executable file of the source code in response to the registration; a first script creation unit that creates a test environment creation script including an instruction for creating a test environment for the communication test by the simulator based on the test scenario and simulator information indicating information related to a simulator that simulates the second device; a second script creation unit that creates a test execution script including an instruction for the second device to transmit transmission data to the first device and perform the communication test based on the test scenario; and a first transmission unit that transmits the test environment construction script and the test execution script to the second device; a second transmitting unit that transmits an installation instruction for the executable file to the first device; Test support system.

2. The simulation device includes: a construction unit that constructs the test environment by executing the test environment construction script; a test execution unit that executes the test execution script to transmit transmission data from the second device to the first device and executes the communication test; an acquisition unit that acquires an execution result of the first device when the transmission data is received; a test result creation unit that creates test results of the communication test from the execution results; a registration unit that registers information including the test results in the repository, The test support system according to claim 1 .

3. The test result creation unit Waiting until there is a change in the execution result; creating the test result based on the execution result when the change occurs and a correct execution result expected for the execution result; The test support system according to claim 2 .

4. The test result creation unit When the number of times of waiting exceeds a predetermined number of times, the test result is created based on the execution result when the number of times of waiting exceeds the predetermined number of times and a correct execution result expected for the execution result. The test support system according to claim 3 .

5. The simulation device includes: an abnormality detection unit that detects the occurrence of an abnormality in the first device during the communication test; a third transmission unit that, when the occurrence of the abnormality is detected, transmits to the test support device a replacement instruction for replacing some of the modules constituting the executable file with mocks; The test support system according to claim 1 .

6. The third transmission unit transmitting the replacement instruction, which includes identification information of the module in which the abnormality has occurred or identification information of a module determined in accordance with a predetermined priority, to the test support device; The test support system according to claim 5 .

7. 1. A test support method used in a test support system including a first device, a simulation device equipped with a simulator that simulates a second device that communicates with the first device, and a test support device that supports a communication test between the first device and the second device, comprising: The test support device an executable file creation procedure for creating an executable file of the source code when the source code to be the target of the communication test and a test scenario representing a scenario of the communication test are registered in a repository of a version control system, triggered by the registration; a first script creation procedure for creating a test environment creation script including an instruction for creating a test environment for the communication test using the simulator, based on the test scenario and simulator information indicating information related to a simulator that simulates the second device; a second script creation step of creating a test execution script based on the test scenario, the test execution script including an instruction for the second device to transmit transmission data to the first device and perform the communication test; a first transmission step of transmitting the test environment construction script and the test execution script to the second device; a second transmitting step of transmitting an installation instruction for the executable file to the first device; Testing support methods.

8. A program used in a test support system including a first device, a simulation device equipped with a simulator that simulates a second device that communicates with the first device, and a test support device that supports a communication test between the first device and the second device, The test support device includes: an executable file creation procedure for creating an executable file of the source code when the source code to be the target of the communication test and a test scenario representing a scenario of the communication test are registered in a repository of a version control system, triggered by the registration; a first script creation procedure for creating a test environment creation script including an instruction for creating a test environment for the communication test using the simulator, based on the test scenario and simulator information indicating information related to a simulator that simulates the second device; a second script creation step of creating a test execution script based on the test scenario, the test execution script including an instruction for the second device to transmit transmission data to the first device and perform the communication test; a first transmission step of transmitting the test environment construction script and the test execution script to the second device; a second transmission step of transmitting an installation instruction for the executable file to the first device; program.

Images