Program development support system and program development method

Abstract

We provide a program development support system that can shorten testing time. [Solution] The test item list contains a list of test items and the number of times each test item failed to produce the expected result. The sequence control unit extracts the test items to be performed from the test item list and rearranges the extracted test items based on the number of failed attempts stored in the test item list. The test execution instruction unit sends a command to the test execution unit to actually perform the tests based on the order rearranged by the sequence control unit.

Description

Technical Field

[0001] The present invention relates to a program development support system and a program development method.

Background Art

[0002] In recent years, in the software development field, automation of tests using a continuous integration (CI) system has been progressing. Automation of tests using a CI system is a common technique in the development of web-based software, and is also gradually being introduced in the development of embedded software. A major difference between embedded software and web-based software is that in embedded software, it is necessary to download software to a target board and execute tests.

[0003] When software is developed by multiple people, a CI system is shared by multiple people. When multiple people commit (upload) source code to a software configuration management (SCM) server at the same time, usually, tests are executed in the order of commitment. In the development of embedded software, the target board itself is often developed in parallel. Therefore, it is difficult to increase the number of target boards available during development. If the exclusive time of the target board becomes long in tests related to one source code, the start of tests related to other source codes will be delayed.

[0004] A technique for shortening the test execution time by determining the execution order of test items based on the dependency relationship of nodes (functions) is known (Patent Document 1).

Prior Art Documents

Patent Documents

[0001] , , , , ,

[0002] , , ,

[0003] , , ,

[0004] , , , ,

Patent Document 1

Summary of the Invention

[0006] The test method described in Patent Document 1 is effective for comprehensive unit testing. However, in coupling tests with connected target substrates, comprehensive function-level testing is performed infrequently, and the test method described in Patent Document 1 is not very effective in reducing test time. In particular, in tests against functional specifications, it is often impossible to define dependencies. The object of the present invention is to provide a program development support system and a program development method that can reduce test time. [Means for solving the problem]

[0007] According to one aspect of the present invention, A list of test items, and a test item list that stores the number of times the expected result was not obtained for each test item, A sequence control unit extracts test items to be performed from the aforementioned list of test items, and sorts the extracted test items based on the number of failed attempts stored in the aforementioned list of test items. A test execution instruction unit sends a test execution command to a test execution unit that actually performs the tests, based on the order sorted by the sequence control unit. A program development support system equipped with these features is provided.

[0008] According to another aspect of the present invention, When testing source code, extract the test items to be performed from the test item list. The extracted test items are sorted based on the number of times a failure occurred when the test item was previously performed. A method for developing programs that perform tests based on the sorted order is provided. [Effects of the Invention]

[0009] By rearranging test items based on the number of failures (NGs) for each test item, the execution order of test items that are likely to fail can be increased. This makes it possible to shorten the overall test time. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram of the program development support system according to the first embodiment. [Figure 2] Figure 2 is a diagram showing an example of the contents of the association table 20. [Figure 3] Figure 3 is a diagram showing an example of the contents of the test item list table 21. [Figure 4] Figure 4 is a schematic diagram showing an example of multiple branches. [Figure 5] Figure 5 is a flowchart showing the procedure of processing performed by the sequence control unit 10. [Figure 6] Figure 6 is a schematic diagram showing an example of the order of test items to explain the function of the sequence control unit 10. [Figure 7] Figure 7 shows an example of an image displayed on the display screen 32. [Figure 8] Figure 8A is an example of a weighted table based on the number of failures, where the evaluation item is the number of failures. Figure 8B is an example of a correlation table based on the degree of correlation between the evaluation item and the changes made in the source code. Figure 8C is an example of a weighted table based on the average test duration, where the evaluation item is the average test duration. [Figure 9] Figure 9 is a diagram illustrating the procedure by which the sequence control unit 10 (Figure 1) of the program development support system according to the second embodiment determines the execution order of test items. [Figure 10] Figure 10 is a diagram showing an example of an image indicating the test progress displayed on the display screen 32 of terminal 31. [Figure 11] Figure 11 is a flowchart showing the procedure for the program development method according to the third embodiment. [Modes for carrying out the invention]

[0011] Referring to FIGS. 1 to 6, a program development support system and a program development method according to the first embodiment will be described.

[0012] FIG. 1 is a block diagram of a program development support system according to the first embodiment. A developer creates or modifies source code 35 using terminal 31 and commits it to SCM server 30. An execution file is downloaded to test execution unit 51 of target board 50, and a test is executed.

[0013] The program development support system includes an order control unit 10, a test execution instruction unit 11, an association setting unit 12, an input device 13, a progress notification unit 14, an association table 20, and a test item list 21. The association table 20 and the test item list 21 are realized by securing a storage area in the storage unit of the program development system. The functions of the order control unit 10, the test execution instruction unit 11, the association setting unit 12, and the progress notification unit 14 are realized by the computer executing a program stored in the storage unit.

[0014] First, the association table 20 will be described with reference to FIG. 2. FIG. 2 is a chart showing an example of the content of the association table 20. The association table 20 associates each of a plurality of source code specifying information for specifying source code with a plurality of test items to be performed to confirm the normality of the source code. The source code committed by the developer to the SCM server 30 is specified by the source code specifying information. The source code specifying information is, for example, the file name of the source code that realizes a specific function. When the source code that realizes a specific function is composed of a plurality of files, the folder name of the folder in which the plurality of files are stored is used as the source code specifying information.

[0015] Each row in FIG. 2 corresponds to source code identification information, and each column corresponds to a test item. The circled symbols shown in FIG. 2 mean that the test item in the corresponding column is associated with the source code identification information in the corresponding row. In the example shown in FIG. 2, for example, test items C and D are associated with the source code identification information c.

[0016] Next, the test item list 21 will be described with reference to FIG. 3. FIG. 3 is a chart showing an example of the content of the test item list 21. In the test item list 21, the number of NG times when the expected result was not obtained is stored for each of a plurality of test items. In the test item list 21, the cumulative number of NG times from the initial stage of development, the number of NG times that occurred after a certain period retroactively from the current time (for example, the number of NG times in the past one month), and the number of NG times for each branch are stored.

[0017] Next, the branch will be described with reference to FIG. 4. FIG. 4 is a schematic diagram showing an example of a plurality of branches that occur during development. A branch records the flow of the program change history. Since the branched branches are not affected by other branches, multiple changes can be made simultaneously during program development.

[0018] As shown in FIG. 4, for example, a development branch Br3 branches from a master branch Br1 with stable operation, and a plurality of function branches Br4 and Br5 branch from the development branch Br3 corresponding to a plurality of functions for which additional changes are to be made. Function X is implemented in the function branch Br4, and function Y is implemented in the function branch Br5. The tests are performed independently for each of the function branches Br4 and Br5.

[0019] Once testing of feature branch Br4 is complete, feature branch Br4 is merged into development branch Br3. After the merge, regression testing is performed on development branch Br3. Once testing of feature branch Br5 is complete, feature branch Br5 is merged into development branch Br3. When all feature branches are merged into development branch Br3, release branch Br2 branches off from development branch Br3. Regression testing is performed on both release branch Br2 and development branch Br3.

[0020] If the regression tests on development branch Br3 fail, a bug fix branch Br6 branches off from development branch Br3, and the bug is fixed in bug fix branch Br6. Once the bug fix is ​​complete, bug fix branch Br6 is merged into development branch Br3. After that, development branch Br3 is merged into release branch Br2. Once the regression tests on release branch Br2 are completed, release branch Br2 is merged into master branch Br1, and master branch Br1 is released.

[0021] In this way, program testing is performed in parallel for each branch. Test Item List 21 (Figure 3) stores the number of failures (NGs) for each test in the multiple branches generated during development. The cumulative number of NGs and the number of NGs for the past month in Test Item List 21 (Figure 3) represent the number of NGs from the tests performed for each test item across all branches.

[0022] Next, the functions of the sequence control unit 10 (Figure 1) will be explained with reference to Figures 5 and 6. Figure 5 is a flowchart showing the procedure of processing performed by the sequence control unit 10. Figure 6 is a schematic diagram showing an example of the order of test items to explain the functions of the sequence control unit 10.

[0023] When new source code is committed (uploaded as an update) to the SCM server 30 (Figure 1), the sequence control unit 10 retrieves the update from the SCM server 30 (step S1). After retrieving the update, it retrieves the test items that should be performed to verify the health of the update (step S2). The test items to be performed are pre-registered for each update. For example, when testing the update of source code identification information c (Figure 2), five test items, from test item A to test item E, are retrieved, as shown in Figure 6.

[0024] Next, the sequence control unit 10 retrieves test items associated with the source code identification information of the update program from the association table 20 (Figure 2) (step S3). For example, if the update program to be tested is identified by source code identification information c (Figure 2), test items C and D are retrieved as associated test items. In Figure 6, test items associated with the source code identification information of the update program are shown with relatively thick solid lines.

[0025] The sequence control unit 10 refers to the test item list 21 (Figure 3) and sorts the test items to be performed, obtained in step S2, based on the number of failed attempts (NG) (step S4). For example, the cumulative number of failed attempts can be used as the criterion for sorting. Alternatively, the number of failed attempts over the past month may be used, or the number of failed attempts of the branch currently performing the test may be used. In Figure 6, the numbers in parentheses next to the test items represent the cumulative number of failed attempts (Figure 3). In this example, sorting by the number of failed attempts results in the order of test items E, C, A, B, and D.

[0026] Next, the sequence control unit 10 shifts the order of the associated test items obtained in step S3 relative to the unassociated test items (step S5). For example, since test items C and D are associated with source code identification information c, the order of test items C and D is shifted relative to test items E, A, and B, as shown in Figure 6. As a result, the order after rearrangement is test items C, D, E, A, and B.

[0027] Once the order of the test items has been rearranged, the sequence control unit 10 notifies the test execution instruction unit 11 of the order in which the test items should be executed (step S6).

[0028] Next, the function of the test execution instruction unit 11 (Figure 1) will be explained. Once the source code identification information specifying the source code 35 to be tested is determined, the test execution instruction unit 11 creates an executable file from the entire source code, including the source code identified by the source code identification information, and downloads it to the test execution unit 51 of the target board 50.

[0029] The test execution instruction unit 11 obtains a test case for each test item. A test case is information that combines the execution conditions, input data, expected output or results, etc., that are prepared when testing the source code, and one or more test cases are prepared for each test item. The contents of the test cases are stored in advance in the test item list table 21.

[0030] The test execution instruction unit 11 sends a test execution command to the test execution unit 51 of the target board 50 based on the order sorted by the sequence control unit 10. The test execution unit 51 executes multiple test items in the commanded order. When executing one test item, it executes all test cases associated with that test item.

[0031] The test execution unit 51 executes the test items instructed by the test execution instruction unit 11 and returns the execution results to the test execution instruction unit 11. This return includes the execution results for each test case.

[0032] The test execution instruction unit 11 receives the test results from the test execution unit 51 and determines the test results. If the execution results of all test cases meet the expected results, the test result for that test item is determined to be "OK". If even one test case does not meet the expected results, the test result for that test item is determined to be "NG". If the test result does not meet the expected results (is NG), the number of NGs for the corresponding test item in the test item list table 21 is updated. Specifically, 1 is added to the current number of NGs for the test item whose test result is NG. Furthermore, each time the test execution instruction unit 11 receives a reply from the test execution unit 51, it notifies the progress notification unit 14 of the test results.

[0033] Next, the function of the progress notification unit 14 (Figure 1) will be explained with reference to Figure 7. The progress notification unit 14 displays information indicating the source code identification information and the progress of the tests currently being performed on the display screen 32 of the terminal 31. Figure 7 is a diagram showing an example of an image displayed on the display screen 32.

[0034] Rectangular shapes representing the test items to be performed are displayed from left to right in the order the tests will be performed. Each test item is associated with information indicating whether or not it is related to the source code being tested, and the number of failures. The current progress of the test is displayed as a progress bar. As the test progresses, the progress bar extends.

[0035] In addition to displaying information indicating the progress of the test on the display screen 32 of the terminal 31, the progress notification unit 14 may also have a function to notify the developer of the progress of the test via email, chat, etc. For example, it may notify the developer of messages such as "Test items related to the added / modified source code have completed successfully," or "The test failed."

[0036] Next, the function of the association setting unit 12 (Figure 1) will be explained. The software administrator inputs information to associate source code identification information with test items by operating the input device 13. Based on the input information, the association setting unit 12 adds a new row to the association table 20 (Figure 2) corresponding to the input source code identification information and associates the test items related to the new source code identification information. If the input source code identification information is already included in the association table 20, the association table 20 is updated based on the input content.

[0037] Next, we will describe the excellent effects of the first embodiment. In the first embodiment, in step S4 (Figure 5), the multiple test items to be performed are rearranged in order of the number of failures. Furthermore, in step S5 (Figure 4), the order of test items related to the source code to be tested is moved up. Test items that have had a high number of failures in past tests, or test items related to modified source code or newly added source code, are likely to result in failures during testing.

[0038] If a test item fails, testing of the target source code is interrupted, and testing of the next source code begins. By prioritizing the test items that are most likely to fail among multiple test items, the time until testing is interrupted due to a failure in a single source code is likely to be shortened. As a result, the time that the target board 50 is exclusively used for testing a specific source code is reduced. Overall, this results in the excellent effect of increased utilization efficiency of the target board 50.

[0039] Furthermore, in the first embodiment, the progress of the tests is displayed on the display screen 32 (Figure 7) of the developer's terminal 31. Therefore, if the tests of several test items are completed successfully, the developer can know that some test items that are prone to failing have been completed successfully.

[0040] Next, a modified example of the first embodiment will be described. In the first embodiment, step S5 (Figure 5) advances the order of test items related to the source code identification information to be tested, but this process may be omitted. If the process in step S5 is omitted, as shown in the second row of Figure 6, the test items will be tested in order from those with the most failures. Even in this case, test items that have a relatively high probability of failing will be tested preferentially, resulting in a reduction in test time.

[0041] Next, with reference to Figures 8A to 10, a program development support system according to the second embodiment will be described. In the first embodiment, as shown in Figure 6, test items are sorted by the number of failures, and then the order of test items associated with the source code to be tested is advanced. In contrast, in the second embodiment, evaluation values ​​are obtained for each test item using various evaluation criteria, and the test items are sorted based on these evaluation values.

[0042] Figure 8A is a diagram showing an example of a weighted table based on the number of failures (NGs) for each evaluation item. Test items with a higher number of NGs are assigned higher evaluation values. For example, test items with fewer than 10 NGs, test items with 10 to less than 100 NGs, and test items with 100 or more NGs are assigned evaluation values ​​of 1, 2, and 3, respectively.

[0043] Figure 8B is an example of a correlation table where evaluation items are assigned based on their relevance to changes in the source code. Test items with a higher correlation to the changes are assigned higher evaluation values. For example, test items that do not use the changes in the source code, test items that use the changes, and test items for the functionality implemented by the changes in the source code are assigned evaluation values ​​of 1, 2, and 3, respectively.

[0044] Figure 8C is a diagram showing an example of a test time weighting table where the evaluation item is the average test duration. The test time weighting table stores evaluation values ​​for each of the multiple test items based on the length of the average time required to execute the test. Test items with shorter average execution times are assigned higher evaluation values. For example, test items with an average execution time of 1 hour or more, test items with an average execution time of 10 minutes or more but less than 1 hour, and test items with an average execution time of less than 10 minutes are assigned evaluation values ​​of 1, 2, and 3, respectively.

[0045] Figure 9 is a diagram illustrating the procedure by which the sequence control unit 10 (Figure 1) of the program development support system according to the second embodiment determines the execution order of test items. The sequence control unit 10 obtains evaluation values ​​for each test item and evaluation item for the source code identification information of the target of test. Then, for each test item, it calculates a test execution order evaluation value by multiplying multiple evaluation values. The test execution order is assigned in order from the test items with the largest test execution order evaluation values. In the example shown in Figure 9, the test execution order evaluation values ​​for test items A, B, C, D, and E are 3, 2, 18, 2, and 12, respectively. In this case, the test execution order of test item C, which has the largest test execution order evaluation value, is set to 1. Furthermore, the test execution orders of test items E, A, B, and D are set to 2, 3, 4, and 5, respectively, in descending order of test execution order evaluation values.

[0046] In other words, the sequence control unit 10 has the function of rearranging multiple test items based on the degree of relevance of the source code identification information to each of the test items shown in Figure 8B. Furthermore, the sequence control unit 10 has the function of rearranging multiple test items based on the evaluation values ​​stored in the test time weighting table (Figure 8C).

[0047] Figure 10 shows an example of an image showing the test progress displayed on the display screen 32 of terminal 31. Similar to the first embodiment (Figure 7), rectangular shapes representing the test items to be performed are displayed in order from left to right in the order the tests are executed. In the second embodiment, each test item is associated with a test execution order evaluation value. The current test progress is displayed as a progress bar. As the test progresses, the progress bar extends.

[0048] Next, we will describe the excellent effects of the second embodiment. In the second embodiment, the evaluation value of test items is determined by referring to the NG count weighting table (Figure 8A), so test items with a high number of NGs are executed with priority over test items with a low number of NGs. Furthermore, the evaluation value of test items is determined by referring to the relevance table (Figure 8B), so test items that are strongly related to the modified source code are executed with priority over test items that are weakly related. As a result, similar to the first embodiment, the execution order of test items that are likely to result in NGs is increased, and the overall test time is shortened.

[0049] Furthermore, in the second embodiment, since the evaluation value of test items is determined by referring to the test time weighting table (Figure 8C), test items with shorter average test durations are executed with priority over test items with longer average test durations. While test items with longer average test durations are being executed, the execution of other test items will be delayed. By prioritizing the execution of test items with shorter average test durations, if a defect is inherent in the source code under test, it is possible to find out early that the test result will be NG. As a result, the test time is shortened.

[0050] Next, a program development method according to the third embodiment will be described with reference to Figure 11. The program development method according to the third embodiment uses the program development support system according to the first or second embodiment.

[0051] Figure 11 is a flowchart showing the procedure for the program development method according to the third embodiment. When testing the source code, test items to be performed are extracted from the test item list 21 (Figure 3) (step SA1). This extraction is performed, for example, by the sequence control unit 10 (Figure 1).

[0052] Next, the extracted test items are sorted based on the number of times a test item failed in the past (Step SA2). This sorting is performed by the sequence control unit 10 (Figure 1) by referring to the test item list 21 (Figure 3).

[0053] Next, the tests are executed based on the order sorted in step SA2 (step SA3). The tests are executed by the test execution unit 51 (Figure 1) in response to commands from the test execution instruction unit 11.

[0054] Next, a modified example of the third embodiment will be described. In step SA2, when sorting multiple test items, sorting may be performed not only based on the number of failures but also on information regarding whether or not the source code to be tested is used. This information can be obtained, for example, by referring to association table 20 (Figure 2). Alternatively, when sorting the extracted test items, sorting may be performed not only based on the number of failures but also on information regarding the average time required to perform each test item.

[0055] Next, we will describe the excellent effects of the third embodiment. In the third embodiment, as in the first and second embodiments, the excellent effect of reducing test time is obtained.

[0056] The embodiments described above are illustrative, and it goes without saying that partial substitution or combination of the configurations shown in different embodiments is possible. Similar effects and benefits from similar configurations in multiple embodiments will not be mentioned sequentially for each embodiment. Furthermore, the present invention is not limited to the embodiments described above. For example, it will be obvious to those skilled in the art that various modifications, improvements, and combinations are possible. [Explanation of Symbols]

[0057] 10 Sequence Control Unit 11 Test Execution Instruction Unit 12 Association Setting Section 13 Input device 14. Interim progress notification section 20 Association Table 21. List of Test Items 30. Software Configuration Management (SCM) Server 31 devices 32. Device display screen 35 Source Code 50 Target substrates 51 Test Execution Unit

Claims

1. A test item list table that stores a list of test items and the number of times each test item failed to produce the expected result. A sequence control unit extracts test items to be performed from the aforementioned list of test items, and sorts the extracted test items based on the number of failures stored in the aforementioned list of test items. A test execution instruction unit sends a test execution command to a test execution unit that actually performs the tests, based on the order sorted by the sequence control unit. A program development support system equipped with the following features.

2. The program development support system according to claim 1, wherein the test execution instruction unit receives test results from the test execution unit, determines the test results, and updates the number of NGs for the corresponding test item in the test item list if the test results do not meet the expected results.

3. The system further includes an association table that associates each of the multiple source code identification pieces with test items that should be performed to confirm the integrity of the source code identified by the source code identification piece. The program development support system according to claim 1 or 2, wherein the sequence control unit has the function of rearranging the extracted multiple test items based on the number of NGs, and then referring to the association table to advance the order of the test items associated with the source code to be tested.

4. The program development support system according to claim 1 or 2, wherein the sequence control unit further has a function to rearrange a plurality of test items based on the degree of association between each of the test items and a plurality of source code identification pieces of information that identify the source code.

5. The system further includes a test time weighting table that stores evaluation values ​​for each of the aforementioned multiple test items based on the average time required to perform the test. The program development support system according to claim 4, further comprising the sequence control unit having a function to rearrange a plurality of test items based on evaluation values ​​stored in the test time weighting table.

6. The program development support system according to claim 1 or 2, wherein the test item list stores the cumulative number of NGs from the early stages of development, and the number of NGs that have occurred since a certain period prior to the present.

7. The program development support system according to claim 1 or 2, wherein the test item list further stores the number of NGs that occurred for each of the multiple branches generated during development.

8. When testing source code, extract the test items to be performed from the test item list. The extracted test items are sorted based on the number of times the test item failed in the past. A program development method that performs tests based on the sorted order.

9. The program development method according to claim 8, wherein when sorting multiple extracted test items, the sorting is performed based on information such as whether or not the source code to be tested is used, in addition to the number of times the test item failed in the past.

10. A program development method according to claim 8 or 9, wherein when sorting a plurality of extracted test items, the sorting is performed based on information regarding the average time required to perform the test items, in addition to the number of failures that occurred when the test items were performed in the past.

Images