Information processing device and test case extraction method

Abstract

A process for extracting a test case for performing a test for performance verification based on a requested specification for a system to be developed performs sentence analysis of a request sentence describing the requested specification, and identifies the semantic role of a target phrase included in the request sentence. The process acquires a definition corresponding to the role of the target phrase from a dictionary database, and identifies a system performance to be verified and the target range of a measurement condition on the basis of the definition. Further, the process acquires history data of the test from a test asset database in which the system performance and the measurement condition of an already-executed test are associated with each other. The present processing collates the completed measurement condition included in the history data and the target range of the identified measurement condition, and extracts a measurement condition to be included in the test case. Thus, a test for performance verification based on the requested specification can be efficiently carried out even when there is ambiguity in the request sentence describing the requested specification for the system.

Description

Information processing device and test case extraction method

[0001] The present invention relates to an information processing apparatus and a test case extraction method for extracting test cases for carrying out tests for performance verification based on required specifications for a system.

[0002] In the development process of various systems that process software programs, testing is performed to verify whether the system satisfies the required specifications in order to ensure the quality of the system. As an example of a technology to support such testing work, a technology has been proposed that uses a keyword library that accumulates keywords for test scenarios and automatically generates test cases in data format from the keywords to improve the efficiency of testing.

[0003] Japanese Patent Application Laid-Open No. 2022-126094

[0004] Here, in the development process of such a system, particularly in the upstream process, the requirements text describing the required specifications can contain ambiguity. In such cases, even if the same object is referred to, the expressions used in the requirements text and the keywords managed in the system may differ, resulting in no direct connection. A system test can be expressed as a combination of the system performance of the system to be verified and the measurement conditions under which that performance is to be confirmed. Currently, when automatically selecting a system test that conforms to the requirements text from existing tests, it is difficult to identify the system performance and measurement conditions from the requirements text due to issues such as the use of relevant keywords.

[0005] Therefore, one aspect of the present invention aims to enable efficient implementation of tests for performance verification based on required specifications, even when the required specifications of a system are described with ambiguity.

[0006] In one aspect of the present invention, a test case extraction process is performed to perform a test for performance verification based on a system requirement specification. The process includes reading input data including a requirement sentence describing the requirement specification, analyzing the requirement sentence, and identifying the semantic roles of target phrases included in the requirement sentence. The process also includes retrieving definitions corresponding to the identified roles for each target phrase from a dictionary database in which definitions of system-related phrases are associated with each role. The process then includes identifying, based on the definitions, the system performance to be verified by the test and the target range of measurement conditions for implementing the test. The process also includes retrieving historical data of tests performed on system performance from a test asset database in which system performance of previously performed tests and already-implemented measurement conditions for the tests are stored in association with each other. The process also includes extracting measurement conditions to be included in test cases by comparing the already-implemented measurement conditions included in the historical data with the target range of measurement conditions identified based on the definitions of the target phrases included in the requirement sentence.

[0007] According to one aspect of the present invention, even if the requirement statement describing the system requirement specifications contains ambiguity, it is possible to efficiently implement tests for performance verification based on the requirement specifications.

[0008] 1 is a diagram showing an example of a performance verification device. FIG. 2 is a diagram showing an example of requirement sentence data. FIG. 3 is a diagram showing an example of a dictionary database. FIG. 4 is a diagram showing an example of requirement condition data. FIG. 5 is a diagram showing an example of a test case table of a test asset database. FIG. 6 is a diagram showing an example of a test history table of a test asset database. FIG. 7 is a diagram showing an example of test condition data. FIG. 8 is a diagram showing an example of test result data. A flowchart showing an example of performance verification processing. An explanatory diagram showing an example of test measurement conditions. An explanatory diagram showing a specific data example of request identification processing. An explanatory diagram showing the target range of test measurement conditions and specific data example of history data in map format. FIG. 9 is a diagram showing the hardware configuration of an information processing device constituting the performance verification device.

[0009] Hereinafter, with reference to the accompanying drawings, an embodiment for carrying out the present invention will be described in detail. [Technical Area to which the Present Embodiment is Applied] First, to facilitate understanding of the present embodiment, an example of a technical area (background) to which the present embodiment is applied will be described. The present embodiment relates to development support for various systems in which programs are executed on information processing devices (including microcomputers). As an example, in the development of a vehicle system, development is carried out based on a development process that applies a model known as a V-shaped development model, which is based on a process model such as A-SPICE (Automotive Software Process Improvement and Capability dEtermination). In the V-shaped development model, development is carried out in the following order from upstream to downstream: (1) requirement definition for the system to be developed, (2) system architecture design, (3) software requirements analysis, (4) software architecture design, and (5) software detailed design. Then, for each of the processes (1) to (5), verification work is carried out in the order from downstream to upstream: (6) software unit verification, (7) software integration testing, (8) software qualification testing, (9) system integration testing, and (10) system qualification testing.

[0010] In the development process described above, for example, in the upstream process (1), a requirements specification document describing the system's requirements (requirements) is analyzed, and the system requirements that the system must meet are defined. The requirements documents describing the requirements contained in such requirements specifications are often more abstract and vague than detailed design documents, such as those created in the software detailed design process (5). Therefore, requirements analysis in process (1) relies heavily on the skills of analysts. Furthermore, when verifying the system's suitability in process (10), which verifies whether process (1) was performed correctly, the correspondence between test cases and requirements is not directly linked because it is mediated by the requirements definition process. Therefore, the more ambiguous the requirements wording, the less clear the correspondence with tests. When the requirements wording itself is added or changed, it is difficult to determine what system performance should be verified and under what conditions. Normally, by going through each step of the V-shape, system requirements are defined, software is developed and verified in accordance with those requirements, and then the system behavior is observed, so this ambiguity does not make verification impossible. However, when it is necessary to evaluate the performance of the current system when requirements change before implementing system modifications, problems arise in which it is not possible to identify which past tests should be performed.

[0011] Furthermore, when a system is dynamic and operates in a variety of scenarios, it is important to repeatedly test and verify the software running on that system by changing the system's usage scenario settings. For example, in the case of a system installed in a vehicle, vehicles are used in a wide variety of scenarios. Therefore, it is necessary to test and evaluate whether each software control operates normally and achieves the required performance under the measurement conditions of the onboard equipment corresponding to the various usage scenarios. For example, in the performance verification of an electric vehicle system powered by a motor, examples of measurement conditions include parameters related to the system state, such as motor rotation speed x load conditions, battery voltage conditions, motor temperature conditions, motor control mode (rotation speed-based, torque-based), inverter control mode (modulation method, carrier frequency), and diagnostic status (normal or functional limitations due to detected faults). When combined with special phenomena such as road slippage due to rain or ice, or sudden changes in road load on uneven roads, the number of measurement conditions based on usage scenarios is countless.

[0012] In addition, even if we take the motor rotation speed x load condition as one of these conditions, the question is how densely measurement points should be taken within the closed space defined by the maximum and minimum rotation speeds and maximum and minimum loads in the plane space spanned by these two elements as vectors. Since it is not possible to take measurement points infinitely finely, measurements are taken at grid points divided into certain granularities, such as in units of 100 rpm or 10 N-meters. The usual method for system verification is to vary the way these grid points are taken depending on the purpose. In other words, a high density is required for verification that has a significant impact on system performance, such as torque accuracy. Therefore, in system verification, a wide range of measurement conditions are measured by varying the grid point density depending on the system performance being verified, resulting in a complex data structure.

[0013] Consider a scenario in which the above system experiences a malfunction that only occurs under a specific usage scenario (operating condition). System verification involves attempting a malfunction reproduction test, but before doing so, it is important to immediately determine whether previous tests using the usage scenario in which the malfunction occurred have been conducted. Because malfunctions are typically reported in natural language, analyzing which performance and under which measurement conditions malfunctions requires analysis by an engineer who understands the system architecture. When a malfunction occurs, a system is expected to automatically determine whether past measurement data is available and, if necessary, automatically execute the data. However, no such system has been developed to date. Similarly, when a request is made to improve the system's performance under specific conditions, the system may need to check for past test data related to the desired performance under the desired conditions. However, there is no system that automatically determines whether relevant past test data exists at the time of the request written in natural language, and automatically executes the data if it has not already been executed.

[0014] The present embodiment, described below, solves the problems in this technical background. First, the ambiguity of the requirements text is reduced by analyzing the requirements text created in the upstream process of the system development process. Then, in this embodiment, the system performance and measurement conditions that should be verified by testing to satisfy the required specifications can be clearly identified from the requirements text. Furthermore, in this embodiment, the measurement conditions that need to be tested, i.e., the measurement conditions that should be included in the test cases, can be automatically extracted within the target range of the identified system performance and measurement conditions.

[0015] 1 is a diagram showing an example of a performance verification device 10 that performs performance verification of a vehicle system in this embodiment. The performance verification device 10 is an information processing device that includes a processor that executes a program and a storage device that stores data used for executing the program. The performance verification device 10 includes a requirement identification unit 11, a condition extraction unit 12, and a test execution unit 13, whose functions are realized when the processor loads and executes the program from memory.

[0016] The requirement identification unit 11 analyzes requirement sentences and identifies the range of system performance and measurement conditions to be verified by testing in order to satisfy the requirement specifications indicated by the requirement sentences. The requirement identification unit 11 includes a sentence analysis unit 11A and a definition acquisition unit 11B. The sentence analysis unit 11A reads requirement sentence data 21 as input data and identifies the semantic roles of phrases (which may include a single word or a phrase combining multiple words) contained in the requirement sentences in the requirement specifications described in the requirement sentences. The phrases whose roles are to be identified are, for example, nouns or noun phrases, but are not limited thereto. In this embodiment, such roles are classified into three roles: "condition," "observation target," and "expected behavior." In the following description, the phrases whose roles are to be identified in the requirement sentences are referred to as target phrases.

[0017] The definition acquisition unit 11B accesses the dictionary database 22 stored in the nonvolatile memory of the storage device of the performance verification device 10 and acquires dictionary information corresponding to each target phrase included in the request sentence. The performance verification device 10 can be implemented on a physical server or on a cloud server via a network; either method is acceptable. The definition acquisition unit 11B then acquires definitions corresponding to the role of each target phrase from the acquired dictionary information. Based on the definitions, the definition acquisition unit 11B identifies the target ranges of system performance and measurement conditions to be verified by testing in order to satisfy the required specifications indicated by the request sentence. The definition acquisition unit 11B then generates requirement data 23 including the target ranges of these system performance and measurement conditions. The requirement data 23 is stored in the nonvolatile memory of the storage device of the performance verification device 10 or in a temporarily used volatile memory.

[0018] The condition extraction unit 12 extracts measurement conditions for testing to verify that the required specifications are satisfied, based on the system performance and the target range of measurement conditions indicated by the requirement data 23 generated by the requirement identification unit 11. In other words, the measurement conditions are measurement conditions to be included in test cases for verifying that the required specifications are satisfied. The condition extraction unit 12 includes a history search unit 12A and a condition matching unit 12B. The history search unit 12A accesses a test case table 24A in a test asset database 24 stored in the nonvolatile memory of the storage device of the performance verification device 10, and searches for and acquires data (records) that match the system performance indicated by the requirement data 23. Then, the history search unit 12A accesses a test history table 24B associated with the data, and searches for and acquires history data whose measurement conditions fall within the range of the measurement conditions indicated by the requirement data 23.

[0019] The condition matching unit 12B extracts measurement conditions for the test case based on the search results for the history data in the test history table 24B. Specifically, the condition matching unit 12B matches the measurement conditions included in the history data already stored in the test history table 24B (i.e., the measurement conditions already implemented in tests that have already been implemented) with the target range of measurement conditions indicated by the requirement condition data 23. Then, the condition matching unit 12B extracts, as measurement conditions to be included in the test case, measurement conditions that may be included in the target range of measurement conditions indicated by the requirement condition data 23, excluding the measurement conditions of the history data already stored in the test history table 24B. These measurement conditions are measurement conditions for which tests have not yet been implemented. The condition matching unit 12B outputs test condition data 25 that includes the system performance indicated by the requirement condition data 23 and the measurement conditions that have not yet been implemented.

[0020] The test execution unit 13 obtains data corresponding to the test cases of the system performance indicated by the test condition data 25 from the test case table 24A. Then, the test execution unit 30 executes the test under the measurement conditions indicated by the test condition data 25 based on the test scenario (automated test script) associated with the data. The test execution unit 30 then generates a test report including test result data 26 containing the execution results of the test. The execution results of the test indicated by the test result data 26 are reflected in the test history table 24B of the test asset database 24.

[0021] Next, the data used by the requirement identification unit 11, condition extraction unit 12, and test execution unit 13 will be described. The requirement statement data 21 is data in which requirement statements included in requirement specifications for system development are stored in a format that can be read by an information processing device. Figure 2 shows an example of the requirement statement data 21. Note that, when the requirement statement data 21 includes multiple requirement statements, it may be in a format such as CSV (Comma Separated Values) and may include a requirement ID for identifying the requirement statement, the requirement statement, the chapter to which the requirement statement belongs in the requirement specification, and the like.

[0022] The dictionary database 22 is a database in which definitions (specific semantic content and quantitative information related to the term) are associated with terms related to the system being developed. FIG. 3 shows an example of the data structure of the dictionary database 22. As shown in FIG. 3, the dictionary database 22 associates a definition with each term as dictionary information for each semantic role in the requirements specification. In this embodiment, these roles are classified into three categories: "condition," "observation target," and "expected behavior." Note that the role classification is not limited to this example and can be arbitrarily set based on the content and characteristics of the system requirements. Furthermore, it is not necessary for all terms to be associated with definitions of all three roles; some terms may be associated with only one of the three definitions. The dictionary information in the dictionary database 22 can be registered arbitrarily throughout the system development process. In particular, registering dictionary information in the dictionary database 22 in downstream processes of the system development process allows for more detailed definitions of terms. Such a dictionary database 22 can be created in a format such as JSON (JavaScript Object Notation).

[0023] The requirement data 23 includes the range of system performance and measurement conditions to be verified by testing in order to satisfy the requirement specifications indicated by the requirement statement. Fig. 4 shows an example of the requirement data 23. In detail, the system performance may include target performance indicating the performance itself to be verified by testing, an observation target in the test, and verification criteria that serve as the basis for determining whether the test execution results satisfy the requirement specifications.

[0024] The test asset database 24 is a relational database and includes a test case table 24A and a test history table 24B. The test case table 24A is a table that stores data including various information related to test cases. As illustrated in FIG. 5 , the test case table 24A has test history table items that record the test case identifier, the system performance to be tested (target performance, observation target, verification criteria), the automated test script, and the measurement conditions under which tests have already been performed. The data in the test history table items is associated with the test history table 24B, which will be described later. Note that, at least for test cases that have already been tested, the definitions in the dictionary database 22 and the system performance in the test case table 24A of the test asset database 24 are synchronized and have a corresponding relationship.

[0025] The test history table 24B is a table that stores historical data on the measurement conditions under which tests were performed in the test cases indicated by the data in the test case table 24A. As shown in FIG. 6, the test history table 24B has fields for the identifiers of the measurement points in the measurement conditions, the measurement parameters, the output values ​​resulting from the test, and the verification results indicating whether the output values ​​meet the verification criteria for the test. In the specific example of the test history table 24B shown in FIG. 6, there are two types of measurement parameters: torque (motor load) and motor rotation speed. The number of measurement parameter fields can vary depending on the content of the test case.

[0026] The test condition data 25 is data obtained by extracting measurement conditions that have not yet been implemented from the target range of measurement conditions indicated in the required condition data 23. For example, when the system performance is electricity consumption calculation accuracy, the measurement conditions in the test condition data 25 include the system performance (target performance, observation target, and verification standard) and a combination of measurement parameter values ​​of torque and motor rotation speed as measurement conditions to be included in the test case, as illustrated in FIG. 7 . The test result data 26 is data indicating the results of a test conducted under the measurement conditions included in the test condition data 25. For example, when a test is conducted based on the test condition data 25 illustrated in FIG. 7 , the test result data 26 may include the following content: That is, as illustrated in FIG. 8 , the test result data 26 includes an electricity consumption accuracy output value (%) and a verification result (Passed or Failed) based on the output value, which are associated with the measurement conditions in the test result data 26 and are the result of conducting a test for a combination of measurement parameter values ​​of torque and motor rotation speed.

[0027] 9 is a flowchart showing the performance verification process executed by the performance verification device 10 in this embodiment. In step 1001 (denoted as S1001 in the figure, the same applies below), the sentence analysis unit 11A of the request identification unit 11 reads the request sentence data 21 from the storage device. Then, the sentence analysis unit 11A performs sentence analysis processing on the request sentence included in the request sentence data 21.

[0028] Here, the text analysis process includes, for example, the following three processes: (1) Morphological Analysis The text analysis unit 11A divides the request text into morphemes, which are the smallest units that have meaning in a language, and performs morphological analysis to determine the part of speech and inflections of each morpheme. Existing technology can be used for the morphological analysis; for example, analysis tools such as MeCab and JUMAN are provided, and an existing corpus or the like can also be used as a dictionary to be used in the analysis process.

[0029] (2) Syntax Analysis Next, the sentence analysis unit 11A performs syntactic analysis (dependency analysis) to analyze the relationships between each word in the request sentence based on the results of the morphological analysis. Existing technology can be used for the syntactic analysis, and for example, a tool based on a machine learning algorithm can be used. For example, one such tool is SyntaxNet, which is a recurrent neural network (RNN) model built using TensorFlow and is specialized for natural language understanding (NLU).

[0030] (3) Phrase Role Analysis (Semantic Analysis) The sentence analysis unit 11A identifies the role of a target phrase included in a request sentence (whether it is a "condition," "observation target," or "expected action") in the request sentence based on the results of the morphological analysis and syntactic analysis. The role classification at this time is to be consistent with the role classification that can be set in the dictionary information of the dictionary database 22. Such role analysis processing can be realized, for example, by deep learning based on a machine learning algorithm. As a specific example of an existing technology that realizes such deep learning, a neural network framework written in Python or the like can be used.

[0031] As an example of deep learning processing, training data can be prepared in which the roles of words used in a requirement sentence are classified into "condition," "observation target," and "expected behavior." A learning process can be performed using the training data to generate a learning model. The input layer of deep learning can include, for example, the target phrase itself, the grammatical structure of the requirement sentence, the chapter to which the requirement sentence belongs in the requirements specification, and keywords contained in the requirement sentence. Information indicating the grammatical structure of the requirement sentence is, for example, information indicating the types of words before and after the target phrase. As a specific example, this information can be used for analysis, such as determining that if a phrase such as "in" or "when" is present immediately after the target phrase, the role is likely to be "condition." The output layer can be information indicating the role of the aforementioned phrase, i.e., "condition," "observation target," or "expected behavior." Although not shown in FIG. 1 , the training data and learning model data used in the deep learning are also stored in the storage device of the performance verification device 10.

[0032] In step 1002, the definition acquisition unit 11B accesses the dictionary database 22 and searches for and acquires dictionary information corresponding to each of the target phrases included in the request sentence. The dictionary information extracted from the dictionary database 22 includes definitions corresponding to all or part of the roles of the target phrases in the request sentence, i.e., "condition," "observation target," and "expected action," as shown in FIG. 3 . If dictionary information corresponding to the target phrase does not exist in the dictionary database 22, the dictionary database 22 may be searched using synonyms of the target phrase. Then, for each of the target phrases included in the request sentence, for which dictionary information has been extracted from the dictionary database 22, the definition acquisition unit 11B acquires, from the dictionary information, a definition corresponding to the role of the target phrase identified by the sentence analysis unit 11A.

[0033] In step 1003, the definition acquisition unit 11B identifies definitions corresponding to the target range of the test system performance (target performance, observation target, verification criteria) and measurement conditions from the definitions of the target words and phrases included in the request sentence, and generates the requirement condition data 23. For example, in the definitions corresponding to the roles of "condition," "observation target," and "expected behavior," the "observation target" may be the target performance or observation target of the system performance to be tested, the "expected behavior" may be the verification criteria, and the "condition" may be the target range of the measurement conditions. In step 1004, the history search unit 12A of the condition extraction unit 12 accesses the test case table 24A, searches for data corresponding to the system performance in the requirement condition data 23, and acquires the data. The data includes information indicating a correspondence with the test history table 24B, which records the measurement conditions under which the test corresponding to the system performance in the requirement condition data 23 was performed.

[0034] In step 1005, the history search unit 12A searches the test history table 24B associated with the acquired data and obtains information on measurement conditions that have already been tested and are included in the target range of the measurement conditions in the requirement condition data 23. In step 1006, the condition comparison unit 12B determines whether a test for a measurement condition included in the target range of the measurement conditions in the requirement condition data 23 has not yet been performed. Specifically, the condition comparison unit 12B compares the measurement conditions included in the history data already stored in the test history table 24B with the target range of the measurement conditions indicated by the requirement condition data 23. If there is a measurement condition that may be included in the target range of the measurement conditions indicated by the requirement condition data 23 but whose history data is not stored in the test history table 24B, the condition comparison unit 12B determines that the test has not yet been performed. If the test has not yet been performed, the process proceeds to step 1006 (Yes); if not, the process ends.

[0035] In step 1007, the condition matching unit 12B extracts measurement conditions to be currently executed within the target range of the measurement conditions in the requirement condition data 23 for a test corresponding to the system performance of the requirement condition data 23. For example, when executing a test by varying the value of a measurement parameter by a predetermined number, the condition matching unit 12B performs the following process. That is, the condition matching unit 12B extracts measurement parameters that differ by the predetermined number within the target range of the measurement conditions in the requirement condition data 23 and that do not exist in the measurement conditions of the history data in the test history table 24B. In other words, the condition matching unit 12B extracts, as measurement conditions to be included in a test case, measurement parameters that differ by the predetermined number within the target range of the measurement conditions in the requirement condition data 23, other than the measurement conditions already stored as history data in the test history table 24B. The condition matching unit 12B then generates test condition data 25 that includes the measurement conditions to be currently executed. Note that in this embodiment, the processing of steps 1001 to 1007 of the performance verification process up to this point is referred to as the test case extraction process.

[0036] In step 1008, the test execution unit 13 obtains data corresponding to the system performance included in the test condition data 25 from the test case table 24A. This data includes information indicating an automatic test script. In step 1009, the test execution unit 13 generates a test scenario by modifying the automatic test script so as to execute tests for measurement conditions included in the test condition data 25 that have not yet been executed, and executes the test. The test execution unit 13 then compares the output values ​​obtained as a result of the test execution with the verification criteria, and verifies whether the execution results satisfy the verification criteria. Furthermore, the test execution unit 13 generates test result data 26 that includes the measurement conditions under which the test was executed, the output values ​​of the test execution results under the measurement conditions, and the verification results.

[0037] In step 1010, the test execution unit 13 updates the test history table 24B with the measurement conditions under which the test was performed, the output values ​​of the execution results of the test under those measurement conditions, and the verification results, which are included in the test result data 26. Specifically, the test execution unit 13 registers data including the measurement conditions, the output values ​​of the execution results of the test, and the verification results in the test history table 24B.

[0038] [Specific Example of Performance Verification Processing] Next, the above-mentioned performance verification processing will be explained using specific data examples. First, let us assume that the system requirements are to improve the electric fuel economy of a hybrid vehicle with an inverter-driven permanent magnet synchronous motor (PMSM), specifically, to verify the accuracy of the calculation of electric fuel economy when the vehicle is traveling uphill and at high speeds and high loads. The background to these requirements is that when the hybrid vehicle is traveling on a highway with many uphill lanes, the motor's energy efficiency is high, making it desirable to actively utilize the motor. However, this is also a region where the gradient of change in motor efficiency is high, requiring high estimation accuracy. Furthermore, motor efficiency is inextricably linked to the rate of dissipation into thermal energy. Therefore, electric fuel economy accuracy is important for the system to determine how long the motor can be driven within that range from its thermal limit. Figure 10 illustrates the "high-speed driving x hill-climbing condition" for the requirements. Under these driving conditions, it is desirable to accurately determine whether torque derating is necessary to address motor overheating issues and to appropriately determine the optimal balance between motor drive and engine drive. For this reason, during the vehicle development process, it is necessary to verify performance under various measurement conditions that meet these driving conditions.

[0039] FIG. 11 illustrates a specific example of processing by the sentence analysis unit 11A and the definition acquisition unit 11B when the request sentence data 21 is a sentence such as, "I would like to confirm whether the accuracy of the calculation of electric fuel consumption when traveling uphill and at high RPM and high load is sufficient." As a result of the sentence analysis process performed by the sentence analysis unit 11A on the request sentence, the request sentence can be divided into, for example, "I would like to confirm whether the accuracy of the calculation of electric fuel consumption when traveling uphill and at high RPM and high load is sufficient." The part "traveling uphill and at high RPM and high load" can be analyzed as a modifier (M) due to the inclusion of the word "when." The part "electric fuel consumption calculation accuracy" can be analyzed as a subject (S) based on word order, part of speech, etc. The part "sufficient" can be analyzed as a complement (C) based on word order, part of speech, etc.

[0040] Then, based on the analysis results, the role of each phrase is further identified. For example, "driving uphill and at high RPM and high load" can be identified as having the role of "condition" in the request sentence because it is included in the modifier and because it uses the word "when." "Electricity consumption calculation accuracy" can be identified as having the role of "object of observation" in the request sentence because it is included in the subject. "Sufficient" can be identified as having the role of "expected behavior" in the request sentence because it is a complement and is included in the predicate.

[0041] Next, for each of the terms, the definition acquisition unit 11B searches the dictionary database 22 using each term as a key to extract dictionary information. At this time, the definition acquisition unit 11B may refer to a database (not shown) storing synonymous terms, convert the term included in the request sentence into another term with a similar meaning, and then search the dictionary database 22. In this specific example, dictionary information is extracted from the dictionary database 22 shown in FIG. 3 for each of the terms "climbing conditions," "high rotation and high load," and "electricity consumption calculation accuracy," which are converted from "driving on an uphill road." Each dictionary information includes the roles of the terms "condition," "observation target," and "expected behavior," as well as definitions associated with each of these roles. Specifically, for the term "climbing conditions," the definition of the role "condition"—"torque: 300 to 400 Nm, rotation speed: 2000 to 8000 rpm"—is extracted. For the phrase "high rotation, high load," the definition of the role "condition" is extracted: "torque: 200 to 400 Nm, rotation speed: 6000 to 10000 rpm." For the phrase "electricity consumption calculation accuracy," first, "Signal A," the definition of the role "observation target," is extracted. Here, the phrase extracted as the role "expected operation" is "sufficient," which means sufficiency of performance. In this case, the definition acquisition unit 11B acquires the definition of the role "expected operation" set in the dictionary database 22 for the phrase "electricity consumption calculation accuracy" whose role is "observation target." That is, the definition acquisition unit 11B extracts the definition of the role "expected operation" of the phrase "electricity consumption calculation accuracy," which is "when torque is 0 to 100 Nm: ±2%, when torque is 100 to 400 Nm: ±3%."

[0042] The definition acquisition unit 11B then generates requirement data 23 including definitions extracted from the dictionary database 22 for each of these terms. FIG. 4 shows an example of requirement data 23 generated based on the definitions acquired from the dictionary database 22 shown in FIG. 3. The definition acquisition unit 11B sets the target performance, among the system performances in the requirement data 23, to the term "electricity consumption calculation accuracy" itself, identified as having the role "observation target." The definition acquisition unit 11B also sets the observation target to "Signal A," which is the definition of the role "observation target" for the term "electricity consumption calculation accuracy." Furthermore, the definition acquisition unit 11B sets the verification criteria to "when torque is 0 to 100 Nm: ±2%, when torque is 100 to 400 Nm: ±3%," which is the definition of the role "expected operation" for the term "electricity consumption calculation accuracy." Furthermore, the definition acquisition unit 11B sets the target range of measurement conditions in the requirement data 23 as follows: That is, the definition acquisition unit 11B identifies a condition (AND condition) that satisfies both the definition of the phrase "hill climbing condition" whose role is identified as "condition" and the definition of the phrase "high rotation, high load." In this example, the definition acquisition unit 11B sets "torque: 300 to 400 Nm, rotation speed: 6000 to 8000 rpm" as the target range of the measurement conditions in the required condition data 23.

[0043] The history search unit 12A of the condition extraction unit 12 then searches the test case table 24A of the test asset database 24 based on the requirement condition data 23. If data corresponding to the system performance (target performance, observation target, verification criteria) of the requirement condition data 23 exists in the test case table 24A, the history search unit 12A acquires information from the test history table 24B associated with that data. The history search unit 12A then acquires from the test history table 24B measurement conditions that have already been tested, within the target range of measurement conditions indicated by the requirement condition data 23.

[0044] FIG. 12 shows the data in the test history table 24B, which records the test history for verifying the system performance of the example requirement data 23 described above, in a map format with two measurement conditions, torque and engine speed, as axes. In this map, the vertical axis represents engine speed and the horizontal axis represents torque. The engine speed on the vertical axis assumes a test conducted by varying the engine speed from 0 to 2000 rpm, and then in 2000 rpm increments thereafter, with rows generated for every 2000 rpm. Meanwhile, the torque on the horizontal axis assumes a test conducted by varying the torque from 0 to 100 Nm, and then in 100 Nm increments thereafter, with columns generated for every 100 Nm. The verification results based on the test output values ​​are displayed at the grid points where the engine speed row and torque column intersect. As described above, the target ranges of measurement conditions identified from the requirement text in the requirement text data 21 are as follows: That is, the target range of the measurement conditions for "hill climbing conditions" is "torque: 300-400 Nm, rotation speed: 2000-8000 rpm," and the target range of the measurement conditions for "high rotation, high load" is "torque: 200-400 Nm, rotation speed: 6000-10000 rpm." Furthermore, the target range of the measurement conditions that meets both of these is "torque: 300-400 Nm, rotation speed: 6000-8000 rpm," which is indicated in the required condition data 23.

[0045] Here, the target range of the measurement condition "torque: 300-400 Nm, rotation speed: 6000-8000 rpm" includes the following measurement conditions when the engine rotation speed is changed in increments of 2000 rpm and the torque in increments of 100 Nm, as shown in the shaded area in FIG. 12 . That is, the target range of the measurement conditions includes four measurement conditions: "torque: 300 Nm, rotation speed: 6000 rpm," "torque: 300 Nm, rotation speed: 8000 rpm," "torque: 400 Nm, rotation speed: 6000 rpm," and "torque: 400 Nm, rotation speed: 8000 rpm." Of these measurement conditions, data for "torque: 400 Nm, rotation speed: 8000 rpm" has already been registered in the test history table 24B. Therefore, the condition collation unit 12B extracts the other three measurement conditions as unexecuted measurement conditions, i.e., as measurement conditions to be included in the test case, as shown in FIG. 7 . Then, test condition data 25 including these measurement conditions is generated.

[0046] The test execution unit 13 then searches the test case table 24A to obtain data corresponding to the system performance of the test condition data 25, and identifies the automatic test script indicated by that data. The test execution unit 13 then generates a test scenario by modifying the automatic test script so as to execute tests for measurement conditions included in the test condition data 25 that have not yet been implemented, and executes the test. The test execution unit 13 then generates test result data 26 as shown in FIG. 8. The measurement conditions and test results included in the test result data 26 are reflected in the test history table 24B.

[0047] Effects and Modifications of the Present Embodiment According to the present embodiment, the following process is executed when a test for performance verification based on the requirements specifications of a system under development is performed. First, morphological analysis, syntactic analysis, and semantic analysis are performed on the requirements text describing the requirements specifications. This reduces the ambiguity of the requirements text. Next, the system performance to be verified by the test and the target range of measurement conditions for the test are identified from the requirements text. The identified system performance and target range of measurement conditions are then compared with the measurement conditions of previously performed tests stored in the test history table 24B of the test asset database 24, and measurement conditions to be included in the test case are extracted. This makes it possible to automatically generate test cases from ambiguous requirements text to verify performance that meets the requirements specifications. Furthermore, not only is such test case generation automatic, but test cases that take into account measurement conditions that have already been verified at the current stage can also be generated. This significantly reduces the workload associated with generating test cases for performance verification based on the requirements specifications of a system, thereby improving the efficiency of system development.

[0048] Furthermore, according to this embodiment, the semantic roles of target phrases included in a request sentence are identified, and definitions corresponding to the identified roles for each target phrase are obtained from the dictionary database 22, in which definitions of system-related phrases are associated with each role. Being able to refer to definitions corresponding to the roles of each phrase in this manner provides the following advantages. That is, even for the same phrase, the meaning (target) indicated by the phrase may differ depending on, for example, the context of each request sentence. Therefore, simply referencing a unique definition for a phrase does not necessarily mean that the definition is appropriate for each request sentence. In response to this, in this embodiment, definitions corresponding to roles are associated with phrases in the dictionary database 22. Then, for target phrases included in a request sentence, definitions corresponding to the identified roles for the target phrases are obtained. This makes it possible to generate accurate test cases corresponding to the content of the requirement specifications. Furthermore, in this embodiment, the roles are classified into "conditions," "observation targets," and "expected behaviors," and definitions corresponding to these roles are extracted. This makes it possible to clarify the information necessary for extracting test cases, i.e., what conditions the requirements specifications described in the requirements document assume, what the object of observation is, and how that object of observation should behave.

[0049] Furthermore, according to this embodiment, the measurement conditions included in the target range of the measurement conditions identified based on the requirement statement, excluding the measurement conditions included in the history data stored in the test history table 24B, are extracted as the measurement conditions to be tested. This enables automatic generation of test cases that target only the measurement conditions that have not yet been tested, significantly reducing the effort required to identify such measurement conditions. Furthermore, according to this embodiment, when the measurement conditions include multiple parameters, such as the torque and engine speed mentioned above, the measurement conditions to be tested are extracted from the measurement conditions that include all of the multiple parameters in their target range. In this way, even if the pattern of measurement conditions to be tested is complex, it is possible to accurately extract the measurement conditions necessary for performance verification of the requirement specifications.

[0050] In requirement sentences relating to system performance, such as the example sentence shown above, target phrases with the role of "expected behavior" often have a meaning indicating the sufficiency of performance. In such cases, in this embodiment, for target phrases with the role of "observation target," a definition set in the dictionary database 22 for the target phrase with the role of "expected behavior" is obtained. Then, this definition is identified as the verification criterion for system performance. This makes it possible to generate test cases that more closely match the intended meaning of the sentence.

[0051] As a modification of this embodiment, if the verification result based on the test output value included in the history data of the test history table 24B does not satisfy the verification criteria, the condition extraction unit 12 may further extract the measurement conditions of the history data as measurement conditions to be included in the test case. This makes it possible to retest not only the measurement conditions that have not been tested but also the measurement conditions that did not satisfy the verification criteria in the previous tests, thereby enabling further performance verification.

[0052] Furthermore, when verifying system performance, depending on the required specifications, it may not be necessary to run tests for all measurement conditions included in the target range of the measurement conditions, and running a test for just one measurement condition may be sufficient for verification. Therefore, as a further modification of this embodiment, if there is one or more pieces of history data in the test history table 24B in which the measurement condition is included in the target range of the measurement conditions in the required condition data 23, the condition extraction unit 12 may process the test as if there are no measurement conditions to be tested.

[0053] In this embodiment, the performance verification device 10 includes all of the request identification unit 11, the condition extraction unit 12, and the test execution unit 13. However, this configuration is merely an example, and these components may be implemented in different information processing devices. Furthermore, for example, the request identification unit 11 and the condition extraction unit 12 may be implemented in one information processing device, and the test execution unit 13 may be implemented in another information processing device. Similarly, in this embodiment, the dictionary database 22 and the test asset database 24 are provided, but these do not necessarily have to be included in the same information processing device. For example, these pieces of data may be stored in a storage device provided in a server on the cloud. The same applies to other data.

[0054] In this embodiment, the test asset database 24 stores the test case table 24A and the test history table 24B as a relational database, but data can also be stored in other formats. For example, these data can be stored as a two-dimensional array, a JSON file, or the like. In addition, although a vehicle system is described as an example in this embodiment, the present invention can also be applied to performance verification of various systems other than vehicles.

[0055] [Hardware Configuration] Figure 12 shows an example of the hardware configuration of an information processing device constituting the performance verification device 10 in the above embodiment. This information processing device includes a processor 910, a memory 920, a storage 930, a portable storage medium drive 940, an input / output device 950, and a communication interface 960. The processor 910 includes a control unit, an arithmetic unit, an instruction decoder, etc., and the execution unit executes arithmetic and logic operations using the arithmetic unit in accordance with program instructions decoded by the instruction decoder and in response to control signals output from the control unit. The processor 910 also includes a control register that stores various information used for control, a cache that can temporarily store contents of previously accessed memory 920, etc. Note that the processor 910 may be configured with multiple CPU (Central Processing Unit) cores.

[0056] The memory 920 is a storage device such as a random access memory (RAM), and serves as a main memory into which programs executed by the processor 910 are loaded and which stores data used in the processor 910's processing. The storage 930 is a storage device such as a hard disk drive (HDD) or flash memory, and stores programs and various data. The portable storage medium drive 940 is a device that reads data and programs stored in a portable storage medium 970. The portable storage medium 970 is, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. The processor 910 executes programs stored in the storage 930 or the portable storage medium 970 in cooperation with the memory 920 and the storage 930. The programs executed by the processor 910 and the data to be accessed may be stored in another device capable of communicating with the information processing device. A program that causes the information processing device to execute the test case extraction process described above may also be stored in the portable storage medium 970.

[0057] The input / output device 950 is an input means and output means such as a touch panel, keyboard, display, etc., and accepts operation commands from a user, etc., while outputting the results of processing by the information processing device. The communication interface 960 enables data communication with the outside. The components of the information processing device described above are connected by a bus 980.

[0058] [Others] The above-described embodiments of the present invention are merely some of the possible implementations within the technical scope of the present invention and do not limit the technical scope of the present invention. Furthermore, the functional configurations and physical configurations in each embodiment are not limited to the above-described aspects. For example, functions and physical resources can be integrated and implemented, or conversely, they can be further distributed and implemented. Furthermore, it is also possible to add, delete, or replace part of the configuration with other configurations.

[0059] 10...Performance verification device, 11...Requirement identification unit, 11A...Text analysis unit, 11B...Definition acquisition unit, 12...Condition extraction unit, 12A...History search unit, 12B...Condition matching unit, 13...Test execution unit, 21...Requirement text data, 22...Dictionary database, 23...Requirement condition data, 24...Test asset database, 24A...Test case table, 24B...Test history table, 25...Test condition data, 26...Test result data

Claims

1. An information processing device that extracts test cases for testing performance verification based on requirements specifications for a system, comprising: a requirement identification unit that reads input data including a requirement sentence describing the requirements specifications, analyzes the requirement sentence, identifies the semantic roles of target words included in the requirement sentence, obtains definitions corresponding to the identified roles for each target word from a dictionary database in which definitions of words related to the system are associated with each role, and identifies the system performance to be verified by the test and the target range of measurement conditions for conducting the test based on the definitions; and a condition extraction unit that obtains history data of tests conducted on the system performance from a test asset database in which the system performance of tests that have already been conducted and the measurement conditions already implemented in the tests are stored in association with each other, and compares the implemented measurement conditions included in the history data with the target range of the measurement conditions identified based on the definitions of the target words included in the requirement sentence to extract measurement conditions to be included in the test cases.

2. The information processing device described in claim 1, wherein the condition extraction unit extracts measurement conditions included in the target range of the measurement conditions identified based on the definition of the target phrase included in the request sentence, excluding the measurement conditions that have already been implemented and included in the historical data, as measurement conditions to be included in the test case.

3. The information processing device according to claim 1, wherein, when the measurement conditions include multiple parameters, the condition extraction unit extracts measurement conditions to be included in the test case from measurement conditions whose target range includes all of the multiple parameters.

4. The information processing device of claim 1, wherein the role is one of the conditions, observation target, and expected behavior of the requirement specification indicated by the requirement sentence, and the requirement identification unit identifies, among the target phrases contained in the requirement sentence, the definitions of target phrases whose role is the observation target and the definitions of target phrases whose role is the expected behavior as the system performance, and the definitions of target phrases whose role is a condition as the target range of the measurement conditions.

5. An information processing device as described in claim 4, wherein, when a target phrase whose role is an expected action has a meaning indicating the adequacy of performance, the requirement identification unit obtains a definition set in the dictionary database for the target phrase whose role is the observation target when the role of the target phrase is an expected action, and identifies the definition as the system performance.

6. An information processing device as described in claim 2, wherein, if the execution results of the test included in the history data do not satisfy the verification criteria indicated by the system performance, the condition extraction unit further extracts the executed measurement conditions of the history data as measurement conditions to be included in the test case.

7. An information processing device as described in claim 1, wherein the condition extraction unit processes the test case as if there is no measurement condition to be included in the test case if there is one or more pieces of history data in which the implemented measurement condition is included in the target range of the measurement condition.

8. A test case extraction method for extracting test cases for testing performance verification based on requirements specifications for a system, the method comprising: reading input data including a requirement statement describing the requirements specifications; analyzing the requirement statement; identifying the semantic roles of target words included in the requirement statement; obtaining definitions corresponding to the identified roles for each target word from a dictionary database in which definitions of words related to the system are associated with each role; identifying the system performance to be verified by the test and the target range of measurement conditions for implementing the test based on the definitions; obtaining history data of tests implemented on the system performance from a test asset database in which the system performance of tests that have already been implemented and the measurement conditions already implemented in the tests are associated and stored; and comparing the implemented measurement conditions included in the history data with the target range of the measurement conditions identified based on the definitions of the target words included in the requirement statement to extract measurement conditions to be included in the test cases.

Images