Information processing device, test case selection method, and program

Abstract

In the present invention, a feature amount extraction means 81 extracts, from a first test data set which is a set of test data items corresponding to a test case to be executed by a first model, a feature amount of each of the test data items. A clustering means 82 generates, by performing clustering on the respective extracted feature amounts, clusters of the corresponding test data items. A data extraction means 83 generates, on the basis of the distribution of the test data items included in the respective generated clusters, a second test data set obtained by extracting a part of the test data items from the first test data set.

Description

Information processing device, test case selection method and program

[0001] The present disclosure relates to an information processing device, a test case selection method, and a test case selection program that select test cases to be applied to an updated model.

[0002] In order to reduce the man-hours required for testing an evaluation target, a method for reducing the number of test cases has been proposed so as to ensure the quality of the evaluation target. For example, Patent Document 1 describes an apparatus that can reduce the number of test cases selected as requiring re-execution during regression testing. The apparatus described in Patent Document 1 reduces the number of test cases by not selecting test cases that cannot discover defects caused by program modifications as test cases requiring re-execution.

[0003] Japanese Patent Application Laid-Open No. 2016-164727

[0004] On the other hand, when evaluating a model updated by machine learning, it is preferable to be able to reduce the number of test cases while maintaining the comprehensiveness of expected test cases. The device described in Patent Literature 1 does not uniformly select test cases that cannot discover defects, so it cannot be said that the comprehensiveness of input data can be guaranteed, and it is difficult to say that an updated model can be tested in a way that guarantees the same quality as before the update.

[0005] Therefore, an object of the present disclosure is to provide an information processing device, a test case selection method, and a test case selection program that can comprehensively select test cases that can guarantee the same quality for an updated model as before the update.

[0006] The information processing device according to the present disclosure is characterized by comprising: a feature extraction means for extracting features of each piece of test data from a first test data set, which is a collection of test data corresponding to test cases to be executed by a first model; a clustering means for generating corresponding clusters of test data by clustering each of the extracted features; and a data extraction means for generating a second test data set by extracting some test data from the first test data set based on the distribution of the test data included in each of the generated clusters.

[0007] The test case selection method according to the present disclosure is characterized in that it extracts features for each piece of test data from a first test data set, which is a collection of test data corresponding to the test cases to be executed by a first model, performs clustering on each of the extracted features to generate corresponding clusters of test data, and generates a second test data set by extracting some test data from the first test data set based on the distribution of the test data contained in each of the generated clusters.

[0008] The test case selection program according to the present disclosure is characterized in that it causes a computer to execute a feature extraction process that extracts features of each piece of test data from a first test data set, which is a collection of test data corresponding to the test cases to be executed by a first model; a clustering process that generates corresponding clusters of test data by clustering each of the extracted features; and a data extraction process that generates a second test data set by extracting some test data from the first test data set based on the distribution of the test data contained in each of the generated clusters.

[0009] According to the present disclosure, it is possible to comprehensively select test cases that can guarantee the same quality for an updated model as before the update.

[0010] Fig. 1 is a block diagram showing an example of the configuration of an embodiment of a test case selection device; Fig. 2 is an explanatory diagram showing an example of a process for comparing characteristic differences between models; Fig. 3 is a flowchart showing an example of the operation of a test case selection device; Fig. 4 is a block diagram showing an overview of an information processing device according to the present disclosure; Fig. 5 is a schematic block diagram showing the configuration of a computer according to at least one embodiment;

[0011] Embodiments of the present disclosure will be described below with reference to the drawings. The test case selection device of the present disclosure is a device that selects test cases to be executed on an AI (Artificial Intelligence) model (hereinafter simply referred to as a model). More specifically, the present disclosure narrows down and selects test cases to be executed on a retrained model (hereinafter also referred to as an updated model) from all possible test cases. This aims to select test cases that can ensure the same quality for the updated model as before the update, while reducing the amount of work required to test the retrained model.

[0012] In the following description, data used according to a test case will be referred to as test data. In the present disclosure, a model applied to an in-vehicle AI will be exemplified as a specific example. An example of the processing performed by a model applied to an in-vehicle AI is image recognition processing. In this case, image data captured by an in-vehicle camera or the like can be used as test data. Note that information indicating the situation in which the image was captured (e.g., daytime, nighttime, busy, etc.) may be added to this image data.

[0013] However, the application of the test case selection device of the present disclosure is not limited to the case of learning a model to be applied to an in-vehicle AI, and the test data is not limited to image data. For example, the test data may be text data including multiple attributes.

[0014] In addition, the model assumed in this embodiment may be of any form, and the method for learning the model may also be any. For example, the assumed model may be a model for detecting an object in an image.

[0015] 1 is a block diagram showing an example configuration of an embodiment of a test case selection device according to the present disclosure. The test case selection device 100 according to this embodiment includes a storage unit 10, a feature extraction unit 20, a clustering unit 30, a data extraction unit 40, and an index calculation unit 50. Note that the test case selection device 100 according to this embodiment is a device that performs various types of information processing, and therefore can be called an information processing device.

[0016] The storage unit 10 stores data required when the test case selection device 100 of this embodiment performs various processes. Specifically, the storage unit 10 stores test data corresponding to test cases to be executed by the AI ​​model (model). Note that the contents of the test cases and the test data are arbitrary. In this embodiment, it is assumed that the storage unit 10 stores test data corresponding to test cases assumed in model testing.

[0017] In the following description, the pre-update model may be referred to as a first model, and a set of test data corresponding to the test cases to be executed by the first model may be referred to as a first test data set. Generally, test data corresponding to the test cases to be covered are used for testing pre-shipment models. Therefore, the set of test data corresponding to the test cases to be executed by an initial, unupdated model (i.e., the first test data set) can be said to be a set including all test data corresponding to the expected test cases.

[0018] The updated model is a model obtained by updating the first model, and may be referred to as the second model. A set of test data corresponding to the test cases to be executed by the second model is referred to as the second test data set. In this embodiment, since the purpose is to reduce the number of test cases to be executed on the updated model (second model), the number of test data included in the second test data set is less than the total number of test data.

[0019] The storage unit 10 may also store a model to be tested (more specifically, a pre-update model and a post-update model). The test case selection device 100 may also include a learning unit (not shown) that learns and updates the model. In this case, the storage unit 10 may store the model learned by the learning unit.

[0020] The feature extraction unit 20 extracts features from each piece of test data. Specifically, the feature extraction unit 20 extracts features from each piece of test data included in the first test data set. The feature extraction unit 20 may extract features from the test data using any method. For example, if the test data is image data, the feature extraction unit 20 may extract features using a trained AI model used in general image classification. Alternatively, the feature extraction unit 20 may extract features using an AI model trained specifically for the target domain, or may extract features using a part of the AI ​​model to be evaluated as a feature extractor.

[0021] The clustering unit 30 performs clustering on each of the extracted features to generate clusters of the corresponding test data. The clustering unit 30 may use any method for clustering, and may use any known clustering method, such as k-means or G-means.

[0022] The data extraction unit 40 extracts a portion of test data from the clustered test data according to the content of the test to be performed based on the distribution of test data included in each generated cluster. That is, the data extraction unit 40 generates a second test data set from the first test data set. In this case, the data extraction unit 40 extracts a portion of test data from each cluster to generate the second test data set. The data extraction unit 40 may generate the second test data set based on, for example, the distance from the cluster center determined based on the distribution of test data included in each cluster.

[0023] There are two main methods that the data extraction unit 40 can use to extract test data, depending on the content of the test to be performed, i.e., the quality that is to be guaranteed by the test. Note that the test data extraction methods described below are merely examples, and the extraction method is not limited to these two methods.

[0024] The first method is to extract test data so as to increase the coverage of typical test cases. In this case, the data extraction unit 40 extracts test data that are close to the cluster center. The data extraction unit 40 may extract test data whose distance from the cluster center is smaller than a predetermined threshold, or may extract a predetermined number of test data items closest to the cluster center.

[0025] Furthermore, the data extraction unit 40 may extract test data within a cluster so that the closer the data is to the cluster center, the more test data is extracted, and the farther the data is from the cluster center, the less test data is extracted. By extracting such data, it becomes possible to test on a group of similar test data.

[0026] The second method is to extract test data that increases the coverage of atypical test cases (in other words, makes it difficult to distinguish between similar test data). This can also be said to be a method of extracting test data that is difficult for the user to judge. In this case, the data extraction unit 40 extracts test data that is far from the cluster center. The data extraction unit 40 may extract test data whose distance from the cluster center is greater than a predetermined threshold, or may extract a predetermined number of test data from the farthest points from the cluster center.

[0027] Alternatively, the data extraction unit 40 may extract test data near the boundary of a cluster (at a predetermined distance from the boundary of the cluster). Alternatively, the data extraction unit 40 may extract test data closer to the boundary of another cluster (at a predetermined distance from the boundary of another cluster). Extracting such test data makes it possible to perform tests on test cases that are difficult to determine.

[0028] Note that test data does not need to be extracted every time the model is updated. For example, a portion of the test data extracted from all the test data (i.e., a second test data set) may be used as test data for the index calculation unit 50 (described later) to calculate the index for the updated model.

[0029] The index calculation unit 50 calculates an index that is more favorable as the expected result is obtained, based on the result when the selected test data is applied to the model. The index calculated by the index calculation unit 50 is determined in advance by a user or the like.

[0030] The index calculation unit 50 may calculate the index by any method, and the method is not particularly limited as long as the method calculates a more preferable value so that the expected result is obtained. Furthermore, the index calculation unit 50 may calculate not only one type of index, but also multiple types of indexes. For example, in the case of a model that detects objects in an image, the index calculation unit 50 may calculate an average precision (AP) or an average value of APs, such as mAP, as an index.

[0031] Specifically, the index calculation unit 50 calculates an index (hereinafter referred to as a post-update index) for a result obtained by applying the extracted test data (i.e., the test data included in the second test data set) to the post-update model, and then compares the post-update index with an index (hereinafter referred to as a pre-update index) for a result obtained by applying the test data included in the first test data set to the pre-update model.

[0032] Note that, when a model is updated multiple times during operation, the index calculation unit 50 may calculate an index by applying test data extracted by the data extraction unit 40 to a further updated model (updated model). For example, assume that all test data is used as a first test data set to test a created AI model. Then, assume that extracted test data is used as a second test data set to test an AI model updated after shipment. If the AI ​​model is subsequently further updated, the index calculation unit 50 may calculate an index for the further updated AI model using the extracted test data (i.e., the second test data set described above).

[0033] If it is determined that there is no difference in accuracy (no difference in characteristics) as a result of comparing the updated index with the pre-update index, the index calculation unit 50 may decide to complete the test using the extracted test data. On the other hand, if it is determined that there is a difference in accuracy (a difference in characteristics), the index calculation unit 50 may instruct the data extraction unit 40 to generate a second test data set to which additional test data has been added. Upon receiving this instruction, the data extraction unit 40 may, for example, extract new test data other than the test data already extracted using a method similar to the method described above, and generate a second test data set to which the extracted test data has been further added.

[0034] The method for determining whether there is a difference in accuracy is arbitrary, and the index calculation unit 50 may determine that there is no difference in accuracy when the difference between the indices is within a predetermined range, for example. The index calculation unit 50 may also verify that there is no difference in characteristics using an existing evaluation engine.

[0035] 2 is an explanatory diagram showing an example of a process for comparing differences in model characteristics. Generally, an initial AI model (here, corresponding to the pre-update model) is tested on all test cases (X cases). By calculating indices (here, pre-update indices) for the output, which is the test result, a score value SC1 of multiple accuracy indices can be obtained, as shown in FIG. 2.

[0036] In this state, in this embodiment, the feature extraction unit 20 extracts features from each piece of test data included in a test data set (first test data set) corresponding to all test cases. The feature extraction unit 20 may plot the extracted features in a feature space, as illustrated in FIG. 2 . The clustering unit 30 performs clustering on this plot. For simplicity of explanation, FIG. 2 shows an example in which the features are plotted in a two-dimensional space. However, if there are N features, for example, the feature extraction unit 20 may plot X pieces of test data in an N-dimensional space.

[0037] The data extraction unit 40 extracts a portion of test data from the clustered test data. In the example shown in Fig. 2, the data extraction unit 40 extracts test data close to the center of the cluster as a typical case, as shown in the first method above. By narrowing down the data in this way, typical cases (Y cases) are extracted.

[0038] The index calculation unit 50 inputs test data corresponding to the extracted Y typical cases into the updated AI model and obtains an output. Then, the index calculation unit 50 calculates, for this output, the same indexes (here, updated indexes) as those calculated for the AI ​​model before the update, thereby obtaining score values ​​SC2 of multiple accuracy indexes as shown in FIG.

[0039] The index calculation unit 50 compares the difference in accuracy index before and after updating the AI ​​model, and if there is no difference in accuracy, decides to complete the test with the extracted typical cases (Y cases).

[0040] The feature extraction unit 20, the clustering unit 30, the data extraction unit 40, and the index calculation unit 50 are realized by a computer processor (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) that operates according to a program (a test case selection program).

[0041] For example, the program may be stored in the storage unit 10 of the test case selection device 100, and the processor may read the program and operate as the feature extraction unit 20, the clustering unit 30, the data extraction unit 40, and the index calculation unit 50 in accordance with the program. Also, the functions of the test case selection device 100 may be provided in the form of SaaS (Software as a Service).

[0042] Furthermore, the feature extraction unit 20, the clustering unit 30, the data extraction unit 40, and the index calculation unit 50 may each be realized by dedicated hardware. Furthermore, some or all of the components of each device may be realized by general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and a program.

[0043] Furthermore, when some or all of the components of the test case selection device 100 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in which each device is connected via a communication network.

[0044] Next, the operation of the test case selection device 100 of this embodiment will be described. Fig. 3 is a flowchart showing an example of the operation of the test case selection device 100 of this embodiment. Note that in this operation example, it is assumed that the pre-update index has already been calculated.

[0045] The feature extraction unit 20 extracts features from each piece of test data included in the first test data set (step S11). The clustering unit 30 performs clustering on each extracted feature to generate corresponding clusters of test data (step S12). The data extraction unit 40 generates a second test data set by extracting some test data from the first test data set based on the distribution of test data included in each generated cluster (step S13).

[0046] Thereafter, the post-update index calculated by the index calculation unit 50 is compared with the pre-update index, and it is determined whether or not to add test data.

[0047] As described above, in this embodiment, the feature extraction unit 20 extracts features from each piece of test data included in the first test data set, and the clustering unit 30 performs clustering on each extracted feature to generate corresponding clusters of test data. Then, the data extraction unit 40 generates a second test data set from the first test data set based on the distribution of test data included in each generated cluster. This makes it possible to comprehensively select test cases that can guarantee the same quality for the updated model as before the update.

[0048] Next, an overview of the present disclosure will be described. Fig. 4 is a block diagram illustrating an overview of an information processing device according to the present disclosure. An information processing device 80 (e.g., a test case selection device 100) according to the present disclosure includes: a feature extraction means 81 (e.g., a feature extraction unit 20) that extracts features of each piece of test data from a first test data set (e.g., all test data), which is a collection of test data corresponding to test cases to be executed by a first model (e.g., a pre-update model); a clustering means 82 (e.g., a clustering unit 30) that generates corresponding clusters of test data by clustering each extracted feature; and a data extraction means 83 (e.g., a data extraction unit 40) that generates a second test data set by extracting some test data from the first test data set based on the distribution of the test data included in each generated cluster.

[0049] Such a configuration makes it possible to comprehensively select test cases that can guarantee the same quality for the updated model as before the update.

[0050] The data extracting means 83 may also extract a portion of test data from each cluster to generate a second test data set.

[0051] The data extraction means 83 may also generate the second test data set based on the distance from the cluster center determined based on the distribution of test data included in each cluster.

[0052] The data extracting means 83 may also extract test data whose distance from the center of the cluster is smaller than a predetermined threshold value.

[0053] Conversely, the data extracting means 83 may extract test data closer to the boundary of another cluster.

[0054] Furthermore, the information processing device 80 may include an index calculation means (e.g., the index calculation unit 50) that calculates an index whose value becomes more favorable as an expected result is obtained, based on a result obtained when the test data is applied to the model (e.g., the output of the model).The index calculation means may then calculate an updated index that is an index for a result obtained by applying the test data included in the second test data set to a second model (e.g., an updated model) that is an updated version of the first model.

[0055] In this case, if the data extraction means 83 determines that there is a difference in accuracy between the pre-update index and the post-update index, which are indexes for the results obtained by applying the test data included in the first test data set to the first model, it may generate a second test data set to which further test data has been added.

[0056] The test cases corresponding to the second test data set are executed on the second model, which is an updated version of the first model.

[0057] 5 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 1000 includes a processor 1001, a main memory device 1002, an auxiliary memory device 1003, and an interface 1004. The computer 1000 may also be connected to a computer that executes a mathematical programming solver, an annealing machine, a simulator, or the like.

[0058] The information processing device 80 described above is implemented in a computer 1000. The operations of the above-described processing units are stored in the form of a program (test case selection program) in an auxiliary storage device 1003. The processor 1001 reads the program from the auxiliary storage device 1003, loads it into the main storage device 1002, and executes the above-described processing in accordance with the program.

[0059] In at least one embodiment, the auxiliary storage device 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM (Compact Disc Read-only memory), a DVD-ROM (Read-only memory), and a semiconductor memory connected via the interface 1004. In addition, when this program is distributed to the computer 1000 via a communication line, the computer 1000 that receives the program may load the program into the main storage device 1002 and execute the above-described processing.

[0060] The program may be a program for realizing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in the auxiliary storage device 1003.

[0061] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above-described embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0062] 10 Storage unit 20 Feature extraction unit 30 Clustering unit 40 Data extraction unit 50 Index calculation unit 100 Test case selection device

Claims

1. An information processing device comprising: a feature extraction means for extracting features of each piece of test data from a first test data set, which is a collection of test data corresponding to a test case to be executed by a first model; a clustering means for generating a cluster of corresponding test data by clustering each extracted feature; and a data extraction means for generating a second test data set by extracting a portion of test data from the first test data set based on the distribution of test data contained in each generated cluster.

2. The information processing apparatus according to claim 1, wherein the data extraction means extracts a portion of the test data from each cluster to generate a second test data set.

3. An information processing device according to claim 1 or 2, wherein the data extraction means generates a second test data set based on a distance from a cluster center determined based on a distribution of test data included in each cluster.

4. The information processing device according to claim 1, wherein the data extraction means extracts test data whose distance from the center of the cluster is less than a predetermined threshold value.

5. The information processing device according to any one of claims 1 to 3, wherein the data extraction means extracts test data closer to a boundary of another cluster.

6. An information processing device as described in any one of claims 1 to 5, comprising an index calculation means for calculating an index that calculates a more favorable value the more an expected result is obtained, based on the results obtained when test data is applied to a model, wherein the index calculation means calculates an updated index that is an index for the results obtained by applying test data included in a second test data set to a second model that is an updated model of the first model.

7. An information processing device as described in claim 6, wherein the data extraction means generates a second test data set to which further test data is added when it is determined that there is a difference in accuracy between a pre-update index and a post-update index, which are indexes for the results obtained by applying the test data included in the first test data set to the first model.

8. The information processing device according to any one of claims 1 to 7, wherein the target on which the test cases corresponding to the second test data set are executed is a second model obtained by updating the first model.

9. A test case selection method comprising the steps of: extracting features of each piece of test data from a first test data set, which is a set of test data corresponding to a test case to be executed by a first model; generating a corresponding cluster of test data by performing clustering on each extracted feature; and generating a second test data set by extracting a portion of test data from the first test data set based on the distribution of test data contained in each generated cluster.

10. A program recording medium storing a test case selection program for causing a computer to execute: a feature extraction process for extracting features of each piece of test data from a first test data set, which is a set of test data corresponding to a test case to be executed by a first model; a clustering process for generating a cluster of corresponding test data by clustering each of the extracted features; and a data extraction process for generating a second test data set by extracting a portion of test data from the first test data set based on the distribution of test data contained in each of the generated clusters.

Images