Analysis device, analysis method, and analysis program

Abstract

This allows for the interpretation and evaluation of learned models, and facilitates adjustments. [Solution] An analysis device that analyzes output data of a learning model or a device equipped with a learning model implements a test data acquisition function that acquires test data, a data conversion function that performs a predetermined transformation on the test data, and an evaluation function that inputs the transformed test data, data based on output data obtained by inputting the transformed test data into the learning model or the device equipped with the learning model, and the transformed test data into a predetermined evaluation function to output an evaluation result.

Description

Technical Field

[0001] The present invention relates to an analysis device, an analysis method, and an analysis program.

Background Art

[0002] Patent Document 1 describes an information processing apparatus including: an acquisition unit that acquires a plurality of training examples; a selection unit that selects two or more training examples among the plurality of training examples, the prediction results of which obtained using one or more machine learning models that output prediction results with the examples as inputs are uncertain; and a generation unit that synthesizes the two or more training examples selected by the selection unit to generate artificial examples.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Recently, devices incorporating machine learning models are being used in a variety of tasks. However, these models have vulnerabilities to certain types of attacks. For example, adding a small perturbation to the input of a machine learning model can change the prediction results it outputs. In other words, by maliciously altering the input data to a device or system implementing a machine learning model, the output can be changed. For example, if a small amount of noise is added to an image of a panda and input it into an image recognition system, it may be recognized as a gibbon. In AI-powered autonomous driving systems, attacks are sometimes carried out to deliberately cause errors in recognizing roads and signs. Other attacks include impersonation in facial recognition systems that implement machine learning models, and attacks that use machine learning models in voice recognition to execute arbitrary commands on smart speakers. Therefore, in order to provide devices and systems that implement machine learning models securely, it is necessary to adjust the machine learning models to withstand these attacks.

[0005] Furthermore, trained models may overfit to test data, potentially compromising their robustness in real-world environments that do not match controlled conditions. Additionally, trained models may require periodic updates even after being implemented as a service. From this perspective, post-trained models may need adjustment.

[0006] Furthermore, learning models have the characteristic that, by learning from vast amounts of data on their own and autonomously deriving answers, their thought process is incomprehensible to humans.

[0007] The present invention has been made in view of the above circumstances, and aims to provide an analysis device, an analysis method, and an analysis program that enable the interpretation and evaluation of learning models and facilitate adjustment. [Means for solving the problem]

[0008] To achieve the aforementioned objectives, the analysis apparatus, analysis method, and analysis program of the present invention are characterized by the following:

[0009] [1] An analysis device for analyzing output data of a learning model or a device equipped with a learning model, A test data acquisition function that acquires test data, A data conversion function that performs a predetermined transformation on the aforementioned test data, An evaluation function that inputs the test data after the conversion into the learning model or a device equipped with the learning model, along with the output data obtained, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analytical device that makes this possible.

[0010] [2] An analysis method for analyzing output data of a learning model or a device equipped with a learning model, The test data acquisition step involves obtaining test data, A data conversion step in which a predetermined transformation is performed on the test data, An evaluation step involves inputting the test data after the conversion into the learning model or a device equipped with the learning model, along with data based on output data obtained from that input, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analytical method to achieve this.

[0011] [3] An analysis program for analyzing output data of a learning model or a device equipped with a learning model, wherein a computer A test data acquisition function that acquires test data, A data conversion function that performs a predetermined transformation on the aforementioned test data, An evaluation function that inputs the test data after the conversion into the learning model or a device equipped with the learning model, along with the output data obtained, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analysis program that makes this possible. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide an analysis device, an analysis method, and an analysis program that enable interpretation and evaluation of a learning model and facilitate adjustment.

[0013] In addition, unexpected movements of the learning model that cannot be grasped during the learning of the learning model can be grasped in advance.

Brief Description of the Drawings

[0014] [Figure 1] A block diagram showing an example of the configuration of an analysis system corresponding to at least one embodiment of the present invention. [Figure 2] A block diagram showing the configuration of a server corresponding to at least one embodiment of the present invention. [Figure 3] A conceptual diagram illustrating a learning model corresponding to at least one embodiment of the present invention. [Figure 4] A conceptual diagram showing an example of the configuration of an analysis device corresponding to at least one embodiment of the present invention.

Modes for Carrying Out the Invention

[0015] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Note that various components in the examples of each embodiment described below can be appropriately combined as long as there is no contradiction. In addition, the content described as an example of a certain embodiment may be omitted in other embodiments. Also, operations and processes not related to the characteristic parts of each embodiment may be omitted. Furthermore, the order of various processes constituting various flows and sequences described below is arbitrary as long as there is no contradiction in the processing content.

[0016] Hereinafter, an analysis program executed in a server, which is an example of a computer, will be exemplified and described. However, the computer may be another device such as a user terminal. Also, the analysis system may execute the analysis program as a whole.

[0017] FIG. 1 is a block diagram showing an example of the configuration of an analysis system corresponding to at least one embodiment of the present invention. The analysis system 1 includes a server 10 and a user terminal 20 used by a user of the analysis system 1. User terminals 20A, 20B, and 20C are each an example of the user terminal 20. The configuration of the analysis system 1 is not limited to this. For example, the analysis system 1 may have a configuration in which a single user terminal is used by a plurality of users. The analysis system 1 may include a plurality of servers.

[0018] The server 10 is an example of an analysis device according to an embodiment of the present disclosure.

[0019] The analysis device may analyze the output data of the learning model. For example, when the learning model and the analysis device are integrated, the analysis device may acquire and analyze the output data of the learning model.

[0020] The analysis device may analyze the output data of a device including a learning model. For example, when a device such as the user terminal 20 including a learning model and the analysis device are separate devices, the analysis device may acquire and analyze the output data of the device including the learning model.

[0021] The server 10 and the user terminal 20 are each communicably connected to a communication network 30. The connection between the communication network 30 and the server 10 and the connection between the communication network 30 and the user terminal 20 may be a wired connection or a wireless connection. The communication network 30 may be a LAN network, a Wifi network, an Internet line, an analog telephone line, or a combined line thereof.

[0022] By including the server 10 and the user terminal 20, the analysis system 1 realizes various functions for executing various processes according to a user's operation.

[0023] Server 10 comprises a processor 11, memory 12, and storage device 13. The processor 11 is, for example, a central processing unit such as a CPU (Central Processing Unit) that performs various calculations and controls. If Server 10 is equipped with a GPU (Graphics Processing Unit), some of the various calculations and controls may be performed by the GPU. Server 10 uses the data read into memory 12 to perform various information processing with the processor 11 and stores the obtained processing results in storage device 13 as needed.

[0024] The storage device 13 functions as a storage medium for storing various types of information. The configuration of the storage device 13 is not particularly limited, but from the viewpoint of reducing the processing load on the user terminal 20, it may be configured to store all the various types of information necessary for the control performed by the analysis system 1. Examples of such configurations include HDDs and SSDs. However, the storage device that stores the various types of information only needs to have a storage area that is accessible to the server 10, and for example, it may be configured to have a dedicated storage area outside the server 10.

[0025] The user terminal 20 is managed by the user. The user terminal 20 may be, for example, various measuring devices. The user terminal 20 may also be any device that provides services to the user.

[0026] The user terminal 20 may be equipped with hardware and software for performing various processes by connecting to the communication network 30 and communicating with the server 10. Each of the multiple user terminals 20 may also be configured to communicate directly with each other without going through the server 10.

[0027] The user terminal 20 may include a processor 21, memory 22, and storage device 23. The processor 21 is, for example, a central processing unit such as a CPU (Central Processing Unit) that performs various calculations and controls. If the user terminal 20 is equipped with a GPU (Graphics Processing Unit), some of the various calculations and controls may be performed by the GPU. The user terminal 20 uses the data read into the memory 22 to perform various information processing using the processor 21, and stores the obtained processing results in the storage device 23 as needed. The storage device 23 functions as a storage medium for storing various information.

[0028] Figure 2 is a block diagram showing the configuration of a server corresponding to at least one embodiment of the present invention. The server 10 comprises a test data acquisition unit 101, a data conversion unit 102, an evaluation unit 103, and a report output unit 104. The processor 11 of the server 10 refers to an analysis program held in the storage device 13 and executes the program to functionally realize the test data acquisition unit 101, the data conversion unit 102, the evaluation unit 103, and the report output unit 104.

[0029] Server 10 or user terminal 20 may have a learning model. External devices connected to server 10 or user terminal 20 in a communicative manner may also have a learning model. In this example, we will assume that server 10 has a learning model 105.

[0030] The test data acquisition unit 101 has the function of acquiring test data. The data conversion unit 102 has the function of performing a predetermined conversion on the test data. The evaluation unit 103 has the function of inputting the converted test data, data based on output data obtained by inputting the converted test data into a learning model or a device equipped with a learning model, and the converted test data into a predetermined evaluation function to output an evaluation result. The evaluation unit 103 may also further input the original test data before data conversion, data based on output data obtained by inputting the original test data before data conversion into a learning model or a device equipped with a learning model, and the original test data before data conversion into a predetermined evaluation function to output an evaluation result. The report output unit 104 has the function of outputting a report based on the evaluation result.

[0031] The evaluation function may be a function that outputs the feature importance in the learning model 105. The evaluation function may be a function that outputs a value that shows the average relationship between features and predicted values ​​in the learning model 105.

[0032] Figure 3 is a conceptual diagram illustrating a learning model corresponding to at least one embodiment of the present invention.

[0033] The learning model 105 relating to this disclosure is a model obtained by machine learning using a large amount of training data. The learning model 105 includes a neural network. If the server 10 has the learning model 105, data defining the structure of the neural network and the weight parameters of each node are pre-stored as the learning model 105 in memory 12 or storage device 13.

[0034] When input data is given to the learning model 105, output data indicating the confidence level is output. The confidence level is a probability value representing the likelihood of the judgment result, calculated by the learning model 105 when making a judgment; a higher value indicates a higher likelihood. The confidence level may be a value output from the final layer of the learning model 105 as a normalized value in the range of 0 to 1.

[0035] For example, if an image of a dog is input to the learning model 105, the output data will show a confidence level, such as a 90% probability (confidence level 0.9) that the image is a dog and a 10% probability (confidence level 0.1) that it is a cat.

[0036] As described above, the learning model 105 may be subjected to attacks that alter the output by maliciously modifying the input data, and it may be necessary to adjust the learning model 105 to make it resistant to such attacks. Also, if the learning model 105 is overfitted to the test data, its robustness in real-world environments that do not match the controlled conditions may be compromised. In fact, when the service is provided, different data is used than the data used to train and validate the learning model 105, so situations may occur where the confidence level of the predictions made by the learning model 105 is low. Furthermore, the learning model may require periodic updates even after it has been implemented as a service. From this perspective as well, post-training adjustments to the learning model are necessary. For these reasons, it is necessary to clearly visualize situations in which the confidence level of the output data from the learning model 105 decreases.

[0037] An analysis apparatus according to one embodiment of the present disclosure, Which features did you focus on? • Relationship between features and predicted values This is visualized by quantifying at least one of these factors. By observing the information obtained by quantifying these factors, the user can discover data (prediction results) that were not anticipated during the training of the learning model 105.

[0038] In this example, features refer to information contained in the input data. Features may represent all of the information in the input data, or they may represent only a portion of the information in the input data.

[0039] Figure 4 is a conceptual diagram showing an example of the configuration of an analytical device corresponding to at least one embodiment of the present invention.

[0040] The test data acquisition unit 101 acquires test data. The source of the test data may be the memory of the analysis device, or the test data may be acquired from an external device as seen from the analysis device. For example, the test data may be time-series data output by a measurement device. Also, the test data referred to here may be data after preprocessing has been performed.

[0041] The analysis device may input the acquired test data directly into the learning model 105. However, in order to obtain input information for the evaluation function described later, the analysis device according to one embodiment of this disclosure performs a predetermined transformation on the test data and inputs the transformed data into the learning model 105. The data transformation unit 102 performs a predetermined transformation on the test data.

[0042] The predetermined transformation applied to the test data is a transformation process different from the preprocessing described above. Typically, the features of the input data are modified, and the influence of the modification on the output data from the learning model 105 is quantified using the evaluation function described later. The predetermined transformation may be, for example, a moving average, derivative, median filter, maximum filter, or minimum filter. If the input data is image data, the predetermined transformation may be converting a color image to a grayscale or black and white image, or gradually changing the contrast. The data transformation unit 102 may perform the predetermined transformation on the entire input data or on a portion of the input data.

[0043] The evaluation unit 103 inputs the data based on the output data obtained by inputting the data-converted test data into a learning model or a device equipped with a learning model, along with the data-converted test data, into a predetermined evaluation function. The evaluation unit 103 may further input the data based on the output data obtained by inputting the original test data before data conversion into a learning model or a device equipped with a learning model, along with the original test data before data conversion, into the predetermined evaluation function. The evaluation function outputs the evaluation result.

[0044] (First evaluation function) The first evaluation function outputs the feature importance in the learning model 105. The data transformation unit 102 performs a transformation that shuffles the feature values, creating a state where those features cannot be effectively used. The first evaluation function compares the error between the ground truth data and the predicted data using the input data before transformation with the error between the ground truth data and the predicted data using the input data after shuffling, and outputs the difference between these two errors as the feature importance. Features whose errors increase after shuffling are considered to contribute greatly to the performance of the learning model 105 and are therefore considered to have high importance.

[0045] (Second evaluation function) The second evaluation function may be a function that calculates the average relationship between features and predicted values ​​in the learning model 105. The average relationship is the average relationship between a feature and a predicted value, showing how a certain feature affects the predicted value output by the learning model 105. For example, the second evaluation function may output a value indicating whether or not the features and predicted values ​​are proportional. The second evaluation function may output a value indicating whether or not the features and predicted values ​​are inversely proportional. The so-called correlation coefficient may be used as a value indicating at least one of proportionality and inverse proportionality. The correlation coefficient is a coefficient that approaches 1 as the positive correlation between the features and predicted values ​​is stronger, and approaches -1 as the negative correlation is stronger.

[0046] The second evaluation function may output a value indicating whether or not there is a linear relationship between the feature and the predicted value. The second evaluation function may output a value indicating whether or not there is a nonlinear relationship between the feature and the predicted value.

[0047] The report output unit 104 outputs a report based on the evaluation results. For example, the report output unit 104 outputs a report that includes information indicating whether or not the feature importance values ​​are aligned with the knowledge of the problem that the learning model 105 is trying to solve.

[0048] For example, the report output unit 104 identifies and outputs features whose importance was assumed to be important during the training phase of the learning model 105, but which are output as feature importance values ​​in the evaluation function. Conversely, the report output unit 104 identifies and outputs features whose importance was assumed to be less important during the training phase of the learning model 105, but which are output as feature importance values ​​in the evaluation function. To achieve this, for example, a predetermined value, the assumed feature importance, may be stored in memory 12 or storage device 13. The assumed feature importance is a value that indicates the importance of the features included in the input data that was assumed to be important during the training phase of the learning model 105. The report output unit 104 outputs information as a report indicating features whose absolute difference between the assumed feature importance and the feature importance output by the evaluation function is greater than a predetermined value.

[0049] The report output unit 104 may output values ​​such as the following as a report. • A value (e.g., correlation coefficient) that indicates whether or not there is a proportional relationship between the feature and the predicted value. • A value (e.g., correlation coefficient) that indicates whether or not there is an inverse relationship between the feature and the predicted value. • A value indicating whether or not there is a linear relationship between the feature and the predicted value. • A value indicating whether or not there is a nonlinear relationship between the feature and the predicted value.

[0050] The user can refer to the report output by the report output unit 104 to create an update plan for the learning model 105.

[0051] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. For example, the steps in the methods of this disclosure can be performed in any order as long as they do not cause any inconsistencies. Furthermore, the components of the embodiments described above may be combined in any way without departing from the spirit of the disclosure.

[0052] In each of the embodiments described above, the user terminal 20 and the server 10 execute the various processes described above in accordance with various control programs (e.g., analysis programs) stored in their own storage devices. Furthermore, other computers, not limited to the user terminal 20 and the server 10, may also execute the various processes described above in accordance with various control programs (e.g., analysis programs) stored in their own storage devices.

[0053] Furthermore, the configuration of the analysis system 1 is not limited to the configuration described as an example of the embodiment above. For example, the server may perform some or all of the processes described as being performed by the user terminal, or the user terminal may perform some or all of the processes described as being performed by the server. Alternatively, the user terminal may be equipped with some or all of the storage unit (memory device) provided by the server. In other words, the analysis system 1 may be configured such that one of the user terminals or the server provides some or all of the functions provided by the other.

[0054] Furthermore, the program may be configured to implement some or all of the functions described above as examples of each embodiment in a standalone device that does not include a communication network.

[0055] [Note] The above-described embodiments are written in such a way that at least the following invention can be put into practice by a person with ordinary skill in the art to which the invention pertains.

[0056] [1] An analysis device for analyzing output data of a learning model or a device equipped with a learning model, A test data acquisition function that acquires test data, A data conversion function that performs a predetermined transformation on the aforementioned test data, An evaluation function that inputs the test data after the conversion into the learning model or a device equipped with the learning model, along with the output data obtained, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analytical device that makes this possible.

[0057] [2] The evaluation function is a function that outputs the feature importance in the learning model. [1] The analytical apparatus described above.

[0058] [3] The analysis apparatus according to [1], wherein the evaluation function is a function that outputs a value that shows the average relationship between the features and the predicted values ​​in the learning model.

[0059] [4] To further implement a report output function that generates a report based on the aforementioned evaluation results, [1] The analytical apparatus described above.

[0060] [5] An analysis method for analyzing output data of a learning model or a device equipped with a learning model, The test data acquisition step involves obtaining test data, A data conversion step in which a predetermined transformation is performed on the test data, An evaluation step involves inputting the test data after the conversion into the learning model or a device equipped with the learning model, along with data based on output data obtained from that input, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analytical method to achieve this.

[0061] [6] An analysis program for analyzing output data of a learning model or a device equipped with a learning model, wherein a computer A test data acquisition function that acquires test data, A data conversion function that performs a predetermined transformation on the aforementioned test data, An evaluation function that inputs the test data after the conversion into the learning model or a device equipped with the learning model, along with the output data obtained, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analysis program that makes this possible. [Explanation of Symbols]

[0062] 1. Analysis System 10 servers 11 processors 12 memory 13 Storage device 20, 20A, 20B User Terminals 21 processors 22 memory 23 Storage device 30 Communication Networks 101 Test Data Acquisition Unit 102 Data Conversion Unit 103 Evaluation Department 104 Report Output Section 105 Learning Models

Claims

1. An analysis device for analyzing output data of a learning model or a device equipped with a learning model, A test data acquisition function that acquires test data, A data conversion function that performs a predetermined transformation on the aforementioned test data, An evaluation function that inputs the test data after the conversion into the learning model or a device equipped with the learning model, along with the output data obtained, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analytical device that makes this possible.

2. The evaluation function is a function that outputs the feature importance in the learning model. The analysis apparatus according to claim 1.

3. The analysis apparatus according to claim 1, wherein the evaluation function is a function that outputs a value representing the average relationship between the features and the predicted values ​​in the learning model.

4. To further implement a report output function that generates a report based on the aforementioned evaluation results, The analysis apparatus according to claim 1.

5. An analysis method for analyzing output data of a learning model or a device equipped with a learning model, The test data acquisition step involves obtaining test data, A data conversion step in which a predetermined transformation is performed on the test data, An evaluation step involves inputting the test data after the conversion into the learning model or a device equipped with the learning model, along with data based on output data obtained from that input, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analytical method to achieve this.

6. An analysis program for analyzing output data of a learning model or a device equipped with a learning model, wherein a computer A test data acquisition function that acquires test data, A data conversion function that performs a predetermined transformation on the aforementioned test data, An evaluation function that inputs the test data after the conversion into the learning model or a device equipped with the learning model, along with the output data obtained, and the test data after the conversion into a predetermined evaluation function to output an evaluation result. An analysis program that makes this possible.

Images