Verification device, inference system, verification method and program

Abstract

To provide a verification device capable of presenting a ratio of an input data area of each determination result of an expectation output of an inference model.SOLUTION: A verification device (2) includes an expectation output determination part (24) for determining whether output data of an inference model to an input data area is a violation determination that is not an expectation output, an input area division part (25) for dividing an area determined not to be the violation determination in the input data area, and setting the divided area in a new input data area for the inference model, an area ratio calculation part (27) for calculating an area ratio of each determination result to the input data area, and a display processing part (28) for displaying verification result information showing the area ratio in a display device (6).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a verification device, an inference system, a verification method, and a program. [Background technology]

[0002] A technique for evaluating the validity of an inference model, which is a machine learning model, has been proposed. For example, Patent Document 1 presents a model output region for a data region input to a decision tree ensemble model, which contains at least one violating data sample that violates a verification property. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7059220 Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional technology described in Patent Document 1 presents a range that includes at least one violating data sample that is not the expected output of the inference model, but there is no guarantee that all data samples in the presented range are violating data samples. For this reason, the conventional technology has a problem in that, for example, if there are no violating data samples in most areas within the presented range and violating data samples are present in only a very narrow area, the degree of violation is overestimated, making it difficult to appropriately deal with the violation.

[0005] The present disclosure solves the above problem and aims to provide a verification device that can present the proportion of input data area for each judgment result of the expected output of an inference model. [Means for solving the problem]

[0006] The verification device according to the present disclosure includes an expected output determination unit that determines whether the output data of an inference model for an input data area, which is a numerical range of data, is a violation determination that is not the expected output; an input area division unit that divides areas of the input data area that are determined not to be a violation determination and sets the divided areas as new input data areas for the inference model; an area proportion calculation unit that calculates the proportion of areas for each determination result for the input data area; and a verification result output unit that outputs verification result information indicating the proportion of areas. [Effects of the Invention]

[0007] According to the present disclosure, a verification device determines whether the output data of an inference model for an input data region is a violation judgment that is not the expected output, divides the input data region that is determined not to be a violation judgment, sets the divided region as a new input data region for the inference model, calculates the region proportion for each judgment result for the input data region, and outputs verification result information indicating the region proportion. By outputting the region proportion for each judgment result of the expected output of the inference model, the verification device according to the present disclosure can present the input data region proportion for each judgment result of the expected output of the inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of an inference system according to embodiment 1. FIG. [Figure 2] 10 is a graph showing the determination results of the input data area and the expected output. [Figure 3] 10A and 10B are schematic diagrams illustrating division processing of an input data area and expected output determination results. [Figure 4] FIG. 10 is a schematic diagram showing an outline of expected output determination. [Figure 5] FIG. 10 is a diagram illustrating an example of an expected output determination result. [Figure 6] 6A, 6B, and 6C are diagrams showing examples of displaying the verification results. [Figure 7]3 is a flowchart showing a verification method according to the first embodiment. [Figure 8] 10 is a flowchart showing an area determination process. [Figure 9] 9A and 9B are block diagrams showing a hardware configuration for realizing the functions of the verification device according to the first embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of an inference system according to a second embodiment. [Figure 11] 10A and 10B are diagrams showing the results of acquiring a data area of ​​an expected output determination result and an example of its display; [Figure 12] 10A and 10B are diagrams showing the results of acquiring data samples from the data area of ​​the expected output determination results and examples of their display. [Figure 13] FIG. 10 is a schematic diagram showing an example of a display of a region where a summary has been performed. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiment 1 FIG. 1 is a block diagram showing an example configuration of an inference system 1 according to a first embodiment. In FIG. 1, the inference system 1 is a system that performs inference using a verified inference model, and includes a verification device 2, an inference device 3, an inference model storage device 4, a judgment result storage device 5, and a display device 6. The verification device 2 is a device that verifies an inference model that is a machine learning model, and the inference device 3 is a device that performs inference using the inference model. The inference system 1 is a system in which the verification device 2, the inference device 3, the inference model storage device 4, the judgment result storage device 5, and the display device 6 are connected via a network using wired signal lines or wireless communication. The network is a telecommunications line including the Internet, etc.

[0010] The verification device 2 can prevent the degree of violation from being overestimated or underestimated by presenting the proportion of the input data area for each judgment result of the expected output of the inference model. For example, in the conventional technology described in Patent Document 1, a violation range that includes at least one violating data sample that is not the expected output of the inference model is presented as the verification result. Note that the expected output is a proposition that is expected to be true for the output data of the inference model. Figure 2 is a graph showing the input data domain and the judgment results of the expected output. The input data domain shown in Figure 2 is the data domain defined by input variables x1 and x3. Region A shown in Figure 2 is the data domain that includes region B. All data included in region B are non-compliant data samples that do not meet the expected output of the inference model, i.e., do not satisfy the verification conditions, and region B is a region that includes only non-compliant data samples.

[0011] In the conventional technology described in Patent Document 1, a region A may be proposed. However, there is no guarantee that all data samples within this region A are violations. For example, as shown in FIG. 2, there is a possibility that most of the regions do not contain any violation data samples, and in this case, the degree of violation may be overestimated. Since it is possible that all data samples in area A fall within the violation range, the output data of the inference model when data in area A is input may not be the expected output, i.e., it may be determined to be a violation that does not satisfy the verification conditions. In this case, the output data of the inference model when data included in an area in area A where no violation data samples exist is input is actually the expected output, but the inference is suspended.

[0012] In addition, the output value of the inference model when data in area A is input may be overwritten to appear as if there is no violation. In this case, the output data of the inference model when data included in an area where no violating data samples exist is input may also be overwritten with a different value, even though it is the expected output.

[0013] In addition, there are cases where the inference model is retrained by adding data samples in area A so that the output value of the inference model when data in area A is input is free of violations. In this case, the inference behavior of the inference model when data in an area where no violating data samples exist in area A is input will also change significantly.

[0014] In a case where the test is passed if the hypervolume of the violation range output from the inference model when data contained in area A is input is less than or equal to a specified value, it is possible that the test will not pass even though the hypervolume of the violation range is actually less than or equal to the specified value if the input data area includes an area where no violation data samples exist.

[0015] Therefore, the verification device 2 according to the first embodiment presents the proportion of the input data area for each expected output judgment result of the inference model by outputting the proportion of the area for each expected output judgment result of the inference model. This allows the verification device 2 to prevent problems caused by overestimating or underestimating the degree of violation as described above.

[0016] Figure 3 is an overview diagram showing the division process of the input data area and the results of the expected output judgment, and shows the input data area of ​​the inference model defined by input variables x1 and x3. In Figure 3, the input data area containing only input data whose output data of the inference model is all the expected output is the pass area. The input data area containing only input data whose output data of the inference model is all the expected output and whose output data is judged to be a violation is the violation area. The input data area that is neither the pass area nor the violation area is the unknown area.

[0017] The verification device 2 inputs an input data region into the inference model and determines whether all of the output data of the inference model is a violation judgment that is not the expected output, thereby calculating the proportion of the region for each judgment result relative to the input data region. As shown in the graph on the left side of Figure 3, the proportion of pass regions and the proportion of unknown regions of the input data region to the inference model are 23.9% and 76.1%, respectively. Since the verification device 2 includes unknown regions, i.e., input data regions that were not judged as violations, the verification device 2 divides the unknown regions as shown by the arrows in Figure 3 and sets the divided regions as new input data regions for the inference model.

[0018] Next, the verification device 2 sets each divided area of ​​the unknown area as a new input data area of ​​the inference model, and calculates the proportion of the area for each judgment result to the input data area by determining whether the output data of the inference model is a violation judgment that is not the expected output. For example, as shown in the graph in the center of Figure 3, the proportion of the areas at this time is assumed to be 46.7% of the input data area to the inference model as a pass area, 42.5% as an unknown area, and 0.8% as a violation area. In this case, the verification device 2 determines that an unknown area is included, and further divides these unknown areas as shown by the arrows in Figure 3.

[0019] Next, the verification device 2 sets the newly divided unknown area as a new input data area for the inference model, and calculates the proportion of each area for each judgment result relative to the input data area by determining whether all of the output data of the inference model is a violation judgment that is not the expected output. For example, as shown in the graph on the right side of Figure 3, the proportion of the areas in the input data area to the inference model is assumed to be 98.3% pass, 0% unknown, and 1.7% violation. In this way, the verification device 2 can present the proportion of the area in the input data area for the inference model that corresponds to the expected output judgment result, and can also identify the data value range for each area. This prevents the degree of violation from being overestimated or underestimated.

[0020] The inference model storage device 4 is a storage device that stores inference models. From the perspective of the verification device 2, the inference model stored in the inference model storage device 4 is a machine learning model to be verified, and from the perspective of the inference device 3, it is a machine learning model for inference processing. The inference model storage device 4 is provided outside the verification device 2 and the inference device 3. Note that the inference model storage device 4 only needs to be accessible from the verification device 2 and the inference device 3, and may be a storage device provided in a computer that functions as the verification device 2 or the inference device 3.

[0021] The judgment result storage device 5 is a storage device that stores the expected output judgment result of the expected output judgment unit 24 provided in the verification device 2. The expected output judgment result is a judgment result as to whether or not all of the output data of the inference model for the input data area, which is the numerical range of the data, is a violation judgment that is not the expected output. The judgment result storage device 5 is provided outside the verification device 2. The judgment result storage device 5 only needs to be accessible from the verification device 2, and may be a storage device included in a computer that functions as the verification device 2.

[0022] The verification device 2 is realized, for example, by a computer having a communication unit, a calculation unit, and a memory unit. The communication unit communicates with the inference model memory device 4 or the judgment result memory device 5 via a network using a wired signal line or wireless communication. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark). The communication unit includes an input interface 100 and an output interface 101 in FIGS. 9A and 9B.

[0023] The calculation unit controls the overall operation of the verification device 2. The calculation unit includes an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input region division unit 25, an input region designation unit 26, a region proportion calculation unit 27, and a display processing unit 28. The calculation unit executes an information processing application for verifying the inference model, thereby realizing various functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28. The calculation unit includes the processing circuit 102 of FIG. 9A and the processor 103 of FIG. 9B.

[0024] The storage unit stores information processing applications and information used for the arithmetic processing of the arithmetic unit. The storage unit is a storage device provided in a computer functioning as the verification device 2, and includes storage such as an HDD or SSD, or the memory 104 shown in FIGS. 9A and 9B. The storage unit may be provided outside the verification device 2 as long as it is accessible by the verification device 2.

[0025] The inference model reading unit 21 reads the inference model to be verified from the inference model storage device 4. The inference model is an N-dimensional input X=(x1, x2, . . . , x N )∈R N For 1-dimensional output y∈R or M-dimensional output Y=(y1,y2,...,y M )∈R M where N and M are integers. Examples of inference models include trained machine learning models, neural networks, decision tree models, decision tree ensemble models, support vector machines (SVMs), generalized linear models, generalized additive models (GAMs), Gaussian process regression models (GPRs), naive Bayes, and Gaussian mixture models (GMMs).

[0026] Neural networks include multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), or transformers. Decision tree ensemble models include random forests or gradient boosted trees. Generalized linear models include linear regression or logistic regression.

[0027] The output upper and lower bound calculation unit 22 calculates at least one of the upper and lower bounds of the output data of the inference model for the input data domain. In other words, the inference model outputs data such that the lower bound value ≦ output data ≦ upper bound value is always true. The upper bound is calculated, for example, as follows. When the inference model is a generalized linear model, the output upper and lower bound calculation unit 22 calculates the upper bound as the output value when a variable with a positive coefficient is the upper limit value of the input data region and a variable with a negative coefficient is the lower limit value of the input data region. When the inference model is a single decision tree model, the output upper and lower bound calculation unit 22 calculates the maximum output value of the set of leaf nodes that can be reached by the data samples in the input data domain as the upper bound value. When the inference model is a decision tree ensemble model, the output upper and lower bound calculation unit 22 calculates the sum of the maximum output values ​​for each decision tree among the set of leaf nodes that can be reached by data samples in the input data domain as the upper bound value.

[0028] The lower bound is calculated, for example, as follows. When the inference model is a generalized linear model, the output upper and lower bound calculation unit 22 calculates the lower bound as the output value when a variable with a positive coefficient is the lower limit value of the input data region and a variable with a negative coefficient is the upper limit value of the input data region. When the inference model is a single decision tree model, the output upper and lower bound calculation unit 22 calculates the minimum output value of the set of leaf nodes that can be reached by the data sample in the input data domain as the lower bound. When the inference model is a decision tree ensemble model, the output upper and lower bound calculation unit 22 calculates, as the lower bound value, the sum of the minimum output values for each decision tree among the set of leaf nodes that the data samples within the input data area can reach.

[0029] The expected output specifying unit 23 sets the expected output value of the inference model for the expected output determination unit 24. The expected output is a proposition that is expected to be true for the output data y of the inference model. For example, specifying 50≦y<80 as the range of the output value of regression, specifying 0.5≦y as the range of the output score (output of the sigmoid function) of binary classification, specifying (y1<y2)&(y3<y2) as the output class of multi-class classification, specifying the output of the softmax function's inequality to output class 2 as the expected output, or specifying |yーy’|≦0.1 which is the difference from the output y’=f’(X) of another inference model f’, etc. can be cited.

[0030] For example, the user uses an input device not shown in FIG. 1 to input the conditional expression or upper and lower limit values of the expected output in text. Thereby, the expected output specifying unit 23 sets, as the expected output, the information indicating the conditional expression or upper and lower limit values input by the input device for the expected output determination unit 24. Also, the expected output specifying unit 23 instructs the display processing unit 28 to display, on the screen of the display device 6, a selection screen for selecting a class ID by a drop-down or checkbox, or a selection screen for selecting the range of the output by a parallel coordinate plot. When the user performs a selection operation on the above selection screen using the above input device, the expected output specifying unit 23 receives the selection operation information and sets the selection operation information as the expected output for the expected output determination unit 24.

[0031] The expected output determination unit 24 determines whether all of the output data of the inference model for the input data region is a violation determination that is not the expected output. Figure 4 is a schematic diagram showing an overview of expected output determination, showing a case where the expected output of regression is 50≦y<80. In Figure 4, all output data y included in the numerical range (lower bound of y≧50) & (upper bound of y<80) is also included in the expected output range 50≦y<80, so the input data region for the inference model when the output data y is output is not determined to be a violation and is a pass region. Since all output data y included in the numerical range (upper bound of y<50)|(lower bound of y≧80) is not included in the expected output range 50≦y<80, the input data area for the inference model when the output data y is output is a violation area determined to be a violation. In the numerical range that is not included in either (lower bound of y≧50)&(upper bound of y<80) or (upper bound of y<50)|(lower bound of y≧80), the output data y that is not judged as violating the rule is included. The expected output judgment unit 24 judges that the output data y is included in the unknown region.

[0032] When the expected output of binary classification is 1, i.e., the output score y is 0.5≦y, all output scores y that fall within the score range where the lower bound of the output score y is ≧0.5 are included in the expected output range 0.5≦y, so the input data domain for the inference model when the output score y is output is the pass domain. All output scores y that fall within the numerical range where the upper bound of the output score y is < 0.5 are not included in the expected output range 0.5≦y, so the input data domain for the inference model when the output score y is output is a violation domain. In a score range where neither the lower bound of the output score y≧0.5 nor the upper bound of the output score y<0.5, output data y that is not judged as violating the rule is included. The expected output judgment unit 24 judges that the output data y is included in the unknown region.

[0033] When the expected output of the three-class classification is class 2, that is, when (y1 < y2) & (y2 < y3), all output data y within the numerical range of (upper bound of y1 < lower bound of y2) & (upper bound of y3 < lower bound of y2) are also included in the expected output range of (y1 < y2) & (y2 < y3). Therefore, the input data region for the inference model when the output data y is output is the qualified region. All output data y within the numerical range of (lower bound of y1 ≥ upper bound of y2) | (lower bound of y3 ≥ upper bound of y2) are not included in the expected output range of (y1 < y2) & (y2 < y3). Therefore, the input data region for the inference model when the output data y is output is the violation region. In the numerical range that is not included in either (upper bound of y1 < lower bound of y2) & (upper bound of y3 < lower bound of y2) or (lower bound of y1 ≥ upper bound of y2) | (lower bound of y3 ≥ upper bound of y2), there are output data y that are not violation judgments. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0034] When |y - y'| ≤ 0.1, which is the difference from the output y' = f'(X) of another inference model f', and is within the expected output range, all output data y within the numerical range of (upper bound of y - lower bound of y' ≤ 0.1) & (upper bound of y' - lower bound of y ≤ 0.1) are also included in the expected output range. Therefore, the input data region for the inference model when the output data y is output is the qualified region. All output data y within the numerical range of (lower bound of y - upper bound of y' > 0.1) | (lower bound of y' - upper bound of y > 0.1) are not included in the expected output range of |y - y'| ≤ 0.1 and are violation judgments. Therefore, the input data region for the inference model when the output data y is output is the violation region. In the numerical range that is not included in either (upper bound of y - lower bound of y' ≤ 0.1) & (upper bound of y' - lower bound of y ≤ 0.1) or (lower bound of y - upper bound of y' > 0.1) | (lower bound of y' - upper bound of y > 0.1), there are output data y that are violation judgments that are not the expected output. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0035] The determination result information of the expected output determination unit 24 is stored in the determination result storage device 5. 5 is a diagram showing an example of the expected output determination result, and shows the stored contents of the determination result storage device 5. As shown in FIG. 5, the determination result information includes an input data area and an expected output determination result for each input data. The input data area contains input variables x1, x2, . . . , x N The expected output result is either pass, violate, or unknown. Although input data corresponding to all judgment results is shown, the judgment result storage device 5 may store only specific judgment results as expected output judgment results. For example, it may store only input data for which the expected output judgment result is a violation.

[0036] The input region division unit 25 divides a region of the input data region that is determined not to be a violation judgment, and sets the divided region as a new input data region for the inference model. For example, the input region division unit 25 may divide the input data region at the branches of a decision tree included in the inference model. In this case, the input region division unit 25 divides the input data region into overlapping regions of the data region corresponding to each leaf node of the decision tree and the input data region. In this case, the input region division unit 25 may ignore leaf nodes that cannot be reached in consideration of the constraints of the input data region, or may stop division at an intermediate branch rather than at a terminal leaf node.

[0037] The input region dividing unit 25 may also divide the input data region randomly. For example, the input region dividing unit 25 randomly selects an input variable and determines a division point within the upper and lower limits of the input variable. Furthermore, the input region dividing unit 25 may divide the input data region based on the midpoint of the upper and lower limits generated from a uniform distribution, or the mean, median, or quantile of the training data. The input region division unit 25 may weight the selection of input variables to be divided according to the importance of the features of the inference model.

[0038] Furthermore, the input region division unit 25 may divide the input data region into multiple division candidates and adopt the division that results in the best index after division. For example, the input region division unit 25 adopts the division that results in the smallest difference between the upper and lower bounds of the inference model output for the divided region. Alternatively, the input region dividing unit 25 may adopt a division that minimizes the sum of the sizes of the unknown regions after division.

[0039] The input region division unit 25 may continue recursively dividing the input region until there are no more unknown regions, or may stop dividing the input region midway. For example, the input region division unit 25 stops dividing the input region when the processing time exceeds a specified value, the number of divided regions exceeds a specified value, the size of the unknown region falls below a specified value, or the size of the unknown region becomes smaller than the size of the violation region.

[0040] The input domain designation unit 26 sets the input data domain to the output upper and lower bound calculation unit 22. The input data domain is a data domain in an N-dimensional space that is input to the inference model to be verified. N is an integer. For example, the input data domain is table data defined by an input variable x. The input data domain may be a feature extracted from an image, video, text, audio, graph, or time-series data using a neural network or the like.

[0041] The input data domain is, for example, S={X=(x1,x2,...,x N )∈R N |-3.5≦x1≦5.2,···,15≦x N ≦40} or S = {X∈R N ||X| 2 ≦r 2}, or S=S1∩S2 representing the intersection of multiple domains, or S=S1∪S2 representing the union of multiple domains.

[0042] For example, a user inputs a conditional expression or upper and lower limit values ​​of a hyper-rectangle prism in text form using an input device not shown in FIG. 1 . As a result, input region designation unit 26 sets information indicating the conditional expression or upper and lower limit values ​​of the hyper-rectangle prism input by the input device as an input data region in output upper and lower bound calculation unit 22. Furthermore, input region designation unit 26 instructs display processing unit 28 to display, on the screen of display device 6, a selection screen for selecting a range of input variables in two-dimensional space using a rectangle or a lasso, or a selection screen for selecting the range of each input variable using a parallel coordinate plot. When the user performs a selection operation on the selection screen using the input device, input region designation unit 26 accepts the selection operation information and sets the selection operation information as an input data region in output upper and lower bound calculation unit 22.

[0043] The region proportion calculation unit 27 calculates the proportion of the region for each expected output determination result with respect to the input data region. For example, the region proportion calculation unit 27 may calculate the proportion using the hypervolume of the input data region. When the input data region is a hypercuboid, the region proportion calculation unit 27 may calculate the region proportion using the product of the lengths of the sides of the hypercuboid. Furthermore, the region proportion calculation unit 27 may calculate the region proportion for each expected output determination result with respect to the input data region using the number of data samples of a predetermined data set included in the region.

[0044] The area ratio calculation unit 27 may calculate the ratio of the area for each expected output determination result to the input data area using the number of generated data samples included in the area. The data samples may be generated from a uniform distribution, by perturbing data samples from a given dataset, from a data distribution trained on a given dataset, or by using a generative adversarial network (GAN) trained on a given dataset.

[0045] The sizes of the violation area, the pass area, and the unknown area are |S 違反 |,|S 合格 |,|S 不明 |, the size of the entire input area is |S 全体|=|S 違反 |+|S 合格 |+|S 不明 In this case, the area ratio calculation unit 27 calculates the area ratio by, for example, |S 違反 | / |S 全体 |, |S 合格 | / |S 全体 |, |S 不明 | / |S 全体 |, |S 違反 | / (|S 違反 |+|S 合格 |), or (|S 違反 |+|S 不明 |) / |S 全体 Calculate |.

[0046] The display processing unit 28 is a verification result output unit that outputs verification result information indicating the area ratio. The display processing unit 28 displays the verification result information on the display device 6. The display device 6 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display device.

[0047] The display processing unit 28 outputs display control information for displaying the verification result information to the display device 6. The display device 6 displays the verification result information based on the display control information. For example, the area ratio is displayed as a bar graph, a pie chart, a stacked bar graph, or by color shading. Figure 6A is a diagram showing an example of the display of the verification results, showing the area percentages in a bar graph. In Figure 6A, the area percentages are represented by the length of the bars. The percentage of pass areas is 32.3%, the percentage of unknown areas is 51.3%, and the percentage of non-compliance areas is 16.4%. 6B is a diagram showing an example of the display of the verification results, in which the percentage of the area is shown by the shade of color. In FIG. 6B, the percentage of the violation area is shown by the shade of color. For example, the greater the percentage of violation, the darker the color is displayed.

[0048] The display processing unit 28 may update the display of the area ratios that change according to the division of the input data area in real time. For example, the display processing unit 28 may monotonically decrease the display of the unknown areas or monotonically increase the display of the pass areas and the violation areas over time. Furthermore, the display processing unit 28 may display the change in the proportion of the area over time on the display device 6 as a line graph or animation.

[0049] Furthermore, the display processing unit 28 may display the specified input data region and the expected output together on the display device 6. For example, the display processing unit 28 displays the region ratio on the display device 6 as a parallel coordinate plot, a two-dimensional plot, or text. Figure 6C shows an example of the display of the verification results, where the area ratio is shown in parallel coordinate plot and text. In Figure 6C, the input variables x1, x2, . . . , x N The numerical range for each of these is shown by a rectangle, and the numerical range of the output data y of the inference model for the input data domain is shown by a rectangle. Furthermore, the display processing unit 28 calculates the input variables x1, x2, . . . , x N The numerical ranges of the respective values ​​and the numerical range of the output data y may be displayed in text form on the display device 6 as shown in FIG.

[0050] 1 shows the verification device 2 as having an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input region division unit 25, an input region designation unit 26, a region proportion calculation unit 27, and a display processing unit 28, but the configuration is not limited to this. The verification device 2 only needs to be able to determine the expected output of the inference model, divide the input data region to be input to the inference model, and output the determination results. Therefore, components other than the expected output determination unit 24, the input region division unit 25, and the display processing unit 28 may be provided in an external device accessible from the verification device 2.

[0051] Although the case where the input data area is a two-dimensional space has been described, the input data area is generally a high-dimensional space of two or more dimensions.

[0052] The inference device 3 is realized, for example, by a computer having a communication unit, a calculation unit, and a memory unit. The communication unit communicates with the inference model memory device 4 via a network using a wired signal line or wireless communication. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark).

[0053] The calculation unit controls the overall operation of the inference device 3. The calculation unit includes an inference model reading unit 31 and an inference unit 32. The calculation unit executes an information processing application for performing inference, thereby realizing various functions of the inference model reading unit 31 and the inference unit 32.

[0054] The storage unit stores information processing applications and information used for the calculation processing of the calculation unit. The storage unit is a storage device provided in a computer that functions as the inference device 3, and includes storage such as an HDD or SSD, or memory 104 of Figures 9A and 9B. Note that the storage unit may be provided outside the inference device 3 as long as it is accessible by the inference device 3.

[0055] The inference model reading unit 31 reads the inference model verified by the verification device 2 from the inference model storage device 4. The inference model read by the inference model reading unit 31 is set in the inference unit 32.

[0056] The inference unit 32 performs inference using the inference model read by the inference model reading unit 31. For example, the inference unit 32 may test a complex inference model. In this case, in order to expose the risk of occurrence of defects such as bugs in the complex inference model in pre-operation testing, the inference unit 32 sets the minimum properties that the inference model should satisfy as input data areas and expected outputs, and performs inference.

[0057] For example, the inference unit 32 infers the rent of an apartment building based on the input data area being within a five-minute walk from the station and being less than ten years old, and the expected output of the inference model for this input data area being a rent of 80,000 yen or more. The inference unit 32 also uses blood pressure of 135 or higher and LDL cholesterol of 140 or higher as an input data region, and infers whether or not an examination is necessary, assuming that the expected output of the inference model for this input data region is that an examination is necessary. The inference unit 32 may use a numerical range indicating products whose measurement error is larger than that of the abnormal product sample as an input data area, and may infer abnormality in the product by assuming as an expected output that all products whose measurement error is larger than that of the abnormal product sample are abnormal. If the percentage of violation areas is less than a specified value, the inference device 3 deploys the system to the real environment and starts operation.

[0058] The inference unit 32 may infer the fidelity of a surrogate model that approximates a complex inferred model. For example, the inference device 3 approximates the behavior during inference using a single decision tree model with high interpretability, and allows the user to recognize the behavior during inference. The verification device 2 sets the output of the original inference model and the proxy model to match as the expected output, and calculates the percentage of violation areas where the outputs of the original inference model and the proxy model do not match. The verification device 2 then regards the smallness of the violation area percentage as an indication of the fidelity of the proxy model, and displays it on the display device 6.

[0059] The inference of an existing rule-based system may be replaced with an inference model verified by the verification device 2. For example, the existing system is replaced with a machine learning model with high prediction accuracy. The verification device 2 sets the match between the output of the inference model and the existing system as the expected output, and calculates the percentage of violation areas where the output of the inference model and the existing system do not match. Then, the inference device 3 performs replacement with an inference model whose violation area ratio is less than a specified value. The verification device 2 may quantify the difference between two different versions of an inference model. For example, the difference between inference model A trained using data from a certain period and inference model B trained using data from another period may be quantified. The verification device 2 sets the expected output to be the same between inference model A and inference model B, and calculates the percentage of violation areas where the outputs of inference model A and inference model B do not match. If the percentage of violation areas is less than a specified value, the difference between model A and model B is considered small. This allows, for example, a system that continuously collects and learns data to update an inference model, to update the model only when the difference between the inference models is small. Furthermore, by presenting the violation areas, it is possible to know in what areas the models differ, making it possible to quickly identify and address environmental changes, such as data shifts.

[0060] Next, a verification method according to the first embodiment will be described. FIG. 7 is a flowchart showing the verification method according to the first embodiment. The inference model reading unit 21 reads the inference model from the inference model storage device 4 and sets it in the output upper and lower bound calculation unit 22, and the input domain designation unit 26 sets the input data domain in the output upper and lower bound calculation unit 22. The expected output designation unit 23 sets the expected output of the inference model to the expected output determination unit 24. This series of processes constitutes the processing of step ST1.

[0061] The expected output determination unit 24 clears the area list and adds the set input data area (step ST2). The area list is a list of data areas to be input to the inference model. The expected output determination unit 24 inputs the input data areas registered in the area list to the inference model.

[0062] Next, the expected output determination unit 24 performs an expected output determination process for each input data area in the area list (step ST3). Next, the expected output determination unit 24 determines whether there is an unknown area, i.e., an area that is neither a violation area where all of the output data of the inference model for the input data area is not the expected output, nor a pass area where all of the output data is the expected output (step ST4).

[0063] If an unknown region exists (step ST4; YES), the input region dividing unit 25 divides the unknown region (step ST5). The expected output determining unit 24 clears the region list and adds the divided region as a new input data region of the inference model (step ST6). Subsequently, the expected output determining unit 24 proceeds to step ST3 to determine the expected output of the new input data region, and repeats the subsequent processes.

[0064] If there are no more unknown areas (step ST4; NO), the area ratio calculation unit 27 calculates the area ratio for each expected output judgment result relative to the input data area, and then the display processing unit 28 displays verification result information indicating the area ratio on the display device 6 (step ST7). This allows the verification device 2 to present the proportion of the input data area for each judgment result of the expected output of the inference model.

[0065] FIG. 8 is a flowchart showing the expected output determination process, and shows the detailed process of step ST3 in FIG. The output upper and lower bound calculation unit 22 calculates the upper and lower bounds of the output data of the inference model for the input data domain (step ST1A). The output upper and lower bound calculation unit 22 sets the calculated upper and lower bounds of the output data in the expected output determination unit 24.

[0066] The expected output determination unit 24 determines whether the input data area corresponding to the output data defined by the upper and lower bounds of the inference model is a violation judgment, i.e., whether it is a violation, or whether it is pass or unknown, with respect to the expected output specified by the expected output designation unit 23 (step ST2A). Next, the expected output determination unit 24 stores the input data area and the corresponding expected output determination result in the determination result storage device 5 (step ST3A).

[0067] Next, the hardware configuration for realizing the functions of the verification device 2 will be described. The functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, expected output designation unit 23, expected output determination unit 24, input region division unit 25, input region designation unit 26, region proportion calculation unit 27, and display processing unit 28 provided in the verification device 2 are realized by processing circuits. That is, the verification device 2 includes a processing circuit for executing the processes from step ST1 to step ST7 shown in Fig. 7. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in memory.

[0068] Figure 9A is a block diagram showing a hardware configuration that realizes the functions of the verification device 2. Figure 9B is a block diagram showing a hardware configuration that executes software that realizes the functions of the verification device 2. In Figures 9A and 9B, the input interface 100 is an interface that relays data that the verification device 2 acquires from the inference model storage device 4 or the judgment result storage device 5. The output interface 101 is an interface that relays data that is output from the verification device 2 to the judgment result storage device 5.

[0069] 9A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28 provided in the verification device 2 may be realized by separate processing circuits, or these functions may be realized together by a single processing circuit.

[0070] 9B, the functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, expected output designation unit 23, expected output determination unit 24, input region division unit 25, input region designation unit 26, region ratio calculation unit 27, and display processing unit 28 provided in the verification device 2 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 104.

[0071] The processor 103 reads and executes programs stored in the memory 104, thereby realizing the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28 provided in the verification device 2. For example, the verification device 2 includes a memory 104 for storing programs that, when executed by the processor 103, result in the processing of steps ST1 to ST7 shown in FIG. 7 being executed. These programs cause a computer to execute the procedures or methods of the processing performed by the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28. The memory 104 may be a computer-readable storage medium storing a program for causing the computer to function as an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input area division unit 25, an input area designation unit 26, an area ratio calculation unit 27, and a display processing unit 28.

[0072] Memory 104 may be, for example, a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically-EPROM) (registered trademark), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD, etc.

[0073] Some of the functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, expected output designation unit 23, expected output determination unit 24, input region division unit 25, input region designation unit 26, region proportion calculation unit 27, and display processing unit 28 provided in the verification device 2 may be realized by dedicated hardware, and other functions may be realized by software or firmware. For example, the functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, expected output designation unit 23, and input region designation unit 26 may be realized by a processing circuit 102, which is dedicated hardware, and the functions of the expected output determination unit 24, input region division unit 25, region proportion calculation unit 27, and display processing unit 28 may be realized by a processor 103 reading and executing a program stored in memory 104. In this way, the processing circuit can realize the above functions by hardware, software, firmware, or a combination of these.

[0074] As described above, the verification device 2 according to the first embodiment includes an expected output determination unit 24 that determines whether all of the output data of the inference model for an input data region, which is a numerical range of data input to the inference model to be verified, is a violation judgment that is not the expected output; an input region division unit 25 that divides the input data region that is determined not to be a violation judgment and sets the divided region as a new input data region for the inference model; a region proportion calculation unit 27 that calculates the region proportion for each judgment result for the input data region; and a display processing unit 28 that displays verification result information indicating the region proportion on the display device 6. By outputting the region proportion for each judgment result of the expected output of the inference model, the verification device 2 can present the ratio of the input data region for each judgment result of the expected output of the inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated. In the conventional technology described in Patent Document 1, the input data domain is divided into multiple regions, and a satisfiability determiner determines whether any of the divided regions are satisfiable. If the satisfiability determiner determines that a region is satisfiable, i.e., that it exhibits satisfiability, at least one data sample in the divided region is judged to be a violation. Therefore, the conventional technology described in Patent Document 1 may propose a region that partially includes a data sample judged to be a violation. In contrast, the verification device 2 can present the proportion of the input data domain for each judgment result by dividing the region into regions where all of the output data of the inference model for the input data domain is judged not to be a violation, i.e., regions that include both a pass judgment and a violation judgment.

[0075] In the verification device 2 according to the first embodiment, the expected output determination unit 24 determines whether the output data of the inference model for the input data region is all the expected output (pass), whether the output data of the inference model for the input data region is all the expected output (fail), or whether the output data is none of the expected output (unclear). This allows the verification device 2 to present the proportion of the input data region for each determination result of the expected output of the inference model.

[0076] The verification device 2 according to the first embodiment includes an output upper / lower bound calculation unit 22 that calculates at least one of the upper and lower bounds of the output data of the inference model for the input data domain. The expected output determination unit 24 determines whether the output data domain indicated by the upper or lower bound of the output data of the inference model is a pass, fail, or unclear determination. This allows the verification device 2 to present the proportion of the input data domain for each determination result of the expected output of the inference model.

[0077] In the verification device 2 according to the first embodiment, the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees. The input region division unit 25 divides the region according to the branching condition of one of the decision trees included in the inference model. This allows the verification device 2 to determine the expected output of the inference model for each divided area.

[0078] The inference device 3 according to the first embodiment includes an inference model reading unit 31 that reads the inference model verified by the verification device 2, and an inference unit 32 that performs inference using the inference model. By presenting the proportion of the input data area for each judgment result of the expected output of the inference model, it is possible to prevent the degree of violation from being overestimated or underestimated, thereby enabling the inference device 3 to perform highly accurate inference.

[0079] The inference system 1 according to the first embodiment includes the above-described verification device 2 and inference device 3. An inference system can be provided that can present the proportion of the input data area for each judgment result of the expected output of the inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated.

[0080] The verification method according to the first embodiment includes the steps of: an expected output determination unit 24 determining whether all of the output data of the inference model for the input data region is a violation determination that is not the expected output; an input region division unit 25 dividing the input data region that is determined not to be a violation determination and setting the divided region as a new input data region for the inference model; an region proportion calculation unit 27 calculating the region proportion for each determination result for the input data region; and a display processing unit 28 displaying verification result information indicating the region proportion on the display device 6. By executing the above method, the verification device 2 can present the proportion of the input data region for each determination result of the expected output of the inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated.

[0081] Embodiment 2 FIG. 10 is a block diagram showing an example of the configuration of an inference system 1A according to embodiment 2. In FIG. 10, the inference system 1A is a system that performs inference using a verified inference model, and includes a verification device 2A, an inference device 3, an inference model storage device 4, a judgment result storage device 5, and a display device 6. The verification device 2A is a device that verifies an inference model that is a machine learning model, and the inference device 3 is a device that performs inference using the inference model. The inference system 1 is a system in which the verification device 2, the inference device 3, the inference model storage device 4, the judgment result storage device 5, and the display device 6 are connected via a network using wired signal lines or wireless communication. The network is a telecommunications line including the Internet, etc.

[0082] The verification device 2A presents the region proportion for each judgment result of the expected output of the inference model, as well as the region itself, data samples contained in the region, or summary information of multiple regions. The verification device 2A is realized, for example, by a computer equipped with a communication unit, a calculation unit, and a memory unit. The communication unit communicates with the inference model storage device 4 or the judgment result storage device 5 via a wired signal line or a wireless communication network. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark). The communication unit includes the input interface 100 and output interface 101 shown in Figures 9A and 9B.

[0083] The calculation unit controls the overall operation of the verification device 2A. The calculation unit includes an inference model reading unit 21, an output upper / lower bound calculation unit 22, an expected output specification unit 23, an expected output determination unit 24, an input region division unit 25, an input region specification unit 26, a region proportion calculation unit 27, a display processing unit 28, a region acquisition unit 29-1, a sample acquisition unit 29-2, and a region summarization unit 29-3. The calculation unit executes an information processing application for verifying the inference model, thereby realizing the various functions of the inference model reading unit 21, the output upper / lower bound calculation unit 22, the expected output specification unit 23, the expected output determination unit 24, the input region division unit 25, the input region specification unit 26, the region proportion calculation unit 27, the display processing unit 28, the region acquisition unit 29-1, the sample acquisition unit 29-2, and the region summarization unit 29-3. The calculation unit includes the processing circuit 102 of FIG. 9A and the processor 103 of FIG. 9B.

[0084] The storage unit stores information processing applications and information used for the arithmetic processing of the arithmetic unit. The storage unit is a storage device provided in a computer functioning as the verification device 2, and includes storage such as an HDD or SSD, or the memory 104 shown in FIGS. 9A and 9B. The storage unit may be external to the verification device 2A as long as it is accessible by the verification device 2A.

[0085] The inference model reading unit 21, output upper and lower bound calculation unit 22, expected output designation unit 23, expected output determination unit 24, input area division unit 25, input area designation unit 26, area ratio calculation unit 27 and display processing unit 28 are the same as those in Figure 1, so duplicate explanations will be omitted.

[0086] The area acquisition unit 29-1 acquires the input data area determined by the expected output determination unit 24 from the determination result storage device 5. The display processing unit 28 displays, on the display device 6, information indicating the area acquired by the area acquisition unit 29-1. 11 is a diagram showing the results of acquiring data areas of expected output determination results and an example of their display. As shown in FIG. 11, information indicating a plurality of input data areas is provided by input variables x1, x2, . . . , x NFor example, the area acquisition unit 29-1 acquires information associated with the violation expected output judgment result from among information indicating a plurality of input data areas corresponding to the pass, violation, and unknown expected output judgment results.

[0087] The display processing unit 28 displays a plurality of input data regions corresponding to the violation expected output determination results in a parallel coordinate plot on the display device 6. In FIG. 11, the input variables x1, x2, . . . , x that define any of the input data regions among the plurality of input data regions corresponding to the violation expected output determination results are N The numerical range for each of these is shown by a rectangle, and the numerical range of the output data y of the inference model for this input data domain is shown by a rectangle. Rectangle C of the output data y of the inference model is the range of the expected output, and rectangle D of the output data y of the inference model is the output range corresponding to the selected violation area. By referring to these, the user can understand the degree of violation in the input data.

[0088] The sample acquisition unit 29-2 acquires one or more data samples included in the region acquired by the region acquisition unit 29-1. The display processing unit 28 displays the data samples acquired by the sample acquisition unit 29-2 on the display device 6. 12 is a diagram showing the results of acquiring data samples from the data area of ​​the expected output judgment result and an example of displaying the same. For example, as shown in FIG. 12, the sample acquiring unit 29-2 acquires input variables x1, x2, . . . , x3, which are parameters of a relational expression indicating one input data area among a plurality of input data areas corresponding to the expected output judgment result of a violation. N The information indicating the lower and upper limits of each of the above is obtained as a data sample. 12, the data sample displayed on the display device 6 and corresponding to the violating expected output judgment result has, for example, an input variable x1 whose value is between the lower limit of 0.2 and the upper limit of 0.5. In this way, the display device 6 displays multiple pieces of input data contained in one input data area corresponding to the violating expected output judgment result. By referring to these, the user can understand the input data in the violating area.

[0089] As shown in FIG. 12, the sample acquisition unit 29-2 acquires data samples when each variable defining the input data area takes on a lower limit value or an upper limit value. In addition, the sample acquisition unit 29-2 may generate and acquire data samples from a uniform distribution within the input data region, may acquire training data samples included in the input data region, or may acquire data samples of the vertices of the input data region.

[0090] The region summarization unit 29-3 summarizes the multiple regions acquired by the region acquisition unit 29-1 into a smaller number of regions. The display processing unit 28 displays, on the display device 6, information indicating the regions summarized by the region summarization unit 29-3. If the inference model to be verified is a single decision tree or a decision tree ensemble model consisting of multiple decision trees, the sample acquisition unit 29-2 acquires one or more data samples included in the region acquired by the region acquisition unit 29-1. The region summarizing unit 29-3 learns a single decision tree using the data samples acquired by the sample acquiring unit 29-2, and summarizes the region using the leaf nodes of the learned decision tree.

[0091] For example, this shows a case where violation regions are summarized in the learning of an inference model consisting of a single decision tree. Data samples acquired from violation regions are assigned a teacher label "1," while data samples acquired from pass regions or unknown regions are assigned a teacher label "0." Data samples acquired from unknown regions may also be included in the teacher label "1." The region summarization unit 29-3 collects multiple decision paths from the learning result decision tree model whose output data has a teacher label of "1," and designates the corresponding regions as summary regions. It is also possible to set hyperparameters during decision tree learning so as to limit the depth of the decision tree, the maximum number of leaves, the number of data samples included in a leaf, and the like. The summary indicates the area with simple information, so that the user can easily grasp the input data area that corresponds to the expected output determination result.

[0092] The region summarization unit 29-3 may weight the data samples acquired by the sample acquisition unit 29-2 with sample weights proportional to the region size of the input data region from which the data samples were acquired, and perform learning. This allows multiple input data regions corresponding to expected output determination results to be presented in summary regions that take region size into consideration, allowing the user to easily grasp the input data region corresponding to the expected output determination result.

[0093] The region summarization unit 29-3 outputs the summarized region to the input region designation unit 26 as a new input data region. Using the expected output judgment result for the summarized region, the region proportion calculation unit 27 may calculate the proportion of regions in the summarized region that are pass, violate, or unknown. For example, even if the proportion of non-compliant regions in the original input data region is displayed as 0.8%, it is difficult to easily grasp the degree of non-compliance in multiple input data regions including this region. Therefore, by summarizing multiple input data regions and displaying that the proportion of non-compliant regions in the summarized region is 93.7%, the user can quantitatively understand the degree of non-compliance in the summarized region.

[0094] FIG. 13 is a schematic diagram showing an example of the display of the summarized area, and shows the area where summarization has been performed as a parallel category plot of the area ratio. In FIG. 13, the ratio of the area of any of pass, violation, or unknown is displayed as a strip-shaped category, and the display width of each category is proportional to the area size. The area defined by the input variable x5 where x5≦3.9 includes a pass area and an unknown area. On the other hand, the area defined by the input variable x5 where 3.9<x5 includes a pass area, an unknown area, and a violation area.

[0095] In the area defined by the input variable x5 where 3.9<x5, the area defined by the input variable x8 where x8≦-0.4 includes a pass area, an unknown area, and a violation area. On the other hand, the area defined by the input variable x8 where -0.4<x8 includes a pass area and an unknown area.

[0096] In the area defined by the input variable x8 where x8≦-0.4, the area defined by the input variable x5 where x5≦5.2 includes only a violation area. That is, the ratio of the violation area is 100%. On the other hand, the area defined by the input variable x5 where 5.2<x5 includes a pass area, an unknown area, and a violation area. The ratio of the violation area is 25%. By referring to this display, the user can easily understand the expected output determination result in the area where summarization has been performed.

[0097] As described above, the verification device 2A according to the second embodiment includes an area acquisition unit 29-1 that acquires the input data area determined by the expected output determination unit 24. The display processing unit 28 displays the information indicating the acquired area on the display device 6. Since the input data area for each determination result of the expected output of the inference model is displayed, the user can grasp the degree of violation in the input data.

[0098] The verification device 2A according to the second embodiment includes a sample acquisition unit 29-2 that acquires one or more data samples contained in the area acquired by the area acquisition unit 29-1. The display processing unit 28 displays the acquired data samples on the display device 6. Since the data samples contained in the input data area for each judgment result of the expected output of the inference model are displayed, the user can grasp the degree of violation in the input data for each data sample.

[0099] The verification device 2A according to the second embodiment includes an area summarization unit 29-3 that summarizes the multiple areas acquired by the area acquisition unit 29-1 into a smaller number of areas. The display processing unit 28 displays information indicating the summarized areas on the display device 6. By displaying the areas where multiple input data areas have been summarized into a smaller number for each judgment result of the expected output of the inference model, the user can easily grasp the degree of violation in the input data.

[0100] In the verification device 2A according to the second embodiment, the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees. It includes a sample acquisition unit 29-2 that acquires one or more data samples included in the acquired region. The region summarization unit 29-3 uses the acquired data samples to train a single decision tree and summarizes the region using the leaf nodes of the trained decision tree. Multiple input data regions for each judgment result of the expected output of the inference model are summarized into simple information, allowing the user to easily grasp the degree of violation in the input data.

[0101] It is possible to combine the embodiments, modify any of the components of the embodiments, or omit any of the components of the embodiments.

[0102] The verification device according to the present disclosure can be used, for example, to verify inference models for performing various inferences. [Explanation of symbols]

[0103] 1,1A inference system, 2,2A verification device, 3 inference device, 4 inference model storage device, 5 judgment result storage device, 6 display device, 21,31 inference model reading unit, 22 output upper and lower bound calculation unit, 23 expected output specification unit, 24 expected output judgment unit, 25 input region division unit, 26 input region specification unit, 27 region proportion calculation unit, 28 display processing unit, 29-1 region acquisition unit, 29-2 sample acquisition unit, 29-3 region summarization unit, 32 inference unit, 100 input interface, 101 output interface, 102 processing circuit, 103 processor, 104 memory.

Claims

1. an expected output determination unit that determines whether the output data of the inference model for an input data area, which is a numerical range of data, is a violation determination that is not an expected output; an input area division unit that divides an area of ​​the input data area that is determined not to be a violation judgment and sets the divided area as a new input data area of ​​the inference model; an area ratio calculation unit that calculates an area ratio for each determination result with respect to the input data area; a verification result output unit that outputs verification result information indicating the ratio of the area; A verification device characterized by:

2. The expected output determination unit determines whether the output data of the inference model for the input data area is a pass determination, in which all of the output data is the expected output; a violation determination, in which all of the output data of the inference model for the input data area is not the expected output; or an unclear determination, in which none of the above applies.

2. The verification device according to claim 1.

3. an output upper / lower bound calculation unit that calculates at least one of the upper and lower bounds of the output data of the inference model for the input data domain; The expected output determination unit determines one of the pass determination, the violation determination, and the unclear determination for an output data area indicated by an upper bound or a lower bound of the output data of the inference model.

3. The verification device according to claim 2.

4. The inference model to be verified is a decision tree ensemble model composed of a single decision tree or multiple decision trees, The input region division unit divides the region according to a branching condition of any of the decision trees included in the inference model.

4. The verification device according to claim 3.

5. an area acquisition unit that acquires the input data area determined by the expected output determination unit; The verification result output unit displays information indicating the acquired area on a display device.

2. The verification device according to claim 1.

6. a sample acquisition unit that acquires one or more data samples included in the region acquired by the region acquisition unit; The verification result output unit displays the acquired data sample on the display device.

6. The verification device according to claim 5.

7. a region summarizing unit that summarizes the plurality of regions acquired by the region acquisition unit into a smaller number of regions; The verification result output unit displays information indicating the region where the summary was performed on the display device.

6. The verification device according to claim 5.

8. The inference model to be verified is a decision tree ensemble model composed of a single decision tree or multiple decision trees, a sample acquisition unit configured to acquire one or more data samples included in the acquired region; The region summarization unit learns a single decision tree using the acquired data samples and performs region summarization using leaf nodes of the learned decision tree.

8. The verification device according to claim 7.

9. A verification device according to any one of claims 1 to 8; an inference device including an inference model reading unit that reads the inference model verified by the verification device, and an inference unit that performs inference using the inference model; An inference system characterized by:

10. A verification method using a verification device, an expected output determination unit determining whether the output data of the inference model for the input data domain, which is a numerical range of data, is a violation determination that is not the expected output; an input area division unit dividing an area of ​​the input data area that is determined not to be a violation judgment, and setting the divided area as a new input data area of ​​the inference model; an area ratio calculation unit calculating an area ratio for each determination result with respect to the input data area; a step in which the verification result output unit outputs verification result information indicating the ratio of the area; A verification method comprising:

11. On the computer, A process of determining whether the output data of the inference model for the input data area, which is the numerical range of the data, is a violation judgment that is not the expected output; A process of dividing an area of ​​the input data area that is determined not to be a violation judgment and setting the divided area as a new input data area of ​​the inference model; A process of calculating a ratio of an area for each determination result to the input data area; and outputting verification result information indicating the ratio of the area.

Images