Verification device, inference system, verification method and program

Abstract

To provide a verification device capable of presenting a ratio of an input data area of each determination result of an expectation output of an inference model.SOLUTION: A verification device (2) includes an expectation output determination part (24) for determining whether output data of an inference model to an input data area is a violation determination that is not an expectation output, an input area division part (25) for dividing an area determined not to be the violation determination in the input data area, and setting the divided area in a new input data area for the inference model, an area ratio calculation part (27) for calculating an area ratio of each determination result to the input data area, and a display processing part (28) for displaying verification result information showing the area ratio in a display device (6).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a verification device, an inference system, a verification method, and a program. [Background technology]

[0002] A technique for evaluating the validity of an inference model, which is a machine learning model, has been proposed. For example, Patent Document 1 presents a model output region for a data region input to a decision tree ensemble model, the region including at least one violating data sample that violates a verification property. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7059220 Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional technology described in Patent Document 1 presents a range that includes at least one violating data sample that is not the expected output of the inference model, but there is no guarantee that all data samples in the presented range are violating data samples. For this reason, the conventional technology has a problem that, for example, when violating data samples are not present in most areas within the presented range and violating data samples are present only in a very narrow area, the degree of violation is overestimated, making it difficult to appropriately deal with the violation.

[0005] The present disclosure is intended to solve the above problem, and aims to provide a verification device that can present the proportion of input data area for each judgment result of the expected output of an inference model. [Means for solving the problem]

[0006] The verification device according to the present disclosure includes an expected output judgment unit that judges whether the output data of an inference model for an input data area, which is a numerical range of data, is a violation judgment that is not the expected output, an input area division unit that divides an area of ​​the input data area that is judged not to be a violation judgment and sets the divided area as a new input data area of ​​the inference model, an area proportion calculation unit that calculates the area proportion for each judgment result for the input data area, and a verification result output unit that outputs verification result information indicating the area proportion. Effect of the Invention

[0007] According to the present disclosure, it is determined whether the output data of the inference model for the input data area is a violation judgment that is not the expected output, the area of ​​the input data area that is determined to be not a violation judgment is divided, the divided area is set as a new input data area of ​​the inference model, the area ratio for each judgment result for the input data area is calculated, and verification result information indicating the area ratio is output. By outputting the area ratio for each judgment result of the expected output of the inference model, the verification device according to the present disclosure can present the input data area ratio for each judgment result of the expected output of the inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of an inference system according to embodiment 1. FIG. [Diagram 2] 11 is a graph showing the determination results of the input data area and the expected output. [Diagram 3] 13 is a schematic diagram showing a division process of an input data area and an expected output determination result. FIG. [Figure 4] FIG. 13 is a schematic diagram showing an outline of expected output determination. [Diagram 5] FIG. 13 is a diagram showing an example of an expected output determination result. [Figure 6] 6A, 6B, and 6C are diagrams showing examples of displaying the verification results. [Figure 7]4 is a flowchart showing a verification method according to the first embodiment. [Figure 8] 13 is a flowchart showing an area determination process. [Figure 9] 9A and 9B are block diagrams showing a hardware configuration for implementing the functions of the verification device according to the first embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of the configuration of an inference system according to a second embodiment. [Figure 11] 13A and 13B are diagrams showing an example of the expected output determination result data area and its display. [Figure 12] 13A and 13B are diagrams showing the results of acquiring data samples from a data area of ​​expected output determination results and examples of their display. [Figure 13] FIG. 13 is a schematic diagram showing a display example of a region where a summary has been performed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Embodiment 1 FIG. 1 is a block diagram showing a configuration example of an inference system 1 according to a first embodiment. In FIG. 1, the inference system 1 is a system that performs inference using a verified inference model, and includes a verification device 2, an inference device 3, an inference model storage device 4, a judgment result storage device 5, and a display device 6. The verification device 2 is a device that verifies an inference model that is a machine learning model, and the inference device 3 is a device that performs inference using the inference model. The inference system 1 is a system in which the verification device 2, the inference device 3, the inference model storage device 4, the judgment result storage device 5, and the display device 6 are connected via a network using wired signal lines or wireless communication. The network is a telecommunication line including the Internet and the like.

[0010] The verification device 2 can prevent the degree of violation from being over- or underestimated by presenting the proportion of the input data area for each judgment result of the expected output of the inference model. For example, in the conventional technology described in Patent Document 1, a violation range that includes at least one violating data sample that is not the expected output of the inference model is presented as the verification result. Note that the expected output is a proposition that is expected to be true for the output data of the inference model. Figure 2 is a graph showing the input data domain and the judgment results of the expected output. The input data domain shown in Figure 2 is a data domain defined by input variables x1 and x3. Domain A shown in Figure 2 is a data domain that includes domain B. All data included in domain B is not the expected output of the inference model, that is, it is a violation data sample that does not satisfy the verification conditions, and domain B is a domain that includes only violation data samples.

[0011] In the conventional technology described in Patent Document 1, a region A may be proposed. There is no guarantee that all data samples in this region A are violations. For example, as shown in FIG. 2, there is a possibility that no violation data samples exist in most of the regions, and in this case, the degree of violation may be overestimated. Since there is a possibility that all data samples in area A are included in the violation range, the output data of the inference model when data in area A is input may not be the expected output, i.e., it may be determined to be a violation that does not satisfy the verification conditions. In this case, the inference of the output data of the inference model when data included in an area in area A where no violating data samples exist is input is actually the expected output, but the inference is withheld.

[0012] In addition, the output value of the inference model when data in area A is input may be overwritten to make it appear as if there is no violation. In this case, the output data of the inference model when data included in an area where no violating data sample exists is input may also be overwritten to a different value, even though it is the expected output.

[0013] In addition, there are cases where the inference model is retrained by adding data samples in area A so that there are no violations in the output values ​​of the inference model when data in area A is input. In this case, the inference behavior of the inference model when data in an area where no violating data samples exist in area A is input also changes significantly.

[0014] In the case where the test is passed if the hyper-volume of the violation range output from the inference model when data contained in area A is input is below a specified value, it is possible that the test will not pass even if the hyper-volume of the violation range is actually below the specified value if the input data area includes an area where no violation data samples exist.

[0015] Therefore, the verification device 2 according to the first embodiment presents the ratio of the input data area for each expected output judgment result of the inference model by outputting the ratio of the area for each expected output judgment result of the inference model. This allows the verification device 2 to suppress the occurrence of problems caused by overestimating or underestimating the degree of violation as described above.

[0016] Fig. 3 is a schematic diagram showing the division process of the input data area and the expected output judgment result, showing the input data area of ​​the inference model defined by the input variables x1 and x3. In Fig. 3, an input data area that includes only input data whose output data of the inference model is all the expected output is a pass area. An input data area that includes only input data whose output data of the inference model is all the non-expected output is a violation area. An input data area that is neither a pass area nor a violation area is an unknown area.

[0017] The verification device 2 inputs an input data region into the inference model, and determines whether all of the output data of the inference model is a violation judgment that is not the expected output, thereby calculating the ratio of the region for each judgment result to the input data region. As shown in the graph on the left side of Fig. 3, the ratio of pass regions is 23.9% and the ratio of unknown regions is 76.1% of the input data region to the inference model. Since the verification device 2 includes an unknown region, i.e., an input data region that was not judged to be a violation, the verification device 2 divides the unknown region as shown by the arrow in Fig. 3, and sets the divided region as a new input data region of the inference model.

[0018] Next, the verification device 2 sets each divided area of ​​the unknown area as a new input data area of ​​the inference model, and calculates the ratio of the area for each judgment result to the input data area by judging whether the output data of the inference model is a violation judgment that is not the expected output. For example, as shown in the graph in the center of Figure 3, the ratio of the areas at this time is 46.7% of the input data area to the inference model as a pass area, 42.5% as an unknown area, and 0.8% as a violation area. In this case, the verification device 2 judges that an unknown area is included, and further divides these unknown areas as shown by the arrows in Figure 3.

[0019] Next, the verification device 2 sets the newly divided unknown area as a new input data area of ​​the inference model, and calculates the ratio of the area for each judgment result to the input data area by judging whether all of the output data of the inference model is a violation judgment that is not the expected output. For example, as shown in the graph on the right side of FIG. 3, the ratio of the pass area is 98.3%, the ratio of the unknown area is 0%, and the ratio of the violation area is 1.7% in the input data area to the inference model. In this way, the verification device 2 can present the ratio of the area corresponding to the expected output judgment result in the input data area for the inference model, and can also specify the data numerical range of each area. This makes it possible to prevent the degree of violation from being overestimated or underestimated.

[0020] The inference model storage device 4 is a storage device that stores an inference model. From the perspective of the verification device 2, the inference model stored in the inference model storage device 4 is a machine learning model to be verified, and from the perspective of the inference device 3, it is a machine learning model for inference processing. The inference model storage device 4 is provided outside the verification device 2 and the inference device 3. Note that the inference model storage device 4 only needs to be accessible from the verification device 2 and the inference device 3, and may be a storage device provided in a computer that functions as the verification device 2 or the inference device 3.

[0021] The judgment result storage device 5 is a storage device that stores the expected output judgment result of the expected output judgment unit 24 included in the verification device 2. The expected output judgment result is a judgment result of whether or not all of the output data of the inference model for the input data area, which is the numerical range of the data, is a violation judgment that is not the expected output. Moreover, the judgment result storage device 5 is provided outside the verification device 2. Note that the judgment result storage device 5 only needs to be accessible from the verification device 2, and may be a storage device included in a computer that functions as the verification device 2.

[0022] The verification device 2 is realized, for example, by a computer including a communication unit, a calculation unit, and a storage unit. The communication unit communicates with the inference model storage device 4 or the judgment result storage device 5 via a network using a wired signal line or wireless communication. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark). The communication unit includes an input interface 100 and an output interface 101 in FIGS. 9A and 9B.

[0023] The calculation unit controls the overall operation of the verification device 2. The calculation unit includes an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output judgment unit 24, an input region division unit 25, an input region designation unit 26, a region proportion calculation unit 27, and a display processing unit 28. The calculation unit executes an information processing application for verifying the inference model, thereby realizing various functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output judgment unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28. The calculation unit includes a processing circuit 102 in FIG. 9A and a processor 103 in FIG. 9B.

[0024] The storage unit stores an information processing application and information used for the arithmetic processing of the arithmetic unit. The storage unit is a storage device provided in a computer functioning as the verification device 2, and includes a storage such as an HDD or SSD, or the memory 104 of FIGS. 9A and 9B. Note that the storage unit may be provided outside the verification device 2 as long as it is accessible by the verification device 2.

[0025] The inference model reading unit 21 reads the inference model to be verified from the inference model storage device 4. The inference model is an N-dimensional input X=(x1, x2, . . . , x N )∈R N For 1-dimensional output y∈R or M-dimensional output Y=(y1,y2,...,y M )∈R M where N and M are integers. For example, inference models include trained machine learning models, neural networks, decision tree models, decision tree ensemble models, support vector machines (SVMs), generalized linear models, generalized additive models (GAMs), Gaussian process regression models (GPRs), naive Bayes, and Gaussian mixture models (GMMs).

[0026] Neural networks include Multi-Layer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Transformers. Decision tree ensemble models include Random Forests or Gradient Boosting Trees. Generalized linear models include Linear Regression or Logistic Regression.

[0027] The output upper and lower bound calculation unit 22 calculates at least one of the upper and lower bounds of the output data of the inference model for the input data domain. In other words, the inference model outputs data such that the lower bound value≦output data≦upper bound value is always true. The upper bound is calculated, for example, as follows. When the inference model is a generalized linear model, the output upper and lower bound calculation unit 22 calculates the upper bound as the output value when a variable whose coefficient is a positive value is the upper limit value of the input data domain and a variable whose coefficient is a negative value is the lower limit value of the input data domain. When the inference model is a single decision tree model, the output upper and lower bound calculation unit 22 calculates, as the upper bound, the maximum output value among the set of leaf nodes that a data sample in the input data domain can reach. When the inference model is a decision tree ensemble model, the output upper and lower bound calculation unit 22 calculates, as the upper bound, the sum of the maximum output values ​​for each decision tree among the set of leaf nodes that a data sample in the input data domain can reach.

[0028] The method for calculating the lower bound is, for example, as follows. When the inference model is a generalized linear model, the output upper and lower bound calculation unit 22 calculates the lower bound as the output value when a variable whose coefficient is a positive value is the lower limit value of the input data domain and a variable whose coefficient is a negative value is the upper limit value of the input data domain. When the inference model is a single decision tree model, the output upper and lower bound calculation unit 22 calculates, as the lower bound, the minimum output value among the set of leaf nodes that a data sample in the input data domain can reach. When the inference model is a decision tree ensemble model, the output upper and lower bound calculation unit 22 calculates, as the lower bound value, the sum of the minimum output values for each decision tree among the set of leaf nodes that the data samples within the input data region can reach.

[0029] The expected output specifying unit 23 sets the expected output value of the inference model for the expected output determination unit 24. The expected output is a proposition that is expected to be true for the output data y of the inference model. For example, specifying 50 ≦ y < 80 as the range of the output value of regression, specifying 0.5 ≦ y as the range of the output score (output of the sigmoid function) of binary classification, specifying (y1 < y2) & (y3 < y2) as the output class of multi-class classification, specifying the output of the softmax function's inequality to output class 2 as the expected output, or specifying |y - y'| ≦ 0.1 which is the difference from the output y' = f'(X) of another inference model f', etc. can be cited.

[0030] For example, the user uses an input device (not shown in FIG. 1) to input the conditional expression or upper and lower limit values of the expected output in text. Thereby, the expected output specifying unit 23 sets the information indicating the conditional expression or upper and lower limit values input by the input device as the expected output for the expected output determination unit 24. Also, the expected output specifying unit 23 instructs the display processing unit 28 to display, on the screen of the display device 6, a selection screen for selecting a class ID with a drop-down or checkbox, or a selection screen for selecting the range of the output with a parallel coordinate plot. When the user performs a selection operation on the above selection screen using the above input device, the expected output specifying unit 23 receives the selection operation information and sets the selection operation information as the expected output for the expected output determination unit 24.

[0031] The expected output judgment unit 24 judges whether all of the output data of the inference model for the input data region is a violation judgment that is not the expected output. Fig. 4 is a schematic diagram showing an overview of expected output judgment, and shows a case where the expected output of regression is 50≦y<80. In Fig. 4, all output data y included in the numerical range of (lower bound of y≧50) & (upper bound of y<80) is also included in the expected output range 50≦y<80, so the input data region for the inference model when the output data y is output is not judged as a violation and is a pass region. Since all output data y included in the numerical range (upper bound of y < 50) | (lower bound of y ≧ 80) is not included in the expected output range 50≦y<80, the input data domain for the inference model when the output data y is output is a violation domain determined to be a violation. In the numerical range not included in either (lower bound of y≧50)&(upper bound of y<80) or (upper bound of y<50)|(lower bound of y≧80), the output data y that is not a violation is included. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0032] When the expected output of binary classification is 1, i.e., the output score y is 0.5≦y, all output scores y included in the score range where the lower bound of the output score y is ≧0.5 are included in the expected output range 0.5≦y, so the input data domain for the inference model when the output score y is output is the pass domain. All output scores y that fall within the numerical range where the upper bound of the output score y is <0.5 are not included in the expected output range 0.5≦y, so the input data domain for the inference model when the output score y is output is a violation domain. In a score range where neither the lower bound of the output score y≧0.5 nor the upper bound of the output score y<0.5, output data y that is not a violation determination is included. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0033] When the expected output of the three-class classification is class 2, i.e., (y1 < y2) & (y2 < y3), all output data y within the numerical range of (the upper bound of y1 < the lower bound of y2) & (the upper bound of y3 < the lower bound of y2) are included in the expected output range of (y1 < y2) & (y2 < y3). Therefore, the input data region for the inference model when the output data y is output is the qualified region. All output data y within the numerical range of (the lower bound of y1 ≥ the upper bound of y2) | (the lower bound of y3 ≥ the upper bound of y2) are not included in the expected output range of (y1 < y2) & (y2 < y3). Therefore, the input data region for the inference model when the output data y is output is the violation region. In the numerical range that is not included in either (the upper bound of y1 < the lower bound of y2) & (the upper bound of y3 < the lower bound of y2) or (the lower bound of y1 ≥ the upper bound of y2) | (the lower bound of y3 ≥ the upper bound of y2), there are output data y that are not violation judgments. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0034] When |y - y'| ≤ 0.1, which is the difference from the output y' = f'(X) of another inference model f', is the expected output range, all output data y within the numerical range of (the upper bound of y - the lower bound of y' ≤ 0.1) & (the upper bound of y' - the lower bound of y ≤ 0.1) are also included in the expected output range. Therefore, the input data region for the inference model when the output data y is output is the qualified region. All output data y within the numerical range of (the lower bound of y - the upper bound of y' > 0.1) | (the lower bound of y' - the upper bound of y > 0.1) are not included in the expected output range of |y - y'| ≤ 0.1 and are violation judgments. Therefore, the input data region for the inference model when the output data y is output is the violation region. In the numerical range that is not included in either (the upper bound of y - the lower bound of y' ≤ 0.1) & (the upper bound of y' - the lower bound of y ≤ 0.1) or (the lower bound of y - the upper bound of y' > 0.1) | (the lower bound of y' - the upper bound of y > 0.1), there are output data y that are violation judgments that are not the expected output. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0035] The determination result information of the expected output determination unit 24 is stored in the determination result storage device 5. 5 is a diagram showing an example of the expected output judgment result, and shows the stored contents of the judgment result storage device 5. As shown in FIG. 5, the judgment result information includes an input data area and an expected output judgment result for each input data. The input data area includes input variables x1, x2, . . ., x N The data area is defined by the lower and upper limits of the respective data. The expected output judgment result is pass, violation, or unknown. Although input data corresponding to all judgment results are shown, the judgment result storage device 5 may store only specific judgment results as expected output judgment results. For example, it may store only input data for which the expected output judgment result is a violation.

[0036] The input region division unit 25 divides a region of the input data region that is determined not to be a violation judgment, and sets the divided region as a new input data region of the inference model. For example, the input region division unit 25 may divide the input data region at the branch of a decision tree included in the inference model. In this case, the input region division unit 25 divides the input data region into overlapping regions of the data region corresponding to each leaf node of the decision tree and the input data region. At this time, the input region division unit 25 may ignore leaf nodes that cannot be reached in consideration of the constraints of the input data region, or may stop the division at an intermediate branch rather than at a terminal leaf node.

[0037] The input region division unit 25 may also divide the input data region randomly. For example, the input region division unit 25 randomly selects an input variable and determines a division point within the upper and lower limits of the input variable. Furthermore, the input region division unit 25 may divide the input data region based on the midpoint of the upper and lower limits generated from a uniform distribution, the average value, the median, or a quantile of the learning data. The input domain division unit 25 may weight the selection of input variables to be divided according to the importance of the features of the inference model.

[0038] Furthermore, the input region division unit 25 may divide the input data region into a plurality of division candidates and adopt the division that provides the best index after division. For example, the input region division unit 25 adopts the division that provides the smallest difference between the upper and lower bounds of the inference model output for the divided region. Alternatively, the input region division unit 25 may adopt a division that minimizes the sum of the sizes of the unknown regions after division.

[0039] The input region division unit 25 may continue the division recursively until there are no more unknown regions, or may discontinue the division midway. For example, the input region division unit 25 stops the division when the processing time is equal to or exceeds a specified value, the number of divided regions is equal to or exceeds a specified value, the size of the unknown region is less than a specified value, or the size of the unknown region is smaller than the size of the violation region.

[0040] The input domain designation unit 26 sets an input data domain to the output upper and lower bound calculation unit 22. The input data domain is a data domain in an N-dimensional space that is input to the inference model to be verified. N is an integer. For example, the input data domain is table data defined by an input variable x. The input data domain may be a feature extracted from an image, video, text, audio, graph, or time-series data using a neural network or the like.

[0041] The input data domain is, for example, S={X=(x1,x2,...,x N )∈R N |-3.5≦x1≦5.2,···,15≦x N ≦40} or S = {X∈R N ||X| 2 ≦r 2}, or S=S1∩S2 representing the intersection of multiple domains, or S=S1∪S2 representing the union of multiple domains.

[0042] For example, a user inputs a conditional expression or upper and lower limit values ​​of a hyperrectangle in the input data region in text using an input device not shown in FIG. 1. As a result, the input region designation unit 26 sets information indicating the conditional expression or upper and lower limit values ​​of the hyperrectangle input by the input device in the output upper and lower bound calculation unit 22 as an input data region. The input region designation unit 26 also instructs the display processing unit 28 to display on the screen of the display device 6 a selection screen for selecting the range of an input variable in a two-dimensional space with a rectangle or a lasso, or a selection screen for selecting the range of each input variable in a parallel coordinate plot. When the user performs a selection operation on the selection screen using the input device, the input region designation unit 26 accepts the selection operation information and sets the selection operation information in the output upper and lower bound calculation unit 22 as an input data region.

[0043] The area ratio calculation unit 27 calculates the ratio of the area for each expected output judgment result for the input data area. For example, the area ratio calculation unit 27 may calculate the ratio using the hypervolume of the input data area. When the input data area is a hypercuboid, the area ratio calculation unit 27 may calculate the area ratio using the product of the lengths of the sides of the hypercuboid. Furthermore, the area ratio calculation unit 27 may calculate the area ratio for each expected output judgment result for the input data area using the number of data samples of a predetermined data set included in the area.

[0044] The area ratio calculation section 27 may calculate the ratio of the area for each expected output determination result to the input data area by using the number of generated data samples included in the area. The data samples may be generated from a uniform distribution, by perturbing data samples from a given dataset, from a data distribution trained on a given dataset, or using a generative adversarial network (GAN) trained on a given dataset.

[0045] The sizes of the violation area, pass area, and unknown area are |S 違反 |,|S 合格 |,|S 不明 |, the size of the entire input area is |S 全体|=|S 違反 |+|S 合格 |+|S 不明 In this case, the area ratio calculation unit 27 calculates the area ratio by, for example, |S 違反 | / |S 全体 |, |S 合格 | / |S 全体 |, |S 不明 | / |S 全体 |, |S 違反 | / (|S 違反 |+|S 合格 |), or (|S 違反 |+|S 不明 |) / |S 全体 Calculate |.

[0046] The display processing unit 28 is a verification result output unit that outputs verification result information indicating the area ratio. The display processing unit 28 displays the verification result information on the display device 6. The display device 6 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display device.

[0047] The display processing unit 28 outputs display control information for displaying the verification result information to the display device 6. The display device 6 displays the verification result information based on the display control information. For example, the area ratio is displayed as a bar graph, a pie chart, a stacked bar graph, or by color shading. FIG. 6A is a diagram showing an example of a display of the verification result, and shows the area ratio in a bar graph. In FIG. 6A, the area ratio is represented by the length of the bar. The percentage of the pass area is 32.3%, the percentage of the unknown area is 51.3%, and the percentage of the violation area is 16.4%. Fig. 6B is a diagram showing an example of the display of the verification result, in which the proportion of the area is shown by the shade of color. In Fig. 6B, the proportion of the violation area is shown by the shade of color. For example, the greater the proportion of the violation, the darker the color is displayed.

[0048] The display processing unit 28 may update in real time the display of the area ratio that changes according to the division of the input data area. For example, the display processing unit 28 may monotonically decrease the display of the unknown area or monotonically increase the display of the pass area and the violation area with the passage of time. Furthermore, the display processing unit 28 may display the change in the ratio of the area over time on the display device 6 as a line graph or animation.

[0049] Furthermore, the display processing unit 28 may display the specified input data region and the expected output together on the display device 6. For example, the display processing unit 28 displays the ratio of the region on the display device 6 as a parallel coordinate plot, a two-dimensional plot, or text. FIG. 6C shows an example of the display of the verification results, in which the proportion of the region is shown in parallel coordinate plot and text. In FIG. 6C, input variables x1, x2, . . . , x N The numerical range of each of the above is shown by a rectangle, and the numerical range of the output data y of the inference model for the input data domain is shown by a rectangle. Furthermore, the display processing unit 28 displays the input variables x1, x2, . . . , x N The numerical ranges of each of the above and the numerical range of the output data y may be displayed in text form on the display device 6 as shown in FIG.

[0050] 1 shows a configuration in which the verification device 2 includes an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input region division unit 25, an input region designation unit 26, a region ratio calculation unit 27, and a display processing unit 28, but the configuration is not limited to this. The verification device 2 only needs to determine the expected output of the inference model, divide the input data region to be input to the inference model, and output the determination result. Therefore, the components other than the expected output determination unit 24, the input region division unit 25, and the display processing unit 28 may be included in an external device accessible from the verification device 2.

[0051] Although the input data domain is a domain in a two-dimensional space in the above embodiment, the input data domain is generally a domain in a high-dimensional space having two or more dimensions.

[0052] The inference device 3 is realized by, for example, a computer including a communication unit, a calculation unit, and a storage unit. The communication unit communicates with the inference model storage device 4 via a network using a wired signal line or wireless communication. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark).

[0053] The calculation unit controls the overall operation of the inference device 3. The calculation unit includes an inference model reading unit 31 and an inference unit 32. The calculation unit executes an information processing application for performing inference, thereby realizing various functions of the inference model reading unit 31 and the inference unit 32.

[0054] The storage unit stores an information processing application and information used for the arithmetic processing of the arithmetic unit. The storage unit is a storage device provided in a computer functioning as the inference device 3, and includes a storage such as an HDD or SSD, or memory 104 of Figures 9A and 9B. Note that the storage unit may be provided outside the inference device 3 as long as it is accessible by the inference device 3.

[0055] The inference model reading unit 31 reads the inference model verified by the verification device 2 from the inference model storage device 4. The inference model read by the inference model reading unit 31 is set in the inference unit 32.

[0056] The inference unit 32 performs inference using the inference model read by the inference model reading unit 31. For example, the inference unit 32 may perform testing of a complex inference model. In this case, in order to expose the risk of occurrence of defects such as bugs in the complex inference model in a test before operation, the inference unit 32 sets properties that are to be satisfied as the minimum input data area and expected output in the inference model and performs inference.

[0057] For example, the inference unit 32 infers the rent for an apartment based on the input data range being within a 5-minute walk from the station and being less than 10 years old, and the expected output of the inference model for this input data range being a rent of 80,000 yen or more. In addition, the inference unit 32 takes blood pressure of 135 or more and LDL cholesterol of 140 or more as the input data region, and infers whether or not an examination is necessary, assuming that the expected output of the inference model for this input data region is that an examination is necessary. The inference unit 32 may take a numerical range indicating products whose measurement error is larger than the abnormal product sample as an input data area, and may infer an abnormality in the product as an expected output that all products whose measurement error is larger than the abnormal product sample are abnormal. If the ratio of violation areas is less than a specified value, the inference device 3 deploys the solution in the real environment and starts operation.

[0058] The inference unit 32 may infer the fidelity of a surrogate model that approximates a complex inference model. For example, the inference device 3 approximates the inference behavior with a single decision tree model with high interpretability and allows the user to recognize the behavior during inference. The verification device 2 sets the output of the original inference model and the surrogate model to match each other as an expected output, and calculates the percentage of violation areas where the outputs of the original inference model and the surrogate model do not match. The verification device 2 then regards the smallness of the violation area percentage as the fidelity of the surrogate model and displays it on the display device 6.

[0059] The inference of an existing rule-based system may be replaced with an inference model verified by the verification device 2. For example, the existing system is replaced with a machine learning model with high prediction accuracy. The verification device 2 sets the match between the output of the inference model and the existing system as the expected output, and calculates the percentage of violation areas where the output of the inference model and the existing system do not match. Then, the inference device 3 performs replacement with an inference model whose violation area ratio is less than a specified value. The verification device 2 may quantify the difference between two inference models with different versions. For example, the difference between an inference model A trained on data for a certain period and an inference model B trained on data for another period is quantified. The verification device 2 sets the output of the inference model A and the inference model B to match as an expected output, and calculates the percentage of violation areas where the outputs of the inference model A and the inference model B do not match. If the violation area percentage is less than a specified value, the difference between the model A and the model B is considered to be small. This allows, for example, in a system that continuously collects and learns data to update an inference model, to update the model only when the difference between the inference models is small. Also, by presenting the violation area, it is possible to know in what areas the models differ, and it is possible to quickly detect and deal with environmental changes such as data shifts.

[0060] Next, a verification method according to the first embodiment will be described. FIG. 7 is a flowchart showing the verification method according to the first embodiment. The inference model reading unit 21 reads the inference model from the inference model storage device 4 and sets it in the output upper and lower bound calculation unit 22, and the input domain designation unit 26 sets the input data domain in the output upper and lower bound calculation unit 22. The expected output designation unit 23 sets the expected output of the inference model to the expected output determination unit 24. This series of processes constitutes step ST1.

[0061] The expected output determination unit 24 clears the area list and adds the set input data area (step ST2). The area list is a list of data areas to be input to the inference model. The expected output determination unit 24 inputs the input data area registered in the area list to the inference model.

[0062] Next, the expected output determination unit 24 performs an expected output determination process for each input data area in the area list (step ST3). Next, the expected output determination unit 24 determines whether there is an unknown area, that is, an area that is neither a violation area in which all of the output data of the inference model for the input data area is not the expected output, nor a pass area in which all of the output data is the expected output (step ST4).

[0063] If an unknown region exists (step ST4; YES), the input region division unit 25 divides the unknown region (step ST5). The expected output determination unit 24 clears the region list and adds the divided region as a new input data region of the inference model (step ST6). Next, the expected output determination unit 24 proceeds to step ST3 to perform expected output determination for the new input data region, and repeats the subsequent processes.

[0064] If there are no more unknown areas (step ST4; NO), the area ratio calculation unit 27 calculates the area ratio for each expected output judgment result relative to the input data area, and then the display processing unit 28 displays verification result information indicating the area ratio on the display device 6 (step ST7). This enables the verification device 2 to present the proportion of the input data area for each judgment result of the expected output of the inference model.

[0065] FIG. 8 is a flowchart showing the expected output determination process, and shows the detailed process of step ST3 in FIG. The output upper and lower bound calculation unit 22 calculates the upper and lower bounds of the output data of the inference model for the input data domain (step ST1A). The output upper and lower bound calculation unit 22 sets the calculated upper and lower bounds of the output data in the expected output determination unit 24.

[0066] The expected output determination unit 24 determines whether the input data area corresponding to the output data defined by the upper and lower bounds of the inference model is a violation judgment for the expected output specified by the expected output designation unit 23, i.e., whether it is a violation, or pass or unknown (step ST2A). Next, the expected output determination unit 24 stores the input data area and the corresponding expected output determination result in the determination result storage device 5 (step ST3A).

[0067] Next, a hardware configuration for realizing the functions of the verification device 2 will be described. The functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region ratio calculation unit 27, and the display processing unit 28 provided in the verification device 2 are realized by a processing circuit. That is, the verification device 2 includes a processing circuit for executing the processes from step ST1 to step ST7 shown in Fig. 7. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in a memory.

[0068] Fig. 9A is a block diagram showing a hardware configuration for realizing the functions of the verification device 2. Fig. 9B is a block diagram showing a hardware configuration for executing software for realizing the functions of the verification device 2. In Figs. 9A and 9B, an input interface 100 is an interface that relays data that the verification device 2 acquires from the inference model storage device 4 or the judgment result storage device 5. An output interface 101 is an interface that relays data that is output from the verification device 2 to the judgment result storage device 5.

[0069] 9A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these. The functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region ratio calculation unit 27, and the display processing unit 28 included in the verification device 2 may be realized by separate processing circuits, or these functions may be realized together by one processing circuit.

[0070] 9B, the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output judgment unit 24, the input area division unit 25, the input area designation unit 26, the area ratio calculation unit 27, and the display processing unit 28 included in the verification device 2 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 104.

[0071] The processor 103 reads out and executes the programs stored in the memory 104, thereby realizing the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output judgment unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28 of the verification device 2. For example, the verification device 2 includes a memory 104 for storing a program that, when executed by the processor 103, results in the processing of steps ST1 to ST7 shown in FIG. 7 being executed. These programs cause a computer to execute the procedures or methods of the processing performed by the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output judgment unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28. The memory 104 may be a computer-readable storage medium storing programs for causing a computer to function as an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input area division unit 25, an input area designation unit 26, an area ratio calculation unit 27 and a display processing unit 28.

[0072] Memory 104 may be, for example, a non-volatile or volatile semiconductor memory such as a Random Access Memory (RAM), a Read Only Memory (ROM), a flash memory, an Erasable Programmable Read Only Memory (EPROM), an Electrically-EPROM (EEPROM) (registered trademark), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD, etc.

[0073] Some of the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output judgment unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28 included in the verification device 2 may be realized by dedicated hardware, and the other parts may be realized by software or firmware. For example, the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, and the input region designation unit 26 may be realized by a processing circuit 102 which is dedicated hardware, and the functions of the expected output judgment unit 24, the input region division unit 25, the region proportion calculation unit 27, and the display processing unit 28 may be realized by the processor 103 reading and executing a program stored in the memory 104. In this way, the processing circuit can realize the above functions by hardware, software, firmware, or a combination of these.

[0074] As described above, the verification device 2 according to the first embodiment includes an expected output determination unit 24 that determines whether all of the output data of the inference model for the input data area, which is the numerical range of data to be input to the inference model to be verified, is a violation determination that is not the expected output, an input area division unit 25 that divides the area of ​​the input data area that is determined not to be a violation determination and sets the divided area as a new input data area of ​​the inference model, an area ratio calculation unit 27 that calculates the area ratio for each judgment result for the input data area, and a display processing unit 28 that displays verification result information indicating the area ratio on the display device 6. By outputting the area ratio for each judgment result of the expected output of the inference model, the verification device 2 can present the input data area ratio for each judgment result of the expected output of the inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated. In the conventional technology described in the above-mentioned Patent Document 1, the input data area is divided into a plurality of areas, and a satisfiability determiner determines whether there is any satisfiability in the divided areas. When the satisfiability determiner determines that there is satisfiability, that is, that the satisfiability is indicated, at least one data sample in the divided area is determined to be a violation. Therefore, the conventional technology described in Patent Document 1 may propose an area that partially includes a data sample determined to be a violation. In response to this, the verification device 2 is capable of presenting the proportion of the input data area for each judgment result by dividing an area in which all of the output data of the inference model for the input data area is determined not to be a violation judgment, that is, an area including those determined to be passed and those determined to be violated.

[0075] In the verification device 2 according to the first embodiment, the expected output judgment unit 24 judges whether the result is a pass judgment, in which all of the output data of the inference model for the input data area are the expected output, a violation judgment, in which all of the output data of the inference model for the input data area are not the expected output, or an unknown judgment, in which none of the above applies. This enables the verification device 2 to present the proportion of the input data area for each judgment result of the expected output of the inference model.

[0076] The verification device 2 according to the first embodiment includes an output upper / lower bound calculation unit 22 that calculates at least one of the upper and lower bounds of the output data of the inference model for the input data domain. The expected output determination unit 24 determines whether the output data domain indicated by the upper or lower bound of the output data of the inference model is passed, violated, or unknown. This allows the verification device 2 to present the proportion of the input data domain for each determination result of the expected output of the inference model.

[0077] In the verification device 2 according to the first embodiment, the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees. The input region division unit 25 divides the region according to the branching condition of any of the decision trees included in the inference model. This enables the verification device 2 to determine the expected output of the inference model for each divided area.

[0078] The inference device 3 according to the first embodiment includes an inference model reading unit 31 that reads the inference model verified by the verification device 2, and an inference unit 32 that performs inference using the inference model. By presenting the proportion of the input data area for each judgment result of the expected output of the inference model, it is possible to prevent the degree of violation from being overestimated or underestimated, thereby enabling the inference device 3 to perform highly accurate inference.

[0079] The inference system 1 according to the first embodiment includes the above-mentioned verification device 2 and inference device 3. It is possible to provide an inference system that can present the ratio of the input data area for each judgment result of the expected output of the inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated.

[0080] The verification method according to the first embodiment includes a step in which the expected output determination unit 24 determines whether all of the output data of the inference model for the input data area are violation determinations that are not expected outputs, a step in which the input area division unit 25 divides the areas of the input data area that are determined not to be violation determinations and sets the divided areas as new input data areas of the inference model, a step in which the area ratio calculation unit 27 calculates the area ratio for each determination result for the input data area, and a step in which the display processing unit 28 displays verification result information indicating the area ratio on the display device 6. The verification device 2 executes the above method, so that the ratio of the input data area for each determination result of the expected output of the inference model can be presented. This makes it possible to prevent the degree of violation from being overestimated or underestimated.

[0081] Embodiment 2 FIG. 10 is a block diagram showing a configuration example of an inference system 1A according to the second embodiment. In FIG. 10, the inference system 1A is a system that performs inference using a verified inference model, and includes a verification device 2A, an inference device 3, an inference model storage device 4, a judgment result storage device 5, and a display device 6. The verification device 2A is a device that verifies an inference model that is a machine learning model, and the inference device 3 is a device that performs inference using the inference model. The inference system 1 is a system in which the verification device 2, the inference device 3, the inference model storage device 4, the judgment result storage device 5, and the display device 6 are connected via a network using wired signal lines or wireless communication. The network is a telecommunication line including the Internet and the like.

[0082] The verification device 2A presents the area ratio for each judgment result of the expected output of the inference model, as well as the area itself, the data samples contained in the area, or summary information of multiple areas. The verification device 2A is realized by, for example, a computer equipped with a communication unit, a calculation unit, and a storage unit. The communication unit communicates with the inference model storage device 4 or the judgment result storage device 5 via a network using a wired signal line or wireless communication. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark). The communication unit includes the input interface 100 and the output interface 101 of FIG. 9A and FIG. 9B.

[0083] The calculation unit controls the overall operation of the verification device 2A. The calculation unit includes an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output judgment unit 24, an input region division unit 25, an input region designation unit 26, a region ratio calculation unit 27, a display processing unit 28, a region acquisition unit 29-1, a sample acquisition unit 29-2, and a region summarization unit 29-3. The calculation unit executes an information processing application for verifying the inference model, thereby realizing various functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output judgment unit 24, the input region division unit 25, the input region designation unit 26, the region ratio calculation unit 27, the display processing unit 28, the region acquisition unit 29-1, the sample acquisition unit 29-2, and the region summarization unit 29-3. The calculation unit includes a processing circuit 102 in FIG. 9A and a processor 103 in FIG. 9B.

[0084] The storage unit stores an information processing application and information used for the arithmetic processing of the arithmetic unit. The storage unit is a storage device provided in a computer functioning as the verification device 2, and includes a storage such as an HDD or SSD, or the memory 104 of Fig. 9A and Fig. 9B. Note that the storage unit may be provided outside the verification device 2A as long as it is accessible by the verification device 2A.

[0085] In addition, the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output judgment unit 24, the input area division unit 25, the input area designation unit 26, the area ratio calculation unit 27 and the display processing unit 28 are the same as those in Figure 1, so duplicated explanations will be omitted.

[0086] The area acquisition unit 29-1 acquires the input data area determined by the expected output determination unit 24 from the determination result storage device 5. The display processing unit 28 displays on the display device 6 information indicating the area acquired by the area acquisition unit 29-1. 11 is a diagram showing an example of the display of the data area of ​​the expected output judgment result. As shown in FIG. 11, the information indicating a plurality of input data areas includes input variables x1, x2, . . . , x NFor example, the area acquisition unit 29-1 acquires information associated with the expected output judgment result of violation from among information indicating a plurality of input data areas corresponding to the expected output judgment results of pass, violation, and unknown.

[0087] The display processing unit 28 displays a plurality of input data regions corresponding to the violation expected output judgment results in a parallel coordinate plot on the display device 6. In FIG. 11, input variables x1, x2, . . . , x that define any of the input data regions corresponding to the violation expected output judgment results are N The numerical range for each of these is shown by a rectangle, and the numerical range of the output data y of the inference model for this input data domain is shown by a rectangle. Rectangle C of the output data y of the inference model is the range of the expected output, and rectangle D of the output data y of the inference model is the output range corresponding to the selected violation area. By referring to these, the user can understand the degree of violation in the input data.

[0088] The sample acquisition unit 29-2 acquires one or more data samples included in the region acquired by the region acquisition unit 29-1. The display processing unit 28 displays on the display device 6 the data samples acquired by the sample acquisition unit 29-2. 12 is a diagram showing an example of the display of a data sample obtained from a data area of ​​an expected output judgment result. For example, as shown in FIG. 12, the sample acquisition unit 29-2 acquires input variables x1, x2, . . . , x3, which are parameters of a relational expression indicating one input data area among a plurality of input data areas corresponding to a violation expected output judgment result. N The lower and upper limits of each of the above are obtained as data samples. 12, the data sample corresponding to the violating expected output judgment result displayed on the display device 6 has, for example, an input variable x1 having a value between the lower limit of 0.2 and the upper limit of 0.5. In this manner, the display device 6 displays a plurality of input data contained in one input data area corresponding to the violating expected output judgment result. By referring to these, the user can grasp the input data in the violating area.

[0089] As shown in FIG. 12, the sample acquisition unit 29-2 acquires a data sample when each variable defining the input data area takes a lower limit value or an upper limit value. In addition, the sample acquisition unit 29-2 may generate and acquire data samples from a uniform distribution within the input data domain, may acquire learning data samples included in the input data domain, or may acquire data samples of the vertices of the input data domain.

[0090] The region summarization unit 29-3 summarizes the multiple regions acquired by the region acquisition unit 29-1 into a smaller number of regions. The display processing unit 28 displays on the display device 6 information indicating the regions summarized by the region summarization unit 29-3. When the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees, the sample acquisition unit 29-2 acquires one or more data samples included in the area acquired by the area acquisition unit 29-1. The region summarization unit 29-3 learns a single decision tree using the data samples acquired by the sample acquisition unit 29-2, and summarizes the region using the leaf nodes of the learned decision tree.

[0091] For example, a case where a violation region is summarized in the learning of an inference model composed of a single decision tree is shown. A teacher label "1" is assigned to a data sample obtained from a violation region, and a teacher label "0" is assigned to a data sample obtained from a pass region or an unknown region. Also, a teacher label "1" may be included in a data sample obtained from an unknown region. The region summarization unit 29-3 collects multiple decision paths in the decision tree model of the learning result, in which the teacher label of the output data is "1", and sets the corresponding region as a summary region. In addition, hyperparameters for decision tree learning may be set so as to limit the depth of the decision tree, the maximum number of leaves, the number of data samples contained in a leaf, and the like. Since the area is shown by the summary using simple information, the user can easily grasp the input data area that corresponds to the expected output determination result.

[0092] The region summarization unit 29-3 may weight the data samples acquired by the sample acquisition unit 29-2 with sample weights proportional to the region size of the input data region from which the data samples are acquired, and perform learning. This allows multiple input data regions corresponding to expected output judgment results to be presented in a summary region that takes the region size into consideration, allowing the user to easily grasp the input data region corresponding to the expected output judgment result.

[0093] The region summarization unit 29-3 outputs the summarized region to the input region designation unit 26 as a new input data region. Using the expected output judgment result for the summarized region, the region ratio calculation unit 27 may calculate the ratio of the region that is either pass, violation, or unknown for the summarized region. For example, even if the ratio of violation regions in the original input data region is displayed as 0.8%, the degree of violation in multiple input data regions including this region cannot be easily grasped. Therefore, by summarizing multiple input data regions and displaying that the ratio of violation regions in the summarized region is 93.7%, the user can quantitatively understand the degree of violation in the summarized region.

[0094] FIG. 13 is a schematic diagram showing an example of the display of the summarized area, in which the area where summarization has been performed is shown as a parallel category plot of the area ratio. In FIG. 13, the ratio of the area of either pass, violation, or unknown is displayed as a band-shaped category, and the display width of each category is proportional to the area size. The area defined by the input variable x5 where x5 ≦ 3.9 includes a pass area and an unknown area. On the other hand, the area defined by the input variable x5 where 3.9 < x5 includes a pass area, an unknown area, and a violation area.

[0095] In the area defined by the input variable x5 where 3.9 < x5, the area defined by the input variable x8 where x8 ≦ -0.4 includes a pass area, an unknown area, and a violation area. On the other hand, the area defined by the input variable x8 where -0.4 < x8 includes a pass area and an unknown area.

[0096] In the area defined by the input variable x8 where x8 ≦ -0.4, the area defined by the input variable x5 where x5 ≦ 5.2 includes only a violation area. That is, the ratio of the violation area is 100%. On the other hand, the area defined by the input variable x5 where 5.2 < x5 includes a pass area, an unknown area, and a violation area. The ratio of the violation area is 25%. By referring to this display, the user can easily understand the expected output determination result in the area where summarization has been performed.

[0097] As described above, the verification device 2A according to the second embodiment includes an area acquisition unit 29-1 that acquires the input data area determined by the expected output determination unit 24. The display processing unit 28 displays the information indicating the acquired area on the display device 6. Since the input data area for each determination result of the expected output of the inference model is displayed, the user can grasp the degree of violation in the input data.

[0098] The verification device 2A according to the second embodiment includes a sample acquisition unit 29-2 that acquires one or more data samples included in the region acquired by the region acquisition unit 29-1. The display processing unit 28 displays the acquired data samples on the display device 6. Since the data samples included in the input data region for each judgment result of the expected output of the inference model are displayed, the user can grasp the degree of violation in the input data for each data sample.

[0099] The verification device 2A according to the second embodiment includes an area summarization unit 29-3 that performs summarization by combining multiple areas acquired by the area acquisition unit 29-1 into a smaller number of areas. The display processing unit 28 displays information indicating the summarized areas on the display device 6. By displaying the areas in which multiple input data areas have been summarized into a smaller number for each judgment result of the expected output of the inference model, the user can easily grasp the degree of violation in the input data.

[0100] In the verification device 2A according to the second embodiment, the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees. The device includes a sample acquisition unit 29-2 that acquires one or more data samples included in the acquired region. The region summarization unit 29-3 learns a single decision tree using the acquired data samples, and summarizes the region using the leaf nodes of the learned decision tree. Since multiple input data regions for each judgment result of the expected output of the inference model are summarized into simple information, the user can easily grasp the degree of violation in the input data.

[0101] It is possible to combine the embodiments, modify any of the components of the embodiments, or omit any of the components of the embodiments.

[0102] The verification device according to the present disclosure can be used, for example, to verify inference models for performing various inferences. [Explanation of symbols]

[0103] 1,1A inference system, 2,2A verification device, 3 inference device, 4 inference model storage device, 5 judgment result storage device, 6 display device, 21,31 inference model reading unit, 22 output upper and lower bound calculation unit, 23 expected output designation unit, 24 expected output judgment unit, 25 input area division unit, 26 input area designation unit, 27 area ratio calculation unit, 28 display processing unit, 29-1 area acquisition unit, 29-2 sample acquisition unit, 29-3 area summary unit, 32 inference unit, 100 input interface, 101 output interface, 102 processing circuit, 103 processor, 104 memory.

Claims

1. an expected output determination unit that determines whether the output data of the inference model for an input data area, which is a numerical range of data, is a violation determination that is not an expected output; an input area division unit that divides an area of ​​the input data area that is determined not to be a violation judgment and sets the divided area as a new input data area of ​​the inference model; an area ratio calculation unit that calculates an area ratio for each determination result with respect to the input data area; A verification result output unit that outputs verification result information indicating the ratio of the area. A verification device comprising:

2. The expected output determination unit determines whether the output data of the inference model for the input data area is a pass determination, that is, all of the output data of the inference model for the input data area is the expected output, a violation determination, that all of the output data of the inference model for the input data area is not the expected output, or an unknown determination, that is, none of the above.

2. The verification device according to claim 1.

3. an output upper / lower bound calculation unit that calculates at least one of the upper and lower bounds of the output data of the inference model for the input data domain; The expected output determination unit determines whether the output data area indicated by the upper or lower bound of the output data of the inference model is a pass determination, a violation determination, or an unclear determination.

3. The verification device according to claim 2.

4. The inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees, The input region division unit divides a region according to a branching condition of any of the decision trees included in the inference model.

4. The verification device according to claim 3.

5. an area acquisition unit that acquires the input data area determined by the expected output determination unit; The verification result output unit displays information indicating the acquired area on a display device.

2. The verification device according to claim 1.

6. a sample acquisition unit that acquires one or more data samples included in the region acquired by the region acquisition unit; The verification result output unit displays the acquired data sample on the display device.

6. The verification device according to claim 5.

7. a region summarizing unit that summarizes the multiple regions acquired by the region acquisition unit into a smaller number of regions; The verification result output unit displays information indicating the area where the summary has been performed on the display device.

6. The verification device according to claim 5.

8. The inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees, a sample acquisition unit for acquiring one or more data samples included in the acquired region; The region summarization unit learns a single decision tree using the acquired data samples, and performs region summarization using leaf nodes of the learned decision tree.

8. The verification device according to claim 7.

9. A verification device according to any one of claims 1 to 8, an inference device including an inference model reading unit that reads the inference model verified by the verification device, and an inference unit that performs inference using the inference model; 1. An inference system comprising:

10. A verification method using a verification device, comprising: An expected output determination unit determines whether the output data of the inference model for an input data area, which is a numerical range of data, is not the expected output, which is a violation determination; An input area division unit divides an area of ​​the input data area that is determined not to be a violation judgment, and sets the divided area as a new input data area of ​​the inference model; a step of an area ratio calculation unit calculating an area ratio for each determination result with respect to the input data area; A step in which a verification result output unit outputs verification result information indicating the ratio of the area.

23. A verification method comprising:

11. On the computer, A process of determining whether the output data of the inference model for an input data area, which is a numerical range of data, is a violation judgment that is not an expected output; A process of dividing an area of ​​the input data area that is determined not to be a violation judgment, and setting the divided area as a new input data area of ​​the inference model; A process of calculating a ratio of an area for each determination result to the input data area; and outputting verification result information indicating the ratio of the area.

12. A verification device for verifying an output of an inference model, comprising: an input data area designation unit that designates an input data area, which is the range of data to be input into the inference model; an output range designation unit that designates an output range which is the range of the output of the inference model; a determination unit that determines, for each data region of the input data region input to the inference model, whether the output of the inference model is a first region that is included in the output range, or a second region that is not included in the output range; an area ratio calculation unit that calculates a ratio of the first area and a ratio of the second area in the input data area; an output unit that outputs information on the ratio of the first region and information on the ratio of the second region; A verification device comprising:

13. The area ratio calculation unit calculates a ratio of a third area, which includes, among the input data area input to the inference model, a data area in which it is unclear whether the output from the inference model is included in the output range, or a data area in which at least a portion of the output from the inference model is not included in the output range, The output unit outputs information on the ratio of the first region, information on the ratio of the second region, and information on the ratio of the third region.

13. The verification device according to claim 12.

14. When either the ratio of the first region or the ratio of the second region is changed, the output unit outputs information on the ratio of the first region or information on the ratio of the second region.

13. The verification device according to claim 12.

15. When either the first area or the second area is specified, the output unit outputs information on the range of each variable that defines the input data area corresponding to the specified area.

13. The verification device according to claim 12, wherein:

16. When a range of a first variable is specified among the variables defining the input data area, the output unit outputs information on the ratio of the first area and information on the ratio of the second area within the specified range of the first variable.

13. The verification device according to claim 12.

17. A verification method for a verification device that verifies an output of an inference model, comprising: A step of specifying an input data area, which is the range of data to be input into the inference model, by an input data area specification unit; A step of specifying an output range, which is the range of the output of the inference model, by an output range specification unit; A step of determining, for each data region of the input data region input to the inference model, whether the output of the inference model is a first region that is included in the output range, or a second region that is not included in the output range, by a determination unit; calculating a ratio of the first area and a ratio of the second area in the input data area by an area ratio calculation unit; and outputting information on the ratio of the first region and information on the ratio of the second region by an output unit.

23. A verification method comprising:

18. On the computer, A process of specifying an input data domain, which is the range of data to be input to the inference model; A process of specifying an output range, which is the range of the output of the inference model; A process of determining, for each data region of the input data region input to the inference model, whether the output of the inference model is a first region that is included in the output range, or a second region that is not included in the output range; A process of calculating a ratio of the first area and a ratio of the second area in the input data area; a program for executing a process of outputting information on the ratio of the first area and information on the ratio of the second area.

Images