Verification device, inference device, inference system, and verification method

Abstract

A verification device (2) comprises: an expected output determination unit (24) that determines whether or not all of output data of an inference model for an input data area constitutes a determination of violation, which is a determination of all of the output data not being an expected output; an input area division unit (25) that divides, from the input data area, an area determined not to constitute a determination of violation, and sets the divided area as a new input data area for the inference model; an area ratio calculation unit (27) that calculates a ratio of the area for each determination result to the input data area; and a display processing unit (28) that displays verification result information indicating the ratio of the area on a display device (6).

Description

Verification device, inference device, inference system, and verification method

[0001] The present disclosure relates to a verification device, an inference device, an inference system, and a verification method.

[0002] A technique for evaluating the validity of an inference model, which is a machine learning model, has been proposed. For example, in Patent Literature 1, a technique is proposed in which a model output region for a data region input to a decision tree ensemble model is presented, which includes at least one violating data sample that violates a verification property.

[0003] Patent No. 7059220

[0004] The conventional technology described in Patent Literature 1 presents a range that includes at least one violating data sample that is not the expected output of the inference model, but there is no guarantee that all data samples in the presented range are violating data samples. For this reason, the conventional technology has a problem in that, for example, if there are no violating data samples in most areas of the presented range and only a very narrow area contains violating data samples, the degree of violation is overestimated, making it difficult to appropriately deal with the violation.

[0005] The present disclosure solves the above problem and aims to provide a verification device that can present the proportion of input data area for each judgment result of the expected output of an inference model.

[0006] The verification device according to the present disclosure includes an expected output determination unit that determines whether all of the output data of an inference model for an input data area, which is a numerical range of data, is a violation determination that is not the expected output; an input area division unit that divides areas of the input data area that are determined not to be a violation determination and sets the divided areas as new input data areas of the inference model; an area proportion calculation unit that calculates the proportion of areas for each determination result for the input data area; and a verification result output unit that outputs verification result information indicating the proportion of areas.

[0007] According to the present disclosure, a verification device determines whether all of the output data of an inference model for an input data region is a violation judgment that is not the expected output, divides the input data region that is determined not to be a violation judgment, sets the divided region as a new input data region for the inference model, calculates the region proportion for each judgment result for the input data region, and outputs verification result information indicating the region proportion.By outputting the region proportion for each judgment result of the expected output of the inference model, the verification device according to the present disclosure can present the region proportion of the input data region for each judgment result of the expected output of the inference model.This makes it possible to prevent the degree of violation from being overestimated or underestimated.

[0008] 1 is a block diagram showing an example of the configuration of an inference system according to embodiment 1. FIG. 1 is a graph showing the input data area and the determination results of the expected output. FIG. 2 is a schematic diagram showing the division process of the input data area and the expected output determination results. FIG. 3 is a schematic diagram showing an overview of the expected output determination. FIG. 4 is a diagram showing an example of the expected output determination results. FIG. 6A, FIG. 6B, and FIG. 6C are diagrams showing an example of the display of verification results. FIG. 7A is a flowchart showing the verification method according to embodiment 1. FIG. 8A is a flowchart showing the area determination process. FIG. 9A and FIG. 9B are block diagrams showing a hardware configuration that realizes the functions of the verification device according to embodiment 1. FIG. 10A is a block diagram showing an example of the configuration of an inference system according to embodiment 2. FIG. 11A is a diagram showing the acquisition result of the data area of ​​the expected output determination results and an example of its display. FIG. 12A is a diagram showing the acquisition result of the data sample from the data area of ​​the expected output determination results and an example of its display. FIG. 13 is a schematic diagram showing an example of the display of the summarized area.

[0009] Embodiment 1. Figure 1 is a block diagram showing an example configuration of an inference system 1 according to embodiment 1. In Figure 1, the inference system 1 is a system that performs inference using a verified inference model, and includes a verification device 2, an inference device 3, an inference model storage device 4, a judgment result storage device 5, and a display device 6. The verification device 2 is a device that verifies an inference model that is a machine learning model, and the inference device 3 is a device that performs inference using the inference model. The inference system 1 is a system in which the verification device 2, the inference device 3, the inference model storage device 4, the judgment result storage device 5, and the display device 6 are connected via a network using wired signal lines or wireless communication. The network is a telecommunications line including the Internet, etc.

[0010] The verification device 2 can prevent the degree of violation from being overestimated or underestimated by presenting the proportion of the input data area for each judgment result of the expected output of the inference model. For example, in the conventional technology described in Patent Document 1, a violation range including at least one violation data sample that is not the expected output of the inference model is presented as the verification result. Note that the expected output is a proposition that is expected to be true for the output data of the inference model. Figure 2 is a graph showing the input data area and the judgment result of the expected output. The input data area shown in Figure 2 is a proposition that is expected to be true for the input variable x １ and the input variable x ３ Area A shown in FIG. 2 is a data area that includes area B. All data included in area B are non-compliant data samples that are not the expected output of the inference model, i.e., do not satisfy the verification conditions, and area B is an area that includes only non-compliant data samples.

[0011] In the conventional technology described in Patent Document 1, an area A may be proposed. There is no guarantee that all data samples within this area A are violations. For example, as shown in FIG. 2, there is a possibility that no violation data samples exist in most areas, and in this case, the degree of violation may be overestimated. Since all data samples in area A may fall within the violation range, the output data of the inference model when data in area A is input may not be the expected output, i.e., it may be determined to be a violation that does not satisfy the verification conditions. In this case, even though the output data of the inference model when data included in an area in area A where no violation data samples exist is input is actually the expected output, the inference is suspended.

[0012] In addition, the output value of the inference model when data in area A is input may be overwritten to appear as if there is no violation. In this case, the output data of the inference model when data included in an area where no violating data samples exist is input may also be overwritten with a different value, even though it is the expected output.

[0013] In addition, the inference model may be retrained by adding data samples in area A so that the output value of the inference model when data in area A is input will no longer violate the rules. In this case, the inference behavior of the inference model will also change significantly when data from an area where no violating data samples exist is input.

[0014] In a case where the test is passed if the hypervolume of the violation range output from the inference model when data contained in area A is input is below a specified value, it is possible that the test will not pass even though the hypervolume of the violation range is actually below the specified value if the input data area includes an area where no violation data samples exist.

[0015] Therefore, the verification device 2 according to the first embodiment presents the proportion of the input data area for each expected output judgment result of the inference model by outputting the proportion of the area for each expected output judgment result of the inference model. This allows the verification device 2 to prevent problems caused by overestimating or underestimating the degree of violation as described above.

[0016] FIG. 3 is a schematic diagram showing the division process of the input data area and the expected output determination result. １ and the input variable x ３ 3 shows the input data region of the inference model defined by the following equation. In FIG. 3, an input data region that contains only input data in which all output data of the inference model are expected outputs is a pass region. An input data region that contains only input data that is judged to be a violation, in which all output data of the inference model are not expected outputs, is a violation region. An input data region that is neither a pass region nor a violation region is an unknown region.

[0017] The verification device 2 inputs an input data area into the inference model and determines whether all of the output data of the inference model is a violation judgment that is not the expected output, thereby calculating the proportion of the area for each judgment result relative to the input data area. As shown in the graph on the left side of Figure 3, the proportion of pass areas and the proportion of unknown areas of the input data area to the inference model are 23.9% and 76.1%, respectively. Since the verification device 2 includes unknown areas, i.e., input data areas that were not judged to be violations, the verification device 2 divides the unknown areas as shown by the arrows in Figure 3 and sets the divided areas as new input data areas for the inference model.

[0018] Next, the verification device 2 sets each divided area of ​​the unknown area as a new input data area of ​​the inference model, and calculates the proportion of the area for each judgment result relative to the input data area by determining whether the output data of the inference model is a violation judgment that is not the expected output. For example, as shown in the graph in the center of Figure 3, the proportion of the areas in the input data area to the inference model is assumed to be 46.7% pass, 42.5% unknown, and 0.8% violation. In this case, the verification device 2 determines that an unknown area is included and further divides these unknown areas as indicated by the arrows in Figure 3.

[0019] Next, the verification device 2 sets the newly divided unknown area as a new input data area for the inference model, and calculates the proportion of each area for each judgment result relative to the input data area by determining whether all of the output data of the inference model is a violation judgment that is not the expected output. For example, as shown in the graph on the right side of Figure 3, the proportion of pass areas in the input data area to the inference model is 98.3%, the proportion of unknown areas is 0%, and the proportion of violation areas is 1.7%. In this way, the verification device 2 can present the proportion of areas corresponding to the expected output judgment results in the input data area for the inference model and can also identify the data numerical range for each area. This prevents the degree of violation from being overestimated or underestimated.

[0020] The inference model storage device 4 is a storage device that stores inference models. From the perspective of the verification device 2, the inference model stored in the inference model storage device 4 is a machine learning model to be verified, and from the perspective of the inference device 3, it is a machine learning model for inference processing. The inference model storage device 4 is provided outside the verification device 2 and the inference device 3. Note that the inference model storage device 4 only needs to be accessible from the verification device 2 and the inference device 3, and may be a storage device provided in a computer that functions as the verification device 2 or the inference device 3.

[0021] The judgment result storage device 5 is a storage device that stores the expected output judgment results of the expected output judgment unit 24 provided in the verification device 2. The expected output judgment result is a judgment result as to whether or not all of the output data of the inference model for the input data area, which is the numerical range of the data, is a violation judgment that is not the expected output. In addition, the judgment result storage device 5 is provided outside the verification device 2. Note that the judgment result storage device 5 only needs to be accessible from the verification device 2, and may be a storage device provided in a computer that functions as the verification device 2.

[0022] The verification device 2 is realized, for example, by a computer having a communication unit, a calculation unit, and a memory unit. The communication unit communicates with the inference model memory device 4 or the judgment result memory device 5 via a network using a wired signal line or wireless communication. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark). The communication unit includes the input interface 100 and output interface 101 of Figures 9A and 9B.

[0023] The calculation unit controls the overall operation of the verification device 2. The calculation unit includes an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input region division unit 25, an input region designation unit 26, a region proportion calculation unit 27, and a display processing unit 28. The calculation unit executes an information processing application for verifying the inference model, thereby realizing the various functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28. The calculation unit includes the processing circuit 102 of FIG. 9A and the processor 103 of FIG. 9B.

[0024] The storage unit stores information processing applications and information used for the arithmetic processing of the arithmetic unit. The storage unit is a storage device provided in the computer functioning as the verification device 2, and includes storage such as an HDD or SSD, or the memory 104 shown in FIGS. 9A and 9B. Note that the storage unit may be provided outside the verification device 2 as long as it is accessible by the verification device 2.

[0025] The inference model reading unit 21 reads the inference model to be verified from the inference model storage device 4. The inference model is an N-dimensional input X=(x １ , x ２ , ..., x Ｎ )∈R Ｎ For one-dimensional output y∈R or M-dimensional output Y=(y １ , y ２ , ..., y Ｍ )∈R Ｍ where N and M are integers. For example, inference models include trained machine learning models, neural networks, decision tree models, decision tree ensemble models, support vector machines (SVMs), generalized linear models, generalized additive models (GAMs), Gaussian process regression models (GPRs), naive Bayes, and Gaussian mixture models (GMMs).

[0026] Neural networks include multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers. Decision tree ensemble models include random forests and gradient boosted trees. Generalized linear models include linear regression and logistic regression.

[0027] The output upper / lower bound calculation unit 22 calculates at least one of the upper and lower bounds of the output data of the inference model for the input data domain. That is, the inference model outputs data such that the lower bound value ≦ output data ≦ upper bound value is always true. The upper bounds are calculated, for example, as follows: When the inference model is a generalized linear model, the output upper / lower bound calculation unit 22 calculates, as the upper bound, the output value when a variable with a positive coefficient is the upper limit value of the input data domain and a variable with a negative coefficient is the lower limit value of the input data domain. When the inference model is a single decision tree model, the output upper / lower bound calculation unit 22 calculates, as the upper bound, the maximum output value of a set of leaf nodes that can be reached by a data sample in the input data domain. When the inference model is a decision tree ensemble model, the output upper / lower bound calculation unit 22 calculates, as the upper bound, the sum of the maximum output values ​​for each decision tree of a set of leaf nodes that can be reached by a data sample in the input data domain.

[0028] The lower bounds are calculated, for example, as follows: When the inference model is a generalized linear model, the output upper / lower bound calculation unit 22 calculates, as the lower bound, the output value when a variable with a positive coefficient is at the lower limit of the input data domain and a variable with a negative coefficient is at the upper limit of the input data domain. When the inference model is a single decision tree model, the output upper / lower bound calculation unit 22 calculates, as the lower bound, the minimum output value among the leaf node sets that can be reached by data samples in the input data domain. When the inference model is a decision tree ensemble model, the output upper / lower bound calculation unit 22 calculates, as the lower bound, the sum of the minimum output values ​​for each decision tree among the leaf node sets that can be reached by data samples in the input data domain.

[0029] The expected output specification unit 23 sets the expected output value of the inference model to the expected output determination unit 24. The expected output is a proposition that is expected to be true for the output data y of the inference model. For example, 50≦y<80 is specified as the range of the output value of regression, 0.5≦y is specified as the range of the output score (output of the sigmoid function) of binary classification, and (y<80) is specified as the output class of multi-class classification. １ <y ２ ) & (y ３ <y ２), specifying class 2 as the expected output using the inequality in the output of the softmax function, or specifying |y-y'|≦0.1, which is the difference with the output y'=f'(X) of another inference model f'.

[0030] For example, a user inputs a conditional expression or upper and lower limit values ​​for the expected output in text using an input device not shown in Fig. 1. As a result, the expected output designation unit 23 sets information indicating the conditional expression or upper and lower limit values ​​input by the input device as the expected output in the expected output determination unit 24. The expected output designation unit 23 also instructs the display processing unit 28 to display, on the screen of the display device 6, a selection screen for selecting a class ID using a drop-down or check boxes, or a selection screen for selecting an output range using a parallel coordinate plot. When the user performs a selection operation on the selection screen using the input device, the expected output designation unit 23 accepts the selection operation information and sets the selection operation information as the expected output in the expected output determination unit 24.

[0031] The expected output determination unit 24 determines whether all of the output data of the inference model for the input data region is a violation judgment that is not the expected output. FIG. 4 is a schematic diagram illustrating an overview of expected output determination, showing a case where the expected output of the regression is 50≦y<80. In FIG. 4, all output data y included in the numerical range (lower bound of y≧50) & (upper bound of y<80) is also included in the expected output range 50≦y<80, so the input data region for the inference model when the output data y is output is not judged as a violation and is a pass region. All output data y included in the numerical range (upper bound of y<50) | (lower bound of y≧80) is not included in the expected output range 50≦y<80, so the input data region for the inference model when the output data y is output is a violation region that is judged as a violation. Numerical ranges that are not included in either (lower bound of y≧50) & (upper bound of y<80) or (upper bound of y<50) | (lower bound of y≧80) include output data y that is not judged as a violation. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0032] When the expected output of binary classification is 1, i.e., when the output score y is 0.5≦y, all output scores y included in the score range where the lower bound of the output score y is ≧0.5 are included in the expected output range 0.5≦y, so the input data region for the inference model when the output score y is output is the pass region. All output scores y included in the numerical range where the upper bound of the output score y is <0.5 are not included in the expected output range 0.5≦y, so the input data region for the inference model when the output score y is output is the violation region. A score range where neither the lower bound of the output score y is ≧0.5 nor the upper bound of the output score y is <0.5 includes output data y that is not judged as a violation. The expected output judgment unit 24 judges that the output data y is included in the unknown region.

[0033] The expected output of the three-class classification is class 2, i.e., (y １ <y ２ ) & (y ２ <y ３ ), then (y １ Upper bound of <y ２ lower bound of y ３ Upper bound of <y ２ All output data y included in the numerical range of (the lower bound of y) is within the range of expected output (y １ <y ２ ) & (y ２ <y ３ ), the input data domain for the inference model when the output data y is output is the pass domain. １ Lower bound ≧ y ２ upper bound of y ３ Lower bound ≧ y ２ All output data y included in the numerical range of the expected output (y １ <y ２ ) & (y ２ <y ３ ), the input data domain for the inference model when the output data y is output is a violation domain. １ Upper bound of <y ２ lower bound of y ３ Upper bound of <y ２ lower bound of y １ Lower bound ≧ y２ upper bound of y ３ Lower bound ≧ y ２ In the case where the numerical range is not included in either of the upper bounds of the expected output y, the output data y is not judged to be in violation. The expected output judging unit 24 judges that the output data y is included in the unknown region.

[0034] If |y - y'| ≦ 0.1, which is the difference between the output y' = f'(X) of another inference model f', is the expected output range, then all output data y included in the numerical range of (upper bound of y - lower bound of y' ≦ 0.1) & (upper bound of y' - lower bound of y ≦ 0.1) is also included in the expected output range, so the input data range for the inference model when the output data y is output is a pass range. All output data y included in the numerical range of (lower bound of y - upper bound of y' > 0.1) | (lower bound of y' - upper bound of y > 0.1) is not included in the expected output range |y - y'| ≦ 0.1, which is a violation judgment, so the input data range for the inference model when the output data y is output is a violation range. In a numerical range that is not included in either (upper bound of y - lower bound of y' ≦ 0.1) & (upper bound of y' - lower bound of y ≦ 0.1) or (lower bound of y - upper bound of y' > 0.1) | (lower bound of y' - upper bound of y > 0.1), output data y that is determined to be a violation and not an expected output is included. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0035] The judgment result information of the expected output judgment unit 24 is stored in the judgment result storage device 5. FIG. 5 is a diagram showing an example of the expected output judgment result, and shows the stored contents of the judgment result storage device 5. As shown in FIG. 5, the judgment result information includes an input data area and an expected output judgment result for each input data. The input data area includes input variables x, which are each parameter of the relational expression indicating the input data. １ , x ２ , ..., x Ｎ The expected output judgment result is either pass, violation, or unknown. Although the input data corresponding to all judgment results is shown, the judgment result storage device 5 may store only specific judgment results as the expected output judgment result. For example, it may store only input data for which the expected output judgment result is violation.

[0036] The input region division unit 25 divides a region of the input data region that is determined not to be a violation judgment, and sets the divided region as a new input data region for the inference model. For example, the input region division unit 25 may divide the input data region at the branches of a decision tree included in the inference model. In this case, the input region division unit 25 divides the input data region into overlapping regions of the data region corresponding to each leaf node of the decision tree and the input data region. In this case, the input region division unit 25 may ignore leaf nodes that cannot be reached in consideration of the constraints of the input data region, or may stop division at an intermediate branch rather than at a terminal leaf node.

[0037] The input region division unit 25 may also divide the input data region randomly. For example, the input region division unit 25 may randomly select an input variable and determine a division point within the upper and lower limits of the input variable. Furthermore, the input region division unit 25 may divide the input data region based on the midpoint of the upper and lower limits generated from a uniform distribution, or the mean value, median, or quantile of the training data. The input region division unit 25 may weight the selection of input variables to be divided according to the importance of the feature of the inference model.

[0038] Furthermore, the input region division unit 25 may divide the input data region into multiple division candidates and adopt the division that results in the best index after division. For example, the input region division unit 25 adopts the division that minimizes the difference between the upper and lower bounds of the inference model output for the divided region. The input region division unit 25 may also adopt the division that minimizes the sum of the sizes of the unknown regions after division.

[0039] The input region division unit 25 may continue recursively dividing the input region until there are no more unknown regions, or may stop dividing halfway through. For example, the input region division unit 25 stops dividing the input region when the processing time exceeds a specified value, the number of divided regions exceeds a specified value, the size of the unknown region falls below a specified value, or the size of the unknown region becomes smaller than the size of the violation region.

[0040] The input domain designation unit 26 sets the input data domain to the output upper and lower bound calculation unit 22. The input data domain is a data domain in an N-dimensional space that is input to the inference model to be verified. N is an integer. For example, the input data domain is table data defined by an input variable x. The input data domain may be feature quantities extracted from images, videos, text, audio, graphs, time-series data, or the like using a neural network or the like.

[0041] The input data domain is, for example, S={X=(x １ , x ２ , ..., x Ｎ )∈R Ｎ |-3.5≦x １ ≦5.2, ..., 15≦x Ｎ ≦40} or S = {X∈R Ｎ ||X| ２ ≦r ２}, or S=S1∩S2 representing the intersection of multiple domains, or S=S1∪S2 representing the union of multiple domains.

[0042] For example, a user inputs a conditional expression or upper and lower limit values ​​of a hyper-rectangle prism in text form using an input device not shown in FIG. As a result, the input region designation unit 26 sets information indicating the conditional expression or upper and lower limit values ​​of the hyper-rectangle prism input via the input device as an input data region in the output upper and lower bound calculation unit 22. The input region designation unit 26 also instructs the display processing unit 28 to display, on the screen of the display device 6, a selection screen for selecting a range of input variables in two-dimensional space using a rectangle or a lasso, or a selection screen for selecting the range of each input variable using a parallel coordinate plot. When the user performs a selection operation on the selection screen using the input device, the input region designation unit 26 accepts the selection operation information and sets the selection operation information as an input data region in the output upper and lower bound calculation unit 22.

[0043] The region proportion calculation unit 27 calculates the proportion of the region for each expected output determination result with respect to the input data region. For example, the region proportion calculation unit 27 may calculate the proportion using the hypervolume of the input data region. When the input data region is a hypercuboid, the region proportion calculation unit 27 may calculate the region proportion using the product of the side lengths of the hypercuboid. Furthermore, the region proportion calculation unit 27 may calculate the region proportion for each expected output determination result with respect to the input data region using the number of data samples of a predetermined data set included in the region.

[0044] The region proportion calculation unit 27 may calculate the proportion of the region for each expected output determination result relative to the input data region using the number of generated data samples included in the region. The data samples may be generated from a uniform distribution, may be generated by perturbing data samples of a predetermined data set, may be generated from a data distribution learned from a predetermined data set, or may be generated using a generative adversarial network (GAN) trained with a predetermined data set.

[0045] The sizes of the violation area, the pass area, and the unknown area are respectively |S 違反 |, |S 合格 |, |S 不明 |, the size of the entire input area is |S 全体 |=|S 違反 |+|S 合格 |+|S 不明 In this case, the area ratio calculation unit 27 calculates the area ratio by, for example, |S 違反 | / |S 全体 |, |S 合格 | / |S 全体 |, |S 不明 | / |S 全体 |, |S 違反 | / (|S 違反 |+|S 合格 |), or (|S 違反 |+|S 不明 |) / |S 全体 Calculate |.

[0046] The display processing unit 28 is a verification result output unit that outputs verification result information indicating the area ratio. The display processing unit 28 displays the verification result information on the display device 6. The display device 6 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display device.

[0047] The display processing unit 28 outputs display control information for displaying the verification result information to the display device 6. The display device 6 displays the verification result information based on the display control information. For example, the area percentages are displayed as a bar graph, a pie chart, a stacked bar graph, or by color shading. FIG. 6A is a diagram showing an example display of the verification results, in which the area percentages are shown as a bar graph. In FIG. 6A, the area percentages are represented by the length of the bar. The pass area percentage is 32.3%, the unknown area percentage is 51.3%, and the violation area percentage is 16.4%. FIG. 6B is a diagram showing an example display of the verification results, in which the area percentages are shown by color shading. In FIG. 6B, the violation area percentage is represented by color shading. For example, the larger the violation percentage, the darker the color displayed.

[0048] The display processing unit 28 may update the display of the area ratios that change in response to the division of the input data area in real time. For example, the display processing unit 28 may monotonically decrease the display of the unknown areas or monotonically increase the display of the pass areas and violation areas over time. The display processing unit 28 may also display the time-varying change in the area ratios on the display device 6 as a line graph or animation.

[0049] Furthermore, the display processing unit 28 may also display the specified input data region and the expected output on the display device 6. For example, the display processing unit 28 displays the region ratio on the display device 6 as a parallel coordinate plot, a two-dimensional plot, or text. FIG. 6C is a diagram showing an example of a display of the verification result, in which the region ratio is shown as a parallel coordinate plot and text. In FIG. 6C, the input variable x that defines the input data region １ , x ２ , ..., x ＮThe numerical range of each of the input variables x is shown by a rectangle, and the numerical range of the output data y of the inference model for the input data domain is shown by a rectangle. １ , x ２ , ..., x Ｎ The numerical ranges of the respective values ​​and the numerical range of the output data y may be displayed in text form on the display device 6 as shown in FIG.

[0050] 1 shows the verification device 2 as including an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input region division unit 25, an input region designation unit 26, a region proportion calculation unit 27, and a display processing unit 28, but the configuration is not limited to this. The verification device 2 only needs to be able to determine the expected output of the inference model, divide the input data region to be input to the inference model, and output the determination results. Therefore, components other than the expected output determination unit 24, the input region division unit 25, and the display processing unit 28 may be provided in an external device accessible from the verification device 2.

[0051] Although the input data area is a two-dimensional space area in the above example, the input data area is generally a high-dimensional space area of ​​two or more dimensions.

[0052] The inference device 3 is realized, for example, by a computer including a communication unit, a calculation unit, and a storage unit. The communication unit communicates with the inference model storage device 4 via a network using a wired signal line or wireless communication. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark).

[0053] The calculation unit controls the overall operation of the inference device 3. The calculation unit includes an inference model reading unit 31 and an inference unit 32. The calculation unit executes an information processing application for performing inference, thereby realizing various functions of the inference model reading unit 31 and the inference unit 32.

[0054] The storage unit stores information processing applications and information used for the calculation processing of the calculation unit. The storage unit is a storage device provided in a computer functioning as the inference device 3, and includes storage such as an HDD or SSD, or memory 104 of Figures 9A and 9B. Note that the storage unit may be provided outside the inference device 3 as long as it is accessible by the inference device 3.

[0055] The inference model reading unit 31 reads the inference model verified by the verification unit 2 from the inference model storage device 4. The inference model read by the inference model reading unit 31 is set in the inference unit 32.

[0056] The inference unit 32 performs inference using the inference model read by the inference model reading unit 31. For example, the inference unit 32 may test a complex inference model. In this case, in order to expose the risk of occurrence of defects such as bugs in the complex inference model in pre-operation testing, the inference unit 32 sets the minimum properties that are desired to be satisfied as input data areas and expected outputs in the inference model and performs inference.

[0057] For example, the inference unit 32 may use an input data area within a five-minute walk from a station and within the last ten years of construction as the input data region, and infer the rent for an apartment building assuming that the expected output of the inference model for this input data region is a rent of 80,000 yen or more. The inference unit 32 may also use an input data area of ​​blood pressure of 135 or higher and LDL cholesterol of 140 or higher as the input data region, and infer the need for testing assuming that the expected output of the inference model for this input data region is that testing is required. The inference unit 32 may use an input data region of a numerical range indicating products whose measurement error is larger than that of the abnormal product sample, and infer a product abnormality by assuming that all products whose measurement error is larger than that of the abnormal product sample are abnormal as the expected output. If the percentage of violation regions is less than a specified value, the inference device 3 deploys the system to the real environment and begins operation.

[0058] The inference unit 32 may infer the fidelity of a proxy model that approximates a complex inference model. For example, the inference device 3 approximates a complex inference model using a single decision tree model with high interpretability, and allows the user to recognize the behavior during inference. The verification device 2 sets the expected output to be the match between the outputs of the original inference model and the proxy model, and calculates the percentage of violation areas where the outputs of the original inference model and the proxy model do not match. The verification device 2 then regards the smallness of the violation area percentage as the fidelity of the proxy model and displays it on the display device 6.

[0059] The inference of an existing rule-based system may be replaced with an inference model verified by the verification device 2. For example, the existing system may be replaced with a machine learning model with high predictive accuracy. The verification device 2 sets the match between the output of the inference model and the existing system as the expected output and calculates the percentage of violation areas where the output of the inference model and the existing system do not match. The inference device 3 then performs replacement with an inference model whose violation area percentage is less than a specified value. The verification device 2 may also quantify the difference between two inference models of different versions. For example, the verification device 2 quantifies the difference between inference model A trained with data from a certain period and inference model B trained with data from another period. The verification device 2 sets the match between the outputs of inference model A and inference model B as the expected output and calculates the percentage of violation areas where the outputs of inference model A and inference model B do not match. If the violation area percentage is less than a specified value, the difference between model A and model B is considered small. This allows, for example, a system that continuously collects and learns data to update an inference model, allowing the model to be updated only when the difference between the inference models is small. Furthermore, by presenting the violation areas, it is possible to know in what areas there are differences in the model, and for example, it is possible to quickly identify and deal with environmental changes such as data shifts.

[0060] Next, a verification method according to the first embodiment will be described. Fig. 7 is a flowchart showing the verification method according to the first embodiment. An inference model reading unit 21 reads an inference model from the inference model storage device 4 and sets it in the output upper and lower bound calculation unit 22, and an input domain designation unit 26 sets an input data domain in the output upper and lower bound calculation unit 22. An expected output designation unit 23 sets the expected output of the inference model to the expected output determination unit 24. This series of processes constitutes step ST1.

[0061] The expected output determination unit 24 clears the area list and adds the set input data area (step ST2). The area list is a list of data areas to be input to the inference model. The expected output determination unit 24 inputs the input data area registered in the area list to the inference model.

[0062] Next, the expected output determination unit 24 performs an expected output determination process for each input data region in the region list (step ST3). Subsequently, the expected output determination unit 24 determines whether there is a region that is neither a violation region in which all of the output data of the inference model for the input data region is the expected output, nor a pass region in which all of the output data is the expected output, i.e., an unknown region (step ST4).

[0063] If an unknown region exists (step ST4; YES), the input region dividing unit 25 divides the unknown region (step ST5). The expected output determining unit 24 clears the region list and adds the divided region as a new input data region of the inference model (step ST6). Next, the expected output determining unit 24 proceeds to step ST3 to determine the expected output of the new input data region, and repeats the subsequent processes.

[0064] If there are no more unknown areas (step ST4; NO), the area ratio calculation unit 27 calculates the area ratio for each expected output judgment result relative to the input data area, and then the display processing unit 28 displays verification result information indicating the area ratio on the display device 6 (step ST7). This allows the verification device 2 to present the input data area ratio for each expected output judgment result of the inference model.

[0065] 8 is a flowchart showing the expected output determination process, showing the detailed process of step ST3 in FIG. 7. The output upper and lower bound calculation unit 22 calculates the upper and lower bounds of the output data of the inference model for the input data domain (step ST1A). The output upper and lower bound calculation unit 22 sets the calculated upper and lower bounds of the output data in the expected output determination unit 24.

[0066] The expected output determination unit 24 determines whether the input data area corresponding to the output data defined by the upper and lower bounds of the inference model is in violation of the expected output specified by the expected output designation unit 23, i.e., whether it is in violation, pass, or is unknown (step ST2A). Next, the expected output determination unit 24 stores the input data area and the corresponding expected output determination result in the determination result storage device 5 (step ST3A).

[0067] Next, we will explain the hardware configuration that realizes the functions of the verification device 2. The functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, expected output specification unit 23, expected output determination unit 24, input region division unit 25, input region specification unit 26, region proportion calculation unit 27, and display processing unit 28 provided in the verification device 2 are realized by processing circuits. That is, the verification device 2 includes a processing circuit for executing the processes from step ST1 to step ST7 shown in Figure 7. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in memory.

[0068] Fig. 9A is a block diagram showing a hardware configuration that realizes the functions of the verification device 2. Fig. 9B is a block diagram showing a hardware configuration that executes software that realizes the functions of the verification device 2. In Figs. 9A and 9B, the input interface 100 is an interface that relays data that the verification device 2 acquires from the inference model storage device 4 or the judgment result storage device 5. The output interface 101 is an interface that relays data that is output from the verification device 2 to the judgment result storage device 5.

[0069] 9A , the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28 provided in the verification device 2 may be realized by separate processing circuits, or these functions may be realized together by a single processing circuit.

[0070] 9B, the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output specification unit 23, the expected output determination unit 24, the input region division unit 25, the input region specification unit 26, the region ratio calculation unit 27, and the display processing unit 28 provided in the verification device 2 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 104.

[0071] The processor 103 reads and executes programs stored in the memory 104 to realize the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28 provided in the verification device 2. For example, the verification device 2 includes a memory 104 for storing programs that, when executed by the processor 103, result in the processing of steps ST1 to ST7 shown in FIG. 7 being executed. These programs cause the computer to execute the procedures or methods of the processing performed by the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, and the display processing unit 28. The memory 104 may be a computer-readable storage medium storing a program for causing the computer to function as an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input area division unit 25, an input area designation unit 26, an area ratio calculation unit 27, and a display processing unit 28.

[0072] The memory 104 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically-EPROM) (registered trademark), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD.

[0073] Some of the functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, expected output designation unit 23, expected output determination unit 24, input region division unit 25, input region designation unit 26, region proportion calculation unit 27, and display processing unit 28 provided in the verification device 2 may be realized by dedicated hardware, and other functions may be realized by software or firmware. For example, the functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, expected output designation unit 23, and input region designation unit 26 may be realized by a processing circuit 102, which is dedicated hardware, and the functions of the expected output determination unit 24, input region division unit 25, region proportion calculation unit 27, and display processing unit 28 may be realized by the processor 103 reading and executing a program stored in memory 104. In this way, the processing circuit can realize the above functions by hardware, software, firmware, or a combination of these.

[0074] As described above, the verification device 2 according to the first embodiment includes an expected output determination unit 24 that determines whether all of the output data of the inference model for an input data domain, which is a numerical range of data input to the inference model to be verified, is a violation judgment that is not the expected output; an input domain division unit 25 that divides the input data domain that is determined not to be a violation judgment and sets the divided domain as a new input data domain for the inference model; a domain proportion calculation unit 27 that calculates the proportion of the domain for each judgment result relative to the input data domain; and a display processing unit 28 that displays verification result information indicating the proportion of the domain on the display device 6. By outputting the proportion of the domain for each judgment result of the expected output of the inference model, the verification device 2 can present the proportion of the input data domain for each judgment result of the expected output of the inference model. This prevents the degree of violation from being overestimated or underestimated. Note that the prior art described in the aforementioned Patent Document 1 also divides the input data domain into multiple domains and uses a satisfiability determiner to determine whether any of the divided domains are satisfiable. If the satisfiability judger judges that a condition is satisfiable, i.e., that the condition indicates sufficiency, at least one data sample in the divided region is judged to be a violation. Therefore, the conventional technology described in Patent Document 1 may propose a region that partially includes a data sample judged to be a violation. In contrast, the verification device 2 can present the proportion of the input data region for each judgment result by dividing the region into regions where all of the output data of the inference model for the input data region is judged not to be a violation, i.e., regions that include both a pass judgment and a violation judgment.

[0075] In the verification device 2 according to the first embodiment, the expected output determination unit 24 determines whether the output data of the inference model for the input data area is all the expected output, whether the output data of the inference model for the input data area is all the expected output, whether the output data is all the expected output, or whether the output data is all the expected output, or whether the output data is all the expected output, or whether the output data is none of the above. This allows the verification device 2 to present the proportion of the input data area for each determination result of the expected output of the inference model.

[0076] The verification device 2 according to the first embodiment includes an output upper / lower bound calculation unit 22 that calculates at least one of the upper and lower bounds of the output data of the inference model for the input data region. The expected output determination unit 24 determines whether the output data region indicated by the upper or lower bound of the output data of the inference model is a pass, a violation, or an unknown determination. This allows the verification device 2 to present the proportion of the input data region for each determination result of the expected output of the inference model.

[0077] In the verification device 2 according to the first embodiment, the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees. The input region division unit 25 divides the region according to the branching condition of one of the decision trees included in the inference model. This allows the verification device 2 to determine the expected output of the inference model for each divided region.

[0078] The inference device 3 according to the first embodiment includes an inference model reading unit 31 that reads the inference model verified by the verification device 2, and an inference unit 32 that performs inference using the inference model. By presenting the proportion of the input data area for each judgment result of the expected output of the inference model, it is possible to prevent the degree of violation from being overestimated or underestimated, and therefore the inference device 3 can perform highly accurate inference.

[0079] The inference system 1 according to the first embodiment includes the above-described verification device 2 and inference device 3. It is possible to provide an inference system that can present the proportion of the input data area for each judgment result of the expected output of an inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated.

[0080] The verification method according to the first embodiment includes the steps of: an expected output determination unit 24 determining whether all of the output data of the inference model for the input data region is a violation determination that is not the expected output; an input region division unit 25 dividing the input data region that is determined not to be a violation determination and setting the divided region as a new input data region for the inference model; an region proportion calculation unit 27 calculating the region proportion for each determination result for the input data region; and a display processing unit 28 displaying verification result information indicating the region proportion on the display device 6. By executing the above method, the verification device 2 can present the proportion of the input data region for each determination result of the expected output of the inference model. This makes it possible to prevent the degree of violation from being overestimated or underestimated.

[0081] Embodiment 2. Figure 10 is a block diagram showing an example configuration of an inference system 1A according to embodiment 2. In Figure 10, the inference system 1A is a system that performs inference using a verified inference model, and includes a verification device 2A, an inference device 3, an inference model storage device 4, a judgment result storage device 5, and a display device 6. The verification device 2A is a device that verifies an inference model that is a machine learning model, and the inference device 3 is a device that performs inference using the inference model. The inference system 1 is a system in which the verification device 2, the inference device 3, the inference model storage device 4, the judgment result storage device 5, and the display device 6 are connected via a network using wired signal lines or wireless communication. The network is a telecommunications line including the Internet, etc.

[0082] The verification device 2A presents the area ratio for each judgment result of the expected output of the inference model, as well as the area itself, data samples contained in the area, or summary information for multiple areas. The verification device 2A is realized, for example, by a computer equipped with a communication unit, a calculation unit, and a memory unit. The communication unit communicates with the inference model storage device 4 or the judgment result storage device 5 via a wired signal line or a wireless communication network. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark). The communication unit includes the input interface 100 and output interface 101 shown in Figures 9A and 9B.

[0083] The calculation unit controls the overall operation of the verification device 2A. The calculation unit includes an inference model reading unit 21, an output upper and lower bound calculation unit 22, an expected output designation unit 23, an expected output determination unit 24, an input region division unit 25, an input region designation unit 26, a region proportion calculation unit 27, a display processing unit 28, a region acquisition unit 29-1, a sample acquisition unit 29-2, and a region summarization unit 29-3. The calculation unit executes an information processing application for verifying the inference model, thereby realizing the various functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output designation unit 23, the expected output determination unit 24, the input region division unit 25, the input region designation unit 26, the region proportion calculation unit 27, the display processing unit 28, the region acquisition unit 29-1, the sample acquisition unit 29-2, and the region summarization unit 29-3. The calculation unit includes the processing circuit 102 of FIG. 9A and the processor 103 of FIG. 9B.

[0084] The storage unit stores information processing applications and information used for the arithmetic processing of the arithmetic unit. The storage unit is a storage device provided in the computer functioning as the verification device 2, and includes storage such as an HDD or SSD, or the memory 104 shown in Figures 9A and 9B. Note that the storage unit may be provided outside the verification device 2A as long as it is accessible by the verification device 2A.

[0085] The inference model reading unit 21, output upper and lower bound calculation unit 22, expected output designation unit 23, expected output determination unit 24, input area division unit 25, input area designation unit 26, area ratio calculation unit 27 and display processing unit 28 are the same as those in Figure 1, so duplicate explanations will be omitted.

[0086] The area acquisition unit 29-1 acquires the input data area determined by the expected output determination unit 24 from the determination result storage device 5. The display processing unit 28 displays information indicating the area acquired by the area acquisition unit 29-1 on the display device 6. FIG. 11 is a diagram showing the acquisition result of the data area of ​​the expected output determination result and an example of its display. As shown in FIG. 11, the information indicating the multiple input data areas is stored in the form of input variables x, which are the parameters of the relational expressions indicating the respective input data areas. １ , x ２ , ..., x Ｎ For example, the area acquisition unit 29-1 acquires information associated with the violation expected output judgment result from among information indicating a plurality of input data areas corresponding to the pass, violation, and unknown expected output judgment results.

[0087] The display processing unit 28 displays a plurality of input data regions corresponding to the violation expected output determination results in a parallel coordinate plot on the display device 6. In FIG. 11, the input variable x １ , x ２ , ..., x Ｎ The numerical ranges for each of the input data regions are shown by rectangles, and the numerical range of the inference model's output data y for this input data region is shown by rectangles. Rectangle C for the inference model's output data y is the expected output range, and rectangle D for the inference model's output data y is the output range corresponding to the selected violation region. By referring to these, the user can understand the degree of violation in the input data.

[0088] The sample acquisition unit 29-2 acquires one or more data samples included in the region acquired by the region acquisition unit 29-1. The display processing unit 28 displays the data samples acquired by the sample acquisition unit 29-2 on the display device 6. FIG. 12 is a diagram showing the acquisition result of a data sample from the data region of the expected output judgment result and an example of its display. For example, as shown in FIG. 12, the sample acquisition unit 29-2 acquires input variables x , which are parameters of a relational expression indicating one input data region out of multiple input data regions corresponding to a violation expected output judgment result. １ , x ２ , ..., x Ｎ 12, the data sample corresponding to the violation expected output judgment result displayed on the display device 6 is, for example, the input variable x １ takes a value between the lower limit of 0.2 and the upper limit of 0.5. In this way, the display device 6 displays multiple input data included in one input data area corresponding to the expected output judgment result of the violation. By referring to these, the user can understand the input data in the violation area.

[0089] The sample acquisition unit 29-2 acquires data samples when each variable defining the input data region takes a lower limit value or an upper limit value, as shown in Fig. 12. The sample acquisition unit 29-2 may also generate and acquire data samples from a uniform distribution within the input data region, acquire training data samples included in the input data region, or acquire data samples at the vertices of the input data region.

[0090] The region summarization unit 29-3 summarizes the multiple regions acquired by the region acquisition unit 29-1 into a smaller number of regions. The display processing unit 28 displays information indicating the regions summarized by the region summarization unit 29-3 on the display device 6. When the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees, the sample acquisition unit 29-2 acquires one or more data samples included in the region acquired by the region acquisition unit 29-1. The region summarization unit 29-3 learns a single decision tree using the data samples acquired by the sample acquisition unit 29-2, and summarizes the region using the leaf nodes of the learned decision tree.

[0091] For example, the example illustrates a case where violation regions are summarized in the learning of an inference model consisting of a single decision tree. Data samples acquired from violation regions are assigned a teacher label "1," while data samples acquired from pass regions or unknown regions are assigned a teacher label "0." Data samples acquired from unknown regions may also be labeled "1." The region summarization unit 29-3 collects multiple decision paths from the learning result decision tree model whose output data has a teacher label of "1," and identifies the corresponding region as a summary region. Hyperparameters may be set during decision tree learning to limit the depth of the decision tree, the maximum number of leaves, the number of data samples contained in each leaf, and so on. Because the summary provides simple information about the region, the user can easily identify the input data region corresponding to the expected output judgment result.

[0092] The region summarization unit 29-3 may perform learning by weighting the data samples acquired by the sample acquisition unit 29-2 with sample weights proportional to the region size of the input data region from which the data samples were acquired. This allows multiple input data regions corresponding to expected output determination results to be presented in summary regions that take region size into consideration, allowing the user to easily grasp the input data region that corresponds to the expected output determination result.

[0093] The region summarization unit 29-3 outputs the summarized region to the input region designation unit 26 as a new input data region. Using the expected output judgment result for the summarized region, the region proportion calculation unit 27 may calculate the proportion of regions in the summarized region that are either pass, violate, or unknown. For example, even if the proportion of violation regions in the original input data region is displayed as 0.8%, it is difficult to easily grasp the degree of violation in multiple input data regions including this region. Therefore, by summarizing multiple input data regions and displaying that the proportion of violation regions in the summarized region is 93.7%, the user can quantitatively understand the degree of violation in the summarized region.

[0094] Figure 13 is a schematic diagram showing an example of a summarized region display, showing the summarized region as a parallel category plot of region percentages. In Figure 13, the percentage of regions that are either pass, fail, or unknown is displayed as a band of categories, with the width of each category proportional to the region size. ５ Input variable x≦3.9 ５ The area defined by includes the pass area and the unknown area. ５ Input variable x ５ The regions defined by include pass regions, unknown regions, and violation regions.

[0095] 3.9<x ５ Input variable x ５ In the region defined by ８ Input variable x≦−0.4 ８ The area defined by includes the pass area, the unknown area, and the violation area. ８ Input variable x ８ The area defined by includes a pass area and an unknown area.

[0096] x ８ Input variable x≦−0.4 ８ In the region defined by ５ Input variable x≦5.2 ５ The area defined by 5.2<x includes only the violation area. In other words, the percentage of the violation area is 100%. ５Input variable x ５ The area defined by includes a pass area, an unknown area, and a violation area. The percentage of the violation area is 25%. By referring to this display, the user can easily understand the expected output judgment result for the area where the summary was performed.

[0097] As described above, the verification device 2A according to the second embodiment includes an area acquisition unit 29-1 that acquires the input data area determined by the expected output determination unit 24. The display processing unit 28 displays information indicating the acquired area on the display device 6. Since the input data area for each determination result of the expected output of the inference model is displayed, the user can grasp the degree of violation in the input data.

[0098] The verification device 2A according to the second embodiment includes a sample acquisition unit 29-2 that acquires one or more data samples contained in the region acquired by the region acquisition unit 29-1. The display processing unit 28 displays the acquired data samples on the display device 6. Since the data samples contained in the input data region for each judgment result of the expected output of the inference model are displayed, the user can grasp the degree of violation in the input data for each data sample.

[0099] The verification device 2A according to the second embodiment includes a region summarization unit 29-3 that summarizes the multiple regions acquired by the region acquisition unit 29-1 into a smaller number of regions. The display processing unit 28 displays information indicating the summarized regions on the display device 6. By displaying the regions where multiple input data regions have been summarized into a smaller number for each determination result of the expected output of the inference model, the user can easily grasp the degree of violation in the input data.

[0100] In the verification device 2A according to the second embodiment, the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees. The device includes a sample acquisition unit 29-2 that acquires one or more data samples contained in the acquired region. The region summarization unit 29-3 uses the acquired data samples to train a single decision tree and summarizes the region using the leaf nodes of the trained decision tree. Multiple input data regions for each judgment result of the expected output of the inference model are summarized into simple information, allowing the user to easily grasp the degree of violation in the input data.

[0101] It is possible to combine the embodiments, modify any of the components of the embodiments, or omit any of the components of the embodiments.

[0102] The verification device according to the present disclosure can be used, for example, to verify inference models for performing various inferences.

[0103] 1, 1A Inference system, 2, 2A Verification device, 3 Inference device, 4 Inference model storage device, 5 Judgment result storage device, 6 Display device, 21, 31 Inference model reading unit, 22 Output upper and lower bound calculation unit, 23 Expected output designation unit, 24 Expected output determination unit, 25 Input region division unit, 26 Input region designation unit, 27 Region proportion calculation unit, 28 Display processing unit, 29-1 Region acquisition unit, 29-2 Sample acquisition unit, 29-3 Region summarization unit, 32 Inference unit, 100 Input interface, 101 Output interface, 102 Processing circuit, 103 Processor, 104 Memory.

Claims

1. A verification device comprising: an expected output judgment unit that judges whether all of the output data of an inference model for an input data domain, which is a numerical range of data, is a violation judgment that is not the expected output; an input domain division unit that divides domains of the input data domain that are judged not to be a violation judgment and sets the divided domains as new input data domains of the inference model; a domain proportion calculation unit that calculates the domain proportion for each judgment result for the input data domain; and a verification result output unit that outputs verification result information indicating the domain proportion.

2. The verification device described in claim 1, characterized in that the expected output judgment unit judges either a pass judgment in which all of the output data of the inference model for the input data area is the expected output, a violation judgment in which all of the output data of the inference model for the input data area is not the expected output, or an unknown judgment in which none of the above applies.

3. The verification device described in claim 2, further comprising an output upper / lower bound calculation unit that calculates at least one of an upper bound or a lower bound of the output data of the inference model for the input data domain, and the expected output determination unit determines either the pass judgment, the violation judgment, or the unknown judgment for the output data domain indicated by the upper bound or lower bound of the output data of the inference model.

4. The verification device described in claim 3, characterized in that the inference model to be verified is a single decision tree or a decision tree ensemble model consisting of multiple decision trees, and the input region division unit divides the region according to the branching conditions of any of the decision trees included in the inference model.

5. The verification device according to claim 1, further comprising an area acquisition unit that acquires the input data area determined by the expected output determination unit, and the verification result output unit displays information indicating the acquired area on a display device.

6. The verification device according to claim 5, further comprising a sample acquisition unit that acquires one or more data samples contained in the area acquired by the area acquisition unit, and the verification result output unit displays the acquired data samples on the display device.

7. The verification device according to claim 5, further comprising an area summarization unit that performs summarization by combining multiple areas acquired by the area acquisition unit into a smaller number of areas, and the verification result output unit displays information indicating the summarized areas on the display device.

8. The verification device described in claim 7, characterized in that the inference model to be verified is a single decision tree or a decision tree ensemble model composed of multiple decision trees, and the device is equipped with a sample acquisition unit that acquires one or more data samples contained in the acquired region, and the region summarization unit learns a single decision tree using the acquired data samples and summarizes the region using the leaf nodes of the learned decision tree.

9. An inference device comprising: an inference model reading unit that reads the inference model verified by a verification device described in any one of claims 1 to 7; and an inference unit that performs inference using the inference model.

10. An inference system comprising: a verification device according to any one of claims 1 to 7; and an inference device according to claim 9.

11. A verification method using a verification device, comprising: a step in which an expected output judgment unit judges whether all of the output data of the inference model for an input data domain that is a numerical range of data is a violation judgment that is not the expected output; a step in which an input domain division unit divides an area of ​​the input data domain that is judged not to be a violation judgment and sets the divided area as a new input data domain of the inference model; a step in which an area proportion calculation unit calculates the area proportion for each judgment result for the input data domain; and a step in which a verification result output unit outputs verification result information indicating the area proportion.

Images