Verification device, inference device, inference system, and verification method

Abstract

This verification device (2) is provided with: a desired output determination unit (24) that determines whether or not there is a violation determination in which all output data from an inference model in an input data region is not a desired output; an input region division unit (25) that divides a region determined not to be a violation determination among the input data regions, and sets the divided region as a new input data region of the inference model; a region proportion calculation unit (27) that calculates the proportion of each calculation determination result region with respect to the input data region; and a display processing unit (28) that displays verification result information indicating the proportion of the region on a display device (6).

Description

Technical Field

[0001] This disclosure relates to a verification device, a reasoning device, a reasoning system, and a verification method. Background Technology

[0002] A technique for evaluating the reasonableness of an inference model as a machine learning model is proposed. For example, in Patent Document 1, a region containing at least one violation data sample with a violation verification property is suggested in the model output region relative to the data region input into the decision tree ensemble model.

[0003] Patent Document 1: Japanese Patent No. 7059220

[0004] Although the prior art described in Patent Document 1 suggests a range of at least one non-compliant data sample that is not the expected output of the inference model, it does not guarantee that all data samples within the suggested range are non-compliant data samples. Therefore, the prior art has the following problems: for example, if non-compliant data samples are not present in most areas of the suggested range, or if non-compliant data samples exist only in a very narrow area, the degree of non-compliance may be overestimated, making it difficult to properly address the non-compliance. Summary of the Invention

[0005] This disclosure is intended to solve the above-mentioned problems, and aims to provide a verification device that can indicate the proportion of the input data region for each decision result of the expected output of the reasoning model.

[0006] The verification apparatus disclosed herein includes: an expected output determination unit that determines whether all output data of the inference model relative to the input data region is not the expected output, wherein the input data region is a numerical range of data; an input region segmentation unit that segments the input data region into regions that are determined not to be violations and sets the segmented regions as new input data regions for the inference model; a region ratio calculation unit that calculates the ratio of each calculated determination result region relative to the input data region; and a verification result output unit that outputs verification result information representing the region ratio.

[0007] According to this disclosure, a violation determination is made to determine whether the output data of the inference model relative to the input data region is entirely different from the expected output. The regions within the input data region that are determined not to be violations are segmented, and these segmented regions are set as new input data regions for the inference model. The proportion of each calculated violation result region relative to the input data region is calculated, and verification result information representing the proportion of the regions is output. By outputting the proportion of each violation result region of the expected output of the inference model, the verification apparatus of this disclosure can indicate the proportion of the input data region for each violation result of the expected output of the inference model. Therefore, it is possible to suppress situations where the degree of violation is overestimated or underestimated. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating a configuration example of the reasoning system involved in Implementation Method 1.

[0009] Figure 2 It is a chart that represents the input data range and the expected output judgment result.

[0010] Figure 3 This is a summary diagram representing the segmentation of the input data region and the expected output judgment result.

[0011] Figure 4 This is a summary diagram representing a general outline of the expected output determination.

[0012] Figure 5 This is a diagram representing an example of the expected output judgment result.

[0013] Figure 6A , Figure 6B as well as Figure 6C This is a diagram showing an example of the verification results.

[0014] Figure 7 This is a flowchart illustrating the verification method involved in Implementation Method 1.

[0015] Figure 8 This is a flowchart representing the region determination process.

[0016] Figure 9A as well as Figure 9B This is a block diagram illustrating the hardware configuration that enables the functionality of the verification device described in Implementation Method 1.

[0017] Figure 10 This is a block diagram illustrating a configuration example of the reasoning system involved in Implementation Method 2.

[0018] Figure 11 This is a diagram showing the obtained results of the data area where the expected output judgment result is expected, and an example of its display.

[0019] Figure 12 It is a graph representing the results obtained from the data sample of the data area from which the judgment result is expected, and its display example.

[0020] Figure 13 This is a summary diagram showing a display example of a region that has been summarized. Detailed Implementation

[0021] Implementation method 1.

[0022] Figure 1 This is a block diagram illustrating a configuration example of the reasoning system 1 according to embodiment 1. Figure 1 In this system, inference system 1 is a system that uses a validated inference model to perform inference, and includes a validation device 2, an inference device 3, an inference model storage device 4, a decision result storage device 5, and a display device 6. The validation device 2 is used to validate the inference model, which serves as a machine learning model, and the inference device 3 is used to perform inference using the inference model. Inference system 1 is a system in which the validation device 2, inference device 3, inference model storage device 4, decision result storage device 5, and display device 6 are connected via wired signal lines or wireless communication networks. The network includes electrical communication lines such as the Internet.

[0023] The verification device 2 can suppress the overestimation or underestimation of the degree of violation by indicating the proportion of the input data region for each judgment result of the expected output of the inference model. For example, in the prior art described in Patent Document 1, the violation range containing at least one violation data sample that is not the expected output of the inference model is indicated as a verification result. Here, the expected output refers to a proposition that is expected to be true for the output data of the inference model.

[0024] Figure 2 It is a chart that represents the input data range and the expected output judgment result. Figure 2 The input data area shown is the data area defined by input variables x1 and x3. Figure 2 Region A shown is a data region that contains Region B. All data contained in Region B is not the expected output of the inference model; that is, it is a violation data sample that does not meet the verification conditions. Region B is a region that contains only violation data samples.

[0025] In the prior art described in Patent Document 1, there exists a proposed region A. This region A lacks the guarantee that all data samples within the region are non-compliant. For example, as... Figure 2 As shown, there may be no samples of non-compliant data in most areas, in which case the degree of non-compliance may be overestimated.

[0026] Since all data samples in region A may be included within the violation range, sometimes the output data of the inference model when data from region A is input is also judged as not the expected output, i.e., a violation that does not meet the verification conditions. In this case, although the output data of the inference model when data from a region in region A that does not contain violation data samples would have been the expected output, its inference is also suspended.

[0027] Additionally, sometimes the output of the inference model when data from region A is input is rewritten in a non-violation manner. In this case, even though the output of the inference model when data from a region containing no violation data samples is input would have been the expected output, it may still be rewritten to other values.

[0028] Additionally, sometimes, to ensure that the output of the inference model is no longer violating rules when data from region A is input, additional data samples from region A are added to relearn the inference model. In this case, the inference behavior of the inference model is significantly different when data from a region within region A that does not contain violating data samples is input.

[0029] In the case where the test is passed if the overvolume of the violation range output by the inference model is below the specified value when the data contained in region A is input, it may fail even if the overvolume of the violation range is below the specified value because the input data region contains a region where there are no violation data samples.

[0030] Therefore, the verification device 2 according to Embodiment 1 indicates the proportion of the input data region for each expected output decision result of the inference model by indicating the proportion of the region of each expected output decision result of the inference model. Thus, the verification device 2 can suppress undesirable situations caused by overestimating or underestimating the degree of violation as described above.

[0031] Figure 3 This is a summary diagram representing the segmentation of the input data region and the expected output decision result, showing the input data region of the inference model defined by input variables x1 and x3. Figure 3 In this context, an input data region containing only the input data whose output data is entirely the expected output of the inference model is considered a qualified region. An input data region containing only the input data whose output data is entirely different from the expected output and is therefore deemed a violation is considered a violation region. An input data region that is neither a qualified region nor a violation region is considered an unknown region.

[0032] Verification device 2 inputs the input data region into the inference model. It calculates the proportion of each calculated region relative to the input data region by determining whether all output data of the inference model is a violation that is not the expected output. The proportions of the regions at this point are as follows: Figure 3 As shown in the chart on the left, the proportion of qualified regions in the input data area to the inference model is 23.9%, while the proportion of unknown regions is 76.1%. Because it includes unknown regions—that is, input data areas that were not subject to violation checks—the verification device 2, as... Figure 3 The unknown region is divided as shown by the arrow, and the divided region is set as the new input data region for the inference model.

[0033] Next, verification device 2 sets each segmented region of the unknown area as a new input data region for the inference model. It calculates the ratio of each region to the input data region by determining whether the output data of the inference model is a violation judgment that is not the expected output. The region ratio at this time is, for example, as shown below. Figure 3 As shown in the central chart, the proportion of qualified regions in the input data area to the inference model is 46.7%, the proportion of unknown regions is 42.5%, and the proportion of illegal regions is 0.8%. In this case, the verification device 2 determines that it contains unknown regions, such as... Figure 3 The arrows in the image further divide these unidentified areas.

[0034] Next, verification device 2 sets the segmented regions of the newly segmented unknown regions as the new input data regions of the inference model. It calculates the ratio of each calculated region to the input data region by determining whether all output data of the inference model is a violation that is not the expected output. For example, the ratio of the regions is... Figure 3 As shown in the chart on the right, the proportion of qualified regions in the input data area of ​​the inference model is 98.3%, the proportion of unknown regions is 0%, and the proportion of illegal regions is 1.7%. Thus, the verification device 2 can indicate the proportion of regions in the input data area of ​​the inference model that correspond to the expected output judgment result, and can also determine the data value range of each region. Therefore, it can prevent the degree of violation from being evaluated too high or too low.

[0035] The inference model storage device 4 is a storage device for storing inference models. The inference model stored in the inference model storage device 4 is a machine learning model of the verification object for the verification device 2, and a machine learning model used for inference processing for the inference device 3. Furthermore, the inference model storage device 4 is located external to both the verification device 2 and the inference device 3. The inference model storage device 4 only needs to be accessible from both the verification device 2 and the inference device 3, and can also be a storage device possessed by the computer that functions as either the verification device 2 or the inference device 3.

[0036] The determination result storage device 5 is a storage device for the expected output determination results of the expected output determination unit 24 included in the verification device 2. The expected output determination result is a determination result of whether all the output data of the inference model relative to the input data region, which is the numerical range of the data, is not the expected output, thus determining a violation.

[0037] Furthermore, the determination result storage device 5 is located outside the verification device 2. The determination result storage device 5 can be accessed from the verification device 2, or it can be a storage device provided by the computer that functions as the verification device 2.

[0038] The verification device 2 is implemented, for example, by a computer having a communication unit, a computing unit, and a storage unit. The communication unit communicates with the inference model storage device 4 or the decision result storage device 5 via a wired signal line or a wireless communication-based network. For example, the communication unit is a communication device capable of mobile communication using communication methods such as LTE, 3G, 4G, or 5G. Alternatively, the communication unit can also be a short-range wireless communication mechanism such as Bluetooth (registered trademark).

[0039] Ministry of Communications includes Figure 9A as well as Figure 9B The input interface 100 and the output interface 101.

[0040] The arithmetic unit controls the overall operation of the verification device 2. The arithmetic unit includes an inference model input unit 21, an output upper and lower bound calculation unit 22, a desired output specification unit 23, a desired output determination unit 24, an input region segmentation unit 25, an input region specification unit 26, a region ratio calculation unit 27, and a display processing unit 28. The arithmetic unit executes an information processing application program for verifying the inference model to realize the various functions of the inference model input unit 21, the output upper and lower bound calculation unit 22, the desired output specification unit 23, the desired output determination unit 24, the input region segmentation unit 25, the input region specification unit 26, the region ratio calculation unit 27, and the display processing unit 28. The arithmetic unit includes... Figure 9A The processing circuit 102 and Figure 9B The processor 103.

[0041] The storage unit stores information used by the information processing applications and the computing unit for computation. The storage unit is a storage device included in the computer that functions as verification device 2, including storage devices such as HDDs or SSDs. Figure 9A as well as Figure 9B The memory 104, etc. The memory unit can be located outside the verification device 2 as long as it is accessible to the verification device 2.

[0042] The inference model reading unit 21 is used to read the inference model of the verification object from the inference model storage device 4. The inference model refers to the inference model relative to the N-dimensional input X = (x1, x2, ..., x...). N )∈R N Returns a one-dimensional output y∈R or an M-dimensional output Y = (y1, y2, ..., y3) M )∈R M The function f is defined by N and M, where N and M are integers. For example, inference models include trained machine learning models, neural networks, decision tree models, decision tree ensemble models, support vector machines (SVM), generalized linear models, generalized additive models (GAM), Gaussian process regression models (GPR), Naive Bayes, and Gaussian mixture models (GMM), etc.

[0043] Neural networks include multilayer perceptrons (MLP), convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformers. Decision tree ensemble models include random forests or gradient boosting trees. Generalized linear models include linear regression or logistic regression.

[0044] The output upper and lower bound calculation unit 22 calculates at least one of the upper or lower bounds of the output data of the inference model relative to the input data region. That is, the data where the lower bound of the inference model output is always true and the output data is always true if the lower bound is less than or equal to the upper bound.

[0045] For example, the method for calculating the upper bound is as follows.

[0046] When the inference model is a generalized linear model, the upper bound value is the output value calculated by the upper bound calculation unit 22 when the variable with a positive coefficient takes the upper limit value of the input data area and the variable with a negative coefficient takes the lower limit value of the input data area.

[0047] When the inference model is a single decision tree model, the upper and lower bound calculation unit 22 calculates the maximum output value in the set of leaf nodes that the data samples within the input data region can reach as the upper bound value.

[0048] When the inference model is a decision tree ensemble model, the upper and lower bound calculation unit 22 calculates the sum of the maximum output values ​​of each decision tree in the set of leaf nodes reachable by the data samples within the input data region as the upper bound value.

[0049] For example, the method for calculating the lower bound is as follows.

[0050] When the inference model is a generalized linear model, the lower bound value is the output value calculated by the upper and lower bound calculation unit 22 when the variable with a positive coefficient takes the lower limit value of the input data area and the variable with a negative coefficient takes the upper limit value of the input data area.

[0051] When the inference model is a single decision tree model, the output upper and lower bound calculation unit 22 calculates the minimum output value in the set of leaf nodes that the data samples within the input data region can reach as the lower bound value.

[0052] When the inference model is a decision tree ensemble model, the output upper and lower bound calculation unit 22 calculates the sum of the minimum output values ​​of each decision tree in the set of leaf nodes reachable by the data samples within the input data region as the lower bound value.

[0053] The expected output designation unit 23 sets the expected output value of the inference model for the expected output determination unit 24. The expected output refers to a proposition that the output data y of the inference model is expected to be true. For example, it can be listed that 50≤y<80 is specified as the range of regression output values, 0.5≤y is specified as the range of output scores (output of the sigmoid function) for binary classification, (y1<y2)&(y3<y2) is specified as the output category for multi-class classification, the case of output category 2 of the soft maxima function output is specified as the expected output, or the difference between the output y'=f'(X) of another inference model f' is |y-y'|≤0.1, etc.

[0054] For example, users use in Figure 1 The input device (not shown) allows users to input the desired output condition or upper / lower limit value via text input. The desired output specifying unit 23 then sets the desired output as the desired output to the desired output determining unit 24, representing the condition or upper / lower limit value input by the input device. Furthermore, the desired output specifying unit 23 instructs the display processing unit 28 to display a selection screen on the display device 6, either a selection screen for choosing a category ID via a dropdown or checkbox, or a selection screen for choosing an output range via a parallel coordinate graph. If the user performs a selection operation on the selection screen using the input device, the desired output specifying unit 23 accepts the selection operation information and sets it as the desired output to the desired output determining unit 24.

[0055] The expected output determination unit 24 determines whether the output data of the inference model relative to the input data region is all non-expected output, which constitutes a violation. Figure 4 This is a summary diagram representing the expected output determination, showing the case where the expected output of the regression is 50 ≤ y < 80. Figure 4 In this case, since all output data y included in the numerical range of (lower bound of y ≥ 50) & (upper bound of y < 80) are also included in the expected output range of 50 ≤ y < 80, the input data region of the inference model when the output data y is output is not judged as a violation, but is a qualified region.

[0056] Since all output data y included in the numerical range of (upper bound of y < 50) | (lower bound of y ≥ 80) are not included in the expected output range of 50 ≤ y < 80, the input data region of the inference model when the output data y is output is the violation region that is judged to be a violation.

[0057] The output data y is not included in any of the numerical ranges of (lower bound of y ≥ 50) & (upper bound of y < 80) and (upper bound of y < 50) | (lower bound of y ≥ 80). The output determination unit 24 determines that the output data y is contained in the unknown region.

[0058] When the expected output of binary classification is 1, that is, the output score y is 0.5≤y, since all output scores y included in the range of the expected output 0.5≤y are included in the range of the lower bound of the output score y ≥ 0.5, the input data region of the inference model when the output score y is output is a qualified region.

[0059] Since all output scores y included in the range of values ​​where the upper bound of the output score y is < 0.5 are not included in the expected output range of 0.5 ≤ y, the input data region for the inference model when the output score y is output is a violation region.

[0060] The output data y is not included in the range of scores that are neither the lower bound ≥ 0.5 nor the upper bound < 0.5. The output determination unit 24 determines that the output data y is included in the unknown region.

[0061] When the expected output of the three-class classification is category 2, i.e. (y1 < y2) & (y2 < y3), since all output data y included in the numerical range of (upper bound of y1 < lower bound of y2) & (upper bound of y3 < lower bound of y2) are also included in the range of (y1 < y2) & (y2 < y3) which is the expected output, the input data region of the inference model when the output data y is output is a qualified region.

[0062] Since all output data y included in the numerical range of (lower bound of y1 ≥ upper bound of y2) | (lower bound of y3 ≥ upper bound of y2) are not included in the expected output range (y1 < y2) & (y2 < y3), the input data region of the inference model when the output data y is output is a violation region.

[0063] The output data y is not included in any of the following numerical ranges: (upper bound of y1 < lower bound of y2) & (upper bound of y3 < lower bound of y2) and (lower bound of y1 ≥ upper bound of y2) | (lower bound of y3 ≥ upper bound of y2). The output determination unit 24 determines that the output data y is included in the unknown region.

[0064] When the expected output range is |y - y'| ≤ 0.1, which is the difference between the output y' = f'(X) of another inference model f', the input data region of the inference model when the output data y is output is a qualified region because all the output data y included in the numerical range of (upper bound of y - lower bound of y' ≤ 0.1) & (upper bound of y' - lower bound of y ≤ 0.1) are also included in the expected output range.

[0065] Since all output data y included in the range of (lower bound of y - upper bound of y' > 0.1) | (lower bound of y' - upper bound of y > 0.1) is not included in the expected output range | y - y' | ≤ 0.1, the input data region of the inference model when the output data y is output is a violation region.

[0066] The output data y is included in the range of values ​​that are not included in (upper bound of y - lower bound of y' ≤ 0.1) & (upper bound of y' - lower bound of y ≤ 0.1) and (lower bound of y - upper bound of y' > 0.1) | (lower bound of y' - upper bound of y > 0.1), which is considered a violation and is not the expected output. The expected output determination unit 24 determines that the output data y is included in the unknown region.

[0067] The determination result information of the expected output determination unit 24 is stored in the determination result storage device 5.

[0068] Figure 5 This diagram illustrates an example of the expected output judgment result, showing the storage contents of the judgment result storage device 5. For example... Figure 5 As shown, the judgment result information includes the input data region and the expected output judgment result relative to each input data. The input data region consists of input variables x1, x2, ..., xn, which are parameters of the relational expression representing the input data. N Each has its own lower and upper limit defining the data range. The expected output result is qualified, non-compliant, or unclear.

[0069] Although the input data corresponding to all the judgment results is shown, the judgment result storage device 5 can also store only specific judgment results as the expected output judgment result. For example, it can store only the input data for which the expected output judgment result is a violation.

[0070] The input region segmentation unit 25 segments the input data region into regions that are determined not to be violations, and sets the segmented regions as new input data regions for the inference model. For example, the input region segmentation unit 25 may segment the input data region according to the branches of the decision tree contained in the inference model. In this case, the input region segmentation unit 25 segments the data region corresponding to each leaf node of the decision tree and the overlapping region of the input data region. At this time, the input region segmentation unit 25 may consider the limitations of the input data region and ignore unreachable leaf nodes, or it may stop segmenting at intermediate branches rather than terminal leaf nodes.

[0071] Furthermore, the input region segmentation unit 25 can randomly segment the input data region. For example, the input region segmentation unit 25 randomly selects an input variable and determines the segmentation point within the upper and lower limits of that input variable. Also, the input region segmentation unit 25 can segment the input data region based on the midpoint of the lower limit generated according to a uniform distribution, the mean, median, or quantile of the learning data.

[0072] The input region segmentation unit 25 can also weight the selection of input variables to be segmented based on the importance of the feature quantities of the inference model.

[0073] Furthermore, the input region segmentation unit 25 can segment the input data region using multiple segmentation candidates and select the segmentation index as the optimal segmentation. For example, the input region segmentation unit 25 may select the segmentation with the smallest difference between the upper and lower bounds of the inference model output for the segmented region.

[0074] Alternatively, the input region segmentation unit 25 may use a segmentation method where the sum of the sizes of the segmented unknown regions is the smallest.

[0075] The input region segmentation unit 25 can recursively continue segmenting until there are no more unknown regions, or it can terminate the segmentation midway. For example, the input region segmentation unit 25 may stop segmenting if the processing time is more than a predetermined value, the number of segmented regions is more than a predetermined value, the size of the unknown region is less than a predetermined value, or the size of the unknown region is less than the size of the violation region.

[0076] The input region designation unit 26 sets the input data region to the output upper and lower bound calculation unit 22. The input data region is a data region in the N-dimensional space of the inference model of the verification object. N is an integer. For example, the input data region is table data defined by the input variable x. The input data region may also be feature quantities extracted from images, videos, text, sound, charts, or time series data using neural networks or the like.

[0077] The input data region can also be, for example, S = {X = (x1, x2, ..., xn)}, representing a hypercube. N )∈R N| -3.5≤x1≤5.2,…,15≤x N ≤40}, representing the hypersphere S = {X∈R} N ||X| 2 ≤r 2}, representing the product set of multiple regions S = S1 ∩ S2, or representing the sum set of multiple regions S = S1 ∪ S2.

[0078] For example, users use in Figure 1 The input device (not shown) inputs the upper and lower limits of the conditional expression or hypercube of the input data area as text. Therefore, the input area designator 26 sets the information representing the upper and lower limits of the conditional expression or hypercube input by the input device as the input data area to the output upper and lower limit calculation unit 22. Furthermore, the input area designator 26 instructs the display processing unit 28 to display a selection screen on the display device 6 for selecting the range of input variables in two-dimensional space using a rectangle or lasso, or for selecting the range of each input variable on a parallel coordinate graph. If the user performs a selection operation on the selection screen using the input device, the input area designator 26 accepts the selection operation information and sets the selection operation information as the input data area to the output upper and lower limit calculation unit 22.

[0079] The region proportion calculation unit 27 calculates the proportion of each region for which a desired output judgment result is desired relative to the input data region. For example, the region proportion calculation unit 27 can calculate the proportion using the hypervolume of the input data region. When the input data region is a hypercube, the region proportion calculation unit 27 can calculate the proportion of the region using the product of the lengths of the sides of the hypercube. Alternatively, the region proportion calculation unit 27 can also calculate the proportion of each region for which a desired output judgment result is desired relative to the input data region using the number of data samples contained in the region of a given dataset.

[0080] The region proportion calculation unit 27 can also use the number of generated data samples contained in the region to calculate the proportion of each region with the desired output judgment result relative to the input data region.

[0081] Data samples can be generated based on a uniform distribution, or by perturbing the data samples of a given dataset. They can also be generated based on a data distribution learned from a given dataset, or by using a generative adversarial network (GAN) learned from a given dataset.

[0082] If the dimensions of the violation area, the qualified area, and the unclear area are respectively set as |S 违规 |、|S 合格 |、|S 不明 |, then the overall size of the input region is determined by |S 整体 | = |S 违规|+|S 合格 |+|S 不明 | Obtained. In this case, the regional proportion calculation unit 27 calculates, for example, |S 违规 | / |S 整体 |、|S 合格 | / |S 整体 |、|S 不明 | / |S 整体 |、|S 违规 | / (|S) 违规 |+|S 合格 |), or (|S) 违规 |+|S 不明 |) / |S 整体 | As a proportion of the region.

[0083] The display processing unit 28 is a verification result output unit that outputs verification result information on the proportion of the display area. The display processing unit 28 displays the verification result information on the display device 6. The display device 6 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display device.

[0084] The display processing unit 28 outputs display control information for displaying verification result information to the display device 6. The display device 6 displays the verification result information based on the display control information. For example, it displays the proportion of the area using bar charts, pie charts, stacked bar charts, or color variations.

[0085] Figure 6A This is a graph showing examples of verification results, with bar charts indicating the proportions of the regions. Figure 6A In the diagram, the proportion of areas is represented by the length of the bars. The proportion of qualified areas is 32.3%, the proportion of unclear areas is 51.3%, and the proportion of non-compliant areas is 16.4%.

[0086] Figure 6B This is a graph showing examples of verification results, using color shades to indicate the proportion of the area. Figure 6B In the code, the proportion of the violation area is represented by the shade of color. For example, the larger the proportion of the violation, the darker the color.

[0087] The display processing unit 28 can also update the display ratio of the area that changes according to the division of the input data area in real time. For example, the display processing unit 28 can reduce the display of unknown areas monotonically over time, and can also increase the display of qualified areas and illegal areas monotonically.

[0088] In addition, the display processing unit 28 can also display the time-varying changes of the area's proportion on the display device 6 through line graphs or animations.

[0089] Furthermore, the display processing unit 28 can also display the specified input data area and the desired output on the display device 6 together. For example, the display processing unit 28 can display the proportion of the area on the display device 6 as a parallel coordinate graph, a two-dimensional chart, or text.

[0090] Figure 6C This is a graph showing an example of the verification results, using a parallel coordinate plot and text to indicate the proportions of the regions. Figure 6C In the diagram, the input variables x1, x2, ..., x3 represent the specified input data area using rectangles. N Each has its own numerical range, represented by a rectangle indicating the numerical range of the output data y of the inference model relative to the input data region.

[0091] Furthermore, the display processing unit 28 can, as shown in Figure 6, process the input variables x1, x2, ..., x... N The respective numerical ranges and the numerical range of the output data y are displayed as text on the display device 6.

[0092] exist Figure 1 The verification device 2 is shown to have a configuration including an inference model reading unit 21, an output upper and lower bound calculation unit 22, a desired output specifying unit 23, a desired output determination unit 24, an input region segmentation unit 25, an input region specifying unit 26, a region ratio calculation unit 27, and a display processing unit 28, but it is not limited to this. The verification device 2 only needs to be able to determine the desired output of the inference model, segment the input data region input to the inference model, and output the determination result. Therefore, the configuration other than the desired output determination unit 24, the input region segmentation unit 25, and the display processing unit 28 can also be equipped by an external device that can be accessed from the verification device 2.

[0093] Furthermore, although the case of an input data region being a two-dimensional space is shown, the input data region is usually a high-dimensional space of two dimensions or more.

[0094] The inference device 3 is implemented, for example, by a computer having a communication unit, a computing unit, and a storage unit. The communication unit communicates with the inference model storage device 4 via a wired signal line or a wireless communication-based network. For example, the communication unit is a communication device capable of mobile communication using communication methods such as LTE, 3G, 4G, or 5G. Alternatively, the communication unit can also be a short-range wireless communication mechanism such as Bluetooth (registered trademark).

[0095] The computation unit controls the overall operation of the inference device 3. The computation unit includes an inference model reading unit 31 and an inference unit 32. The computation unit implements various functions of the inference model reading unit 31 and the inference unit 32 by executing an information processing application program for performing inference.

[0096] The storage unit stores information used by information processing applications and by the arithmetic unit for computation. The storage unit is the storage device within a computer that functions as the inference device 3, including storage devices such as HDDs or SSDs. Figure 9A as well as Figure 9B The memory 104, etc. The storage unit can be any storage unit that can be accessed by the inference device 3, or it can be located outside the inference device 3.

[0097] The inference model reading unit 31 is used to read the inference model verified by the verification device 2 from the inference model storage device 4. The inference model read by the inference model reading unit 31 is set to the inference unit 32.

[0098] The reasoning unit 32 uses the reasoning model read in by the reasoning model reading unit 31 to perform reasoning. For example, the reasoning unit 32 can conduct experiments on complex reasoning models. In this case, in order to reveal the risk of defects such as loopholes in the complex reasoning model during the pre-application experiment, the reasoning unit 32 sets the properties that are minimum expected to be satisfied as the input data area and the desired output to the reasoning model, and then performs reasoning.

[0099] For example, the reasoning unit 32 will use the area within a 5-minute walk of the station and within the last 10 years of construction as the input data area, and assume that the expected output of the reasoning model relative to this input data area is a rent of 80,000 yen or more, to infer the rent of the apartment.

[0100] In addition, the inference unit 32 takes blood pressure above 135 and LDL cholesterol above 140 as the input data region, and assumes that the expected output of the inference model relative to the input data region is that an inspection is required, in order to infer whether an inspection is required.

[0101] The inference unit 32 can also use the range of values ​​representing products with measurement errors larger than those of abnormal samples as the input data area, and use the expected output that all products with measurement errors larger than those of abnormal samples are abnormal, to infer the abnormality of the products.

[0102] Specifically, for the reasoning device 3, if the proportion of the violation area is less than the specified value, it is deployed to the actual environment and put into use.

[0103] The reasoning unit 32 can infer the fidelity of the proxy model of an approximately complex reasoning model.

[0104] For example, the inference device 3 approximates a single decision tree model with a high probability of explanation and enables the user to identify the actions during inference. The verification device 2 sets the expected output as the case where the outputs of the original inference model and the surrogate model are consistent, and calculates the proportion of violation areas where the outputs of the original inference model and the surrogate model are inconsistent. Then, the verification device 2 considers the degree to which the proportion of violation areas is small as the fidelity of the surrogate model and displays it on the display device 6.

[0105] The reasoning of an existing rule-based system can also be replaced by a reasoning model validated by validation device 2. For example, an existing system can be replaced by a machine learning model with high prediction accuracy.

[0106] Verification device 2 sets the expected output as the case where the output of the inference model is consistent with that of the existing system, and calculates the proportion of the violation area where the output of the inference model is inconsistent with that of the existing system.

[0107] Then, the inference device 3 performs the replacement using an inference model where the proportion of the violation area is less than the specified value.

[0108] The difference between two different versions of inference models can also be quantified using verification device 2. For example, the difference between inference model A learned from data of one period and inference model B learned from data of another period can be quantified. Verification device 2 sets the expected output as the case where the outputs of inference model A and inference model B are consistent, and calculates the proportion of violation areas where the outputs of inference model A and inference model B are inconsistent. If the proportion of violation areas is less than a predetermined value, the difference between model A and model B is considered small. Thus, for example, in a system that continuously collects and learns data to update the inference model, the model can be updated only when the difference between the inference models is small. Furthermore, by highlighting violation areas, it is possible to know in which areas the model differs, for example, to detect and respond to environmental changes such as data shifts as early as possible.

[0109] Next, the verification method involved in Implementation Method 1 will be described.

[0110] Figure 7 This is a flowchart illustrating the verification method involved in Implementation Method 1.

[0111] The inference model reading unit 21 reads the inference model from the inference model storage device 4 and sets it to the output upper and lower bounds calculation unit 22. The input region specifying unit 26 sets the input data region to the output upper and lower bounds calculation unit 22. The desired output specifying unit 23 sets the desired output of the inference model to the desired output determination unit 24. This series of processes constitutes step ST1.

[0112] The expected output determination unit 24 clears the region list and adds the set input data regions (step ST2). The region list is a list of data regions input into the inference model. The expected output determination unit 24 inputs the input data regions registered in the region list into the inference model.

[0113] Next, the expected output determination unit 24 performs expected output determination processing on each input data region in the region list (step ST3).

[0114] Next, the expected output determination unit 24 determines whether there is a region that is either a violation region where all output data of the inference model relative to the input data region is not the expected output or a qualified region where all output data is the expected output, i.e., whether there is an unknown region (step ST4).

[0115] If an unknown region exists (step ST4; Yes), the input region segmentation unit 25 segments the unknown region (step ST5). The expected output determination unit 24 clears the region list and adds the segmented region as a new input data region for the inference model (step ST6). Next, the expected output determination unit 24 moves to step ST3 to determine the expected output of the new input data region, and repeats the subsequent processing.

[0116] If there is no unknown area (step ST4; no), the area ratio calculation unit 27 calculates the ratio of each area with the desired output judgment result to the input data area. Then, the display processing unit 28 displays the verification result information representing the area ratio on the display device 6 (step ST7).

[0117] Thus, the verification device 2 can indicate the proportion of the input data region for each decision result of the expected output of the inference model.

[0118] Figure 8 This is a flowchart illustrating the expected output determination process. Figure 7 The detailed processing of step ST3.

[0119] The output upper and lower bound calculation unit 22 calculates the upper and lower bounds of the output data of the inference model relative to the input data region (step ST1A). The output upper and lower bound calculation unit 22 sets the calculated upper and lower bounds of the output data to the expected output determination unit 24.

[0120] The expected output determination unit 24 determines whether the input data region corresponding to the expected output specified by the expected output designation unit 23 and the output data defined by the upper and lower bounds of the inference model is a violation, i.e., a violation, a pass, or an unknown (step ST2A).

[0121] Next, the expected output determination unit 24 stores the input data area and the corresponding expected output determination result in the determination result storage device 5 (step ST3A).

[0122] Next, the hardware configuration for implementing the functions of verification device 2 will be described.

[0123] The functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, desired output specifying unit 23, desired output determination unit 24, input region segmentation unit 25, input region specifying unit 26, region ratio calculation unit 27, and display processing unit 28 provided in the verification device 2 are implemented by a processing circuit. That is, the verification device 2 has the functions for performing... Figure 7 The processing circuit shown in steps ST1 to ST7 can be dedicated hardware or a CPU (Central Processing Unit) that executes a program stored in memory.

[0124] Figure 9A This is a block diagram showing the hardware configuration that enables the functionality of verification device 2. Figure 9B This is a block diagram illustrating the hardware configuration that executes the software that implements the functions of verification device 2. Figure 9A as well as Figure 9B In this interface, input interface 100 is used to relay data obtained by the verification device 2 from the inference model storage device 4 or the decision result storage device 5. Output interface 101 is used to relay data output from the verification device 2 to the decision result storage device 5.

[0125] In the processing circuit Figure 9A In the case of the dedicated hardware processing circuit 102 shown, the processing circuit 102 is, for example, equivalent to a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a circuit composed of combinations thereof. The functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the desired output specifying unit 23, the desired output determining unit 24, the input region segmentation unit 25, the input region specifying unit 26, the region ratio calculation unit 27, and the display processing unit 28 provided in the verification device 2 can be implemented by individual processing circuits, or they can all be implemented by a single processing circuit.

[0126] In the processing circuit Figure 9BIn the case of the processor 103 shown, the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the desired output specifying unit 23, the desired output determining unit 24, the input region segmentation unit 25, the input region specifying unit 26, the region ratio calculation unit 27, and the display processing unit 28 provided by the verification device 2 are implemented by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 104.

[0127] The processor 103 implements the functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, expected output specifying unit 23, expected output determination unit 24, input region segmentation unit 25, input region specifying unit 26, region ratio calculation unit 27, and display processing unit 28 of the verification device 2 by reading and executing the program stored in the memory 104. For example, the verification device 2 includes a memory 104 for storing the results of execution by the processor 103. Figure 7 The program shown describes the processing steps ST1 to ST7. These programs cause the computer to execute the processing steps or methods performed by the inference model reading unit 21, the output upper and lower bound calculation unit 22, the desired output specifying unit 23, the desired output determining unit 24, the input region segmentation unit 25, the input region specifying unit 26, the region ratio calculation unit 27, and the display processing unit 28. The memory 104 may also be a computer-readable storage medium storing programs for enabling the computer to function as the inference model reading unit 21, the output upper and lower bound calculation unit 22, the desired output specifying unit 23, the desired output determining unit 24, the input region segmentation unit 25, the input region specifying unit 26, the region ratio calculation unit 27, and the display processing unit 28.

[0128] Memory 104 includes, for example, non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically-EPROM) (registered trademark), as well as disks, floppy disks, optical disks, compressed optical disks, mini disks, DVDs, etc.

[0129] Alternatively, some of the functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, desired output designation unit 23, desired output determination unit 24, input region segmentation unit 25, input region designation unit 26, region ratio calculation unit 27, and display processing unit 28 of the verification device 2 can be implemented using dedicated hardware, while other parts can be implemented using software or firmware. For example, the functions of the inference model reading unit 21, output upper and lower bound calculation unit 22, desired output designation unit 23, and input region designation unit 26 can be implemented by the processing circuit 102, which is dedicated hardware, and the functions of the desired output determination unit 24, input region segmentation unit 25, region ratio calculation unit 27, and display processing unit 28 can be implemented by the processor 103 reading and executing the program stored in the memory 104. In this way, the processing circuit can implement the above functions through hardware, software, firmware, or a combination thereof.

[0130] As described above, the verification device 2 according to Embodiment 1 includes: a desired output determination unit 24, which determines whether a violation determination is made if all output data of the inference model relative to the input data region is not the desired output, where the input data region is the numerical range of data input to the inference model of the verification object; an input region segmentation unit 25, which segments the regions in the input data region that are determined not to be violation determinations, and sets the segmented regions as new input data regions for the inference model; a region ratio calculation unit 27, which calculates the ratio of each calculated determination result region relative to the input data region; and a display processing unit 28, which displays verification result information representing the region ratio on a display device 6. By using the ratio of the region of each determination result of the desired output of the inference model, the verification device 2 can indicate the ratio of the input data region of each determination result of the desired output of the inference model. This helps to prevent the degree of violation from being evaluated too high or too low.

[0131] In the prior art described in Patent Document 1, the input data region is divided into multiple regions, and a satisfiability determiner is used to determine whether any of the divided regions are unsatisfiable. If the satisfiability determiner determines that a region is satisfiable (i.e., exhibits satisfiability), at least one data sample within the divided region is a data sample deemed to be non-compliant. Therefore, the prior art described in Patent Document 1 may propose regions that partially include data samples deemed to be non-compliant. In contrast, the verification device 2, by dividing the region into regions that include those deemed to be entirely free of non-compliant output data relative to the input data region (i.e., regions deemed compliant) and those deemed non-compliant, can indicate the proportion of the input data region for each determination result.

[0132] In the verification apparatus 2 according to Embodiment 1, the expected output determination unit 24 determines any one of the following: a pass determination that all output data of the inference model relative to the input data area is the expected output, a violation determination that all output data of the inference model relative to the input data area is not the expected output, or an undetermined determination that does not fall into any of these categories. Thus, the verification apparatus 2 can indicate the proportion of the input data area for each determination result of the expected output of the inference model.

[0133] In the verification apparatus 2 according to Embodiment 1, an output upper and lower bound calculation unit 22 is provided, which calculates at least one of the upper or lower bound of the output data of the inference model relative to the input data region. The expected output determination unit 24 determines whether the output data region indicated by the upper or lower bound in the output data of the inference model is a pass / fail determination, a violation determination, or an unclear determination. Thus, the verification apparatus 2 can indicate the proportion of the input data region for each determination result of the expected output of the inference model.

[0134] In the verification apparatus 2 according to Embodiment 1, the reasoning model of the verification object is a decision tree ensemble model composed of a single decision tree or multiple decision trees. The input region segmentation unit 25 segments the region according to the branching conditions of any decision tree contained in the reasoning model.

[0135] Thus, the verification device 2 is able to determine the expected output of the reasoning model for each segmented region.

[0136] In the inference apparatus 3 according to Embodiment 1, there are: an inference model reading unit 31 for reading inference models verified by the verification unit 2; and an inference unit 32 for performing inference using the inference model. Since the proportion of the input data area of ​​each judgment result of the expected output of the inference model can be indicated, the degree of violation can be suppressed from being evaluated too high or too low, so the inference apparatus 3 can perform inference with high accuracy.

[0137] The inference system 1 according to Embodiment 1 includes the verification device 2 and the inference device 3 described above. It provides an inference system capable of indicating the proportion of the input data region for each decision result that corresponds to the expected output of the inference model. This prevents situations where the degree of violation is evaluated too high or too low.

[0138] The verification method according to Embodiment 1 includes the following steps: A desired output determination unit 24 determines whether a violation is found where all output data of the inference model relative to the input data region is not the desired output; an input region segmentation unit 25 segments the regions in the input data region that are determined not to be violations, and sets the segmented regions as new input data regions for the inference model; a region ratio calculation unit 27 calculates the ratio of each calculated determination result region relative to the input data region; and a display processing unit 28 displays verification result information indicating the region ratio on a display device 6. By executing the above method, the verification device 2 can display the ratio of the input data region for each determination result of the desired output of the inference model. This prevents the degree of violation from being evaluated too high or too low.

[0139] Implementation method 2.

[0140] Figure 10 This is a block diagram illustrating a configuration example of the reasoning system 1A according to Embodiment 2. Figure 10 In this system, inference system 1A is a system that performs inference using a validated inference model, and includes a validation device 2A, an inference device 3, an inference model storage device 4, a decision result storage device 5, and a display device 6. The validation device 2A is used to validate the inference model, which serves as a machine learning model, and the inference device 3 is used to perform inference using the inference model. Inference system 1 is a system in which the validation device 2, inference device 3, inference model storage device 4, decision result storage device 5, and display device 6 are connected via wired signal lines or a wireless communication network. The network includes electrical communication lines such as the Internet.

[0141] In addition to indicating the region proportion of each decision result in the expected output of the inference model, the verification device 2A also indicates the region itself, the data samples contained in the region, or summary information of multiple regions. The verification device 2A is implemented, for example, by a computer equipped with a communication unit, a computing unit, and a storage unit. The communication unit communicates with the inference model storage device 4 or the decision result storage device 5 via a wired signal line or a wireless communication-based network. For example, the communication unit is a communication device capable of mobile communication methods such as LTE, 3G, 4G, or 5G. Alternatively, the communication unit can also be a short-range wireless communication mechanism such as Bluetooth (registered trademark). The communication unit includes... Figure 9A as well as Figure 9B The input interface 100 and the output interface 101.

[0142] The arithmetic unit controls the overall operation of the verification device 2A. The arithmetic unit includes an inference model input unit 21, an output upper and lower bound calculation unit 22, a desired output specification unit 23, a desired output determination unit 24, an input region segmentation unit 25, an input region specification unit 26, a region ratio calculation unit 27, a display processing unit 28, a region acquisition unit 29-1, a sample acquisition unit 29-2, and a region summarization unit 29-3. The arithmetic unit executes an information processing application program for verifying the inference model to realize the various functions of the inference model input unit 21, the output upper and lower bound calculation unit 22, the desired output specification unit 23, the desired output determination unit 24, the input region segmentation unit 25, the input region specification unit 26, the region ratio calculation unit 27, the display processing unit 28, the region acquisition unit 29-1, the sample acquisition unit 29-2, and the region summarization unit 29-3. The arithmetic unit includes... Figure 9A The processing circuit 102 and Figure 9B The processor 103.

[0143] The storage unit stores information used by the information processing applications and the computing unit for computation. The storage unit is a storage device included in the computer that functions as verification device 2, including storage devices such as HDDs or SSDs. Figure 9A as well as Figure 9B The memory 104, etc. The memory unit can be located outside the verification device 2A, as long as it is accessible to the verification device 2A.

[0144] Among them, the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output specifying unit 23, the expected output determination unit 24, the input region segmentation unit 25, the input region specifying unit 26, the region ratio calculation unit 27, and the display processing unit 28 are... Figure 1 Similarly, repeated explanations are omitted.

[0145] The region acquisition unit 29-1 is used to acquire the input data region determined by the expected output determination unit 24 from the determination result storage device 5. The display processing unit 28 displays information representing the region acquired by the region acquisition unit 29-1 on the display device 6.

[0146] Figure 11 This is a diagram showing the obtained data area and its display example, representing the expected output judgment result. For example... Figure 11 As shown, the information representing multiple input data regions consists of the input variables x1, x2, ..., x3, which serve as parameters in the relational expression representing each input data region. N Information that represents the lower and upper limits of each. For example, the region acquisition unit 29-1 acquires information related to the expected output judgment result of violation from among the information representing multiple input data regions corresponding to the expected output judgment results of qualified, unqualified, and unclear results.

[0147] The display processing unit 28 displays multiple input data areas corresponding to the expected output judgment result of the violation on the display device 6 using a parallel coordinate graph. Figure 11 In the diagram, rectangles represent any one of the multiple input data regions corresponding to the expected output judgment results of the rules and violations, where the input variables x1, x2, ..., x are given. N The numerical range of each input data region is represented by a rectangle, which indicates the numerical range of the output data y of the inference model relative to that input data region.

[0148] The rectangle C representing the output data y of the inference model is the expected output range, and the rectangle D representing the output range corresponding to the selected violation region. By referring to these, users can understand the degree of violation in the input data.

[0149] The sample acquisition unit 29-2 acquires one or more data samples contained in the region acquired by the region acquisition unit 29-1. The display processing unit 28 displays the data samples acquired by the sample acquisition unit 29-2 on the display device 6.

[0150] Figure 12 This is a graph representing the results obtained from data samples within the data region from which the desired output judgment result is expected, and a display example thereof. For example, as shown... Figure 12 As shown, the sample acquisition unit 29-2 acquires the input variables x1, x2, ..., x... N The lower and upper limits of each variable are used as data samples, and the input variables are x1, x2, ..., x. N These are the parameters representing the relationship between one of the multiple input data regions corresponding to the expected output judgment result of the violation.

[0151] exist Figure 12 In this context, for the data sample displayed on display device 6 corresponding to the expected output judgment result of the violation, for example, the input variable x1 takes a value between the lower limit of 0.2 and the upper limit of 0.5. Thus, display device 6 displays multiple input data contained in an input data area corresponding to the expected output judgment result of the violation. By referring to these, the user can understand the input data of the violation area.

[0152] like Figure 12 As shown, the sample acquisition unit 29-2 acquires data samples when each variable in the input data region is given a lower or upper limit value.

[0153] In addition, the sample acquisition unit 29-2 can generate and acquire data samples based on a uniform distribution within the input data region, or it can acquire learning data samples contained in the input data region, or it can acquire data samples of the vertices of the input data region.

[0154] The region aggregation unit 29-3 aggregates multiple regions acquired by the region acquisition unit 29-1 into a smaller number of regions. The display processing unit 28 displays information indicating the regions aggregated by the region aggregation unit 29-3 on the display device 6.

[0155] When the reasoning model of the verification object is a decision tree ensemble model consisting of a single decision tree or multiple decision trees, the sample acquisition unit 29-2 acquires one or more data samples contained in the region acquired by the region acquisition unit 29-1.

[0156] The region aggregation unit 29-3 uses data samples obtained by the sample acquisition unit 29-2 to learn a single decision tree, and uses the leaf nodes of the learned decision tree to aggregate regions.

[0157] For example, the example of summarizing violation regions during the learning of an inference model composed of a single decision tree is shown. Data samples obtained from violation regions are assigned a teaching label "1", while data samples obtained from qualified or unknown regions are assigned a teaching label "0". Additionally, data samples obtained from unknown regions may contain a teaching label "1". The region summarization unit 29-3 collects multiple decision paths with a teaching label of "1" for the output data in the decision tree model from which the learning results are obtained, and sets the corresponding regions as summarization regions.

[0158] Hyperparameters can also be set during decision tree learning to limit the depth of the decision tree, the maximum number of leaves, the number of data samples contained in a leaf, etc.

[0159] Because the information is presented in a simplified manner through summarization, users can easily grasp the input data area that corresponds to the desired output judgment result.

[0160] The region aggregation unit 29-3 can learn by weighting the data samples acquired by the sample acquisition unit 29-2 with sample weights proportional to the region size of the input data region that serves as the source of the data samples. Therefore, since multiple input data regions corresponding to the desired output judgment result can be displayed in the aggregation region that takes into account the region size, the user can easily grasp the input data region corresponding to the desired output judgment result.

[0161] The region aggregation unit 29-3 outputs the aggregated region as a new input data region to the input region designation unit 26. Using the expected output judgment result regarding the aggregated region, the region ratio calculation unit 27 calculates the proportion of any one of the aggregated regions—qualified, non-compliant, or unclear. For example, even if the proportion of non-compliant regions in the original input data region is displayed as 0.8%, it is not easy to grasp the degree of non-compliance among multiple input data regions containing that region. Therefore, by aggregating multiple input data regions, the proportion of non-compliant regions in the aggregated region is displayed as 93.7%, allowing the user to quantitatively understand the degree of non-compliance in the aggregated region.

[0162] Figure 13 This is a summary diagram showing examples of summarized regions, representing summarized regions as parallel category diagrams showing region proportions. Figure 13 In this context, the proportion of any area that is qualified, non-compliant, or unclear is displayed as a band-shaped category, with the display width of each category proportional to the area size. The area defined by the input variable x5 (x5 ≤ 3.9) includes qualified and unclear areas. Conversely, the area defined by the input variable x5 (3.9 < x5) includes qualified, unclear, and non-compliant areas.

[0163] Within the region defined by the input variable x5 (3.9 < x5), the region defined by the input variable x8 (x8 ≤ -0.4) includes qualified regions, unclear regions, and non-compliant regions. Conversely, the region defined by the input variable x8 (-0.4 < x8) includes qualified regions and unclear regions.

[0164] Within the region defined by the input variable x8 (x8 ≤ -0.4), the region defined by the input variable x5 (x5 ≤ 5.2) contains only non-compliant regions. That is, the proportion of non-compliant regions is 100%. On the other hand, the region defined by the input variable x5 (5.2 < x5) contains compliant regions, unclear regions, and non-compliant regions. The proportion of non-compliant regions is 25%.

[0165] By referring to this display, users can easily understand the expected output judgment results in the summarized area.

[0166] As described above, the verification device 2A according to Embodiment 2 includes a region acquisition unit 29-1, which acquires the input data region determined by the expected output determination unit 24. The display processing unit 28 displays information representing the acquired region on the display device 6. Since the input data region of each determination result of the expected output of the inference model is displayed, the user can grasp the degree of violation in the input data.

[0167] The verification apparatus 2A according to Embodiment 2 includes a sample acquisition unit 29-2, which acquires one or more data samples contained in a region acquired by the region acquisition unit 29-1. The display processing unit 28 displays the acquired data samples on the display device 6. Since the data samples contained in the input data region of each determination result of the expected output of the inference model are displayed, the user can grasp the degree of violation in the input data according to each data sample.

[0168] The verification device 2A according to Embodiment 2 includes a region aggregation unit 29-3, which aggregates multiple regions acquired by the region acquisition unit 29-1 into a smaller number of regions. The display processing unit 28 displays information indicating the aggregated regions on the display device 6. By displaying the aggregated multiple input data regions of each judgment result of the expected output of the inference model into a smaller number of regions, the user can easily grasp the degree of violation in the input data.

[0169] In the verification apparatus 2A according to Embodiment 2, the reasoning model of the verification object is a decision tree ensemble model composed of a single decision tree or multiple decision trees. A sample acquisition unit 29-2 is provided to acquire one or more data samples contained in the acquired region. A region aggregation unit 29-3 uses the acquired data samples to learn a single decision tree and uses the leaf nodes of the learned decision trees to aggregate the regions. Since multiple input data regions for each decision result of the expected output of the reasoning model are aggregated into simple information, the user can easily grasp the degree of violation in the input data.

[0170] Furthermore, combinations of various embodiments or modifications of any constituent elements of each embodiment are possible, or any constituent elements of each embodiment may be omitted.

[0171] Industrial availability

[0172] The verification apparatus disclosed herein can be used, for example, to verify reasoning models for various types of reasoning.

[0173] Explanation of reference numerals in the attached figures

[0174] 1. 1A...Inference system; 2. 2A...Verification device; 3...Inference device; 4...Inference model storage device; 5...Judgment result storage device; 6...Display device; 21. 31...Inference model input unit; 22...Output upper and lower bound calculation unit; 23...Desired output specification unit; 24...Desired output judgment unit; 25...Input region segmentation unit; 26...Input region specification unit; 27...Region ratio calculation unit; 28...Display processing unit; 29-1...Region acquisition unit; 29-2...Sample acquisition unit; 29-3...Region summarization unit; 32...Inference unit; 100...Input interface; 101...Output interface; 102...Processing circuit; 103...Processor; 104...Memory.

Claims

1. A verification device, characterized in that, have: The expected output determination unit determines whether a violation occurs when all the output data of the inference model relative to the input data region is not the expected output, where the input data region is the numerical range of the data. The input region segmentation unit segments the regions in the input data region that are determined not to be violations, and sets the segmented regions as new input data regions for the inference model. The region ratio calculation unit calculates the ratio of the region of each judgment result relative to the input data region; as well as The verification result output section outputs verification result information representing the proportion of the region.

2. The verification device according to claim 1, characterized in that, The expected output determination unit determines any one of the following: The following are considered as valid judgments: all output data of the inference model relative to the input data region are expected outputs; all output data of the inference model relative to the input data region are not expected outputs; or an undefined judgment as a case that does not belong to any of these categories.

3. The verification device according to claim 2, characterized in that, The system includes an output upper and lower bound calculation unit that calculates at least one of the upper or lower bounds of the output data of the inference model relative to the input data region. The expected output determination unit determines any one of the following: the pass / fail determination, the violation determination, or the unclear determination, for the output data region indicated by the upper or lower bound of the output data of the inference model.

4. The verification device according to claim 3, characterized in that, The reasoning model of the verification object is a decision tree ensemble model consisting of a single decision tree or multiple decision trees. The input region segmentation unit segments the region based on the branching conditions of any decision tree contained in the inference model.

5. The verification device according to claim 1, characterized in that, It includes a region acquisition unit that acquires the input data region determined by the desired output determination unit. The verification result output unit displays information about the obtained area on the display device.

6. The verification device according to claim 5, characterized in that, It includes a sample acquisition unit that acquires one or more data samples contained within a region acquired by the region acquisition unit. The verification result output unit displays the obtained data sample on the display device.

7. The verification device according to claim 5, characterized in that, It includes a region aggregation unit that performs aggregation by merging multiple regions acquired by the region acquisition unit into a smaller number of regions. The verification result output unit displays information representing the summarized regions on the display device.

8. The verification device according to claim 7, characterized in that, The reasoning model of the verification object is a decision tree ensemble model consisting of a single decision tree or multiple decision trees. The verification device includes a sample acquisition unit that acquires one or more data samples contained in the acquired area. The regional aggregation unit uses the acquired data samples to learn a single decision tree, and uses the leaf nodes of the learned decision tree to aggregate the regions.

9. A reasoning device, characterized in that, have: The inference model input unit reads in the inference model verified by the verification device according to any one of claims 1 to 7; and The reasoning department performs reasoning using the aforementioned reasoning model.

10. A reasoning system, characterized in that, It comprises the verification device according to any one of claims 1 to 7 and the inference device according to claim 9.

11. A verification method, which is a verification method related to a verification device, characterized in that, The following steps are required: The expected output determination unit determines whether the violation judgment is that all the output data of the inference model for the input data region is not the expected output, where the input data region is the numerical range of the data. The input region segmentation part cuts out the regions in the input data region that are determined not to be violations, and sets the segmented regions as the new input data regions of the inference model; The region ratio calculation unit calculates the ratio of the region of each judgment result relative to the input data region; as well as The verification result output section outputs verification result information representing the proportion of the region.

Images