Verification device, inference device, inference system and verification procedure

Abstract

A verification device (2) includes: an expected output determination unit (24) to determine whether all output data of an inference model for an input data range constitutes a violation in that the output data are not expected outputs; an input range splitting unit (25) to split a range determined to be non-violating within the input data range and to designate a split range as the input data range that is new to the inference model; a range ratio calculation unit (27) to calculate a ratio of a range for each of the determination results with respect to the input data range; and a display processing unit (28) to display verification result information indicating the ratio of the range on a display device (6).

Description

TECHNICAL AREA

[0001] The present disclosure relates to a verification device, an inference device, an inference system and a verification procedure. STATE OF THE ART

[0002] A technique for evaluating the validity of an inference model, which is a machine learning model, has been proposed. For example, patent literature 1 presents a section containing at least one violation data sample that violates or contravenes the verification property among the model output sections for data areas input into the decision tree ensemble model. REFERENCE LIST PATENT LITERATURE

[0003] Patent literature 1: Japanese patent no. 7059220 SUMMARY OF THE INVENTIONAL PROBLEM

[0004] The conventional technique described in patent literature 1 represents a scope containing at least one infringing data sample that is not an expected output of an inference model, but there is no guarantee that all data samples in the represented scope are infringing. For this reason, the conventional technique has the problem that, for example, if the infringing data sample is not present in most areas within the represented scope and is present only in a very narrow area, the degree of infringement is overestimated and it is difficult to address the infringement appropriately.

[0005] The present disclosure solves the above problem, and its aim is to obtain a verification device that is able to represent a ratio of an input data range for each determination result of an expected output of an inference model. TECHNICAL SOLUTION

[0006] A verification device according to the present disclosure comprises: an expected output determination unit to determine whether all output data of an inference model for an input data range, which is a numerical scope or range of values ​​of data, constitute a violation such that the output data is not an expected output; an input range division unit to divide a range determined to be non-violating within the input data range and to designate a divided range as the input data range that is new to the inference model; a range ratio calculation unit to calculate a ratio of a range for each of the determination results with respect to the input data range; and a verification result output unit to output verification result information indicating the ratio of the range. ADVANTAGEOUS EFFECTS OF THE INVENTION

[0007] According to the present disclosure, it is determined whether all output data of an inference model for an input data range violates the requirement that the output data is not the expected output. This is done by splitting a range that has been determined not to violate the requirement, designating a split range as the input data range that is new to the inference model, calculating a ratio of the range for each of the determination results with respect to the input data range, and outputting verification result information indicating the range ratio. By outputting the range ratio for each determination result of the expected output of the inference model, the verification device according to the present disclosure can represent the ratio of the input data range for each determination result of the expected output of the inference model.This helps prevent the degree of injury or violation from being overestimated or underestimated. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram showing a configuration example of an inference system according to a first embodiment. Fig. Figure 2 is a diagram illustrating the results of determining an input data range and an expected output. Fig. Figure 3 is a schematic diagram showing the division processing of an input data range and a determination of the expected output. Fig. Figure 4 is a schematic diagram that provides an overview of the determination of the expected output. Fig. Figure 5 is a representation that provides an example of the result of determining the expected output. Fig. 6A, Fig. 6B and Fig. 6C are illustrations that show examples of how verification results are displayed. Fig. Figure 7 is a flowchart showing a verification procedure according to the first embodiment. Fig. Figure 8 is a flowchart illustrating the range determination processing. Fig. 9A and Fig. Figure 9B are block diagrams illustrating a hardware configuration that implements the functions of the verification device according to the first embodiment. Fig. Figure 10 is a block diagram showing a configuration example of an inference system according to a second embodiment. Fig. Figure 11 is a representation that shows a procurement result of a data area of ​​a determination result of the expected output and a display example for it. Fig. Figure 12 is a representation that shows a procurement result of a data sample from the data value range of the determination result of the expected output and a display example for it. Fig. Figure 13 is a schematic representation showing an example of an area where a summary has been performed. DESCRIPTION OF THE EXAMPLES First example of implementation

[0008] Fig. Figure 1 is a block diagram showing a configuration example of an inference system 1 according to a first embodiment. Fig. Inference system 1 is a system that performs inference using a verified inference model and comprises a verification device 2, an inference device 3, an inference model storage device 4, a determination result storage device 5, and a display device 6. The verification device 2 is a device that verifies an inference model, which is a machine learning model, and the inference device 3 is a device that performs inference using the inference model. The inference system 1 is a system in which the verification device 2, the inference device 3, the inference model storage device 4, the determination result storage device 5, and the display device 6 are connected via a wired signal line or a network through wireless communication. The network is, for example, a telecommunications line, the internet, or similar.

[0009] By displaying the ratio of an input data range to each determination result of an expected output of the inference model, the verification device 2 can prevent the degree of infringement from being overestimated or underestimated. For example, in a conventional technique described in patent literature 1, an infringement scope containing at least one infringement data sample that is not an expected output of the inference model is presented as a verification result. It should be noted that the expected output is a statement that is expected to be true for the output data of the inference model.

[0010] Fig. Figure 2 is a diagram showing the result of determining the input data range and the expected output. The diagram in Fig. The input data range shown in section 2 is a data range defined by an input variable x1 and an input variable x3. A Fig. 2. The depicted area A is a data area that includes an area B. All data contained in area B is a violation data sample that is not the expected output of the inference model, i.e., it does not satisfy the verification condition, and area B is an area that contains only the violation data sample.

[0011] In the conventional technique described in patent literature 1, a region A can be proposed. Within this region A, there is no guarantee that all data samples in this region constitute an infringement. As in Fig. As shown in Figure 2, for example, in most areas no violation data may be available, so the degree of the violation or infringement may be overestimated.

[0012] Since there is a possibility that all data samples fall within the violation scope in area A, the output data of the inference model, if the data is entered into area A, will also be determined as not the expected output, i.e., as a violation that does not satisfy the verification condition. In this case, although the output data of the inference model, if the data is in an area where the violation data sample is not present in area A, is originally the expected output, the conclusion or inference is suspended.

[0013] Furthermore, there is a case where the output value of the inference model is overwritten when the data is fed into range A, thus preventing a violation. In this case, it is possible that the output data of the inference model will be overwritten with a different value even if the data is contained in the range where the violation data sample is not present, although the output data was originally the expected output.

[0014] Furthermore, the inference model can be relearned by adding a data sample to range A in such a way that there is no violation in the output value of the inference model when the data is inserted into range A. In this case, the inference behavior of the inference model also changes significantly when the data from the range where the violation data sample is not present is inserted into range A.

[0015] In a case where a test passes if the violation scope hypervolume output by the inference model when inputting the data contained in region A is equal to or less than a predetermined value, there is a possibility that the test may fail even though the violation scope hypervolume is initially equal to or less than the predetermined value, because the region where the violation data sample is not present is contained within the input data region.

[0016] Accordingly, the verification device 2, according to the first embodiment, outputs the ratio of the range for each expected output determination result of the inference model, thus representing the ratio of the input data range for each determination result of the expected output of the inference model. In this way, the verification device 2 can suppress the occurrence of an error due to an overestimation or underestimation of the degree of violation described above.

[0017] Fig. Figure 3 is a schematic diagram illustrating the division processing of the input data range and the determination result for the expected output, and illustrates the input data range of the inference model, which is defined by the input variables x1 and x3. Fig. 3 is an input data region containing only input data where all output data of the inference model is the expected output; this is a pass region. An input data region containing only input data where all output data of the inference model is found to be non-expected is a violation region. The range of input data that is neither a pass region nor a violation region is an unknown region.

[0018] Verification device 2 inputs the input data range into the inference model and determines whether all output data of the inference model violate the expected output or not, calculating the ratio of the range for each determination result to the input data range. As shown in the diagram on the left of Fig. As shown in Figure 3, it is assumed that the ratio of the throughput area at this time is 23.9% and the ratio of an unknown area is 76.1% in the input data area to the inference model. Since an unknown area, i.e., an input data area in which no determination of a violation has yet been made, is present, the verification device 2 divides the unknown area, as indicated by an arrow in Figure 3. Fig. 3 indicated, and sets the split area as the new input data area of ​​the inference model.

[0019] Subsequently, the verification device 2 sets each shared portion of the unknown range as a new input data range for the inference model and determines whether the output data of the inference model violates or contradicts the expected output, thereby calculating the ratio of the range to the input data range for each determination result. As shown in the middle diagram of Fig. As shown in Figure 3, for example, it is assumed that the ratio of the throughput area at this point in time is 46.7%, the ratio of the unknown area is 42.5%, and the ratio of the violation area is 0.8% in the input data area for the inference model. In this case, the verification device 2 detects that unknown areas are present and further subdivides these unknown areas, as indicated by arrows in Figure 3. Fig. 3 indicated.

[0020] Next, verification device 2 sets a divided portion of the newly divided unknown area as the new input data area of ​​the inference model and determines whether all output data of the inference model violate the inference model in that the output data is not the expected output, thereby calculating the ratio of the area to the input data area for each determination result. As shown in the graph on the right of Fig. As shown in Figure 3, for example, it is assumed that the ratio of the pass area is 98.3%, the ratio of the unknown area is 0%, and the ratio of the violation area is 1.7% in the input data range for the inference model. As described above, the verification device 2 can represent the ratio of the range in accordance with the determination result of the expected output among the input data ranges for the inference model and can also specify a numerical data range for each range. This prevents the degree of violation or infringement from being overestimated or underestimated.

[0021] The inference model storage device 4 is a storage device that stores an inference model. From the perspective of the verification device 2, the inference model stored in the inference model storage device 4 is a machine learning model to be verified, and from the perspective of the inference device 3, it is a machine learning model for inference processing. Furthermore, the inference model storage device 4 is located outside of both the verification device 2 and the inference device 3. It is important to note that the inference model storage device 4 only needs to be accessible by the verification device 2 and the inference device 3 and can be a storage device contained within a computer that functions as either the verification device 2 or the inference device 3.

[0022] The determination result storage device 5 is a storage device that stores a determination result of the expected output of a determination unit 24 for an expected output contained in the verification device 2. The determination result of the expected output is a determination result as to whether all output data of the inference model for the input data range, which is a numerical scope of the data, is non-compliant in that the output data is not the expected output.

[0023] Furthermore, the result storage device 5 is located outside the verification device 2. It should be noted that the result storage device 5 only needs to be accessible from the verification device 2 and can be a storage device contained within a computer that functions as the verification device 2.

[0024] The verification device 2 is implemented, for example, by a computer comprising a communication unit, a processing unit, and a storage unit. The communication unit communicates with the inference model storage device 4 or the determination result storage device 5 via a wired signal line or a network through wireless communication. The communication unit is, for example, a communication device suitable for mobile communication via a communication system such as LTE, 3G, 4G, or 5G. Furthermore, the communication unit can be a short-range wireless communication device such as Bluetooth (registered trademark).

[0025] The communication unit comprises an input interface 100 and an output interface 101 in the Fig. 9A and Fig. 9B.

[0026] The computing unit controls the overall operation of the verification device 2. The computing unit comprises an inference model reading unit 21, an output upper and lower limit computing unit 22, a labeling unit 23 for an expected output, the determination unit 24 for an expected output, an input area division unit 25, an input area labeling unit 26, an area ratio computing unit 27, and a display processing unit 28.The computing unit, which executes an information processing application to verify the inference model, implements various functions of the inference model reading unit 21, the output upper and lower bounds computing unit 22, the labeling unit 23 for an expected output, the determination unit 24 for an expected output, the input range division unit 25, the input range labeling unit 26, the range ratio computing unit 27, and the display processing unit 28. The computing unit includes a processing circuit 102. Fig. 9A and a 103 processor Fig. 9B.

[0027] The storage unit stores, for example, an information processing application and information used for the computational processing of the computing unit. The storage unit is a storage device contained in a computer, which functions as the verification device 2, and includes storage such as an HDD or an SSD, a storage capacity of 104 in Fig. 9A and Fig. 9B or similar. It should be noted that the storage unit only needs to be accessible by verification device 2 and may also be located outside of verification device 2.

[0028] The inference model reading unit 21 reads the inference model to be verified from the inference model storage device 4. The inference model is a function f that produces a one-dimensional output y ∈ R or an M-dimensional output Y = (x1, x2, ..., x N ) ∈ RM for an N-dimensional input X = (y1, y2, ..., y M ) ∈ RN delivers. Here, N and M are integers. The inference model includes, for example, a trained machine learning model, a neural network, a decision tree model, a decision tree ensemble model, a support vector machine (SVM), a generalized linear model, a generalized additive model (GAM), a Gaussian process regression model (GPR), a naive Bayesian model, a Gaussian mixture model (GMM), and the like.

[0029] Examples of neural networks include multilayer perceptrons (MLPs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, and the like. Examples of decision tree ensemble models include random forests, gradient-boosting trees, and the like. Examples of generalized linear models include linear regression and logistic regression.

[0030] The output upper and lower bound calculation unit 22 calculates at least one upper or lower bound of the output data of the inference model for the input data range. This means that the inference model outputs data where the lower bound ≤ output data ≤ upper bound is always true.

[0031] One method for calculating the upper limit is, for example, the following.

[0032] In a case where the inference model is a generalized linear model, the output upper and lower bound computation unit 22 calculates an output value as the upper bound if a variable with a positive coefficient assumes an upper bound of the input data range and a variable with a negative coefficient assumes a lower bound of the input data range.

[0033] In a case where the inference model is a single decision tree model, the output upper and lower bound calculation unit 22 calculates as the upper bound the maximum output value of a leaf node set that can be achieved by the data sample in the input data area.

[0034] In a case where the inference model is a decision tree ensemble model, the output upper and lower bound calculation unit 22 calculates as the upper bound a sum of the maximum output values ​​for the respective decision trees of a leaf node set that can be reached by the data sample in the input data area.

[0035] One method for calculating the lower limit is, for example, the following.

[0036] In a case where the inference model is a generalized linear model, the output upper and lower bound computation unit 22 calculates an output value as the lower bound if a variable with a positive coefficient assumes a lower bound of the input data range and a variable with a negative coefficient assumes an upper bound of the input data range.

[0037] In a case where the inference model is a single decision tree model, the output upper and lower bound calculation unit 22 calculates as the lower bound the minimum output value of a leaf node set that can be achieved by the data sample in the input data area.

[0038] In a case where the inference model is a decision tree ensemble model, the output upper and lower bound calculation unit 22 calculates as a lower bound a sum of minimum output values ​​for the respective decision trees under a leaf node set that can be reached by the data sample in the input data area.

[0039] The designation unit 23 for the expected output specifies an expected output value of the inference model in the determination unit 24 for an expected output. The expected output is a statement that is expected to be true for the output data y of the inference model. Examples include defining or designating 50 ≤ y < 80 as the range of values ​​for the output value of the regression, defining or designating 0.5 ≤ y as the range of values ​​for the output values ​​of a binary classification (output of a sigmoid function), designating (y1 < y2) & (y3 < y2) as the output class of a multi-class classification, designating the output of class 2 by an inequality of the output of a softmax function as the expected output, designating |y-y'| ≤ 0.1, which is the difference to the output y' = f'(X) of another inference model f', and the like.

[0040] For example, a user uses an input device (in Fig. (1 not shown), to enter a conditional expression or upper and lower limits of the expected output in text form. Thus, the expected output designation unit 23 sets as the expected output information specifying the conditional expression or the upper and lower limits entered by the input device into the expected output designation unit 24. Furthermore, the expected output designation unit 23 instructs the display unit 28 to display on the display device 6 a selection screen for choosing a class ID by means of a drop-down menu or a checkbox, or a selection screen for choosing an output range by means of a parallel coordinate representation.When the user performs a selection operation on the selection screen using the input device, the labeling unit 23 for an expected output receives the information about the selection operation and sets the information about the selection operation in the destination unit 24 for an expected output as the expected output.

[0041] The determination unit 24 for an expected output determines whether all output data of the inference model for the input data range violate the expected output or not. Fig. Figure 4 is a schematic diagram that provides an overview of the expected output determination and shows a case where the expected output of the regression is 50 ≤ y < 80. Fig. 4. All output data y that are contained within a numerical range (lower limit of y ≥ 50) & (upper limit of y < 80) are also contained within the expected output range 50 ≤ y < 80, and thus the input data range for the inference model is not determined as a violation when the output data y are output and is a passable range.

[0042] All output data y that fall within a numerical range (upper limit of y < 50) | (lower limit of y ≥ 80) are not within the expected output range 50 ≤ y < 80, and thus the input data range for the inference model, when the output data y are output, is a violation range in which a violation determination has been made.

[0043] The output data y, for which no violation was detected, lies within a numerical range that is not included in any of the following cases: (lower limit of y ≥ 50) & (upper limit of y < 80) and (upper limit of y < 50) | (lower limit of y ≥ 80). The determination unit 24 for an expected output determines that the output data y is contained within an unknown range of values.

[0044] In a case where the expected output of the binary classification is 1, i.e., the output score y is 0.5 ≤ y, all output scores y that are contained in a score range where the lower bound ≥ 0.5 of the output score y is contained in the expected output score range 0.5 ≤ y, and thus the input data range, with respect to the inference model, when the output score y is output, is a pass-through range.

[0045] Since all output values ​​y that lie within a numeric range where the upper limit < 0.5 of the output value y is satisfied are not contained within the expected output range 0.5 ≤ y, the input data range for the inference model is a violation range when outputting the output value y.

[0046] The evaluation range, which is neither the lower limit ≥ 0.5 of the output rating y nor the upper limit < 0.5 of the output rating y, includes the output data y that are classified as non-violating. The determination unit 24 for an expected output establishes that the output data y is contained within an unknown range of values.

[0047] In a case where the expected output of a three-class classification is class 2, i.e., (y1 < y2) & (y2 < y3), all output data y that lies within a numeric range of values ​​of (upper bound of y1 < lower bound of y2) & (upper bound of y3 < lower bound of y2) are also contained within the expected output range (y1 < y2) & (y2 < y3), and thus the input data range for the inference model, when the output data y are output, is a traverse range.

[0048] All output data y that is contained in a numeric range of values ​​of (lower limit of y1 ≥ upper limit of y2) |(lower limit of y3 ≥ upper limit of y2) are not contained in the expected output range of values ​​(y1 < y2) & (y2 < y3), and thus the input data range for the inference model when the output data y is output is a violation range.

[0049] The output data y, for which no violation was detected, lies within a numerical range of values ​​that is not contained in any of the following cases: (upper limit of y1 < lower limit of y2) & (upper limit of y3 < lower limit of y2) and (lower limit of y1 ≥ upper limit of y2) | (lower limit of y3 ≥ upper limit of y2). The determination unit 24 for an expected output determines that the output data y is contained within an unknown range.

[0050] In a case where |y-y'| ≤ 0,1, which is a difference to the output y' = f'(X) of another inference model f', is an expected output range, then all output data y that is contained in the numerical range of (upper bound of y - lower bound of y' ≤ 0,1) & (upper bound of y' - lower bound of y ≤ 0,1) is also contained in the expected output range, and thus the input data range for the inference model, when the output data y is output, is a pass-through range.

[0051] All output data y that is contained in a numeric range of values ​​from (lower limit of y - upper limit of y' > 0.1) | (lower limit of y' - upper limit of y > 0.1) are not contained in the expected output range |y - y'| ≤ 0.1 and are classified as violating, and thus the input data range for the inference model is a violation range when outputting the output data y.

[0052] Output data y, which was found to be non-expected, lies within a numerical range of values ​​that is not contained in any of the following: (upper bound of y - lower bound of y ≤ 0.1) & (upper bound of y' - lower bound of y ≤ 0.1) and (lower bound of y - upper bound of y' > 0.1) | (lower bound of y' - upper bound of y > 0.1). The determination unit 24 for an expected output determines that the output data y lies within an unknown range.

[0053] The information about the determination result of the determination unit 24 for an expected output is stored in the determination result storage device 5.

[0054] Fig. Figure 5 is a representation that shows an example of the determination result of the expected output and illustrates the content stored in the determination result storage device 5. As in Fig. As shown in Figure 5, the information about the determination result includes the input data range and the determination result of the expected output for each input data point. The input data range is a data range defined by a lower bound and an upper bound for each of the input variables x1, x2, ..., x N The parameters of the relational expression that specify the input data are defined. The expected result of the result determination is "Pass", "Violation", or "Unknown".

[0055] Although the input data corresponding to all determination results are displayed, the determination result storage device 5 can only store one specific determination result as the determination result of the expected output. For example, only input data whose determination result of the expected output is a violation can be stored.

[0056] The input-area subdivision unit 25 divides the area determined to be non-violating within the input data area and designates the divided area as the new input data area of ​​the inference model. For example, the input-area subdivision unit 25 can subdivide the input data area by a branch of a decision tree contained in the inference model. In this case, the input-area subdivision unit 25 divides into overlapping portions of a data area corresponding to each leaf node of the decision tree and the input data area. At this point, the input-area subdivision unit 25 can ignore a leaf node that cannot be reached due to the constraints of the input data area, or it can terminate the division at an intermediate branch instead of at the leaf node at one end.

[0057] Furthermore, the Input Range Division Unit 25 can randomly divide the input data range. For example, the Input Range Division Unit 25 randomly selects an input variable and determines division points within the upper and lower bounds of the input variable. Additionally, the Input Range Division Unit 25 can divide the input data range with respect to a midpoint of the upper and lower bounds generated from the uniform distribution, a mean value, a median value, or a quantile point of the training data.

[0058] The input area division unit 25 can weight the selection of input variables to be divided according to the importance of the feature of the inference model.

[0059] Furthermore, input range partition unit 25 can divide the input data range by a plurality of partition candidates and choose the partition that yields the best index after the partition. For example, input range partition unit 25 uses a partition that minimizes the difference between an upper and a lower bound of the inference model for the partitioned range.

[0060] Furthermore, the input area division unit 25 can perform a division that minimizes the sum of the sizes of unknown areas after the division.

[0061] The input area division unit 25 can recursively continue the division until no unknown area remains, or it can abort the division halfway through. For example, the input area division unit 25 stops the division if the processing time is equal to or greater than a predetermined value, the number of divided areas is equal to or greater than a predetermined value, the size of the unknown area is less than a predetermined value, or the size of the unknown area is less than the size of the violation area.

[0062] The input range designation unit 26 defines the input data range in the output upper and lower bound calculation unit 22. The input data range is a data range in an N-dimensional space that is input into the inference model to be tested. N is an integer. The input data range might, for example, consist of tabular data defined by the input variable x. The input data range could be a feature extracted from an image, video, text, sound, graph, time series data, or similar source using a neural network or similar algorithm.

[0063] The input data area can e.g. B. S = {X = (x1, x2, ..., x N ) ∈ R N|-3 .5 ≤ x1 ≤ 5.2, ..., 15 ≤ x N ≤ 40}, which represents a hyperrectangle, S = {X ∈ R N|| X |2 ≤ r 2}, which represents a hypersphere, S = S1 ∩ S2, which represents a product of several regions, or S = S1 U S2, which represents a union of several regions.

[0064] For example, the user enters information via a [unclear] in Fig. 1. An input device (not shown) provides a conditional expression of the input data range or upper and lower limits of the hyperrectangle in text form. The input range designator 26 thus inserts information specifying the conditional expression or the upper and lower limits of the hyperrectangle input to the output upper and lower limit calculation unit 22 as the input data range. Furthermore, the input range designator 26 instructs the display processing unit 28 to display a selection screen for selecting the range of the input variables by means of a rectangle or a wheel in two-dimensional space, or a selection screen for selecting the range of each input variable by means of a parallel coordinate representation, on the screen of the display device 6.When the user performs a selection operation on the selection screen using the input device, the input area labeling unit 26 receives the selection operation information and sets the selection operation information as an input data area in the output calculation unit 22 for the upper and lower limits.

[0065] The Area Ratio Calculation Unit 27 calculates the ratio of an area to the input data area for each outcome of the expected output determination. For example, the Area Ratio Calculation Unit 27 can calculate the ratio based on the hypervolume of the input data area. If the input data area is a hyperrectangle, the Area Ratio Calculation Unit 27 can calculate the area ratio based on the product of the side lengths of the hyperrectangle. Furthermore, the Area Ratio Calculation Unit 27 can calculate the area ratio to the input data area for each outcome of the expected output determination using the number of data samples from a predetermined dataset contained within the area.

[0066] The range ratio calculation unit 27 can calculate the ratio of the range for each determination result of the expected output to the input data range using the number of generated data samples contained in the range.

[0067] The data sample can be generated from a uniform distribution, it can be generated by perturbing a data sample of a given data set, it can be generated from a data distribution learned from a given data set, or it can be generated using a Generative Adversarial Network (GAN) learned from a given data set.

[0068] Assuming that the sizes of the violation area, the passage area, and the unknown area are |S violation |, |S pass | and |S unknown | if |S en-tire |= |S violation |+ |S pass |+ |S unknown In this case, the area ratio calculation unit 27 calculates, for example, |S violation | / |S entire |, |S pass | / |S entire |, |S unk- nown | / |S entire |, |S violation | / (| Sviolation |+ |S pass |), or (|S violation |+ |S unknown |) / |S en- tire | as a ratio of an area.

[0069] The display processing unit 28 is a verification result output unit for outputting verification result information that specifies a ratio of a range. The display processing unit 28 displays the verification result information on the display device 6. The display device 6 is, for example, a liquid crystal display (LCD) or an organic electroluminescent (EL) display device.

[0070] The display processing unit 28 outputs display control information to the display device 6 for displaying the verification result information. The display device 6 displays the verification result information based on the display control information. The ratio of the range is represented, for example, in a band chart, a pie chart, a stacked bar chart, or a color gradient.

[0071] Fig. Figure 6A is a representation that provides an example of how a verification result is displayed and illustrates the relationships between the ranges using a band diagram. Fig. Figure 6A represents the ratio of an area to the length of a strip. The ratio of the throughput area is 32.3%, the ratio of the unknown area is 51.3%, and the ratio of the violation area is 16.4%.

[0072] Fig. Figure 6B is a diagram that shows an example of how the verification result is displayed and illustrates the relationships between the areas through color shading. Fig. In section 6B, the ratio of violation areas is represented by a color shading. For example, the greater the ratio of violations, the darker the displayed color.

[0073] The display processing unit 28 can update the display of the ratio of the area that changes according to the division of the input data area in real time. For example, the display unit 28 can monotonically decrease the display of the unknown area over time or monotonically increase the display of the pass area and the violation area.

[0074] In addition, the display processing unit 28 can display a change in an area ratio over time on the display device 6 as a line graph or animation.

[0075] Furthermore, the display processing unit 28 can display the designated input data range and the expected output together on the display device 6. For example, the display processing unit 28 displays the ratio of the range on the display device 6 as a parallel coordinate diagram, as a two-dimensional diagram, or as text.

[0076] Fig. Figure 6C is a representation that provides an example of how to display the verification result and illustrates the relationship of the area using a parallel coordinate diagram and text. Fig. 6C will assign a numerical range of values ​​to each of the input variables x1, x2, ..., x N , which define the input data range, is shown by a rectangle, and a numerical value range of the output data y of the inference model for the input data range is shown by a rectangle.

[0077] Furthermore, the display processing unit 28 can adjust the numerical value range of each of the input variables x1, x2, ..., x N and display the numerical value range of the output data y as text on the display device 6, as in Fig. 6 shown.

[0078] Fig. Figure 1 illustrates a configuration in which the verification device 2 includes, but is not limited to, the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output determination unit 24, the expected output labeling unit 23, the input range division unit 25, the input range determination unit 26, the range ratio calculation unit 27, and the display processing unit 28. The verification device 2 only needs to be able to determine the expected output of the inference model, subdivide the input data range fed into the inference model, and output a determination result. Therefore, with the exception of the expected output determination unit 24, the input range division unit 25, and the display processing unit 28, the configuration can be contained in an external device accessible from the verification device 2.

[0079] It should be noted that, although the case has been described in which the input data domain is a region of a two-dimensional space, the input data domain is generally a region of a higher-dimensional space of the second or higher order.

[0080] The inference device 3 is implemented, for example, by a computer comprising a communication unit, a processing unit, and a storage unit. The communication unit communicates with the inference model storage 4 via a wired signal line or a network via wireless communication. The communication unit is, for example, a communication device suitable for mobile communication via a communication system such as LTE, 3G, 4G, or 5G. Furthermore, the communication unit can be a short-range wireless communication device such as Bluetooth (registered trademark).

[0081] The computing unit controls the overall operation of the inference device 3. The computing unit comprises an inference model reading unit 31 and an inference unit 32. Various functions of the inference model reading unit 31 and the inference unit 32 are implemented by the computing unit, which runs an information processing application to perform inferences.

[0082] The storage unit stores, for example, an information processing application and information used for the computational processing of the computing unit. The storage unit is a storage device contained in a computer that functions as an inference device 3 and includes storage such as an HDD or an SSD, the memory 104 in Fig. 9A and Fig. 9B or similar. It should be noted that the storage unit only needs to be accessible from inference device 3 and may also be located outside of inference device 3.

[0083] The inference model reading unit 31 reads the inference model verified by the verification device 2 from the inference model storage device 4. The inference model read by the inference model reading unit 31 is stored in the inference unit 32.

[0084] The inference unit 32 performs the inference using the inference model read by the inference model reading unit 31. The inference unit 32 can, for example, test a complex inference model. In this case, to demonstrate the risk of an error occurring, such as a bug in a complex inference model, in a pre-operational test, the inference unit 32 sets a property that must be satisfied as an input data range and an expected output in the inference model and performs an inference.

[0085] For example, inference unit 32 specifies an area within a five-minute walk of a train station and within 10 years of construction as its input data range and derives the rent of an apartment under the assumption that the expected output of the inference model for this input data range is a rent of 80,000 yen or more.

[0086] Furthermore, inference unit 32 specifies a blood pressure of 135 or more and an LDL cholesterol of 140 or more as the input data range and infers the need for an investigation from the assumption that the expected output of the inference model for this input data range requires an investigation.

[0087] The inference unit 32 can take as input a numerical range of values ​​indicating a product with a measurement error greater than that of an anomalous product sample and infer the anomaly of the product, assuming that an expected output is that all products with a measurement error greater than that of the anomalous product sample are anomalous.

[0088] If the ratio of the violation area is smaller than the specified value, the inference device 3 is deployed in the actual environment and begins operation.

[0089] The inference unit 32 can derive the accuracy of a proxy model that approximates a complicated inference model.

[0090] For example, inference device 3 prompts the user to detect the behavior at the time of inference by approximating it with a single, highly interpretable decision tree model. Verification device 2 determines that the outputs of the original inference model and the proxy model are the expected output and calculates the ratio of the violation region, where the outputs of the original inference model and the proxy model do not match. Verification device 2 then considers the smallness of the violation region ratio as the fidelity of the proxy model and displays the fidelity on display device 6.

[0091] The inference of the existing rule-based system can be replaced by the inference model verified by verification device 2. For example, the existing system is replaced by a machine learning model with high predictive accuracy.

[0092] Verification device 2 specifies as its expected output that the outputs of the inference model and the existing system match, and calculates a ratio of a violation range in which the outputs of the inference model and the existing system do not match.

[0093] Then, the inference device 3 performs a replacement using the inference model where the ratio of the violation area is smaller than the specified value.

[0094] A difference between two inference models with different versions can be quantified by Verification Device 2. For example, the difference between inference model A, learned from data of a specific time period, and inference model B, learned from data of a different time period, is quantified. Verification Device 2 specifies that the outputs of inference model A and inference model B are expected to match and calculates the ratio of the violation range, where the outputs of inference model A and inference model B do not match. If the violation range ratio is smaller than the specified value, the difference between model A and model B is considered small.For example, in a system that continuously collects data, learns, and updates its inference model, the model can be updated only when the difference between the inference models is small. Furthermore, by visualizing the violation area, it is possible to identify where the model exhibits a deviation, and it is possible, for example, to detect and address environmental changes such as data shifts at an early stage.

[0095] The following describes a verification procedure according to the first embodiment.

[0096] Fig. Figure 7 is a flowchart illustrating the verification procedure according to the first embodiment.

[0097] The inference model read unit 21 reads the inference model from the inference model storage device 4 and places the inference model in the output upper and lower bound computation unit 22, and the input range label unit 26 places the input data range in the output upper and lower bound computation unit 22. The expected output label unit 23 specifies the expected output of the inference model in the expected output determination unit 24. A set of these processes constitutes the ST1 stage process.

[0098] The determination unit 24 for an expected output deletes a range list and adds the configured input data range (step ST2). The range list is a list of data ranges that are entered into the inference model. Determination unit 24 for an expected output enters the input data range registered in the range list into the inference model.

[0099] Next, the determination unit 24 for an expected output performs processing to determine the expected output for each input data range in the range list (step ST3).

[0100] Subsequently, for an expected output, the determination unit 24 determines whether there is a range that is neither a violation range, where all output data of the inference model for the input data range is not the expected output, nor a pass range, where all output data is the expected output, i.e., an unknown range (step ST4).

[0101] If an unknown range exists (step ST4; YES), the input range division unit 25 splits the unknown range (step ST5). The expectation output determination unit 24 clears the range list and adds the split range as a new input data range to the inference model (step ST6). Subsequently, the expectation output determination unit 24 proceeds to step ST3, determines the expected output of the new input data range, and repeats the subsequent processing.

[0102] If there is no unknown range (step ST4; NO), the range ratio calculation unit 27 calculates the ratio of the range for each expected output determination result with respect to the input data range, and subsequently the display processing unit 28 displays verification result information showing the ratio of the range on the display device 6 (step ST7).

[0103] In this way, the verification device 2 can represent the ratio of the input data range for each determination result to the expected output of the inference model.

[0104] Fig. Figure 8 is a flowchart illustrating the determination processing of the expected output and the detailed processing of step ST3 in Fig. 7 shows.

[0105] The output upper and lower bound calculation unit 22 calculates the upper and lower bounds of the output data of the inference model for the input data range (step ST1A). The output upper and lower bound calculation unit 22 defines the calculated upper and lower bounds of the output data in the determination unit 24 for an expected output.

[0106] The designation unit 24 for an expected output determines whether the input data range corresponding to the output data defined in the upper bound and lower bound of the inference model constitutes a violation or not with respect to the expected output designated by the designation unit 23 for an expected output, i.e., whether it constitutes a violation or whether it exists or is unknown (step ST2A).

[0107] Next, the determination unit 24 stores the input data area and the corresponding determination result of the expected output in the determination result storage device 5 (step ST3A) for an expected output.

[0108] The following describes a hardware configuration that implements the functions of the verification device 2.

[0109] The functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the labeling unit 23 for an expected output, the determination unit 24 for an expected output, the input range division unit 25, the input range labeling unit 26, the range ratio calculation unit 27, and the display processing unit 28, which are contained in the verification device 2, are implemented by a processing circuit. That is, the verification device 2 contains a processing circuit for performing the functions described in Fig. The processing shown in step 7 is from step ST1 to step ST7. The processing circuit can be dedicated hardware or a central processing unit (CPU) that executes a program stored in main memory.

[0110] Fig. Figure 9A is a block diagram showing a hardware configuration that implements the functions of the verification device 2. Fig. Figure 9B is a block diagram showing a hardware configuration for running software to implement the functions of the verification device 2. Fig. 9A and Fig. 9B is the input interface 100, an interface that forwards data acquired by the verification device 2 from the inference model storage device 4 or the determination result storage device 5. The output interface 101 is an interface that forwards the data output by the verification device 2 to the determination result storage device 5.

[0111] In a case where the processing circuit 102 is in Fig. Where the dedicated hardware shown in Figure 9A is the processing circuit 102, for example, corresponds to a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof. The functions of the inference model read unit 21, the output upper and lower bound calculation unit 22, the expected output labeling unit 23, the expected output determination unit 24, the input range division unit 25, the input range labeling unit 26, the range ratio calculation unit 27, and the display processing unit 28, which are contained in the verification device 2, can be implemented by separate processing circuits, or these functions can be implemented together by one processing circuit.

[0112] In a case where the processing circuit of the in Fig. The functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output labeling unit 23, the expected output determination unit 24, the input range division unit 25, the input range determination unit 26, the range ratio calculation unit 27, and the display processing unit 28, which are contained in the verification device 2, are implemented by software, firmware, or a combination of software and firmware. It should be noted that the software or firmware is described as a program and stored in the working memory 104.

[0113] The processor 103 reads and executes the program stored in memory 104, thereby implementing the functions of the inference model read unit 21, the output upper and lower bound calculation unit 22, the expected output labeling unit 23, the expected output determination unit 24, the input area division unit 25, the input area determination unit 26, the area ratio calculation unit 27, and the display processing unit 28, which are contained in the verification device 2. The verification device 2 includes, for example, a memory 104 for storing a program which, when executed by the processor 103, performs the functions described in Fig. The processing steps ST1 to ST7 shown in Figure 7 are carried out. These programs cause a computer to execute a procedure or processing method that is performed by the inference model reading unit 21, the output upper and lower bound calculation unit 22, the designation unit 23 for an expected output, the determination unit 24 for an expected output, the input area division unit 25, the input area determination unit 26, the area ratio calculation unit 27, and the display processing unit 28.The memory 104 can be a computer-readable storage medium on which a program is stored that causes a computer to operate as an inference model reading unit 21, as an output upper and lower bound calculation unit 22, as a designation unit 23 for an expected output, as a determination unit 24 for an expected output, as an input area division unit 25, as an input area determination unit 26, as an area ratio calculation unit 27 and as a display processing unit 28.

[0114] Memory 104 corresponds to a non-volatile or volatile semiconductor memory such as random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM) or electrical EPROM (EEPROM) (registered trademark), a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisc or a DVD.

[0115] Some of the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the designation unit 23 for an expected output, the determination unit 24 for an expected output, the input range division unit 25, the input range determination unit 26, the range ratio calculation unit 27 and the display processing unit 28, which are contained in the verification device 2, can be implemented by special hardware, and the other part can be implemented by software or firmware.For example, the functions of the inference model reading unit 21, the output upper and lower bound calculation unit 22, the expected output labeling unit 23, and the input range determination unit 26 can be implemented by the processing circuit 102, which is dedicated hardware. The functions of the expected output determination unit 24, the input range division unit 25, the range ratio calculation unit 27, and the display processing unit 28 can be implemented by the processor 103, which reads and executes a program stored in memory 104. As described above, the processing circuit can implement the aforementioned functions through hardware, software, firmware, or a combination thereof.

[0116] As described above, the verification device 2 according to the first embodiment includes the determination unit 24 for expected output to determine whether all output data of an inference model for an input data range, which is a numerical range of values ​​of data input into the inference model to be verified, constitute a violation, such that the output data is not an expected output; the input range division unit 25 to divide a range determined to be non-violating within the input data range and to designate a divided range as the input data range that is new to the inference model; the range ratio calculation unit 27 to calculate a ratio of a range for each of the determination results with respect to the input data range; and the display processing unit 28 to display verification result information indicating the ratio of the ranges.to be displayed on the display device 6. By outputting the range ratio for each determination result of the expected output of the inference model, the verification device 2 can display the ratio of the input data range for each determination result of the expected output of the inference model. This prevents the degree of infringement or violation from being overestimated or underestimated.

[0117] It should be noted that even in the conventional technique described in patent literature 1, the input data area is subdivided into a plurality of areas, and a feasibility determination unit determines whether any of the subdivided areas lack feasibility. If the feasibility determination device determines that feasibility exists, i.e., feasibility is indicated, at least one data sample in the subdivided area is identified as infringing. Therefore, in the conventional technique described in patent literature 1, it is possible that an area may be proposed that partially includes the data sample identified as infringing.On the other hand, the verification device 2 can represent the ratio of the input data range for each determination result by dividing the range in which all output data of the inference model for the input data range are determined to be non-violating, i.e., the range that includes the range containing the data determined to be passed and the data determined to be violating.

[0118] In the verification device 2 according to the first embodiment, the determination unit 24 determines one of the following for an expected output: pass determination, where all output data of the inference model for the input data range is an expected output; violation determination, where all output data of the inference model for the input data range is not an expected output; and unknown determination, which corresponds to neither. In this way, the verification device 2 can represent the ratio of the input data range to the expected output of the inference model for each determination result.

[0119] The verification device 2 according to the first embodiment comprises the output upper and lower bound calculation unit 22 for calculating at least one upper or lower bound of the output data of the inference model for the input data range. The determination unit 24 for an expected output determines one of the following: the pass determination, the violation determination, and the unknown output determination for an output data range specified by the upper or lower bound of the output data of the inference model. In this way, the verification device 2 can represent the ratio of the input data range for each determination result of the expected output of the inference model.

[0120] In the verification device 2 according to the first embodiment, the inference model to be verified is a decision tree ensemble model containing a single decision tree or a plurality of decision trees. The input area partitioning unit 25 divides the area according to a branching condition of a decision tree contained in the inference model.

[0121] In this way, the verification device 2 can determine the expected output of the inference model for each of the subdivided areas.

[0122] The inference device 3 according to the first embodiment comprises the inference model reading unit 31, which reads the inference model verified by the verification device 2, and the inference unit 32, which performs an inference using the inference model. By displaying the relationship between the input data range for each determination result and the expected output of the inference model, it is possible to prevent the degree of violation from being overestimated or underestimated, thus enabling the inference device 3 to perform a very accurate inference.

[0123] The inference system 1 according to the first embodiment comprises the verification device 2 and the inference device 3. It is possible to present an inference system that is capable of representing a relationship to an input data range for each determination result of an expected output of an inference model. This prevents the degree of infringement or violation from being overestimated or underestimated.

[0124] The verification procedure according to the first embodiment comprises the following steps: Determining, by the determination unit 24 for an expected output, whether all output data of an inference model for an input data range constitutes a violation such that the output data is not an expected output; Splitting, by the input range splitting unit 25, a range determined to be non-violating within the input data range and designating a split range as a new input data range of the inference model; Calculating, by the range ratio calculation unit 27, a ratio of a range for each of the determination results with respect to the input data range; and Displaying, by the display processing unit 28, verification result information showing the ratio of the ranges on the display device 6.By performing the above procedure, the verification device 2 can represent the ratio of the input data range for each determination result to the expected output of the inference model. This prevents the degree of infringement or violation from being overestimated or underestimated. Second embodiment

[0125] Fig. Figure 10 is a block diagram showing a configuration example of an inference system 1A according to a second embodiment. Fig. Inference system 1A, defined as a system that performs inference using a verified inference model, comprises a verification device 2A, an inference device 3, an inference model storage device 4, a determination result storage device 5, and a display device 6. Verification device 2A verifies an inference model, which is a machine learning model, and inference device 3 performs inference using the inference model. Inference system 1 is a system in which verification device 2, inference device 3, inference model storage device 4, determination result storage device 5, and display device 6 are connected via a wired signal line or a network through wireless communication. The network could be, for example, a telecommunications line, the internet, or similar.

[0126] In addition to displaying a range ratio for each determination result of an expected output of the inference model, the verification device 2A represents a range itself, a data sample contained within the range, or summary information about a plurality of ranges. The verification device 2A is implemented, for example, by a computer comprising a communication unit, a processing unit, and a storage unit. The communication unit communicates with the inference model storage device 4 or the determination result storage device 5 via a wired signal line or a network via wireless communication. The communication unit is, for example, a communication device suitable for mobile communication via a communication system such as LTE, 3G, 4G, or 5G.Furthermore, the communication unit can be a short-range wireless communication unit such as Bluetooth (registered trademark). The communication unit includes an input interface 100 and an output interface 101. Fig. 9A and Fig. 9B.

[0127] The computing unit controls the entire operation of the verification device 2A. The computing unit comprises an inference model reading unit 21, an output upper and lower bounds computing unit 22, a labeling unit 23 for an expected output, a determination unit 24 for an expected output, an input range division unit 25, an input range determination unit 26, a range ratio computing unit 27, a display processing unit 28, a range procurement unit 29-1, a sample procurement unit 29-2, and a range summarization unit 29-3.By executing an information processing application to verify the inference model, the computing unit implements various functions of the inference model reading unit 21, the output upper and lower bounds computing unit 22, the expected output labeling unit 23, the expected output determination unit 24, the input range division unit 25, the input range determination unit 26, the range ratio computing unit 27, the display processing unit 28, the range procurement unit 29-1, the sample procurement unit 29-2, and the range summarization unit 29-3. The computing unit includes the processing circuit 102. Fig. 9A and the 103 processor from Fig. 9B.

[0128] The storage unit stores, for example, an information processing application and information used for the computational processing of the computing unit. The storage unit is a storage device contained in a computer, which functions as the verification device 2, and includes storage such as an HDD or an SSD, a storage capacity of 104 in Fig. 9A and Fig. 9B or similar. It should be noted that the storage unit only needs to be accessible by verification device 2A and may also be located outside of verification device 2A.

[0129] It should be noted that the inference model reading unit 21, the output upper and lower bound calculation unit 22, the labeling unit 23 for an expected output, the determination unit 24 for an expected output, the input area division unit 25, the input area determination unit 26, the area ratio calculation unit 27, and the display processing unit 28 are those in Fig. 1 are similar, so a redundant description is omitted.

[0130] The area procurement unit 29-1 takes the input data area determined by the determination unit 24 for the expected output from the determination result storage device 5. The display processing unit 28 displays information on the display device 6 indicating an area procured by the area procurement unit 29-1.

[0131] Fig. Figure 11 is a representation showing a procurement result of a data range, a determination result of the expected output, and a display example for it. As in Fig. Figure 11 shows the information that specifies a plurality of input data ranges, information that defines the lower and upper bounds of each of the input variables x1, x2, ..., x N Specify the parameters of a relational expression that specifies each input data range. For example, the range procurement unit 29-1 acquires information associated with an expected output determination result of a violation, from information specifying a plurality of input data ranges corresponding to the expected output determination results of the pass, violation, and unknown.

[0132] The display processing unit 28 displays the majority of the input data areas corresponding to the determination result of the expected output of the violation on the display device 6 as parallel coordinate diagrams. Fig. 11 is a numerical range of values ​​for each of the input variables x1, x2, ..., x N , which define one of the several input data ranges corresponding to the determination result of the expected output of the violation, is shown by a rectangle, and a numerical range of values ​​of the output data y of the inference model for this input data range is shown by a rectangle.

[0133] Rectangle C of the inference model's output data y represents a range of expected output values, and rectangle D of the inference model's output data y represents an output area corresponding to the selected violation area. Using this information, the user can assess the severity of the violation in the input data.

[0134] The sample acquisition unit 29-2 acquires one or more data samples contained within the area acquired by the area acquisition unit 29-1. The display processing unit 28 displays the data sample acquired by the sample acquisition unit 29-2 on the display device 6.

[0135] Fig. Figure 12 is a representation showing a procurement result of a data sample from the data value range of the determined result of the expected output and a display example for it. For example, as in Fig. As shown in Figure 12, the sample acquisition unit 29-2 acquires information as a data sample that defines a lower bound and an upper bound for each of the input variables x1, x2, ..., x N specify the parameters of a relational expression that specifies an input data range from a plurality of input data ranges that correspond to the determination result of the expected output of the violation.

[0136] In Fig. 12 The input variable x1 in the data sample, corresponding to the determination result of the expected output of the violation displayed on the display device 6, assumes, for example, a value between a lower limit of 0.2 and an upper limit of 0.5. In this way, the display device 6 shows a variety of input data contained within an input data range that corresponds to the determination result of the expected output of the violation. The user can use this to determine the input data of the violation range.

[0137] As in Fig. As shown in Figure 12, the sample acquisition unit 29-2 procures a data sample in a case where each variable defining the input data range assumes a lower bound or an upper bound. Furthermore, the sample acquisition unit 29-2 can generate and acquire a data sample from a uniform distribution in the input data range, acquire a training data sample contained in the input data range, or acquire a data sample of a vertex of the input data range.

[0138] The area aggregation unit 29-3 aggregates the majority of the areas procured by the area procurement unit 29-1 into a smaller number of areas. The display processing unit 28 displays information about an area aggregated by the area aggregation unit 29-3 on the display device 6.

[0139] In a case where the inference model to be tested is a decision tree ensemble model containing a single decision tree or a plurality of decision trees, the sample procurement unit 29-2 acquires one or more data samples contained in the area procured by the area procurement unit 29-1.

[0140] The domain summarization unit 29-3 learns the individual decision tree using the data samples acquired by the sample acquisition unit 29-2 and summarizes the domain using a leaf node of the learned decision tree.

[0141] As an example, a case is presented in which a violation area is aggregated during the learning of an inference model with a single decision tree. A teacher label "1" is assigned to a data sample from the violation area, and a teacher label "0" is assigned to a data sample from a pass area or an unknown area. It can also be included in the teacher label "1" of the data sample obtained from the unknown area. The area aggregation unit 29-3 collects a multitude of determination paths in which the teacher label of the output data is "1" in a decision tree model of a learning outcome and designates the corresponding area as an aggregate area.

[0142] It should be noted that a hyperparameter can be set at the time of learning the decision tree to limit the depth of the decision tree, the maximum number of leaves, the number of data samples contained in a leaf, etc.

[0143] Since the range is displayed through simple information in the summary, the user can easily grasp the range of input data that corresponds to the determination result of the expected output.

[0144] The area summarization unit 29-3 can weight and learn the data sample acquired by the sample acquisition unit 29-2 with a sample weight proportional to the area size of the input data area that is the data source. In this way, a plurality of input data areas corresponding to the determination result of the expected output can be represented in the summarization area, taking the area size into account, so that the user can easily grasp the input data area that corresponds to the determination result of the expected output.

[0145] The range summarization unit 29-3 outputs the summarized range as a new input data range to the input range labeling unit 26. Using the result of the determination of the expected output for the summarized range, the range ratio calculation unit 27 can calculate the ratio between any of the ranges Pass, Violation, and Unknown for the summarized range. Even if, for example, it is displayed that the ratio of the Violation range in the original input data range is 0.8%, the degree of violation in the majority of the input data ranges that include this range cannot be easily grasped. Therefore, by summarizing a majority of input data ranges and displaying that the ratio or proportion of the Violation range in the summarized range is 93.7%, the user can quantitatively understand the degree of violation in the summarized range.

[0146] Fig. Figure 13 is a representation that provides an example of the aggregated area and shows the area in which the aggregation was performed as a parallel category representation of the area ratio. Fig. 13. The relationship between the areas "Pass," "Violation," or "Unknown" is displayed as band-shaped categories, with the display width of each category being proportional to the size of the area. The area defined by the input variable x5 where x5 ≤ 3.9 includes a pass area and an unknown area. Conversely, the area defined by the input variable x5 where x5 < 3.9 includes a pass area, an unknown area, and a violation area.

[0147] Within the range defined by the input variable x5 (3.9 < x5), the range defined by the input variable x8 (x8 ≤ -0.4) includes a pass-through region, an unknown region, and a violation region. Conversely, the range defined by the input variable x8 (-0.4 < x8) includes a pass-through region and an unknown region.

[0148] In the range defined by the input variable x8, where x8 ≤ -0.4, only the violation area within the range defined by the input variable x5, where x5 ≤ 5.2, is included. This means the violation area comprises 100%. Conversely, the range defined by the input variable x5, where 5.2 < x5, includes a passable area, an unknown area, and a violation area. The violation area comprises 25%.

[0149] This display allows the user to easily understand the result of the determination of the expected output in the summarized area.

[0150] As described above, the verification device 2A according to the second embodiment includes the range acquisition unit 29-1, which acquires an input data range determined by the determination unit 24 for an expected output. The display processing unit 28 displays information about the acquired range on the display device 6. Since the input data range is displayed for each determination result of the expected output of the inference model, the user can ascertain the degree of violation in the input data.

[0151] The verification device 2A according to the second embodiment comprises the sample acquisition unit 29-2, which acquires one or more data samples contained in the area acquired by the area acquisition unit 29-1. The display processing unit 28 displays the acquired data sample on the display device 6. Since the data samples contained in the input data area are displayed for each determination result of the expected output of the inference model, the user can ascertain the degree of violation in the input data for each data sample.

[0152] The verification device 2A according to the second embodiment includes the area summarization unit 29-3, which summarizes a plurality of areas acquired by the area acquisition unit 29-1 into a smaller number of areas. The display processing unit 28 displays information about a summarized area on the display device 6. By displaying areas obtained by summarizing a plurality of input data areas into a smaller number for each determination of the expected output of the inference model, the user can easily grasp the degree of violation in the input data.

[0153] In the verification device 2A according to the second embodiment, the inference model to be verified is a decision tree ensemble model containing a single decision tree or a plurality of decision trees. A sample acquisition unit 29-2, which acquires one or more data samples contained in an acquired domain, is provided. A domain summarization unit 29-3 learns a single decision tree using the acquired data samples and summarizes the domains using a leaf node of the learned decision tree. Since the majority of the input data domains are summarized as simple information for each determination of the expected output of the inference model, the user can easily grasp the degree of violation in the input data.

[0154] It should be noted that combinations of the individual embodiments, modifications of any components of the individual embodiments, or omissions of any components in the individual embodiments are possible. INDUSTRIAL APPLICABILITY

[0155] The verification device according to the present disclosure can, for example, be used to verify an inference model for performing various inferences. REFERENCE MARK LIST 1, 1A Inference System, 2, 2A Verification device, 3 Inference device, 4 Inference model storage device, 5. Result storage device, 6 Display device, 21, 31 Inference model reading unit, 22 Output upper and lower limits calculation unit, 23 Designation unit for an expected output, 24 Unit of determination for an expected output, 25 Input area division unit, 26 Input area label unit, 27 Area ratio calculation unit, 28 Display processing unit, 29-1 Area Procurement Unit, 29-2 Sample procurement unit, 29-3 Area Summary Unit, 32 Inference unit, 100 input interfaces, 101 Output interface, 102 Processing circuit, 103 processor, 104 storage QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 7059220

[0003]

Claims

Verification device comprising: an expected output determination unit for determining whether all output data of an inference model for an input data range, which is a numeric data range, constitute a violation such that the output data is not an expected output; an input range division unit for dividing a range determined to be non-violating within the input data range and for designating a divided range as the input data range that is new to the inference model; a range ratio calculation unit for calculating a ratio of a range for each of the determination results with respect to the input data range; and a verification result output unit for outputting verification result information indicating the ratio of the range. Verification device according to claim 1, wherein the determination unit for an expected output determines one of the following: pass determination, in which all output data of the inference model for the input data domain is an expected output; violation determination, in which all output data of the inference model for the input data domain is not an expected output; and unknown determination, which is neither the pass determination nor the violation determination. Verification device according to claim 2, comprising: an output upper and lower bound calculation unit for calculating at least one upper bound or one lower bound of output data of the inference model for the input data range, wherein the determination unit for an expected output determines one of the pass determination, violation determination and unknown determination for an output data range indicated by the upper bound or the lower bound among the output data of the inference model. Verification device according to claim 3, wherein the inference model to be verified is a decision tree ensemble model containing a single decision tree or a plurality of decision trees, and the input area division unit divides an area in accordance with a branching condition of any decision tree contained in the inference model. Verification device according to claim 1, comprising: a range procurement unit for procuring the input data range determined by the determination unit for an expected output, wherein the verification result output unit displays information about the procured range on a display device. Verification device according to claim 5, comprising: a sample procurement unit for procuring one or more data samples contained in the area procured by the area procurement unit, wherein the output unit for the verification result displays the procured data samples on the display device. Verification device according to claim 5, comprising: an area summarization unit for summarizing a plurality of areas procured by the area procurement unit into a smaller number of areas, wherein the verification result output unit displays information indicating an area in which a summarization was performed on the display device. Verification device according to claim 7, wherein the inference model to be verified is a decision tree ensemble model containing a single decision tree or a plurality of decision trees, the verification device comprises a sample acquisition unit to acquire one or more data samples contained in an acquired domain, and the domain summarization unit learns the individual decision tree using the acquired data samples and summarizes the domains using a leaf node of the learned decision tree. Inference device comprising: an inference model reading unit for reading the inference model verified by the verification device according to any one of claims 1 to 7; and an inference unit for performing inference using the inference model. Inference system comprising: the verification device according to any one of claims 1 to 7; and the inference device according to claim 9. A verification procedure performed by a verification device, comprising the steps of: determining, by a determination unit for expected outputs, whether all output data of an inference model for an input data range, which is a numeric range of values ​​of data, are in violation of the requirement that the output data are not expected outputs; dividing a range determined to be non-violating in the input data range, by an input range division unit, and designating a divided range as the input data range that is new to the inference model; calculating, by a range ratio calculation unit, a ratio of a range for each of the determination results with respect to the input data range; and outputting, by a verification result output unit, verification result information indicating the ratio of the range.

Images