Verification device, verification method, and program

Abstract

This verification device comprises: an inference model loading unit (101) that loads an inference model relating to a trained decision tree model; a change determining unit (103) that determines whether or not a reached leaf of the decision tree model changes in response to a change in a feature input into the loaded inference model; and an output change characteristic determining unit (104) that, on the basis of the determination result from the change determining unit (103), determines whether or not a change in an output value of the inference model violates an output change characteristic.

Description

[Technical Field]

[0001] This disclosure relates to a verification device, a verification method, and a program. [Background technology]

[0002] Conventionally, techniques have been known to train inference models while checking the possible input-output relationships in order to guarantee the monotonicity of the output values ​​of the inference model (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 06-242804 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the prior art disclosed in Patent Document 1 cannot address the verification of output change characteristics for a trained model.

[0005] This disclosure was made to solve the above-mentioned problems and aims to provide a verification device that enables verification of output change characteristics for a trained model. [Means for solving the problem]

[0006] The verification device relating to this disclosure is characterized by comprising: an inference model loading unit that loads an inference model relating to a trained decision tree model; a change determination unit that determines whether or not the reachable leaf of the decision tree model changes due to a change in the feature quantities input to the loaded inference model; and an output change characteristic determination unit that determines whether or not the change in the output value of the inference model violates the output change characteristic based on the determination result by the change determination unit. [Effects of the Invention]

[0007] According to this disclosure, the configuration described above makes it possible to verify the output change characteristics of the trained model. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of the verification device according to Embodiment 1. [Figure 2] This flowchart shows an example of the operation of the verification device according to Embodiment 1. [Figure 3] This flowchart shows an example of the operation of the change detection unit in Embodiment 1. [Figure 4] This figure shows an example of an inference model loaded by the inference model loading unit in Embodiment 1. [Figure 5] Figures 5A and 5B illustrate an example of the operation of the leaf arrival determination unit in Embodiment 1. [Figure 6] Figures 6A and 6B illustrate an example of the operation of the output change characteristic determination unit in Embodiment 1. [Figure 7] Figures 7A and 7B illustrate an example of the operation of the verification result display processing unit in Embodiment 1. [Figure 8] Figures 8A and 8B show examples of the hardware configuration of the verification device according to Embodiment 1. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the drawings. Embodiment 1. Figure 1 shows an example configuration of a verification system equipped with a verification device 1 according to Embodiment 1. The verification system includes, for example, a verification device 1, a storage device 2, and a display device 3, as shown in Figure 1.

[0010] As shown in, for example, FIG. 1, the verification device 1 includes an inference model loading unit 101, an expected output specifying unit 102, a change determination unit 103, an output change characteristic determination unit 104, a verification result storage processing unit 105, and a verification result display processing unit 106.

[0011] The inference model loading unit 101 loads an inference model related to a learned decision tree model.

[0012] Examples of the inference model loaded by the inference model loading unit 101 include a decision tree model and a decision tree ensemble model. In addition, not limited to the above, as the inference model loaded by the inference model loading unit 101, for example, a model in which a neural network or the like is surrogate to a decision tree-based model can also be used.

[0013] When the inference model is a decision tree model, the output value of the reached leaf becomes the output value of the inference model. When the inference model is a decision tree ensemble model, a value integrated by adding or averaging the output values of the reached leaves for each decision tree becomes the output value of the inference model.

[0014] Information indicating the inference model loaded by this inference model loading unit 101 is output to the change determination unit 103.

[0015] The expected output specifying unit 102 specifies an output change characteristic, a feature amount, and a range of the feature amount. The output change characteristic specified by this expected output specifying unit 102 is a change characteristic expected for the output value of the inference model loaded by the inference model loading unit 101.

[0016] Note that the expected output specifying unit 102 may specify an output change characteristic, a feature amount, and a range of the feature amount according to an operation by the user. Alternatively, the expected output specification unit 102 may automatically specify the output change characteristics, features, and range of features. For example, the expected output specification unit 102 may automatically specify the output change characteristics, features, and range of features according to predetermined rules.

[0017] The output change characteristics specified by the expected output specification unit 102 include, for example, characteristics related to monotonicity, characteristics related to the shape of the graph showing the output value, or characteristics related to the limitation of the slope of the output value. Characteristics related to monotonicity include monotonically increasing and monotonically decreasing.

[0018] The information indicating the output change characteristics specified by the expected output specification unit 102 is output to the output change characteristic determination unit 104. Furthermore, information indicating the feature quantities and ranges of the feature quantities specified by the expected output specification unit 102 is output to the change determination unit 103.

[0019] The change determination unit 103 determines whether the reach of the decision tree model changes due to a change in the features input to the inference model, based on the inference model loaded by the inference model loading unit 101 and the features and feature ranges specified by the expected output specification unit 102. Information indicating the determination result by the change determination unit 103 is output to the output change characteristic determination unit 104.

[0020] This change determination unit 103 includes, for example, a branch threshold extraction unit 1031 and a leaf arrival determination unit 1032, as shown in Figure 1.

[0021] The branch threshold extraction unit 1031 extracts branch thresholds for features to be input to the inference model, based on the inference model loaded by the inference model loading unit 101 and the features and feature ranges specified by the expected output specification unit 102. In this process, the branch threshold extraction unit 1031 extracts all branch thresholds for the specified features within the specified feature range from the loaded inference model. Note that these branch thresholds also include the minimum and maximum values ​​of the specified features. The branch threshold information extracted by the branch threshold extraction unit 1031 is output to the leaf arrival determination unit 1032.

[0022] The leaf arrival determination unit 1032 determines whether the leaf arrival changes due to a change in the branch threshold, based on the inference model loaded by the inference model loading unit 101 and the branch threshold extracted by the branch threshold extraction unit 1031.

[0023] Furthermore, the leaf arrival determination unit 1032 detects the amount of change between the output value of the leaf arrival before the change in the branch threshold and the output value of the leaf arrival after the change in the branch threshold. Furthermore, if the destination leaf determination unit 1032 determines that the destination leaf remains unchanged, it sets the amount of change due to the output value of the unchanging destination leaf to zero.

[0024] Furthermore, if the decision tree model is a decision tree ensemble model that includes multiple decision trees, the destination leaf determination unit 1032 may exclude a decision tree from the determination target of the output change characteristic determination unit 104 if there is a decision tree in which the destination leaf does not change at all branch thresholds.

[0025] Furthermore, if there are multiple ranges of feature quantities in which the reached leaf does not change with respect to a change in the branch threshold, the arrival leaf determination unit 1032 may associate information indicating each of these ranges of feature quantities with information indicating that the amount of change in those cases is zero, and output this information to the output change characteristic determination unit 104.

[0026] Furthermore, the leaf arrival determination unit 1032 may detect the range of feature quantities that reach the leaf arrival, which changes in accordance with the change in the branch threshold. Alternatively, the leaf arrival determination unit 1032 may detect the range of feature quantities that reach a leaf that does not change with the change in the branch threshold.

[0027] Information indicating the determination result by the leaf arrival determination unit 1032 is output to the output change characteristic determination unit 104.

[0028] The output change characteristic determination unit 104 determines whether the change in the output value of the inference model read by the inference model reading unit 101 violates the output change characteristic, based on the output change characteristic specified by the expected output specification unit 102 and the determination result by the change determination unit 103.

[0029] In particular, if the change determination unit 103 determines that the reached leaf has changed, the output change characteristic determination unit 104 determines whether the change in the output value of the inference model in response to the change in the feature for which the reached leaf has been determined to have changed violates the output change characteristic. In this case, the output change characteristic determination unit 104 detects, among the reachable leaves that change in accordance with the change in the branch threshold, the reachable leaf whose change in the output value of the inference model violates the output change characteristic. If the decision tree model is a decision tree ensemble model, the output change characteristic determination unit 104 first detects feasible reachable leaf combinations among the reachable leaf combinations that change in accordance with the change in the branch threshold, and then detects reachable leaf combinations whose change in the output value of the inference model violates the output change characteristic.

[0030] The information indicating the determination result by the output change characteristic determination unit 104 is output to the verification result storage processing unit 105 and the verification result display processing unit 106.

[0031] The information indicating the determination result by the output change characteristic determination unit 104 includes, for example, information indicating at least one of the following: a reachable leaf that is determined to violate the output change characteristic; a feature that reaches the reachable leaf that is determined to violate the output change characteristic, the range of feature specified by the expected output specification unit 102; and the amount of change in the output value of the inference model. Furthermore, the information indicating the determination result by the output change characteristic determination unit 104 may also include, in addition to the above, information indicating the range of feature quantities that reach the destination leaf determined to be in violation of the output change characteristic, other than the feature quantities specified by the expected output specification unit 102.

[0032] Furthermore, the information indicating the determination result by the output change characteristic determination unit 104 may also include, in addition to the above, information indicating, for example, the reachable leaf which is determined not to violate the output change characteristic, the range of feature quantities that reach the reachable leaf which does not violate the output change characteristic and which are specified by the expected output specification unit 102, and at least one of the change in the output value of the inference model. Furthermore, the information indicating the determination result by the output change characteristic determination unit 104 may also include, in addition to the above, information indicating the range of feature quantities other than those specified by the expected output specification unit 102 that reach the destination leaf determined not to violate the output change characteristics.

[0033] The verification result storage processing unit 105 stores information indicating the determination result by the output change characteristic determination unit 104 in the storage device 2.

[0034] In this case, the verification result storage processing unit 105 stores in the storage device 2 information indicating at least one of the following: the destination leaf determined by the output change characteristic determination unit 104 to be in violation of the output change characteristics; the range of feature quantities that reach the destination leaf determined to be in violation of the output change characteristics, which are specified by the expected output specification unit 102; and the amount of change in the output value of the inference model. Note that the range of feature quantities referred to here may be only the branch threshold, or it may be a range that includes the values ​​between branch thresholds. In addition to the above, the verification result storage processing unit 105 may also store in the storage device 2 information indicating the range of feature quantities other than those specified by the expected output specification unit 102, which are feature quantities necessary to reach a leaf that has been determined to violate the output change characteristics.

[0035] Furthermore, the verification result storage processing unit 105 may also store in the storage device 2 information indicating at least one of the following: the reachable leaf determined by the output change characteristic determination unit 104 as not violating the output change characteristics; the range of feature quantities that reach the reachable leaf determined not to violate the output change characteristics, which are specified by the expected output specification unit 102; and the amount of change in the output value of the inference model. In addition, the verification result storage processing unit 105 may also store in the storage device 2 information indicating the range of feature quantities other than those specified by the expected output specification unit 102 that reach the destination leaf determined not to violate the output change characteristics.

[0036] The verification result display processing unit 106 displays information indicating the determination result by the output change characteristic determination unit 104 on the display device 3.

[0037] In this case, the verification result display processing unit 106 causes the display device 3 to display information indicating at least one of the following: the destination leaf determined by the output change characteristic determination unit 104 to violate the output change characteristics; the range of features that reach the destination leaf determined to violate the output change characteristics, other than the features specified by the expected output specification unit 102; and the amount of change in the output value of the inference model. Note that the range of features referred to here may be only the branch threshold, or it may be a range that includes the values ​​between branch thresholds.

[0038] Furthermore, the verification result display processing unit 106 may cause the display device 3 to display information indicating at least one of the following: the range of feature quantities that reach the destination leaf determined by the output change characteristic determination unit 104 to violate the output change characteristics, other than the feature quantities specified by the expected output specification unit 102, or the degree of violation. Furthermore, information indicating the degree of the violation could include, for example, the excessive volume of the violation area.

[0039] Figure 1 shows the case where the verification device 1 is equipped with an expected output designation unit 102. However, the expected output designation unit 102 is not an essential component of the verification device 1 and may not be provided in the verification device 1. For example, if there is only one option for the output change characteristics and feature quantities, or if the range of the feature quantities can be handled by, for example, setting it to the entire range, then the expected output specification unit 102 is not necessary for the verification device 1.

[0040] Furthermore, the storage device 2 is a storage device that stores information indicating the verification results by the verification device 1. In Figure 1, the storage device 2 is located outside the verification device 1, but the storage device 2 only needs to be accessible from the verification device 1, and may be a storage device provided by a computer that functions as the verification device 1.

[0041] This storage device 2 may be, for example, a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), or EEPROM (Electrically EPROM), or a magnetic disk, flexible disk, optical disk, compact disk, minidisc, or DVD (Digital Versatile Disc).

[0042] Furthermore, the display device 3 is a display device that shows information indicating the verification results by the verification device 1. In Figure 1, the display device 3 is located outside the verification device 1, but the display device 3 only needs to be accessible from the verification device 1, and may be a display device provided by the computer that functions as the verification device 1.

[0043] This display device 3 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display device.

[0044] The verification device 1 is implemented, for example, by a computer equipped with a communication unit, an arithmetic unit, and a memory unit. The communication unit communicates with the memory device 2 or the display device 3 via a wired signal line or a wireless communication network. For example, the communication unit is a communication device capable of mobile communication using communication methods such as LTE, 3G, 4G, or 5G. Alternatively, the communication unit may be a short-range wireless communication means such as Bluetooth®. The communication unit includes the input interface 54 and output interface 55 shown in Figures 8A and 8B.

[0045] The arithmetic unit controls the overall operation of the verification device 1. The arithmetic unit comprises an inference model reading unit 101, an expected output specification unit 102, a change determination unit 103, an output change characteristic determination unit 104, a verification result storage processing unit 105, and a verification result display processing unit 106. The arithmetic unit executes an information processing application for verifying the inference model, thereby realizing the various functions of the inference model reading unit 101, the expected output specification unit 102, the change determination unit 103, the output change characteristic determination unit 104, the verification result storage processing unit 105, and the verification result display processing unit 106. The arithmetic unit includes the processing circuit 51 in Figure 8A or the processor 52 in Figure 8B.

[0046] The memory unit stores information processing applications and information used in the calculations of the arithmetic unit. The memory unit is a storage device provided by the computer that functions as the verification device 1, and includes storage such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), or the memory 53 shown in Figures 8A and 8B. The memory unit only needs to be accessible by the verification device 1 and may be located outside the verification device 1.

[0047] Next, an example of the operation of the verification device 1 according to Embodiment 1 shown in Figure 1 will be explained with reference to Figure 2.

[0048] In an example of the operation of the verification device 1 according to Embodiment 1 shown in Figure 1, for example, as shown in Figure 2, first the inference model loading unit 101 loads an inference model relating to a trained decision tree model (step ST101).

[0049] Furthermore, the expected output specification unit 102 specifies the output change characteristics, the feature quantities, and the range of the feature quantities (step ST102).

[0050] Next, the branch threshold extraction unit 1031 extracts branch thresholds for the features to be input to the inference model based on the inference model loaded by the inference model loading unit 101 and the features and feature ranges specified by the expected output specification unit 102 (step ST103). At this time, the branch threshold extraction unit 1031 extracts all branch thresholds for the specified features, including the minimum and maximum values, from the loaded inference model, within the range of the specified features.

[0051] Next, the leaf arrival determination unit 1032 determines whether the leaf arrival changes due to a change in the branch threshold, based on the inference model read by the inference model reading unit 101 and the branch threshold extracted by the branch threshold extraction unit 1031 (step ST104).

[0052] Furthermore, the leaf arrival determination unit 1032 detects the amount of change between the output value of the leaf arrival before the change in the branch threshold and the output value of the leaf arrival after the change in the branch threshold. Furthermore, if the destination leaf determination unit 1032 determines that the destination leaf remains unchanged, it sets the amount of change due to the output value of the unchanging destination leaf to zero.

[0053] Furthermore, if the decision tree model is a decision tree ensemble model that includes multiple decision trees, the destination leaf determination unit 1032 may exclude a decision tree from the determination target of the output change characteristic determination unit 104 if there is a decision tree in which the destination leaf does not change at all branch thresholds.

[0054] Furthermore, if there are multiple ranges of feature quantities in which the reached leaf does not change with respect to a change in the branch threshold, the arrival leaf determination unit 1032 may associate information indicating each of these ranges of feature quantities with information indicating that the amount of change in those cases is zero, and output this information to the output change characteristic determination unit 104.

[0055] Furthermore, the leaf arrival determination unit 1032 may detect the range of feature quantities that reach the leaf arrival, which changes in accordance with the change in the branch threshold. Alternatively, the leaf arrival determination unit 1032 may detect the range of feature quantities that reach a leaf that does not change with the change in the branch threshold.

[0056] A detailed example of the operation of this leaf arrival determination unit 1032 will be described later.

[0057] Next, the output change characteristic determination unit 104 determines whether the change in the output value of the inference model read by the inference model reading unit 101 violates the output change characteristic, based on the output change characteristic specified by the expected output specification unit 102 and the determination result by the change determination unit 103 (reached leaf determination unit 1032) (step ST105).

[0058] In particular, if the change determination unit 103 determines that the reached leaf has changed, the output change characteristic determination unit 104 determines whether the change in the output value of the inference model in response to the change in the feature for which the reached leaf has been determined to have changed violates the output change characteristic. In this case, the output change characteristic determination unit 104 detects, among the reachable leaves that change in accordance with the change in the branch threshold, the reachable leaf whose change in the output value of the inference model violates the output change characteristic. If the decision tree model is a decision tree ensemble model, the output change characteristic determination unit 104 first detects feasible reachable leaf combinations among the reachable leaf combinations that change in accordance with the change in the branch threshold, and then detects reachable leaf combinations whose change in the output value of the inference model violates the output change characteristic.

[0059] Next, the verification result storage processing unit 105 stores information indicating the determination result by the output change characteristic determination unit 104 in the storage device 2 (step ST106).

[0060] In this process, the verification result storage processing unit 105 stores in the storage device 2 information indicating at least one of the following: the destination leaf determined by the output change characteristic determination unit 104 to be in violation of the output change characteristics; the range of the feature quantities that reach the destination leaf determined to be in violation of the output change characteristics, which are specified by the expected output specification unit 102; and the amount of change in the output value of the inference model. In addition to the above, the verification result storage processing unit 105 may also store in the storage device 2 information indicating the range of feature quantities other than those specified by the expected output specification unit 102, which are feature quantities necessary to reach a leaf that has been determined to violate the output change characteristics.

[0061] Furthermore, the verification result storage processing unit 105 may also store in the storage device 2 information indicating at least one of the following: the reachable leaf determined by the output change characteristic determination unit 104 as not violating the output change characteristics; the range of feature quantities that reach the reachable leaf determined not to violate the output change characteristics, which are specified by the expected output specification unit 102; and the amount of change in the output value of the inference model. In addition, the verification result storage processing unit 105 may also store in the storage device 2 information indicating the range of feature quantities other than those specified by the expected output specification unit 102 that reach the destination leaf determined not to violate the output change characteristics.

[0062] Furthermore, the verification result display processing unit 106 displays information indicating the determination result by the output change characteristic determination unit 104 on the display device 3 (step ST107).

[0063] In this case, the verification result display processing unit 106 causes the display device 3 to display information indicating at least one of the following: the destination leaf determined by the output change characteristic determination unit 104 to be in violation of the output change characteristics; the range of features that reach the destination leaf determined to be in violation of the output change characteristics, other than the features specified by the expected output specification unit 102; and the amount of change in the output value of the inference model.

[0064] Furthermore, the verification result display processing unit 106 may cause the display device 3 to display information indicating at least one of the following: the range of feature quantities that reach the destination leaf determined by the output change characteristic determination unit 104 to violate the output change characteristics, other than the feature quantities specified by the expected output specification unit 102, or the degree of violation.

[0065] Next, an example of the operation of the leaf determination unit 1032 in Embodiment 1 shown in Figure 1 will be explained with reference to Figure 3. Figure 3 shows an example of operation when the inference model loaded into the inference model loading unit 101 is a decision tree ensemble model.

[0066] In the example of operation of the leaf arrival determination unit 1032 in Embodiment 1 shown in Figure 1, for example as shown in Figure 3, the leaf arrival determination unit 1032 first performs initialization (step ST201). That is, the leaf arrival determination unit 1032 sets the identification number (i) of the decision tree to 1 and the identification number (j) of the branch threshold of the feature specified by the expected output specification unit 102 to 1.

[0067] Next, the leaf arrival determination unit 1032 extracts all the ranges that can reach a leaf for each feature other than those specified by the expected output specification unit 102 for the i-th decision tree (step ST202).

[0068] Next, the leaf arrival determination unit 1032 determines that for the i-th decision tree, the branch threshold of the feature specified by the expected output specification unit 102 is a j In the case of reachable leaves and a j+1 Extract the reachable leaves in each case (step ST203). Furthermore, the destination leaf may also change due to features other than those specified by the expected output specification unit 102.

[0069] Next, the leaf arrival determination unit 1032 determines that for the i-th decision tree, the branch threshold of the feature specified by the expected output specification unit 102 is a j In the case of a j+1Determine if the reachable leaf may change depending on the case (step ST204).

[0070] In step ST204, if the destination leaf determination unit 1032 determines that the destination leaf for the i-th decision tree will not change, that is, the destination leaf will not change at all, the sequence proceeds to step ST208.

[0071] On the other hand, in step ST204, if the destination leaf determination unit 1032 determines that the destination leaf may change, it designates the destination leaf before and after the change as the change leaf and detects the amount of change in the output value of the change leaf (step ST205).

[0072] Next, the leaf arrival determination unit 1032 determines that the branching threshold in the feature quantity specified by the expected output specification unit 102 is a j In the case of a j+1 In both cases, determine if there is a common decision path, that is, if the destination leaf does not change (step ST206).

[0073] In step ST206, if the destination leaf determination unit 1032 determines that the destination leaf may not change, it designates that destination leaf as a constant leaf and sets the change in value at the constant leaf to zero (step ST207).

[0074] Next, the leaf arrival determination unit 1032 determines whether processing has been performed for all branch thresholds of the feature quantities specified by the expected output specification unit 102 (step ST208). In step ST208, if the leaf arrival determination unit 1032 determines that no processing has been performed for any branch thresholds, it adds 1 to j (step ST209). The sequence then returns to step ST203 and repeats the above process.

[0075] On the other hand, in step ST208, if the leaf arrival determination unit 1032 determines that it has processed all branch thresholds, it determines whether it has processed all decision trees (step ST210).

[0076] In step ST210, if the reachable leaf determination unit 1032 determines that no processing has been performed on any of the decision trees, it adds 1 to i (step ST211). The sequence then returns to step ST202 and repeats the above process.

[0077] On the other hand, in step ST210, if the leaf arrival determination unit 1032 determines that processing has been performed for all decision trees, the sequence ends.

[0078] Next, a specific example of the operation of the verification device 1 according to Embodiment 1 shown in Figure 1 will be explained with reference to Figures 4 to 7.

[0079] Here, for example, as shown in Figure 4, we assume that the inference model loaded by the inference model loading unit 101 is a decision tree ensemble model using three decision trees (Tree1, Tree2, Tree3). Furthermore, here, if the change in the output value of the reached leaf before and after the change in the branch threshold is a positive value, we assume that the output value of the reached leaf in the range between that branch threshold is monotonically increasing, and if the change in the output value of the reached leaf before and after the change in the branch threshold is a negative value, we assume that the output value of the reached leaf in the range between that branch threshold is monotonically decreasing. In Figures 4-7, "Leaf" indicates the reached leaf, and "Value" indicates the output value of the reached leaf.

[0080] Furthermore, suppose the expected output specification unit 102 specifies, as an output change characteristic, that the output value of the decision tree ensemble model increases monotonically with increasing feature quantities. Also, suppose the expected output specification unit 102 specifies the feature quantities and the range of feature quantities, for example, as shown in equation (1) below. In equation (1), x1, x2, ..., x Nrepresents a feature quantity, and X represents a group of feature quantities including x1, x2, …, x N The feature quantity group is shown. X = (x1, x2, ···, x N ) -3.5 ≤ x1 ≤ 5.2, …, 15 ≤ x N ≤ 40 (1)

[0081] Incidentally, hereinafter, the case where the verification device 1 verifies that the output value of the decision tree ensemble model monotonically increases with respect to the increase in the feature quantity (x1) specified by the expected output specifying unit 102 is shown, but the verification for other feature quantities is the same.

[0082] In this case, first, the branch threshold extraction unit 1031 extracts the branch threshold of the feature quantity based on the feature quantity specified by the expected output specifying unit 102 and the range of the feature quantity. For example, the branch threshold extraction unit 1031 extracts a branch threshold as shown in the following formula (2) for x1 in formula (1). In formula (2), a1, a2, …, a N represents the branch threshold in x1, and A represents a group of branch thresholds including a1, a2, …, a N The branch threshold group is shown. A = (a1, a2, ···, a N ) = (-inf, 3, 5, ···, inf) (2)

[0083] Next, the reached leaf determination unit 1032 determines whether the reached leaf changes as the branch threshold changes based on the inference model read by the inference model reading unit 101 and the branch threshold extracted by the branch threshold extraction unit 1031. <​​​​​​Also, in Tree1, when 8 < x2 ≤ 15, whether x1 = 3 or x1 = 5, the reachable leaf is reachable leaf 1-3. Therefore, even if x1 changes from 3 to 5, the output value of the reachable leaf does not change, and the change amount is 0. Also, the decision path in this case is a common decision path. Also, in Tree1, when 15 < x2 ≤ inf, whether x1 = 3 or x1 = 5, the reachable leaf is reachable leaf 1-4. Therefore, even if x1 changes from 3 to 5, the output value of the reachable leaf does not change, and the change amount is 0. Also, the decision path in this case is a common decision path.

[0085] Also, as shown in FIG. 5, in Tree2, when -inf < x2 ≤ 5 and x1 = 3, the reachable leaf is reachable leaf 2-1, but when x1 = 5, the reachable leaf is reachable leaf 2-3. Also, the change amount of the output value of the reachable leaf due to the change of x1 from 3 to 5 is +3. Also, in Tree2, when 5 < x2 ≤ 10 and x1 = 3, the reachable leaf is reachable leaf 2-2, but when x1 = 5, the reachable leaf is reachable leaf 2-3. Also, the change amount of the output value of the reachable leaf due to the change of x1 from 3 to 5 is +2. Also, in Tree2, when 10 < x2 ≤ inf and x1 = 3, the reachable leaf is reachable leaf 2-2, but when x1 = 5, the reachable leaf is reachable leaf 2-4. Also, the change amount of the output value of the reachable leaf due to the change of x1 from 3 to 5 is +4.

[0086] Also, as shown in FIG. 5, in Tree3, whether x1 = 3 or x1 = 5, the reachable leaf is reachable leaf 3-1. Therefore, even if x1 changes from 3 to 5, the output value of the reachable leaf does not change, and the change amount is 0. Also, the decision path in this case is a common decision path.

[0087] In Figure 5B, the solid thick line path shows the determined path taken only when x1=3, the dashed thick line path shows the determined path taken only when x1=5, and the dashed thin line path shows the determined path (common path) taken in both cases of x1=3 and x1=5.

[0088] In other words, in the example shown in Figure 5, the arrival leaf determination unit 1032 detects arrival leaves 1-1 and 1-2, arrival leaves 2-1 and 2-3, arrival leaves 2-2 and 2-3, and arrival leaves 2-2 and 2-4 as arrival leaves that change when x1 changes from 3 to 5. Furthermore, the leaf arrival determination unit 1032 detects the change in output value due to the change from leaf arrival 1-1 to leaf arrival 1-2 (-3), the change in output value due to the change from leaf arrival 2-1 to leaf arrival 2-3 (+3), the change in output value due to the change from leaf arrival 2-2 to leaf arrival 2-3 (+2), and the change in output value due to the change from leaf arrival 2-2 to leaf arrival 2-4 (+4).

[0089] In the example shown in Figure 5, Tree 3 is a decision tree in which the reachable leaf remains the same at all branch thresholds. In this case, the reachable leaf determination unit 1032 may exclude Tree 3 from the determination target of the output change characteristic determination unit 104. Figure 5A shows a table with Tree 3 excluded. This makes it possible to narrow down the determination target of the output change characteristic determination unit 104, thereby reducing the processing load.

[0090] Also, as shown in FIG. 5A, for Tree1, reachable leaf 1-3 and reachable leaf 1-4 are reachable leaves that do not change with the change of the branch threshold. Thus, for reachable leaves that do not change with the change of the branch threshold, the change amount of the output value is zero. In this case, the reachable leaf determination unit 1032 may associate the information indicating the range of the feature amount (-inf < x1 ≦ inf, 8 < x2 ≦ 15) reaching reachable leaf 1-3 and the range of the feature amount (-inf < x1 ≦ inf, 15 < x2 ≦ inf) reaching reachable leaf 1-4 with the information indicating that the change amount at that time is zero, and output it to the output change characteristic determination unit 104. Thereby, when the change amount is zero, it becomes possible to summarize the information into one, and the processing load can be reduced.

[0091] Next, the output change characteristic determination unit 104 determines whether the change in the output value of the inference model read by the inference model loading unit 101 violates the output change characteristic based on the output change characteristic specified by the expected output specifying unit 102 and the determination result by the change determination unit 103 (reachable leaf determination unit 1032). Here, the inference model read by the inference model loading unit 101 is a decision tree ensemble model, and it is assumed that the output value of the inference model is the sum value of the output values in each decision tree. In this case, the output change characteristic determination unit 104 detects the sum value of the change amounts of the output values of Tree1 to Tree3 as the change amount of the output value of the decision tree ensemble model.

[0092] <009>In this case, as shown in FIG. 6, when changing from reachable leaf 1-1 to reachable leaf 1-2 and changing from reachable leaf 2-2 to reachable leaf 2-3, the change amount of the output value of the decision tree ensemble model becomes a negative value (-1), and in the case of other combinations, the change amount of the output value becomes a positive value.

[0093] Also, here, the output change characteristic determination unit 104 determines whether the output value of the decision tree ensemble model increases monotonically with the increase of the feature amount (x1). In the case of Figure 6, the output change characteristic determination unit 104 determines that when the destination leaf changes from 1-1 to 1-2, and from 2-2 to 2-3, the change in the output value of the decision tree ensemble model is a negative value (-1), thus violating the principle of monotonic increase.

[0094] Next, the verification result display processing unit 106 displays information indicating the determination result by the output change characteristic determination unit 104 on the display device 3. In this case, the verification result display processing unit 106 displays information indicating at least one of the following: the destination leaf determined by the output change characteristic determination unit 104 to be in violation of the output change characteristics; the range of the feature quantities that reach the destination leaf determined to be in violation of the output change characteristics, which are specified by the expected output specification unit 102; and the amount of change in the output value of the inference model.

[0095] Furthermore, the verification result display processing unit 106 may cause the display device 3 to display information indicating at least one of the following: the range of feature quantities that reach the destination leaf determined by the output change characteristic determination unit 104 to violate the output change characteristics, other than the feature quantities specified by the expected output specification unit 102, or the degree of violation.

[0096] For example, Figures 7A and 7B show examples of screens displayed on the display device 3 by the verification result display processing unit 106. In Figure 7A, the tree structure of the inference model loaded by the inference model loading unit 101 is displayed, and the parts of the reached leaves that the output change characteristic determination unit 104 has determined to be in violation of the output change characteristics are highlighted and displayed differently from the other reached leaves. In the example in Figure 7A, reached leaves 1-1 and 1-2 of Tree 1, and reached leaves 2-2 and 2-3 of Tree 2 are reached leaves that have been determined to be in violation of the output change characteristics, and these parts of the reached leaves are highlighted by hatching.

[0097] Furthermore, for example, the decision path to reach a leaf that has been determined by the output change characteristic determination unit 104 to be in violation of the output change characteristics may also be highlighted. Furthermore, for example, the range of each feature that reaches a leaf that the output change characteristic determination unit 104 has determined to be in violation of the output change characteristic may be displayed as text or the like.

[0098] Furthermore, in Figure 7B, the output values ​​of the inference model are shown by solid lines when features other than those specified by the expected output specification unit 102 are fixed to some value, and the specified features are changed. The portion of this solid line that slopes in the negative direction is the part that violates the output change characteristic (monotonically increasing in the example shown in the figure). Moreover, the greater the slope in the negative direction, the greater the degree of violation. Furthermore, in Figure 7B, the ultracuboidal volume of the region related to the violation of the output change characteristics is shown as a rectangular region. The larger this rectangular region, the greater the degree of the violation. Furthermore, Figure 7B shows the rate of change in the output value of the inference model, indicated by the number of black circles. A larger number of black circles indicates a greater degree of violation.

[0099] As described above, according to this embodiment 1, the verification device 1 includes an inference model loading unit 101 that loads an inference model relating to a trained decision tree model, a change determination unit 103 that determines whether or not the reachable leaf of the decision tree model changes due to changes in the feature quantities input to the loaded inference model, and an output change characteristic determination unit 104 that determines whether or not the change in the output value of the inference model violates the output change characteristics based on the determination result by the change determination unit 103. Furthermore, according to this embodiment 1, the change determination unit 103 includes a branch threshold extraction unit 1031 that extracts branch thresholds for feature quantities to be input to the inference model, and a destination leaf determination unit 1032 that determines whether or not the destination leaf changes based on the change in the branch threshold extracted by the branch threshold extraction unit 1031. Furthermore, according to this embodiment 1, the leaf arrival determination unit 1032 detects the amount of change between the output value of the leaf arrival before the change in the branch threshold and the output value of the leaf arrival after the change in the branch threshold. Furthermore, according to this embodiment 1, if the change determination unit 103 determines that the reached leaf has changed, the output change characteristic determination unit 104 determines whether the change in the output value of the inference model in response to the change in the feature quantity for which the reached leaf has been determined to have changed violates the output change characteristic. Furthermore, according to this embodiment 1, if the change determination unit 103 determines that the destination leaf has not changed, it sets the output value of the destination leaf that has not changed to zero. As a result, the verification device 1 according to Embodiment 1 makes it possible to verify the output change characteristics of a trained model.

[0100] Furthermore, according to this embodiment 1, the inference model is a decision tree ensemble model that includes multiple decision trees, and the reach leaf determination unit 1032 excludes a decision tree from the determination target of the output change characteristic determination unit 104 if there is a decision tree in which the reach leaf does not change at all branch thresholds. As a result, the verification device 1 according to Embodiment 1 can reduce the processing load.

[0101] Furthermore, according to this embodiment 1, the system includes an expected output specification unit 102 that specifies output change characteristics, feature quantities, and ranges of feature quantities. The change determination unit 103 determines whether the reachable leaf changes due to changes in the feature quantities input to the inference model, based on the specified feature quantities and ranges of feature quantities. The output change characteristic determination unit 104 determines whether the change in the output value of the inference model violates the specified output change characteristics. As a result, in the verification device 1 according to Embodiment 1, when there are options for output change characteristics, feature quantities, and ranges of feature quantities, it becomes possible to specify force change characteristics, feature quantities, and ranges of feature quantities, respectively.

[0102] Furthermore, according to this embodiment 1, the system includes a verification result storage processing unit 105 that stores in the storage device 2 information indicating at least one of the following: a reachable leaf determined to be in violation of the output change characteristics, the range of feature quantities that reach the reachable leaf determined to be in violation of the output change characteristics, and the amount of change in the output value of the inference model. As a result, the verification device 1 according to Embodiment 1 can store not only information on whether or not the output change characteristics are violated, but also information regarding violations of the output change characteristics in the storage device 2.

[0103] Furthermore, according to this embodiment 1, the system includes a verification result display processing unit 106 that performs processing to display on the display device 3 information indicating at least one of the following: the reachable leaf determined to violate the output change characteristics, the range of features that reach the reachable leaf determined to violate the output change characteristics, and the amount of change in the output value of the inference model. Furthermore, according to this embodiment 1, the verification result display processing unit 106 controls the display device 3 to display information indicating at least one of the following: a range of feature quantities different from the feature quantities needed to reach the leaf that the output change characteristic determination unit 104 has determined to be in violation of the output change characteristics, or the degree of the violation. As a result, in the verification device 1 according to Embodiment 1, it is possible to display not only whether or not the output change characteristics are violated, but also information regarding violations of the output change characteristics on the display device 3.

[0104] Furthermore, according to this embodiment 1, the output change characteristics are characteristics related to monotonicity. This makes it possible to verify the reliability of a trained machine learning model, particularly the trend of changes in the monotonic output value, using the verification device 1 according to Embodiment 1.

[0105] Furthermore, according to this embodiment 1, the verification method includes the steps of: the inference model loading unit 101 loading an inference model relating to a trained decision tree model; the change determination unit 103 determining whether the reach of the decision tree model changes due to changes in the feature quantities input to the loaded inference model; and the output change characteristic determination unit 104 determining whether the change in the output value of the inference model violates the output change characteristics based on the determination result by the change determination unit 103. This makes it possible to verify the output change characteristics of a trained model in the verification method according to Embodiment 1.

[0106] Furthermore, according to this embodiment 1, the program causes the computer to perform the following processes: load an inference model relating to a trained decision tree model; determine whether the reachable leaf of the decision tree model changes due to changes in the features input to the loaded inference model; and determine, based on the determination result, whether the change in the output value of the inference model violates the output change characteristics. This makes it possible to verify the output change characteristics of the trained model in the program according to Embodiment 1.

[0107] Finally, with reference to Figure 8, an example of the hardware configuration of the verification device 1 according to Embodiment 1 will be described. The functions of the inference model reading unit 101, expected output specification unit 102, change determination unit 103, output change characteristic determination unit 104, verification result storage processing unit 105, and verification result display processing unit 106 in the verification device 1 are realized by the processing circuit 51. The processing circuit 51 may be dedicated hardware as shown in Figure 8A, or it may be a processor (Central Processing Unit, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in memory 53, as shown in Figure 8B.

[0108] Furthermore, in Figures 8A and 8B, the input interface 54 is an interface that relays data acquired by the verification device 1 from the storage device 2 or the display device 3. The output interface 55 is an interface that relays data output from the verification device 1 to the storage device 2 or the display device 3.

[0109] If the processing circuit 51 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each part, the inference model reading unit 101, the expected output specification unit 102, the change determination unit 103, the output change characteristic determination unit 104, the verification result storage processing unit 105, and the verification result display processing unit 106, may be implemented individually by the processing circuit 51, or the functions of each part may be implemented together by the processing circuit 51.

[0110] When the processing circuit 51 is a processor 52, the functions of the inference model reading unit 101, expected output specification unit 102, change determination unit 103, output change characteristic determination unit 104, verification result storage processing unit 105, and verification result display processing unit 106 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in memory 53. The processing circuit 51 realizes the functions of each unit by reading and executing the programs stored in memory 53. In other words, the verification device 1 is equipped with memory for storing programs that, when executed by the processing circuit 51, result in the execution of each step shown in Figures 2 and 3, for example. Furthermore, these programs can be said to cause the computer to execute the procedures and methods of the inference model reading unit 101, expected output specification unit 102, change determination unit 103, output change characteristic determination unit 104, verification result storage processing unit 105, and verification result display processing unit 106. Here, memory 53 includes, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidiscs, or DVDs (Digital Versatile Discs).

[0111] Furthermore, some of the functions of the inference model loading unit 101, expected output specification unit 102, change determination unit 103, output change characteristic determination unit 104, verification result storage processing unit 105, and verification result display processing unit 106 may be implemented using dedicated hardware and some using software or firmware. For example, the inference model loading unit 101 can be implemented using a dedicated hardware processing circuit 51, while the expected output specification unit 102, change determination unit 103, output change characteristic determination unit 104, verification result storage processing unit 105, and verification result display processing unit 106 can be implemented by the processing circuit 51 reading and executing a program stored in memory 53.

[0112] Thus, the processing circuit 51 can realize each of the above-mentioned functions through hardware, software, firmware, or a combination thereof.

[0113] Furthermore, any component of the embodiment can be modified, or any component of the embodiment can be omitted. [Industrial applicability]

[0114] The verification device described herein enables verification of output change characteristics for a trained model and is suitable for use in verification devices and the like. [Explanation of Symbols]

[0115] 1 Verification device, 2 Storage device, 3 Display device, 51 Processing circuit, 52 Processor, 53 Memory, 54 Input interface, 55 Output interface, 101 Inference model reading unit, 102 Expected output specification unit, 103 Change determination unit, 104 Output change characteristic determination unit, 105 Verification result storage processing unit, 106 Verification result display processing unit, 1031 Branch threshold extraction unit, 1032 Reached leaf determination unit.

Claims

1. An inference model loading unit loads an inference model related to a pre-trained decision tree model, A change determination unit determines whether the reach of the decision tree model changes based on changes in the features input to the loaded inference model, Based on the determination result by the change determination unit, an output change characteristic determination unit determines whether or not the change in the output value of the inference model violates the output change characteristics, A verification device equipped with the following features.

2. The aforementioned change determination unit, A branch threshold extraction unit that extracts branch thresholds for the feature quantities to be input to the inference model, The system includes a leaf arrival determination unit that determines whether or not the leaf arrival changes based on the change in the branch threshold extracted by the branch threshold extraction unit. The verification apparatus according to claim 1, characterized by the features described above.

3. The aforementioned decision tree model is a decision tree ensemble model that includes multiple decision trees. The aforementioned leaf arrival determination unit excludes a decision tree from the output change characteristic determination unit's determination target if there is a decision tree in which the leaf arrival remains unchanged at all branch thresholds. The verification apparatus according to feature 2.

4. The leaf arrival determination unit detects the amount of change between the output value of the leaf arrival before the change in the branch threshold and the output value of the leaf arrival after the change in the branch threshold. The verification apparatus according to feature 2.

5. The leaf arrival determination unit detects the amount of change between the output value of the leaf arrival before the change in the branch threshold and the output value of the leaf arrival after the change in the branch threshold. The verification apparatus according to claim 3.

6. The output change characteristic determination unit, when it determines that the reached leaf has changed, determines whether the change in the output value of the inference model in response to the change in the feature quantity for which the reached leaf has been determined to have changed violates the output change characteristic. The verification apparatus according to feature 1.

7. If the change determination unit determines that the destination leaf has not changed, it sets the change in the output value of the destination leaf that has not changed to zero. The verification apparatus according to feature 6.

8. The system includes an expected output specification unit that specifies the output change characteristics, the feature quantities, and the range of the feature quantities, The change determination unit determines, based on the specified feature quantity and the specified range of the feature quantity, whether or not the reached leaf changes due to a change in the feature quantity input to the inference model. The output change characteristic determination unit determines whether the change in the output value of the inference model violates the specified output change characteristic. The verification apparatus according to any one of claims 1 to 7, characterized in that

9. The verification result storage processing unit includes a verification result storage processing unit that stores in a storage device information indicating at least one of the following: the reachable leaf determined to be in violation of the output change characteristics; the range of the feature quantity that reaches the reachable leaf determined to be in violation of the output change characteristics; and the amount of change in the output value of the inference model. The verification apparatus according to any one of claims 1 to 7, characterized in that

10. The system includes a verification result display processing unit that performs processing to display on a display device information indicating at least one of the following: the reachable leaf determined to be in violation of the output change characteristics; the range of the feature quantities that reach the reachable leaf determined to be in violation of the output change characteristics; and the amount of change in the output value of the inference model. The verification apparatus according to any one of claims 1 to 7, characterized in that

11. The verification result display processing unit controls the display device to display information indicating at least one of the following: a range of feature quantities different from the feature quantities needed to reach the leaf that the output change characteristic determination unit has determined to be in violation of the output change characteristic, or the degree of the violation. The verification apparatus according to claim 10, characterized in that it is a verification apparatus.

12. The aforementioned output change characteristics are characteristics related to monotonicity. The verification apparatus according to any one of claims 1 to 7, characterized in that

13. The inference model loading unit performs the step of loading an inference model relating to a pre-trained decision tree model, The change determination unit determines whether the reach of the decision tree model changes due to a change in the feature quantities input to the loaded inference model, The output change characteristic determination unit determines, based on the determination result by the change determination unit, whether or not the change in the output value of the inference model violates the output change characteristics. A verification method having the following characteristics.

14. On the computer, The process involves loading an inference model related to a pre-trained decision tree model, A process to determine whether the reachable leaf of the decision tree model changes based on the changes in the features input to the loaded inference model, A program for executing a process that determines, based on the judgment result, whether or not the change in the output value of the inference model violates the output change characteristics.

Images