Verification device, verification method, and program

Abstract

This verification device comprises: an inference model loading unit (101) that loads an inference model relating to a trained decision tree model; a change determining unit (103) that determines whether or not a reached leaf of the decision tree model changes in response to a change in a feature input into the loaded inference model; and an output change characteristic determining unit (104) that, on the basis of the determination result from the change determining unit (103), determines whether or not a change in an output value of the inference model violates an output change characteristic.

Description

Verification device, verification method, and program

[0001] The present disclosure relates to a verification device, a verification method, and a program.

[0002] Conventionally, a technique is known in which a model is trained while checking the possible input-output relationships in order to ensure monotonicity of the output values ​​of an inference model (see, for example, Patent Document 1).

[0003] Japanese Patent Application Publication No. 06-242804

[0004] However, conventional techniques such as those disclosed in Patent Document 1 cannot handle verification of output change characteristics for trained models.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a verification device that is capable of verifying the output change characteristics of a trained model.

[0006] The verification device according to the present disclosure is characterized by comprising an inference model reading unit that reads an inference model related to a trained decision tree model, a change determination unit that determines whether the reached leaf of the decision tree model changes due to a change in the feature input to the read inference model, and an output change characteristic determination unit that determines whether a change in the output value of the inference model violates the output change characteristic based on the determination result by the change determination unit.

[0007] According to the present disclosure, the above-described configuration makes it possible to verify the output change characteristics of a trained model.

[0008] FIG. 1 is a diagram illustrating an example of the configuration of a verification device according to embodiment 1. FIG. 2 is a flowchart illustrating an example of the operation of a verification device according to embodiment 1. FIG. 3 is a flowchart illustrating an example of the operation of a change determination unit according to embodiment 1. FIG. 4 is a diagram illustrating an example of an inference model read by an inference model reading unit according to embodiment 1. FIG. 5A and FIG. 5B are diagrams illustrating an example of the operation of a reached leaf determination unit according to embodiment 1. FIG. 6A and FIG. 6B are diagrams illustrating an example of the operation of an output change characteristic determination unit according to embodiment 1. FIG. 7A and FIG. 7B are diagrams illustrating an example of the operation of a verification result display processing unit according to embodiment 1. FIG. 8A and FIG. 8B are diagrams illustrating an example of the hardware configuration of a verification device according to embodiment 1.

[0009] Hereinafter, the embodiments will be described in detail with reference to the drawings. Embodiment 1. Fig. 1 is a diagram showing an example of the configuration of a verification system including a verification device 1 according to embodiment 1. As shown in Fig. 1, the verification system includes, for example, the verification device 1, a storage device 2, and a display device 3.

[0010] As shown in Figure 1, the verification device 1 includes an inference model reading unit 101, an expected output specification unit 102, a change determination unit 103, an output change characteristic determination unit 104, a verification result storage processing unit 105, and a verification result display processing unit 106.

[0011] The inference model reading unit 101 reads an inference model related to a trained decision tree model.

[0012] Examples of the inference model read by the inference model reading unit 101 include a decision tree model and a decision tree ensemble model. In addition to the above, the inference model read by the inference model reading unit 101 can also be, for example, a neural network or the like surrogated to a decision tree-based model.

[0013] If the inference model is a decision tree model, the output value of the reached leaf becomes the output value of the inference model. If the inference model is a decision tree ensemble model, the output value of the inference model becomes the integrated value obtained by summing or averaging the output values ​​of the reached leaves for each decision tree.

[0014] Information indicating the inference model read by this inference model reading unit 101 is output to the change determination unit 103.

[0015] The expected output specification unit 102 specifies the output change characteristic, the feature value, and the range of the feature value. The output change characteristic specified by the expected output specification unit 102 is the change characteristic expected for the output value of the inference model read by the inference model reading unit 101.

[0016] The expected output specifying unit 102 may specify the output change characteristic, the feature amount, and the range of the feature amount in response to a user operation. Alternatively, the expected output specifying unit 102 may specify the output change characteristic, the feature amount, and the range of the feature amount automatically. For example, the expected output specifying unit 102 may specify the output change characteristic, the feature amount, and the range of the feature amount automatically in accordance with a predetermined rule.

[0017] The output change characteristics specified by the expected output specifying unit 102 include, for example, characteristics related to monotonicity, characteristics related to the shape of a graph showing the output value, characteristics related to limiting the slope of the output value, etc. Examples of the characteristics related to monotonicity include monotonic increase and monotonic decrease.

[0018] Information indicating the output change characteristic specified by the expected output specification unit 102 is output to the output change characteristic determination unit 104. Information indicating the feature amount and the range of the feature amount specified by the expected output specification unit 102 is output to the change determination unit 103.

[0019] The change determination unit 103 determines whether or not a change in the feature input to the inference model will change the reached leaf of the decision tree model, based on the inference model read by the inference model reading unit 101 and the feature and feature range specified by the expected output specification unit 102. Information indicating the determination result by this change determination unit 103 is output to the output change characteristic determination unit 104.

[0020] The change determination unit 103 includes, for example, a branch threshold extraction unit 1031 and a reached leaf determination unit 1032 as shown in FIG.

[0021] The branch threshold extraction unit 1031 extracts branch thresholds for feature quantities to be input to the inference model based on the inference model read by the inference model reading unit 101 and the feature quantities and ranges of feature quantities specified by the expected output specification unit 102. In this case, the branch threshold extraction unit 1031 extracts all branch thresholds for specified feature quantities within the range of the specified feature quantities from the read inference model. Note that these branch thresholds also include the minimum and maximum values ​​of the specified feature quantities. Information indicating the branch thresholds extracted by the branch threshold extraction unit 1031 is output to the reached leaf determination unit 1032.

[0022] The reached leaf determination unit 1032 determines whether the reached leaf changes due to a change in the branch threshold based on the inference model read by the inference model reading unit 101 and the branch threshold extracted by the branch threshold extraction unit 1031.

[0023] The reached leaf determination unit 1032 detects the amount of change between the output value of the reached leaf before the change in the branching threshold and the output value of the reached leaf after the change in the branching threshold. If the reached leaf determination unit 1032 determines that the reached leaf remains unchanged, it sets the amount of change due to the output value of the unchanged reached leaf to zero.

[0024] In addition, if the decision tree model is a decision tree ensemble model including multiple decision trees, the reached leaf determination unit 1032 may exclude a decision tree whose reached leaf does not change for all branching thresholds from the determination targets of the output change characteristic determination unit 104.

[0025] In addition, if there are multiple ranges of features in which the reached leaf does not change with a change in the branch threshold, the reached leaf determination unit 1032 may associate information indicating each range of features with information indicating that the change at that time is zero, and output the information to the output change characteristic determination unit 104.

[0026] The reached leaf determination unit 1032 may also detect a range of feature amounts reaching a reached leaf that changes with a change in the branching threshold, or may detect a range of feature amounts reaching a reached leaf that does not change with a change in the branching threshold.

[0027] Information indicating the determination result by the reaching leaf determining section 1032 is output to the output change characteristic determining section 104 .

[0028] The output change characteristic determination unit 104 determines whether the change in the output value of the inference model read by the inference model reading unit 101 violates the output change characteristic based on the output change characteristic specified by the expected output designation unit 102 and the determination result by the change determination unit 103.

[0029] In particular, when the change determination unit 103 determines that the reached leaf will change, the output change characteristic determination unit 104 determines whether a change in the output value of the inference model in response to a change in the feature determined to result in a change in the reached leaf violates the output change characteristic. At this time, the output change characteristic determination unit 104 detects, among the reached leaves that change with a change in the branching threshold, a reached leaf whose change in the output value of the inference model violates the output change characteristic. Note that, when the decision tree model is a decision tree ensemble model, the output change characteristic determination unit 104 detects a feasible combination of reached leaves among the combinations of reached leaves that change with a change in the branching threshold, and then detects a combination of reached leaves whose change in the output value of the inference model violates the output change characteristic.

[0030] Information indicating the determination result by the output change characteristic determining unit 104 is output to the verification result storage processing unit 105 and the verification result display processing unit 106 .

[0031] The information indicating the determination result by the output change characteristic determination unit 104 includes, for example, information indicating at least one of a destination leaf determined to violate the output change characteristic, a feature amount that reaches the destination leaf determined to violate the output change characteristic and that is specified by the expected output designation unit 102, and a change in the output value of the inference model. In addition to the above, the information indicating the determination result by the output change characteristic determination unit 104 may also include information indicating a range of feature amounts that reach the destination leaf determined to violate the output change characteristic and that is other than the feature amount specified by the expected output designation unit 102.

[0032] Furthermore, the information indicating the determination result by the output change characteristic determination unit 104 may also include, in addition to the above, information indicating at least one of a reached leaf determined not to violate the output change characteristic, a feature amount reaching a reached leaf that does not violate the output change characteristic and that is specified by the expected output designation unit 102, and a change in the output value of the inference model. Furthermore, the information indicating the determination result by the output change characteristic determination unit 104 may also include, in addition to the above, information indicating a range of feature amounts reaching a reached leaf determined not to violate the output change characteristic and that is other than the feature amount specified by the expected output designation unit 102.

[0033] The verification result storage processing unit 105 stores information indicating the determination result by the output change characteristic determining unit 104 in the storage device 2 .

[0034] At this time, the verification result storage processing unit 105 stores, in the storage device 2, information indicating at least one of the reached leaf determined by the output change characteristic determination unit 104 to violate the output change characteristic, the range of the feature value specified by the expected output designation unit 102, which is a feature value reaching the reached leaf determined to violate the output change characteristic, and the change in the output value of the inference model. Note that the range of the feature value here may be only the branching threshold, or may be a range including values ​​between the branching thresholds. In addition to the above, the verification result storage processing unit 105 may also store, in the storage device 2, information indicating the range of the feature value other than the feature value specified by the expected output designation unit 102, which is a feature value for reaching the reached leaf determined to violate the output change characteristic.

[0035] The verification result storage processing unit 105 may also store, in the storage device 2, information indicating at least one of a reached leaf determined by the output change characteristic determination unit 104 not to violate the output change characteristic, and a feature amount reaching the reached leaf determined not to violate the output change characteristic, the range of the feature amount specified by the expected output designation unit 102, and the change in the output value of the inference model. In addition to the above, the verification result storage processing unit 105 may also store, in the storage device 2, information indicating a range of feature amounts reaching the reached leaf determined not to violate the output change characteristic, other than the feature amount specified by the expected output designation unit 102.

[0036] The verification result display processing unit 106 causes the display device 3 to display information indicating the determination result by the output change characteristic determining unit 104 .

[0037] At this time, the verification result display processing unit 106 displays on the display device 3 information indicating at least one of the reached leaf determined by the output change characteristic determination unit 104 to violate the output change characteristic, the range of feature quantities that reach the reached leaf determined to violate the output change characteristic and that are other than the feature quantities specified by the expected output specification unit 102, and the amount of change in the output value of the inference model. Note that the range of feature quantities here may be only the branch threshold value, or may be a range including values ​​between the branch threshold values.

[0038] The verification result display processing unit 106 may also cause the display device 3 to display information indicating at least one of the range of feature quantities reaching the reached leaf that have been determined by the output change characteristic determination unit 104 to violate the output change characteristic, other than the feature quantity specified by the expected output specification unit 102, or the degree of violation. Note that the information indicating the degree of violation may include, for example, the hypervolume of the violation region.

[0039] 1 shows a case where the verification device 1 is provided with an expected output designation unit 102. However, the expected output designation unit 102 is not an essential component of the verification device 1, and does not have to be provided in the verification device 1. For example, if there is only one option for the output change characteristic and feature, or if the range of the feature can be handled by, for example, setting it to the entire range, the verification device 1 does not need the expected output designation unit 102.

[0040] 1, the storage device 2 is provided outside the verification device 1, but the storage device 2 may be a storage device provided in a computer that functions as the verification device 1 as long as it is accessible from the verification device 1.

[0041] The storage device 2 is, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0042] 1, the display device 3 is provided outside the verification device 1, but the display device 3 may be a display device provided in a computer that functions as the verification device 1 as long as it is accessible from the verification device 1.

[0043] The display device 3 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display device.

[0044] The verification device 1 is realized, for example, by a computer including a communication unit, a calculation unit, and a storage unit. The communication unit communicates with the storage device 2 or the display device 3 via a network using a wired signal line or wireless communication. For example, the communication unit is a communication device capable of mobile communication using a communication method such as LTE, 3G, 4G, or 5G. The communication unit may also be a short-range wireless communication means such as Bluetooth (registered trademark). The communication unit includes the input interface 54 and the output interface 55 shown in FIGS. 8A and 8B .

[0045] The calculation unit controls the overall operation of the verification device 1. The calculation unit includes an inference model reading unit 101, an expected output designation unit 102, a change determination unit 103, an output change characteristic determination unit 104, a verification result storage processing unit 105, and a verification result display processing unit 106. The calculation unit executes an information processing application for verifying the inference model, thereby realizing various functions of the inference model reading unit 101, the expected output designation unit 102, the change determination unit 103, the output change characteristic determination unit 104, the verification result storage processing unit 105, and the verification result display processing unit 106. The calculation unit includes the processing circuit 51 of FIG. 8A or the processor 52 of FIG. 8B.

[0046] The storage unit stores information processing applications and information used for the arithmetic processing of the arithmetic unit. The storage unit is a storage device provided in the computer functioning as the verification device 1, and includes storage such as a hard disk drive (HDD) or a solid state drive (SSD), or the memory 53 shown in FIGS. 8A and 8B. Note that the storage unit may be external to the verification device 1 as long as it is accessible by the verification device 1.

[0047] Next, an example of the operation of the verification device 1 according to the first embodiment shown in FIG. 1 will be described with reference to FIG.

[0048] In an example of operation of the verification device 1 according to the first embodiment shown in FIG. 1, as shown in FIG. 2, the inference model reading unit 101 first reads an inference model related to a trained decision tree model (step ST101).

[0049] Furthermore, the expected output designation unit 102 designates the output change characteristic, the feature amount, and the range of the feature amount (step ST102).

[0050] Next, the branch threshold extraction unit 1031 extracts branch thresholds for the feature quantities to be input to the inference model based on the inference model read by the inference model reading unit 101 and the feature quantities and ranges of the feature quantities specified by the expected output specification unit 102 (step ST103). At this time, the branch threshold extraction unit 1031 extracts all branch thresholds for the specified feature quantities, including the minimum and maximum values, from the read inference model within the range of the specified feature quantities.

[0051] Next, the reached leaf determination unit 1032 determines whether the reached leaf changes due to a change in the branch threshold based on the inference model read by the inference model reading unit 101 and the branch threshold extracted by the branch threshold extraction unit 1031 (step ST104).

[0052] The reached leaf determination unit 1032 detects the amount of change between the output value of the reached leaf before the change in the branching threshold and the output value of the reached leaf after the change in the branching threshold. If the reached leaf determination unit 1032 determines that the reached leaf remains unchanged, it sets the amount of change due to the output value of the unchanged reached leaf to zero.

[0053] In addition, if the decision tree model is a decision tree ensemble model including multiple decision trees, the reached leaf determination unit 1032 may exclude a decision tree whose reached leaf does not change for all branching thresholds from the determination targets of the output change characteristic determination unit 104.

[0054] In addition, when there are multiple ranges of features in which the reached leaf does not change with a change in the branch threshold, the reached leaf determination unit 1032 may associate information indicating each range of features with information indicating that the change at that time is zero, and output the information to the output change characteristic determination unit 104.

[0055] The reached leaf determination unit 1032 may also detect a range of feature amounts reaching a reached leaf that changes with a change in the branching threshold, or may detect a range of feature amounts reaching a reached leaf that does not change with a change in the branching threshold.

[0056] A detailed example of the operation of this reached leaf determining unit 1032 will be described later.

[0057] Next, the output change characteristic determination unit 104 determines whether the change in the output value of the inference model read by the inference model reading unit 101 violates the output change characteristic based on the output change characteristic specified by the expected output designation unit 102 and the determination result by the change determination unit 103 (reached leaf determination unit 1032) (step ST105).

[0058] In particular, when the change determination unit 103 determines that the reached leaf will change, the output change characteristic determination unit 104 determines whether a change in the output value of the inference model in response to a change in the feature determined to result in a change in the reached leaf violates the output change characteristic. At this time, the output change characteristic determination unit 104 detects, among the reached leaves that change with a change in the branching threshold, a reached leaf whose change in the output value of the inference model violates the output change characteristic. Note that, when the decision tree model is a decision tree ensemble model, the output change characteristic determination unit 104 detects a feasible combination of reached leaves among the combinations of reached leaves that change with a change in the branching threshold, and then detects a combination of reached leaves whose change in the output value of the inference model violates the output change characteristic.

[0059] Next, the verification result storage processing unit 105 stores information indicating the determination result by the output change characteristic determining unit 104 in the storage device 2 (step ST106).

[0060] At this time, the verification result storage processing unit 105 stores, in the storage device 2, information indicating at least one of the reached leaf determined by the output change characteristic determination unit 104 to violate the output change characteristic, and the feature amount reaching the reached leaf determined to violate the output change characteristic, which is the range of the feature amount specified by the expected output designation unit 102, and the change amount of the output value of the inference model. In addition to the above, the verification result storage processing unit 105 may also store, in the storage device 2, information indicating the range of the feature amount for reaching the reached leaf determined to violate the output change characteristic, other than the feature amount specified by the expected output designation unit 102.

[0061] The verification result storage processing unit 105 may also store, in the storage device 2, information indicating at least one of a reached leaf determined by the output change characteristic determination unit 104 not to violate the output change characteristic, and a feature amount reaching the reached leaf determined not to violate the output change characteristic, the range of the feature amount specified by the expected output designation unit 102, and the change in the output value of the inference model. In addition to the above, the verification result storage processing unit 105 may also store, in the storage device 2, information indicating a range of feature amounts reaching the reached leaf determined not to violate the output change characteristic, other than the feature amount specified by the expected output designation unit 102.

[0062] Furthermore, the verification result display processing unit 106 causes the display device 3 to display information indicating the determination result by the output change characteristic determining unit 104 (step ST107).

[0063] At this time, the verification result display processing unit 106 displays on the display device 3 information indicating at least one of the reached leaf that has been determined by the output change characteristic determination unit 104 to violate the output change characteristic, the range of features that reach the reached leaf that has been determined to violate the output change characteristic and that are other than the features specified by the expected output designation unit 102, and the amount of change in the output value of the inference model.

[0064] In addition, the verification result display processing unit 106 may display on the display device 3 information indicating at least one of the range of features that reach the destination leaf that has been determined by the output change characteristic determination unit 104 to violate the output change characteristic, other than the feature specified by the expected output designation unit 102, or the degree of violation.

[0065] Next, an example of the operation of the reached leaf determination unit 1032 in the first embodiment shown in Fig. 1 will be described with reference to Fig. 3. Note that Fig. 3 shows an example of the operation when the inference model read into the inference model reading unit 101 is a decision tree ensemble model.

[0066] In the example of operation of the reached leaf determining unit 1032 in the first embodiment shown in Fig. 1, the reached leaf determining unit 1032 first performs initialization (step ST201) as shown in Fig. 3. That is, the reached leaf determining unit 1032 sets the identification number (i) of the decision tree to 1 and the identification number (j) of the branch threshold of the feature specified by the expected output specifying unit 102 to 1.

[0067] Next, the reached leaf determining unit 1032 extracts all ranges reaching the reached leaf for each feature other than the feature specified by the expected output specifying unit 102 for the i-th decision tree (step ST202).

[0068] Next, the reached leaf determining unit 1032 determines whether the branch threshold of the feature quantity specified by the expected output specifying unit 102 is a ｊ Reached leaf and a ｊ＋１ In this case, the reached leaf is extracted (step ST203). Note that the reached leaf may change depending on a feature other than the feature specified by the expected output specifying unit 102.

[0069] Next, the reached leaf determining unit 1032 determines whether the branch threshold of the feature quantity specified by the expected output specifying unit 102 is a ｊ In the case of a ｊ＋１ It is determined whether the reached leaf changes between the cases (step ST204).

[0070] In step ST204, if the reached leaf determining section 1032 determines that the reached leaf does not change for the i-th decision tree, that is, the reached leaf does not change at all, the sequence proceeds to step ST208.

[0071] On the other hand, in step ST204, if the reached leaf determination unit 1032 determines that the reached leaf may change, it sets the reached leaves before and after the change as changed leaves, and detects the amount of change in the output value of the changed leaf (step ST205).

[0072] Next, the reached leaf determination unit 1032 determines whether the branch threshold in the feature quantity specified by the expected output specification unit 102 is a ｊ In the case of a ｊ＋１ It is determined whether there is a common decision path between the cases (1) and (2), that is, whether there is a case where the arrival leaf does not change (step ST206).

[0073] In step ST206, if the reached leaf determination unit 1032 determines that the reached leaf may remain unchanged, it sets the reached leaf as an unchanged leaf and sets the amount of change in value of the unchanged leaf to zero (step ST207).

[0074] Next, the reached leaf determination unit 1032 determines whether or not all branch thresholds of the feature quantities specified by the expected output specification unit 102 have been processed (step ST208). If the reached leaf determination unit 1032 determines in step ST208 that all branch thresholds have not been processed, it adds 1 to j (step ST209). Thereafter, the sequence returns to step ST203, and the above processing is repeated.

[0075] On the other hand, if it is determined in step ST208 that the process has been performed for all branching thresholds, the reached leaf determining unit 1032 determines whether the process has been performed for all decision trees (step ST210).

[0076] In step ST210, if the reached leaf determining unit 1032 determines that the process has not been performed for all decision trees, it adds 1 to i (step ST211). After that, the sequence returns to step ST202, and the above process is repeated.

[0077] On the other hand, if the reached leaf determining unit 1032 determines in step ST210 that all decision trees have been processed, the sequence ends.

[0078] Next, a specific example of the operation of the verification device 1 according to the first embodiment shown in FIG. 1 will be described with reference to FIGS.

[0079] Here, for example, as shown in Figure 4, the inference model read by the inference model reading unit 101 is assumed to be a decision tree ensemble model using three decision trees (Tree 1, Tree 2, Tree 3). Also, here, if the change in the output value of the reached leaf before and after the branching threshold is changed is a positive value, it is assumed that the output value of the reached leaf in the range between the branching thresholds is monotonically increasing, and if the change in the output value of the reached leaf before and after the branching threshold is changed is a negative value, it is assumed that the output value of the reached leaf in the range between the branching thresholds is monotonically decreasing. Also, in Figures 4 to 7, "Leaf" indicates the reached leaf, and "Value" indicates the output value of the reached leaf.

[0080] Furthermore, it is assumed that the expected output specification unit 102 specifies, as the output change characteristic, a characteristic in which the output value of the decision tree ensemble model increases monotonically with an increase in the feature amount. Furthermore, it is assumed that the expected output specification unit 102 specifies the feature amount and the range of the feature amount, for example, as shown in the following formula (1). In formula (1), x １ , x ２ , ..., x Ｎ indicates the feature, and X indicates x １ , x ２ , ..., x Ｎ X=(x １ , x ２ , ..., x Ｎ ) -3.5≦x １ ≦5.2, ..., 15≦x Ｎ≦40 (1)

[0081] In the following description, the verification device 1 determines whether the feature (x １ This example shows the case where we verify that the output value of the decision tree ensemble model increases monotonically with an increase in the number of features. However, the same applies to verification of other features.

[0082] In this case, first, the branch threshold extraction unit 1031 extracts a branch threshold for the feature based on the feature and the range of the feature specified by the expected output specification unit 102. For example, the branch threshold extraction unit 1031 extracts a branch threshold for the feature based on x in Equation (1). １ For the above, a branch threshold is extracted as shown in the following formula (2). １ , a ２ , ..., a Ｎ Ha x １ A denotes the branching threshold at a １ , a ２ , ..., a Ｎ A=(a １ , a ２ , ..., a Ｎ )=(-inf, 3, 5,..., inf) (2)

[0083] Next, the reached leaf determination unit 1032 determines whether the reached leaf changes with the change in the branch threshold based on the inference model read by the inference model reading unit 101 and the branch threshold extracted by the branch threshold extraction unit 1031. Here, for each decision tree shown in FIG. １ When the value of the sigma is changed from 3 to 5, the decision path is as shown in FIG.

[0084] That is, as shown in FIG. 5, in Tree 1, −inf<x ２ When x≦8, １ In the case of x = 3, the reached leaf is reached leaf 1-1, but １ In the case of x = 5, the reached leaf is reached leaf 1-2. １ When x changes from 3 to 5, the change in the output value of the reached leaf is −3. ２ If ≦15, then x１ Even if = 3, x １ Even if x = 5, the reached leaf is reached leaf 1-3. １ Even if x changes from 3 to 5, the output value of the reached leaf does not change, and the amount of change is 0. In this case, the decision path is a common decision path. In addition, in Tree 1, 15<x ２ If ≦inf, then x １ Even if = 3, x １ Even if x = 5, the reached leaf is reached leaf 1-4. １ Even if the value of the output of the reached leaf changes from 3 to 5, the output value of the reached leaf does not change, and the amount of change is 0. Also, the decision path in this case is a common decision path.

[0085] Furthermore, as shown in FIG. 5, in Tree 2, −inf<x ２ When x≦5, １ In the case of x = 3, the reached leaf is reached leaf 2-1, but １ In the case of x = 5, the reached leaf is the reached leaf 2-3. １ When x changes from 3 to 5, the change in the output value of the reached leaf is +3. ２ When x≦10, １ In the case of x = 3, the reached leaf is reached leaf 2-2, but １ In the case of x = 5, the reached leaf is the reached leaf 2-3. １ When x changes from 3 to 5, the change in the output value of the reached leaf is +2. ２ If x≦inf, １ In the case of x = 3, the reached leaf is reached leaf 2-2, but １ In the case of x = 5, the reached leaf is the reached leaf 2-4. １ When the value of the output of the reached leaf changes from 3 to 5, the change amount of the output of the reached leaf is +4.

[0086] Also, as shown in FIG. 5, in Tree 3, x １ Even if = 3, x １ Even if x = 5, the reached leaf is reached leaf 3-1. １Even if the value of the output of the reached leaf changes from 3 to 5, the output value of the reached leaf does not change, and the amount of change is 0. Also, the decision path in this case is a common decision path.

[0087] In FIG. 5B, the thick solid line path is x １ The bold dashed path indicates the decision path that is taken only when x = 3. １ The dashed line indicates the decision path that is taken only when x = 5. １ = 3 and x １ .times. ...

[0088] That is, in the example of FIG. 5, the reached leaf determining unit 1032 determines whether x １ The reached leaf determining unit 1032 detects reached leaf 1-1 and reached leaf 1-2, reached leaf 2-1 and reached leaf 2-3, reached leaf 2-2 and reached leaf 2-3, and reached leaf 2-2 and reached leaf 2-4 as reached leaves that change when the number of reached leaves changes from 3 to 5. The reached leaf determining unit 1032 also detects the amount of change in the output value due to the change from reached leaf 1-1 to reached leaf 1-2 (-3), the amount of change in the output value due to the change from reached leaf 2-1 to reached leaf 2-3 (+3), the amount of change in the output value due to the change from reached leaf 2-2 to reached leaf 2-3 (+2), and the amount of change in the output value due to the change from reached leaf 2-2 to reached leaf 2-4 (+4).

[0089] In the example of Fig. 5, Tree 3 is a decision tree in which the reached leaf does not change for all branching thresholds. In this case, the reached leaf determination unit 1032 may exclude Tree 3 from the targets of determination by the output change characteristic determination unit 104. Fig. 5A shows a table in which Tree 3 is excluded. This makes it possible to narrow down the targets of determination by the output change characteristic determination unit 104, thereby reducing the processing load.

[0090] As shown in FIG. 5A, for Tree 1, reaching leaf 1-3 and reaching leaf 1-4 are reaching leaves that do not change with a change in the branching threshold. In this way, the amount of change in the output value of the reaching leaves that do not change with a change in the branching threshold is zero. In this case, the reaching leaf determination unit 1032 determines the range of the feature amount that reaches reaching leaf 1-3 (-inf<x１ ≦inf, 8<x ２ ≦15) and the range of features that reach leaves 1-4 (-inf<x １ ≦inf, 15<x ２ ≦inf) and information indicating that the amount of change at that time is zero may be associated with each other and output to the output change characteristic determining unit 104. This makes it possible to combine information into one when the amount of change is zero, thereby reducing the processing load.

[0091] Next, the output change characteristic determination unit 104 determines whether or not a change in the output value of the inference model read by the inference model reading unit 101 violates the output change characteristic, based on the output change characteristic specified by the expected output specification unit 102 and the determination result by the change determination unit 103 (reached leaf determination unit 1032). Here, it is assumed that the inference model read by the inference model reading unit 101 is a decision tree ensemble model, and the output value of the inference model is the sum of the output values ​​of each decision tree. In this case, the output change characteristic determination unit 104 detects the sum of the changes in the output values ​​of Tree 1 to Tree 3 as the amount of change in the output value of the decision tree ensemble model.

[0092] In this case, as shown in Figure 6, when changing from reaching leaf 1-1 to reaching leaf 1-2 and from reaching leaf 2-2 to reaching leaf 2-3, the change in the output value of the decision tree ensemble model becomes a negative value (-1), and for other combinations, the change in the output value becomes a positive value.

[0093] Here, the output change characteristic determining unit 104 determines the feature amount (x １ 6, when the reaching leaf 1-1 is changed to the reaching leaf 1-2 and when the reaching leaf 2-2 is changed to the reaching leaf 2-3, the output change characteristic determining unit 104 determines that the change in the output value of the decision tree ensemble model is a negative value (-1), violating the monotonic increase.

[0094] Next, the verification result display processing unit 106 displays information indicating the judgment result by the output change characteristic judgment unit 104 on the display device 3. At this time, the verification result display processing unit 106 displays information indicating at least one of the reached leaf that has been judged by the output change characteristic judgment unit 104 to violate the output change characteristic, the feature that reaches the reached leaf that has been judged to violate the output change characteristic and that is specified by the expected output specification unit 102, and the amount of change in the output value of the inference model.

[0095] In addition, the verification result display processing unit 106 may display on the display device 3 information indicating at least one of the range of features that reach the destination leaf that has been determined by the output change characteristic determination unit 104 to violate the output change characteristic, other than the feature specified by the expected output designation unit 102, or the degree of violation.

[0096] 7A and 7B show examples of screens displayed on the display device 3 by the verification result display processing unit 106. In Fig. 7A, the tree structure of the inference model read by the inference model reading unit 101 is displayed, and the parts of the reaching leaves that have been determined by the output change characteristic determination unit 104 to violate the output change characteristic are highlighted and displayed differently from the parts of the other reaching leaves. In the example of Fig. 7A, reaching leaves 1-1 and 1-2 of Tree 1 and reaching leaves 2-2 and 2-3 of Tree 2 are reaching leaves that have been determined to violate the output change characteristic, and the parts of these reaching leaves are highlighted by hatching.

[0097] Further, for example, a decision path reaching a destination leaf that has been determined to violate the output change characteristics by the output change characteristics determination unit 104 may also be highlighted. Further, for example, the range of each feature amount reaching a destination leaf that has been determined to violate the output change characteristics by the output change characteristics determination unit 104 may be displayed in text or the like.

[0098] Furthermore, in Figure 7B, the output value of the inference model is shown by a solid line when feature quantities other than those specified by the expected output specification unit 102 are fixed to some value and the specified feature quantity is changed. The negatively sloping portions of this solid line are portions that violate the output change characteristic (monotonically increasing in the example shown). The greater the negative slope, the greater the degree of violation. Also, in Figure 7B, the hypercuboid volume of the area involved in the violation of the output change characteristic is shown by a rectangular area. The larger this rectangular area, the greater the degree of violation. Also, in Figure 7B, the rate of change of the output value of the inference model is shown by the number of black circles. The greater the number of black circles, the greater the degree of violation.

[0099] As described above, according to the first embodiment, the verification device 1 includes an inference model reading unit 101 that reads an inference model related to a trained decision tree model, a change determination unit 103 that determines whether a reached leaf of the decision tree model changes due to a change in a feature value input to the read inference model, and an output change characteristic determination unit 104 that determines whether a change in an output value of the inference model violates an output change characteristic based on the determination result by the change determination unit 103. Also, according to the first embodiment, the change determination unit 103 includes a branch threshold extraction unit 1031 that extracts a branch threshold for the feature value input to the inference model, and a reached leaf determination unit 1032 that determines whether a reached leaf changes due to a change in the branch threshold extracted by the branch threshold extraction unit 1031. Also, according to the first embodiment, the reached leaf determination unit 1032 detects the amount of change between the output value of the reached leaf before the change in the branch threshold and the output value of the reached leaf after the change in the branch threshold. Furthermore, according to this first embodiment, when the change determination unit 103 determines that the reached leaf will change, the output change characteristic determination unit 104 determines whether or not a change in the output value of the inference model in response to the change in the feature determined to change the reached leaf violates the output change characteristic. Also, according to this first embodiment, when the change determination unit 103 determines that the reached leaf will not change, it sets the output value of the unchanged reached leaf to zero. As a result, the verification device 1 according to the first embodiment can verify the output change characteristic of a trained model.

[0100] Furthermore, according to the first embodiment, the inference model is a decision tree ensemble model including multiple decision trees, and if there is a decision tree whose reached leaf does not change for all branching thresholds, the reached leaf determination unit 1032 excludes the decision tree from the determination targets of the output change characteristic determination unit 104. This makes it possible to reduce the processing load in the verification device 1 according to the first embodiment.

[0101] Furthermore, according to this first embodiment, the verification device 1 is provided with an expected output specification unit 102 that specifies the output change characteristic, the feature, and the range of the feature, and the change determination unit 103 determines whether the reached leaf changes due to a change in the feature input to the inference model based on the specified feature and the specified range of the feature, and the output change characteristic determination unit 104 determines whether a change in the output value of the inference model violates the specified output change characteristic. As a result, in the verification device 1 according to the first embodiment, when there are options for the output change characteristic, the feature, and the range of the feature, it is possible to specify each of the force change characteristic, the feature, and the range of the feature.

[0102] Furthermore, according to the first embodiment, the verification result storage processing unit 105 is provided, which performs processing to store information indicating at least one of the reached leaf determined to violate the output change characteristic, the range of feature values ​​reaching the reached leaf determined to violate the output change characteristic, and the amount of change in the output value of the inference model in the storage device 2. This makes it possible for the verification device 1 according to the first embodiment to store in the storage device 2 not only whether or not the output change characteristic is violated, but also information regarding the violation of the output change characteristic.

[0103] Moreover, according to the first embodiment, the verification device 1 includes a verification result display processing unit 106 that performs processing to display, on the display device 3, information indicating at least one of the reached leaf determined to violate the output change characteristic, the range of feature quantities reaching the reached leaf determined to violate the output change characteristic, and the amount of change in the output value of the inference model. Moreover, according to the first embodiment, the verification result display processing unit 106 controls the display device 3 to display information indicating at least one of the range of feature quantities different from the feature quantities required to reach the reached leaf determined by the output change characteristic determination unit 104 to violate the output change characteristic, or the degree of violation. As a result, the verification device 1 according to the first embodiment can display, on the display device 3, not only whether or not the output change characteristic is violated, but also information regarding the violation of the output change characteristic.

[0104] Furthermore, according to the first embodiment, the output change characteristics are characteristics related to monotonicity, which allows the verification device 1 according to the first embodiment to verify the reliability of the trained machine learning model, particularly the tendency of monotonic output value changes.

[0105] Moreover, according to this first embodiment, the verification method includes the steps of an inference model reading unit 101 reading an inference model related to a trained decision tree model, a change determination unit 103 determining whether or not a reached leaf of the decision tree model changes due to a change in a feature input to the read inference model, and an output change characteristic determination unit 104 determining whether or not a change in an output value of the inference model violates the output change characteristic based on the determination result by the change determination unit 103. As a result, the verification method according to the first embodiment makes it possible to verify the output change characteristic of a trained model.

[0106] Furthermore, according to the first embodiment, the program causes a computer to execute the following processes: loading an inference model related to a trained decision tree model; determining whether a reached leaf of the decision tree model changes due to a change in a feature input to the loaded inference model; and determining whether a change in an output value of the inference model violates an output change characteristic based on the determination result. As a result, the program according to the first embodiment makes it possible to verify the output change characteristic of a trained model.

[0107] Finally, an example of the hardware configuration of the verification device 1 according to the first embodiment will be described with reference to Fig. 8. The functions of the inference model reading unit 101, expected output specifying unit 102, change determining unit 103, output change characteristic determining unit 104, verification result storage processing unit 105, and verification result display processing unit 106 in the verification device 1 are realized by a processing circuit 51. The processing circuit 51 may be dedicated hardware as shown in Fig. 8A, or may be a processor (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in a memory 53 as shown in Fig. 8B.

[0108] 8A and 8B, the input interface 54 is an interface that relays data that the verification device 1 acquires from the storage device 2 or the display device 3. The output interface 55 is an interface that relays data that is output from the verification device 1 to the storage device 2 or the display device 3.

[0109] When the processing circuit 51 is dedicated hardware, the processing circuit 51 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each of the inference model reading unit 101, the expected output specifying unit 102, the change determining unit 103, the output change characteristic determining unit 104, the verification result storage processing unit 105, and the verification result display processing unit 106 may be realized individually by the processing circuit 51, or the functions of each unit may be realized collectively by the processing circuit 51.

[0110] When the processing circuit 51 is a processor 52, the functions of the inference model reading unit 101, expected output designation unit 102, change determination unit 103, output change characteristic determination unit 104, verification result storage processing unit 105, and verification result display processing unit 106 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuit 51 realizes the functions of each unit by reading and executing the programs stored in the memory 53. That is, the verification device 1 includes a memory for storing programs that, when executed by the processing circuit 51, result in the execution of, for example, each step shown in FIGS. 2 and 3 . Furthermore, these programs can also be said to cause a computer to execute the procedures and methods of the inference model reading unit 101, expected output designation unit 102, change determination unit 103, output change characteristic determination unit 104, verification result storage processing unit 105, and verification result display processing unit 106. Here, examples of the memory 53 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0111] It is also possible to realize some of the functions of the inference model reading unit 101, the expected output designation unit 102, the change determination unit 103, the output change characteristic determination unit 104, the verification result storage processing unit 105, and the verification result display processing unit 106 with dedicated hardware and some of the functions with software or firmware. For example, the inference model reading unit 101 can be realized by a processing circuit 51 as dedicated hardware, and the expected output designation unit 102, the change determination unit 103, the output change characteristic determination unit 104, the verification result storage processing unit 105, and the verification result display processing unit 106 can be realized by the processing circuit 51 reading and executing programs stored in the memory 53.

[0112] In this way, the processing circuitry 51 can realize each of the above-described functions by hardware, software, firmware, or a combination of these.

[0113] Any of the components of the embodiments may be modified or omitted.

[0114] The verification device according to the present disclosure is capable of verifying the output change characteristics of a trained model, and is suitable for use as a verification device, etc.

[0115] 1 Verification device, 2 Storage device, 3 Display device, 51 Processing circuit, 52 Processor, 53 Memory, 54 Input interface, 55 Output interface, 101 Inference model reading unit, 102 Expected output designation unit, 103 Change determination unit, 104 Output change characteristic determination unit, 105 Verification result storage processing unit, 106 Verification result display processing unit, 1031 Branch threshold extraction unit, 1032 Reached leaf determination unit.

Claims

1. An inference model loading unit loads an inference model related to a pre-trained decision tree model, A change determination unit determines whether the reach of the decision tree model changes based on changes in the features input to the loaded inference model, Based on the determination result by the change determination unit, an output change characteristic determination unit determines whether the change in the output value of the inference model violates the output change characteristics, A verification device equipped with the following features.

2. The aforementioned change determination unit, A branch threshold extraction unit that extracts branch thresholds for the feature quantities to be input to the inference model, The system includes a leaf arrival determination unit that determines whether or not the leaf arrival changes based on the change in the branch threshold extracted by the branch threshold extraction unit. The verification apparatus according to claim 1, characterized by the features described above.

3. The aforementioned decision tree model is a decision tree ensemble model that includes multiple decision trees. The aforementioned leaf arrival determination unit excludes a decision tree from the output change characteristic determination unit's determination target if there is a decision tree in which the leaf arrival remains unchanged at all branch thresholds. The verification apparatus according to feature 2.

4. The leaf arrival determination unit detects the amount of change between the output value of the leaf arrival before the change in the branch threshold and the output value of the leaf arrival after the change in the branch threshold. The verification apparatus according to feature 2.

5. The leaf arrival determination unit detects the amount of change between the output value of the leaf arrival before the change in the branch threshold and the output value of the leaf arrival after the change in the branch threshold. The verification apparatus according to claim 3.

6. The output change characteristic determination unit, when it determines that the reached leaf has changed, determines whether the change in the output value of the inference model in response to the change in the feature quantity for which the reached leaf has been determined to have changed violates the output change characteristic. The verification apparatus according to feature 1.

7. If the change determination unit determines that the destination leaf has not changed, it sets the change in the output value of the destination leaf that has not changed to zero. The verification apparatus according to feature 6.

8. The system includes an expected output specification unit that specifies the output change characteristics, the feature quantities, and the range of the feature quantities, The change determination unit determines, based on the specified feature quantity and the specified range of the feature quantity, whether or not the reached leaf changes due to a change in the feature quantity input to the inference model. The output change characteristic determination unit determines whether the change in the output value of the inference model violates the specified output change characteristic. The verification apparatus according to any one of claims 1 to 7, characterized in that

9. The verification result storage processing unit includes a verification result storage processing unit that stores in a storage device information indicating at least one of the following: the reachable leaf determined to be in violation of the output change characteristics; the range of the feature quantity that reaches the reachable leaf determined to be in violation of the output change characteristics; and the amount of change in the output value of the inference model. The verification apparatus according to any one of claims 1 to 7, characterized in that

10. The system includes a verification result display processing unit that performs processing to display on a display device information indicating at least one of the following: the reachable leaf determined to be in violation of the output change characteristics; the range of the feature quantities that reach the reachable leaf determined to be in violation of the output change characteristics; and the amount of change in the output value of the inference model. The verification apparatus according to any one of claims 1 to 7, characterized in that

11. The verification result display processing unit controls the display device to display information indicating at least one of the following: a range of feature quantities different from the feature quantities needed to reach the leaf that the output change characteristic determination unit has determined to be in violation of the output change characteristic, or the degree of the violation. The verification apparatus according to claim 10, characterized in that it is a verification apparatus.

12. The aforementioned output change characteristics are characteristics related to monotonicity. The verification apparatus according to any one of claims 1 to 7, characterized in that

13. The inference model loading unit performs the step of loading an inference model relating to a pre-trained decision tree model, The change determination unit determines whether the reach of the decision tree model changes due to a change in the feature quantities input to the loaded inference model, The output change characteristic determination unit determines, based on the determination result by the change determination unit, whether or not the change in the output value of the inference model violates the output change characteristics. A verification method having the following characteristics.

14. On the computer, The process involves loading an inference model related to a pre-trained decision tree model, A process to determine whether the reachable leaf of the decision tree model changes based on the changes in the features input to the loaded inference model, A program for executing a process that determines, based on the judgment result, whether or not the change in the output value of the inference model violates the output change characteristics.