COMPATIBILITY EVALUATION DEVICE, COMPATIBILITY EVALUATION METHOD, AND PROGRAM
The compatibility evaluation device and method address AI behavior inconsistencies by using a generalized backward compatibility index to ensure smooth transitions during updates, enhancing AI integration efficiency.
Patent Information
- Application Number
- JP2023503257
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-03
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-03-03
AI Technical Summary
Existing AI updating methods improve accuracy but may result in unpredictable changes in AI behavior, necessitating time and effort to understand and adapt business operations.
A compatibility evaluation device and method using a generalized backward compatibility index to assess the agreement and disagreement between pre-update and post-update AI predictors, incorporating metrics like Backward Trust Compatibility (BTC) and Backward Error Compatibility (BEC) to ensure seamless integration.
Enables effective evaluation of AI compatibility, reducing the need for extensive adaptation by selecting highly compatible post-update predictors, thereby minimizing operational disruptions and costs.
Smart Images

Figure 0007593473000014 
Figure 0007593473000015 
Figure 0007593473000016
Abstract
Description
[Technical field]
[0001] The present disclosure relates to techniques for evaluating predictors. [Background technology]
[0002] In the operation of AI (Artificial Intelligence), in order to adapt and improve the performance of the AI to changes in the environment, it is essential to update the AI by re-learning using new data. When updating the AI, it is required that the accuracy of the updated AI is improved compared to before the update. Patent Document 1 discloses a method for reducing deterioration of a model when updating a model generated by machine learning. Patent Document 2 discloses a method for evaluating the similarity of the structure of a predictive model before and after re-learning as the similarity of the properties of the predictive model when re-learning the predictive model. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2019-204190 A [Patent Document 2] International Publication No. WO2016 / 151618 Summary of the Invention [Problem to be solved by the invention]
[0004] Even if an AI update improves accuracy, the AI's behavior may be different before and after the update. For example, an updated AI may not be able to correctly answer data that the AI currently in operation was able to correctly answer. In this case, the AI operator may need to spend time and effort to understand the quirks of the updated AI, or changes may be required to business operations in response to the AI's predictions.
[0005] One objective of the present disclosure is to provide a technique for evaluating the compatibility of predictors. [Means for solving the problem]
[0006] In one aspect of the present disclosure, a compatibility assessment device includes: obtaining means for obtaining outputs of the first predictor and the second predictor for the evaluation data; An indicator determination means for determining a generalized backward compatibility indicator defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; The present invention also includes a calculation means for calculating a score indicating compatibility between the first predictor and the second predictor using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index.
[0007] In another aspect of the present disclosure, a compatibility evaluation method includes: Obtaining outputs of the first predictor and the second predictor for the evaluation data; determining a generalized backward compatibility index defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; A score indicating compatibility between the first predictor and the second predictor is calculated using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index.
[0008] In yet another aspect of the present disclosure, program teeth, Obtaining outputs of the first predictor and the second predictor for the evaluation data; determining a generalized backward compatibility index defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; The method causes a computer to execute a process of calculating a score indicating compatibility between the first predictor and the second predictor using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index. Effect of the Invention
[0009] According to the present disclosure, the compatibility of predictors can be evaluated. [Brief description of the drawings]
[0010] [Figure 1] An example of prediction results for evaluation data of the AI before and after the update is shown below. [Diagram 2] 1 is a block diagram showing an overall configuration of a compatibility evaluation device according to a first embodiment. [Diagram 3] 1 is a block diagram showing a hardware configuration of a compatibility evaluation device according to a first embodiment. [Figure 4] 1 is a block diagram showing a functional configuration of a compatibility evaluation device according to a first embodiment. [Diagram 5] 11 is a flowchart of a compatibility evaluation process according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing a functional configuration of a compatibility evaluation device according to a second embodiment. [Figure 7] 10 is a flowchart of a process performed by the compatibility evaluation device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. <Compatibility evaluation index> (Predictor compatibility) When updating (relearning) an AI using new data, the update is done to improve accuracy, but at that time, compatibility of the AI becomes an issue. Compatibility refers to the degree of agreement between the correct answers / incorrect answers of the AI before the update and the correct answers / incorrect answers of the AI after the update.
[0012] One of the indicators of compatibility is the Backward Trust Compatibility (BTC) score (hereafter referred to as "BTC"). BTC is the percentage of data that the AI before the update can answer correctly that the AI after the update can also answer correctly, and the higher the BTC, the higher the compatibility.
[0013] Figure 1 shows examples of prediction results for evaluation data of the pre-update AI and two post-update AIs. The pre-update AI is the AI currently in operation. The two post-update AIs were obtained by re-learning the pre-update AI, but are different AIs generated by changing hyperparameters, etc. In Figure 1, a check mark indicates that the prediction result is correct.
[0014] As shown in the figure, the pre-update AI correctly answered four of the evaluation data 1 to 7, with an accuracy of 4 / 7. In contrast, the first post-update AI and the second post-update AI both have an accuracy of 5 / 7, which is an improvement over the pre-update AI. On the other hand, the first post-update AI correctly answered three evaluation data indicated by stars (★) out of the four evaluation data that the pre-update AI answered correctly, with a BTC score of 3 / 4. In contrast, the second post-update AI only answered two of the four evaluation data that the pre-update AI answered correctly, with a BTC score of 2 / 4. Therefore, although the two post-update AIs have the same accuracy, the first post-update AI, which has higher compatibility (BTC score), is evaluated as being better.
[0015] Another indicator of compatibility is the Backward Error Compatibility (BEC) score (hereafter referred to as "BEC"). BEC is the percentage of errors made by the AI before the update on data that is incorrectly made by the AI after the update. A higher BEC score indicates higher compatibility.
[0016] Thus, when updating an AI by re-learning, it is necessary to consider not only accuracy but also compatibility with the pre-update AI. Below, we propose a generalized backward compatibility index that can be applied to various tasks.
[0017] (Generalized backward compatibility indicator) The generalized backward compatibility index is a generalized index of compatibility indexes such as BTC and BEC mentioned above. An example of the generalized backward compatibility index is explained below.
[0018] (Example 1) The first example is the most basic generalized backward compatibility index. Let h be a predictor and a pair of inputs and outputs (X, Y), then
number
[0019]
number
[0020] The four relations have the following meanings: · CC (Correct Compatibility) (h1, h2) indicates the proportion of evaluation data in which predictor h1 outputs the correct answer and predictor h2 outputs the correct answer, out of all evaluation data. ·EC (Error Compatibility) (h1, h2) indicates the proportion of evaluation data in which predictor h1 outputs an incorrect answer and predictor h2 outputs an incorrect answer, out of all evaluation data. IC1(Imcompatibility-1)(h1,h2) indicates the proportion of evaluation data in which predictor h1 outputs the correct answer and predictor h2 outputs the incorrect answer, out of all evaluation data. IC2(Imcompatibility-2)(h1,h2) indicates the percentage of evaluation data in which predictor h1 outputs an incorrect answer and predictor h2 outputs a correct answer out of all evaluation data.
[0021] Specifically, the above four relational expressions are given as follows:
number
[0022] In equation (1), coefficient a 11 , b 10 , b 11 If we set ∑ to 1 and the other coefficients to 0, the GBC score in equation (1) is consistent with the BTC score. Therefore, the above GBC encompasses BTC.
[0023] In addition, in equation (1), coefficient a 00 , b 00 , b 10 If we set ∑ i = 1 and the other coefficients ∑ i = 0, the GBC score in equation (1) is consistent with the BEC score. Therefore, the above GBC encompasses the BEC.
[0024] In this way, by using the above generalized backward compatibility metric (GBC), it is possible to define an appropriate compatibility metric depending on the task of the predictor by changing the coefficients (weights) in equation (1).
[0025] Next, we will show an example of the score calculation formula using the GBC in Example 1. Now, set the input as follows:
number
[0026] GBC score estimateGBC ∧ is given by the following formula. For convenience, the letter "X" is written with " ∧ " symbol is "X ∧ " should be written.
number
[0027] In addition, each relational expression CC ∧, E.C. ∧ , IC1 ∧ , IC2 ∧ is given by replacing the expected values in equations (2) to (5) with the sample mean as follows:
number
[0028] (Example 2) In the first example above, coefficients (weights) are set for the four relational expressions CC, EC, IC1, and IC2, as shown in equation (1). In contrast, in the second example, coefficients (weights) are set for each class y predicted by the predictors h1 and h2. The GBC score in the second example is given by the following equation.
[0029]
number
[0030]
number
[0031] In the second example, GBC allows various existing binary classification indices that can be expressed as linear fractional expressions to be constructed in the context of backward compatibility. For example, by adjusting the weights of GBC shown in equation (11), a compatibility index that is effective for imbalanced binary classification can be constructed. If compatibility is not taken into consideration, the F-score (Y = 1 is the positive class, Y = 0 is the negative class) for binary classification Y ∈ {0,1} is as follows:
[0032]
number
[0033] On the other hand, the F value that takes compatibility into consideration (called "BC-F") is a 11,1 =b 11,1 =2, b 11,0 =b 00,1 = 1 and the remaining coefficients are set to "0", we get:
number
[0034] (Example 3) The third example is an example of a compatibility index other than the linear fractional expressions like the first and second examples. Consider a binary classification task where you want the score ranking of the predictor before the update to match that of the predictor after the update. If the predictor assigns real numbers to "-1" and "+1", you can obtain the following compatibility index.
[0035]
number
[0036] This compatibility index is a relational expression that shows the magnitude relationship between the output of the predictor before the update when evaluation data X whose correct answer is "+1" and evaluation data X' whose correct answer is "-1" are input.
number
number
[0037] (Application to regression tasks) In the above first and second examples, the predictor executes a classification task, but GBC can also be applied to a predictor that executes a regression task. In that case, if the difference between the predicted value output by the predictor for the evaluation data and the actual value corresponding to the evaluation data is equal to or less than a predetermined threshold, the predicted value is considered to be correct, and if it is greater than the threshold, the predicted value is considered to be incorrect, and the GBC of the first or second example can be applied.
[0038] First Embodiment [Overall configuration] 2 is a block diagram showing the overall configuration of a compatibility evaluation device according to a first embodiment. The compatibility evaluation device 100 evaluates the compatibility of two predictors and outputs a compatibility score. As shown in the figure, the same evaluation data is input to the two predictors h1 and h2. In a typical example, the predictor h1 is a predictor currently in operation, i.e., a pre-update predictor, and the predictor h2 is a post-update predictor.
[0039] The predictor h1 and the predictor h2 output predicted values for the input evaluation data to the compatibility evaluation device 100. The compatibility evaluation device 100 outputs a compatibility score indicating the compatibility between the output of the predictor h1 and the output of the predictor h2, using the above-mentioned generalized backward compatibility measure (GBC).
[0040] [Hardware configuration] 3 is a block diagram showing a hardware configuration of the compatibility evaluation device 100. The compatibility evaluation device 100 includes an interface 101, a processor 102, a memory 103, a recording medium 104, an input unit 105, and a display unit .
[0041] An interface (IF) 101 receives predicted values from the predictors h1 and h2. The IF 101 also outputs to an external device the compatibility score calculated by the compatibility evaluation device 100. The IF is an example of an acquisition unit.
[0042] The processor 102 is a computer such as a CPU, and executes a prepared program to control the entire compatibility evaluation device 100. The processor 102 may be a GPU or a Field-Programmable Gate Array (FPGA). Specifically, the processor 102 executes a compatibility evaluation process, which will be described later.
[0043] The memory 103 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 103 stores information on the generalized backward compatibility index, coefficients (weights) for each index number, etc. The memory 103 is also used as a working memory while the processor 102 is executing various processes.
[0044] The recording medium 104 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the compatibility evaluation device 100. The recording medium 104 records various programs executed by the processor 102. When the compatibility evaluation device 100 executes processing, the programs recorded on the recording medium 104 are loaded into the memory 103 and executed by the processor 102.
[0045] The input unit 105 is, for example, a keyboard, a mouse, etc., and is used when a user gives and inputs various instructions. The display unit 106 is, for example, a liquid crystal display device, etc., and displays various information to the user.
[0046] [Function configuration] FIG. 4 is a block diagram showing a functional configuration of the compatibility evaluation device 100. In terms of functions, the compatibility evaluation device 100 includes an evaluation index determination unit 110 and a score calculation unit 120. An index number is input to the evaluation index determination unit 110. The index number is a number that specifies a compatibility index used in evaluating compatibility. The index number is determined, for example, based on a task of a predictor to be updated. The evaluation index determination unit 110 determines a compatibility index (hereinafter also referred to as an "evaluation index") to be actually used for evaluation based on a generalized backward compatibility index (GBC) shown in equation (1) or equation (11) based on the input index number, and outputs the compatibility index to the score calculation unit 120.
[0047] The index number is determined in advance in correspondence with the combination of coefficients (weights) included in the formula (1). For example, when the compatibility index number "1" corresponds to BTC, the combination of coefficients "coefficient a 11 =b 10 =b 11 Therefore, when the user inputs the compatibility index number "1", the evaluation index determination unit 110 sets "coefficient a = 1, other coefficients = 0" in advance. 11 =b 10 =b 11 = 1, other coefficients = 0" into equation (1) to generate an evaluation index showing the BTC score.
[0048] The score calculation unit 120 uses the determined evaluation index to calculate and output a compatibility score from the predicted values output by the predictors h1 and h2. For example, the score calculation unit 120 substitutes the predicted values output by the predictors into equations (7) to (10) to obtain the values of the four relational expressions CC(h1,h2), EC(h1,h2), IC1(h1,h2), and IC2(h1,h2), and substitutes them into the evaluation indexes such as equation (6) to calculate and output the GBC score.
[0049] The evaluation index determination unit 110 is an example of an index determination means, and the score calculation unit 120 is an example of a calculation means.
[0050] [Compatibility evaluation process] Fig. 5 is a flowchart of the compatibility evaluation process executed by the compatibility evaluation device 100. This process is realized by the processor 102 shown in Fig. 3 executing a program prepared in advance and operating as each element shown in Fig. 4.
[0051] First, the compatibility evaluation device 100 receives an index number input by a user (step S11). Next, the evaluation index determination unit 110 determines an evaluation index based on the input index number (step S12). For example, when the GBC of the first or second example described above is used as the evaluation index, the evaluation index determination unit 110 obtains each coefficient (weight) corresponding to the index number and substitutes it into formula (1) or formula (11) to determine the evaluation index.
[0052] Next, the score calculation unit 120 obtains the predicted values output by the predictors h1 and h2 for the evaluation data (step S13), inputs them into the evaluation index determined in step S12, calculates the compatibility score (GBC score), and outputs it (step S14). In this way, the compatibility score indicating the compatibility between the predictors h1 and h2 is obtained. Then, the process ends.
[0053] [Use Case] GBC can be used as an index to evaluate the compatibility of multiple updated predictors with different hyperparameters and seeds when updating a predictor. By selecting the one with high compatibility with the pre-update predictor from among the multiple updated predictors generated, it is possible to reduce the cost of procedural changes associated with changes in the behavior of the AI after the update.
[0054] In addition, when data changes due to seasonality occur, GBC can be used to check whether there are past forecast models that are highly compatible with the current forecast model. If there is a past forecast model that is highly compatible with the current forecast model and has high accuracy, the current forecast model can be switched to that forecast model, making it possible to switch to a forecast model suitable for that season without incurring the cost of re-learning.
[0055] In addition, if the business's KPIs (Key Performance Indicators) change during AI operation, GBC can be used to build compatibility indicators that focus on items that the new KPIs emphasize (for example, the class you want to get correct), which can be useful for continued AI operation.
[0056] [Constructing a predictor using GBC] In the above example, GBC is used to evaluate the compatibility of predictors when updating, but instead, GBC can be used in training predictors. In this case, when training a predictive model, GBC is added as a regularizer to the error function used in normal training. Specifically, as with existing generalized binary classification indices, an upper bound for GBC can be constructed by replacing the indicator function with a loss function (squared loss or hinge loss). Then, a predictive model is trained to minimize the combination of the constructed upper bound and the error function of normal binary classification. By using the predictor before the update and the additionally collected data as inputs and regularizing GBC, a new predictor with high backward compatibility suitable for the target task can be constructed.
[0057] <Second embodiment> Next, a second embodiment of the present disclosure will be described. Fig. 6 is a block diagram showing a functional configuration of a compatibility evaluation device 70 according to the second embodiment. The compatibility evaluation device 70 includes an acquisition unit 71, an index determination unit 72, and a calculation unit 73.
[0058] 7 is a flowchart of the process performed by the compatibility evaluation device 70. The acquisition means 71 acquires the outputs of the first predictor and the second predictor for the evaluation data (step S41). The index determination means 72 determines a generalized backward compatibility index defined by a combination of a plurality of relational expressions indicating the relationship between the output of the first predictor and the output of the second predictor (step S42). The calculation means 73 calculates a score indicating the compatibility between the first predictor and the second predictor using the output of the first predictor, the output of the second predictor, and the generalized backward compatibility index (step S43).
[0059] According to the compatibility evaluation device 70 of the second embodiment, it is possible to evaluate the compatibility of predictors using an appropriate compatibility index according to the task of the predictor.
[0060] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.
[0061] (Appendix 1) obtaining means for obtaining outputs of the first predictor and the second predictor for the evaluation data; An indicator determination means for determining a generalized backward compatibility indicator defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; A calculation means for calculating a score indicating compatibility between the first predictor and the second predictor using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index; A compatibility evaluation device comprising:
[0062] (Appendix 2) The compatibility assessment device according to claim 1, wherein the generalized backward compatibility index is expressed by arithmetic operations of a plurality of weighted relational expressions.
[0063] (Appendix 3) A designation means for receiving a designation of a compatibility index; The index determination means sets a weight for each of the plurality of relational expressions based on the designation, and determines an evaluation index from the generalized backward compatibility index; 3. The compatibility evaluation device according to claim 2, wherein the calculation means calculates the score using the evaluation index.
[0064] (Appendix 4) The above relation is: A first equation indicating a rate at which the output of the first predictor and the output of the second predictor are both correct; A second equation indicating a rate at which both the output of the first predictor and the output of the second predictor are incorrect; A third equation indicating the proportion of times that the output of the first predictor is incorrect and the output of the second predictor is correct; and a fourth equation indicating a proportion in which the output of the first predictor is correct and the output of the second predictor is incorrect.
[0065] (Appendix 5) The first predictor and the second predictor perform a regression analysis; The compatibility evaluation device described in Appendix 4, wherein the calculation means determines that the output is correct when a difference between a predicted value that is the output of the first predictor and the second predictor and an actual value corresponding to the predicted value is equal to or less than a predetermined threshold, and determines that the output is incorrect when the difference is greater than the threshold.
[0066] (Appendix 6) the relational expression indicates a magnitude relationship of the output of the first predictor for two pieces of evaluation data, and a magnitude relationship of the output of the second predictor for the two pieces of evaluation data, The compatibility evaluation device according to claim 1, wherein the calculation means calculates as the score an expected value at which a magnitude relationship between the output of the first predictor and a magnitude relationship between the output of the second predictor coincide.
[0067] (Appendix 7) Obtaining outputs of the first predictor and the second predictor for the evaluation data; determining a generalized backward compatibility index defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; A compatibility evaluation method for calculating a score indicating compatibility between the first predictor and the second predictor using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index.
[0068] (Appendix 8) Obtaining outputs of the first predictor and the second predictor for the evaluation data; determining a generalized backward compatibility index defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; A recording medium having a program recorded thereon to cause a computer to execute a process of calculating a score indicating compatibility between the first predictor and the second predictor using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index.
[0069] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-mentioned embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]
[0070] 100 Compatibility Evaluation Device 101 Interface 102 processors 103 Memory 104 Recording media 105 Input section 106 Display section 110 Evaluation index determination unit 120 Score calculation unit
Claims
1. obtaining means for obtaining outputs of the first predictor and the second predictor for the evaluation data; An indicator determination means for determining a generalized backward compatibility indicator defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; A calculation means for calculating a score indicating compatibility between the first predictor and the second predictor using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index; A compatibility evaluation device comprising:
2. 2. The compatibility evaluation device according to claim 1, wherein the generalized backward compatibility index is expressed by four arithmetic operations of a plurality of weighted relational expressions.
3. A designation means for receiving a designation of a compatibility index; The index determination means sets a weight for each of the plurality of relational expressions based on the designation, and determines an evaluation index from the generalized backward compatibility index; The compatibility evaluation device according to claim 2 , wherein the calculation means calculates the score using the evaluation index.
4. The above relation is: A first equation indicating a rate at which the output of the first predictor and the output of the second predictor are both correct; A second equation indicating a rate at which both the output of the first predictor and the output of the second predictor are incorrect; A third equation indicating a ratio of the output of the first predictor being incorrect and the output of the second predictor being correct; and a fourth equation indicating a ratio of the output of the first predictor being correct and the output of the second predictor being incorrect.
5. The first predictor and the second predictor perform a regression analysis; The compatibility evaluation device according to claim 4, wherein the calculation means determines that the output is correct when a difference between a predicted value that is the output of the first predictor and the second predictor and an actual value corresponding to the predicted value is equal to or smaller than a predetermined threshold, and determines that the output is incorrect when the difference is greater than the threshold.
6. the relational expression indicates a magnitude relationship of the output of the first predictor with respect to two pieces of evaluation data, and a magnitude relationship of the output of the second predictor with respect to the two pieces of evaluation data, 2 . The compatibility evaluation device according to claim 1 , wherein the calculation means calculates, as the score, an expected value at which a magnitude relationship between the output of the first predictor and a magnitude relationship between the output of the second predictor coincide.
7. A computer-implemented method for evaluating compatibility, comprising: Obtaining outputs of the first predictor and the second predictor for the evaluation data; determining a generalized backward compatibility index defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; A compatibility evaluation method for calculating a score indicating compatibility between the first predictor and the second predictor using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index.
8. Obtaining outputs of the first predictor and the second predictor for the evaluation data; determining a generalized backward compatibility index defined by a combination of a plurality of relational expressions indicating a relationship between an output of the first predictor and an output of the second predictor; A program that causes a computer to execute a process of calculating a score indicating compatibility between the first predictor and the second predictor using an output of the first predictor, an output of the second predictor, and the generalized backward compatibility index.
Citation Information
Patent Citations
Learning support device and learning support method
JP2019204190A
Learning model evaluation method, learning method, device, and program
JP2020004178A
Comparing models
US8296257B1
Predictive model updating system, predictive model updating method, and predictive model updating program
WO2016151618A1
