Drift detection device and drift detection method

Abstract

A drift detection device (100) detects drift occurring in an AI model and comprises: a failure rate calculation unit (13) for calculating a failure rate of a device within a prescribed period; a failure rate change detection unit (14) for detecting a failure rate change amount, which is a time-series change amount in the failure rate; and a determination unit (15) for determining that drift has occurred in input data of the AI model when the failure rate change amount of the device reaches a prescribed determination criterion.

Description

Drift detection device and drift detection method

[0001] The present disclosure relates to a drift detection device and a drift detection method.

[0002] In the field of network operations, automation of failure analysis using AI (artificial intelligence) models is being promoted to identify the location of failures in target equipment. In AI models, the accuracy of output characteristics can decrease due to a phenomenon called drift, in which the characteristics of the handled data change. When drift occurs, the AI ​​model must be retrained according to the change in output characteristics. Therefore, it is necessary to detect with high accuracy whether drift has occurred in the AI ​​model.

[0003] Non-Patent Document 1 proposes a method for detecting drift by using statistical techniques to determine whether there is a trend change in the distribution of feature values ​​(indicators that have a high contribution rate to analysis results) during AI learning and operation. Specifically, it is determined that drift has occurred when there is a large difference between the probability distribution during learning and the probability distribution during operation.

[0004] Learning underConcept Drift: A Review (Jie Lu, et al. IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 12, Dec.2019, pp. 2346-2363.)

[0005] However, the technology disclosed in the above-mentioned Non-Patent Document 1 is based on the premise that the features are known, and when performing AI model analysis that includes feature design such as deep learning, it is not possible to compare the probability distribution during learning and the probability distribution during operation from the perspective of the features, and therefore it is not possible to detect drift that needs to be re-learned with high accuracy.

[0006] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide a drift detection device and a drift detection method that are capable of detecting the occurrence of drift in an AI model, including feature design, with high accuracy.

[0007] A drift detection device according to one aspect of the present disclosure is a drift detection device that detects drift occurring in an AI model, and includes a failure rate calculation unit that calculates the failure rate of a device within a predetermined period of time, a failure rate change detection unit that detects a failure rate change amount, which is the amount of change in the failure rate over time, and a determination unit that determines that drift has occurred in the input data of the AI ​​model when the failure rate change amount of the device reaches a predetermined determination criterion.

[0008] A drift detection method according to one aspect of the present disclosure is a drift detection method for detecting drift occurring in an AI model, in which a failure rate calculation unit calculates the failure rate of a device within a predetermined period of time, a failure rate change detection unit detects a failure rate change amount, which is a time-series change amount of the failure rate, and a determination unit determines that drift has occurred in the input data of the AI ​​model when the failure rate change amount of the device reaches a predetermined determination criterion.

[0009] According to the present disclosure, it becomes possible to detect the occurrence of drift with high accuracy in an AI model that includes feature design.

[0010] FIG. 1 is a block diagram showing the configuration of a drift detection device according to an embodiment. FIG. 2A is a graph showing the number of alarms generated in devices A and B before drift occurs and failures occurring in devices A and B. FIG. 2B is a graph showing the number of alarms expected to occur in devices A and B after drift occurs and failures occurring in devices A and B. FIG. 3 is a graph showing failures occurring in devices A and B before drift occurs, with feature value X on the horizontal axis and feature value Y on the vertical axis. FIG. 4 is a graph showing a failure rate based on the number of failures occurring in devices A and B before drift occurs, as shown in FIG. 2A. FIG. 5 is a graph showing a failure rate based on the number of failures occurring in devices A and B after drift occurs, as shown in FIG. 2B. FIG. 6A is an explanatory diagram showing an example of device information. FIG. 6B is an explanatory diagram showing an example of information for determining a change in failure rate. FIG. 7 is a flowchart showing the processing procedure of a drift detection device according to this embodiment. FIG. 8 is a block diagram showing the hardware configuration of this embodiment.

[0011] Hereinafter, an embodiment will be described with reference to the drawings. First, drift occurring in an AI model will be described. Figures 2A and 2B are graphs showing the output of the AI ​​model. Figure 2A is a graph showing the output before drift occurs, and Figure 2B is a graph showing the output expected after drift occurs.

[0012] FIG. 2A shows failures occurring in devices A and B before drift occurs, with the vertical axis representing the number of alarms issued by device A and the horizontal axis representing the number of alarms issued by device B. In FIG. 2A, failures occurring in device A are indicated by the symbol "◯" and failures occurring in device B are indicated by the symbol "△". From the graph shown in FIG. 2A, it can be seen that as the number of alarms issued by device B increases, the number of failures occurring in device B increases and the failure rate rises. It can also be seen that as the number of alarms issued by device A increases, the number of failures occurring in device A increases and the failure rate rises. That is, the failure rate of device A is high in the upper left region of curve L1 shown in FIG. 2A, and the failure rate of device B is high in the lower right region.

[0013] On the other hand, when drift occurs in the AI ​​model, the number of failures increases in the region where the number of alarms is low, for example, for device B. Therefore, the output of the AI ​​model is expected to be a graph in which the number of failures of device B increases in region R1 where the number of alarms of device B is low, as shown in Figure 2B. In other words, when drift occurs in the AI ​​model, if the existing AI model shown in Figure 2A is used without re-learning the AI ​​model, it will be difficult to accurately identify the failure of device B.

[0014] Therefore, if drift occurs in the AI ​​model, it is necessary to retrain the AI ​​model based on new input data. In the example shown in Figures 2A and 2B, the feature quantities shown on the vertical and horizontal axes are set to the number of alarms issued by devices A and B, so the occurrence of drift can be detected by comparing the distributions of the graphs shown in Figures 2A and 2B from the perspective of the feature quantities. In other words, the user can easily recognize that retraining of the AI ​​model is necessary.

[0015] Fig. 3 shows failures that occur in devices A and B before drift occurs, with the horizontal axis representing feature value X and the vertical axis representing feature value Y. In Fig. 3, failures that occur in device A are indicated by the symbol "◯" and failures that occur in device B are indicated by the symbol "Δ". As shown in Fig. 3, as feature value X increases, the failure rate of device B increases, and as feature value Y increases, the failure rate of device A increases.

[0016] However, because the feature quantities X and Y are not specified for the AI ​​model shown in Fig. 3, it is not possible to determine whether or not drift has occurred in the AI ​​model. That is, in an AI model in which the feature quantities are self-evident as shown in Fig. 2A and Fig. 2B, drift can be detected based on changes in the output of the AI ​​model, but in an AI model in which the feature quantities are not self-evident, such as deep learning, it is difficult to detect drift.

[0017] In this embodiment, the occurrence of drift that requires re-learning is detected based on the fact that there is a correlation between the feature values ​​used in the AI ​​model and the failure rate of the device, as will be described in detail below.

[0018] Fig. 4 is a graph showing the failure rates based on the number of failures that occurred for devices A and B before the occurrence of the drift shown in Fig. 2A. In Fig. 2A, the number of failures that occurred for both devices A and B is the same (the number of "◯" and "△" is the same), so as shown in Fig. 4, both devices A and B have the same failure rate x1.

[0019] FIG. 5 is a graph showing the failure rates based on the number of failures of devices A and B after the occurrence of drift shown in FIG. 2B. In FIG. 2B, the number of failures of device B is greater than that of device A, so as shown in FIG. 5, the failure rate of device A is x2 and the failure rate of device B is x3 (x3 > x2). When drift occurs in the AI ​​model, the failure rate of the device increases. In other words, from the relationship between FIG. 2A and FIG. 4 and the relationship between FIG. 2B and FIG. 5, it can be understood that the feature quantities used in the AI ​​model to identify the failure location have a correlation with the failure rate of the device.

[0020] In this embodiment, drift occurring in an AI model is detected by monitoring the amount of change over time in the device failure rate. This makes it possible to detect drift with high accuracy in an AI model whose features are not self-evident, and use this as an indicator of whether or not to retrain the AI ​​model.

[0021] The following describes an embodiment with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a drift detection device 100 according to an embodiment. As shown in Fig. 1, the drift detection device 100 includes a processing unit 1 and a storage unit 2.

[0022] The processing unit 1 includes an information management unit 11 , an operating time calculation unit 12 , a failure rate calculation unit 13 , a failure rate change detection unit 14 , and a determination unit 15 .

[0023] The information management unit 11 collects various information about a device (not shown) that is to be monitored for failure occurrence from the device. Based on the collected information, the information management unit 11 groups the devices by desired items. Items to be grouped can be, for example, device name, vendor name, model name, purpose, and installation date (see FIG. 6A described later).

[0024] The operating time calculation unit 12 calculates the average operating time of the devices included in each group grouped by the information management unit 11. For example, if a group contains 10 devices, the operating time calculation unit 12 calculates the average operating time of the 10 devices from the time they started operating until the present (see FIG. 6B).

[0025] The failure rate calculation unit 13 collects failure information for each fixed operating period (hereinafter referred to as a "sampling period") from devices to be monitored for failure occurrence (e.g., device A and device B), and calculates the number of failures for each sampling period for each group. That is, the failure rate calculation unit 13 calculates the failure rate of the devices within a predetermined period (e.g., multiple sampling periods). The failure rate calculation unit 13 also calculates the failure rate for each predetermined sampling period.

[0026] The failure rate calculation method performed by the failure rate calculation unit 13 involves, for example, collecting the number of failures up to the present for each device and aggregating the number of failures for each sampling period. The failure rate calculation unit 13 then adds up the aggregated number of failures for each group. As a result, data such as "number of failures in group 1 for the past eight sampling periods: 0, 1, 0, 0, 1, 0, 2, 4" is obtained. The failure rate calculation unit 13 then further calculates the failure rate for each sampling period.

[0027] The failure rate change detection unit 14 detects the amount of change in the failure rate over time based on the failure rate calculated in each sampling period by the failure rate calculation unit 13. For example, the failure rate change detection unit 14 detects the amount of change in the failure rate over multiple sampling periods over time.

[0028] The determination unit 15 determines whether a failure rate change that satisfies a predetermined criterion has occurred in the target device based on the amount of failure rate change detected by the failure rate change detection unit 14. Hereinafter, the occurrence of a failure rate change that satisfies the predetermined criterion will be referred to as "reaching the determination criterion." When the amount of failure rate change reaches the determination criterion, the determination unit 15 determines that drift has occurred in the AI ​​model, and outputs this determination result to the outside. The determination result includes information about the device that has reached the determination criterion, the degree of change, etc.

[0029] The determination unit 15 determines that the change in the failure rate has reached the determination criterion, for example, when the failure rate detected in each sampling period is gradually increasing (e.g., increasing exponentially). Generally, the failure rate of a device remains stable for a long period of time. However, after a certain period of time, the failure rate tends to increase at an accelerated rate due to aging. The determination unit 15 determines that the change in the failure rate has reached the determination criterion when the failure rate for each sampling period is gradually increasing. "A gradual increase in the failure rate" refers to, for example, a gradual increase in the difference in the failure rates, such as d1 < d2 < d3, where d1 is the difference between the failure rates of the first and second sampling periods, d2 is the difference between the failure rates of the second and third sampling periods, and d3 is the difference between the failure rates of the third and fourth sampling periods.

[0030] Furthermore, it can be inferred that groups with similar model names or similar uses have approximately the same time period for changes in failure rate. Therefore, when the amount of change in failure rate for a device belonging to one group reaches the judgment criterion, the judgment unit 15 judges that the amount of change in failure rate for this device has reached the judgment criterion if there is a device belonging to another group with a similar model name or use to the first group and that device has a longer operating period than the device in the first group. The judgment unit 15 outputs this judgment result to the outside. In other words, the judgment unit 15 judges that drift has occurred in the AI ​​model when the amount of change in failure rate reaches a predetermined judgment criterion.

[0031] The storage unit 2 stores, for example, the information shown in FIG. 6A as device information that is the target of fault detection. The storage unit 2 stores, for example, the information shown in FIG. 6B as fault detection data generated in each device. As shown in FIG. 6A, the device information includes "device name," "vendor name," "model name," "purpose (core device, edge device, etc.)," ​​"introduction date," "group to which the device belongs," etc. As shown in FIG. 6B, the fault detection data includes "group name to which the device belongs," "average operating time," "number of faults per certain period," "time when the judgment criterion was reached," etc.

[0032] Next, the processing procedure of the drift detection device 100 according to this embodiment will be described with reference to the flowchart shown in FIG. 7. First, in step S11, the information management unit 11 acquires device information about devices that are the target of fault detection and groups the devices by similar devices. The grouping results are stored in the storage unit 2. The number of groups is defined as "N." That is, the device information for the first to Nth groups is stored in the storage unit 2.

[0033] In step S12, the operating time calculation unit 12 calculates the average operating period of each group grouped in the process of step S11.

[0034] In step S13, the failure rate calculation unit 13 calculates the number of failures for each sampling period for each group.

[0035] In step S14, the failure rate calculation unit 13 sets the number I of each group to "I=1".

[0036] In step S15, the failure rate calculation unit 13 calculates the failure rate from the number of failures of the devices included in the first group, and the failure rate change detection unit 14 calculates the amount of change in the failure rate for each sampling period. The determination unit 15 determines whether the amount of change in the failure rate has reached the determination criterion. For example, if the failure rate has gradually increased over the past several sampling periods, it is determined that the amount of change in the failure rate has reached the determination criterion. If the amount of change in the failure rate has reached the determination criterion (if there is a change), the process proceeds to step S17; if not (if there is no change), the process proceeds to step S16.

[0037] In step S16, the judgment unit 15 compares the operating times of devices in the first group that has been judged to have reached the judgment criterion with those of devices in other similar groups, and judges whether the amount of change in failure rate of the devices in the other groups has reached the judgment criterion. As described above, when the amount of change in failure rate of a device in one group has reached the judgment criterion, if there is a device in another group that has a model name or use similar to that of the first group and that has an operating period longer than that of the device in the first group, the judgment unit 15 judges that the amount of change in failure rate of this device has reached the judgment criterion.

[0038] In step S17, the determination unit 15 outputs the determination result to the user P. The user P recognizes that drift has occurred in the AI ​​model based on the determination result.

[0039] That is, when the determination unit 15 determines that "change has occurred" in the processing of step S15, it determines that a change in failure rate has occurred in the devices belonging to that group, which is linked to the occurrence of drift that requires re-learning of the AI ​​model. Also, even when the determination unit 15 determines that "no change" in the processing of step S15, if the amount of change in failure rate in a group similar to this group in model name or use reaches the determination criterion, it determines that a change in failure rate has occurred in the devices belonging to this group, which is linked to the occurrence of drift that requires re-learning of the AI ​​model.

[0040] In step S18, the processing unit 1 sets the variable I indicating the group number to "I = I + 1" and returns to step S15. In step S19, if "I > N", the processing ends. In this way, it becomes possible to detect drift that occurs in the AI ​​model that estimates the occurrence of failures based on changes in the failure rate of each device.

[0041] As described above, the drift detection device 100 according to this embodiment is a drift detection device 100 that detects drift that occurs in an AI model, and includes a failure rate calculation unit 13 that calculates the failure rate of the device within a predetermined period of time, a failure rate change detection unit 14 that detects a failure rate change amount, which is the amount of change in the failure rate over time, and a determination unit 15 that determines that drift has occurred in the input data of the AI ​​model when the failure rate change amount of the device reaches a predetermined determination criterion.

[0042] In this embodiment, the feature used in the AI ​​model for identifying the failure location has a characteristic of being correlated with the failure rate of the device, and when the change in the failure rate reaches a judgment criterion, it is determined that drift has occurred in the AI ​​model. This makes it possible to detect the occurrence of drift with high accuracy in an AI model that includes feature design. This allows the user to easily recognize whether or not re-learning of the AI ​​model is necessary, making it possible to estimate device failures using the AI ​​model with high accuracy.

[0043] In this embodiment, the failure rate calculation unit 13 calculates the failure rate for each predetermined sampling period, and the failure rate change detection unit 14 detects a time-series change in the failure rate based on the failure rate calculated for each sampling period. This makes it possible to detect a time-series change in the failure rate with high accuracy, and to detect the occurrence of drift with high accuracy.

[0044] In this embodiment, when the determination unit 15 determines that the failure rate for each sampling period is gradually increasing, it is determined that the amount of change in the failure rate has reached the determination criterion, making it possible to detect the occurrence of drift with high accuracy.

[0045] The drift detection apparatus 100 of the present embodiment described above can be implemented as a general-purpose computer system including, for example, a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906, as shown in Fig. 8. The memory 902 and the storage 903 are storage devices. In this computer system, the CPU 901 executes a predetermined program loaded onto the memory 902, thereby realizing each function of the drift detection apparatus 100.

[0046] The drift detection apparatus 100 may be implemented by one computer or by multiple computers, or may be a virtual machine implemented on a computer.

[0047] The program for the drift detection device 100 can be stored in a computer-readable recording medium such as a HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), DVD (Digital Versatile Disc), or can be distributed via a network. The computer-readable recording medium is, for example, a non-transitory recording medium.

[0048] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the present disclosure.

[0049] REFERENCE SIGNS LIST 1 Processing unit 2 Storage unit 11 Information management unit 12 Operating time calculation unit 13 Failure rate calculation unit 14 Failure rate change detection unit 15 Determination unit 100 Drift detection device

Claims

1. A drift detection device that detects drift occurring in an AI model, comprising: a failure rate calculation unit that calculates the failure rate of a device within a specified period; a failure rate change detection unit that detects a failure rate change amount, which is the amount of change in the failure rate over time; and a determination unit that determines that drift has occurred in the input data of the AI ​​model when the failure rate change amount of the device reaches a specified determination criterion.

2. The drift detection device according to claim 1, wherein the failure rate calculation unit calculates the failure rate for each predetermined sampling period, and the failure rate change detection unit detects the amount of change in the failure rate based on the failure rate calculated for each sampling period.

3. The drift detection device according to claim 2, wherein the determination unit determines that the amount of change in the failure rate has reached the determination criterion when the failure rate for each sampling period is gradually increasing.

4. A drift detection method for detecting drift occurring in an AI model, comprising: a failure rate calculation unit that calculates the failure rate of a device within a predetermined period; a failure rate change detection unit that detects a failure rate change amount, which is a time-series change amount of said failure rate; and a judgment unit that judges that drift has occurred in the input data of said AI model when the failure rate change amount of said device reaches a predetermined judgment criterion.

Images