Information processing device

Abstract

This information processing device for calculating ground-truth data for change points in time series data includes an arithmetic unit which calculates a set of candidate change points in the time series data by using a change detection model, trains an abnormality detection model during a training period corresponding to the candidate change point for each candidate change point in the set of candidate change points, calculates abnormality detection accuracy of the abnormality detection model during an evaluation period corresponding to the candidate change point, and determines a set of candidate change points in which the abnormality detection accuracy satisfies a predetermined condition, as ground-truth data for the change points.

Description

Information processing device

[0001] The present invention relates to a technique for detecting change points in time-series data.

[0002] A technique for detecting changes in the state of a system consisting of one or more devices using time-series data representing the state of the system at each point in time has been known. Here, the "system state" refers to the operating state of the system expressed by quantitative variables such as the "number of accesses" and the "number of users."

[0003] Conventional change detection techniques include, for example, a change detection technique using the subspace method (Non-Patent Document 1) and a change detection technique based on Bayes' theorem (Non-Patent Document 2). However, the change detection technique using the subspace method is constrained by the requirement that the time series to be detected must be a stationary process. Furthermore, the change detection technique based on Bayes' theorem is constrained by the requirement that the time series to be detected must be independently and identically distributed (iid).

[0004] Furthermore, a change detection method has been proposed for time series data that does not satisfy stationarity constraints or iid constraints, such as when the data exhibits periodic fluctuations (Patent Document 1).

[0005] One possible use of change detection techniques is to determine when to update (re-learn) the AI ​​model built into the system.

[0006] WO2023 / 084787

[0007] T. Ide and K. Tsuda, "Change-Point Detection Using Krylov Subspace Learning," in Procs. of the SIAM International Conference on Data Mining, Apr. 2007.R. P. Adams and D. J. C. MacKay, "Bayesian Online Changepoint Detection," arXiv:0710.3742, Oct. 2007.C. D. Stefano et al., "To Reject or Not to Reject: That is the Question - An Answer in Case of Neural Classifiers," IEEE Trans. Syst. Manag. Cyber, Vol.30, No.1, pp.84-94, Feb. 2000.S. Byers and A. E. Raftery, "Nearest-Neighbor Clutter Removal for Estimating Features in Spatial Point Processes," Journal Amer. Statis. Assoc. Vol.93, No.442, pp.577-584, Jun. 1998.D. Yu et al., "FindOut: Finding Outliers in Very Large Datasets," Knowl. Inform. Syst. Vol.4, No.4, pp.387-412, 2002.E. Eskin, "Anomaly Detection over Noisy Data using Learned Probability Distributions," in Proc. of ICML, Stanford, CA, USA, Jun. 2000, pp.255-262.A. Arning et al., "A Linear Method for Deviation Detection in Large Databases," in Proc. of KDD, Portland, OR, USA, Aug.1996. pp.164-169.M. Sakurada and T. Yairi, "Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction," In Proc. of MLSDA, New York, NY, USA, Dec. 2014, pp.4-11.

[0008] Consider an anomaly detection model as an example of an AI model. Non-Patent Documents 3 to 8 disclose various anomaly detection methods. In any anomaly detection method, when the normal state of the data changes, the anomaly detection model learned from past data becomes obsolete. Therefore, it is necessary to use a change detection model to detect change points in the normal state, discard data before the change point, and then re-learn the anomaly detection model.

[0009] To perform change detection with high accuracy, it is necessary to evaluate the accuracy of the change detection model and tune its hyperparameters using correct data for change points. However, with conventional technology, the only way to create correct data is for it to be created manually, and the points that are selected as correct data for change points are left to the subjective judgment of the creator.

[0010] The present invention has been made in view of the above points, and has as its object to provide a technique that makes it possible to create correct data on change points without involving human subjectivity.

[0011] According to the disclosed technology, there is provided an information processing device that calculates correct data for change-points in time-series data, the information processing device including a calculation unit that calculates a set of candidate change-points in the time-series data using a change detection model, learns an anomaly detection model for each candidate change-point in the set of candidate change-points during a learning period corresponding to the candidate change-point, calculates the anomaly detection accuracy of the anomaly detection model during an evaluation period corresponding to the candidate change-point, and determines the set of candidate change-points whose anomaly detection accuracy satisfies a predetermined condition as correct data for the change-points.

[0012] The disclosed technology provides a technology that makes it possible to create correct data on change points without involving human subjectivity.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS It is a figure for explaining an outline of an embodiment. It is a block diagram of an information processing device 100. It is a flowchart showing the operation of the information processing device 100. It is a figure which shows an example of a hardware configuration of the device.

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0015] Below, the conventional technology and its problems will be described in more detail, and then the technology according to this embodiment will be described.

[0016] (Regarding the Prior Art) As mentioned above, one possible use of change detection techniques is to use them to determine the timing of updating (relearning) an AI model built into a system. The reason for this is that when the system state changes, the AI ​​model built into the system is likely to become obsolete, making it necessary to update the AI ​​model in response to the changes.

[0017] Here, we consider an anomaly detection model as an example of an AI model. Various anomaly detection methods have been proposed, including statistical analysis-based methods, machine learning-based methods, and deep learning-based methods.

[0018] For example, Non-Patent Document 3 (Stefano 2000 et al.) discloses a classification-based method that learns a normal / abnormal classification model based on data with correct labels. Non-Patent Document 4 (Byers 1998 et al.) discloses a nearest neighbor-based method that uses the distance from nearby data or relative data density as an anomaly score.

[0019] Non-Patent Document 5 (Yu2002 et al.) discloses a clustering-based method that considers data that do not belong to a cluster to be anomalous, and uses the distance from the nearest centroid as an anomaly score. Non-Patent Document 6 (Eskin2000 et al.) discloses a statistical method that considers data that occur in a low probability region of a probability model to be anomalous.

[0020] Furthermore, Non-Patent Document 7 (Arning 1996 et al.) discloses an information theoretic method in which a subset that reduces complexity the most by excluding it from the entire set of data is determined to be abnormal. Non-Patent Document 8 (Sakurada 2014 et al.) discloses a deep learning-based method in which time series prediction is performed using a neural network and normal / abnormal classification is performed based on the prediction error.

[0021] Both anomaly detection methods either learn a normal / abnormal classification model from past data and classify normal / abnormal conditions, or learn the normal state and use the degree of deviation from that as an anomaly score.

[0022] In any anomaly detection method, if the normal state of the data changes, the anomaly detection model learned from past data becomes outdated. Therefore, it is necessary to detect changes in the normal state using a change detection method, discard data before the change point, and then re-train the anomaly detection model. Note that an anomaly detection model is a normal / abnormal classification model or a model of the normal state.

[0023] By repeatedly training the anomaly detection model over time while staggering the training period, it is possible to discard past data and prevent the anomaly detection model from becoming outdated. However, if the training period is long, the effects of data prior to the change point will be felt for a long period of time when a change occurs. Conversely, if the training period is short, the anomaly detection model will become sensitive and will react strongly to subtle changes in trends.

[0024] (About the Issues) As mentioned above, it is necessary to use a change detection method to detect changes in the normal state, discard data before the change point, and then retrain the anomaly detection model.

[0025] Change detection techniques are implemented using a change detection model. To achieve high accuracy in change detection, the change detection model must be evaluated for accuracy and hyperparameter tuning must be performed.

[0026] Ground-truth data for change points is required to evaluate the accuracy of change detection models and tune their hyperparameters. The purpose of change detection is to discard data from before the change point and retrain the anomaly detection model to improve anomaly detection accuracy. Therefore, it is desirable to create ground-truth data for change points that will maximize anomaly detection accuracy, and then use that ground-truth data to tune the hyperparameters.

[0027] However, with conventional technology, the only way to create correct data is for the data creator to create it, and the points that are selected as correct data for change points depend on the creator's subjective judgment. Therefore, with conventional technology, it is difficult to create correct data for change points that will lead to maximizing the accuracy of anomaly detection.

[0028] (Outline of the embodiment) In this embodiment, the information processing device 100 creates correct answer data for change points without incorporating human subjectivity. This makes it possible to create correct answer data for change points that leads to maximizing the accuracy of anomaly detection. Furthermore, the information processing device 100 creates correct answer data for change points and performs hyperparameter tuning of the change detection model. An overview of the processing executed by the information processing device 100 is as follows. To make the processing content easier to understand, the description will be made with appropriate reference to FIG. 1. FIG. 1 shows specific examples of time-series data, change-point candidates, correct answer data at the time of anomaly occurrence, a learning period, an evaluation period, etc.

[0029] Here, we consider time series data f(t) of one or more dimensions. Also, we consider a change detection model φ(π ｋ ) is given. The change detection model φ(π ｋ ) in ｋ is the k-th tuned combination of hyperparameters, and π ０ is the initial value combination of hyperparameters.

[0030] The information processing device 100 uses a motion detection model φ(π ｋ ) to determine the change flag c ｋ (t) is calculated, and c ｋ The time τ when (t) = 1 ｋ (i) A change point candidate set T consisting of (i=1, 2, ...) ｋ In FIG. 1, three change point candidates A, B, and C are shown.

[0031] Change flag c ｋ (t) is a flag that indicates whether a time point is a change point or not, generated by performing post-processing on the change score that indicates the degree of change at each time point of the time series data. ｋ (t) is the change score d calculated by the existing technology ｋ (t), and the calculation method will be described later.

[0032] The information processing device 100 calculates a change point candidate set T ｋ The following process (S1 to S2) is performed for all elements of the anomaly detection model ψ ｋ (i) Anomaly detection accuracy α(ψ ｋ (i), τ ｋ (i)) is the preset lower limit value α － The time point τ at which the value is equal to or greater than ｋ (i) Set U ｋ is determined as the correct data for the change point.

[0033] S1 (Step 1): The information processing device 100 selects a candidate τ ｋ The learning period [τ ｋ (i), τ ｋ (i)+Δτ] (Δτ is a given constant), and an anomaly detection model ψ is calculated based on the time series data (learning data) for the relevant period. ｋ (i) is learned. In FIG. 1, the learning period is indicated by D.

[0034] S2: The information processing device 100 selects the candidate τ ｋ The evaluation period [τ ｋ (i) + Δτ, τ ｋ (i+1)] is set and the anomaly detection model ψ is applied to the time series data (evaluation data) for that period. ｋ(i) is applied, and the application result is compared with the correct data V at the time of an abnormality occurrence in the period in question, thereby determining the abnormality detection accuracy α(ψ ｋ (i), τ ｋ In FIG. 1, the evaluation period is indicated by E.

[0035] Next, the information processing device 100 calculates a change point candidate set T ｋ The correct data U of the change points obtained by S1 to S2 executed for all elements of ｋ Using this, we perform hyperparameter tuning of the change detection model φ(π) and find the optimal hyperparameter combination π ＊ Calculate this π ＊ is π ｋ＋１ This becomes:

[0036] The information processing device 100 continues the above process (T ｋ Created by U ｋ (processing up to hyperparameter tuning using k = k + 1, π ｋ＋１ = π ＊ and execute it repeatedly.

[0037] An example of the termination determination using the termination condition is as follows:

[0038] Example 1: The information processing device 100 calculates the average value ave(α(τ ｋ (i))) is the target value α ＋ If this is achieved, the process ends.

[0039] Example 2: The information processing device 100 calculates the increase in the average anomaly detection accuracy “ave(α(τ ｋ (i)))−ave(α(τ ｋ－１ (i)))) is the lower limit of accuracy increase Δα － If it falls below this value, the process ends.

[0040] Example 3: The information processing device 100 ends the process when the number of repetitions k reaches the upper limit.

[0041] <How to obtain the change flag> Change detection model φ(π ｋ ) is the change score at time t calculated by ｋ (t), change flag cｋ (t). The change score d ｋ (t) can be calculated as the "change point score" disclosed in Patent Document 1 (WO2023 / 084787), for example. The method for calculating the change score (or change point score) is not limited to the technique disclosed in Patent Document 1, and any existing technique can be used.

[0042] In the technology disclosed in Patent Document 1, the change score is calculated as the degree of change from the past period to the current period based on the cluster transition tensors for the past period and the current period. Specifically, the degree of change is the distance between the cluster transition tensor in the past period and the cluster transition tensor in the current period.

[0043] Change flag c ｋ (t) is the change score d ｋ (t) is a 0-1 flag converted from (t), and the change score d ｋ This flag is obtained by processing such that the value is set to 1 when (t) exceeds a predetermined threshold, and the value is set to 0 otherwise.

[0044] Change score d ｋ The threshold value of (t) is, for example, the change score d ｋ It is possible to set a percentile value of (t) for a certain period in the past.

[0045] Furthermore, the change detection model in this embodiment may be any model that has learnable hyperparameters, receives time-series data as input, and outputs a change score at each time point t. Furthermore, the change detection model in this embodiment may be a model that has learnable hyperparameters, receives time-series data as input, and outputs a change flag at each time point t. Furthermore, the change point detection device disclosed in Patent Document 1 may be used as the change detection model in this embodiment.

[0046] (Device Configuration Example) Fig. 2 shows an example of the functional configuration of the information processing device 100. As shown in Fig. 2, the information processing device 100 includes an input unit 101, a change point candidate set creation unit 102, a change point correct data creation unit 103, a hyperparameter tuning unit 104, a termination determination unit 105, and an output unit 106. Note that the "change point candidate set creation unit 102, change point correct data creation unit 103, hyperparameter tuning unit 104, and termination determination unit 105" may be referred to as a calculation unit 110.

[0047] The input unit 101 inputs data to be used in processing. The change-point candidate set creation unit 102 creates a change-point candidate set using change flags. The change-point correct data creation unit 103 creates correct data of change points using an anomaly detection model.

[0048] The hyperparameter tuning unit 104 uses the correct data for the change points to tune the hyperparameters of the change detection model. The termination determination unit 105 performs termination determination based on termination determination conditions. The output unit 106 outputs the correct data for the change points and the tuning results of the change detection model (optimal hyperparameter combination). Note that the output unit 106 may output only the correct data for the change points or only the tuning results of the change detection model (optimal hyperparameter combination).

[0049] (Processing Flow) Next, the operation of the information processing apparatus 100 having the above configuration will be described with reference to the flowchart of FIG.

[0050] <S101> In S101, the input unit 101 inputs the following input data to the change point candidate set creation unit 102.

[0051] Time series data f(t) Hyperparameter initial value combination π of the change detection model ０ Learning period length Δτ of the anomaly detection model Correct data V at the time of anomaly occurrence Lower limit value α of anomaly detection accuracy － ・Target value α for anomaly detection accuracy ＋ (or the lower limit of the increase in the abnormality detection accuracy Δα － , or the upper limit k of the number of hyperparameter tuning ＋) <S102> In S102, the change point candidate set creation unit 102 creates a change detection model φ(π ｋ ) to calculate the change flag c at each point in time of the time series data f(t). ｋ (t) is calculated, and c ｋ The time τ when (t) = 1 ｋ (i) A change point candidate set T consisting of (i=1, 2, ...) ｋ The change point candidate set creating unit 102 creates the created change point candidate set T ｋ to the change point correct data creation unit 103.

[0052] <S103> In S103, the change point correct data creation unit 103 creates a change point candidate set T ｋ For all elements of the anomaly detection model ψ, the following steps S1 and S2 are performed. ｋ (i) Anomaly detection accuracy α(ψ ｋ (i), τ ｋ (i)) is the preset lower limit value α － The time point τ at which the value is equal to or greater than ｋ (i) Set U ｋ The change-point correct data creating unit 103 outputs the created change-point correct data to the hyperparameter tuning unit 104.

[0053] S1: Candidate τ ｋ The learning period [τ ｋ (i), τ ｋ (i)+Δτ] (Δτ is a given constant), and the anomaly detection model ψ ｋ Learn (i).

[0054] S2: Candidate τ ｋ The evaluation period [τ ｋ (i) + Δτ, τ ｋ (i+1)] and set the anomaly detection model ψ ｋ (i) is applied, and the anomaly detection accuracy α(ψ ｋ (i), τ ｋ (i)) is calculated.

[0055] In addition, the anomaly detection model ψ ｋRegarding (i), any model may be used as long as it can classify data at each time point into normal / abnormal or calculate an anomaly score. ｋ (i) may be based on any of statistical analysis, machine learning, and deep learning, or may be a model using a method other than "statistical analysis, machine learning, and deep learning."

[0056] Anomaly detection accuracy α(ψ ｋ (i), τ ｋ Regarding (i), the accuracy evaluation index to be used is not limited to a specific one. For example, the accuracy evaluation index may be Accuracy, Precision, Recall, or F-score.

[0057] <S104> In S104, the hyperparameter tuning unit 104 tunes the correct data U ｋ Using the change detection model φ(π ｋ ) and tune the hyperparameters to find the optimal hyperparameter combination π ＊ The hyperparameter tuning unit 104 outputs the calculated optimal hyperparameter combination to the termination determination unit 105.

[0058] Optimal hyperparameter combination π ＊ The calculation method is not limited to a specific method, but for example, it can be calculated by hyperparameter tuning using a Bayesian optimization framework with change detection accuracy (F-score, etc.) as an objective function.

[0059] <S105> In S105, the termination determination unit 105 determines whether the termination condition is satisfied. If satisfied, the process proceeds to S106. If not satisfied, the process proceeds to S107. ｋ＋１ = π ＊ Then, the process returns to step S102.

[0060] Specifically, the termination conditions include the following examples 1 to 3, and any of the termination conditions from examples 1 to 3 may be used.

[0061] Example 1: Change point candidate set Tｋ The average value of the anomaly detection accuracy for all elements of ｋ (i), τ ｋ (i))) is the target value α ＋ have achieved the following.

[0062] Example 2: The increase in the average anomaly detection accuracy ave(α(τ ｋ (i)))−ave(α(τ ｋ－１ (i))) is the lower limit of accuracy increase Δα － is below.

[0063] Example 3: The number of repetitions k is the upper limit k ＋ has reached this level.

[0064] <S106> In S106, the output unit 106 outputs the correct data U ｋ and the tuning result of the change detection model φ(π) (optimal hyperparameter combination) π ＊ Output.

[0065] (Hardware Configuration Example) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0066] That is, the information processing device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the information processing device 100. The program can be recorded on a computer-readable recording medium (such as a portable memory) and can be saved or distributed. The program can also be provided via a network such as the Internet or email.

[0067] Fig. 4 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 4 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0068] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0069] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0070] (Effects of the embodiment) As described above, the technology described in this embodiment makes it possible to create correct data of change points to be used for accuracy evaluation / hyperparameter tuning of a change detection model without involving human subjectivity.

[0071] In addition, it is possible to create correct data for change points that meet the purpose of change detection (such as maximizing the detection accuracy of the anomaly detection model).

[0072] Furthermore, by detecting changes using a model with hyperparameter tuning based on appropriate ground truth data, it is possible to appropriately determine the timing for re-learning the anomaly detection model.

[0073] The following additional notes are provided regarding the above-described embodiments.

[0074] <Additional Notes> (Additional Item 1) An information processing device that calculates ground truth data for change-points in time-series data, comprising a calculation unit that uses a change detection model to calculate a set of change-point candidates in the time-series data, trains an anomaly detection model for each change-point candidate in the set of change-point candidates during a learning period corresponding to the change-point candidate, calculates the anomaly detection accuracy of the anomaly detection model during an evaluation period corresponding to the change-point candidate, and determines a set of change-point candidates whose anomaly detection accuracy satisfies a predetermined condition as the ground truth data for the change-point. (Additional Item 2) The information processing device according to Additional Item 1, wherein the predetermined condition is that the anomaly detection accuracy is equal to or greater than a predetermined lower limit. (Additional Item 3) The information processing device according to Additional Item 1, wherein the calculation unit performs hyperparameter tuning of the change detection model using the ground truth data for the change-points to calculate an optimal combination of hyperparameters for the change detection model. (Supplementary Item 4) The information processing device according to Supplementary Item 3, wherein the calculation unit repeatedly calculates the change-point candidate set, the anomaly detection accuracy, and the ground truth data of the change-points while updating hyperparameters of the change detection model through the hyperparameter tuning, until a termination condition is satisfied.

[0075] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0076] REFERENCE SIGNS LIST 100 Information processing device 101 Input unit 102 Change point candidate set creation unit 103 Change point correct data creation unit 104 Hyperparameter tuning unit 105 End determination unit 106 Output unit 110 Calculation unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing device that calculates correct answer data for change-points in time-series data, the information processing device comprising: a calculation unit that calculates a set of change-point candidates in the time-series data using a change detection model; for each change-point candidate in the set of change-point candidates, trains an anomaly detection model during a learning period corresponding to the change-point candidate; calculates the anomaly detection accuracy of the anomaly detection model during an evaluation period corresponding to the change-point candidate; and determines the set of change-point candidates whose anomaly detection accuracy satisfies a predetermined condition as the correct answer data for the change-point.

2. The information processing device according to claim 1, wherein the predetermined condition is that the anomaly detection accuracy exhibits a value equal to or greater than a preset lower limit value.

3. The information processing device according to claim 1, wherein the calculation unit performs hyperparameter tuning of the change detection model using the correct data of the change points to calculate an optimal combination of hyperparameters of the change detection model.

4. The information processing device according to claim 3, wherein the calculation unit repeatedly calculates the change-point candidate set, the anomaly detection accuracy, and the correct data for the change points while updating the hyperparameters of the change detection model through the hyperparameter tuning, until a termination condition is met.

Images