APPARATUS, METHOD AND SYSTEM FOR CONCEPT DRIFT DETECTION - Patent application

Abstract

To determine whether or not a concept drift is present in seasonal time-series data.SOLUTION: A concept drift detection device includes: a data input unit which receives a time-series dataset including a set of past time-series data and a set of current time-series data; a baseline model generation unit which generates a baseline model on the basis of subsets of the set of past time-series data; a feature extraction unit which extracts a set of past data features, a set of baseline data features, and a set of current data features; a distance calculation unit which calculates a baseline distance and a current distance; and a concept drift detection unit which determines whether or not a concept drift is present between the set of current time-series data and the set of past time-series data, on the basis of baseline statistics and the current distance.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates generally to concept drift detection, and more specifically, to detecting concept drift in seasonal time series data. [Background technology]

[0002] In recent years, with the advancement of digitalization in a wide variety of business fields, utilizing digital data to gain insights has become increasingly important in business operations. AI (Artificial Intelligence) and ML (Machine Learning) technologies are examples of tools used for data analysis and insight generation.

[0003] AIOPs (AI for Information Technology (IT) Operations) is an industry trend that uses machine learning methods and data analysis of IT system data to assist human operators in managing IT system operations. AIOps solutions aim to maintain high reliability, availability, and security of IT system environment operations while reducing TCO (Total Cost of Operation) and OpEx (Operation Expenses) through the use of AI.

[0004] Several AIOps solutions involve making predictions about future outcomes by using Machine Learning (ML) trained predictive models. These predictive models are generally trained on historical data and then used to generate predictions based on a given input data set. Such predictive models have a wide range of applications, including healthcare, risk assessment, failure prediction, and growth forecasting.

[0005] However, predictive models can be susceptible to concept drift. In general, concept drift refers to the concept that the statistical properties of the target variable that a predictive model is trying to predict change in an unexpected way over time. This negatively impacts the accuracy of the predictive model as the statistical properties of the target variable diverge from the data on which the predictive model was originally trained. Detecting and mitigating concept drift is important to maintain accurate predictive models.

[0006] Conventionally, methods for detecting concept drift have been proposed. For example, Cavalcante et al. (Non-Patent Document 1) write that "A time series is a set of observations taken over a certain sampling interval. Several real-world dynamic processes, such as stock price fluctuations, exchange rates, and temperature, can be modeled as time series. Time series, as special data streams, may exhibit concept drift, which adversely affects the analysis and prediction of time series. Explicit drift detection methods based on monitoring time series features may provide a better understanding of how concepts evolve over time than methods based on monitoring the prediction errors of base predictors. In this paper, we propose an online explicit drift detection method called FEDD (Feature Extraction For Explicit Concept Drift Detection) that identifies concept drift in time series by monitoring time series features. Computational experiments have shown that FEDD outperforms error-based approaches in several linear and nonlinear artificial time series with rapid and gradual concept drift." [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] CAVALCANTE, R, MINKU, LL & OLIVEIRA, A 2016, FEDD: Feature Extraction for Explicit Concept Drift Detection in Time Series. in Proceedings of 2016 IEEE International Joint Conference on. IEEE Xplore, Vancouver, Canada, pp. 740-747. https: / / doi.org / 10.1109 / IJCNN.2016.7727274 Summary of the Invention [Problem to be solved by the invention]

[0008] Non-Patent Document 1 discloses a technique for detecting concept drift in stationary time series data by calculating the distance between statistical features of past and current time series data. A single-value drift detection threshold is calculated based on an exponentially weighted moving average control chart with known baseline mean and standard deviation distances.

[0009] However, the technique disclosed in Non-Patent Document 1 is applicable to concept drift detection in independent and identically distributed stationary time series data, but is not applicable to non-stationary seasonal time series data (e.g., performance data of an IT system). More specifically, Non-Patent Document 1 describes the use of a single-value drift detection threshold for concept drift detection, but such a single-value drift detection threshold is not effective for accurately determining concept drift in seasonal time series data that requires separate baseline statistics for each of multiple seasonal periods.

[0010] Therefore, an object of the present disclosure is to provide an apparatus, method, and system for concept drift detection that can determine the presence or absence of concept drift in seasonal time series data. [Means for solving the problem]

[0011] One representative example of the present disclosure relates to a concept drift detection device for detecting concept drift in a time series dataset, the concept drift detection device including a data input unit for receiving a time series dataset including a set of past time series data related to a first time period and a set of current time series data related to a second time period following the first time period, a baseline model generation unit for generating a baseline model based on a subset of the set of past time series data, a baseline model generation unit for dividing the set of past time series data into a set of past windows, dividing the set of current time series data into a set of current windows, dividing the set of baseline data created by the baseline model into a set of baseline windows, and extracting baseline data features from the set of baseline windows. a feature extraction unit that calculates a set of past data features from the set of past windows and a set of current data features from the set of current windows; a distance calculation unit that calculates baseline distances between a subset of the set of past data features and a subset of baseline data features associated with a corresponding time frame and calculates current distances between the subset of the set of current data features and the subset of baseline data features associated with the corresponding time frame; and a concept drift detection unit that calculates baseline statistics indicative of a criterion for determining concept drift based on the baseline distances and determines whether or not there is concept drift between the current set of time series data and the past set of time series data based on the baseline statistics and the current distances. Effect of the Invention

[0012] According to the present disclosure, it is possible to provide an apparatus, method, and system for concept drift detection that can determine the presence or absence of concept drift in seasonal time series data.

[0013] Other objects, configurations and effects will become apparent from the following description of the preferred embodiment of the invention. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 is a block diagram illustrating an example of a computing architecture for implementing embodiments of the present disclosure. [Diagram 2] FIG. 2 is a diagram showing an example of a configuration of a concept drift detection system according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a graph showing an example of concept drift in time-series data according to an embodiment of the present invention. [Figure 4] FIG. 4 illustrates an example set of seasonal time series data according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a flowchart illustrating an example of a concept drift detection process according to the present disclosure. [Figure 6] FIG. 6 is a block diagram illustrating an example of data flow in a concept drift detection apparatus 250 according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] As described herein, aspects of the present disclosure relate to detecting concept drift in seasonal time series data. Concept drift detection is an important research topic related to monitoring the performance of ML-trained predictive models. Many ML-trained models assume that the input-output relationships observed for target data are static and will remain the same for all future data. If this assumption fails for any reason, concept drift is said to have occurred. Here, the term "concept" refers to the target variable or quantity being predicted. Deviation from this concept, i.e., concept drift, can lead to a decrease in the performance of the ML-trained predictive model. In general, to detect concept drift, it is necessary to compare the current concept with some baseline concept and then detect the concept drift based on the error of the predictive model's performance and the characteristics of the actual data distribution. If the comparison between the current concept and the baseline concept differs by more than a threshold, it is determined that concept drift has occurred.

[0016] One scenario in which concept drift detection is important involves the analysis of IT system data using ML-trained predictive models. Because system configurations and workload distributions of IT systems, such as data storage facilities, change frequently, these configuration changes can affect the input-output relationships of observed IT system data, causing concept drift and negatively impacting the performance of ML-trained predictive models. To reduce operational expenses and total cost of ownership, it is desirable to automatically monitor and update the performance of these ML-trained predictive models.

[0017] Accordingly, aspects of the present disclosure relate to apparatus, methods, and systems for concept drift detection capable of determining the presence or absence of concept drift in seasonal time series data. As described in detail below, aspects of the present disclosure relate to generating separate baseline statistics for each seasonal period of a set of time series data. Further aspects relate to smoothing feature distances and reducing outliers using a distance smoothing operation. Further aspects relate to splitting the time series data into rolling time windows to obtain statistical features for shorter seasonal time frames.

[0018] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. It should be understood that the embodiments described herein do not limit the invention according to the claims, and that each of the elements and combinations thereof described in relation to the embodiments are not strictly necessary to implement the aspects of the present invention.

[0019] In the following description and the related drawings, various aspects are disclosed. These aspects can be changed to alternative aspects without departing from the scope of the present disclosure. Furthermore, well-known elements of the present disclosure may not be described in detail or may be omitted without obscuring the relevant matters of the present disclosure.

[0020] The terms "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Similarly, the term "aspects of the disclosure" does not require that all aspects of the disclosure include a particular feature, advantage or mode of operation.

[0021] Moreover, many aspects are described in terms of sequences of operations to be performed by, for example, elements of a computing device. It will be appreciated that various operations described herein may be performed by specific circuitry (e.g., an application specific integrated circuit (ASIC)), by program instructions executed by one or more processors, or a combination of both. Moreover, the sequences of operations described herein may be considered to be fully embodied in any form of computer-readable storage medium having stored therein a corresponding set of computer instructions that, when executed, cause an associated processor to perform the functions described herein. Thus, various aspects of the present disclosure may be embodied in several different forms, all of which are contemplated to be within the scope of the claims.

[0022] Referring to the drawings, Figure 1 shows a high level block diagram of a computer system 100 for implementing various embodiments of the present disclosure. The mechanisms and apparatus of various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 100 include one or more processors 102, memory 104, terminal interface 112, storage interface 113, I / O (input / output) device interface 114, and network interface 115. These components may be communicatively connected directly or indirectly via a memory bus 106, an I / O bus 108, a bus interface unit 109, and an I / O bus interface unit 110.

[0023] Computer system 100 may include one or more general purpose programmable central processing units (CPUs) 102A and 102B, collectively referred to herein as processors 102. In some embodiments, computer system 100 may include multiple processors, and in other embodiments, computer system 100 may be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include one or more levels of on-board cache.

[0024] In some embodiments, memory 104 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing or encoding data and programs. In some embodiments, memory 104 represents the entire virtual memory of computer system 100 and may also include virtual memory of other computer systems coupled to computer system 100 or connected via a network. While memory 104 may be conceptually considered as a single monolithic entity, in other embodiments, memory 104 may be a more complex configuration, such as a hierarchical structure of caches and other memory devices. For example, memory may exist in multiple levels of caches, and these caches may be further divided by function. Thus, one cache may hold instructions and another cache may hold non-instruction data used by one or more processors. Memory may be further distributed and associated with different CPUs or sets of CPUs, as known in any of the so-called non-uniform memory access (NUMA) computer architectures.

[0025] Memory 104 may store all or a portion of various programs, modules, and data structures for handling data transfers as described herein. For example, memory 104 may store concept drift detection application 150. In an embodiment, concept drift detection application 150 may include instructions or statements that execute on processor 102 or that are interpreted by instructions or statements that execute on processor 102 to perform functions as described further below. In an embodiment, concept drift detection application 150 may include instructions or statements that execute on processor 102 or that are interpreted by instructions or statements that execute on processor 102 to perform functions as described further below. In some embodiments, the concept drift detection application 150 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, the concept drift detection application 150 may include data other than instructions or descriptions. In some embodiments, a camera, sensor, or other data input device (not shown) may be provided to communicate directly with the bus interface unit 109, the processor 102, or other hardware of the computer system 100. In such a configuration, the need for the processor 102 to access the memory 104 and the concept drift detection application 150 may be reduced.

[0026] Computer system 100 may include a bus interface unit 109 for communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may couple to I / O bus 108 for transferring data to and from various I / O units. I / O bus interface unit 110 may communicate with a number of I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs), via I / O bus 108. Display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to display device 126. Computer system 100 may also include one or more sensors or other devices configured to collect data and provide the data to processor 102. By way of example, computer system 100 may include biometric sensors to collect heart rate data, stress level data, etc., environmental sensors to collect humidity data, temperature data, pressure data, etc., and motion sensors to collect acceleration data, movement data, etc. Other types of sensors may also be used. Display memory may be dedicated memory for buffering video data. Display system 124 may be connected to a display device 126, such as a standalone display screen, a computer monitor, a television, a tablet or handheld device, etc. In some embodiments, display device 126 may include one or more speakers for playing audio. Alternatively, one or more speakers for playing audio may be connected to the I / O interface unit. In other embodiments, one or more functions provided by display system 124 may be implemented in an integrated circuit that includes processor 102. Furthermore, one or more functions provided by bus interface unit 109 may be implemented in an integrated circuit that includes processor 102.

[0027] The I / O interface unit provides the ability to communicate with various storage and I / O devices. For example, the terminal interface unit 112 may be attached to user I / O devices 116, such as user output devices (e.g., a video display, speakers, and / or a television set, etc.) and user input devices (e.g., a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device, etc.). A user may use a user interface to enter input data or instructions into the user I / O devices 116 and the computer system 100 by manipulating the user input devices, and to receive output data using the user output devices. The user interface may be displayed on a display, played through speakers, or printed via a printer, for example, via the user I / O devices 116.

[0028] Storage interface 113 allows attachment of one or more disk drives or direct access storage device 117 (usually a magnetic disk drive storage device, but may also be an array of disk drives arranged to appear as a single large storage device to a host computer, or other storage devices including solid state drives such as flash memory). In some embodiments, storage device 117 may be implemented as any secondary storage device. Contents of memory 104 may be stored in storage device 117 and retrieved as needed. Input / output device interface 114 provides an interface to any of a variety of input / output devices such as printers, fax machines, etc. Network interface 115 provides one or more communication paths from computer system 100 to other digital devices and computer systems. These communication paths may include, for example, one or more networks 130.

[0029] 1 shows a particular bus structure providing direct communication paths between processor 102, memory 104, bus interface unit 109, display system 124, and I / O bus interface unit 110, in other embodiments, computer system 100 may include different buses or communication paths that may be arranged in various forms, such as point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, parallel and redundant paths, or any other suitable type of configuration. Additionally, while I / O bus interface unit 110 and I / O bus 108 are each shown as single units, in reality computer system 100 may include multiple I / O bus interface units 110 and / or multiple I / O buses 108. Additionally, although multiple I / O interface units are shown isolating I / O bus 108 from other communication paths running to various I / O devices, in other embodiments some or all of the I / O devices are directly connected to one or more system I / O buses.

[0030] In various embodiments, computer system 100 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer, etc. In other embodiments, computer system 100 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable type of electronic device.

[0031] Next, an example of a configuration of a concept drift detection system according to an embodiment of the present disclosure will be described with reference to FIG.

[0032] 2 is a diagram illustrating an example of a configuration of a concept drift detection system 200 according to an embodiment of the present disclosure. As illustrated in FIG. 2, the concept drift detection system 200 includes a client device 210, a data storage device 220 that stores a time series data set 230, a communication network 240, and a concept drift detection device 250. The client device 210, the data storage device 220, and the concept drift detection device 250 may be communicatively connected via the communication network 240. Here, the communication network 240 may include the Internet, a LAN (Local Area Network) connection, a MAN (Metropolitan Area Network) connection, a WAN (Wide Area Network) connection, or the like.

[0033] The client device 210 is a device configured to receive and transmit information to the concept drift detection apparatus 250 and / or the data store 220. The client device 210 may be used by an owner or manager of the data store 220 and the time series dataset 230 to send a concept drift detection request to the concept drift detection apparatus 250 to initiate an analysis of the time series dataset 230. In an embodiment, the client device 210 may include a personal computing device (e.g., a smartphone, a tablet, a smartwatch, a laptop computer), etc. The client device 210 may be configured to receive commands and instructions from a user via a user interface.

[0034] Data storage 220 is a device configured to store and maintain time series data set 230. In certain embodiments, data storage may include a hard disk drive, random access memory, read only memory, erasable programmable read only memory, or flash memory, static random access memory, etc. In some embodiments, data storage 220 may include multiple distributed cloud storage systems.

[0035] The time series dataset 230 is the data that is analyzed by the concept drift detector 250. Here, the time series dataset 230 refers to a collection of data points indexed in time at successive equally spaced time points. By way of example, the time series dataset 230 may include data related to weather observations, IT systems, economics, healthcare, or any other information that can be modeled in terms of a time sequence. In one embodiment, the time series data set 230 may include a set of historical time series data associated with a first time period and a set of current time series data associated with a second time period subsequent to the first time period, where the first and second time periods may refer to any time period such as a year, a month, a week, a day, an hour, etc.

[0036] The time series dataset 230 may include internal structures or relationships between the time series data points, such as autocorrelation, trends, or seasonality. It is desirable to recognize and take these internal structures and relationships into account when training an ML predictive model. More specifically, as described herein, aspects of the present disclosure relate to when the time series dataset 230 is seasonal. Here, seasonality refers to the presence of fluctuations in the data that occur at specific regular intervals. Seasonality may be caused by a variety of factors, such as changes in system configuration, weather, employee work schedules, etc., and consists of cyclical, repetitive, and generally regular and predictable patterns in the time series dataset 230.

[0037] The concept drift detection device 250 is a device configured to analyze the time series data set 230 and determine the presence or absence of concept drift. For example, the concept drift detection device 250 may determine the presence of concept drift between a set of past time series data and a set of current time series data included in the time series data set 230. As shown in FIG. 2, the concept drift detection device 250 mainly includes a data input unit 252, a baseline model generation unit 254, a feature extraction unit 256, a distance calculation unit 258, and a concept drift detection unit 260. However, the present disclosure is not limited thereto, and the concept drift detection device may include other functional units (for example, a distance smoothing unit described later).

[0038] The data input 252 is a functional unit configured to receive the time series dataset 230 from the data store 220. As described herein, the time series dataset 230 may include a set of past time series data and a set of current time series data. In an embodiment, the concept drift detector 250 may receive the time series dataset 230 transmitted over the communication network 240. In an embodiment, the concept drift detector 250 may be granted access to the data store 220 to perform analysis of the time series dataset 230 without transmitting the time series dataset 230 over the communication network 240.

[0039] The baseline model generator 254 is a functional unit configured to generate a baseline model based on a subset of a set of historical time series data, where, by way of example, the baseline model may include a trained machine learning model configured to generate a set of forecasted time series data based on a set of current time series data.

[0040] The feature extraction unit 256 is a functional unit configured to divide a set of past time series data into a set of past windows, divide a set of current time series data into a set of current windows, divide a baseline model into a set of baseline windows, and then calculate a set of baseline data features from the set of baseline windows, calculate a set of past data features from the set of past windows, and calculate a set of current data features from the set of current windows.

[0041] The distance calculation unit 258 is a functional unit configured to calculate baseline distances between a subset of the set of past data features and a subset of baseline data features associated with a corresponding time frame, and to calculate current distances between a subset of the set of current data features and the subset of baseline data features associated with a corresponding time frame.

[0042] The concept drift detection unit 260 is a functional unit that calculates a baseline indicating a criterion for determining concept drift based on the baseline distance, and determines whether or not there is concept drift between the current set of time series data and the past set of time series data based on the baseline and the current distance.

[0043] The above-described functional parts of the concept drift detection apparatus 250 may be implemented as software modules configured to run on a computer system (e.g., software modules of the concept drift detection application 150 of the computer system 100 shown in FIG. 1). In other embodiments, the above-described functional parts of the concept drift detection apparatus 250 may be implemented as dedicated hardware units.

[0044] 2 shows an example of the configuration of the concept drift detection system 200, but the present disclosure is not limited to this. For example, the concept drift detection device 250 and the data storage device 220 may be integrated into one hardware unit, or the client device 210, the data storage device 220, and the concept drift detection device 250 may all be implemented on the same local area network.

[0045] According to the concept drift detection system 200 shown in FIG. 2, it is possible to perform concept drift detection that can determine the presence or absence of concept drift in seasonal time series data.

[0046] Next, an example of a graph showing concept drift will be described with reference to FIG.

[0047] 3 is a graph 300 illustrating an example of concept drift in time series data according to an embodiment of the present disclosure. The graph 300 shows both real values ​​315 and predicted values ​​325 of a particular target variable (i.e., concept) over time. As an example, the real values ​​315 may represent actual measured I / O traffic load of a storage device, and the predicted values ​​325 may represent I / O traffic load of the storage device predicted by an ML prediction model.

[0048] As shown in graph 300, the progression of the predicted values ​​325 closely corresponds to the progression of the actual values ​​315 over the course of the first time period 310, but over the course of the second time period 320, the progression of the predicted values ​​325 deviates from the progression of the actual values ​​315. Here, it can be said that concept drift has occurred in the actual values ​​315, resulting in a decrease in the predictive accuracy of the ML prediction model. This concept drift may have occurred as a result of, for example, a change in the configuration of the storage device that affects the I / O traffic load.

[0049] Accordingly, aspects of the present disclosure relate to detecting such concept drift in seasonal time series data and updating forecasting models to improve forecast accuracy.

[0050] Next, an example of seasonal time series data will be described with reference to FIG.

[0051] As described herein, aspects of the present disclosure relate to detecting concept drift in seasonal time series data. Here, seasonality refers to the presence of fluctuations in data that occur at specific regular intervals. Seasonality can be caused by a variety of factors, such as changes in system configuration, weather, etc., and consists of cyclical, repetitive, and generally regular and predictable patterns in a time series dataset. To improve the accuracy of forecasting models that analyze time series data with seasonal characteristics, it is desirable to take seasonality into account.

[0052] FIG. 4 illustrates an example of a set of seasonal time series data 400 according to an embodiment of the present disclosure. As described herein, seasonal time series data naturally includes periodic and repetitive patterns of variation. Here, the periodic and repetitive patterns of variation present in seasonal time series data are referred to as seasonal time patterns. As an example, as shown in FIG. 4, the set of seasonal time series data 400 includes three seasonal time patterns 410, 420, and 430. Each of the three seasonal time series patterns corresponds to a predetermined period. As an example, as shown in FIG. 4, each of the three seasonal time series patterns may correspond to a predetermined period of "one week", but the present disclosure is not limited thereto, and the predetermined period may be one second, one minute, one hour, one day, multiple days, one year, or any other period.

[0053] Additionally, each seasonal time pattern 410, 420, 430 includes one or more seasonal time windows 412, where a seasonal time window 412 refers to a series of consecutive seasonal time pattern points 422 that make up part of the seasonal time pattern 410, 420, 430. Each seasonal time window 412 in a particular seasonal time pattern 410, 420, 430 corresponds to a seasonal time window in another seasonal time pattern of the set of seasonal time series data 400. As an example, if each seasonal time pattern 410, 420, 430 represents a week, then "Monday" may represent a seasonal time window 412 in each of the three seasonal time patterns 410, 420, 430.

[0054] The seasonal time pattern points 422 refer to particular points in the evolution of the seasonal time patterns 410, 420, 430 and correspond to particular time features, where a time feature refers to a predetermined recurring point in time or period, and may include a second, a minute, an hour, a day, a year, or any other period. The seasonal time patterns 410, 420, 430 may be defined by the seasonal time pattern points 422 corresponding to any suitable time feature. The time features of a particular seasonal time pattern 410, 420, 430 depend on the type of seasonality observed and the observed interval of the time series.

[0055] As an example, if each seasonal time pattern 410, 420, 430 represents a week, each seasonal time pattern point 422 may correspond to a day of the week (in this case there are 7 seasonal time pattern points 422 per seasonal time pattern), an hour (in this case there are 168 time pattern points 422 per seasonal time pattern), a minute (in this case there are 10,080 time pattern points 422 per seasonal time pattern), etc. Two seasonal time pattern points 422 are considered equal if they have the same time feature value (i.e., the same value on the vertical axis). Similarly, two seasonal time windows 412 are considered equal if their start and end seasonal time pattern points are equal.

[0056] An example of seasonality can be observed in the use of computing resources of IT systems used in large business environments. In such a scenario, the seasonality present in a time series dataset representing the use of IT system computing resources is heavily influenced by users' working hours. For example, the use of IT system computing resources increases significantly during employees' working hours (e.g., Monday to Friday, 9:00 a.m. to 5:00 p.m.) and decreases outside these time frames. Thus, users' working hours give rise to both weekly and daily seasonality in the time series dataset representing the use of IT system computing resources.

[0057] Next, an example of a concept drift detection process according to the present disclosure will be described with reference to FIG.

[0058] 5 is a flow chart illustrating an example of a concept drift detection process 500 according to the present disclosure. The concept drift detection process 500 illustrates a method for detecting concept drift in seasonal time series data, and may be executed by various functional units of a concept drift detection device according to the present disclosure (e.g., the concept drift detection device 250 illustrated in FIG. 2). As illustrated in FIG. 5, the concept drift detection process may start at step S501 and end at step S599.

[0059] First, in step S510, a data input unit (e.g., the data input unit 252 of the concept drift detection device 250) inputs a set of past time series data (X past ) and the current set of time series data (X current ) where the time series dataset may be sent from a client device to a data input of the concept drift detection device or may be retrieved from a local or distributed storage device accessible to the concept drift detection device. As described herein, the time series dataset may include data related to weather observations, IT systems, economics, healthcare, or any other information that can be modeled in a temporal order. In one embodiment, the time series data set may include a set of historical time series data associated with a first time period and a set of current time series data associated with a second time period subsequent to the first time period, where the first and second time periods may refer to any time period such as a year, a month, a week, a day, an hour, etc.

[0060] Next, in step S520, a baseline model generator (e.g., the baseline model generator 254 of the concept drift detection device 250) generates a baseline model based on a subset of the set of past time series data. The baseline model may be configured to process the subset of the set of past time series data to generate a set of baseline data. As described below, this baseline data is used to create both a baseline distance and a current distance to determine the presence or absence of concept drift. The baseline data generated by the baseline model may include a baseline seasonal pattern that provides one data value for each combination of seasonal features, or actual past values ​​to which seasonal time features are aligned with actual data (past or present) against which they are compared.

[0061] The baseline model can be implemented in various configurations. For example, in one embodiment, the baseline model may be implemented as an ML (Machine Learning) prediction model trained to detect seasonal patterns in time series data and predict output. Here, in one embodiment, in the training phase, the ML model may be trained based on seasonal time features (e.g., one minute, one hour, day of the week, etc.). Then, in the inference phase, the ML model can input the seasonal time features and output seasonal time patterns. Furthermore, in another embodiment, in the training phase, the ML model may be trained based on time series data. Then, in the inference phase, the ML model can generate a time series as an output based on input data (e.g., recent past data of the time series). In this way, by outputting the time series predicted using the ML model as baseline data, it becomes possible to determine the presence or absence of concept drift based on the performance error of the prediction model for the past data with respect to the current data (i.e., the prediction accuracy of the ML model for the past data is compared with the prediction accuracy for the current data, and the presence or absence of concept drift is determined based on this comparison).

[0062] Furthermore, in some embodiments, the set of past time series data may be processed without using a machine learning model. In that case, the baseline model may be a statistical model configured to average the set of past time series data according to the seasonal time characteristics of the data (i.e., calculate the average of the values ​​of the past time series data of the set of past time series data at the corresponding seasonal time pattern point) and output the seasonal time pattern. Furthermore, in other embodiments, the baseline model may be configured to shift the set of past time series data and the set of current time series data by one or more seasonal time pattern lengths in the past (i.e., shift by N) before using them as a basis for comparison with the actual past and current data, and output the baseline data time series. In this way, directly calculating the baseline data from the set of past time series data allows the presence or absence of concept drift to be determined based on the observed deviation of the seasonal time pattern of the actual data (i.e., determine the magnitude of deviation of the past time series data and the current time series data from the baseline data, and determine that concept drift exists if the deviation of the current time series data is larger). As described herein, various types of time series data can be used as input, and the present disclosure is not limited to the case where a trained ML prediction model is available, although for ease of explanation, the following describes the case where a trained machine learning prediction model is used as the baseline model.

[0063] Next, in step S530, the feature extraction unit (e.g., the feature extraction unit 256 of the concept drift detection device 250) divides the set of past time series data into a set of past windows, divides the set of current time series data into a set of current windows, and divides the set of baseline data into a set of baseline windows. Here, the expression "divide into windows" means dividing the past time series data, the current time series data, and the baseline data into divided parts of fixed length shorter than the seasonal time pattern of each data set. For example, the feature extraction unit can divide the set of past time series data, the set of current time series data, and the set of baseline data into rolling windows with a fixed time difference window of length "l", where l is less than the length of the seasonal time pattern. Here, the windows may partially overlap. For example, the first window may have a window length from time 1 to the length of l, and the second window may have a window length from time 2 to the length of l+1. The amount of overlap of the windows can be freely configured. As another example, a first window may have a window length from time 1 to time l, and a second window may have a window length from time 10 to time l+10. In this way, by dividing the data into windows that are shorter in length than the seasonal time patterns, it becomes possible to detect smaller concept drift changes in the seasonal time patterns that occur only at specific seasonal time pattern points or time windows.

[0064] Next, in step S540, the feature extractor calculates a set of baseline data features from the set of baseline windows, calculates a set of past data features from the set of past windows, and calculates a set of current data features from the current window. Here, calculating the set of baseline data features, the set of past data features, and the set of current data features may include extracting statistical features for each window. More specifically, the feature extractor may obtain a feature vector for each window from a first time point (i~1) to a second time point i of a time series having a length M, where l≦i≦M (or from a first time feature (j~1) to j of a seasonal time pattern having a length N, where 1≦j≦N). The statistical features calculated here may include the mean, standard deviation, betweenness, variance, interquartile range, kurtosis, skewness, median absolute deviation, maximum value, minimum value, etc.

[0065] Thus, a feature vector is computed for each window of three types of input data: a set of past data features computed directly from the set of past windows, a set of baseline data features computed from the baseline data output by the baseline model, (either the time series data or the seasonal time pattern data), and a set of current data features computed directly from the set of current windows. Thus, one feature vector is computed for each window, and there will be a set of feature vectors for the set of windows.

[0066] Next, in step S550, the distance calculation unit (e.g., the distance calculation unit 258 of the concept drift detection device 250) calculates baseline distances between a subset of the set of past data features and a subset of the baseline data features associated with the corresponding time frame, and calculates current distances between a subset of the set of current data features and a subset of the baseline data features associated with the corresponding time frame. More specifically, the distance calculation unit may calculate baseline distances between past data feature vectors in the set of past data features and baseline data feature vectors in the set of baseline data features associated with the corresponding time frame, and calculate current distances between current data feature vectors in the set of current data features and baseline data feature vectors in the set of baseline data features associated with the corresponding time frame. Here, "data feature vectors associated with a corresponding time frame" refers to data feature vectors obtained for substantially the same period (e.g., April 1, 2021, from 8:00 a.m. to 8:00 p.m.). The baseline distance is a quantitative measure of the relative similarity between the past data feature vector and the baseline data feature vector. Similarly, the current distance is a quantitative measure of the relative similarity between the current data feature vector and the baseline data feature vector. In one embodiment, the baseline distance and the current distance may be calculated using the following formulas:

number

[0067] Here, A and B represent two different feature vectors. For example, when calculating the baseline distance, A may be the feature vector of a time window of the past data, and B may be the feature vector of a time window of the baseline data. Also, when calculating the current distance, A may be the feature vector of a time window of the current data, and B may be the feature vector of a time window of the baseline data. In this way, the baseline distance can be calculated between the feature vectors of the past data and the feature vectors of the baseline data, and the current distance can be calculated between the feature vectors of the current data and the feature vectors of the baseline data. By calculating the distance between the feature vectors obtained from the individual windows (e.g., as opposed to calculating the distance between the actual data), it is possible to reduce the error tendency to smaller drifts / inconsistencies in the time series.

[0068] Note that if the baseline data feature vector consists of only one seasonal time pattern period, the baseline data feature vector is iteratively assigned based on the corresponding time features of the past or current data feature vector. Furthermore, the output of this step is the past time series data X past and the current time series data X current For each available time point i in the set of, ...

[0069] Next, in step S560, the distance smoothing unit 259 (for example, the distance smoothing unit 259 shown in FIG. 6) performs distance smoothing on the baseline distance time series and the current distance time series calculated in step S550. Here, the distance smoothing unit may perform distance smoothing on each data point in the baseline distance time series and the current distance time series based on the value of the time close to that point (for example, by emphasizing a predetermined weight on the most recent observation value). More specifically, the distance smoothing unit may apply an EWM (Exponentially Weighted Moving Average) and variance technique to the baseline distance time series and the current distance time series. The distance smoothing unit may calculate the EWM average using the following formula:

number

number

[0070] where x i is the current distance value at a particular time point i corresponding to a time window, and the parameter 0≦λ≦1 represents the weight given to recent data compared to previous data. In addition, when smoothing the current distance time series, the distance smoother may use the already smoothed values ​​of the baseline distance time series as input to improve the smoothing effect on the first observation (where the correct matching of time series features needs to be considered). In this way, a set of smoothed baseline distance data (EWM smoothed mean / variance) and a set of smoothed current distance data (EWM smoothed mean) can be obtained. Smoothing the baseline distance time series and the current distance time series can reduce noise, false positives, and false negatives in concept drift detection.

[0071] Next, in step S570, the concept drift detector calculates baseline statistics based on the baseline distance data (e.g., the smoothed baseline distance data obtained in step S560). Here, the baseline statistics indicate a criterion for determining concept drift. In one embodiment, the baseline statistics may be a seasonal baseline indicating a criterion for determining concept drift for each set of seasonal time pattern points of the baseline distance data belonging to the same seasonal time feature (e.g., minute, hour, day of week). The concept drift detector may calculate the seasonal baseline by calculating an overall mean and standard deviation for each set of seasonal time pattern points in the baseline distance time series.

[0072] More specifically, the concept drift detector calculates all the EWM-smoothed average values ​​μ i and the variance value σ 2For each seasonal time feature j, a seasonal baseline may be calculated by averaging j In this way, we can obtain a single baseline mean and standard deviation value for each seasonal time feature j. j may be calculated using the following formula:

number

number

[0073] Next, in step S580, the concept drift detection unit determines whether or not there is a concept drift between the current set of time series data and the past set of time series data based on the baseline statistics (e.g., seasonal baseline) and the current distance data (e.g., the smoothed current distance data obtained in step S560). Here, the concept drift detection unit calculates a seasonal threshold for each seasonal time feature by a statistical thresholding method using the seasonal mean and standard deviation indicated by the seasonal baseline calculated in step S570, and then calculates the seasonal threshold for the current distance μ based on the seasonal threshold. currenti For a smoothed average seasonal time pattern point i of currenti It may be determined whether there is concept drift for point i of the EWM smoothed average of

number

[0074] Here, the parameters “k” and “σ” are the standard deviations by which the current mean can deviate from the baseline mean without being considered as concept drift. jAs an example, k may be set to a value of 3, although values ​​less than 3 may be appropriate if distance smoothing in step S560 is applied. currenti The smoothed average of is the baseline average μ of the seasonal time characteristics. j and k and standard deviation σ j If the current distance μ is greater than the sum of the products of the products (i.e., seasonal thresholds), the concept drift detector determines that there is a concept drift between the current set of time series data and the past set of time series data for the seasonal time pattern point i. Thus, the concept drift detector determines that the current distance μ currenti It may output a Boolean value that is true if the smoothed mean of satisfies the seasonal threshold, and false otherwise.

[0075] In some embodiments, the concept drift detector may be configured to output a concept drift notification indicating the presence or absence of concept drift between the current set of time series data and the past set of time series data for the seasonal time pattern points. In some embodiments, the concept drift detector may be configured to output the concept drift notification when it determines that concept drift exists for a predefined number of (consecutive) seasonal time pattern points of the smoothed current distance. That is, to determine that concept drift exists, the smoothed current distance needs to exceed the seasonal threshold for more than a certain time (one or more time steps) or for more than several short-term time windows belonging to the same seasonal time pattern (e.g., in case of weekly seasonality, when concept drift is detected on the same day of the week for several consecutive weeks, etc.).

[0076] If concept drift is detected, the concept drift detector may update the set of historical time series data and retrain the baseline model. If concept drift is not detected, the concept drift detector may update the set of historical time series data and the baseline model with additional data to further improve performance.

[0077] Thus, the concept drift detection process 500 described with reference to Figure 5 can determine the presence or absence of concept drift in seasonal time series data. Note that unlike existing methods that use a single threshold to detect concept drift in stationary data, the concept drift detection process 500 determines baseline statistics for each of a set of seasonal time pattern points of baseline distance, thereby enabling concept drift to be detected in seasonal time series data.

[0078] Next, an example of the data flow in the concept drift detection device 250 according to the present disclosure will be described with reference to FIG.

[0079] FIG. 6 is a block diagram illustrating an example of data flow in a concept drift detection apparatus 250 according to the present disclosure.

[0080] First, the data input unit 252 receives the time series data set 230. Here, the time series data set 230 may be sent to the data input unit 252 of the concept drift detection device 250 from a client device (e.g., the client device 210 shown in FIG. 2) or may be retrieved from a local or distributed storage device accessible to the concept drift detection device 250. As described herein, the time series data set 230 may include a set of past time series data 602 and a set of current time series data 604.

[0081] The baseline model generator 254 then generates a baseline model 606 based on a subset of the historical time series data set 602. The baseline model 606 may be configured to process the subset of the historical time series data set 602 to generate a set of baseline data. As described below, this baseline data is used to create both baseline distance data and current distance data to determine whether concept drift exists.

[0082] Next, the feature extraction unit 256 divides the set of past time series data 602 into a set of past windows, divides the set of current time series data 604 into a set of current windows, and divides the set of baseline data into a set of baseline windows, and then calculates a set of baseline data features 610 from the set of baseline windows, a set of past data features 608 from the set of past windows, and a set of current data features 612 from the set of current windows, where calculating the set of baseline data features, the set of past data features, and the set of current data features may include extracting statistical features for each window.

[0083] The distance calculator 258 then calculates a baseline distance 614 between the subset of the set of past data features 608 and the subset of the baseline data features 610 associated with the corresponding time frame, and calculates a current distance 616 between the subset of the set of current data features 612 and the subset of the baseline data features 610 associated with the corresponding time frame, where the baseline distance 614 is a quantitative measure of the relative similarity between the set of past data features 608 and the set of baseline data features 610. Similarly, the current distance 616 is a quantitative measure of the relative similarity between the set of current data features 612 and the set of baseline data features 610.

[0084] Next, the distance smoothing unit 259 performs distance smoothing on the baseline distance 614 and the current distance 616 calculated by the distance calculation unit 258. Here, the distance smoothing unit 259 may perform distance smoothing on each point of the baseline distance 614 and the current distance 616, and calculate a smoothed baseline distance 618 and a smoothed current distance 620.

[0085] Next, the concept drift detector 260 calculates a seasonal baseline 622 based on the smoothed baseline distance 618. More specifically, the concept drift detector 260 may calculate a seasonal baseline for each seasonal time feature by calculating the overall mean and standard deviation for each set of seasonal time pattern points at the baseline distance. The concept drift detector 260 may then perform concept drift detection 624 based on the seasonal baseline 622 and the smoothed current distance 620.

[0086] In one embodiment, the concept drift detector 260 may output a concept drift notification indicating whether there is a concept drift between the current set of time series data 604 and the past set of time series data 602. The concept drift detector 260 may send the concept drift notification to a client device (e.g., client device 210 shown in FIG. 2) of a client that owns or manages the time series data set 230. In one embodiment, the concept drift detector 260 may be configured to output the concept drift notification if it determines that concept drift exists for a predetermined number of (consecutive) seasonal time pattern points of the smoothed current distance 620.

[0087] Further, in some embodiments, if concept drift is detected, the concept drift detector 260 may be configured to update the baseline model 606 based on a second time series data set. This second time series data set may include a time series data set acquired after the historical time series data 602 included in the time series data set 230. The baseline model generator 254 may use this second time series data set to retrain the ML model that becomes the baseline model 606 to align the baseline model 606 with characteristics of the current data and minimize or mitigate the detection of concept drift. Similarly, if concept drift is not detected, the concept drift detector 260 may be configured to update the set of historical time series data 602 and the baseline model 606 based on newly acquired time series data.

[0088] According to the embodiments described herein, an apparatus, method, and system for concept drift detection can be provided, which can determine the presence or absence of concept drift in seasonal time series data. It should be noted that unlike existing methods that use a single threshold to detect concept drift in stationary data, the concept drift detection process of the present disclosure determines baseline statistics for each set of seasonal time pattern points of baseline distance, which allows for the detection of concept drift in seasonal time series data.

[0089] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0090] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures with instructions recorded in grooves, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical digital signals transmitted through wires, per se.

[0091] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams that illustrate methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0092] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, and executed via the processor of the computer or other programmable data processing apparatus to provide a machine implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer readable storage medium having the instructions stored thereon becomes an article of manufacture including instructions implementing an aspect of the function / acts specified in a block or blocks of the flowcharts and / or block diagrams.

[0093] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, to generate a computer-implemented process.

[0094] Embodiments of the present disclosure may be provided to end users via a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as a computing capability that provides an abstraction between computing resources and their underlying technical architecture (e.g., servers, storage, networks), allowing convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly deployed and released with minimal administrative effort or service provider intervention. Thus, cloud computing allows users to access virtual computing resources (such as storage, data, applications, and even complete virtualized computing systems) in the "cloud" regardless of the underlying physical systems (or the location of those systems) used to provide the computing resources.

[0095] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may be executed in a different order than that described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special purpose hardware-based system that executes the specified functions or operations, or executes a combination of special purpose hardware and computer instructions.

[0096] Although the above is directed to exemplary embodiments, other / further embodiments of the present invention may be devised without departing from the basic scope of the present invention, the scope of the present disclosure being defined by the following claims. The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to allow those skilled in the art to easily understand the embodiments disclosed herein.

[0097] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the various embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly indicates otherwise. A "set," "group," "part," and the like are intended to include one or more. As used herein, the terms "comprise" and / or "may include" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof. In the foregoing detailed description of exemplary aspects of the various embodiments, reference is made to the accompanying drawings, which are a part of this specification and in which like reference numerals represent like elements, showing by way of example specific exemplary embodiments for practicing the various aspects. Reference has been made to the above-mentioned references. Although these embodiments have been described in sufficient detail to enable one skilled in the art to practice the embodiments, other embodiments may be used, and logical, mechanical, electrical changes, and the like, may be made without departing from the scope of the various embodiments. In the above description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. However, the various embodiments may be practiced without these specific details. Also, in other places, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the embodiments. [Explanation of symbols]

[0098] 100 Computer Systems 102 processors 104 Memory 106 Memory Bus 108 I / O bus 109 Bus IF 110 I / O bus interface 112 Terminal Interface 113 Storage Interface 114 I / O device interface 115 Network Interface 116 User I / O Device 117 Storage Device 124 Display System 126 Display device 130 Network 150 Concept Drift Detection Applications 200 Concept Drift Detection System 210 Client device 220 Data storage device 230 Time Series Datasets 240 Communication Network 250 Conceptual Drift Detection Device 252 Data Entry Section 254 Baseline model generation unit 256 Feature Extraction Unit 258 Distance calculation section 259 Distance smoothing section 260 Concept Drift Detection Unit

Claims

1. 1. A concept drift detection apparatus for detecting concept drift in a time series data set, comprising: The concept drift detection device comprises: a data input for receiving a time series data set comprising a set of historical time series data relating to a first time period and a set of current time series data relating to a second time period subsequent to the first time period; a baseline model generator for generating a baseline model based on a subset of the set of historical time series data; a feature extraction unit that divides the set of past time series data into a set of past windows, divides the set of current time series data into a set of current windows, divides the set of baseline data created by the baseline model into a set of baseline windows, and calculates a set of baseline data features by obtaining feature vectors from the set of baseline windows, calculates a set of past data features by obtaining feature vectors from the set of past windows, and calculates a set of current data features by obtaining feature vectors from the set of current windows; a distance calculation unit for calculating a baseline distance between feature vectors between a subset of the set of past data features and a subset of baseline data features associated with a corresponding time frame, and for calculating a current distance between feature vectors between a subset of the set of current data features and a subset of baseline data features associated with a corresponding time frame; a concept drift detection unit that calculates a baseline statistic indicating a criterion for determining concept drift based on the baseline distance, and determines whether or not there is a concept drift between the current time series data set and the past time series data set based on the baseline statistic and the current distance; A conceptual drift detection device comprising:

2. 2. The concept drift detection device of claim 1, further comprising a distance smoothing unit that smoothes the baseline distance and the current distance using an exponentially weighted moving average technique.

3. The baseline model is a trained machine learning model that generates a set of predictive time series data based on a subset of the set of historical time series data as the baseline data; 2. The conceptual drift detection apparatus of claim 1.

4. the time series data set is seasonal time series data comprising a set of seasonal time patterns that repeat periodically over a predetermined period of time; each seasonal time pattern of the set of seasonal time patterns includes a set of seasonal time pattern points corresponding to a set of time features; 2. The concept drift detection device according to claim 1 .

5. The concept drift detection unit calculating as said baseline statistic a seasonal baseline that indicates a basis for determining concept drift for each of said set of seasonal time pattern points of said baseline distance; determining that concept drift exists for the first seasonal time pattern point of the current distance if the first seasonal time pattern point of the current distance exceeds a statistical threshold with respect to the seasonal baseline; 5. The concept drift detection device according to claim 4.

6. The concept drift detection unit outputting a concept drift notification upon determining that concept drift exists for a predetermined number of seasonal time pattern points of the current distance; 6. The concept drift detection device according to claim 5.

7. The baseline model generating unit updating the baseline model based on a second time series data set if it is determined that concept drift exists for a predetermined number of seasonal time pattern points of the current distance; 7. The concept drift detection device according to claim 6.

8. A concept drift detection method for detecting concept drift in a time series data set in a concept drift detection device including a data input unit, a baseline model generation unit, a feature extraction unit, a distance calculation unit, and a concept drift detection unit, comprising: The concept drift detection method includes: said data input receiving a time series data set comprising a set of historical time series data relating to a first time period and a set of current time series data relating to a second time period subsequent to said first time period; generating a baseline model based on a subset of the set of historical time series data; said feature extraction unit dividing said set of historical time series data into a set of historical windows; the feature extraction unit dividing the current set of time series data into a set of current windows; said feature extractor dividing a set of baseline data produced by said baseline model into a set of baseline windows; the feature extractor computing a set of baseline data features by obtaining feature vectors from the set of baseline windows; the feature extraction unit calculating a set of past data features by obtaining feature vectors from the set of past windows; the feature extractor computing a set of current data features by obtaining feature vectors from the set of current windows; the distance calculation unit calculating baseline distances between feature vectors between a subset of the set of past data features and a subset of baseline data features associated with a corresponding time frame; said distance calculator calculating current distances between feature vectors between a subset of said set of current data features and a subset of baseline data features associated with a corresponding time frame; a step of the concept drift detection unit calculating a baseline statistic indicating a criterion for determining concept drift based on the baseline distance; determining whether there is a concept drift between the current set of time series data and the past set of time series data based on the baseline statistics and the current distance; A concept drift detection method comprising:

9. 1. A concept drift detection system for detecting concept drift in a time series dataset, comprising: The concept drift detection system comprises: A client device; a data storage device for storing a time series data set including a set of past time series data relating to a first time period and a set of current time series data relating to a second time period subsequent to the first time period; a concept drift detection device for detecting concept drift in the time series data set; Including, The concept drift detection device comprises: a data input for receiving the time series data set from a data storage device; a baseline model generator for generating a baseline model based on a subset of the set of historical time series data; a feature extraction unit that divides the set of past time series data into a set of past windows, divides the set of current time series data into a set of current windows, divides a set of baseline data created by the baseline model into a set of baseline windows, and calculates a set of baseline data features by obtaining feature vectors from the set of baseline windows, calculates a set of past data features by obtaining feature vectors from the set of past windows, and calculates a set of current data features from the set of current windows; a distance calculation unit for calculating a baseline distance between feature vectors between a subset of the set of past data features and a subset of baseline data features associated with a corresponding time frame, and for calculating a current distance between feature vectors between a subset of the set of current data features and a subset of baseline data features associated with a corresponding time frame; a concept drift detection unit that calculates a baseline statistic indicating a criterion for determining a concept drift based on the baseline distance, determines the presence or absence of a concept drift between the current time series data set and the past time series data set based on the baseline statistic and the current distance, and outputs a concept drift notification to the client device when it is determined that a concept drift exists between the current time series data set and the past time series data set; The concept drift detection system comprises:

Images