Outlier detection method, program, data processing device, and data processing system

The proposed method improves outlier detection accuracy by calculating and comparing ratios between data points in time-series data, specifically addressing the issue of human error-induced inaccuracies in existing methods, thereby enhancing the precision of data cleansing and prediction.

WO2024247102A1PCT designated stage expired Publication Date: 2024-12-05NT T INC

Patent Information

Application Number
PCT/JP2023/020086
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing outlier detection methods, such as the 3σ method and Hampel filter, inaccurately identify normal observed values as outliers due to human errors in manual data entry, leading to low accuracy in detecting abnormal values.

Method used

A method that calculates and compares ratios between consecutive data points in time-series data, excluding the first and last data points, to determine outliers using a predetermined threshold, thereby reducing the detection of incorrectly input normal values as outliers.

Benefits of technology

This approach enhances the accuracy of detecting abnormal values by minimizing the incorrect identification of normal values as outliers, allowing for more precise data cleansing and prediction in time-series analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023020086_05122024_PF_FP_ABST
    Figure JP2023020086_05122024_PF_FP_ABST
Patent Text Reader

Abstract

This outlier detection method comprises: a computation step for sequentially detecting outliers in data that excludes initial and final data from time-sequence data; and an outlier assessment step in which it is determined whether individual data values included in the time-sequence data are outliers on the basis of computation results from the computation step for detecting outliers. The computation step for detecting outliers includes: a ratio calculation step; and a comparison step in which values obtained in the ratio computation step are compared with prescribed threshold values. In the ratio calculation step, calculated are the values of a first ratio, which is the ratio between first data to be assessed and second data that is located immediately before the first data in time sequence, and a second ratio, which is the ratio between the first data and third data that is located immediately after the first data in time sequence.
Need to check novelty before this filing date? Find Prior Art

Description

Outlier detection method, program, data processing device, and data processing system

[0001] The present invention relates to an outlier detection method, a program, a data processing device, and a data processing system.

[0002] In various fields, observation data is periodically acquired over a certain period of time, the acquired data is recorded in a database in chronological order, the recorded time-series data is analyzed, and the future behavior of the observed subject is predicted based on the analysis results. For example, in a periodic health checkup, data representing measurements of the subject's height, weight, blood pressure, etc., as well as results of blood tests and electrocardiograms, are acquired, and the acquired time-series data is analyzed to predict the subject's future health condition.

[0003] For some reason, time series data may contain outliers that deviate significantly from the expected distribution of variables (for example, a normal distribution). For example, while an observed signal is being acquired over time, a noise signal with a momentary large value may be mixed in (hereinafter, outliers among actually observed values, such as noise signals, will be specifically referred to as "anomalous observed values," while actually observed values ​​that are not outliers will be referred to as "normal observed values").

[0004] In order to analyze recorded time series data and make accurate predictions, it is necessary to perform preprocessing to remove such anomalous observations in advance. Specifically, for each observed object and each data item, data values ​​that significantly deviate from the expected distribution are detected from the recorded time series data, the detected data are determined to be outlier data, and the determined outlier data are removed (data cleansing).

[0005] Known methods for detecting outlier data include the 3σ method, the Hampel filter method, and the moving Hampel filter method (Non-Patent Document 1). The 3σ method is a method for detecting, when a data series of variables assumed to follow a single normal distribution, values ​​that are three or more standard deviations (σ) away from the mean value of the data series as outliers. The Hampel filter method is a method that addresses the problem caused by the mean value and standard deviation being influenced by outliers in the 3σ method. The Hampel filter method sets a threshold value as the median of the data series ±1.4826 × median absolute deviation (MAD) × 3, and detects values ​​outside this threshold as outliers. The moving Hampel filter method is a method that uses a moving window on a time series for time series data in which variables are not assumed to always follow a single normal distribution. That is, a moving window is set that covers the time series data from a certain point in time to the past n points of data, and the Hampel filter method is applied to a set of n consecutive data points within the moving window. Next, the moving window is moved, and the Hampel filter method is applied to a new set of n consecutive data points. The Moving Hampel filter method is a method that repeats this process to detect outliers throughout the entire time series data.

[0006] “[Introduction to Signal Processing] No Machine Learning Required! Outlier Detection (Anomaly Detection) with Hampel Filter -Python-”, [online], October 9, 2019, [Retrieved April 18, 2023], Internet <URL: https: / / cpp-learning.com / hampel-filter / >

[0007] When recording time series data into a database, individual observation data is often manually entered into the database. Due to human error (incorrect input) inherent in such manual work, the recorded data values ​​may become outliers even though the actual observed values ​​were normal observation values. When applying the aforementioned outlier detection methods to such time series data, the incorrectly input normal observation values ​​(hereinafter referred to as "incorrectly input normal observation values") are also detected as outliers, resulting in low accuracy in detecting abnormal observation values.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide an outlier detection method, a program, a data processing device, and a data processing system that can more accurately detect abnormal observations among outliers.

[0009] According to a first aspect of the present invention, an outlier detection method includes a calculation step for detecting outliers sequentially for data excluding the first and last data in time series data, and an outlier determination step for determining whether or not a value of each data item included in the time series data is an outlier based on a calculation result of the calculation step for detecting outliers, wherein the calculation step for detecting outliers includes a ratio calculation step and a comparison step for comparing the value obtained in the ratio calculation step with a predetermined threshold, and the ratio calculation step calculates a first ratio which is the ratio between a first data item to be determined and a second data item located immediately before it in the time series, and a second ratio which is the ratio between the first data item and a third data item located immediately after it in the time series.

[0010] According to a second aspect of the present invention, the program causes a computer to execute a calculation step for detecting outliers, sequentially for data excluding the first and last data in time series data, and an outlier determination step for determining whether or not a value of each data item included in the time series data is an outlier based on the calculation result of the calculation step. The calculation step for detecting outliers includes a ratio calculation step and a comparison step for comparing the value obtained in the ratio calculation step with a predetermined threshold value. The ratio calculation step calculates a first ratio, which is the ratio between a first data item to be determined and a second data item located immediately before it in the time series, and a second ratio, which is the ratio between the first data item and a third data item located immediately after it in the time series.

[0011] According to a third aspect of the present invention, a data processing device includes a database that records time series data and an outlier detection unit, the outlier detection unit including a data acquisition unit, a calculation unit, and a judgment unit, the data acquisition unit acquires the recorded time series data from the database, the calculation unit performs a calculation for outlier detection on the time series data acquired by the data acquisition unit, and the judgment unit judges whether or not a value of each data included in the time series data is an outlier based on the calculation result of the calculation unit, the calculation for outlier detection includes a ratio calculation step and a comparison step that compares the value obtained in the ratio calculation step with a predetermined threshold, and the ratio calculation step calculates a first ratio which is the ratio between a first data item to be judged and a second data item located immediately before it in the time series, and a second ratio which is the ratio between the first data item and a third data item located immediately after it in the time series.

[0012] According to a fourth aspect of the present invention, a data processing system includes a database that records time series data and an outlier detection unit connected to the database via a network, the outlier detection unit including a data acquisition unit, a calculation unit, and a judgment unit, wherein the data acquisition unit acquires the recorded time series data from the database, the calculation unit performs a calculation for outlier detection on the time series data acquired by the data acquisition unit, and the judgment unit judges whether or not a value of each data included in the time series data is an outlier based on a calculation result of the calculation unit, the calculation for outlier detection includes a ratio calculation step and a comparison step that compares the value obtained in the ratio calculation step with a predetermined threshold, and the ratio calculation step calculates a first ratio which is a ratio between a first data item to be judged and a second data item located immediately before it in the time series, and a second ratio which is a ratio between the first data item and a third data item located immediately after it in the time series.

[0013] According to the present invention, it is possible to detect abnormal observed values ​​with higher accuracy.

[0014] FIG. 1 is a diagram showing the configuration of a data processing device according to a first embodiment of the present invention. FIG. 2 is a diagram showing the configuration of an outlier detection unit provided in the data processing device according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of time series data (health checkup data). FIG. 4 is a diagram showing the flow of an outlier detection method according to the first embodiment of the present invention. FIG. 5 is a diagram showing the configuration of a data processing system according to a second embodiment of the present invention. FIG. 6 is a diagram showing an example of the present invention. FIG. 7 is a diagram showing another example of time series data (attendance data).

[0015] The following describes embodiments of the present invention with reference to the drawings, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0016] First Embodiment A first embodiment of the present invention will be described below with reference to FIGS. 1A to 3. FIG.

[0017] [Data Processing Apparatus] As shown in FIG. 1A, the data processing apparatus 1 includes a database (DB) 10 and an outlier detection unit 11.

[0018] Time-series data is recorded in the database 10. The time-series data is, for example, health checkup data. The health checkup data is, for example, tabular data including an ID identifying the person undergoing the health checkup, the health checkup date indicating the date on which the health checkup was conducted, and test values ​​representing the results of each test obtained in the health checkup. The outlier detection unit 11 executes an outlier detection method (described below) according to this embodiment on the time-series data acquired from the database 10.

[0019] As shown in FIG. 1B , the outlier detection unit 11 includes a data acquisition unit 110, a calculation unit 111, a determination unit 112, an output unit 113, and a storage unit 114. The data acquisition unit 110 acquires recorded time-series data from the database 10. The calculation unit 111 performs a predetermined calculation (described below) for outlier detection on the time-series data acquired by the data acquisition unit 110. The determination unit 112 determines whether or not each data value included in the time-series data is an outlier based on the calculation result of the calculation unit 111. The output unit 113 outputs whether or not the time-series data includes outlier data, and if so, which data is the outlier data, based on the determination result of the determination unit 112. The storage unit 114 stores programs and parameters (e.g., threshold value a described below) used in the outlier detection method according to this embodiment.

[0020] [Outlier Detection Method] The outlier detection method according to this embodiment will be described below using health checkup data as an example of time-series data. Focus is placed on time-series data for one test item of a certain subject (hereinafter referred to as "outlier detection time-series data"). The outlier detection is considered to be performed from the outlier detection time-series data.

[0021] Figure 2 shows an example of time-series data for outlier detection. The time-series data for outlier detection is a set of the date of a health checkup and the test value of hemoglobin level, which is one of the blood test items in the health checkup. The table on the left of Figure 2 is a representation of the time-series data for outlier detection in tabular form. The graph on the left of Figure 2 is a representation of the time-series data for outlier detection in graphical form.

[0022] For ease of explanation, the outlier detection time-series data will be referred to as d. The outlier detection time-series data d is a list (list length: l) of test values ​​acquired at multiple discrete times (l points in total (l≧3). In the outlier detection time-series data illustrated in FIG. 2 , l=10) arranged in chronological order, and is represented by a one-dimensional array d[t] (t=1, 2, ..., l). d[i] is the ith test value from the beginning of the outlier detection time-series data d. Here, the test values ​​are natural numbers or positive real numbers.

[0023] 3 is a flow diagram showing the outlier detection method according to this embodiment. In the outlier detection unit 11, the data acquisition unit 110 first acquires recorded time-series data d[t] (t=1, . . . , l) from the database 10 (step S101).

[0024] Next, the following steps 102, 103, and 104 are repeated for each of i=2, . . . , l-1 in turn.

[0025] The calculation unit 111 calculates the values ​​of F1(i), F2(i), G1(i), and G2(i) given by the following equations (1) to (4) (step 102). F1(i) is the ratio of the data d[i] to be judged to the data d[i-1] located immediately before it in the time series. F2(i) is the ratio of the data d[i] to be judged to the data d[i+1] located immediately after it in the time series. G1(i) is the reciprocal of F1(i). G2(i) is the reciprocal of F2(i). F1(i)=d[i] / d[i-1]... (1) F2(i)=d[i] / d[i+1]... (2) G1(i)=d[i-1] / d[i]... (3) G2(i)=d[i+1] / d[i]... (4)

[0026] Next, the calculation unit 111 compares a predetermined threshold value a with each of the values ​​F1(i), F2(i), G1(i), and G2(i) (step 103).

[0027] Next, based on the result of the comparison made in the immediately preceding step 103, the determination unit 112 determines that d[i] is an outlier if the condition given by the following logical formula (5) is satisfied (formula (5) is true), and determines that d[i] is not an outlier if the condition is not satisfied (step 104): (F1(i)>a ∧ F2(i)>a) ∨ (G1(i)>a ∧ G2(i)>a) ... (5)

[0028] When step 104 is completed for i=l-1, the output unit 113 outputs i and d[i] as outliers if d[i] is determined to be an outlier for each of i=2, ..., l-1. If d[i] is not determined to be an outlier for any of i, a message indicating that there is no outlier is output (step 105).

[0029] The threshold value a is determined in advance so that the proportion of erroneously input normal observed values ​​detected among outliers is as large as possible. For example, the above-described outlier detection method is performed using test time-series data created by artificially inputting erroneous values ​​into some of the normal observed values ​​of given observation data, while trial-changing the value of a, and the proportion of erroneously input normal observed values ​​detected for each value of a is determined, and the optimal value of a is determined.

[0030] For example, if an incorrectly input normal observed value is caused by a one-digit shift when inputting a normal observed value (e.g., 1000 → 100, 12.3 → 123), setting a to approximately 8 to 9 makes it possible to detect data that deviates from the distribution in a spike-like manner.

[0031] [Program] The outlier detection method according to this embodiment is stored in the form of a computer-readable program in the storage unit 114. The outlier detection method according to this embodiment is performed by reading and executing this program and parameters (including the threshold value a) in the outlier detection unit 11. The program and parameters may be stored in the database 110 or in another external storage medium connected to the data processing device 1 by wire or wirelessly.

[0032] Second Embodiment A second embodiment of the present invention will be described below with reference to Fig. 4. Description of components common to the first embodiment will be omitted.

[0033] 4, the data processing system 2 includes a database (DB) 10 and an outlier detection unit 11. Unlike the first embodiment of the present invention, the database 10 and the outlier detection unit 11 are not provided in the same data processing device, but are connected via a network such as the Internet or a LAN.

[0034] This configuration allows the database 10 and the outlier detection unit 11 to be configured separately, thereby making it possible to miniaturize a device that includes only the outlier detection unit 11. Furthermore, an existing database can be used as the database 10.

[0035] <Example> Hereinafter, an example of the present invention will be described with reference to Figs.

[0036] The time series data (health check data) d[t] (t = 1, ..., 10) shown in Figure 2 are all normal observation values. Some of this time series data was recorded with incorrect data input, resulting in the time series data d'[t] (t = 1, ..., 10) shown in Figure 5. The incorrectly input data are d'[4] = 137 and d'[7] = 1.60 (the corresponding normal observation values ​​are d[4] = 13.7 and d[7] = 16.0, respectively).

[0037] In order to pick out the erroneous input normal observation values, the outlier detection method according to the first embodiment of the present invention was applied to the time series data d′[t] shown in FIG. 5. The value of the threshold a was set to 8.

[0038] As a result, for t=4, F1(4) = d'[4] / d'[3] = 9.320, F2(4) = d'[4] / d'[5] = 9.013, both of which are greater than the threshold a=8. Also, for t=7, G1(7) = d'[6] / d'[7] = 9.188, G2(7) = d'[8] / d'[7] = 9.5, both of which are greater than the threshold a=8. Therefore, the condition of equation (5) is satisfied for t=4 and 7. On the other hand, the condition of equation (5) is not satisfied for other t's.

[0039] As described above, by applying the outlier detection method according to the first embodiment of the present invention to the time series data d'[t] shown in Figure 5, it is possible to detect outliers d'[4] and d'[7].

[0040] In data processing using conventional methods such as the Hampel filter, even if the data is a normal observed value, it may be removed as an outlier. For example, in the time series data shown in Figure 5, if d'[2] (a normal observed value that is not erroneously input) is 16.3, using the Hampel filter or the like will remove not only d'[4] and d'[7], but also d'[2].

[0041] In contrast, if the outlier detection method according to the first embodiment of the present invention is used, even in the above case, it is possible to appropriately remove only d'[4] and d'[7]. In this way, the outlier detection method according to the first embodiment of the present invention reduces the risk of removing excessive normal observed values, making it possible to perform more accurate predictions based on a larger number of normal observed values.

[0042] While the preferred embodiments of the present invention have been described above with reference to the accompanying drawings, it goes without saying that the present invention is not limited to these examples. The shapes and combinations of the components shown in the above examples are merely examples, and various modifications can be made based on design requirements, etc., without departing from the spirit of the present invention.

[0043] For example, in the first embodiment of the present invention, medical checkup data (FIG. 2) is used as an example of time-series data, but the present invention is not limited to this example and may be, for example, attendance data showing the working hours of a worker for each working day, as shown in FIG. 5. Other time-series data that can be used include financial data showing whether stock prices are rising or falling, meteorological data such as temperature and rainfall, store sales, and the number of hits to a website, as well as various other data that fluctuate over time.

[0044] For example, in the first embodiment of the present invention, a digit misalignment is given as an example of an incorrect input, and the value of a corresponding to such an incorrect input is set to approximately 8 to 9. However, the value of a can be set appropriately depending on the degree of dispersion of the time series data and the severity of the change over time (whether it is stable or fluctuates wildly).

[0045] REFERENCE SIGNS LIST 1... data processing device 2... data processing system 10... database (DB) 11... outlier detection unit 110... data acquisition unit 111... calculation unit 112... determination unit 113... output unit 114... storage unit

Claims

1. An outlier detection method comprising: a calculation step for detecting outliers sequentially for time-series data excluding the first and last data; and an outlier determination step for determining whether or not a value of each data included in the time-series data is an outlier based on a calculation result of the calculation step for detecting outliers, wherein the calculation step for detecting outliers includes a ratio calculation step and a comparison step for comparing a value obtained in the ratio calculation step with a predetermined threshold, and the ratio calculation step calculates a first ratio which is a ratio between a first data item to be determined and a second data item located immediately before it in the time series, and a second ratio which is a ratio between the first data item and a third data item located immediately after it in the time series.

2. The outlier detection method according to claim 1, wherein in the comparison step, a comparison is made as to whether or not each of the first ratio and the second ratio is greater than a predetermined threshold value, and in the outlier determination step, if both the first ratio and the second ratio are greater than the predetermined threshold value, it is determined that the first data is an outlier.

3. The outlier detection method according to claim 1, wherein in the comparison step, a third ratio, which is the reciprocal of the first ratio, and a fourth ratio, which is the reciprocal of the second ratio, are compared to determine whether they are greater than a predetermined threshold value, and in the outlier determination step, if both the third ratio and the fourth ratio are greater than the predetermined threshold value, it is determined that the first data is an outlier.

4. A program for causing a computer to execute a calculation step for detecting outliers for time-series data excluding the first and last data, and an outlier determination step for determining whether or not a value of each data item included in the time-series data is an outlier based on a result of the calculation step, wherein the calculation step for detecting outliers includes a ratio calculation step and a comparison step for comparing a value obtained in the ratio calculation step with a predetermined threshold, and wherein the ratio calculation step calculates a first ratio which is a ratio between a first data item to be determined and a second data item located immediately before it on the time series, and a second ratio which is a ratio between the first data item and a third data item located immediately after it on the time series.

5. The program according to claim 4, wherein in the comparison step, a comparison is made as to whether each of the first ratio and the second ratio is greater than a predetermined threshold value, and in the outlier determination step, if both the first ratio and the second ratio are greater than the predetermined threshold value, the first data is determined to be an outlier.

6. The program according to claim 4, wherein in the comparison step, a third ratio, which is the reciprocal of the first ratio, and a fourth ratio, which is the reciprocal of the second ratio, are compared to determine whether they are greater than a predetermined threshold value, and in the outlier determination step, if both the third ratio and the fourth ratio are greater than the predetermined threshold value, the first data is determined to be an outlier.

7. A data processing device comprising: a database for recording time series data; and an outlier detection unit, the outlier detection unit comprising a data acquisition unit, a calculation unit, and a judgment unit, the data acquisition unit acquires the recorded time series data from the database, the calculation unit performs a calculation for outlier detection on the time series data acquired by the data acquisition unit, and the judgment unit judges whether or not a value of each data included in the time series data is an outlier based on a calculation result of the calculation unit, the calculation for outlier detection comprising a ratio calculation step and a comparison step for comparing a value obtained in the ratio calculation step with a predetermined threshold, and the ratio calculation step calculates each of a first ratio which is a ratio between a first data to be judged and a second data located immediately before it in the time series, and a second ratio which is a ratio between the first data and a third data located immediately after it in the time series.

8. A data processing system comprising: a database for recording time series data; and an outlier detection unit connected to the database via a network, the outlier detection unit comprising a data acquisition unit, a calculation unit, and a judgment unit, the data acquisition unit acquires the recorded time series data from the database, the calculation unit performs a calculation for outlier detection on the time series data acquired by the data acquisition unit, and the judgment unit judges whether or not a value of each data included in the time series data is an outlier based on a calculation result of the calculation unit, the calculation for outlier detection comprising a ratio calculation step and a comparison step for comparing a value obtained in the ratio calculation step with a predetermined threshold, and the ratio calculation step calculates each of a first ratio which is a ratio between a first data to be judged and a second data located immediately before it in the time series, and a second ratio which is a ratio between the first data and a third data located immediately after it in the time series.

Citation Information

Patent Citations

  • Vital information processing server, method for processing vital information, and program

    JP2015204038A

  • Segment display reader and method

    JP2019091202A

  • Systems and methods for modifying adaptive dosing regimens

    JP2021522575A

Cited By

  • A water pump data processing system and method based on tunnel inclined shaft

    CN119783006A

  • Tailing pond risk early warning and disposal whole-process collaborative management platform

    CN119886820A

  • Tailings pond risk warning and disposal full-process collaborative management platform

    CN119886820B