Data compression device, data compression method, and data compression program

Abstract

A data compression device (100) that compresses discrete time series data including a plurality of continuous values, the continuous values being constituted by one or more data points representing the same value that are consecutive in the time series, includes a noise determination section (11). When each continuous value included in the time series data is taken as a target continuous value, the noise determination section (11) determines the target continuous value to be noise in a case where the duration of the target continuous value is less than a specified duration and the difference between the value of the target continuous value and the value of the data point immediately preceding the start point of the target continuous value in the time series is equal to or less than a difference threshold. The noise determination section (11) determines the target continuous value to be not noise in a case where the duration of the target continuous value is equal to or more than the specified duration and the difference between the value of the target continuous value and the value of the data point immediately preceding the start point of the target continuous value in the time series is equal to or less than the difference threshold. The noise determination section (11) generates compressed data that is data in which each continuous value determined to be noise is not represented, each continuous value determined to be not noise is represented, and the time series data is compressed.

Description

Technical Field

[0001] This disclosure relates to a data compression apparatus, a data compression method, and a data compression program. Background Technology

[0002] There are techniques for compressing discrete time-series data containing many continuous values. Patent Document 1 discloses an example of such a technique.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2001-165712 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] According to the technology disclosed in Patent Document 1, time series data is compressed according to a predetermined time, that is, time series data is compressed without considering the length of the period during which the same value is consecutive. Furthermore, in this technology, when a value change occurs that is larger than the dead band width, data corresponding to the value change is preserved. Therefore, this technology has the following problem: when a value changes within a range within the dead band width within a predetermined time period, even if the value change is not considered to be caused by noise, sometimes the compressed time series data does not contain information indicating the value change.

[0008] Here, since machine learning uses massive amounts of data, it is preferable to maximize the data compression rate. Furthermore, machine learning emphasizes points of change. Therefore, even when using noise-reduced data for learning, the impact on accuracy is generally small. Thus, when compressing discrete time-series data containing many continuous values ​​for the purpose of generating data used in machine learning, it is preferable to remove noise while retaining information about the data within the continuous period of the same value, even when the value varies within a dead zone width.

[0009] The purpose of this disclosure is to provide a technique for compressing discrete time series data containing many continuous values, which, while eliminating noise, retains information representing the data within the continuous period of the same value, even when the value varies within a range of a difference threshold and the period of continuity of the same value is longer than a specified period.

[0010] Methods for solving problems

[0011] The data compression apparatus of the present invention compresses discrete time series data containing multiple continuous values, wherein the continuous values ​​are composed of one or more data points representing the same value consecutively in the time series. The data compression apparatus includes a noise determination unit. When the continuous values ​​included in the time series data are assumed to be object continuous values, if the duration of the object continuous value is less than a specified period and the difference between the value of the object continuous value and the value of a data point immediately preceding the start point of the object continuous value in the time series is less than a difference threshold, the noise determination unit determines that the object continuous value is noise. If the duration of the object continuous value is greater than or equal to the specified period and the difference between the value of the object continuous value and the value of a data point immediately preceding the start point of the object continuous value in the time series is less than the difference threshold, the noise determination unit determines that the object continuous value is not noise. The noise determination unit generates compressed data, which is data that does not represent the continuous values ​​determined to be noise and data that represents the continuous values ​​determined not to be noise, and is data obtained by compressing the time series data.

[0012] Invention Effects

[0013] According to this disclosure, if the duration of a continuous value is less than a specified period and the difference between the value of the continuous value and the value of the data point immediately preceding the start point of the continuous value in the time series is less than a difference threshold, the noise determination unit determines the continuous value to be noise. Conversely, if the duration of a continuous value is more than or equal to the specified period and the difference between the value of the continuous value and the value of the data point immediately preceding the start point of the continuous value in the time series is less than a difference threshold, the noise determination unit determines the continuous value to be non-noise. Furthermore, the noise determination unit generates compressed data, which contains data representing each continuous value determined not to be noise and data representing each continuous value determined not to be noise, and is data obtained by compressing time series data. Here, the time series data is discrete data containing multiple continuous values.

[0014] Therefore, according to this disclosure, in the technique for compressing discrete time series data containing a large number of continuous values, it is possible to eliminate noise while retaining information representing the data during the continuous period of the same value, even when the value varies within a range within a difference threshold and the period of the same value being continuous is longer than a specified period. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating a structural example of the data compression system 90 in Embodiment 1.

[0016] Figure 2This is a diagram illustrating a structural example of the data compression system 90 in Embodiment 1.

[0017] Figure 3 This is a diagram illustrating the processing of the noise determination unit 11 in Embodiment 1.

[0018] Figure 4 The diagrams illustrate the processing of the noise determination unit 11 in Embodiment 1, (a) is a diagram illustrating the stable state and the transition state, and (b) is a diagram representing the function corresponding to the performance of the air conditioner.

[0019] Figure 5 This is a diagram illustrating the processing of the noise determination unit 11 in Embodiment 1.

[0020] Figure 6 This is a diagram illustrating an example of the hardware structure of the data compression device 100 in Embodiment 1.

[0021] Figure 7 This is a flowchart illustrating the processing of the data compression system 90 in Implementation Method 1.

[0022] Figure 8 This is a flowchart illustrating the processing of the noise determination unit 11 in Embodiment 1.

[0023] Figure 9 This is a flowchart illustrating the processing of the noise determination unit 11 in Embodiment 1.

[0024] Figure 10 This is a diagram illustrating a hardware structure example of the data compression device 100, a variation of Embodiment 1.

[0025] Figure 11 This is a diagram illustrating a structural example of the data compression system 90 in Embodiment 2.

[0026] Figure 12 This is a diagram illustrating a structural example of the statistical selection unit 30 in Implementation Method 2.

[0027] Figure 13 This is a diagram illustrating the specified period 12 and the differential threshold 13 of Implementation Method 2.

[0028] Figure 14 This is a flowchart illustrating the processing of the data compression system 90 in Implementation Method 2.

[0029] Figure 15 This is a flowchart illustrating the processing of the statistical selection unit 30 in Implementation Method 2.

[0030] Figure 16 This is a flowchart illustrating the processing of the specified period calculation unit 32 in Embodiment 2.

[0031] Figure 17This is a flowchart illustrating the processing of the differential threshold calculation unit 34 in Embodiment 2.

[0032] Figure 18 This is a diagram illustrating a structural example of the data compression system 90 in Embodiment 3.

[0033] Figure 19 This is a diagram illustrating a structural example of the parameter determination unit 40 in Embodiment 3.

[0034] Figure 20 This is a diagram illustrating a structural example of the statistical selection unit 30 in Implementation Method 3.

[0035] Figure 21 This is a diagram illustrating the processing of the data compression system 90 in Embodiment 3.

[0036] Figure 22 This is a flowchart illustrating the processing of the data compression system 90 in Implementation Method 3.

[0037] Figure 23 This is a flowchart illustrating the processing of the data compression system 90 in Implementation Method 3.

[0038] Figure 24 This is a flowchart illustrating the processing of the statistical selection unit 30 in Implementation Method 3.

[0039] Figure 25 This is a flowchart illustrating the processing of the parameter determination unit 40 in Embodiment 3.

[0040] Figure 26 This is a flowchart illustrating the processing of the parameter determination unit 40 in Embodiment 3. Detailed Implementation

[0041] In the description and accompanying drawings of the embodiments, the same reference numerals are used to label the same elements and corresponding elements. The descriptions of elements labeled with the same reference numerals are appropriately omitted or simplified. The arrows in the figures mainly indicate data flow or processing flow. In addition, "part" or "device" may be appropriately replaced with "system", "circuit", "process", "sequence", "processing" or "line".

[0042] Implementation method 1.

[0043] Hereinafter, this embodiment will be described in detail with reference to the accompanying drawings.

[0044] In this embodiment, the objective is to balance reducing data volume by removing noise that has little or no impact on machine learning, and ensuring that the accuracy of a machine learning model based on noise-removed data is equal to the accuracy of a machine learning model using the original data, in order to generate data for use in the learning process of machine learning. Furthermore, in this embodiment, the objective is to reduce the data volume by removing weak noise from the time series data and then compressing it to lower the storage costs of the time series data. The time series data in this embodiment consists of discrete values ​​and contains multiple continuous values. That is, the time series data is discrete. Continuous values ​​consist of one or more consecutive data points representing the same value in the time series.

[0045] As a specific example, continuous values ​​are becoming more prevalent in IoT (Internet of Things) use cases. This is primarily due to factors such as low sensor sensitivity (resolution) or intentional rounding to reduce data volume. Furthermore, outliers are sometimes also represented as continuous values. The value of an outlier differs from the value of any data point adjacent to it.

[0046] Structural description

[0047] Figure 1 and Figure 2 This illustrates a structural example of the data compression system 90 in this embodiment.

[0048] The data compression device 100 compresses time-series data. The data compression device 100 can, for example... Figure 1 As shown, this can be achieved through cloud system 1, or as follows: Figure 2 The data compression device 100 and the data recovery device 200 can also be integrated into one unit.

[0049] The following describes a specific example of how the data compression device 100 is implemented through the cloud system 1.

[0050] The case where the data compression device 100 is implemented by the edge system 19 is the same as the case where the data compression device 100 is implemented by the cloud system 1. The recovery method 22 is data representing a method for recovering the compressed data 14.

[0051] The time-series data 5 acquired by each sensor 2 is transmitted to the cloud system 1 via network 6. The time-series data 5 records the sensor values ​​acquired by each sensor 2 in a time sequence. The time-series data 5 is equivalent to the raw data.

[0052] The cloud system 1 includes a data receiving unit 7, a data compression device 100, a database 15, and a data recovery device 200.

[0053] The data receiving unit 7 receives time series data 5.

[0054] The data compression device 100 includes an encoding unit 9 and a noise determination unit 11.

[0055] The encoding unit 9 receives time series data 5 from the data receiving unit 7 and generates encoded data 10 by encoding the received time series data 5.

[0056] As a specific example, the encoding unit 9 generates encoded data 10 using differential encoding. Differential encoding is a method of compressing discrete time-series data based on the difference between adjacent data points. In differential encoding, each data point is preserved if the difference between it and the immediately preceding data point in the time series is not zero, and is deleted if the difference between it and the immediately preceding data point in the time series is zero.

[0057] The noise determination unit 11 determines whether a portion of the data points in the encoded data 10 are noise based on a specified period 12 and a differential threshold 13. The data points determined to be noise are deleted from the encoded data 10 to generate compressed data 14. The noise determination unit 11 can also generate compressed data 14 using differential encoding. The noise determination unit 11 stores the generated compressed data 14 in the database 15. Compressed data 14 does not represent consecutive values ​​that the noise determination unit 11 determines are noise, but rather represents consecutive values ​​that the noise determination unit 11 determines are not noise. It is data obtained by compressing the time series data 5. Compressed data 14 can also be data obtained by further compressing the encoded data 10.

[0058] In this embodiment, a specified period 12 and a difference threshold 13 are introduced to determine whether each data point is noise. The specified period 12 and the difference threshold 13 can also be values ​​calculated based on domain knowledge. Furthermore, the noise determination unit 11 can use time series data 5 instead of the encoded data 10. The compressed data 14 is equivalent to data representing the features extracted from the original data.

[0059] If the duration of a certain continuous value is shorter than the specified period 12, the noise determination unit 11 determines that the continuous value as noise. However, even if a continuous value is determined to be noise based on the specified period 12, the noise determination unit 11 will not determine that continuous value as noise if the difference (variation) between the value of the continuous value and the value of the data point immediately preceding the start point of the continuous value in the time series is greater than the difference threshold 13.

[0060] When setting each continuous value contained in the time series data 5 as an object continuous value, if the duration of the object continuous value is less than a specified period 12 and the difference between the value of the object continuous value and the value of the data point immediately preceding the start point of the object continuous value in the time series is less than a difference threshold 13, the noise determination unit 11 determines the object continuous value to be noise. Conversely, if the duration of the object continuous value is greater than or equal to the specified period 12 and the difference between the value of the object continuous value and the value of the data point immediately preceding the start point of the object continuous value in the time series is less than a difference threshold 13, the noise determination unit 11 determines the object continuous value to be non-noise. Furthermore, if the difference between the value of the object continuous value and the value of the data point immediately preceding the start point of the object continuous value in the time series is greater than a difference threshold 13, the noise determination unit 11 determines the object continuous value to be non-noise.

[0061] Figure 3 This is a diagram illustrating a specific example of the processing by the noise determination unit 11. Figure 3 In the specified period 12, the value is 3, and the differential threshold 13 is 0.5. Figure 3 The original data and the saved data are shown in Figure 5. The original data consists of the data points shown in time series data 5. The saved data consists of the data points from the data points shown in time series data 5 that are saved in the compressed data 14, and are determined based on differential coding.

[0062] The duration of data D1 is 0, meaning it is less than the specified period 12. Furthermore, the difference between the value of data D1 and the value of the continuous value S1 is 0.5, meaning it is below the difference threshold 13. Here, the value of the continuous value S1 is the value of the data point immediately preceding data D1 in the time series. Data D1 is also a continuous value. Therefore, the noise determination unit 11 determines data D1 as noise and deletes data D1 from the encoded data 10. Furthermore, in Figure 3 In the example shown, the value at time t1 is 1.5 in time series data 5 and encoded data 10, but the value at time t1 is 1 in the data obtained by restoring the compressed data 14. That is, in compressed data 14, data D1 is actually treated as data D1'.

[0063] Here, the duration of each continuous value S1 and continuous value S2 is 2. However, since data D1 is processed as data D1', continuous value S1, data D1', and continuous value S2 are processed together as continuous value S1'. The duration of continuous value S1' is 6, which is a specified period of 12 or more. Therefore, the noise determination unit 11 does not determine continuous value S1' as noise. Furthermore, the noise determination unit 11 may also consider continuous value S1 and continuous value S2 as a single continuous data point.

[0064] The duration of the continuous value S3 is 3, which is more than or equal to the specified period 12. Therefore, the noise determination unit 11 does not determine the continuous value S3 as noise.

[0065] The duration of continuous value S4 is 2 seconds. However, the difference between the value of continuous value S4 and the value of continuous value S3 is greater than the difference threshold 13. Therefore, the noise determination unit 11 does not determine continuous value S4 as noise. Similarly, the noise determination unit 11 does not determine continuous values ​​S5 and S6 as noise.

[0066] Database 15 is the database that stores the compressed data 14 generated by the storage.

[0067] The data recovery device 200 includes a recovery section 17.

[0068] The restoration unit 17 retrieves the compressed data 14 from the database 15 and uses the retrieved compressed data 14 as the restoration data 18 to restore it into time series data. At this time, the restoration unit 17 uses a decoding algorithm corresponding to the algorithm used to encode the compressed data 14.

[0069] use Figure 4 This section explains an example of calculating a specified period 12 and a difference threshold 13 when time series data 5 is a series of room temperature data from an air conditioner. Here, the specified period 12 and the difference threshold 13 are calculated based on domain knowledge. Furthermore, in... Figure 4 In the example shown, the purpose is to detect an abnormal state based on assumptions such as the performance of the air conditioner and the nature of the room where the air conditioner is installed.

[0070] When an air conditioner is running, the room temperature goes through a transitional state before reaching a steady state. The transitional state is when the room temperature rises or falls towards the set temperature. The steady state is when the room temperature remains stationary at the set temperature.

[0071] Then, as Figure 4 As shown in (a), after the air conditioning stops due to reaching a steady state, the room temperature gradually decreases or increases over z minutes. As a result, if the set temperature deviates from the room temperature threshold, the air conditioner switches back to a transition state. Here, time x i+z With time x i The difference between them is z minutes. In addition, the specified period 12 is set to 10 minutes. When this threshold is set to α, the range of change of room temperature naturally, i.e., without any environmental factor, decreasing or increasing can be defined as "|(set temperature) - (room temperature)| < α".

[0072] When α is set to the room temperature that can return to the set temperature within a specified period of 12 based on the air conditioner's capability, it becomes "α≤ (differential threshold 13)".

[0073] If the room temperature value changes within a specified period 12, exceeding the differential threshold 13, it is considered that some environmental factor such as ventilation or human entry and exit has occurred, and compressed data 14 is generated by saving the changed room temperature value.

[0074] Furthermore, when setting the differential threshold to 13, based on the air conditioner's capacity (for example, 2.8kW for cooling and 3.6kW for heating) and the size of the room where the air conditioner is installed, it is sufficient to know "how many minutes it takes for the air conditioner to raise or lower the room temperature by x degrees". Here, if the change in room temperature y can be defined as a function y=f(x) of the air conditioner's performance f and time x, then... Figure 4 As shown in (b), the maximum variation per unit time can be determined. Figure 4 In (b), (x) i+1 -x i ) corresponds to a unit of time, "|f(x) i )-f(x i+1 "|" represents the maximum room temperature variation assumed based solely on the performance of the air conditioner per unit time.

[0075] Here, with the temperature sensor measuring room temperature in units of 0.5 degrees, and considering time x... i ~time x i+1 The corresponding difference threshold 13 is shown in [Mathematical Formula 1]. As a specific example, when the room temperature changes at a certain time point as f(x1=1)=15.5 and f(x2=2)=18.5, the difference threshold 13 based on domain knowledge is shown in [Mathematical Formula 2].

[0076] [Mathematical Expression 1]

[0077] [Mathematical Expression 2]

[0078] As a use case other than air conditioners, such as Figure 5 As shown, data from vibration sensors used to detect tool wear on machine tools installed in factories and other similar locations can be cited as examples.

[0079] Here, because the vibration waveform changes periodically, therefore, as Figure 5 As shown on the left, it does not include continuous values. However, when transforming the vibration waveform using Fourier transform or similar methods, such as... Figure 5 As shown on the right, sometimes data containing a large number of certain continuous values ​​may appear. By removing weak noise from such data based on a specified period 12 and a difference threshold 13 and then compressing it, the data volume can be reduced. In addition, information representing normal and abnormal waveforms is retained separately in the compressed data 14.

[0080] Figure 6 An example of the hardware structure of the data compression apparatus 100 according to this embodiment is shown. The data compression apparatus 100 is configured as a computer. The data compression apparatus 100 may also be configured as multiple computers.

[0081] As shown in this figure, the data compression device 100 is a computer equipped with hardware such as a processor 51, a memory 52, an auxiliary storage device 53, an input / output interface 54, and a communication device 55. These hardware components are appropriately connected via signal lines 59.

[0082] Processor 51 is an IC (Integrated Circuit) that performs computational processing and controls the hardware of the computer. As a specific example, processor 51 is a CPU (Central Processing Unit), DSP (Digital Signal Processor), or GPU (Graphics Processing Unit).

[0083] The data compression device 100 may also have multiple processors to replace the processor 51. The multiple processors share the functions of the processor 51.

[0084] Memory 52 is typically a volatile storage device, specifically RAM (Random Access Memory). Memory 52 is also referred to as main storage or main memory. Data stored in memory 52 is stored in auxiliary storage device 53 as needed.

[0085] The auxiliary storage device 53 is typically a non-volatile storage device, such as ROM (Read Only Memory), HDD (Hard Disk Drive), or flash memory. Data stored in the auxiliary storage device 53 is loaded into the memory 52 as needed.

[0086] The memory 52 and the auxiliary storage device 53 can also be configured as a single unit.

[0087] The IF54 input / output port is used to connect input and output devices. For example, the IF54 input / output port is a USB (Universal Serial Bus) terminal. For example, input devices include a keyboard and mouse. For example, an output device is a monitor.

[0088] The communication device 55 is a receiver and a transmitter. As a specific example, the communication device 55 is a communication chip or a NIC (Network Interface Card).

[0089] When communicating with other devices, the various parts of the data compression device 100 may also appropriately use the input / output IF 54 and the communication device 55.

[0090] Auxiliary storage device 53 stores a data compression program. The data compression program is a program that enables the computer to perform the functions of each component of the data compression device 100. The data compression program is loaded into memory 52 and executed by processor 51. The functions of each component of the data compression device 100 are implemented through software.

[0091] Data used during the execution of the data compression program and data obtained by executing the data compression program are appropriately stored in a storage device. Each part of the data compression apparatus 100 appropriately utilizes a storage device. As a specific example, the storage device comprises at least one of a memory 52, an auxiliary storage device 53, a register within the processor 51, and a cache memory within the processor 51. Furthermore, the terms "data" and "information" sometimes have the same meaning. The storage device may also be independent of the computer.

[0092] The functions of memory 52 and auxiliary storage device 53 can also be implemented by other storage devices.

[0093] Data compression programs can also be recorded on non-volatile recording media that can be read by a computer. Specific examples of non-volatile recording media include optical discs or flash memory. Data compression programs can also be provided as program products.

[0094] Description of the action

[0095] The sequence of actions of the data compression system 90 is equivalent to a data compression method. Furthermore, the program that implements the actions of the data compression system 90 is equivalent to a data compression program.

[0096] Figure 7 This is a flowchart illustrating an example of the processing by the data compression device 100. (Using...) Figure 7 Explain the processing of the data compression device 100.

[0097] (Step S101)

[0098] The data receiving unit 7 receives the time series data 5 and sends the received time series data 5 to the encoding unit 9.

[0099] (Step S102)

[0100] The encoding unit 9 receives time series data 5 from the data receiving unit 7, generates encoded data 10 by encoding the received time series data 5, and sends the generated encoded data 10 to the noise determination unit 11.

[0101] (Step S103)

[0102] Input the specified period 12 into the noise determination unit 11.

[0103] (Step S104)

[0104] Input the differential threshold 13 into the noise determination unit 11.

[0105] (Step S105)

[0106] The noise determination unit 11 receives the encoded data 10 and uses the received encoded data 10, the input specified period 12, and the difference threshold 13 to generate compressed data 14. At this time, the noise determination unit 11 detects continuous values ​​whose duration is shorter than the specified period 12, and deletes each continuous value from the encoded data 10 whose difference between the value of the continuous value and the value of the data point immediately preceding the start point of the continuous value in the time series is less than the difference threshold 13.

[0107] (Step S106)

[0108] The noise determination unit 11 outputs the generated compressed data 14.

[0109] Figure 8 and Figure 9 This is a flowchart illustrating an example of the processing by the noise determination unit 11. Using Figure 8 and Figure 9 Explain the processing of the noise determination unit 11.

[0110] (Step S121)

[0111] Input the encoded data 10 into the noise determination unit 11.

[0112] (Step S122)

[0113] Input the specified period 12 into the noise determination unit 11.

[0114] (Step S123)

[0115] Input the differential threshold 13 into the noise determination unit 11.

[0116] (Step S124)

[0117] The noise determination unit 11 sets the constant P to the value shown in the specified period 12, sets the constant D to the value shown in the differential threshold 13, and sets the encoded data 10 to (X, Y). Here, X[k] (1 ≤ k ≤ n) represents the k-th moment in the time series of the time series data 5, and Y[k] represents the value of the time series data 5 at the k-th moment in the time series. n is the maximum value of the subscripts of the time series data 5. E[k] (0 ≤ E[k] ≤ n) represents the subscript of X corresponding to the k-th data point of the encoded data 10 in the time series.

[0118] (Step S125)

[0119] The noise determination unit 11 substitutes 1 for the variable i and sets E’ to an empty set. The E determined to be noise in the encoded data 10 is sequentially stored in E’.

[0120] (Step S126)

[0121] When i < n is satisfied, the noise determination unit 11 proceeds to step S127. Otherwise, the noise determination unit 11 proceeds to step S131.

[0122] (Step S127)

[0123] When X[E[i + 1] - 1] - X[E[i]] < P is satisfied, the noise determination unit 11 proceeds to step S128. Otherwise, the noise determination unit 11 proceeds to step S130.

[0124] (Step S128)

[0125] When |Y[E[i + 1] - 1] - Y[E[i]]| ≤ D is satisfied, the noise determination unit 11 proceeds to step S129. Otherwise, the noise determination unit 11 proceeds to step S130.

[0126] (Step S129)

[0127] The noise determination unit 11 appends E[i] to E’.

[0128] (Step S130)

[0129] The noise determination unit 11 increments the value of the variable i by 1.

[0130] (Step S131)

[0131] The noise determination unit 11 sets E” to the difference set of E and E’. Here, A\B represents the difference set obtained by subtracting set B from set A.

[0132] (Step S132)

[0133] The noise determination unit 11 sets the compressed data 14 as (X[E”], Y[E”]).

[0134] (Step S133)

[0135] The noise determination unit 11 outputs the generated compressed data 14.

[0136] Explanation of the effects of Implementation Method 1

[0137] As described above, according to this embodiment, when the time series data contains a large number of continuous values, it does not save each data point that has passed a specified time as shown in Patent Document 1. Therefore, it is possible to further compress the time series data.

[0138] Furthermore, according to this embodiment, even when there is a variation below the difference threshold 13, if the continuous value following the variation persists for a specified period of 12 or more, the information representing that continuous value is retained in the compressed data 14. Therefore, important points of change in machine learning are not discarded. In machine learning, values ​​with larger variations (i.e., values ​​with larger variances) tend to have a greater impact on prediction. This embodiment actively retains points with large variations and only removes noise that has little or no impact on the accuracy of the machine learning model. Therefore, according to this embodiment, accuracy can be maintained while reducing the amount of data and lowering data storage costs.

[0139] In the restoration of the compressed data 14, a conventional differential encoding restoration algorithm is used as a specific example. Therefore, no special data restoration device is required in this embodiment. Furthermore, the setting values ​​of the specified period 12 and the differential threshold 13 do not affect the data restoration; therefore, the specified period 12 and the differential threshold 13 may not be retained as metadata.

[0140] Furthermore, the process of generating compressed data 14 sometimes achieves the same effect as preprocessing in machine learning. Therefore, based on the machine learning model (learning algorithm), improved accuracy can be expected.

[0141] Other structures

[0142] <Variation Example 1>

[0143] Figure 10 This describes a hardware structure example of the data compression device 100 in this modified example.

[0144] The data compression device 100 includes a processing circuit 58, which replaces the processor 51, the processor 51 and the memory 52, the processor 51 and the auxiliary storage device 53, or the processor 51, the memory 52 and the auxiliary storage device 53.

[0145] The processing circuit 58 is hardware that implements at least a portion of the components of the data compression device 100.

[0146] The processing circuit 58 can be dedicated hardware, or it can be a processor that executes the program stored in the memory 52.

[0147] In the case where the processing circuit 58 is dedicated hardware, as a specific example, the processing circuit 58 is a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

[0148] The data compression device 100 may also have multiple processing circuits that replace the processing circuit 58. The multiple processing circuits share the functions of the processing circuit 58.

[0149] In the data compression device 100, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0150] As a specific example, the processing circuit 58 is implemented by hardware, software, firmware, or a combination thereof.

[0151] The processor 51, memory 52, auxiliary storage device 53, and processing circuit 58 are collectively referred to as the "processing circuit". That is, the functions of each functional component of the data compression device 100 are realized through the processing circuit.

[0152] The data compression device 100 in other embodiments may also have the same structure as this modified example.

[0153] Implementation method 2.

[0154] Hereinafter, the differences from the embodiments described above will be explained with reference to the accompanying drawings.

[0155] Structural description

[0156] Figure 11 This illustrates a structural example of the data compression system 90 in this embodiment.

[0157] The data compression apparatus 100 of this embodiment also includes a statistics selection unit 30, a specified period calculation unit 32, and a difference threshold calculation unit 34. Furthermore, the data compression apparatus 100 can also be implemented using an edge system 19.

[0158] Figure 12 The structure of the statistic selection unit 30 is shown as an example. The statistic selection unit 30 appropriately selects statistic 31 and statistic 33 from the statistic list 301.

[0159] Statistic 31 is the statistic used when calculating a specified period 12.

[0160] Statistic 33 is the statistic used when calculating the difference threshold 13.

[0161] Furthermore, statistics 31 and 33 can be different statistics from each other.

[0162] List of statistics 301 is a list representing various statistics. As a specific example, list of statistics 301 consists of data representing the "arithmetic mean", "median", "most frequent value", "truncated mean", and "weighted average", respectively.

[0163] The specified period calculation unit 32 calculates the specified period 12 based on the duration of each continuous value contained in each time series data 5. Specifically, the specified period calculation unit 32 uses the statistic 31, past time series data 35, and time series data 5 as inputs to calculate the specified period 12. The past time series data 35 is data restored from the compressed data 14 generated in the past. The past time series data 35 can also be time series data 5 obtained in the past.

[0164] As a specific example, when the specified period 12 is determined by the arithmetic mean, the specified period calculation unit 32 calculates the specified period 12 by [mathematical formula 3]. Figure 13 When the time series data 5 shown is the input, the specified period calculation unit 32 calculates the specified period 12 as shown in [Mathematical Formula 4].

[0165] [Mathematical Expression 3]

[0166] [Mathematical Expression 4]

[0167] The difference threshold calculation unit 34 calculates the difference threshold 13 based on the difference between the value of each continuous value contained in each time series data 5 and the value of the data point immediately preceding the start point of each continuous value in the time series. Specifically, the difference threshold calculation unit 34 calculates the difference threshold 13 by taking the statistic 33, the past time series data 35, and the time series data 5 as inputs.

[0168] As a specific example, when the difference threshold 13 is determined by the arithmetic mean, the difference threshold calculation unit 34 calculates the difference threshold 13 using [Mathematical Formula 5]. Figure 13 When the time series data 5 shown is the input, the calculation unit 32 for the specified period calculates the difference threshold 13 as shown in [Mathematical Formula 6].

[0169] [Mathematical Expression 5]

[0170] [Mathematical Expression 6] Description of the action

[0171] Figure 14 This is a flowchart illustrating an example of the processing of a data compression system 90. (Using...) Figure 14 Explain the processing of data compression system 90.

[0172] (Step S201)

[0173] In the statistics selection section 30, statistics 31 and 33 are selected.

[0174] (Step S202)

[0175] Time series data 5 and past time series data 35 are input to the specified period calculation unit 32 and the difference threshold calculation unit 34, respectively.

[0176] (Step S203)

[0177] The specified period calculation unit 32 calculates the specified period 12.

[0178] (Step S204)

[0179] The differential threshold calculation unit 34 calculates the differential threshold 13.

[0180] Figure 15 This is a flowchart illustrating an example of the processing in the statistics selection unit 30. (Using...) Figure 15 Explain the processing of the statistical selection section 30.

[0181] (Step S221)

[0182] Enter the list of statistics 301 into the statistics selection section 30.

[0183] (Step S222)

[0184] The statistic selection unit 30 selects the statistic to be used when calculating the specified period 12 from the statistics shown in the statistics list 301 as statistic 31.

[0185] (Step S223)

[0186] The statistic selection unit 30 outputs the selected statistic 31 to the specified period calculation unit 32.

[0187] (Step S224)

[0188] The statistic selection unit 30 selects the statistic used in calculating the difference threshold 13 from the statistics shown in the statistic list 301 as statistic 33.

[0189] (Step S225)

[0190] The statistic selection unit 30 outputs the selected statistic 33 to the difference threshold calculation unit 34.

[0191] Figure 16 This is a flowchart illustrating an example of the processing by the calculation unit 32 during a specified period. (Using...) Figure 16 Explain the processing of the calculation unit 32 during the specified period.

[0192] (Step S241)

[0193] Input the statistic 31 into the calculation unit 32 for the specified period.

[0194] (Step S242)

[0195] Input time series data 5 into the calculation unit 32 for the specified period.

[0196] (Step S243)

[0197] Input past time series data 35 into the calculation unit 32 for the specified period.

[0198] (Step S244)

[0199] During the specified period, the calculation unit 32 substitutes the time series data 5 into the variable Seq.

[0200] (Step S245)

[0201] If past time series data 35 exists, the specified period calculation unit 32 proceeds to step S246. Otherwise, the specified period calculation unit 32 proceeds to step S247.

[0202] (Step S246)

[0203] During the specified period, the calculation unit 32 appends past time series data 35 to the variable Seq.

[0204] (Step S247)

[0205] The calculation unit 32 extracts n consecutive values ​​from the time series data shown by the variable Seq as period[i] (1≤i≤n).

[0206] (Step S248)

[0207] The specified period calculation unit 32 calculates the specified period 12 based on the input statistics 31.

[0208] (Step S249)

[0209] The specified period calculation unit 32 outputs the calculated specified period 12 to the noise determination unit 11.

[0210] Figure 17 This is a flowchart illustrating an example of the processing by the difference threshold calculation unit 34. Using Figure 17 Explain the processing of the differential threshold calculation unit 34.

[0211] (Step S261)

[0212] Input the statistic 33 into the differential threshold calculation unit 34.

[0213] (Step S262)

[0214] Input time series data 5 into the differential threshold calculation unit 34.

[0215] (Step S263)

[0216] Input past time series data 35 into the differential threshold calculation unit 34.

[0217] (Step S264)

[0218] The difference threshold calculation unit 34 substitutes the time series data 5 into the variable Seq.

[0219] (Step S265)

[0220] If past time series data 35 is available, the difference threshold calculation unit 34 proceeds to step S266. Otherwise, the difference threshold calculation unit 34 proceeds to step S267.

[0221] (Step S266)

[0222] The difference threshold calculation unit 34 appends past time series data 35 to the variable Seq.

[0223] (Step S267)

[0224] The difference threshold calculation unit 34 extracts the difference of m data points from the time series data shown by variable Seq as diff[i] (1≤i≤m).

[0225] (Step S268)

[0226] The differential threshold calculation unit 34 calculates the differential threshold 13 based on the input statistic 33.

[0227] (Step S269)

[0228] The differential threshold calculation unit 34 outputs the calculated differential threshold 13 to the noise determination unit 11.

[0229] Explanation of the effects of Implementation Method 2

[0230] As described above, according to this embodiment, a specified period 12 and a difference threshold 13 are calculated based on time series data and selected statistics. Therefore, it is not necessary to determine the specified period 12 and the difference threshold 13 separately for each data series, i.e., no domain knowledge is required.

[0231] Furthermore, according to this embodiment, the number of user parameters is reduced, thus improving ease of use.

[0232] Implementation method 3.

[0233] Hereinafter, the differences from the embodiments described above will be explained with reference to the accompanying drawings.

[0234] Structural description

[0235] Figure 18 This illustrates a structural example of the data compression system 90 in this embodiment.

[0236] The data compression apparatus 100, compared to the data compression apparatus 100 of Embodiment 2, also includes a parameter determination unit 40. Furthermore, the data compression apparatus 100 of Embodiment 1 may also include a parameter determination unit 40.

[0237] Figure 19 An example of the structure of the parameter determination unit 40 is shown. The parameter determination unit 40 includes a model generation unit 401, an inference unit 404, and an accuracy determination unit 407.

[0238] The parameter determination unit 40 determines whether to use the calculated specified period 12 and the calculated difference threshold 13, respectively, based on the accuracy of the machine learning model obtained by learning using the time series data 5 and the accuracy of the machine learning model obtained by learning using the data obtained by restoring the compressed data 14.

[0239] The model generation unit 401 uses time series data 5 to generate a comparison model 402 and uses restored data 41 to generate a judgment object model 403. The restored data 41 is data obtained by the restoration unit 17 restoring the compressed data 14, and is the time series data used as the judgment object. The comparison model 402 and the judgment object model 403 are both machine learning models.

[0240] The model generation unit 401 may also use past time series data 35 when generating the comparison model 402. The past time series data 35 is time series data generated by the restoration unit 17 retrieving past compressed data 14 from the database 15 and restoring the retrieved compressed data 14.

[0241] The inference unit 404 calculates inference result 405 by performing inference using comparison model 402, and calculates inference result 406 by performing inference using decision object model 403. Inference result 405 and inference result 406 are predicted values, respectively. Inference result 406 is equivalent to the decision object.

[0242] The accuracy determination unit 407 measures the inference error by using inference results 405 and 406, and determines whether the accuracy of the machine learning model corresponding to the compressed data 14 is acceptable.

[0243] Figure 20 This section illustrates a structural example of the statistics selection unit 30 in this embodiment. The statistics selection unit 30 also includes a usage completion table 302.

[0244] Use-out table 302 stores the data of each statistic determined by the parameter determination unit 40, and manages the data of each statistic after use. For example... Figure 20 As shown, the use-out table 302 can also maintain a group of statistics corresponding to the specified period 12 and the statistics corresponding to the difference threshold 13. As the use-out table 302, there can also be a use-out table corresponding to the specified period 12 and a use-out table corresponding to the difference threshold 13.

[0245] The statistic selection unit 30 selects a statistic that is not included in the used table 302 from the statistic list 301. The statistic selection unit 30 can also select a combination of statistics that are not included in the used table 302 if the group of statistics is still in the used table 302.

[0246] The statistics selection unit 30 records each used statistics in the usage completion table 302.

[0247] Figure 21 An image illustrating the processing of the data compression system 90 in this embodiment.

[0248] The parameter determination unit 40 uses both the original data and the time series data obtained by restoring the compressed data 14 to create machine learning models. Based on the difference in inference accuracy between the two machine learning models, it determines whether to adopt the specified period 12 and the difference threshold 13 used when generating the compressed data 14. If the parameter determination unit 40 sets the specified period 12 and the difference threshold 13 to be disregarded, the statistic selection unit 30 selects the statistics used in the calculation of the specified period 12 and the difference threshold 13. The parameter determination unit 40 can perform the adoption determination only for the specified period 12 and the difference threshold 13, or it can perform the adoption determination for both the specified period 12 and the difference threshold 13 simultaneously, or it can perform the adoption determination for the specified period 12 and the difference threshold 13 sequentially.

[0249] Furthermore, the machine learning model compared with the machine learning model corresponding to the restored data 41 can also be a machine learning model obtained by learning from previously collected data. The parameter determination unit 40 can also use a combination of a specified period 12 and a difference threshold 13 that minimizes the error relative to the accuracy of the machine learning model corresponding to the original data.

[0250] Description of the action

[0251] Figure 22 and Figure 23 This is a flowchart illustrating an example of the processing of a data compression system 90. (Using...) Figure 22 and Figure 23 Explain the processing of data compression system 90.

[0252] (Step S301)

[0253] The restoration unit 17 generates restoration data 41 by restoring the compressed data 14 generated by the noise determination unit 11.

[0254] (Step S302)

[0255] Input time series data 5 into parameter determination unit 40.

[0256] (Step S303)

[0257] The restoration unit 17 retrieves the previously generated compressed data 14 from the database 15 and restores the retrieved compressed data 14 into past time series data 35.

[0258] (Step S304)

[0259] The parameter determination unit 40 measures the accuracy of the machine learning model corresponding to the restored data 41, and determines whether to use the current specified period 12 and the difference threshold 13 based on the measurement results.

[0260] (Step S305)

[0261] With the current specified period 12 and differential threshold 13 employed, the data compression system 90 proceeds to step S106. Without employing them, the data compression system 90 repeatedly executes the processes described in this flowchart.

[0262] Figure 24 This is a flowchart illustrating an example of the processing in the statistics selection unit 30. (Using...) Figure 24 Explain the processing of the statistical selection section 30.

[0263] (Step S321)

[0264] Input the used table 302 into the statistics selection section 30.

[0265] (Step S322)

[0266] If the selected statistics 31 and 33 in Table 302 have been used up, the statistic selection unit 30 returns to the process of selecting each statistic. Otherwise, the statistic selection unit 30 proceeds to step S323.

[0267] (Step S323)

[0268] The statistic selection unit 30 records the selected statistics 31 and 33 in the use-out table 302.

[0269] Figure 25 and Figure 26 This is a flowchart illustrating an example of the processing by the parameter determination unit 40. (Using...) Figure 25 and Figure 26 Explain the processing of parameter determination unit 40.

[0270] (Step S341)

[0271] Input time series data 5 into parameter determination unit 40.

[0272] (Step S342)

[0273] Input past time series data 35 into parameter determination unit 40.

[0274] (Step S343)

[0275] Input the restored data 41, which is the object of determination, into the parameter determination unit 40.

[0276] (Step S344)

[0277] The model generation unit 401 substitutes the time series data 5 into the variable BaseSeq.

[0278] (Step S345)

[0279] If past time series data 35 is available, the model generation unit 401 proceeds to step S346. Otherwise, the model generation unit 401 skips step S346.

[0280] (Step S346)

[0281] Model generation unit 401 appends past time series data 35 to the variable BaseSeq.

[0282] (Step S347)

[0283] The model generation unit 401 divides the data shown by the variable BaseSeq into training data and evaluation data.

[0284] (Step S348)

[0285] The model generation unit 401 generates a comparison model 402 by performing machine learning with TrainData as input.

[0286] (Step S349)

[0287] The inference unit 404 generates an inference result 405 by performing inference on the comparison model 402 generated by the model generation unit 401 and inputting TestData.

[0288] (Step S350)

[0289] The model generation unit 401 divides the data shown in the restored data 41 into training data and evaluation data.

[0290] (Step S351)

[0291] The model generation unit 401 generates a decision object model 403 by performing machine learning with TrainData as input.

[0292] (Step S352)

[0293] The inference unit 404 generates an inference result 406 by performing inference on the judgment object model 403 generated by the model generation unit 401 and inputting TestData.

[0294] (Step S353)

[0295] The accuracy determination unit 407 calculates the error of the inference result 406 relative to the inference result 405. As a specific example, the error index is RSME (Root Mean Squared Error), MSE (Mean Squared Error), MAE (Mean Absolute Error), or MAPE (Mean Absolute Percentage Error).

[0296] (Step S354)

[0297] If the calculated error is above a predetermined threshold, the accuracy determination unit 407 proceeds to step S355. Otherwise, the accuracy determination unit 407 proceeds to step S356.

[0298] (Step S355)

[0299] The accuracy determination unit 407 sets whether to adopt or not 43 to not adopt.

[0300] (Step S356)

[0301] The accuracy determination unit 407 sets whether to adopt or not 43 to adopt.

[0302] (Step S357)

[0303] The parameter determination unit 40 outputs whether to adopt or not 43.

[0304] Explanation of the effects of implementation method 3

[0305] As described above, according to this embodiment, the decision to adopt the specified period 12 and the difference threshold 13 is based on the error between the machine learning model corresponding to the compressed data 14 and the machine learning model corresponding to the original data. Therefore, according to this embodiment, it is possible to automatically determine whether the specified period 12 and the difference threshold 13 should be used to minimize the reduction in the inference accuracy of the machine learning model.

[0306] Other implementation methods

[0307] It is possible to freely combine the above-described embodiments, modify any constituent elements of each embodiment, or omit any constituent elements in each embodiment.

[0308] Furthermore, the implementation method is not limited to the methods shown in Implementation Methods 1 to 3, and various modifications can be made as needed. The order of explanation using flowcharts and other methods can also be appropriately changed.

[0309] Label Explanation

[0310] 1: Cloud system; 2: Sensor; 5: Time series data; 6: Network; 7: Data receiving unit; 9: Encoding unit; 10: Encoded data; 11: Noise determination unit; 12: Specified period; 13: Differential threshold; 14: Compressed data; 15: Database; 17: Restoration unit; 18: Restored data; 19: Edge system; 22: Restoration method; 30: Statistic selection unit; 301: Statistic list; 302: Use-up table; 31: Statistic; 32: Specified period calculation unit; 33: Statistic; 34: Differential threshold calculation unit 35: Past time series data; 40: Parameter determination unit; 401: Model generation unit; 402: Comparison model; 403: Decision object model; 404: Inference unit; 405, 406: Inference results; 407: Accuracy determination unit; 41: Restored data; 43: Adoption or not; 51: Processor; 52: Memory; 53: Auxiliary storage device; 54: Input / output IF; 55: Communication device; 58: Processing circuit; 59: Signal line; 90: Data compression system; 100: Data compression device; 200: Data restoration device.

Claims

1. A data compression apparatus for compressing discrete time-series data containing multiple continuous values, wherein the continuous values ​​are composed of one or more data points representing the same value consecutively in the time series, wherein, The data compression device includes a noise detection unit. When the continuous values ​​contained in the time series data are assumed to be continuous values ​​of the object, If the duration of the continuous value of the object is less than a specified period, and the difference between the value of the continuous value of the object and the value of the data point immediately preceding the start point of the continuous value of the object in the time series is less than a difference threshold, the noise determination unit determines the continuous value of the object to be noise. If the duration of the continuous value of the object is longer than the specified period, and the difference between the value of the continuous value of the object and the value of the data point immediately preceding the start point of the continuous value of the object in the time series is less than the difference threshold, the noise determination unit determines that the continuous value of the object is not noise. The noise determination unit generates compressed data, which includes data that does not represent consecutive values ​​determined to be noise and data that represents consecutive values ​​determined not to be noise, and is data generated by compressing the time series data.

2. The data compression device according to claim 1, wherein, If the difference between the value of the continuous value of the object and the value of the data point immediately preceding the start point of the continuous value of the object in the time series is greater than the difference threshold, the noise determination unit determines that the continuous value of the object is not noise.

3. The data compression apparatus according to claim 1 or 2, wherein, The noise determination unit generates the compressed data through differential encoding.

4. The data compression apparatus according to any one of claims 1 to 3, wherein, The specified period and the differential threshold are values ​​calculated based on domain knowledge.

5. The data compression apparatus according to any one of claims 1 to 3, wherein, The data compression device also includes: A specified period calculation unit calculates the specified period based on the duration of each consecutive value contained in the time series data; as well as The difference threshold calculation unit calculates the difference threshold based on the difference between the value of each consecutive value contained in the time series data and the value of the data point immediately preceding the start point of each consecutive value in the time series.

6. The data compression apparatus according to claim 5, wherein, The data compression device further includes a statistics selection unit, which selects the statistics used when calculating the specified period and the statistics used when calculating the difference threshold.

7. The data compression apparatus according to claim 5 or 6, wherein, The data compression device further includes a parameter determination unit, which determines whether to use a calculated specified period and a calculated difference threshold respectively, based on the accuracy of a machine learning model obtained by learning from the data restored from the compressed data and the accuracy of a machine learning model obtained by learning from the time series data.

8. A data compression method, executed as a data compression device of a computer, wherein the data compression device compresses discrete time-series data containing multiple continuous values, the continuous values ​​being composed of one or more data points representing the same value consecutively in the time series, wherein... When the continuous values ​​contained in the time series data are assumed to be continuous values ​​of the object, If the duration of the continuous value of the object is less than a specified period, and the difference between the value of the continuous value of the object and the value of the data point immediately preceding the start point of the continuous value of the object in the time series is less than a difference threshold, the data compression device determines the continuous value of the object to be noise. If the duration of the continuous object value is longer than the specified period, and the difference between the value of the continuous object value and the value of the data point immediately preceding the start point of the continuous object value in the time series is less than the difference threshold, the data compression device determines that the continuous object value is not noise. The data compression device generates compressed data, which includes data that does not represent continuous values ​​determined to be noise and data that does not represent continuous values ​​determined to be noise, and is data generated by compressing the time series data.

9. A data compression program, executed as a data compression device of a computer, the data compression device compressing discrete time-series data containing multiple continuous values, wherein the continuous values ​​are composed of one or more data points representing the same value consecutively in the time series, wherein... The data compression program causes the data compression device to perform the following noise determination processing: When the continuous values ​​contained in the time series data are assumed to be continuous values ​​of the object, If the duration of the continuous value of an object is less than a specified period, and the difference between the value of the continuous value of the object and the value of the data point immediately preceding the start point of the continuous value of the object in the time series is below a difference threshold, the continuous value of the object is determined to be noise. If the duration of the continuous value of the object is longer than the specified period, and the difference between the value of the continuous value of the object and the value of the data point immediately preceding the start point of the continuous value of the object in the time series is less than the difference threshold, the continuous value of the object is determined to be not noise. The compressed data is generated, which includes data that does not represent continuous values ​​determined to be noise and data that does not represent continuous values ​​determined to be noise, and is data generated by compressing the time series data.

Images