Data storage system, data storage method, and data storage program

Abstract

This data storage system (90), which stores irreversibly compressed data, comprises an irreversible compression device (9). The irreversible compression device (9) includes: a smoothness determination unit (18) which determines smoothness according to the rarity of events indicated by target data, as target smoothness; and a data smoothness unit (22) which generates smoothed target data by smoothing the target data by means of the target smoothness.

Description

Data storage system, data storage method, and data storage program

[0001] The present disclosure relates to a data storage system, a data storage method, and a data storage program.

[0002] Since the amount of data used in AI (Artificial Intelligence) development is generally enormous, it is necessary to compress the data. Patent Document 1 discloses a technology for compressing data.

[0003] Patent No. 4719667

[0004] Databases provided by cloud systems and the like are often used to store data used in AI development. Considering data storage costs, it is desirable to perform lossy compression at as high a compression rate as possible. Meanwhile, in AI development, data indicating rare events generally has a relatively high utility value, while data indicating common events generally has a relatively low utility value. Therefore, in order to compress data at as high a compression rate as possible while retaining the characteristics of rare events, it is preferable to determine the data compression rate according to the rarity of the event indicated by the data. However, Patent Document 1 does not disclose a technology for determining the data compression rate according to the rarity of the event indicated by the data.

[0005] The present disclosure aims to realize a data storage system that stores data used in AI development by lossy compressing it, and that determines the compression rate of the data depending on the rarity of the event represented by the data.

[0006] The data storage system according to the present disclosure is a data storage system that stores lossy compressed data, and includes a lossy compression device that includes: a smoothness determination unit that determines a target smoothness based on the rarity of the event indicated by the target data; and a data smoothing unit that generates smoothed target data by smoothing the target data with the target smoothness.

[0007] According to the present disclosure, a smoothing determination unit determines a smoothing degree according to the rarity of an event indicated by the target data, and a data smoothing unit smooths the target data using the determined smoothing degree. Here, the smoothing degree corresponds to a compression rate. Therefore, according to the present disclosure, a data storage system that stores data used in AI development by lossy compressing the data can be realized, in which the data compression rate is determined according to the rarity of the event indicated by the data.

[0008] FIG. 1 is a diagram for explaining an overview of the first embodiment. FIG. 2 is a diagram showing an example of the configuration of a data storage system 90 according to the first embodiment. FIG. 3 is a diagram for explaining functions of the data storage system 90 according to the first embodiment. FIG. 4 is a diagram for explaining a probability distribution according to the first embodiment. FIG. 5 is a diagram for explaining smoothing according to the first embodiment. FIG. 6 is a diagram for explaining smoothing according to the first embodiment. FIG. 7 is a diagram for explaining smoothing according to the first embodiment. FIG. 8 is a diagram for explaining smoothing according to the first embodiment. FIG. 9 is a diagram for explaining smoothing according to the first embodiment. FIG. 10 is a diagram for explaining smoothing according to the first embodiment. FIG. 11 is a diagram showing an example of the hardware configuration of a lossy compression device 9 according to the first embodiment. A flowchart showing the processing of a data receiving unit 7 according to the first embodiment. A flowchart showing the processing of a data interpolation unit 10 according to the first embodiment. A flowchart showing the processing of a data value calculation unit 12 according to the first embodiment. A flowchart showing the processing of a data value calculation unit 12 according to the first embodiment. A flowchart showing the processing of a smoothness determination unit 18 according to the first embodiment. A flowchart showing the processing of a data smoothing unit 22 according to the first embodiment. A flowchart showing the processing of a singularity extraction unit 24 according to the first embodiment. A flowchart showing the processing of a restoration unit 28 according to the first embodiment. A diagram showing an example of the hardware configuration of a lossy compression device 9 according to a variation of the first embodiment. A diagram showing an example of the configuration of a data storage system 90 according to a second embodiment. 10A and 10B are diagrams for explaining the processing of the smoothness determination unit 18 according to the second embodiment, where (a) is a diagram showing a specific example of a smoothness determination function 19 and (b) is a diagram showing a specific example of a smoothness determination table 20. A flowchart showing the processing of the smoothness determination unit 18 according to the second embodiment. A diagram for explaining weights according to the third embodiment. A diagram showing an example of the configuration of a data storage system 90 according to the fourth embodiment. A diagram for explaining the processing of allocating singularity data 25 according to the fourth embodiment. A diagram showing an example of the configuration of a data storage system 90 according to the fifth embodiment. A flowchart showing the processing of a data value calculation unit 40 according to the fifth embodiment. A flowchart showing the processing of a data storage determination unit 44 according to the fifth embodiment.

[0009] In the description of the embodiments and the drawings, the same elements and corresponding elements are given the same reference numerals. The description of elements given the same reference numerals will be omitted or simplified as appropriate. Arrows in the drawings mainly indicate the flow of data or the flow of processing. Furthermore, "unit" may be read as "circuit," "step," "procedure," "process," or "circuitry" as appropriate.

[0010] Embodiment 1. Hereinafter, this embodiment will be described in detail with reference to the drawings. An overview of this embodiment will be described using FIG. 1. In storing sensor data in the Internet of Things (IoT), a data compression method that balances the storage cost and utilization of sensor data is required. Sensor data is discrete data and time-series data. Here, databases with low storage costs generally have the disadvantages of taking a long time to retrieve data and being charged each time data is input or output. Therefore, it is preferable to use databases with low storage costs as long-term storage databases. On the other hand, databases with high storage costs generally have the advantage of being searchable using standard queries such as SQL (Structured Query Language), allowing for fast data input and output. Furthermore, databases with high storage costs are charged according to the database startup time. Therefore, it is preferable to use databases with high storage costs as utilization databases.

[0011] The long-term storage database and the utilization database are each, as a specific example, databases realized in a cloud system. The utilization database is a database that allows AI (Artificial Intelligence) developers to immediately use data, i.e., a database with fast search speeds. The utilization database stores preprocessed data to make it easier for AI developers to use, and also stores lossy-compressed data to reduce data volume. To reduce storage costs, it is preferable to store data compressed at as high a compression rate as possible in the utilization database. However, simply increasing the data compression rate carries the risk of losing data characteristics and rendering the data useless for AI development. The long-term storage database is a database for backing up and archiving data, and is basically operated to minimize the number of reads by accessing it only in emergency situations. Therefore, there is generally no problem with using a database with low storage costs as a long-term storage database. Furthermore, the long-term storage database stores data generated by losslessly compressing unprocessed data.

[0012] *** Description of Configuration *** Fig. 2 shows an example configuration of a data storage system 90 according to this embodiment. As shown in the figure, the data storage system 90 includes a cloud system 1, a sensor 2, a sensor 3, a sensor 4, and a network 6. The cloud system 1, the sensor 2, the sensor 3, and the sensor 4 are each connected via the network 6 so as to be able to communicate with each other.

[0013] Each of sensors 2, 3, and 4 periodically transmits time-series data 5 to cloud system 1 via network 6. The time-series data 5 is data indicating the measurement results of each sensor. Note that the total number of sensors included in data storage system 90 is not limited to three. Each sensor may be any type of sensor.

[0014] The cloud system 1 includes a data receiving unit 7, a long-term storage database 8, a lossy compression device 9, a utilization database 26, and a restoration device 27. Multiple components of the cloud system 1 may be configured integrally as appropriate. The long-term storage database 8 stores losslessly compressed target data. The target data is, for example, time-series data indicating sensor measurement results. The time-series data is, for example, raw data before processing. The lossy compression device 9 includes a data interpolation unit 10, a data value calculation unit 12, a probability distribution storage unit 14, a smoothness determination unit 18, a data smoothing unit 22, and a singularity extraction unit 24. The utilization database 26 stores lossy compressed data. The restoration device 27 includes a restoration unit 28.

[0015] When the data receiving unit 7 receives the time series data 5 , it stores the received time series data 5 in the long-term storage database 8 and outputs the received time series data 5 to the data interpolation unit 10 .

[0016] When the sampling rate of the time series data 5 is low, the data interpolation unit 10 upsamples the time series data 5 using a moving average, Akima interpolation, spline interpolation, or the like in order to smooth the waveform of the time series data 5. Thereafter, the data interpolation unit 10 outputs the upsampled time series data 5 to each of the data value calculation unit 12 and the data smoothing unit 22 as interpolated time series data 11. Furthermore, when the sampling rate of the time series data 5 is sufficiently high, the data interpolation unit 10 outputs the time series data 5 itself to each of the data value calculation unit 12 and the data smoothing unit 22 as interpolated time series data 11.

[0017] The data value calculation unit 12 calculates either the occurrence probability of a data point indicated by the target data or the occurrence probability of a label assigned to the target data as the rarity of an event indicated by the target data. Specifically, the data value calculation unit 12 externally acquires labels identifying events indicated by the interpolated time series data 11 as domain knowledge 13, and also acquires a probability distribution 15 from the probability distribution storage unit 14. Specific examples of events indicated by the target data include data points or amounts of change indicated by the target data, labels assigned to the target data, or state transitions indicated by the target data. The probability distribution 15 is data indicating the occurrence probability of events corresponding to each label. The probability distribution 15 may be data derived from the collected time series data 5, or data provided as domain knowledge. The data value calculation unit 12 calculates the occurrence probability of an event corresponding to a label corresponding to the interpolated time series data 11 based on the probability distribution 15, and outputs data indicating the calculated occurrence probability as probability 16 to the smoothness determination unit 18. In addition, when there are multiple labels indicated by the interpolated time series data 11, the data value calculation unit 12, as a specific example, determines the label with the smallest corresponding occurrence probability among the multiple labels as the representative label of the interpolated time series data 11, and determines the occurrence probability corresponding to the representative label as probability 16. Furthermore, the data value calculation unit 12 may correct the probability distribution 15 by recalculating the probability distribution based on the probability distribution 15 and the labels corresponding to the interpolated time series data 11 and storing the recalculated probability distribution as probability distribution 17 in the probability distribution storage unit 14. In addition, when it is not possible to assign labels to each event, the data value calculation unit 12 may use a probability density function of the data points of the interpolated time series data 11 instead of the labels. In this case, the probability density function may be provided as domain knowledge 13 or may be estimated from the collected time series data 5 using kernel density estimation or the like. In this case, by replacing the occurrence probability of the event corresponding to each label with the occurrence probability of each data point, processing similar to that when using labels can be realized.

[0018] The smoothness determination unit 18 determines the target smoothness based on the rarity of the event indicated by the target data. Specifically, the smoothness determination unit 18 determines the smoothness 21 based on the probability 16 and the domain knowledge 13 and outputs the determined smoothness 21 to the data smoothing unit 22. The smoothness 21 is data indicating the degree of smoothing. Specific examples of the smoothness 21 include the window size of a moving average, λ (smoothness based on second-order derivatives) of a smoothing spline, or a regularization term (penalty term for outliers) of a multiple regression model. The smoothness determination unit 18 typically determines the smoothness 21 on a file-by-file basis. A specific example of a file is observation data from a single day. The domain knowledge 13 is information indicating, for example, that "if the probability 16 is 0.5 or greater, the smoothness 21 is 100, and if the probability 16 is less than 0.5, the smoothness 21 is 10." Here, 0.5 corresponds to a threshold value. It should be noted that each of the smoothness and threshold can be regarded as empirically obtained domain knowledge.

[0019] The data smoothing unit 22 generates smoothed target data by smoothing the target data using the target smoothness. Specifically, the data smoothing unit 22 smooths the interpolated time series data 11 using the value indicated by the smoothness 21 as a parameter value, and outputs the smoothed interpolated time series data 11 to the singularity extraction unit 24 as smoothed time series data 23. The data smoothing unit 22 smooths the interpolated time series data 11 using, for example, moving average or spline smoothing. The smoothed time series data 23 corresponds to the smoothed target data.

[0020] The singularity extraction unit 24 extracts multiple singularities from the smoothed time series data 23, generates singularity data 25 consisting of the extracted multiple singularities, and outputs the generated singularity data 25 to the utilization database 26. Specific examples of each singularity include a start point, an extreme value (maximum point, minimum point), an inflection point, or an end point. Note that the singularity extraction unit 24 does not need to consider inflection points as singularities if it is desired to reduce the number of singularities.

[0021] The restoration unit 28 retrieves the singularity data 25 from the utilization database 26 and restores the data by interpolating the retrieved singularity data 25 using Akima interpolation, spline interpolation, or the like, with the value indicated by the externally provided sampling rate 29 as a parameter value. The restoration unit 28 outputs the data restored in this manner as restored time-series data 30.

[0022] FIG. 3 is a diagram outlining the functions of this embodiment. In this embodiment, the number of singular points is adjusted by determining the degree of smoothing using domain knowledge 13 in lossy compression. In AI development, data indicating rare events is generally highly useful, while data indicating common events is less useful. A rare event is an event with a relatively low probability of occurrence. A common event is an event with a relatively high probability of occurrence. If the target data indicates a rare combination of multiple events, the combination of multiple events may be considered a rare event, or the rarity extracted by analyzing the target data using a known method may be considered a rare event. When the target data indicates a rare event, the lossy compression device 9 lossily compresses the target data so that relatively many of the characteristics of the target data remain. Specifically, the lossy compression device 9 weakens smoothing of the target data and performs minimal noise removal on the target data. Since preferred data processing methods vary depending on the purpose of AI development, AI developers may select appropriate processing methods. Furthermore, if the target data indicates a common event, the lossy compression device 9 lossy compresses the target data so that only the trend of the target data remains. Specifically, the lossy compression device 9 reduces the amount of data by strongly smoothing the target data. Note that data processing may also be performed in the utilization database 26.

[0023] The domain knowledge 13 indicates, as a specific example, a label or the occurrence probability of an event. As a specific example, a label indicates a positive example, a negative example, an event, or an identifier. Note that the average information amount (entropy) shown in [Equation 1] may be used instead of the occurrence probability of an event. When using entropy, an event with high entropy is considered to be equivalent to an event with low occurrence probability, and an event with low entropy is considered to be equivalent to an event with high occurrence probability.

[0024]

[0025] FIG. 4 is a diagram illustrating a case where the occurrence probability of an event is used as domain knowledge 13. The occurrence probability of each event is, as a specific example, a probability estimated based on a group of time-series data consisting of collected time-series data as shown in FIG. 4. Here, the time-series data is interpolated data, and the random variables represent each event. Note that the time-series data described in this specification may be collectively referred to simply as "time-series data." KDE is an abbreviation for Kernel Density Estimation. In FIG. 4, each event is classified as a rare event if its occurrence probability is equal to or less than a predetermined threshold, and as a common event if its occurrence probability is greater than the predetermined threshold.

[0026] The domain knowledge and smoothing will be specifically described using FIG. 5 . In FIG. 5 , 10 years' worth of measurement data is collected, and the measurement data is divided according to the number of years elapsed since the start of measurement. Here, it is assumed that domain knowledge 13 is given indicating that "if outliers are included in each of the data from the first year and the tenth year since the start of measurement, the data is likely to represent an event with a low probability of occurrence." In this case, if these data actually contain outliers, the lossy compression device 9 weakens the smoothing of these data. It is also assumed that domain knowledge 13 is given indicating that "if outliers are included in each of the data from the second to ninth years since the start of measurement, the outliers are likely to be noise." In this case, if these data actually contain outliers, the lossy compression device 9 strengthens the smoothing of these data.

[0027] FIG. 6 illustrates a process for determining the utility value of each piece of time series data from a single label assigned to each piece of time series data and automatically determining the degree of smoothing for each piece of time series data. In FIG. 6, a representative value of the label corresponding to each piece of time series data is used as the label for each piece of time series data, and each label indicates a positive or negative example. In the example illustrated in FIG. 6, when labeled time series data is given, the lossy compression device 9 calculates a probability distribution of events corresponding to each label, determines the utility value of each piece of labeled time series data using the calculated probability distribution, and automatically determines the degree of smoothing for each piece of time series data based on the determination result. The probability distribution may be provided in advance as domain knowledge 13.

[0028] Fig. 7 shows how the utility value of time series data is determined from multiple labels and the degree of smoothing is automatically determined. In Fig. 7, time series data is prepared for each device, and each time series data is data indicating an event corresponding to each time series data. The occurrence probability of each event may differ for each device. In the example shown in Fig. 7, the lossy compression device 9 calculates the probability distribution of each event for each device and automatically determines the degree of smoothing for each time series data using the calculated probability distribution.

[0029] FIG. 8 shows how the utility value of time series data is determined from the frequency distribution of data points and the degree of smoothing is automatically determined. In FIG. 8, no labels are attached to the time series data. In the example shown in FIG. 8, the lossy compression device 9 calculates a probability distribution of data points based on collected time series data, and automatically determines the degree of smoothing for each time series data using the calculated probability distribution. The probability distribution indicates the occurrence probability of each data point. Note that the probability distribution may be provided in advance as domain knowledge 13.

[0030] 9 shows an example of the hardware configuration of a lossy compression device 9 according to this embodiment. The lossy compression device 9 is made up of a computer. The lossy compression device 9 may be made up of multiple computers. The hardware configurations of the other components that make up the data storage system 90 may be similar to the hardware configuration of the lossy compression device 9.

[0031] As shown in the figure, the lossy compression device 9 is a computer including hardware such as a processor 51, a memory 52, an auxiliary storage device 53, an input / output IF (Interface) 54, and a communication device 55. These pieces of hardware are appropriately connected via signal lines 59.

[0032] The processor 51 is an integrated circuit (IC) that performs arithmetic processing and controls the hardware of the computer. Specific examples of the processor 51 include a central processing unit (CPU), a digital signal processor (DSP), or a graphics processing unit (GPU). The lossy compression device 9 may include multiple processors that replace the processor 51. The multiple processors share the role of the processor 51.

[0033] The memory 52 is typically a volatile storage device, and a specific example is RAM (Random Access Memory). The memory 52 is also called a primary storage device or a main memory. Data stored in the memory 52 is saved in the secondary storage device 53 as needed.

[0034] The auxiliary storage device 53 is typically a non-volatile storage device, and specific examples thereof include a ROM (Read Only Memory), an HDD (Hard Disk Drive), or a flash memory. Data stored in the auxiliary storage device 53 is loaded into the memory 52 as needed. The memory 52 and the auxiliary storage device 53 may be configured integrally.

[0035] The input / output IF 54 is a port to which an input device and an output device are connected. Specific examples of the input / output IF 54 include a USB (Universal Serial Bus) terminal. Specific examples of the input device include a keyboard and a mouse. Specific examples of the output device include a display.

[0036] The communication device 55 is a receiver and a transmitter, and is specifically a communication chip or a NIC (Network Interface Card).

[0037] Each part of the lossy compression device 9 may use the input / output IF 54 and the communication device 55 as appropriate when communicating with other devices.

[0038] The auxiliary storage device 53 stores a data storage program. The data storage program is a program that causes a computer to realize the functions of each unit included in the lossy compression device 9. The data storage program is loaded into the memory 52 and executed by the processor 51. The functions of each unit included in the lossy compression device 9 are realized by software.

[0039] Data used when executing the data storage program and data obtained by executing the data storage program are stored in a storage device as appropriate. Each part of the lossy compression device 9 uses a storage device as appropriate. Specific examples of the storage device include at least one of the memory 52, the auxiliary storage device 53, a register in the processor 51, and a cache memory in the processor 51. Note that the terms "data" and "information" may have the same meaning. The storage device may be independent of the computer. The functions of the memory 52 and the auxiliary storage device 53 may be realized by other storage devices.

[0040] The data storage program may be recorded on a computer-readable non-volatile recording medium. Specific examples of the non-volatile recording medium include an optical disk and a flash memory. The data storage program may be provided as a program product.

[0041] ***Explanation of Operation*** The operation procedure of the data archiving system 90 corresponds to a data archiving method. Also, the program that realizes the operation of the data archiving system 90 corresponds to a data archiving program.

[0042] 10 is a flowchart showing an example of the processing of the data receiving unit 7. The processing of the data receiving unit 7 will be described with reference to FIG.

[0043] (Step S101) The data receiving unit 7 receives the time-series data 5 from each sensor via the network 6.

[0044] (Step S102 ) The data receiving unit 7 outputs the received time series data 5 to the data interpolating unit 10 .

[0045] (Step S103 ) The data receiving unit 7 outputs the received time-series data 5 to the long-term storage database 8 .

[0046] 11 is a flowchart showing an example of the processing of the data interpolation unit 10. The processing of the data interpolation unit 10 will be described with reference to FIG.

[0047] (Step S111) The data interpolation unit 10 receives the time series data 5 from the data receiving unit 7. The data interpolation unit 10 may receive a plurality of pieces of time series data 5. When the data interpolation unit 10 receives a plurality of pieces of time series data 5, the data interpolation unit 10 executes the following process for each piece of the received time series data 5.

[0048] (Step S112) If the sampling rate of the received time series data 5 is lower than the predetermined threshold, the data interpolation unit 10 proceeds to step S113. Otherwise, the data interpolation unit 10 proceeds to step S115.

[0049] (Step S113) The data interpolation unit 10 selects a sampling rate that is equal to or higher than the sampling rate of the received time series data 5.

[0050] (Step S114) The data interpolation unit 10 generates interpolated time series data 11 by up-sampling the received time series data 5 using the selected sampling rate and data interpolation function.

[0051] (Step S115 ) The data interpolation unit 10 converts the received time series data 5 into interpolated time series data 11 .

[0052] (Step S116 ) The data interpolation unit 10 outputs the interpolated time-series data 11 to the data value calculation unit 12 .

[0053] (Step S117 ) The data interpolation unit 10 outputs the interpolated time series data 11 to the data smoothing unit 22 .

[0054] 12 is a flowchart showing an example of the process of the data value calculation unit 12 when a label is used. The process of the data value calculation unit 12 will be described with reference to FIG.

[0055] (Step S121) The data value calculation unit 12 receives the interpolated time series data 11 from the data interpolation unit 10. Note that the data value calculation unit 12 may receive a plurality of interpolated time series data 11. When the data value calculation unit 12 receives a plurality of interpolated time series data 11, it executes the following process for each of the received interpolated time series data 11.

[0056] (Step S122) The data value calculation unit 12 receives the domain knowledge 13 from the outside.

[0057] (Step S123) The data value calculation unit 12 labels the events indicated by the interpolated time-series data 11 based on the received domain knowledge 13.

[0058] (Step S124) The data value calculation unit 12 receives the probability distribution 15 indicating the appearance probability of each label from the probability distribution storage unit 14.

[0059] (Step S125) If the data value calculation unit 12 has been able to acquire the probability distribution 15, the process proceeds to step S127. Otherwise, the data value calculation unit 12 proceeds to step S126.

[0060] (Step S126 ) The data value calculation unit 12 acquires the probability distribution 15 from the domain knowledge 13 .

[0061] (Step S127 ) The data value calculation unit 12 identifies, as the probability 16 , the minimum value of the occurrence probability corresponding to each label associated with the interpolated time-series data 11 based on the probability distribution 15 .

[0062] (Step S128) The data value calculation unit 12 recalculates the occurrence probability of each label based on the probability distribution 15 and each label corresponding to the interpolated time series data 11, and generates a probability distribution 17 based on the probability distribution 15 and the recalculated occurrence probability.

[0063] (Step S129) The data value calculation unit 12 outputs the generated probability distribution 17 to the probability distribution storage unit 14.

[0064] (Step S130) The data value calculation unit 12 outputs the probability 16 to the smoothness determination unit 18.

[0065] 13 is a flowchart showing an example of the processing of the data value calculation unit 12 when using the occurrence probability of a data point. The processing of the data value calculation unit 12 will be described with reference to FIG.

[0066] (Step S131) ​​This step is similar to step S121.

[0067] (Step S132) The data value calculation unit 12 receives the probability distribution 15 representing the occurrence probability of each data point indicated by the interpolated time series data 11 from the probability distribution storage unit 14.

[0068] (Step S133) If the data value calculation unit 12 has been able to acquire the probability distribution 15, the process proceeds to step S135. Otherwise, the data value calculation unit 12 proceeds to step S134.

[0069] (Step S134 ) The data value calculation unit 12 estimates a probability density function using kernel density estimation or the like based on the received interpolated time series data 11 , and sets the estimated probability density function as the probability distribution 15 .

[0070] (Step S135 ) The data value calculation unit 12 identifies, as the probability 16 , the minimum value of the occurrence probability of each data point indicated by the interpolated time-series data 11 based on the probability distribution 15 .

[0071] (Step S136) The data value calculation unit 12 recalculates the occurrence probability of each data point based on the probability distribution 15 and the interpolated time series data 11, and generates the probability distribution 17 based on the probability distribution 15 and the recalculated occurrence probability.

[0072] (Step S137) This step is similar to step S129.

[0073] (Step S138) This step is similar to step S130.

[0074] 14 is a flowchart showing an example of the process of the smoothness determining unit 18. The process of the smoothness determining unit 18 will be described with reference to FIG.

[0075] (Step S141 ) The smoothness determination unit 18 receives the probability 16 from the data value calculation unit 12 .

[0076] (Step S142) The smoothness determining unit 18 receives the domain knowledge 13 from the outside.

[0077] (Step S143 ) The smoothness determining unit 18 identifies the smoothness 21 corresponding to the probability 16 based on the received domain knowledge 13 .

[0078] (Step S144 ) The smoothness determination unit 18 outputs the determined smoothness 21 to the data smoothing unit 22 .

[0079] 15 is a flowchart showing an example of the process of the data smoothing unit 22. The process of the data smoothing unit 22 will be described with reference to FIG.

[0080] (Step S151 ) The data smoothing unit 22 receives the interpolated time-series data 11 from the data interpolation unit 10 .

[0081] (Step S152 ) The data smoothing unit 22 receives the smoothness 21 from the smoothness determination unit 18 .

[0082] (Step S153) The data smoothing unit 22 generates smoothed time series data 23, which is data obtained by smoothing the received interpolated time series data 11, using the received interpolated time series data 11, the received smoothness 21, and a moving average or spline smoothing function, etc.

[0083] (Step S154 ) The data smoothing unit 22 outputs the generated smoothed time-series data 23 to the singularity extraction unit 24 .

[0084] 16 is a flowchart showing an example of the process of the singularity extraction unit 24. The process of the singularity extraction unit 24 will be described with reference to FIG.

[0085] (Step S161 ) The singularity extractor 24 receives the smoothed time-series data 23 from the data smoother 22 .

[0086] (Step S162 ) The singularity extractor 24 includes the start point of the smoothed time-series data 23 in the singularity data 25 .

[0087] (Step S163) The singularity extraction unit 24 includes "extreme value" in "type". "Type" is the type of singularity to be included in the singularity data 25. "Extreme value" indicates that the singularity data 25 includes local maximum points and local minimum points.

[0088] (Step S164) If the singularity extraction unit 24 desires to improve the restoration accuracy, the process proceeds to step S165. Otherwise, the singularity extraction unit 24 proceeds to step S166.

[0089] (Step S165) The singular point extraction unit 24 adds "inflection point" to "type".

[0090] (Step S166 ) The singularity extracting unit 24 extracts data points that match the “type” from the smoothed time-series data 23 using the singularity extracting function, and adds the extracted data points to the singularity data 25 .

[0091] (Step S167 ) The singularity extractor 24 adds the end point of the smoothed time-series data 23 to the singularity data 25 .

[0092] (Step S168 ) The singularity extraction unit 24 outputs the generated singularity data 25 to the utilization database 26 .

[0093] 17 is a flowchart showing an example of the process of the restoration unit 28. The process of the restoration unit 28 will be described with reference to FIG.

[0094] (Step S171 ) The restoration unit 28 receives the singularity data 25 from the utilization database 26 .

[0095] (Step S172) The restoration unit 28 receives the sampling rate 29 from the outside.

[0096] (Step S173) The restoration unit 28 generates restored time series data 30 using the singularity data 25, the sampling rate 29, and a data interpolation function such as Akima interpolation or spline interpolation.

[0097] (Step S174) The restoration unit 28 outputs the generated restored time-series data 30.

[0098] ***Explanation of the Effects of Embodiment 1*** Generally, common events are far more numerous than rare events. Therefore, according to this embodiment, by compressing data corresponding to common events at a high compression rate and compressing data corresponding to rare events at a low compression rate, it is possible to significantly reduce the amount of data while retaining the characteristics of rare events. Furthermore, according to this embodiment, when restoring data, the restoration unit 28 can interpolate data at any granularity using each singularity as a reference. Therefore, according to this embodiment, data can be restored in accordance with the sampling rate required for AI learning. Note that when the restoration unit 28 restores data in accordance with a high sampling rate, improved model accuracy is expected. Furthermore, in this embodiment, if only extreme values ​​are used as singular points, the amount of data can be further reduced. Here, including extreme values ​​and inflection points as singular points improves the accuracy of data restoration. Therefore, according to this embodiment, the data size and restoration accuracy can be appropriately selected depending on the use case of data utilization.

[0099] ***Other Configurations*** <Modification 1> Fig. 18 shows an example of the hardware configuration of a lossy compression device 9 according to this modification. The lossy compression device 9 includes a processing circuit 58 instead of the processor 51, the processor 51 and memory 52, the processor 51 and auxiliary storage device 53, or the processor 51, memory 52, and auxiliary storage device 53. The processing circuit 58 is hardware that realizes at least a part of the components included in the lossy compression device 9. The processing circuit 58 may be dedicated hardware, or may be a processor that executes a program stored in the memory 52.

[0100] When the processing circuitry 58 is dedicated hardware, the processing circuitry 58 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The lossy compression device 9 may include multiple processing circuits that replace the processing circuitry 58. The multiple processing circuits share the role of the processing circuitry 58.

[0101] In the lossy compression device 9, some of the functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0102] The processing circuitry 58 is realized by, for example, hardware, software, firmware, or a combination of these. The processor 51, memory 52, auxiliary storage device 53, and processing circuitry 58 are collectively referred to as "processing circuitry." In other words, the functions of each functional component of the lossy compression device 9 are realized by the processing circuitry. The hardware configurations of the other components that make up the data storage system 90 may be similar to that of this modified example. Lossy compression devices 9 according to other embodiments may also be similar to that of this modified example.

[0103] Second Embodiment The following mainly describes the differences from the above-described embodiment with reference to the drawings.

[0104] ***Description of Configuration*** Figure 19 shows an example configuration of a data storage system 90 according to this embodiment. The smoothness determination unit 18 according to this embodiment determines the target smoothness using data indicating the relationship between the rarity of the event indicated by the target data and the smoothness. Specifically, the smoothness determination unit 18 does not receive the domain knowledge 13, but determines the smoothness 21 using the probability 16 and a smoothness determination function 19 or a smoothness determination table 20. The smoothness determination function 19 and the smoothness determination table 20 each correspond to data indicating the relationship between the rarity of the event indicated by the target data and the smoothness. Note that the domain knowledge 13 may also indicate data indicating the relationship between the rarity of the event indicated by the target data and the smoothness.

[0105] The smoothness determination function 19 is a function that indicates the relationship between the rarity of an event and the smoothness. As a specific example, the smoothness determination function 19 is a function that indicates the relationship between the occurrence probability of an event and the smoothness, or a function that indicates the relationship between a label corresponding to an event and the smoothness. When the smoothness determination function 19 is a function that indicates the relationship between the occurrence probability of an event and the smoothness, the smoothness determination function 19 may be any monotonically increasing function. Figure 20 (a) shows two specific examples of the smoothness determination function 19. In the smoothness determination function 19, the lower the occurrence probability, the weaker the corresponding smoothness, and the higher the occurrence probability, the stronger the corresponding smoothness.

[0106] The smoothness determination table 20 is a table showing the relationship between the rarity of an event and the smoothness. As a specific example, the smoothness determination table 20 is a table showing the relationship between the occurrence probability of an event and the smoothness, or a table showing the relationship between the label corresponding to the event and the smoothness. (b) of FIG. 20 shows a specific example of the smoothness determination table 20. Note that the smoothness determination table 20 may be a table showing labels instead of probabilities.

[0107] ***Explanation of Operation*** Fig. 21 is a flowchart showing an example of the processing of the smoothness determining unit 18. The processing of the smoothness determining unit 18 will be described with reference to Fig. 21 .

[0108] (Step S201) The smoothness determination unit 18 receives the smoothness determination function 19 or the smoothness determination table 20 from the outside.

[0109] (Step S202 ) The smoothness determination unit 18 identifies the smoothness 21 corresponding to the probability 16 based on the received smoothness determination function 19 or smoothness determination table 20 .

[0110] ***Explanation of the Effects of Embodiment 2*** As described above, according to this embodiment, smoothness is determined using the smoothness determination function 19 or the smoothness determination table 20, so there is no need to determine the smoothness from domain knowledge for each piece of time series data, and the smoothness can be determined seamlessly and automatically according to the value of the data.

[0111] Third Embodiment Hereinafter, differences from the above-described embodiments will be mainly described with reference to the drawings.

[0112] *** Description of Configuration *** An example configuration of the data storage system 90 according to this embodiment is the same as the example configuration of the data storage system 90 according to the above-described embodiment. The data smoothing unit 22 according to this embodiment smooths the target data using a weight that becomes larger the more rare the event indicated by the target data. Specifically, the data smoothing unit 22 uses a weighted moving average in smoothing. The weighted moving average is also called a weighted moving average. The weight is used to increase the value of data points with a low probability of occurrence. As a specific example, the weight is the reciprocal of the probability of occurrence. [Mathematical formula 2] shows a specific example of a weighted moving average. Here, PDF is the probability density function of the interpolated time series data 11, and x ｉ are sample points (i=1, . . . , n) within the window in the moving average.

[0113]

[0114] Figure 22 shows how weights are assigned taking into account the value of data points. The left side of Figure 22 shows a window in a moving average, and the right side of Figure 22 shows a probability distribution indicating the occurrence probability of each data point. With a normal moving average, as shown on the left side of Figure 22, when high-value data and low-value data are mixed within the window, the value of the window as a whole may be low. On the other hand, with a weighted moving average in which the weight is the reciprocal of the occurrence probability, the value of the window as a whole is less likely to be low when high-value data and low-value data are mixed within the window.

[0115] ***Explanation of Operation*** The difference between the operation of the data storage system 90 according to this embodiment and the operation of the data storage system 90 according to the previous embodiment will be explained. In step S153, the data smoothing unit 22 uses a weighted moving average instead of a moving average or a spline smoothing function.

[0116] ***Explanation of the Effects of Embodiment 3*** When a simple moving average is used in smoothing, the value of the sample points within the window is not fully taken into consideration. On the other hand, according to this embodiment, a weighted moving average is used instead of a simple moving average, so the value of the sample points within the window is taken into consideration. Therefore, according to this embodiment, the characteristics of the data are more likely to remain even when strong smoothing is applied.

[0117] Fourth Embodiment Hereinafter, differences from the above-described embodiments will be mainly described with reference to the drawings.

[0118] 23 shows an example of the configuration of a data storage system 90 according to this embodiment. The cloud system 1 according to this embodiment includes a utilization database 31, a utilization database 32, and a utilization database 33.

[0119] The utilization database 31, the utilization database 32, and the utilization database 33 correspond to multiple utilization databases that are candidates for storing the singularity data 25. The computational resources of each of the multiple utilization databases may be different from each other. Each of the utilization databases 31, the utilization database 32, and the utilization database 33 is similar to the utilization database 26.

[0120] The singularity extraction unit 24 according to this embodiment selects a utilization database from among a plurality of utilization databases as a storage destination for the singularity data 25. At this time, the singularity extraction unit 24 stores the singularity data 25 in a utilization database with more computational resources as the utilization value of the singularity data 25 increases. As a specific example, the singularity extraction unit 24 distributes and outputs the singularity data 25 to the utilization database 31, the utilization database 32, and the utilization database 33. Note that the number of databases to which the singularity extraction unit 24 outputs is not limited to three. At this time, as a specific example, the singularity extraction unit 24 distributes the singularity data 25 according to the frequency of use of the singularity data, the cumulative number of search hits for the singularity data, or the value (probability 16) of the time-series data corresponding to the singularity data 25. Note that it is estimated that the greater the corresponding cumulative number of search hits, the higher the utilization value of the data.

[0121] FIG. 24 is a diagram illustrating the process of allocating the singularity data 25 according to the cumulative search hit count. As a specific example, the singularity extraction unit 24 allocates the singularity data 25 so that data with a cumulative search hit count of 100 or more is stored in the utilization database 31, data with a cumulative search hit count of 51 to 99 is stored in the utilization database 32, and data with a cumulative search hit count of 50 or less is stored in the utilization database 33. In this example, to increase search speed, more computational resources such as CPU and memory resources are allocated to the utilization database 31. Similarly, fewer computational resources than those of the utilization database 31 are allocated to the utilization database 32, and fewer computational resources than those of the utilization database 32 are allocated to the utilization database 33. By allocating computational resources in this manner, it is possible to reduce database operating costs while improving response performance for searches of data that is used relatively frequently.

[0122] ***Explanation of Operation*** The following explains the differences between the operation of the data storage system 90 according to this embodiment and the operation of the data storage system 90 according to the previous embodiment. In step S168, the singularity extraction unit 24 appropriately allocates the singularity data 25 to the utilization database 31, the utilization database 32, and the utilization database 33.

[0123] ***Explanation of the Effects of Embodiment 4*** As described above, according to this embodiment, by preparing multiple types of databases and allocating the singularity data 25 to each database according to the value of the singularity data 25, it is possible to improve the response performance for searches of data that is used with a relatively high frequency while reducing the operating costs of the database.

[0124] Fifth Embodiment Hereinafter, differences from the above-described embodiments will be mainly described with reference to the drawings.

[0125] ***Description of Configuration*** FIG. 25 shows an example of the configuration of a data storage system 90 according to this embodiment. Note that FIG. 25 excerpts the configuration that is characteristic of this embodiment. The configuration of the cloud system 1 and the like according to this embodiment is basically as described above. The data storage system 90 includes a client device 34. The client device 34 includes a temporary storage area 35, a data receiving unit 36, a data receiving unit 38, a data value calculation unit 40, and a data storage determination unit 44. The client device 34 is communicatively connected to the cloud system 1, and is communicatively connected to the lossy compression device 9 via the data receiving unit 37. The cloud system 1 according to this embodiment includes a data receiving unit 37. The data receiving unit 36, the data receiving unit 37, and the data receiving unit 38 are each similar to the data receiving unit 7.

[0126] In this embodiment, time-series data with relatively high utility value is given priority and temporarily stored in the client device 34 for the purpose of backup in preparation for a temporary network outage, etc. The client device 34 acquires in advance the utility value of the time-series data (such as the probability distribution of data points) calculated on the cloud system 1 side, and determines whether or not to store the acquired time-series data 5 at the time of acquiring the time-series data 5. The client device 34 periodically transmits the time-series data 5 measured by the sensor 2 to the cloud system 1 via the network 6. Note that the number of sensors is not limited to one.

[0127] The temporary storage area 35 stores the temporarily stored data in association with the rarity of the event indicated by the temporarily stored data. Specifically, the temporary storage area 35 stores the time series data 5 and the probability 41. The time series data 5 corresponds to the temporarily stored data.

[0128] The data value calculation unit 40 receives the time series data 5 from the sensor 2. The data value calculation unit 40 also receives the probability distribution 39 representing the occurrence probability of each data point indicated by the time series data 5 from the probability distribution storage unit 14, and identifies the probability corresponding to the data point indicated by the time series data 5 with the lowest occurrence probability based on the received probability distribution 39, and sets the identified probability as probability 41. Thereafter, the data value calculation unit 40 outputs the time series data 5 and the probability 41 to the data storage determination unit 44.

[0129] The data storage determination unit 44 determines whether to store the target data in the temporary storage area 35 based on the result of comparing the rarity of the event indicated by the target data with the rarity of the event indicated by the temporarily stored data. As a specific example, the data storage determination unit 44 receives, from among the time-series data 5 stored in the temporary storage area 35, the time-series data 5 having the highest occurrence probability of the corresponding representative point as the time-series data 42, and also receives, as the probability 43, the probability 41 corresponding to the representative point of the time-series data 42. Thereafter, if there is free space in the temporary storage area 35, the data storage determination unit 44 stores the time-series data 5 and the probability 41 in the temporary storage area 35. If there is no free space in the temporary storage area 35, the data storage determination unit 44 compares the probability 41 and the probability 43, and if the probability 41 is smaller than the probability 43, deletes the time-series data 42 and the probability 43 from the temporary storage area 35 and stores the time-series data 5 and the probability 41 in the temporary storage area 35. Furthermore, if the probability 43 is less than or equal to the probability 41, the data storage determination unit 44 discards the time series data 5 and does not store the time series data 5 in the temporary storage area 35.

[0130] ***Explanation of Operation*** Fig. 26 is a flowchart showing an example of the processing of the data value calculation unit 40. The processing of the data value calculation unit 40 will be described using Fig. 26.

[0131] (Step S401) The data value calculation unit 40 receives the time-series data 5 from the sensor 2.

[0132] (Step S402) The data value calculation unit 40 receives the probability distribution 39 indicating the occurrence probability of each data point indicated by the time-series data 5 from the probability distribution storage unit 14.

[0133] (Step S403) If the data value calculation unit 40 has been able to acquire the probability distribution 39, the process proceeds to step S405. Otherwise, the data value calculation unit 40 proceeds to step S404.

[0134] (Step S404) The data value calculation unit 40 estimates a probability density function using kernel density estimation or the like based on the received time-series data 5, and sets the estimated probability density function as the probability distribution 39.

[0135] (Step S405) The data value calculation unit 40 identifies, as the probability 41, the minimum value of the occurrence probability of each data point indicated by the time-series data 5 based on the probability distribution 39.

[0136] (Step S406) The data value calculation unit 40 outputs the received time-series data 5 and the identified probability 41 to the data storage determination unit 44.

[0137] 27 is a flowchart showing an example of the process of the data storage determination unit 44. The process of the data storage determination unit 44 will be described with reference to FIG.

[0138] (Step S411) The data storage determination unit 44 receives the time-series data 5 and the probability 41 from the data value calculation unit 40.

[0139] (Step S412) The data storage determination unit 44 receives, from among the time series data 5 stored in the temporary storage area 35, the time series data 5 having the highest occurrence probability of the corresponding representative point as the time series data 42, and receives, as the probability 43, the probability 41 corresponding to the representative point of the time series data 42.

[0140] (Step S413) If there is free space in the temporary storage area 35, the data storage determination unit 44 proceeds to step S417. Otherwise, the data storage determination unit 44 proceeds to step S414.

[0141] (Step S414) If Probability 41<Probability 43, the data storage determination unit 44 proceeds to step S416. Otherwise, the data storage determination unit 44 proceeds to step S415.

[0142] (Step S415) The data storage determination unit 44 discards the received time-series data 5.

[0143] (Step S416 ) The data storage determination unit 44 deletes the time-series data 42 and the probability 43 from the temporary storage area 35 .

[0144] (Step S417) The data storage determination unit 44 outputs the time-series data 5 and the probability 41 to the temporary storage area 35.

[0145] ***Explanation of the Effects of Embodiment 5*** As described above, according to this embodiment, data that is relatively useful is stored preferentially in the client device 34, thereby improving the cost-effectiveness of the device.

[0146] ***Other Embodiments*** The above-described embodiments can be freely combined, or any of the components of each embodiment can be modified, or any of the components can be omitted from each embodiment. Furthermore, the embodiments are not limited to those shown in embodiments 1 to 5, and various modifications are possible as needed. The procedures described using flowcharts, etc., can be modified as appropriate.

[0147] 1 Cloud system, 2, 3, 4 Sensor, 5 Time series data, 6 Network, 7 Data receiving unit, 8 Long-term storage database, 9 Lossy compression device, 10 Data interpolation unit, 11 Interpolated time series data, 12 Data value calculation unit, 13 Domain knowledge, 14 Probability distribution storage unit, 15 Probability distribution, 16 Probability, 17 Probability distribution, 18 Smoothing determination unit, 19 Smoothing determination function, 20 Smoothing determination table, 21 Smoothing, 22 Data smoothing unit, 23 Smoothed time series data, 24 Singularity extraction unit, 25 Singularity data, 26 Utilization database, 27 Restoration device, 28 Restoration unit, 29 Sampling rate, 30 Reconstructed time series data, 31, 32, 33 Utilization database, 34 Client device, 35 Temporary storage area, 36, 37, 38 Data receiving unit, 39 Probability distribution, 40 Data value calculation unit, 41 Probability, 42 time series data, 43 probability, 44 data storage determination unit, 51 processor, 52 memory, 53 auxiliary storage device, 54 input / output IF, 55 communication device, 58 processing circuit, 59 signal line, 90 data storage system.

Claims

1. A data storage system for storing lossy compressed data, a smoothness determination unit that determines the smoothness of the object according to the rarity of the event indicated by the object data; a data smoothing unit that generates smoothed target data by smoothing the target data with the target smoothness; A lossy compression device comprising: A data storage system comprising:

2. The data storage system further comprises: a data value calculation unit that calculates, as the rarity of the event indicated by the target data, either the occurrence probability of a data point indicated by the target data or the occurrence probability of a label assigned to the target data; The data storage system of claim 1 .

3. 3. The data storage system according to claim 1, wherein the smoothness determining unit determines the target smoothness using data indicating a relationship between the rarity of the event indicated by the target data and the smoothness.

4. The data storage system according to claim 1 or 2, wherein the data smoothing unit smoothes the target data using a weight that becomes larger as the event indicated by the target data becomes rarer.

5. The lossy compression device further comprises: a singularity extraction unit that extracts a plurality of singularities from the smoothed target data and generates singularity data consisting of the extracted singularities; Equipped with The data storage system further comprises: Multiple utilization databases that are candidates for storing the singularity data Equipped with The computing resources of each of the plurality of utilization databases are different from one another, The data storage system according to claim 1 or 2, wherein the singularity extraction unit selects a utilization database from among the plurality of utilization databases as the storage destination for the singularity data so that the more useful the singularity data is, the more computing resources the singularity data is stored in the utilization database.

6. The data storage system further comprises: a temporary storage area for storing the temporarily stored data in association with the rarity of the event indicated by the temporarily stored data; a data storage determination unit that determines whether or not to store the target data in the temporary storage area based on a result of comparing the rarity of the event indicated by the target data with the rarity of the event indicated by the temporarily stored data; A client device comprising: Equipped with 3. The data storage system according to claim 1, wherein the lossy compression device and the client device are connected so as to be able to communicate with each other.

7. The data storage system further comprises: A long-term storage database that stores the losslessly compressed target data The data storage system according to claim 1 or 2, comprising:

8. The data storage system according to claim 1 or 2, wherein the target data is time-series data indicating measurement results of a sensor.

9. A data storage method executed in a data storage system that stores lossy compressed data, comprising: The computer determines the target smoothness in accordance with the rarity of the event indicated by the target data; A data storage method in which the computer generates smoothed target data by smoothing the target data with the target smoothness.

10. A data storage program executed by a lossy compression device, which is a computer included in a data storage system that stores lossy compressed data, a smoothness determination process for determining the smoothness of the target data in accordance with the rarity of the event indicated by the target data; a data smoothing process for generating smoothed target data by smoothing the target data with the target smoothness; A data storage program that causes the lossy compression device to execute the above.