DATA STORAGE SYSTEM, DATA STORAGE METHOD, AND DATA STORAGE PROGRAM

Abstract

This data storage system (90), which stores irreversibly compressed data, comprises an irreversible compression device (9). The irreversible compression device (9) includes: a smoothness determination unit (18) which determines smoothness according to the rarity of events indicated by target data, as target smoothness; and a data smoothness unit (22) which generates smoothed target data by smoothing the target data by means of the target smoothness.

Description

[Technical field]

[0001] The present disclosure relates to a data storage system, a data storage method, and a data storage program. [Background technology]

[0002] Since the amount of data used in AI (Artificial Intelligence) development is generally huge, the data needs to be compressed. Patent Document 1 discloses a technology for compressing data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4719667 Summary of the Invention [Problem to be solved by the invention]

[0004] In AI development, databases provided in cloud systems and the like are often used as storage destinations for data used in the development of AI. Considering the cost of storing data, it is desirable to perform lossy compression at as high a compression rate as possible. On the other hand, in general, in AI development, the useful value of data indicating rare events is relatively high, and the useful value of data indicating common events is relatively low. Therefore, in order to compress data at as high a compression rate as possible while retaining the characteristics of rare events, it is preferable to determine the compression rate of the data according to the rarity of the event indicated by the data. However, Patent Document 1 does not disclose a technology for determining the compression rate of the data according to the rarity of the event indicated by the data.

[0005] The present disclosure aims to realize a data storage system that lossily compresses and stores data used in AI development, and that determines the compression rate of the data depending on the rarity of the event that the data indicates. [Means for solving the problem]

[0006] The data storage system according to the present disclosure comprises: A data storage system for storing lossy compressed data, comprising: a smoothness determining unit that determines, as an object smoothness, a smoothness according to the rarity of an event represented by the object data; a data smoothing unit that generates smoothed target data by smoothing the target data with the target smoothness; A lossy compression device comprising: Equipped with. Effect of the Invention

[0007] According to the present disclosure, the smoothness determination unit determines the smoothness according to the rarity of the event indicated by the target data, and the data smoothing unit smoothes the target data with the determined smoothness. Here, the smoothness corresponds to the compression rate. Therefore, according to the present disclosure, it is possible to realize a data storage system that stores data used in AI development by lossy compressing the data, and determines the compression rate of the data according to the rarity of the event indicated by the data. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining an overview of a first embodiment. [Diagram 2] FIG. 1 shows an example of the configuration of a data storage system 90 according to a first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining the functions of a data storage system 90 according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining a probability distribution according to the first embodiment. [Diagram 5] FIG. 4 is a diagram for explaining smoothing according to the first embodiment. [Figure 6] FIG. 4 is a diagram for explaining smoothing according to the first embodiment. [Figure 7] FIG. 4 is a diagram for explaining smoothing according to the first embodiment. [Figure 8] FIG. 4 is a diagram for explaining smoothing according to the first embodiment. [Figure 9]2 is a diagram showing an example of the hardware configuration of a lossy compression device 9 according to the first embodiment. [Figure 10] 4 is a flowchart showing a process of a data receiving unit 7 according to the first embodiment. [Figure 11] 4 is a flowchart showing the process of the data interpolation unit 10 according to the first embodiment. [Figure 12] 4 is a flowchart showing the process of a data value calculation unit 12 according to the first embodiment. [Figure 13] 4 is a flowchart showing the process of a data value calculation unit 12 according to the first embodiment. [Figure 14] 4 is a flowchart showing the process of a smoothness determining unit 18 according to the first embodiment. [Figure 15] 4 is a flowchart showing the process of a data smoothing unit 22 according to the first embodiment. [Figure 16] 4 is a flowchart showing the process of a singular point extraction unit 24 according to the first embodiment. [Figure 17] 4 is a flowchart showing the processing of a restoration unit 28 according to the first embodiment. [Figure 18] FIG. 13 is a diagram showing an example of the hardware configuration of a lossy compression device 9 according to a modified example of the first embodiment. [Figure 19] FIG. 13 is a diagram showing an example of the configuration of a data storage system 90 according to a second embodiment. [Figure 20] 13A and 13B are diagrams for explaining the processing of a smoothness determination unit 18 according to the second embodiment, where FIG. 13A shows a specific example of a smoothness determination function 19 and FIG. 13B shows a specific example of a smoothness determination table 20. [Figure 21] 10 is a flowchart showing the process of a smoothness determining unit 18 according to the second embodiment. [Figure 22] FIG. 13 is a diagram for explaining weights according to the third embodiment. [Diagram 23] FIG. 13 is a diagram showing an example of the configuration of a data storage system 90 according to a fourth embodiment. [Figure 24] 13 is a diagram for explaining a process of allocating the singular point data 25 according to the fourth embodiment. FIG. [Diagram 25] FIG. 13 is a diagram showing an example of the configuration of a data storage system 90 according to a fifth embodiment. [Figure 26]13 is a flowchart showing the processing of a data value calculation unit 40 according to the fifth embodiment. [Figure 27] 13 is a flowchart showing the process of a data storage determination unit 44 according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the description of the embodiments and the drawings, the same elements and corresponding elements are given the same reference numerals. Descriptions of elements given the same reference numerals are omitted or simplified as appropriate. Arrows in the drawings primarily indicate data flow or processing flow. In addition, "part" may be read as "circuit," "step," "procedure," "processing," or "circuitry" as appropriate.

[0010] Embodiment 1 Hereinafter, the present embodiment will be described in detail with reference to the drawings. The outline of this embodiment will be described with reference to Fig. 1. In storing sensor data in the Internet of Things (IoT), a data compression method that balances the storage cost and utilization of the sensor data is required. The sensor data is discrete data and time-series data. However, databases with low storage costs generally have the disadvantage that it takes a long time to retrieve data and that a fee is charged each time data is input or output, so it is preferable to use a database with low storage costs as a long-term storage database. On the other hand, databases with high storage costs generally have the advantage of high speed data input / output because they can be searched using standard queries such as SQL (Structured Query Language). In addition, databases with high storage costs are charged according to the database startup time. For this reason, it is preferable to use databases with high storage costs as utilization databases.

[0011] As a specific example, each of the long-term storage database and the utilization database is a database realized in a cloud system. The utilization database is a database that allows AI (Artificial Intelligence) developers to use data immediately, i.e., a database with fast search speed. In the utilization database, data that has been preprocessed to make it easier for AI developers to use the data is stored, and lossy compressed data is stored to reduce the amount of data. Here, it is preferable to store data compressed with as high a compression rate as possible in order to reduce storage costs. However, simply increasing the data compression rate runs the risk of the data becoming unuseful for AI development because the characteristics of the data are lost. Long-term storage databases are databases for backing up and archiving data, and are basically operated to minimize the number of reads by accessing them only in the unlikely event of an emergency. Therefore, there are basically no problems with using a database with low storage costs as a long-term storage database. In addition, data generated by losslessly compressing unprocessed data is stored in long-term storage databases.

[0012] ***Configuration Description*** 2 shows an example of the configuration of a data storage system 90 according to this embodiment. As shown in the figure, the data storage system 90 includes a cloud system 1, a sensor 2, a sensor 3, a sensor 4, and a network 6. The cloud system 1, the sensor 2, the sensor 3, and the sensor 4 are each connected to each other via the network 6 so as to be able to communicate with each other.

[0013] Each of the sensors 2, 3, and 4 periodically transmits time series data 5 to the cloud system 1 via the network 6. The time series data 5 is data indicating the measurement results of each sensor. Note that the total number of sensors included in the data storage system 90 is not limited to three. Each sensor may be any type of sensor.

[0014] The cloud system 1 includes a data receiving unit 7, a long-term storage database 8, a lossy compression device 9, a utilization database 26, and a restoration device 27. The multiple components of the cloud system 1 may be configured integrally as appropriate. The long-term storage database 8 stores losslessly compressed target data. The target data is, for example, time-series data indicating the measurement results of a sensor. The time-series data is, for example, raw data before processing. The lossy compression device 9 includes a data interpolation unit 10, a data value calculation unit 12, a probability distribution storage unit 14, a smoothness determination unit 18, a data smoothing unit 22, and a singular point extraction unit 24. The utilization database 26 stores the lossy compressed data. The restoration device 27 includes a restoration unit 28 .

[0015] When the data receiving unit 7 receives the time series data 5 , it stores the received time series data 5 in a long-term storage database 8 and outputs the received time series data 5 to a data interpolation unit 10 .

[0016] When the sampling rate of the time series data 5 is low, the data interpolation unit 10 upsamples the time series data 5 using a moving average, Akima interpolation, spline interpolation, or the like in order to make the time series data 5 have a smooth waveform. Thereafter, the data interpolation unit 10 outputs the upsampled time series data 5 as the interpolated time series data 11 to each of the data value calculation unit 12 and the data smoothing unit 22. Furthermore, when the sampling rate of the time series data 5 is sufficiently high, the data interpolation unit 10 outputs the time series data 5 itself as the interpolated time series data 11 to each of the data value calculation unit 12 and the data smoothing unit 22.

[0017] The data value calculation unit 12 calculates, as the rarity of the event indicated by the target data, either the occurrence probability of a data point indicated by the target data or the occurrence probability of a label assigned to the target data. As a specific example, the data value calculation unit 12 externally acquires a label that identifies the event indicated by the interpolated time series data 11 as domain knowledge 13, and also acquires a probability distribution 15 from the probability distribution storage unit 14. Specific examples of the event indicated by the target data are a data point or amount of change indicated by the target data, a label assigned to the target data, or a state transition indicated by the target data. The probability distribution 15 is data indicating the occurrence probability of an event corresponding to each label. The probability distribution 15 may be data derived from the collected time-series data 5, or may be data given as domain knowledge. The data value calculation unit 12 calculates the occurrence probability of an event corresponding to a label corresponding to the interpolated time series data 11 based on the probability distribution 15, and outputs data indicating the calculated occurrence probability to the smoothness determination unit 18 as probability 16. Note that when there are multiple labels indicated by the interpolated time series data 11, as a specific example, the data value calculation unit 12 determines the label with the smallest corresponding occurrence probability among the multiple labels as the representative label of the interpolated time series data 11, and determines the occurrence probability corresponding to the representative label as probability 16. In addition, the data value calculation unit 12 may correct the probability distribution 15 by recalculating the probability distribution based on the probability distribution 15 and a label corresponding to the interpolated time series data 11, and storing the recalculated probability distribution in the probability distribution storage unit 14 as the probability distribution 17. If it is not possible to assign a label to each event, the data value calculation unit 12 may use a probability density function of the data points of the interpolated time series data 11 instead of the labels. In this case, the probability density function may be given as domain knowledge 13, or may be estimated from the collected time series data 5 by kernel density estimation or the like. In this case, by replacing the occurrence probability of the event corresponding to each label with the occurrence probability of each data point, it is possible to realize processing similar to that in the case of using labels.

[0018] The smoothness determination unit 18 determines the smoothness as the target smoothness according to the rarity of the event indicated by the target data. Specifically, the smoothness determination unit 18 determines the smoothness 21 based on the probability 16 and the domain knowledge 13, and outputs the determined smoothness 21 to the data smoothing unit 22. The smoothness 21 is data indicating the degree of smoothing. Specific examples of the smoothness 21 include the window size of the moving average, the λ of the smoothing spline (smoothness based on the second derivative), or the regularization term of the multiple regression model (penalty term for outliers). The smoothness determining unit 18 typically determines the smoothness 21 on a file-by-file basis. A file specifically consists of observation data for a certain day. The domain knowledge 13 is information indicating, as a specific example, that "if the probability 16 is 0.5 or more, the smoothness 21 is 100, and if the probability 16 is less than 0.5, the smoothness 21 is 10." Here, 0.5 corresponds to a threshold. Note that each of the smoothness and the threshold can be regarded as empirically obtained domain knowledge.

[0019] The data smoothing unit 22 generates smoothed target data by smoothing the target data with the target smoothness. Specifically, the data smoothing unit 22 smoothes the interpolated time series data 11 using the value indicated by the smoothness 21 as a parameter value, and outputs the smoothed interpolated time series data 11 to the singularity extraction unit 24 as smoothed time series data 23. The data smoothing unit 22 smoothes the interpolated time series data 11 using, for example, moving average or spline smoothing. The smoothed time series data 23 corresponds to the smoothed target data.

[0020] The singularity extraction unit 24 extracts a plurality of singularities from the smoothed time series data 23, generates singularity data 25 consisting of the extracted singularities, and outputs the generated singularity data 25 to a utilization database 26. Specific examples of each singular point include a start point, an extreme value (maximum point, minimum point), an inflection point, and an end point. Note that the singular point extraction unit 24 does not need to treat an inflection point as a singular point if it is desired to reduce the number of singular points.

[0021] The restoration unit 28 extracts the singularity data 25 from the utilization database 26, and restores the data by interpolating the extracted singularity data 25 using Akima interpolation, spline interpolation, or the like with a value indicated by an externally provided sampling rate 29 as a parameter value. The restoration unit 28 outputs the data thus restored as restored time series data 30.

[0022] FIG. 3 is a diagram for explaining an outline of the functions of this embodiment. In this embodiment, in lossy compression, the number of singular points is adjusted by determining the degree of smoothing using domain knowledge 13. Here, in AI development, basically, data showing rare events has high utility value, and data showing common events has low utility value. A rare event is an event with a relatively low occurrence probability. A common event is an event with a relatively high occurrence probability. When a combination of multiple events shown in the target data is rare, the combination of the multiple events may be regarded as a rare event, or the rarity extracted by analyzing the target data using a known method may be regarded as a rare event. When the target data indicates a rare event, the lossy compression device 9 lossily compresses the target data so that relatively many of the characteristics of the target data remain. Specifically, the lossy compression device 9 weakens smoothing of the target data and performs a minimum amount of noise removal on the target data. Note that, regarding the data processing method, since the preferred processing method differs depending on the purpose of AI development, etc., the AI ​​developer may select the processing method as appropriate. Furthermore, when the target data indicates a common event, the lossy compression device 9 lossy compresses the target data so that only the trend of the target data remains. Specifically, the lossy compression device 9 reduces the amount of data by strongly smoothing the target data. Note that data processing may be performed in the utilization database 26.

[0023] The domain knowledge 13 indicates, as a specific example, a label or the occurrence probability of an event. As a specific example, a label indicates a positive example, a negative example, an event, or an identifier. Note that the average amount of information (entropy) shown in [Equation 1] may be used instead of the occurrence probability of an event. When using entropy, an event with high entropy is considered to be equivalent to an event with low occurrence probability, and an event with low entropy is considered to be equivalent to an event with high occurrence probability.

[0024]

number

[0025] FIG. 4 is a diagram for explaining a case where the occurrence probability of an event is used as the domain knowledge 13. The occurrence probability of each event is a probability estimated based on a time series data group consisting of collected time series data as shown in FIG. 4 as a specific example. Here, the time series data is data that has been interpolated, and the random variables indicate each event. Note that each time series data described in this specification may be collectively referred to simply as "time series data." KDE is an abbreviation for Kernel Density Estimation. In addition, in FIG. 4, each event is classified as a rare event if its occurrence probability is equal to or less than a predetermined threshold, and is classified as a common event if its occurrence probability is greater than the predetermined threshold.

[0026] The domain knowledge and smoothing will be specifically described with reference to Fig. 5. In Fig. 5, measurement data for 10 years has been collected, and the measurement data is divided according to the number of years that have passed since the start of measurement. Here, it is assumed that domain knowledge 13 is given that indicates that "if the data from the first year and the tenth year after the start of measurement contain outliers, the data is highly likely to represent an event with a low occurrence probability." In this case, if these data actually contain outliers, the lossy compression device 9 weakens the smoothing of these data. Also, it is assumed that domain knowledge 13 is given that indicates that "if each of the data from the second to ninth years since the start of measurement contains an outlier, the outlier is highly likely to be noise." In this case, if the data actually contains an outlier, the lossy compression device 9 strengthens the smoothing of the data.

[0027] Fig. 6 shows how the utility value of each time series data is judged from a single label assigned to each time series data, and the degree of smoothing for each time series data is automatically determined. In Fig. 6, the representative value of the label corresponding to each time series data is used as the label for each time series data, and each label indicates a positive example or a negative example. 6, when each labeled time series data is given, the lossy compression device 9 calculates a probability distribution of events corresponding to each label, judges the utility of each labeled time series data using the calculated probability distribution, and automatically determines the degree of smoothing of each time series data based on the judgment result. Note that the probability distribution may be given in advance as domain knowledge 13.

[0028] Figure 7 shows how the usefulness of time-series data is judged from multiple labels and the degree of smoothing is automatically determined. In Figure 7, time-series data is prepared for each device, and each piece of time-series data is data indicating an event corresponding to each piece of time-series data. The occurrence probability of each event may differ for each device. In the example shown in FIG. 7, the lossy compression device 9 calculates the probability distribution of each event for each device, and automatically determines the degree of smoothing of each time series data by using the calculated probability distribution.

[0029] Figure 8 shows how the usefulness of time series data is judged from the frequency distribution of data points, and the degree of smoothing is automatically determined. In Figure 8, no labels are attached to each piece of time series data. In the example shown in Fig. 8, the lossy compression device 9 calculates a probability distribution of data points based on collected time series data, and automatically determines the degree of smoothing for each time series data using the calculated probability distribution. The probability distribution indicates the occurrence probability of each data point. The probability distribution may be given in advance as domain knowledge 13.

[0030] 9 shows an example of the hardware configuration of a lossy compression device 9 according to this embodiment. The lossy compression device 9 is made up of a computer. The lossy compression device 9 may be made up of multiple computers. The hardware configurations of the other components constituting the data storage system 90 may be similar to the hardware configuration of the lossy compression device 9.

[0031] As shown in the figure, the lossy compression device 9 is a computer including hardware such as a processor 51, a memory 52, an auxiliary storage device 53, an input / output IF (Interface) 54, and a communication device 55. These pieces of hardware are appropriately connected via signal lines 59.

[0032] The processor 51 is an integrated circuit (IC) that performs arithmetic processing and controls hardware included in a computer. Specific examples of the processor 51 include a central processing unit (CPU), a digital signal processor (DSP), or a graphics processing unit (GPU). The lossy compression device 9 may include a plurality of processors that replace the processor 51. The plurality of processors share the role of the processor 51.

[0033] The memory 52 is typically a volatile storage device, and a specific example is a RAM (Random Access Memory). The memory 52 is also called a primary storage device or a main memory. Data stored in the memory 52 is saved in the auxiliary storage device 53 as necessary.

[0034] The auxiliary storage device 53 is typically a non-volatile storage device, and specific examples thereof include a ROM (Read Only Memory), an HDD (Hard Disk Drive), or a flash memory. Data stored in the auxiliary storage device 53 is loaded into the memory 52 as necessary. The memory 52 and the auxiliary storage device 53 may be integrated into one unit.

[0035] The input / output IF 54 is a port to which an input device and an output device are connected. A specific example of the input / output IF 54 is a Universal Serial Bus (USB) terminal. Specific examples of the input device are a keyboard and a mouse. A specific example of the output device is a display.

[0036] The communication device 55 is a receiver and a transmitter, and is, for example, a communication chip or a network interface card (NIC).

[0037] Each unit of the lossy compression device 9 may use the input / output IF 54 and the communication device 55 as appropriate when communicating with other devices.

[0038] The auxiliary storage device 53 stores a data storage program. The data storage program is a program that causes a computer to realize the functions of each unit of the lossy compression device 9. The data storage program is loaded into the memory 52 and executed by the processor 51. The functions of each unit of the lossy compression device 9 are realized by software.

[0039] Data used when executing the data storage program and data obtained by executing the data storage program are appropriately stored in a storage device. Each part of the lossy compression device 9 uses a storage device as appropriate. As a specific example, the storage device is composed of at least one of the memory 52, the auxiliary storage device 53, a register in the processor 51, and a cache memory in the processor 51. Note that the terms "data" and "information" may have the same meaning. The storage device may be independent of the computer. The functions of the memory 52 and the auxiliary storage device 53 may be realized by other storage devices.

[0040] The data storage program may be recorded in a computer-readable non-volatile recording medium. Specific examples of the non-volatile recording medium include an optical disk and a flash memory. The data storage program may be provided as a program product.

[0041] ***Explanation of Operation*** The operation procedure of data storage system 90 corresponds to a data storage method, and the program that realizes the operation of data storage system 90 corresponds to a data storage program.

[0042] 10 is a flowchart showing an example of the processing of the data receiving unit 7. The processing of the data receiving unit 7 will be described with reference to FIG.

[0043] (Step S101) The data receiving unit 7 receives the time series data 5 from each sensor via the network 6 .

[0044] (Step S102) The data receiving unit 7 outputs the received time series data 5 to a data interpolation unit 10 .

[0045] (Step S103) The data receiving unit 7 outputs the received time series data 5 to a long-term storage database 8 .

[0046] 11 is a flowchart showing an example of the process of the data interpolation unit 10. The process of the data interpolation unit 10 will be described with reference to FIG.

[0047] (Step S111) The data interpolation unit 10 receives the time series data 5 from the data receiving unit 7. The data interpolation unit 10 may receive a plurality of pieces of time series data 5. When the data interpolation unit 10 receives a plurality of pieces of time series data 5, the data interpolation unit 10 executes the following process for each piece of the received time series data 5.

[0048] (Step S112) If the sampling rate of the received time series data 5 is lower than the predetermined threshold, the data interpolation unit 10 proceeds to step S113, otherwise the data interpolation unit 10 proceeds to step S115.

[0049] (Step S113) The data interpolation unit 10 selects a sampling rate that is equal to or greater than the sampling rate of the received time series data 5 .

[0050] (Step S114) The data interpolator 10 generates interpolated time series data 11 by up-sampling the received time series data 5 using a selected sampling rate and a data interpolation function.

[0051] (Step S115) The data interpolation unit 10 converts the received time series data 5 into interpolated time series data 11.

[0052] (Step S116) The data interpolation unit 10 outputs the interpolated time series data 11 to the data value calculation unit 12 .

[0053] (Step S117) The data interpolation unit 10 outputs the interpolated time series data 11 to the data smoothing unit 22 .

[0054] 12 is a flow chart showing an example of the process of the data value calculation unit 12 when a label is used. The process of the data value calculation unit 12 will be explained with reference to FIG.

[0055] (Step S121) The data value calculation unit 12 receives the interpolated time series data 11 from the data interpolation unit 10. The data value calculation unit 12 may receive a plurality of interpolated time series data 11. When the data value calculation unit 12 receives a plurality of interpolated time series data 11, it executes the following process for each of the received interpolated time series data 11.

[0056] (Step S122) The data value calculation unit 12 receives domain knowledge 13 from the outside.

[0057] (Step S123) The data value calculation unit 12 labels the events indicated by the interpolated time series data 11 based on the received domain knowledge 13 .

[0058] (Step S124) The data value calculation unit 12 receives a probability distribution 15 indicating the occurrence probability of each label from the probability distribution storage unit 14.

[0059] (Step S125) If the data value calculation unit 12 is able to acquire the probability distribution 15, the data value calculation unit 12 proceeds to step S127. Otherwise, the data value calculation unit 12 proceeds to step S126.

[0060] (Step S126) The data value calculation unit 12 acquires a probability distribution 15 from the domain knowledge 13 .

[0061] (Step S127) The data value calculation unit 12 specifies, as the probability 16, the minimum value of the occurrence probability corresponding to each label associated with the interpolated time series data 11 based on the probability distribution 15.

[0062] (Step S128) The data value calculation unit 12 recalculates the occurrence probability of each label based on the probability distribution 15 and each label corresponding to the interpolated time series data 11, and generates a probability distribution 17 based on the probability distribution 15 and the recalculated occurrence probability.

[0063] (Step S129) The data value calculation unit 12 outputs the generated probability distribution 17 to the probability distribution storage unit 14.

[0064] (Step S130) The data value calculation unit 12 outputs the probability 16 to the smoothness determination unit 18 .

[0065] 13 is a flow chart showing an example of the process of the data value calculation unit 12 when using the occurrence probability of a data point. The process of the data value calculation unit 12 will be explained with reference to FIG.

[0066] (Step S131) This step is similar to step S121.

[0067] (Step S132) The data value calculation unit 12 receives a probability distribution 15 indicating the occurrence probability of each data point indicated by the interpolated time series data 11 from the probability distribution storage unit 14 .

[0068] (Step S133) If the data value calculation unit 12 is able to acquire the probability distribution 15, the data value calculation unit 12 proceeds to step S135. Otherwise, the data value calculation unit 12 proceeds to step S134.

[0069] (Step S134) The data value calculation unit 12 estimates a probability density function using kernel density estimation or the like based on the received interpolated time series data 11 , and sets the estimated probability density function as a probability distribution 15 .

[0070] (Step S135) The data value calculation unit 12 specifies, as the probability 16, the minimum value of the occurrence probability of each data point indicated by the interpolated time series data 11 based on the probability distribution 15.

[0071] (Step S136) The data value calculation unit 12 recalculates the occurrence probability of each data point based on the probability distribution 15 and the interpolated time series data 11, and generates a probability distribution 17 based on the probability distribution 15 and the recalculated occurrence probability.

[0072] (Step S137) This step is similar to step S129.

[0073] (Step S138) This step is similar to step S130.

[0074] 14 is a flowchart showing an example of the process of the smoothness determining unit 18. The process of the smoothness determining unit 18 will be described with reference to FIG.

[0075] (Step S141) The smoothness determining unit 18 receives the probability 16 from the data value calculating unit 12 .

[0076] (Step S142) The smoothness determining unit 18 receives domain knowledge 13 from the outside.

[0077] (Step S143) The smoothness determiner 18 determines a smoothness 21 corresponding to the probability 16 based on the received domain knowledge 13 .

[0078] (Step S144) The smoothness determining unit 18 outputs the specified smoothness 21 to the data smoothing unit 22 .

[0079] 15 is a flowchart showing an example of the process of the data smoothing unit 22. The process of the data smoothing unit 22 will be described with reference to FIG.

[0080] (Step S151) The data smoothing unit 22 receives the interpolated time series data 11 from the data interpolation unit 10 .

[0081] (Step S152) The data smoothing unit 22 receives the smoothness 21 from the smoothness determining unit 18 .

[0082] (Step S153) The data smoothing unit 22 generates smoothed time series data 23, which is data obtained by smoothing the received interpolated time series data 11, using the received interpolated time series data 11, the received smoothing degree 21, and a moving average or spline smoothing function, etc.

[0083] (Step S154) The data smoothing unit 22 outputs the generated smoothed time series data 23 to a singular point extraction unit 24 .

[0084] 16 is a flowchart showing an example of the process of the singular point extraction unit 24. The process of the singular point extraction unit 24 will be described with reference to FIG.

[0085] (Step S161) The singular point extractor 24 receives the smoothed time series data 23 from the data smoother 22 .

[0086] (Step S162) The singularity extracting unit 24 includes the start point of the smoothed time series data 23 in the singularity data 25 .

[0087] (Step S163) The singularity extraction unit 24 includes "extreme value" in "type". "Type" is the type of singularity to be included in the singularity data 25. "Extreme value" indicates that the singularity data 25 includes local maximum points and local minimum points.

[0088] (Step S164) If it is desired to improve the restoration accuracy, the singularity extraction unit 24 proceeds to step S165, otherwise the singularity extraction unit 24 proceeds to step S166.

[0089] (Step S165) The singular point extraction unit 24 adds "inflection point" to the "type".

[0090] (Step S166) The singularity extraction unit 24 extracts each data point that matches the “type” from the smoothed time series data 23 using a singularity extraction function, and adds each extracted data point to the singularity data 25.

[0091] (Step S167) The singularity extracting unit 24 adds the end point of the smoothed time series data 23 to the singularity data 25 .

[0092] (Step S168) The singularity extraction unit 24 outputs the generated singularity data 25 to a utilization database 26 .

[0093] 17 is a flowchart showing an example of the process of the restoration unit 28. The process of the restoration unit 28 will be described with reference to FIG.

[0094] (Step S171) The restoration unit 28 receives the singularity data 25 from the utilization database 26 .

[0095] (Step S172) The restoration unit 28 receives a sampling rate 29 from an external source.

[0096] (Step S173) The restoration unit 28 generates restored time series data 30 using the singularity data 25, a sampling rate 29, and a data interpolation function such as Akima interpolation or spline interpolation.

[0097] (Step S174) The restoration unit 28 outputs the generated restored time series data 30.

[0098] ***Explanation of the Effects of the First Embodiment*** Generally, common events are far more common than rare events, so according to this embodiment, data corresponding to common events is compressed at a high compression rate and data corresponding to rare events is compressed at a low compression rate, thereby making it possible to significantly reduce the amount of data while retaining the characteristics of rare events. Furthermore, according to this embodiment, when restoring data, the restoration unit 28 can interpolate data at any granularity with each singular point as a reference. Therefore, according to this embodiment, data can be restored in accordance with a sampling rate required for AI learning. It is expected that the accuracy of the model will improve if the restoration unit 28 restores data in accordance with a high sampling rate. In addition, in this embodiment, if the singular points are only extreme values, the amount of data can be further reduced. Here, if the singular points include extreme values ​​and inflection points, the accuracy of data restoration can be improved. Therefore, according to this embodiment, the data size and restoration accuracy can be appropriately selected according to the use case of data utilization.

[0099] ***Other configurations*** <Variation 1> FIG. 18 shows an example of the hardware configuration of a lossy compression device 9 according to this modified example. The lossy compression device 9 includes a processing circuit 58 in place of the processor 51 , the processor 51 and a memory 52 , the processor 51 and an auxiliary storage device 53 , or the processor 51 , the memory 52 , and the auxiliary storage device 53 . The processing circuit 58 is hardware that realizes at least a part of each unit included in the lossy compression device 9 . The processing circuitry 58 may be dedicated hardware, or may be a processor that executes programs stored in the memory 52 .

[0100] When processing circuitry 58 is dedicated hardware, processing circuitry 58 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The lossy compression device 9 may include a plurality of processing circuits that replace the processing circuit 58. The plurality of processing circuits share the role of the processing circuit 58.

[0101] In the lossy compression device 9, some of the functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0102] Processing circuitry 58 is illustratively implemented in hardware, software, firmware, or a combination thereof. The processor 51, the memory 52, the auxiliary storage device 53, and the processing circuit 58 are collectively referred to as the “processing circuitry.” In other words, the functions of the functional components of the lossy compression device 9 are realized by the processing circuitry. The hardware configurations of the other components constituting the data storage system 90 may be the same as those of this modified example. The lossy compression device 9 according to other embodiments may also be the same as that of this modified example.

[0103] Embodiment 2 The following mainly describes the differences from the above-described embodiment with reference to the drawings.

[0104] ***Configuration Description*** FIG. 19 shows an example of the configuration of a data storage system 90 according to this embodiment. The smoothness determination unit 18 according to this embodiment determines the target smoothness using data indicating the relationship between the rarity of the event indicated by the target data and the smoothness. Specifically, the smoothness determination unit 18 does not receive the domain knowledge 13, but determines the smoothness 21 using the probability 16 and a smoothness determination function 19 or a smoothness determination table 20. Each of the smoothness determination function 19 and the smoothness determination table 20 corresponds to data indicating the relationship between the rarity of the event indicated by the target data and the smoothness. Note that the domain knowledge 13 may indicate data indicating the relationship between the rarity of the event indicated by the target data and the smoothness.

[0105] The smoothness determination function 19 is a function that indicates the relationship between the rarity of an event and the smoothness. As a specific example, the smoothness determination function 19 is a function that indicates the relationship between the occurrence probability of an event and the smoothness, or a function that indicates the relationship between a label corresponding to an event and the smoothness. When the smoothness determination function 19 is a function that indicates the relationship between the occurrence probability of an event and the smoothness, the smoothness determination function 19 may be any function that monotonically increases. 20(a) shows two specific examples of the smoothness determination function 19. In the smoothness determination function 19, the lower the occurrence probability, the weaker the corresponding smoothness, and the higher the occurrence probability, the stronger the corresponding smoothness.

[0106] The smoothness determination table 20 is a table showing the relationship between the rarity of an event and the smoothness. As a specific example, the smoothness determination table 20 is a table showing the relationship between the occurrence probability of an event and the smoothness, or a table showing the relationship between a label corresponding to an event and the smoothness. 20(b) shows a specific example of the smoothness decision table 20. Note that the smoothness decision table 20 may be a table showing labels instead of probabilities.

[0107] ***Explanation of Operation*** 21 is a flowchart showing an example of the process of the smoothness determining unit 18. The process of the smoothness determining unit 18 will be described with reference to FIG.

[0108] (Step S201) The smoothness determination unit 18 receives a smoothness determination function 19 or a smoothness determination table 20 from the outside.

[0109] (Step S202) The smoothness determination unit 18 determines a smoothness 21 corresponding to the probability 16 based on the received smoothness determination function 19 or smoothness determination table 20 .

[0110] ***Explanation of the effect of the second embodiment*** As described above, according to this embodiment, the smoothness is specified using the smoothness determination function 19 or the smoothness determination table 20, so there is no need to determine the smoothness from domain knowledge for each time-series data, and the smoothness can be seamlessly and automatically determined according to the value of the data.

[0111] Embodiment 3 The following mainly describes the differences from the above-described embodiment with reference to the drawings.

[0112] ***Configuration Description*** The configuration example of the data storage system 90 according to this embodiment is similar to the configuration example of the data storage system 90 according to the above-described embodiment. The data smoothing unit 22 according to this embodiment smoothes the target data using a weight that takes a larger value as the event indicated by the target data becomes rare. Specifically, the data smoothing unit 22 uses a weighted moving average in the smoothing. The weighted moving average is also called a weighted moving average. The weight is used to increase the value of data points with a low probability of occurrence. As a specific example, the weight is the inverse of the probability of occurrence. [Mathematical formula 2] shows a specific example of the weighted moving average. Here, PDF is the probability density function of the interpolated time series data 11, and x i are the sample points (i=1,…,n) within the window in the moving average.

[0113]

number

[0114] Figure 22 shows how weighting is done taking into account the value of the data points. The left side of Figure 22 shows the window in the moving average, and the right side of Figure 22 shows the probability distribution showing the occurrence probability of each data point. In a normal moving average, the value of the window as a whole may become low when high-value data and low-value data are mixed within the window, as shown on the left side of Fig. 22. On the other hand, in a weighted moving average where the weight is the inverse of the occurrence probability, the value of the window as a whole is unlikely to become low when high-value data and low-value data are mixed within the window.

[0115] ***Explanation of Operation*** The difference between the operation of the data storage system 90 according to this embodiment and the operation of the data storage system 90 according to the previous embodiment will be described. In step S153, the data smoothing unit 22 uses a weighted moving average instead of a moving average or a spline smoothing function.

[0116] ***Explanation of the Effects of the Third Embodiment*** When a simple moving average is used in smoothing, the value of the sample points within the window is not fully taken into consideration. On the other hand, according to this embodiment, a weighted moving average is used instead of a simple moving average, so the value of the sample points within the window is taken into consideration. Therefore, according to this embodiment, the features of the data are more likely to remain even if the smoothing is strong.

[0117] Embodiment 4 The following mainly describes the differences from the above-described embodiment with reference to the drawings.

[0118] ***Configuration Description*** FIG. 23 shows an example of the configuration of a data storage system 90 according to this embodiment. The cloud system 1 according to this embodiment includes a utilization database 31, a utilization database 32, and a utilization database 33.

[0119] The utilization database 31, the utilization database 32, and the utilization database 33 correspond to multiple utilization databases that are candidates for storing the singularity data 25. The computational resources of each of the multiple utilization databases may be different from each other. Each of the utilization databases 31, the utilization database 32, and the utilization database 33 is similar to the utilization database 26.

[0120] The singularity extraction unit 24 according to this embodiment selects a utilization database from among a plurality of utilization databases as a storage destination of the singularity data 25. At this time, the singularity extraction unit 24 stores the singularity data 25 in a utilization database having more computational resources as the utilization value of the singularity data 25 is higher. As a specific example, the singularity extraction unit 24 distributes the singularity data 25 to the utilization database 31, the utilization database 32, and the utilization database 33 and outputs them. The number of databases to which the singularity extraction unit 24 outputs is not limited to three. At this time, as a specific example, the singularity extraction unit 24 distributes the singularity data 25 according to the frequency of use of the singularity data, the cumulative number of search hits for the singularity data, or the value (probability 16) of the time-series data corresponding to the singularity data 25. It is presumed that the greater the corresponding cumulative number of search hits, the higher the utility value of the data.

[0121] FIG. 24 is a diagram for explaining the process of allocating the singularity data 25 according to the cumulative search hit count. As a specific example, the singularity extraction unit 24 allocates the singularity data 25 so that data with a cumulative search hit count of 100 or more is stored in the utilization database 31, data with a cumulative search hit count of 51 to 99 is stored in the utilization database 32, and data with a cumulative search hit count of 50 or less is stored in the utilization database 33. In this example, more computational resources such as CPU and memory resources are allocated to the utilization database 31 so as to increase the search speed. Similarly, less computational resources than those of the utilization database 31 are allocated to the utilization database 32, and less computational resources than those of the utilization database 32 are allocated to the utilization database 33. By allocating computational resources in this manner, it is possible to reduce the operating costs of the database while improving the response performance to searches for data that is used relatively frequently.

[0122] ***Explanation of Operation*** The difference between the operation of the data storage system 90 according to this embodiment and the operation of the data storage system 90 according to the previous embodiment will be described. In step S168, the singularity extraction unit 24 appropriately distributes the singularity data 25 among the utilization database 31, the utilization database 32, and the utilization database 33.

[0123] ***Explanation of the effect of the fourth embodiment*** As described above, according to this embodiment, by preparing a plurality of types of databases and allocating the singularity data 25 to each database according to the value of the singularity data 25, it is possible to improve the response performance to searches for data that is used with a relatively high frequency while reducing the operating costs of the databases.

[0124] Embodiment 5. The following mainly describes the differences from the above-described embodiment with reference to the drawings.

[0125] ***Configuration Description*** Fig. 25 shows a configuration example of a data storage system 90 according to this embodiment. Note that Fig. 25 shows an excerpt of a characteristic configuration of this embodiment. The configuration of the cloud system 1 etc. according to this embodiment is basically as described above. The data storage system 90 includes a client device 34. The client device 34 includes a temporary storage area 35, a data receiving unit 36, a data receiving unit 38, a data value calculation unit 40, and a data storage determination unit 44. The client device 34 is communicatively connected to the cloud system 1, and is communicatively connected to the lossy compression device 9 via the data receiving unit 37. The cloud system 1 according to the present embodiment includes a data receiving unit 37. Each of the data receiving units 36 , 37 , and 38 is similar to the data receiving unit 7 .

[0126] In this embodiment, for the purpose of backup in preparation for a temporary network interruption, etc., time-series data with a relatively high utility value is given priority and temporarily stored in the client device 34. The client device 34 acquires in advance the utility value (probability distribution of data points, etc.) of the time-series data calculated on the cloud system 1 side, and determines whether or not to store the acquired time-series data 5 at the time of acquiring the time-series data 5. The client device 34 periodically transmits the time-series data 5 measured by the sensor 2 to the cloud system 1 via the network 6. It should be noted that the number of sensors is not limited to one.

[0127] The temporary storage area 35 stores the temporarily stored data in association with the rarity of the event indicated by the temporarily stored data. Specifically, the temporary storage area 35 stores the time series data 5 and the probability 41. The time series data 5 corresponds to the temporarily stored data.

[0128] The data value calculation unit 40 receives the time series data 5 from the sensor 2. The data value calculation unit 40 also receives the probability distribution 39 indicating the occurrence probability of each data point indicated by the time series data 5 from the probability distribution storage unit 14, and identifies the probability corresponding to the data point with the lowest occurrence probability among the data points indicated by the time series data 5 based on the received probability distribution 39, and sets the identified probability as probability 41. Thereafter, the data value calculation unit 40 outputs the time series data 5 and the probability 41 to the data storage determination unit 44.

[0129] The data storage determination unit 44 determines whether or not to store the target data in the temporary storage area 35 based on a result of comparing the rarity of the event indicated by the target data with the rarity of the event indicated by the temporarily stored data. As a specific example, the data storage determination unit 44 receives, as the time series data 42, the time series data 5 having the highest occurrence probability of the corresponding representative point among the time series data 5 stored in the temporary storage area 35, and also receives, as the probability 43, the probability 41 corresponding to the representative point of the time series data 42. Thereafter, if there is free space in the temporary storage area 35, the data storage determination unit 44 stores the time series data 5 and the probability 41 in the temporary storage area 35. If there is no free space in the temporary storage area 35, the data storage determination unit 44 compares the probability 41 and the probability 43, and if the probability 41<the probability 43, the data storage determination unit 44 deletes the time series data 42 and the probability 43 from the temporary storage area 35 and stores the time series data 5 and the probability 41 in the temporary storage area 35. Furthermore, if the probability 43≦the probability 41, the data storage determination unit 44 discards the time series data 5 and does not store the time series data 5 in the temporary storage area 35.

[0130] ***Explanation of Operation*** 26 is a flowchart showing an example of the process of the data value calculation unit 40. The process of the data value calculation unit 40 will be explained using FIG.

[0131] (Step S401) The data value calculation unit 40 receives the time-series data 5 from the sensor 2 .

[0132] (Step S402) The data value calculation unit 40 receives the probability distribution 39 indicating the occurrence probability of each data point indicated by the time series data 5 from the probability distribution storage unit 14 .

[0133] (Step S403) If the data value calculation unit 40 is able to acquire the probability distribution 39, the data value calculation unit 40 proceeds to step S405. Otherwise, the data value calculation unit 40 proceeds to step S404.

[0134] (Step S404) The data value calculation unit 40 estimates a probability density function using kernel density estimation or the like based on the received time series data 5 , and sets the estimated probability density function as a probability distribution 39 .

[0135] (Step S405) The data value calculation unit 40 specifies, as the probability 41, the minimum value of the occurrence probability of each data point indicated by the time series data 5 based on the probability distribution 39.

[0136] (Step S406) The data value calculation unit 40 outputs the received time-series data 5 and the identified probability 41 to the data storage determination unit 44.

[0137] 27 is a flowchart showing an example of the process of the data storage determination unit 44. The process of the data storage determination unit 44 will be described with reference to FIG.

[0138] (Step S411) The data storage determination unit 44 receives each of the time series data 5 and the probability 41 from the data value calculation unit 40.

[0139] (Step S412) The data storage determination unit 44 receives as time series data 42 the time series data 5 having the highest occurrence probability of the corresponding representative point from among the time series data 5 stored in the temporary storage area 35, and receives as probability 43 the probability 41 corresponding to the representative point of the time series data 42.

[0140] (Step S413) If there is free space in the temporary storage area 35, the data storage determination unit 44 proceeds to step S417. Otherwise, the data storage determination unit 44 proceeds to step S414.

[0141] (Step S414) If Probability 41<Probability 43, the data storage determination unit 44 proceeds to step S416. Otherwise, the data storage determination unit 44 proceeds to step S415.

[0142] (Step S415) The data storage determination unit 44 discards the received time series data 5.

[0143] (Step S416) The data storage determination unit 44 deletes the time series data 42 and the probability 43 from the temporary storage area 35.

[0144] (Step S417) The data storage determination unit 44 outputs the time series data 5 and the probability 41 to the temporary storage area 35.

[0145] ***Explanation of the effect of the fifth embodiment*** As described above, according to this embodiment, data that is relatively useful is preferentially stored in the client device 34, so that the cost-effectiveness of the device can be improved.

[0146] ***Other embodiments*** The above-described embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted. Moreover, the embodiments are not limited to those shown in the first to fifth embodiments, and various modifications are possible as necessary. The procedures explained using the flowcharts and the like may be modified as appropriate. [Explanation of symbols]

[0147] 1 Cloud system, 2,3,4 Sensor, 5 Time series data, 6 Network, 7 Data receiver, 8 Long-term storage database, 9 Lossy compression device, 10 Data interpolation unit, 11 Interpolated time series data, 12 Data value calculation unit, 13 Domain knowledge, 14 Probability distribution storage unit, 15 Probability distribution, 16 Probability, 17 Probability distribution, 18 Smoothing degree determination unit, 19 Smoothing degree determination function, 20 Smoothing degree determination table, 21 Smoothing degree, 22 Data smoothing unit, 23 Smoothed time series data, 24 Singularity extraction unit, 25 Singularity data, 26 Utilization database, 27 Restoration device, 28 Restoration unit, 29 Sampling rate, 30 Reconstructed time series data, 31,32,33 Utilization database, 34 Client device, 35 Temporary storage area, 36,37,38 Data receiver, 39 Probability distribution, 40 Data value calculation unit, 41 Probability, 42 time series data, 43 probability, 44 data storage determination unit, 51 processor, 52 memory, 53 auxiliary storage device, 54 input / output IF, 55 communication device, 58 processing circuit, 59 signal line, 90 data storage system.

Claims

1. A data storage system for storing lossy compressed data, comprising: a smoothness determining unit that determines, as an object smoothness, a smoothness according to the rarity of an event represented by the object data; a data smoothing unit that generates smoothed target data by smoothing the target data with the target smoothness; A lossy compression device comprising: A data storage system comprising:

2. The data storage system further comprises: a data value calculation unit that calculates, as the rarity of the event represented by the target data, either the occurrence probability of a data point represented by the target data or the occurrence probability of a label assigned to the target data; The data storage system according to claim 1 .

3. 3. The data storage system according to claim 1, wherein the smoothness determining unit determines the object smoothness using data indicating a relationship between the rarity of the event indicated by the object data and the smoothness.

4. The data storage system according to claim 1 or 2, wherein the data smoothing unit smoothes the target data using a weight that has a larger value as the event indicated by the target data becomes rare.

5. The lossy compression device further comprises: A singularity extraction unit that extracts a plurality of singularities from the smoothed target data and generates singularity data consisting of the extracted singularities. Equipped with The data storage system further comprises: A number of utilization databases that are candidates for storing the singularity data Equipped with The computing resources of each of the plurality of utilization databases are different from each other, The data storage system of claim 1 or 2, wherein the singularity extraction unit selects a utilization database from among the multiple utilization databases as a storage destination for the singularity data, so that the higher the utility value of the singularity data, the more computing resources the singularity data is stored in the utilization database.

6. The data storage system further comprises: a temporary storage area for storing the temporarily stored data in association with the rarity of the event indicated by the temporarily stored data; a data storage determination unit that determines whether or not to store the target data in the temporary storage area based on a result of comparing the rarity of the event indicated by the target data with the rarity of the event indicated by the temporarily stored data; A client device comprising: Equipped with 3. The data storage system according to claim 1, wherein the lossy compression device and the client device are connected so as to be able to communicate with each other.

7. The data storage system further comprises: A long-term storage database for storing the losslessly compressed target data The data storage system according to claim 1 or 2, comprising:

8. The data storage system according to claim 1 or 2, wherein the target data is time-series data indicating the measurement results of a sensor.

9. A data storage method executed in a data storage system that stores lossy compressed data, comprising: The computer determines the smoothness of the object according to the rarity of the event represented by the object data; A data storage method in which the computer generates smoothed target data by smoothing the target data with the target smoothness.

10. A data storage program executed by a lossy compression device, which is a computer included in a data storage system that stores lossy compressed data, comprising: A smoothness determination process for determining the smoothness of the target data according to the rarity of the event represented by the target data; a data smoothing process for generating smoothed target data by smoothing the target data with the target smoothness; A data storage program that causes the lossy compression device to execute the above.

Images