Storage system and storage control method

Abstract

The present invention provides a storage system that uses a compression decompressor based on machine learning to perform lossy compression of time series data, which can improve the compression rate. In the storage system, taking the original data (the time series data input to the compression decompressor) as the object, for each of more than one statistic, a statistical value is calculated based on one or more parameters, and taking the decompressed data corresponding to the original data (the time series data output from the compression decompressor) as the object, for each of the above-mentioned more than one statistic, a statistical value is calculated based on one or more parameters. The compression decompressor performs machine learning based on the statistical value calculated for each of more than one statistic with the original data as the object, and the statistical value calculated for each of more than one statistic with the decompressed data as the object.

Description

Technical Field

[0001] The present invention generally relates to storage control for compression and decompression of data. Background Art

[0002] Storage systems capable of reducing the amount of data are already known (e.g., Patent Document 1). Such storage systems generally reduce the amount of data by compression. As one of the existing compression methods, a method is known in which a string with a high frequency of occurrence in a predetermined unit block is defined as a dictionary and replaced with a smaller symbol, such as the run-length method.

[0003] As a technology that can reduce the amount of data more than lossless compression such as the run-length method, some lossy compression technologies are known. For example, in Patent Document 2, sensor data is converted into a power spectrum and a phase spectrum, and the power and phase values ​​are excluded from the recording object at the frequency where the power value is less than a preset threshold value, thereby reducing the amount of data.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2007-199891

[0007] Patent document 2: US6529927B1 Summary of the invention

[0008] Technical problem to be solved by the invention

[0009] In the analysis of sensor data, in order to perform statistical analysis, as a necessary condition for accuracy, it is sometimes required to be able to maintain the probability distribution of the data (for example, the mean and variance of the noise, etc.). However, in Patent Document 2, if it is required to maintain the probability distribution as a necessary condition for accuracy, the problem of poor compression rate will occur. For example, it is assumed that the sensor data is data containing white noise. The technology disclosed in Patent Document 2 is to compress the sensor data by discarding the power and phase values ​​in the frequency band with relatively small power when the power spectrum of the sensor data varies with the frequency band. However, for white noise, its power is evenly distributed in all frequency bands. Therefore, in the case where it is necessary to maintain the statistical properties of the sensor data, the technology of Patent Document 2 cannot discard the values ​​of a specific frequency band, and the compression rate will deteriorate.

[0010] In recent years, compression and decompression devices based on machine learning have become known. An example of such a compression and decompression device is an autoencoder. An autoencoder is composed of a neural network, and by learning data in a specific field, it can achieve compression specifically for data in the same field as the learning data.

[0011] Generally, a neural network learns in such a way that the function value of a loss function is reduced. In a neural network constituting a compressor / decompressor, the loss function is typically set as a function (error function) that constructs the error between input and output as an index.

[0012] The inventors of the present application conducted an in-depth study on a storage system that uses an automatic encoder as a comparative example of a compressor and decompressor, and obtained the following findings.

[0013] When the error function set as the loss function is different from the error function that meets the accuracy requirement, information that is not necessary to meet the accuracy requirement will remain in the compression result, resulting in a poor compression rate. For example, in sensor data, consider the case where the accuracy requirement is to maintain the statistic of the average value every 1 second. In order to meet this accuracy requirement, when the sensor data is divided into 1-second windows, it is sufficient as long as the sum of the values ​​in each window is consistent before and after compression, and there is no need to record each value in the window. In this case, if the mean square error is set as the loss function as an example, information that does not affect the accuracy of the statistic (for example, each value in the window) will be recorded in the compression result, resulting in a poor compression rate.

[0014] The above problems also exist when using machine learning-based compressors and decompressors other than autoencoders to lossily compress time series data other than sensor data.

[0015] Technical means of solving problems

[0016] A storage system for lossy compression of time series data using a compressor and decompressor based on machine learning, wherein, with original data (time series data input to the compressor and decompressor), for each of more than one statistic, a statistic value is calculated based on one or more parameters, and with decompressed data corresponding to the original data (time series data output from the compressor and decompressor), for each of the above-mentioned more than one statistic, a statistic value is calculated based on one or more parameters. The compressor and decompressor performs machine learning based on the statistic value calculated for each of the more than one statistic with the original data as the object and the statistic value calculated for each of the more than one statistic with the decompressed data as the object.

[0017] Effects of the Invention

[0018] According to the present invention, information that does not affect the accuracy requirement of the statistic calculated based on one or more parameters (i.e., information other than the information required to calculate the statistic) can be excluded from the compression result (compressed data). Therefore, under the premise of satisfying the same accuracy requirement of the statistic, the compression rate can be improved compared with the comparative example. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram showing the overall structure of a system according to one embodiment.

[0020] Figure 2 This is a diagram of the structure of the statistics parameter setting screen.

[0021] Figure 3 This is a diagram showing the structure of the statistics quality confirmation screen.

[0022] Figure 4 This is a diagram of the RAM structure.

[0023] Figure 5 This is the structural diagram of the statistics parameter management table.

[0024] Figure 6 It is the structural diagram of the statistical quality management table.

[0025] Figure 7 This is a flowchart of the initial setting process.

[0026] Figure 8 This is a flowchart of the first program setting process.

[0027] Fig. 9 This is a flowchart of the second program setting process.

[0028] Fig.10 This is a flow chart of the data writing process.

[0029] Fig.11 is a flowchart of the statistical quality calculation process.

[0030] Fig.12 This is a flowchart of the data reading process.

[0031] Fig.13 This is a schematic diagram showing an example of the outline of a storage system. DETAILED DESCRIPTION

[0032] In the following description, an "interface device" may be one or more interface devices. The one or more interface devices may be at least one of the following.

[0033] One or more I / O (Input / Output) interface devices. An I / O (Input / Output) interface device is an interface device of at least one of an I / O device and a remote display computer. The I / O interface device of the display computer may be a communication interface device. The at least one I / O device may be any one of an input device such as a user interface device such as a keyboard and a pointing device and an output device such as a display device.

[0034] One or more communication interface devices. The one or more communication interface devices may be one or more communication interface devices of the same type (for example, one or more NICs (Network Interface Cards)), or two or more communication interface devices of different types (for example, a NIC and an HBA (Host Bus Adapter)).

[0035] In addition, in the following description, "memory" refers to one or more memory devices, typically a main memory device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0036] In the following description, "persistent storage device" refers to one or more permanent storage devices. The permanent storage device is typically a non-volatile storage device (eg, auxiliary storage device), specifically, a HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0037] In the following description, a "storage device" may be a physical storage device such as a permanent storage device, or a logical storage device associated with the physical storage device.

[0038] In addition, in the following description, "processor" refers to one or more processing devices. The at least one processing device is typically a microprocessing device such as a CPU (Central Processing Unit), but may also be other types of processing devices such as a GPU (Graphics Processing Unit). The at least one processing device may be a single-core device or a multi-core device. The at least one processing device may also be a processor core. The at least one processing device may also be a generalized processing device such as a hardware circuit capable of performing partial or full processing (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0039] In addition, in the following description, expressions such as "xxx table" are often used to explain that information that can be output for input, but the information can be data of any structure. Therefore, "xxx table" can be said to be "xxx information". In addition, in the following description, the structure of each table is just an example, and a table can also be split into two or more tables, and all or part of two or more tables can also be one table.

[0040] In addition, in the following description, sometimes the processing is described with "program" as the main body, but because the program is executed by the processor and appropriately uses the storage device and / or interface device to perform the specified processing, the main body of the processing can also be described as the processor (or a device having the processor). The program can also be installed from a program source to a device such as a computer. The program source can be, for example, a program distribution server or a computer-readable recording medium (for example, a non-transitory medium). In addition, in the following description, two or more programs can be implemented as one program, and one program can also be implemented as two or more programs.

[0041] In addition, in the following description, the "storage system" can be a locally deployed (On-Premise) storage system, a cloud storage system, or one or all of an edge system (for example, an edge system that collects data from multiple sensors) and a core system (for example, a system that receives data collected from multiple sensors from an edge system).

[0042] In addition, in the following description, an autoencoder is used as an example of a "compressor and decompressor", but a compressor and decompressor other than an autoencoder may also be used. An example of a compressor and decompressor may be a data model. The data model may be any one of a data generation rule, a model, and an execution subject. For example, a data model may be a binary string used to express the regularity of formula processing, the shape of a waveform, a probability distribution, etc. The data model may take the original data (the time series data before compression) as input and the decompressed data (the data obtained by decompressing the original data after compression) as output. As a data model, for example, commonly used neural networks, generative models (for example, GMM (Gaussian Mixture Models), HMM (Hidden MarkovMode1), SCFG (Stochastic Context-Free Grammar), GAN (Generative Adversarial Nets) or VAE (Variational Auto Encoder)), genetic programming, etc. may be used. In addition, in order to reduce the amount of information in the data model, model compression such as Mimic Mode1 may also be applied.

[0043] In addition, in the following description, as an example of "time series data", sensor data including multiple sensor data sets corresponding to multiple moments is used. A "sensor data set" can be, for example, a data set containing a sensor ID, a measurement value, and information representing a moment (for example, a measurement moment or an assigned timestamp). A "data set" is a collection of logical electronic data for a program such as an application, and can also be, for example, any of a record, a file, a key-value pair, and a tuple. The present invention can also be applied to time series data other than sensor data. For example, a collection of images taken by a camera can be regarded as time series data. In this case, the data per unit moment is a three-dimensional tensor. In this case, the sensor data set is, for example, a data set containing a camera ID, a three-dimensional tensor representing an image, and information representing the shooting moment.

[0044] Next, embodiments of the present invention will be described based on the drawings. However, the present invention is not limited to the embodiments described below.

[0045] (1) System structure

[0046] First, use Figure 1 A system configuration to which the present invention is applied will be described. Figure 1 The present invention is applicable to a system example. Figure 1 Take the system as an example.

[0047] The storage node 100, the sensor server 120, and the management server 130 are all connected to the switch 140. The switch 140 connects the storage node 100, the sensor server 120, and the management server 130 to each other. The storage node 100, the sensor server 120, and the management server 130 can be independent computers, or virtual computers such as different virtual machines working on the same computer, or can be composed of a combination of them. In addition, the storage node 100, the sensor server 120, and the management server 130 do not necessarily have to be one each, and can also be multiple. In addition, a communication network (for example, a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet) can also be used instead of the switch 140.

[0048] The sensor server 120 is a computer including hardware resources such as a processor, a memory, and an interface device, and software resources such as a sensor driver, etc. The sensor server 120 collects data (sensor data sets) generated by one or more sensors 110 and sends them to the storage node 100 .

[0049] The management server 130 is a computer including hardware resources such as a processor, a memory, an interface device, and local input and output devices, as well as software resources such as a management program. The management server 130 uses the management program to display the local output device (typically a display device). Figure 2 and Figure 3 Furthermore, the management server 130 has a function of responding to a request from the storage node 100 by inputting the statistics parameter setting screen 200 (see Figure 2 ) is configured to send the function of sending the device settings on the storage node. In addition, the management server 130 may not have a local output device, but instead forward the output to other computers connected through the network. In addition, it may not have a local input device, but instead receive input from other computers via the switch 140.

[0050] The storage node 100 includes a front-end interface 101, a processor 102, a cache memory 103, a RAM 104, a back-end interface 107, and a switch 108 that connects them. The front-end interface 101 and the back-end interface 107 are examples of interface devices. The cache memory 103 and the RAM 104 are examples of storage devices (e.g., memory).

[0051] The front-end interface 101 is a device that connects the storage node 100 to the sensor server 120 and the management server 130 via the switch 140 in a communicable manner.

[0052] The processor 102 controls the entire storage node 100 via the switch 108 based on one or more programs recorded in the program area 105 of the RAM 104 and the management information recorded in the management area 106 of the RAM 104. In addition to being a general-purpose computing processor such as a CPU, the processor 102 may also be an accelerator such as a GPU or an FPGA, or a combination thereof.

[0053] The cache memory 103 is a storage device that temporarily stores data to be written before the compression process is completed. The cache memory 103 may be composed of a volatile element such as a DRAM (Dynamic Random Access Memory), or a non-volatile drive such as a HDD or SSD.

[0054] The backend interface 107 is a device that connects the storage node 100 to a nonvolatile drive 150 such as a HDD or SSD in a communicable manner.

[0055] The front-end interface 101, processor 102, cache 103, RAM 104 and back-end interface 107 described above may be formed by a single semiconductor element as an ASIC or FPGA, or may be formed by interconnecting a plurality of separate ICs (Integrated Circuits). In addition, the storage node 100 and the management server 130 may also be components of the storage system, and further, one or more drives 150 may also be components of the storage system.

[0056] (2) Management screen

[0057] Figure 2 1 is a diagram showing a statistic parameter setting screen 200 output to a local output device of the management server 130 .

[0058] The statistical parameter setting screen 200 is an example of a first interface (eg, a first UI (User Interface)), such as a GUI (Graphical User Interface). The statistical parameter setting screen 200 includes a statistical parameter input area 201 and a finish button 204 .

[0059] The statistic parameter input area 201 is a tabular area for the user to input the type and parameter of the statistic. Each row 202 of the statistic parameter input area 201 corresponds to a combination of the type and parameter of the statistic. Each column 203 of the statistic parameter input area corresponds to the type or parameter of the statistic. Taking one combination as an example, for example, as described below.

[0060] In the column 203 of “statistic type”, the type of statistic such as “mean” or “variance” (hereinafter referred to as “target type” in this paragraph) can be input.

[0061] In the “Sensor” column 203 , the ID of the sensor to be used for calculating the statistic belonging to the target category can be input.

[0062] In the “Number of Samples” column 203 , the number of samples of values ​​used when calculating a statistic belonging to the target category can be input.

[0063] In the “Calculation frequency” column 203 , it is possible to input a time interval for calculating a statistic belonging to the target category.

[0064] Parameters such as "number of samples" or "computation frequency" are just examples, so you can add Figure 2Parameter items other than the illustrated parameter items may be omitted, and at least one of the illustrated parameter items may be omitted. In addition, the statistic parameter input area 201 may be implemented by an interface such as a drop-down list or radio button in addition to an interface for the user to specify the type or parameter of the statistic in text form. In addition, the statistic parameter setting screen 200 may also allow the user to specify the type or parameter of the statistic by using a data description language such as JSON (JavaScript Object Notation) (JavaScript is a registered trademark).

[0065] The completion button 204 is a button for receiving a notification from a user (eg, a system user) that the input to the statistics parameter input area 201 has been completed.

[0066] Figure 3 1 is a diagram showing a statistic quality confirmation screen 300 output to a local output device of the management server 130 .

[0067] The statistics quality confirmation screen 300 is an example of a second interface (e.g., a second UI), such as a GUI (Graphical User Interface). The statistics quality confirmation screen 300 is used to display the size of the error caused by compression to the user for each statistic specified in the statistics parameter input area 201. The statistics quality confirmation screen 300 includes a data ID input area 301, a display button 302, and a statistics quality area 303.

[0068] The data ID input area 301 is an input area for specifying the object data for which the statistical error is to be displayed. In the data ID input area 301, the interval of the time when the sensor 110 generates data can be specified, and the number indicating the number of times the sensor server 120 sends data to the storage node 100 can be specified, and any other information that can determine the range of the sensor data (an example of time series data) can be specified.

[0069] The function of the display button 302 is to allow the user to confirm the information input into the data ID input area 301. When the display button 302 is pressed, the management server 130 sends the information input into the data ID input area 301 to the storage node 100, and sets the value shown in the response of the storage node 100 into the statistical quality area 303.

[0070] The rows 304 of the statistic quality area 303 correspond to the combination of the type of statistic and the parameter, and thus correspond one-to-one to the rows 202 of the statistic parameter input area 201. The columns 305 of the statistic quality area 303 correspond to the errors (e.g., various errors such as mean square error and maximum error) of the statistics of the sensor data determined based on the information input to the data ID input area 301. In addition, the indexing of the statistic is not limited to the mean square error and maximum error shown as examples, and any index may be added.

[0071] (3) RAM structure

[0072] Figure 4 It is a diagram showing the structure of RAM104.

[0073] RAM 104 includes a program area 105 and a management area 106 , wherein program area 105 stores programs executed by processor 102 , and management area 106 stores management information that is read and written by processor 102 according to program commands.

[0074] One or more programs stored in the program area 105 include, for example, an initial setting action program 400, a statistical setting program 401, a loss setting program 402, a data writing program 403, a data reading program 404, a statistical quality calculation program 405, a specified statistical calculation program 406, a statistical calculation program 407, a loss function program 408, a database program 409 and a compression and decompression program 420.

[0075] The initial setting operation program 400 performs initialization of management information. The statistics setting program 401 performs settings of a designated statistics calculation program 406. The loss setting program 402 performs settings of a loss function program 408.

[0076] The data writing program 403 performs learning of a compressor and decompressor, compression of write target data, and writing of the compression result (compressed data) into the nonvolatile drive 150 in response to a request from the sensor server 120 .

[0077] In response to a request from the sensor server 120 or the management server 130 , the data reading program 404 reads the compression result from the nonvolatile drive 150 , decompresses the data, and transmits the decompressed data as a response.

[0078] The statistics quality calculation program 405 calculates the error generated in the data due to the compression and decompression process, with respect to the statistics designated by the user via the statistics parameter setting screen 200 .

[0079] The designated statistic calculation program 406 is a program that calculates the value of a statistic designated by the user via the statistic parameter setting screen 200 for data provided as input to the program.

[0080] The designated statistic calculation program 406 is set by the initial setting operation program 400 based on the statistic designated by the user on the statistic parameter setting screen 200 .

[0081] The statistics calculation program 407 is a collection of programs for calculating statistics such as mean, variance, median, etc. These programs, as arguments, not only obtain the sensor data of the object for calculating the statistics, but also obtain parameters such as the object sensor, number of samples, and calculation frequency. Usually, the size of the sensor data (for example, the number of sensor data sets constituting the sensor data) is larger than the number of samples when calculating the statistics. Therefore, at least one statistics calculation program 407 divides the sensor data into windows according to the specified number of samples and calculates statistics for each window. Therefore, multiple statistical values ​​can be calculated from one sensor data. Therefore, the return value of the statistics calculation program 407 is usually a tensor composed of statistics. In addition, the calculation of statistics is not limited to data at different times. For example, when the data at each moment of the time series data is tensor data such as an image, statistics can also be calculated for the data of the same tensor. In addition, each tensor can also be divided into smaller tensors, and then the statistics are calculated for each sub-tensor.

[0082] The loss function program 408 is a program for calculating the value of the loss function used for learning the neural network using the statistic calculated by the designated statistic calculation program 406. The loss function program 408 is set by the initial setting operation program 400 at the time of initialization.

[0083] The database program 409 is a program of database software for managing compression results. The database program 409 may include a function of distributing and writing compression results to multiple non-volatile drives 150, and a function of distributing and saving data among multiple storage nodes 100 when there are multiple storage nodes 100. In addition, the database program 409 may provide a function and an API (Application Programming Interface) for providing data storage and reading to other programs.

[0084] The compression / decompression program 420 provides the following functions: a function of learning a neural network constituting a compression / decompression device for sensor data based on specified learning data and a loss function, a function of compressing sensor data using the learned compression / decompression device, and a function of decompressing sensor data from compressed data. The compression / decompression program 420 may also include a compression / decompression device.

[0085] The management information stored in the management area 106 includes, for example, a statistic parameter management table 410 and a statistic quality management table 411. In addition, these tables may be implemented by data structures other than tables, such as a hash table or a binary tree.

[0086] Figure 5 2 is a diagram showing the statistic parameter management table 410 .

[0087] The statistics parameter management table 410 stores the parameters of the statistics set by the user via the management server 130. Each row 501 of the statistics parameter management table 410 corresponds to each statistic specified by the user on the statistics parameter setting screen 200 of the management server 130. The column 502 of the statistics parameter management table 410 stores the type of statistics specified by the user on the statistics parameter setting screen 200 of the management server 130. The column 503 of the statistics parameter management table 410 corresponds to the parameters of the statistics specified by the user on the statistics parameter setting screen 200 of the management server 130 (for example, the calculation target sensor, the number of samples when calculating the statistics, the calculation frequency of the statistics, etc.).

[0088] Figure 6 It is a diagram showing the statistic quality management table 411 .

[0089] The statistics quality management table 411 is a table for storing the calculation results of the statistics quality calculation program. Each row 601 of the statistics quality management table 411 corresponds to each statistic specified by the user on the management server 130. In addition, each column 602 of the statistics quality management table corresponds to an index for evaluating the error of the statistic, such as the mean square error or the maximum error, which is preset in the storage node 100. The index is not limited to the mean square error or the maximum error, and any index calculated by a function that calculates a scalar value with two tensors as input can be used.

[0090] (4) Overview of initial setting operation of storage node 100

[0091] Figure 7 This is a flowchart of the initial setting process.

[0092] The initial setting operation program 400 is started by the processor 102 when the storage node 100 is started for the first time, when the user presses the reset button of the storage node 100, or when the management server 130 issues an instruction. (S700)

[0093] In S701, the processor 102 instructs the management server 130 to display the statistics parameter setting screen 200 on the local output device via the switch 108, the front-end interface 101, and the switch 140. After receiving the instruction, the management server 130 outputs the statistics parameter setting screen 200 to the local output device, allowing the user to input the type and parameters of the statistics.

[0094] In S702, the processor 102 requests the management server 130, via the switch 108, the front-end interface 101, and the switch 140, the value input into the statistic parameter input area 201, and receives the response value. In response to the request of the storage node 100, if the user has pressed the completion button 204, the management server 130 sends the parameters in the statistic parameter input area 201 to the storage node 100 as a response. In the case where the completion button 204 is not pressed by the user, the management server 130 waits until the button is pressed, and once the button is pressed, the parameters in the statistic parameter input area 201 are responded to the storage node 100. In addition, S702 is a step based on the pull-type information collection in which the processor 102 requests the management server 130 for parameters, but it can also replace the pull-type information collection, and adopt the push-type information collection in which the management server 130 actively sends the parameters to the storage node 100. In this case, the processor 102 can standby before the front-end interface 101 receives the parameters sent by the management server 130.

[0095] In S703 , the processor 102 stores the parameters acquired in S702 in the statistics parameter management table 410 .

[0096] In S704 , the processor 102 calls the statistics setting program 401 .

[0097] In S705 , the processor 102 calls the loss setting program 402 .

[0098] After S705 , the processor 102 ends the initial setting operation program 400 ( S706 ).

[0099] Figure 8 This is a flowchart of the first program setting process.

[0100] The first program setting process is performed by the statistics setting program 401. The statistics setting program 401 is Figure 7 The process starts with the call of S704 (S800).

[0101] In S801 , the processor 102 writes a program indicating “no operation (NOP)” in a designated program area (not shown) (an area in the RAM 104 ) to initialize the designated program area.

[0102] S802 to S806 are executed for each row 501 of the statistics parameter management table 410. Figure 8 In the description of , it is called "loop target row").

[0103] In S803 , the processor 102 identifies the type of statistics based on the column 502 of the loop target row of the statistics parameter management table 410 .

[0104] In S804 , the processor 102 acquires the parameter of the statistic from the column 503 of the loop target row of the statistic parameter management table 410 .

[0105] In S805 , the processor 102 records “Processing: Calling the statistic calculation program 407 for calculating the statistic value belonging to the statistic type determined in S803 with the parameter obtained in S804 as an argument” in the program in the designated program area (the program written in S801 ).

[0106] In S807 , the processor 102 records “Processing: End the designated statistic calculation program 406 with the statistic value group (one or more statistic values) calculated in S805 as a return value” in the program in the designated program area (the program after the additional recording in S805 ).

[0107] After S807, the processor 102 ends the statistics setting program 401 (S808).

[0108] Thus, the designated statistical quantity calculation program 406 is created in the designated program area.

[0109] Fig. 9 This is a flowchart of the second program setting process.

[0110] The second program setting process is performed by the loss setting program 402. The loss setting program 402 is Figure 7 The process starts with the call of S705 (S900).

[0111] In S901 , the processor 102 writes a program indicating “no operation (NOP)” in the loss function program area (an area in the RAM 104 ) to initialize the loss function program area.

[0112] In S902 , the processor 102 records “Processing: Using the input data of the compressor / decompressor as an argument, calling the designated statistic calculation program 406 ” additionally to the program in the loss function program area (the program written in S901 ).

[0113] In S903, the processor 102 records "Processing: Using the output data of the compressor / decompressor as an argument, calling the designated statistic calculation program 406" in the program in the loss function program area (the program after the additional recording in S902).

[0114] In S904, the processor 102 appends the record "Processing: Let the statistical value group calculated by the processing of the additional record in S902 be statistical value group A, and the statistical value group calculated by the processing of the additional record in S903 be statistical value group B, and one or more errors contained in the error group between them be indexed respectively" to the program in the loss function program area (the program after the additional record in S903). As an example of an indexing method, the mean square error between statistical value group A and statistical value group B can be listed, but the indexing method is not limited to this, as long as it is a function that can calculate a scalar value based on two statistical groups composed of multiple statistics, any method can be used. In addition, the indexing method does not need to be statically set in the storage node 100, and the management server 130 can allow the user to specify it through the statistic parameter setting screen 200. For example, for each statistic n, the user may be allowed to specify the error indexing method f_n and its target value g_n through the statistic parameter setting screen 200, and for the statistic value a_n of the statistic value group A and the statistic value b_n of the statistic value group B, the error is indexed using the specified method (f_n(a_n,b_n)), and the maximum value max(f_n(a_n,b_n),g_n) between the value and the target value g_n is obtained, and the sum of the obtained value to all statistics Sum(max(f_n(a_n,b_n),g_n)) is determined as the final error index. In this case, even if the compressor / decompressor is learned in such a way that the error index f_n(a_n,b_n) of each statistic is less than the target value g_n, the value of the loss function does not decrease. Therefore, the neural network constituting the compressor and decompressor is learned in such a way that, instead of improving the statistic whose error index is already less than the target value, the indexed value f_n(a_n,b_n) of the error of other statistics or statistics n at other moments is improved. In this way, the error fluctuation caused by the moment or feature value can be suppressed, and the proportion of moments and statistics that meet the error target specified by the user can be increased. In addition, each statistic value a_n and b_n can also be a vector with a length in the time direction corresponding to each statistic type, and can also be its value at each moment. In addition, when the data at each moment of the time series data is tensor data such as an image, the tensor can also be decomposed into smaller tensors, and the error is calculated for each sub-tensor, and the maximum value between the calculated value and the target value is used. For example, for time series data composed of multiple images, each image can be divided into image blocks (patch), and the image quality (for example, peak signal-to-noise ratio or multi-scale structured similarity) between the image block a_n of the original image and the image block b_n of the compressed and decompressed image is used as f_n for each image block.At this time, by using the loss function defined by the above method, the compressor and decompressor can be learned so that the image quality between images and between image blocks within an image is uniformly located near the target value.

[0115] In addition, in the loss function, in addition to the term representing the error, a term for improving the compression rate is sometimes added, such as a term representing the entropy of the output value of the compressor. In this case, the trade-off between the compression rate and the error can be adjusted by the ratio of the error term to the compression rate term, that is, the coefficient multiplied by the compression rate term (hereinafter referred to as the compression coefficient). Usually, the compression coefficient is set manually, but one embodiment of the present invention controls the compression coefficient in such a way that, in the case of a statistic with an error less than a target value, the compression coefficient is set to a smaller value, and in other cases, the compression coefficient is set to a larger value. Thus, before the error meets the target, learning is performed to prioritize minimizing the error, and after the target is achieved, learning is performed to prioritize improving the compression rate, so that the best compression decompressor for the user in the trade-off between the error and the compression rate can be obtained. By adopting the above method, the following effect can be obtained, that is, the compression rate is optimized within the range where the error of the statistic specified by the user is less than the specified target value.

[0116] In S905 , the processor 102 records “Processing: End loss function program 408 with the index calculated by the processing additionally recorded in S904 as the return value” in the program in the loss function program area (the program after the additional recording in S904 ).

[0117] After S905, the processor 102 ends the loss setting program 402 (S906).

[0118] Thus, a loss function program 408 is created in the loss function program area.

[0119] So, through Fig. 9 The loss function program 408 set by the step becomes a program that obtains the input data and output data of the compressor and decompressor as input, and outputs the error between the two inputs for the statistics input by the user on the statistics parameter setting screen 200.

[0120] In addition, Fig. 9 In the flowchart of FIG. 4 , the loss function program 408 is set to calculate the error of the statistic specified by the user as the loss function, but the processor 102 may also record the following processing in the loss function program area, wherein the processing adds indicators other than the error of the statistic such as the amount of compressed data and the mean square error between the input data and the output data of the compressor and decompressor to the error of the statistic. In this way, these indicators can be reflected in the loss function.

[0121] (5) Data writing process

[0122] Fig.10 This is a flow chart of the data writing process.

[0123] The data writing process is performed by the data writing program 403. The data writing program 403 is started by the processor 102 when the front-end interface 101 of the storage node 100 receives the writing data from the sensor server 120 (S1000).

[0124] In S1001 , the processor 102 stores the write target data received by the front-end interface 101 in the cache 103 .

[0125] In S1002, the processor 102 learns the neural network that constitutes the compressor and decompressor through the compression and decompression program 420. The learning data used is the sensor data stored in the cache 103 in S1001. The value of the loss function is calculated by executing the loss function program 408 using the combination of the original data (sensor data in the cache 103) and the decompressed data (output data obtained from the compressor and decompressor when the original data is used as the input of the compressor and decompressor) as an argument. When the error back propagation algorithm is used as the learning algorithm of the neural network, the gradient value in the loss function can be calculated using the well-known automatic differentiation method, or a program for calculating partial differential values ​​can be generated in the loss setting program 402 and calculated by executing the program. In addition, S1002 can be skipped and the compressor and decompressor learned when S1002 was executed last time can be reused in subsequent steps.

[0126] In S1003, the processor 102 compresses the sensor data stored in the cache 103 in S1001 by using the compression function provided by the compression and decompression program 420 and the compression and decompression device learned in S1002. The compressed data contains all the information required for data decompression. The information contained in the compressed data varies depending on the method of the compression and decompression device. For example, when the compression and decompression device is composed of an automatic encoder, the compressed data contains the values ​​of the parameters of the decoder and the values ​​of the intermediate vectors.

[0127] In S1004 , the processor 102 calls the statistical quality calculation program 405 using the compressed data obtained in S1003 as an argument.

[0128] In S1005 , the processor 102 enters the compressed data obtained in S1003 into a database provided by the database program 409 .

[0129] In S1006 , the processor 102 obtains all values ​​related to the statistical quality from the statistical quality management table 411 , and enters them into the database provided by the database program 409 .

[0130] After S1006, the processor 102 ends the data writing program 403 (S1007).

[0131] Fig.11 is a flowchart of the statistical quality calculation process.

[0132] The statistical quality calculation process is performed by the statistical quality calculation program 405. The statistical quality calculation program 405 is Fig.10 The process starts with the call of S1004 (S1100).

[0133] In S1101, the processor 102 decompresses the compressed data provided as an argument to obtain sensor data using the decompression function provided by the compression and decompression program 420. The decompression method of the sensor data varies depending on the type of the compression and decompression device. For example, when the compression and decompression device is composed of an automatic encoder, the value of the parameter included in the compressed data is set to the decoder, and the value of the intermediate vector included in the compressed data is input to the decoder to obtain the sensor data.

[0134] S1102 to S1111 are executed for each row 501 of the statistics parameter management table 410. Fig.11 In the description of , it is called "loop target row").

[0135] In S1103 , the processor 102 identifies the type of statistics based on the column 502 of the loop target row of the statistics parameter management table 410 .

[0136] In S1104 , the processor 102 acquires the parameter of the statistic from the column 503 of the loop target row of the statistic parameter management table 410 .

[0137] In S1105 , the processor 102 calls the statistic calculation program 407 corresponding to the statistic type determined in S1103 , using the sensor data decompressed in S1101 and the parameters acquired in S1104 as arguments.

[0138] In S1106 , the processor 102 calls the statistic calculation program 407 corresponding to the statistic type determined in S1103 , using the sensor data in the cache 103 and the parameters acquired in S1104 as arguments.

[0139] S1107 to S1110 are executed for each column 602 of the statistical quality management table 411. Fig.11 In the description of , it is called "loop target column").

[0140] In S1108, the processor 102 calculates the error corresponding to the index of the loop object column based on the tensor of statistics calculated in S1105 and the tensor of statistics calculated in S1106. For example, when the loop object column is a column indicating "mean square error", the processor 102 calculates the mean square error of the two tensors.

[0141] In S1109 , the processor 102 stores the error calculated in S1108 in the loop target column 602 of the row 601 corresponding to the loop target row in the statistic quality management table 411 .

[0142] When the loop of S1102 to S1111 ends, the processor 102 ends the statistic quality calculation program 405 ( S1112 ).

[0143] (6) Sensor data reading and processing

[0144] Fig.12 This is a flowchart of the data reading process.

[0145] The data reading process is performed by the data reading program 404. The data reading program 404 is started by the processor 102 when the front-end interface 101 receives a data reading request from a computer such as the sensor server 120 or the management server 130 (S1200). In addition, the requester of data reading is not limited to the sensor server 120 and the management server 130, but can also be any computer connected to the storage node 100 via the switch 140.

[0146] In S1201 , the processor 102 analyzes the read request received by the front-end interface 101 , and obtains compressed data as a read object from a database provided by the database program 409 .

[0147] In S1202, the processor 102 decompresses the compressed data obtained in S1201 using the decompression function provided by the compression and decompression program 420 to obtain sensor data. The decompression method of the sensor data varies depending on the type of the compression and decompression device. For example, when the compression and decompression device is composed of an automatic encoder, the value of the parameter included in the compressed data is set to the decoder, and the value of the intermediate vector included in the compressed data is input to the decoder to obtain the sensor data.

[0148] In S1203 , the processor 102 sends the sensor data decompressed in S1202 as a response to the requesting computer via the front-end interface 101 .

[0149] After S1203, the processor 102 ends the data reading program 404 (S1204).

[0150] (7) Statistical quality response processing

[0151] When the user presses the display button 302 on the statistic quality confirmation screen 300 , the management server 130 transmits a request for statistic quality together with the value set in the data ID input area 301 to the storage node 100 .

[0152] The processor 102 starts processing for responding the quality of the statistics to the management server 130 when the front-end interface 101 of the storage node 100 receives the request for the quality of the statistics.

[0153] First, the processor 102 obtains the data ID received by the front-end interface 101. Next, the processor 102 obtains the information corresponding to the ID from the statistical quality information entered into the database in S1006 from the database provided by the database program 409. Next, the processor 102 sends the statistical quality response obtained from the database to the management server 130 via the front-end interface 101.

[0154] After receiving the response from the storage node 100 , the management server 130 displays the received statistical quality value in the statistical quality area 303 of the statistical quality confirmation screen 300 .

[0155] Such statistical quality response processing may be performed by, for example, the processor 102 executing at least one of the data reading program 404 and the statistical quality calculation program 405 .

[0156] The storage system according to one embodiment of the present invention has been described above.

[0157] For example, the above description can be referred to Fig.13 The summary is as follows.

[0158] The storage system 50 includes an interface device 58 (e.g., a backend interface 107) and a processor 52 (e.g., a processor 102), wherein the receiving device 58 is connected to a storage device 59 (e.g., a drive 150), and the processor 52 uses a compression and decompression device 13 based on machine learning to perform lossy compression of time series data (e.g., sensor data), and saves the time series data subjected to the lossy compression, i.e., compressed data, to the storage device 59. The input to the compression and decompression device 13 is the time series data before compression, i.e., the original data 16. For the time series data 16, the output given by the compression and decompression device is the data obtained by decompressing the compressed data of the time series data 16, i.e., the decompressed data 17. The processor 52 provides a first interface 55, which is used to receive the input of one or more parameters required for the calculation of each of the one or more statistics. For each of the one or more statistics, the machine learning of the compression and decompression device 13 is performed based on the one or more parameters input through the first interface 55. The time series data 16 is compressed using the compressor / decompressor 13 after machine learning, and the compressed data of the original data 16 is stored in the storage device 59. As a result, information that does not affect the accuracy requirement of the statistic (i.e., information other than the information required to calculate the statistic) can be excluded from the compression result (compressed data), where the statistic is calculated based on the parameters specified by the first interface 55. That is, the compressor / decompressor 13 learns based on the parameters specified by the first interface 55, and the data compressed by the compressor / decompressor 13 after learning does not include information that does not affect the accuracy requirement of the statistic. Therefore, under the premise of meeting the same accuracy requirement of the statistic, the compression rate can be improved compared with the comparative example.

[0159] For example, in lossy compression of sensor data, information such as statistics required by the application is different for each data, but according to the above-mentioned storage system 50, the statistics to be retained in the compression result can be controlled based on the user's designation.

[0160] In addition, the storage system 50 can also be applied to lossy compression of time series data other than sensor data, such as audio data. For example, the first interface 55 can also receive a parameter (frequency: 100 Hz-200 Hz) for the statistic type "power spectrum".

[0161] The processor 52 manages the management information 41 (e.g., information including the statistic parameter management table 410). The management information 41 includes one or more parameters inputted via the first interface for each of one or more statistics. Thus, one or more parameters can be managed for each type of statistic. In addition, the storage system 50 includes, for example, a memory 54 (e.g., RAM 104), and the management information 41 can be stored in the memory 54.

[0162] The processor 52 calculates the statistical value for each of the one or more statistics based on one or more parameters input through the first interface 55, taking the original data 16 as an object. The processor 52 calculates the statistical value for each of the one or more statistics based on one or more parameters input through the first interface 55, taking the decompressed data 17 corresponding to the original data 16 as an object. The compressor 13 performs machine learning based on the statistical value calculated for each of the one or more statistics based on the original data 16 and the statistical value calculated for each of the one or more statistics based on the decompressed data 17. In this way, the machine learning of the compressor 13 can be performed based on one or more parameters input through the first interface 55.

[0163] The machine learning of the compressor and decompressor 13 is performed based on the objective function 20 set in the compressor and decompressor. The processor 52 calculates a statistic value for each of the one or more statistics, with the original data 16 as the object, and calculates another statistic value for the decompressed data 17 as the object, and sets one or more values ​​(errors) related to the difference between the two statistic values ​​as the objective function 20 of the compressor and decompressor 13. One example of the objective function 20 is a loss function. Usually, the loss function uses the average value of the error at different times. However, there is no method to make the error uniform between times. Therefore, there is a problem that the error fluctuates at different times. In addition, in the loss function, sometimes in addition to the term representing the error, a term for improving the compression rate is added. In this case, the trade-off between the compression rate and the error can be adjusted by the ratio of the error term and the compression rate term, that is, the coefficient multiplied by the compression rate term (hereinafter referred to as the compression coefficient). However, under normal circumstances, the relationship between the compression coefficient, the error, and the compression rate is not clear. In order to adjust that relationship, a large number of compressors and decompressors with different compression coefficients must be learned, which takes time. The use of the storage system 50 can solve such a problem. That is, the error fluctuation between moments or between statistics can be reduced. For example, an interface (such as a UI) can be provided to receive a target value of the error specified by a user, and a loss function can be set or the compression coefficient can be automatically adjusted to optimize the compression rate within the range of the target value specified by the interface.

[0164] The one or more statistics may include at least one of a mean and a variance. For at least one of the one or more statistics, the one or more parameters may include a sample size and a calculation frequency. Such parameters may be specified as parameters necessary for calculating the statistic.

[0165] The processor 102 may provide a second interface 56 for displaying the quality of statistics, wherein the quality of statistics is obtained based on the difference between the following two statistical values, one of which is a statistical value calculated for each of more than one statistics based on the original data 16, and the other is a statistical value calculated for each of more than one statistics based on the data 17 obtained by decompressing the compressed data of the compressor and decompressor after machine learning. Thus, the user can know the quality of statistics, which is obtained based on the data 17 compressed and decompressed by the compressor and decompressor 13 after machine learning. In addition, the quality of statistics may be one or more values ​​related to the difference between the following two statistical values, one statistical value is a statistical value calculated for each of more than one statistics based on the original data 16; the other statistical value is a statistical value calculated for each of more than one statistics based on the decompressed data 17 of the compressed data of the original data 16.

[0166] The type of statistic may also be specified via the first interface 55. This allows more accurate specification of information required for calculation of statistics.

[0167] Although one embodiment has been described above, this is merely an example for explaining the present invention, and the scope of the present invention should not be limited to this embodiment. The present invention can also be implemented in various other forms.

[0168] Description of Reference Numerals

[0169] 100 storage nodes

Claims

1. A storage system, comprising: An interface device connected to the storage device; and A processor, which uses a compression and decompression device based on machine learning to perform lossy compression of time series data, and saves the time series data subjected to lossy compression, i.e., compressed data, into the storage device, The input to the compression and decompression device is the original data consisting of the time series data before compression, and the output of the compression and decompression device is the data obtained by decompressing the compressed data of the time series data, that is, the decompressed data. For each of the one or more statistics, the compression and decompression device is machine-learned based on one or more parameters, the time series data is compressed using the compression and decompression device after the machine learning, and the compressed data of the time series data is stored in the storage device, the processor, Taking the original data as an object, for each of the one or more statistics, a statistical value is calculated based on the one or more parameters, Taking the decompressed data corresponding to the original data as an object, for each of the one or more statistics, a statistical value is calculated based on the one or more parameters, The machine learning of the compressor and decompressor is performed in the following manner: the statistical value calculated for each of the one or more statistics with the statistical value calculated for each of the one or more statistics with the decompressed data is compared to exclude from the compression result information that will not affect the accuracy requirement of the statistics calculated based on one or more parameters specified via an interface, wherein the interface receives input of the one or more parameters required to calculate the statistic for each of the one or more statistics.

2. The storage system according to claim 1, wherein: The machine learning of the compressor and decompressor is performed based on the objective function set in the compressor and decompressor. For each of the one or more statistics, the processor sets one or more values ​​related to the difference between the following two statistic values ​​as the objective function of the compression and decompression device, wherein one statistic value is calculated for each of the one or more statistics based on the original data, and the other statistic value is calculated for each of the one or more statistics based on the decompressed data.

3. The storage system according to claim 1, wherein: The one or more statistics include at least one of a mean and a variance, For at least one statistic among the more than one statistics, the more than one parameters include a sample number and a calculation frequency.

4. The storage system according to claim 1, wherein: The processor provides a first interface for receiving input of one or more parameters required to calculate each of the one or more statistics.

5. The storage system according to claim 4, wherein: The processor manages information including one or more parameters input through the first interface for each of the one or more statistics.

6. The storage system according to claim 4, wherein: the processor, Taking the original data as an object, for each of the one or more statistics, a statistical value is calculated based on one or more parameters input through the first interface, Taking decompressed data corresponding to the compressed data of the compression and decompression device after the machine learning as an object, for each of the one or more statistics, a statistical value is calculated based on one or more parameters input through the first interface, A second interface is also provided, which is an interface for displaying statistical quality, wherein the statistical quality corresponds to the difference between the following two statistical values, one of which is calculated for each of the more than one statistics based on the original data, and the other is calculated for each of the more than one statistics based on the decompressed data.

7. The storage system according to claim 6, wherein: The statistical quality is one or more values ​​regarding the difference between the following two statistical values, wherein one statistical value is calculated for each of the one or more statistics based on the original data, and the other statistical value is calculated for each of the one or more statistics based on the decompressed data.

8. The storage system according to claim 4, wherein: The type of the statistic is a type specified by the first interface.

9. A storage control method for a storage system, wherein the storage system compresses data and stores the data in a storage device, wherein: Taking original data consisting of time series data as an object, for each of more than one statistical quantity, a statistical value is calculated based on more than one parameter, wherein the original data is input into a compression and decompression device based on machine learning for lossy compression, Taking decompressed data corresponding to the original data output from the compression / decompression device as an object, for each of the one or more statistics, a statistical value is calculated based on the one or more parameters, The machine learning of the compressor and decompressor is performed in the following manner: the statistical value calculated for each of the one or more statistics with the statistical value calculated for each of the one or more statistics with the decompressed data is compared to exclude from the compression result information that will not affect the accuracy requirement of the statistics calculated based on one or more parameters specified via an interface, wherein the interface receives input of the one or more parameters required to calculate the statistic for each of the one or more statistics.

Images