Data storage system

Abstract

This data storage system is provided with a deviation value detection unit (12), a learning unit (14), and a data storage unit (16). The deviation value detection unit detects, from the input data, a deviation value that is isolated from the main data group and that is generated at a low frequency. The learning unit learns a data generation model by inputting main data obtained by removing the deviation value from the input data into the data generation model using a machine learning model. The data storage unit stores the learned data generation model and the deviation value detected by the deviation value detection unit.

Description

Technical Field

[0001] The present disclosure relates to a data storage system. Background Art

[0002] As described in Patent Document 1, a forwarding system is known, which comprises a data compression unit constructed using a deep generative model such as VAE or GAN, and compresses input data into compressed data in the form of a multidimensional joint probability distribution and sends the compressed data to a data reproduction unit.

[0003] In addition, VAE is the abbreviation of "Variational Auto Encoder", which means variational auto encoder. In addition, GAN is the abbreviation of "Generative Adversarial Net", which means adversarial generative network.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2018-61091

[0005] In the above forwarding system, the amount of compressed data increases as the amount of input data increases. Therefore, there is a problem that the amount of compressed data sent to the data reproduction unit cannot be reduced, and the storage capacity for storing compressed data in the data reproduction unit increases. Summary of the invention

[0006] An object of one aspect of the present disclosure is to provide a data storage system capable of constructing a data generation model that can appropriately restore input data without storing compressed data obtained by compressing input data.

[0007] A data storage system according to one aspect of the present disclosure includes a deviation value detection unit, a learning unit, and a data storage unit.

[0008] Here, the deviation value detection unit detects deviation values ​​that are isolated from the main data group and have a low frequency of occurrence from the input data. In addition, the learning unit inputs the main data after removing the deviation values ​​from the input data into the data generation model using the machine learning model to learn the data generation model. Then, the data storage unit stores the data generation model learned by the learning unit and the deviation values ​​detected by the deviation value detection unit.

[0009] Usually, in the learning of the data generation model, all the obtained data are used for learning. On the other hand, since the deviation data often has characteristics different from the usual data, it is generally difficult to reproduce the same data even if learning is performed, which causes a decrease in accuracy when generating. Furthermore, the data generation model attempts to learn to generate data containing deviation data, so that the accuracy of data other than the deviation data after generation tends to decrease. In addition, the main data is also frequently appearing data, and it is not considered efficient to save all of this data from the perspective of data variety and data volume.

[0010] Therefore, in the technology disclosed in the present invention, in the learning unit, learning is performed by using the main data generated at a high frequency after removing the deviation value from the input data, thereby creating a data generation model that can reconstruct the input data, thereby improving the reconstruction accuracy of the input data by the data generation model. In addition, the deviation value detected by the deviation value detection unit is generally likely to be poor in reconstruction accuracy, so the deviation value is stored in the original state in addition to the data generation model by the data storage unit.

[0011] Therefore, according to the data storage system disclosed in the present invention, the compressed data after the input data is compressed is not forwarded or stored as described in Patent Document 1, but the input data can be restored using the stored data generation model. In addition, by adding the deviation value stored by the data storage unit to the restored input data, the input data restoration accuracy can be improved.

[0012] In addition, as a technology for directly sending and saving a part of the input data as in the data storage system disclosed in the present invention, for example, the technology described in references such as WO2022 / 074794A1 and WO2021 / 250868A1 is known. However, the data storage system disclosed in the present invention is completely different from the technology described in the references in that it saves the data generation model and the deviation value.

[0013] That is, the technology described in the reference document is a technology in which, in a system that compresses input data using an encoder of an auto encoder and transmits it, input data that cannot be normally restored by a decoder of the auto encoder is transmitted without being compressed.

[0014] In this technology, the reason why input data that cannot be normally restored by the encoder is directly sent is that the input data is unlearned input data that is not used for learning of the automatic encoder. That is, when the unlearned input data is compressed and sent, the input data cannot be restored on the receiving side, so the unlearned input data is sent without compression.

[0015] Therefore, in the technology described in the above reference, the data stored on the receiving side are compressed data and unlearned input data, which is similar to the technology of the present disclosure in that the unlearned input data is directly stored.

[0016] However, in the data storage system of the present disclosure, input data other than the deviation value and compressed data are not forwarded or stored, but the data generation model learned using the main data is stored.

[0017] Therefore, according to the data storage system disclosed in the present invention, the amount of data stored in the data storage unit can be reduced to an extremely small storage capacity, compared with the techniques described in Patent Document 1 and the reference documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a block diagram showing the configuration of the data storage system according to the first embodiment.

[0019] Figure 2 This is an explanatory diagram showing the learning action of VAE in the learning unit and the data generation model obtained through the learning.

[0020] Figure 3 It is an explanatory diagram showing the deviation value detected by the deviation value detection unit.

[0021] Figure 4 It is a block diagram showing the configuration of a data storage system according to the second embodiment.

[0022] Figure 5 This is a flowchart showing a data storage process executed by a computer in the third embodiment.

[0023] Figure 6 This is a flowchart showing a data storage process executed by the server in the third embodiment.

[0024] Figure 7 It is a block diagram showing the configuration of a data storage system according to a fourth embodiment.

[0025] Figure 8 This is a block diagram showing the configuration of a data storage system according to the fifth embodiment.

[0026] Fig. 9 This is a flowchart showing the outlier value detection process executed by the learning unit according to the fifth embodiment. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0028] [First embodiment]

[0029] like Figure 1As shown, the data storage system 1 of the present embodiment includes an input unit 2 , an outlier detection unit 12 , a learning unit 14 , a data storage unit 16 , and a storage unit 20 .

[0030] The input unit 2 is a means for inputting a large amount of input data such as image data, a deviation value ratio described later, etc. from the outside to the computer 10 including a CPU, a ROM, and a RAM.

[0031] The deviation value detection unit 12 , the learning unit 14 , and the data storage unit 16 are functions of the computer 10 that are realized by the CPU of the computer 10 executing a predetermined program.

[0032] The storage unit 20 is composed of a storage medium such as a memory, a hard disk, or an SSD, or a server for storing data that can communicate with the computer 10 , and can store various data described below.

[0033] Here, the deviation value detection unit 12 detects deviation values ​​that are isolated from the main data group and have a low frequency of occurrence from the input data, and separates the input data into deviation value data and main data. This separation is performed based on a deviation value ratio that is preset by input from the input unit 2. That is, the deviation value detection unit 12 detects the deviation values ​​and separates them from the input data in such a way that the ratio of the number of deviation values ​​to the number of main data becomes the preset deviation value ratio.

[0034] Then, the deviation value data separated from the input data by the deviation value detection unit 12 is input to the data storage unit 16, and the main data is input to the learning unit 14. The learning unit 14 inputs the main data to the data generation model using the machine learning model to learn the data generation model. Specifically, in this embodiment, the learning unit 14 uses the above-mentioned VAE as the data generation model.

[0035] like Figure 2 As shown, VAE is a learner composed of an encoder 32 and a decoder 34, which are usually composed of a neural network. Then, when the encoding result of the input x is set to z, the encoding result z is converted into a decoding result x' by the decoder 34. Then, in the VAE, the internal parameters of the encoder 32 and the decoder 34 are learned so that the input x becomes the decoding result x'.

[0036] The following describes the learning of VAE. In the following description, the input data is (x1, ..., xn), and the data of the intermediate layer corresponding to each input data is given as (z1, ..., zn). In addition, when capital letters are used such as X and Z, they represent random variables.

[0037] In addition, pe(x) represents the empirical distribution and pβ(x) represents the generalized marginal likelihood, which are defined as follows.

[0038] [Formula 1]

[0039]

[0040] In addition, p(x|z) and q(z|x) are probability distributions obtained by the decoder 34 and the encoder 32, respectively. For example, in the present embodiment, p(x|z) is given by the following equation.

[0041] [Formula 2]

[0042]

[0043] In addition, D represents the dimension of the data. In addition, q(z|x) is a normal distribution determined for each data used in VEA. p(z) is the probability distribution of the target intermediate layer. Although the standard normal distribution can be used, other distributions such as mixed normal distribution can also be used.

[0044] in addition,

[0045] [Formula 3]

[0046] Cβpβ(x|z)βdx

[0047] KL(·||·) is called KL information, and is a function that measures a quantity equivalent to the distance between two probability distributions, as defined by the following formula.

[0048] [Formula 4]

[0049]

[0050] Next, VAE learning is achieved by minimizing the following equation (1).

[0051] [Formula 5]

[0052]

[0053] In formula (1), the first term is a negative log-likelihood, which serves to reduce the reconstruction error. On the other hand, the second term aims to make the intermediate distribution close to the known distribution p(z), and serves as a regularization term. For example, a standard normal distribution can be used as p(z). Then, by minimizing formula (1), the encoder 32 and decoder 34 with a known intermediate distribution (distribution of z) 36 (here, a normal distribution) are learned.

[0054] like Figure 2As described above, the learning unit 14 outputs the decoder part 34 and the normal distribution 38 learned as described above to the data storage unit 16 as the data generation model 30. In addition, the learning unit 14 also outputs the number of main data used for learning the data generation model 30 to the data storage unit 16.

[0055] In this way, by generating random numbers from the normal distribution 38 as the data distribution based on the data generation model 30 output from the learning unit 14 to the data storage unit 16 and inputting them to the decoder 34, the data x' can be output from the decoder 34. In other words, the same data distribution as that constituting the input data x can be restored.

[0056] In addition, the learning of VAE in the learning unit 14 has been recorded in the above-mentioned references and is a well-known technology, so a detailed description is omitted here. In addition, in the present embodiment, VAE is used as the data generation model 30, but for example, a generation model using the above-mentioned GAN, its improved method, diffusion model (Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models, 2020.) and the like can also be used.

[0057] Next, the outlier detection in the outlier detection unit 12 is realized by minimizing the following equation (2).

[0058] [Formula 6]

[0059]

[0060] In formula (2), the first and second terms on the right side have β and πi added as coefficients, which is different from formula (1), but the functions are the same. The first term plays a role in reducing the reconstruction error, and the second term is a term for approaching the known distribution and acts as a regularization term. β is a constant used to obtain a balance between reconstruction error and regularization.

[0061] In addition, the third and fourth terms are necessary to become the upper limit of the left side, but in this embodiment, the influence on the minimization of formula (2) is small, so these terms are ignored. Therefore, in formula (2), the difference from formula (1) is only the existence of the balance coefficient β and the weight πi of each data in the first and second terms.

[0062] The right side is a quantity determined for two probability distributions p(x) and q(x) used in a robust statistics called gamma entropy, and is given by the following equation (3).

[0063] [Formula 7]

[0064]

[0065] In this embodiment, the fact that equation (3) is a value equivalent to the distance having the robustness of two distributions is utilized, and by minimizing the right side of equation (2), the left side also becomes small. In addition, πi is calculated by the following equation (4).

[0066] [Formula 8]

[0067]

[0068] Then, the deviation value detection unit 12 calculates πi of formula (4) for each data xi in coordination with the minimization of formula (2). That is, the deviation value detection unit 12 alternately repeats the minimization of formula (2) and the calculation of πi using formula (4), and uses the πi obtained by the calculation as a parameter representing the normality of each data xi to detect the deviation value.

[0069] For example, Figure 3 This is a histogram of normality πi obtained as a result of performing the above calculation process on input data x obtained by mixing 1% of “1” images with “0” images of a database of handwritten numerals called MNIST.

[0070] As from Figure 3 As can be seen, the gray input data x representing "0" is located in the area with a high normality πi, while the input data x of "1" shown in black is distributed in the area with a normality of "0". In addition, although gray input data x is also distributed in a part of the area with a normality of "0", this is an image different from the usual "0" such as a broken "0".

[0071] Therefore, the outlier detection unit 12 detects input data whose normality πi is lower than the threshold value based on a threshold value set in advance for the normality πi calculated for each input data xi as outlier data and separates it from the main data.

[0072] In addition, in the present embodiment, the threshold for deviation value detection is appropriately changed so that the ratio of main data to deviation value data corresponds to the deviation value ratio input from the input unit 2. Next, the data storage unit 16 stores the data generation model 30 input from the learning unit 14 and the deviation value data 22 input from the deviation value detection unit 12 in a predetermined storage area of ​​the storage unit 20.

[0073] The data storage unit 16 stores the number of main data input from the learning unit 14 and the number of deviation value data 22 in a predetermined storage area of ​​the storage unit 20 as the number of data 24 for generating data in the data generation model 30. The data storage unit 16 may be configured to store the number of input data in the storage unit 20 instead of the number of main data.

[0074] As described above, according to the data storage system 1 of the present embodiment, the data generation model 30 , the outlier data 22 , and the data count 24 of the main data and the outlier data generated by the learning unit 14 are stored in the storage unit 20 .

[0075] Therefore, the input data can be restored using the data generation model 30 stored in the storage unit 20 , without having to transmit or store compressed data obtained by compressing the input data for restoration of the input data as described in Patent Document 1 or the above-mentioned reference documents.

[0076] Furthermore, since the outlier data 22 is stored in the storage unit 20 , the accuracy of data restoration can be improved by adding the outlier data 22 to the data restored using the data generation model 30 .

[0077] Furthermore, when restoring input data using the data generation model 30 , the number of restored input data can be set corresponding to the number of deviation value data based on the number of data 24 stored in the storage unit 20 , so that the original input data used for learning can be restored more accurately.

[0078] In addition, since only the data generation model 30, the deviation value data 22, and the data number 24 are stored in the storage unit 20, it is not necessary to store the compressed data of the input data. Therefore, the amount of data stored in the storage unit 20 can be reduced, and the storage capacity of the storage unit 20 can be reduced.

[0079] Furthermore, since it is not necessary to transfer and store the compressed data of the input data in the storage unit 20 as described in Patent Document 1 and the above-mentioned reference documents, it is also possible to prevent the compressed data including personal information from being leaked to the outside and used maliciously.

[0080] Furthermore, if the number of outlier data stored in the storage unit 20 is increased, the accuracy of restoring data based on the data generation model 30 can be improved, but the amount of outlier data stored in the storage unit 20 will increase.

[0081] In contrast, in the present embodiment, the ratio of the deviation value data 22 separated by the deviation value detection unit 12 to the main data can be specified by the deviation value ratio input from the input unit 2. Therefore, by appropriately setting the deviation value ratio, the user can ensure the accuracy of data restoration based on the data generation model 30 and suppress the amount of deviation value data stored in the storage unit 20. The so-called appropriateness here is, for example, a value determined in consideration of the accuracy desired for the data generation model and the increase in data volume caused by the deviation value.

[0082] [Second embodiment]

[0083] like Figure 4As shown, the data storage system 1 of the present embodiment includes a plurality of clients 40 and a server 50 capable of performing data communication with each of the clients 40 .

[0084] The client 40 and the server 50 are each constituted by a computer including a CPU, a ROM, and a RAM, and are connected via a predetermined communication line, for example, via a network such as the Internet.

[0085] The plurality of clients 40 are provided with the same deviation value detection unit 12 and learning unit 14 as those in the first embodiment. Furthermore, the deviation value data output from the deviation value detection unit 12 and the parameters of the data generation model learned by the learning unit 14 are input to the data transmission unit 42, and the data transmission unit 42 transmits the deviation value data and the parameters of the data generation model to the server 50.

[0086] On the other hand, the server 50 includes the data storage unit 16 and the memory unit 20 similar to those of the first embodiment. The server 50 also includes a data receiving unit 52 that receives transmission data from the plurality of clients 40 and outputs the data to the data storage unit 16 .

[0087] In the server 50, the data storage unit 16 updates the data generation model 30 stored in the storage unit 20 using the parameters of the data generation model in the data sent from each client 40 received by the data reception unit 52. In addition, the data storage unit 16 stores the deviation value data sent from each client 40 in the storage unit 20.

[0088] Here, the parameter update of the data generation model can be performed by, for example, weighting the parameters of each client according to the amount of data each client has, normalizing them so that the overall weight is 1, and then adding them together.

[0089] In addition, in the above structure, the parameters from the client are sent only once, but for example, the parameters of the data generation model updated by the server are sent to the client again, and after the client updates the parameters of the data generation model of each client using its own data, they are sent to the server again. It is also possible to consider an implementation method in which these operations are repeated a predetermined number of times or until the update of the parameters converges.

[0090] That is, the server 50 uses a plurality of clients 40 to learn the data generation model without concentrating the dispersed data in one place. Then, the parameters of each data generation model as the learning result are obtained from each client 40, and the data generation model 30 is updated, thereby realizing so-called federated learning. In addition, for example, federated learning refers to "H.Brendan McMahan,EiderMoore,Daniel Ramage,Seth Hampson,and Blaise Aguera y Arcas.Communication-efficient learning of deep networks from decentralized data.Proceedings ofthe 20th International Conference on Artificial Intelligence and Statistics(AISTATS),Vol.54,,2017.".

[0091] In addition, the server 50 not only generates and updates the data generation model 30 by federated learning, but also obtains the deviation value data from each client 40 and stores it in the storage unit 20. Therefore, according to the data storage system of this embodiment, the same effect as the first embodiment can be obtained. That is, the input data can be restored using the data generation model 30 stored in the storage unit 20 of the server 50, and the deviation value data 22 is added to the restored data, so that the restoration accuracy of the data can be improved.

[0092] Furthermore, since only the data generation model 30 and the deviation value data 22 are stored in the storage unit 20 , the amount of data stored in the storage unit 20 can be reduced, and the storage capacity of the storage unit 20 can be reduced.

[0093] In addition, only the parameters and deviation data of the data generation model learned on the client 40 side are sent from the client 40 to the server 50, and there is no need to send input data containing personal information or compressed data obtained by compressing the input data. Therefore, it is possible to prevent these data from flowing out of the data transmission path from the client 40 to the server 50 and the server 50 to the outside and being maliciously used. In addition, since actual data is not sent from the client to the server, it is also an excellent form from the perspective of privacy protection.

[0094] In addition, the number of main data and the number of deviation value data used to learn the data generation model on the client 40 side can also be sent from the client 40 to the server 50, and on the server 50 side, the data storage unit 16 stores these data numbers in the storage unit 20.

[0095] [Third Embodiment]

[0096] In the first embodiment and the second embodiment, the data storage unit 16 stores the data generation model 30 and the deviation value data 22 learned by the learning unit 14 or each client 40 in the storage unit 20. Therefore, the data generation model and the deviation value data stored in the storage unit 20 are sequentially updated during the input data input period.

[0097] However, for example, in a system configured to collect data over several months or years, the distribution of input data may change due to environmental changes during the period of collecting input data, etc. In this case, although the input data averaged over the entire period of collecting data can be restored in the stored data generation model 30, the input data cannot be restored for each season or each month.

[0098] Therefore, in the data storage system 1 of the present embodiment, the learned data generation model 30 and the deviation value data 22 are stored for each predetermined period such as one month or several weeks. In addition, time information (e.g., a timestamp) indicating the period or time when the data generation model 30 was learned is given to these stored data.

[0099] Specifically, in the data storage system 1 of the first embodiment, the computer 10 is Figure 5 The steps shown in the figure perform data storage processing. That is, in S110, input data and deviation value ratio are obtained from the outside via the input unit 2, and in the next S120, the process of detecting deviation values ​​from the input data based on the obtained deviation value ratio is performed as the deviation value detection unit 12. In addition, in the next S130, the process of learning the data generation model based on the main data after removing the deviation values ​​from the input data is performed as the learning unit 14.

[0100] Next, in S140 , it is determined whether a preset period (eg, one month) has passed since the start of the learning process in S130 . If the preset period has not passed, the process proceeds to S110 and the processes of S110 to S130 are executed again.

[0101] On the other hand, when it is determined in S140 that the set period has passed, the process moves to S150, and the data generation model learned through the learning process of S130 within the set period and the deviation value data detected in S120 are stored in the storage unit 20 together with the timestamp indicating the current learning period.

[0102] Then, in the next S160 , it is determined whether it is the next learning start timing of the data generation model, and the process waits until the next learning start timing comes. When it comes to the next learning start timing, the process moves to S110 and the above series of processes are executed again.

[0103] As a result, in the storage unit 20 , a plurality of data generation models 30 learned in each predetermined period and the deviation value data 22 corresponding to each data generation model 30 are respectively given a time stamp and stored.

[0104] Therefore, when restoring data using the data generation model 30 stored in the storage unit 20 , the user can restore data for a desired period by selecting the data generation model 30 and the deviation value data 22 learned in a desired period.

[0105] In the data storage system 1 of the second embodiment, the server 50 is used to Figure 6 The steps shown in the figure execute the data storage process, and the same effect as the above-mentioned method can be obtained. That is, in the server 50, in S210, the process as the data receiving unit 52 is executed to obtain the parameters of the data generation model and the deviation value data from the client 40.

[0106] Then, in the next S220 , the data storage unit 16 updates the data generation model 30 stored in the storage unit 20 using the parameters of the data generation model acquired in S210 , and stores the deviation value data in the storage unit 20 .

[0107] Next, in S230 , it is determined whether a preset period has passed since the update process of the data generation model based on S210 and S220 was started. If the preset period has not passed, the process proceeds to S210 and the processes of S210 and S220 are executed again.

[0108] On the other hand, if it is determined in S230 that the set period has passed, the process proceeds to S240, and a timestamp indicating the current data collection period is given to the data generation model 30 and the deviation value data 22 stored in the storage unit 20 during the set period, and the data is stored as the data of the collection period.

[0109] Then, in the next S240 , it is determined whether the current time is the next data collection start timing, and the process waits until the next data collection start timing comes. When the next data collection start timing comes, the process moves to S210 and the above series of processes are executed again.

[0110] As a result, in the storage unit 20, a plurality of data generation models 30 learned in each predetermined period and the deviation value data 22 corresponding to each data generation model 30 are respectively given time stamps and stored, and the same effect as the above-mentioned method can be obtained.

[0111] [Fourth Embodiment]

[0112] like Figure 7As shown, the data storage system 1 of this embodiment is Figure 1 The basic configuration of the system of the first embodiment shown is the same as that of the first embodiment, but is different from the first embodiment in that a display unit 4 is provided and the computer 10 functions as a reconstruction accuracy calculation unit 18 .

[0113] The reconstruction accuracy calculation unit 18 calculates the reconstruction accuracy of the data generated by the data generation model 30 learned by the learning unit 14. Figure 2 In the VAE shown, the reconstruction accuracy is calculated based on the difference between the data x input to the encoder 32 during learning and the data x' output from the decoder 34. The difference between the data x input to the encoder 32 during learning and the data x' output from the decoder 34 can be a statistical quantity such as the average value or maximum value of the error.

[0114] Then, the reconstruction accuracy calculation unit 18 outputs the calculated reconstruction accuracy to the display unit 4, and displays it in a predetermined display area of ​​the display unit 4. In addition, the number of detected deviation values, that is, the number of data of deviation value data, is output from the deviation value detection unit 12 to the display unit 4, and the display unit 4 displays the data number in a predetermined display area.

[0115] As a result, in the data storage system 1 of the present embodiment, the reconstruction accuracy of the data generation model 30 learned by the learning unit 14 and the number of data of the outlier detected by the outlier detection unit 12 are displayed on the display unit 4 .

[0116] Therefore, according to the data storage system 1 of this embodiment, the user can set the deviation value ratio in consideration of the reconstruction accuracy of the data generation model 30 displayed on the display unit 4, the number of deviation value data, or the storage capacity of the storage unit 20 storing the deviation value data 22.

[0117] For example, when the reconstruction accuracy of the data generation model 30 is low, by increasing the deviation value ratio, the reconstruction accuracy of the data generation model 30 learned later can be improved. In addition, for example, by reducing the deviation value ratio as the storage capacity of the storage unit 20 increases, the storage capacity of the deviation value data 22 in the storage unit 20 can also be suppressed.

[0118] [Fifth Embodiment]

[0119] In the above-described embodiments, the outlier detection unit 12 calculates the normality πi for each input data xi and detects the input data x whose normality πi is lower than the threshold as outlier data.

[0120] In contrast, in this embodiment, if Figure 8As shown, the function as an outlier detection unit is realized by the outlier detection process of S400 which is repeatedly executed together with the learning process of S300 in the learning unit 14 .

[0121] That is, in this embodiment, in the learning unit 14, the learning process of S300 is executed every time the input data x is input from the input unit 2. In this learning process, for example, Figure 2 In the VAE-based data generation model shown, the encoder 32 and the decoder 34 are learned so that the output data x′ from the decoder 34 is close to the input data x.

[0122] In contrast, in the deviation value detection process of S400, each time the learning process of S300 is executed, Fig. 9 The steps shown determine whether the latest input data x used in the learning process of S300 is outlier data.

[0123] That is, in the deviation detection process of S400, first, in S410, the input data x and output data x' used for learning the data generation model in S300 are obtained, and the absolute value of the difference (x-x') is calculated as a parameter representing the reconstruction accuracy of the input data x.

[0124] Next, in S420, it is determined whether the reconstruction accuracy of the input data x calculated in S410 is lower than a preset threshold. Then, when the difference between the input data x and the output data x' is small and it is determined in S420 that the reconstruction accuracy of the input data x is above the threshold, the deviation value detection process is terminated and the learning process of S300 is moved to.

[0125] On the other hand, when the difference between the input data x and the output data x′ is large and it is determined in S420 that the reconstruction accuracy of the input data x is lower than the threshold, the process moves to S430 and determines that the latest input data x used in the learning process of S300 is a deviation value.

[0126] Furthermore, in S430, the input data x determined to be the outlier value is input to the data storage unit 16 as the outlier value data 22. As a result, the outlier value data 22 is stored in a predetermined storage area of ​​the storage unit 20.

[0127] In addition, when it is determined in S430 that the input data x is an outlier, the process proceeds to S440, where the outlier is removed from the input data, so that a learning process based on the input data after the outlier is removed, in other words, a learning process based on the main data, is performed in S300. Then, when the outlier is removed from the input data in S440, the outlier detection process is terminated, and the process proceeds to the learning process in S300.

[0128] Thus, in the present embodiment, in the learning unit 14, whether the input data x input from the input unit 2 is an outlier is determined based on the reconstruction accuracy of the input data obtained from the learning result of the data generation model. That is, in the present embodiment, input data whose reconstruction accuracy is low and below the threshold value cannot be properly restored using the data generation model, and therefore is determined to be an outlier and is removed from the input data used to learn the data generation model.

[0129] Therefore, the data storage system 1 of this embodiment can also obtain the same effect as the data storage system 1 of the first embodiment described above. In addition, in this embodiment, it is not necessary to extract the deviation value from all the input data used for learning the data generation model, and it is possible to sequentially determine whether the input data is a deviation value during the learning operation in the learning unit 14. Therefore, the time required for extracting the deviation value can be shortened.

[0130] In addition, in this embodiment, as a modification of the deviation value detection unit 12 of the first embodiment, the deviation value detection process performed by the learning unit 14 is described. However, the deviation value detection process of this embodiment can also be applied to the learning unit 14 of the second to fourth embodiments in the same manner as the above, and the same effect as the above can be obtained.

[0131] In the first to fourth embodiments, if the extraction of outliers and the learning of the data generation model are repeatedly performed in the same manner as in the present embodiment, a data generation model with higher reconstruction accuracy can be constructed.

[0132] [Other Embodiments]

[0133] As mentioned above, although embodiment of this disclosure was described, this disclosure is not limited to the said embodiment, Various deformation|transformation can be made and it can be implemented.

[0134] For example, in the first embodiment, a method of constructing a data storage system using one computer 10 is described, and in the second embodiment, a method of constructing a data storage system using a plurality of computers constituting a plurality of clients 40 and a server 50 is described.

[0135] In contrast, the multiple functions implemented by these computers can also be implemented by a special-purpose computer, which is provided by a processor composed of one or more special-purpose hardware logic circuits. Alternatively, it can also be implemented by one or more special-purpose computers, which are composed of a processor and a memory programmed to perform one or more functions and a processor composed of one or more hardware logic circuits. In addition, the computer program that implements the above functions can also be stored as an instruction executed by the computer in a non-transitional tangible recording medium that can be read by the computer. In addition, the method for implementing the above functions does not necessarily need to include software, and all of its functions can also be implemented using one or more hardware.

[0136] In addition, multiple functions of one component in the above-mentioned embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. In addition, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. In addition, part of the structure of the above-mentioned embodiment may be omitted. In addition, at least part of the structure of the above-mentioned embodiment may be added to or replaced by the structure of other above-mentioned embodiments.

[0137] In addition, the technology disclosed herein can be implemented in various forms other than the above-mentioned data storage system, such as a system having a data storage system as a component, a program for causing a computer to function as a data storage system, a non-transitional physical recording medium such as a semiconductor memory recording the program, and a data storage method.

Claims

1. A data storage system, wherein: have: The deviation value detection unit is configured to detect, from the input data, a deviation value that is isolated from the main data group and has a low occurrence frequency; a learning unit configured to input main data obtained by removing the deviation value from the input data into a data generation model using a machine learning model to learn the data generation model; as well as The data storage unit is configured to store the data generation model learned by the learning unit and the deviation value detected by the deviation value detection unit.

2. The data storage system according to claim 1, wherein: A plurality of clients are provided, wherein the client is configured to include the deviation value detection unit and the learning unit, and transmits the parameters of the data generation model obtained by the learning unit and the deviation value detected by the deviation value detection unit, and A server is provided as the data storage unit, and the server is configured to generate and update the data generation model using the parameters transmitted from the plurality of clients, and to store the deviation values ​​transmitted from the plurality of clients.

3. The data storage system according to claim 1 or 2, wherein: The data storage unit is configured to store the number of the input data or the main data and the number of the deviation values.

4. The data storage system according to claim 1 or 2, wherein: An input unit is provided, the input unit being capable of inputting a ratio of the number of the deviation values ​​detected by the deviation value detection unit to the number of the main data, The deviation value detection unit is configured to detect the deviation value so that the ratio between the number of the deviation values ​​and the number of the main data becomes the ratio input from the input unit.

5. The data storage system according to claim 4, wherein: A display unit is provided, and the display unit is configured to display the reconstruction accuracy of the data by the data generation model learned by the learning unit and / or the number of the deviation values ​​detected by the deviation value detection unit.

6. The data storage system according to claim 1 or 2, wherein: The data generation model learned by the learning unit and the deviation value detected by the deviation value detection unit are stored together with time information indicating the period for each preset period.

7. The data storage system according to claim 1 or 2, wherein: For the above deviation value detection unit, Whenever the learning unit implements the learning based on the input data, the reconstruction accuracy of the data by the learned data generation model is calculated. When the reconstruction accuracy is lower than a specified threshold, it is determined that the input data used for the learning is the deviation value, and the learning unit implements the learning based on the main data after removing the deviation value from the input data.

Images