An enterprise data storage management system based on cloud storage

Abstract

The invention discloses an enterprise data storage management system based on cloud storage, belonging to the technical field of data storage, including: a storage division module for dividing the storage space in the cloud server into multiple sub-storage devices according to different storage contents; an allocation module for inputting the data input into the cloud server into the corresponding sub-storage device for storage according to its deviation coefficient; a cleaning module for cleaning the stored data in the sub-storage device irregularly. A space adjustment module for dynamically adjusting the storage space according to the storage conditions of each sub-storage device. Through the space adjustment module according to the storage conditions in each sub-storage device, the invention can automatically dynamically adjust the storage space in each sub-storage device, so as to use the storage space in the cloud server more reasonably, reduce the waste of storage resources, and save storage costs.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and particularly relates to an enterprise data storage management system based on cloud storage. Background Art

[0002] With the rapid growth of enterprise data volume, the traditional local storage method can no longer meet the requirements of rapid expansion and high reliability. Therefore, cloud storage technology has emerged. Cloud storage technology is a data storage technology developed on the basis of cloud computing. It integrates various storage resources in the network through the network and provides them to users in the form of storage services. It combines multiple technologies such as distributed storage, virtualization, and network transmission, and stores data on a remote server cluster through the network, realizing cross-regional, cross-device access and sharing of data.

[0003] When using cloud storage technology to store enterprise data in the corresponding cloud server, generally, the cloud server is first planned and classified into multiple sub-storage devices, and then the enterprise data is stored in the corresponding sub-storage devices; however, due to the variety of enterprise data, data storage errors are likely to occur during storage, which is not convenient for subsequent searching; and due to the growth of stored data, a large amount of storage space is required, and its storage cost may exceed expectations. Therefore, how to more reasonably allocate and utilize the space of each sub-storage device to reduce the output of storage costs has become increasingly important. Summary of the Invention

[0004] The purpose of the present invention is to provide an enterprise data storage management system based on cloud storage to solve the problems faced in the above background art.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] An enterprise data storage management system based on cloud storage, the management system includes:

[0007] A storage device division module, which is used to divide the storage space in the cloud server into multiple sub-storage devices according to different storage contents, and a matching feature set trained by a neural network model is provided in each sub-storage device, and the matching feature set contains multiple matching features related to the storage content of the sub-storage device;

[0008] An allocation module, which is used to input the data input into the cloud server into the corresponding sub-storage device for storage according to its deviation coefficient;

[0009] A cleaning module, which is used to clean the stored data in the sub-storage device irregularly.

[0010] Space adjustment module, which is used to dynamically adjust the storage space according to the storage conditions of each sub-storage device.

[0011] Further, the method for the distribution module to work is as follows:

[0012] The distribution is divided into a first stage and a second stage:

[0013] First stage: Extract multiple matching features of the data input into the cloud server to form an initial matching feature set, and compare the initial matching feature set with the matching feature sets in each sub-storage device to screen out the sub-storage devices containing all the initial matching feature sets;

[0014] Second stage: Obtain multiple matching features of the data input into the cloud service, and based on the number of times the same matching feature appears, obtain the occupancy ratio ρ of each matching feature i , and the occupancy ratio ρ of each matching feature i is compared with the standard occupancy ratio ρ of each matching feature in each sub-storage device thi , and through the formula obtain the deviation value R of each matching feature i ;

[0015] Furthermore, through the formula F = R1 + …… + R i + …… + R n obtain the deviation coefficient F of each sub-storage device;

[0016] Input the data input into the cloud service into the sub-storage device with the smallest deviation coefficient;

[0017] Wherein, n is the number of matching feature items of the data input into the cloud service obtained, R1 is the deviation value of the first matching feature, and i ∈ (1, n).

[0018] Further, the working method of the cleaning module is as follows:

[0019] Obtain the time when the stored data in each sub-storage device is entered into the sub-storage device. When the storage time exceeds the preset storage duration, clean the storage time;

[0020] The data cleaning method is: Remove the expired stored data from the sub-storage device and convert the stored data into a cold storage mode.

[0021] Further, the working method of the space adjustment module is as follows:

[0022] Set multiple adjustment time points, and obtain the storage space size EM in the sub-storage device at the current adjustment time point;

[0023] Through the formula Obtain the storage predicted value EP;

[0024] When EP > 90% of EM, adjust the storage space of the sub - storage;

[0025] Among them, R a is the first predicted value, R s is the second predicted value, and τ1 and τ2 are preset coefficients.

[0026] Furthermore, the method for obtaining the first predicted value R a is as follows:

[0027] Obtain the curve of the storage data varying with time R b from the current adjustment time point t c to the next adjustment time point t ax in the historical storage data, and the curve of the storage data varying with time R a from the previous adjustment time point t b to the current adjustment time point t ay (t);

[0028] Through the formula

[0029]

[0030] Obtain the first predicted value R a ;

[0031] Among them, WR ax (t) is the standard storage data curve varying with time from the preset current adjustment time point t b to the next adjustment time point t c in the historical storage data, and WR ay (t) is the standard storage data curve varying with time from the preset previous adjustment time point t a to the current adjustment time point t b in the historical storage data.

[0032] Furthermore, the method for obtaining the second predicted value R s is as follows:

[0033] Obtain the business generation volume Q of the enterprise, the frequency h of the enterprise's daily business activities, and the change amount P of the enterprise's internal basic data from the previous adjustment time point t a to the current adjustment time point t b ;

[0034] Thus, through the formula

[0035]

[0036] Obtain the second predicted value R s ;

[0037] Among them, α1 and α2 are preset proportionality coefficients, and Q0 is the preset previous adjustment time point t a to the current adjustment time point t b The standard business generation volume of the enterprise within, h(t) is the curve of the frequency of the enterprise's daily business activities changing with time, h(t b ) is the frequency of the enterprise's daily business activities at time t b .

[0038] Furthermore, the method for the space adjustment module to adjust the storage space of the sub-storage is as follows:

[0039] Select the number m of sub-storage with EP > EM * 90%, increase its storage space, obtain the storage deviation value S of each sub-storage through the formula S = EP - EM * 90%, and sort the sub-storage from large to small according to the storage deviation value S as S1, S2... S m , and increase them in sequence according to the sorting order;

[0040] At the same time, select the number k of sub-storage with EP ≤ EM * 90%, reduce its storage space, obtain the storage free value D of each sub-storage through the formula D = EM * 90% - EP, and sort the sub-storage from large to small according to the storage free value D as D1, D2... D k , and reduce them in sequence according to the sorting order.

[0041] Advantages of the present invention:

[0042] The present invention can divide the storage space in the cloud server into multiple sub-storages in advance according to the different contents of the enterprise storage data, and through the allocation module, compare the matching characteristics of the data input into the cloud server with the matching characteristics in each sub-storage to obtain the deviation coefficient of each sub-storage, so as to input the data into the corresponding sub-storage for storage according to the deviation coefficient situation. In this way, the data can be more accurately input into the corresponding storage space, thereby reducing storage errors and facilitating subsequent query and management;

[0043] The present invention can automatically adjust the storage space in each sub-storage dynamically according to the storage situation in each sub-storage through the space adjustment module, so as to use the storage space in the cloud server more reasonably, reduce the waste of storage resources, and save storage costs.

[0044] Of course, it is not necessary for any product implementing the present invention to achieve all the above advantages simultaneously. Description of the Drawings

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a system block diagram of the present invention. Specific embodiments

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0048] In one embodiment, an enterprise data storage management system based on cloud storage is disclosed, as Figure 1 shown. The management system includes:

[0049] A storage division module, which is used to divide the storage space in the cloud server into multiple sub-storage devices according to different storage contents, and a trained matching feature set by a neural network model is provided in each sub-storage device. The matching feature set contains multiple matching features related to the storage content of the sub-storage device.

[0050] An allocation module, which is used to input the data input into the cloud server into the corresponding sub-storage device for storage according to its deviation coefficient.

[0051] A cleaning module, which is used to clean the stored data in the sub-storage device irregularly.

[0052] A space adjustment module, which is used to dynamically adjust the storage space according to the storage conditions of each sub-storage device.

[0053] Through the above technical solution, the present application can, in advance, divide the storage space in the cloud server into multiple sub-storage devices according to different data contents stored by the enterprise, and use the allocation module to compare the matching features of the data input into the cloud server with the matching features in each sub-storage device one by one to obtain the deviation coefficient, so as to input the data into the corresponding sub-storage device for storage according to the deviation coefficient situation. In this way, the data can be more accurately input into the corresponding storage space, reducing storage errors and facilitating subsequent query and management. At the same time, the present application also uses the space adjustment module to automatically dynamically adjust the storage space in each sub-storage device according to the storage situation in each sub-storage device, so as to use the storage space in the cloud server more reasonably, reduce waste of storage resources, and save storage costs.

[0054] As an implementation manner of the present invention, the method for the allocation module to work is as follows:

[0055] The allocation is divided into a first stage and a second stage:

[0056] First stage: Extract multiple matching features of the data input into the cloud server to form an initial matching feature set, and compare the initial matching feature set with the matching feature sets in each sub-storage device to screen out the sub-storage devices containing all the initial matching feature sets;

[0057] Second stage: Obtain multiple matching features of the data input into the cloud service, and obtain the occupation ratio ρ of each matching feature according to the number of times the same matching feature appears. i , the occupation ratio ρ of each matching feature i is compared with the standard occupation ratio ρ of each matching feature in each sub-storage device. thi By the formula to obtain the deviation value R of each matching feature. i ;

[0058] Furthermore, through the formula F = R1 + …… + R i + …… + R n to obtain the deviation coefficient F of each sub-storage device;

[0059] Input the data input into the cloud service into the sub-storage device with the smallest deviation coefficient;

[0060] Wherein, n is the number of matching feature items of the data input into the cloud service obtained, R1 is the deviation value of the first matching feature, and i ∈ (1, n).

[0061] Through the above technical solution, this embodiment provides a specific method for the allocation module to allocate and store the data input into the cloud server. The entire allocation is divided into a first stage and a second stage; the first stage: extract multiple matching features of the data input into the cloud server to form an initial matching feature set, and compare the initial matching feature set with the matching feature sets in each sub-storage device, and filter out the sub-storage devices containing all the initial matching feature sets, so as to find the sub-storage devices related to the data to be stored and narrow the storage selection range; the second stage: obtain multiple matching features of the data input into the cloud service, and based on the number of times the same matching feature appears, obtain the occupancy ratio ρ of each matching feature i , and use the occupancy ratio ρ of each matching feature i to compare with the standard occupancy ratio ρ of each matching feature in each sub-storage device thi , and obtain the deviation value R of each matching feature through the formula ; furthermore, obtain the deviation coefficient F of each sub-storage device through the formula F = R1 + …… + R i i + …… + R n , and input the data input into the cloud service into the sub-storage device with the smallest deviation coefficient; after narrowing the storage selection range, in order to more accurately determine the storage location, further precise division can be carried out. Therefore, first obtain multiple matching features of the data input into the cloud service, and based on the number of times the same matching feature appears, obtain the occupancy ratio ρ of each matching feature i , and use the occupancy ratio ρ of each matching feature i to compare with the standard occupancy ratio ρ of each matching feature in each sub-storage device thi , and obtain the deviation value R of each matching feature through the formula ; it can be seen that the smaller the deviation value, the greater the possibility that the input data is stored in this sub-storage device. Furthermore, obtain the deviation coefficient F of each sub-storage device through the formula F = R1 + …… + R i i + …… + R n through comprehensive calculation. It can be seen that the smaller the overall deviation coefficient, the higher the possibility that the input data is stored in this sub-storage device. Therefore, input the data input into the cloud service into the sub-storage device with the smallest deviation coefficient. In this way, the storage location required for the input data can be accurately determined according to the deviation coefficient between the input data and each sub-storage device, thereby reducing the occurrence of storage errors.

[0062] It should be noted that the standard occupancy ratio ρ of each matching feature in each sub-storage device thi can be determined independently according to the storage content in each sub-storage device based on experience, and will not be elaborated here.

[0063] ​​As an implementation manner of the present invention, the working method of the cleaning module is as follows:

[0064] Obtain the time when the stored data in each sub-storage is entered into the sub-storage. When the storage time exceeds the preset storage duration, clean the storage time.

[0065] The data cleaning method is: remove the expired stored data from the sub-storage and convert the stored data into a cold storage mode.

[0066] Through the above technical solution, this embodiment provides a specific method for the cleaning module to work. In order to better utilize the storage space and reduce waste, it is necessary to clean the data in the storage space irregularly. Specifically, first obtain the time when the stored data in each sub-storage is entered into the sub-storage, and set a storage duration. When the storage time exceeds the preset storage duration, clean the storage time. At this time, it means that the stored data has expired. In order to reduce its occupation of the storage space, remove the expired stored data from the sub-storage and convert the stored data into a cold storage mode, so as to ensure the full utilization of the storage space.

[0067] As an implementation manner of the present invention, the working method of the space adjustment module is: set multiple adjustment time points, and obtain the storage space size EM in the sub-storage at the current adjustment time point.

[0068] Through the formula Obtain the storage prediction value EP;

[0069] When EP > 90% of EM, adjust the storage space of the sub-storage.

[0070] Among them, R a Is the first prediction value, R s Is the second prediction value, τ1 and τ2 are preset coefficients;

[0071] And the method for obtaining the first prediction value R a Is: obtain the storage data change curve R b (t) with time from the historical storage data from the current adjustment time point t c To the next adjustment time point t ax , and the storage data change curve R a From the previous adjustment time point t b To the current adjustment time point t ay (t);

[0072] Through the formula

[0073]

[0074] Obtain the first prediction value Ra ;

[0075] Among them, WR ax (t) is the curve of the standard storage data varying with time within the preset current adjustment time point t b to the next adjustment time point t c ; WR(t) is the curve of the standard storage data varying with time within the preset previous adjustment time point t ay to the current adjustment time point t a ; b

[0076] And the method for obtaining the second predicted value R s is as follows: Obtain the business generation volume Q of the enterprise, the frequency h of the enterprise's daily business activities, and the change amount P of the enterprise's internal basic data within the previous adjustment time point t a to the current adjustment time point t b ;

[0077]

[0078]

[0079]

[0080] Then, the second predicted value R is obtained through the formula s ;

[0081] Among them, α1 and α2 are preset proportionality coefficients, Q0 is the standard business generation volume of the enterprise within the previous adjustment time point t a to the current adjustment time point t b ; h(t) is the curve of the set frequency of the enterprise's daily business activities varying with time, and h(t b ) is the frequency of the enterprise's daily business activities at time t b ;

[0082] The method for the space adjustment module to adjust the storage space of the sub - storage is as follows: Select the number m of sub - storages where EP > EM * 90%, increase their storage space, obtain the storage deviation value S of each sub - storage through the formula S = EP - EM * 90%, and sort the sub - storages S1, S2... S m in descending order according to the storage deviation value S, and increase them sequentially according to the sorting order;

[0082] At the same time, select the number k of sub - storages where EP ≤ EM * 90%, reduce their storage space, obtain the storage idle value D of each sub - storage through the formula D = EM * 90% - EP, and sort the sub - storages D1, D2... D k in descending order according to the storage idle value D, and reduce them sequentially according to the sorting order.

[0083] Through the above technical solution, this embodiment provides a method for dynamically adjusting the storage space by the space adjustment module. First, set multiple adjustment time points, and obtain the storage space size EM in the sub-storage at the current adjustment time point; then obtain the storage data from the historical storage data at the current adjustment time point t b to the next adjustment time point t c The curve R ax (t) of the storage data changing with time within, and the curve R a of the storage data changing with time from the previous adjustment time point t b to the current adjustment time point t ay (t). Through the formula

[0084]

[0085] Obtain the first estimated value R a ; from the formula It represents the difference between the change of the storage data in the same time period (t c -t b ) in the historical data and the change of the standard storage data. The larger the value, the larger the storage data and the larger the storage space required. Similarly, the formula represents the difference between the change of the storage data in the same time period (t b -t a ) in the historical data and the change of the standard storage data. The larger the difference, the larger the storage space required. Therefore, combine the storage data situation of the previous period and the next period at the same time point in the historical storage data for comprehensive analysis to obtain the first estimated value, so that the required storage space can be better predicted according to the historical storage situation; then obtain the previous adjustment time point t a to the current adjustment time point t b The business generation volume Q of the enterprise, the frequency h of the enterprise's daily business activities, and the change amount P of the enterprise's internal basic data within, so as to obtain the second estimated value R

[0086]

[0087] through the formula s ; the formula represents the difference between the frequency of the enterprise's daily activities in the second half of the time and the frequency of the enterprise's daily activities in the first half of the time within the time of t a ~t b . The larger the value, the larger the storage space required; so combine t a ~t bComprehensively analyze the business generation volume Q of the enterprise within a certain period of time, the frequency h of the enterprise's daily business activities, and the change volume P of the enterprise's internal basic data to obtain the second predicted value. It can be seen that during this period, the more the business generation volume Q of the enterprise, the greater the change volume P of the enterprise's internal basic data, and the higher the frequency of the enterprise's daily business activities, the larger the storage space required. Therefore, finally, through the formula obtain the storage predicted value EP, and compare the obtained storage predicted value with the storage space size EM in the sub-storage at the current adjustment time point: when EP > 90% * EM, it indicates that the storage space is insufficient; otherwise, it indicates that the storage space is sufficient, and then make corresponding adjustments to the storage space of the sub-storage. The specific adjustment is as follows: select the number m of sub-storage with EP > 90% * EM, increase its storage space, obtain the storage deviation value S of each sub-storage through the formula S = EP - 90% * EM, and sort the sub-storage from large to small according to the storage deviation value S as S1, S2... S m , and increase them in sequence according to the sorting order. The higher its ranking, the more storage space is required, and the more storage space is increased for this sub-storage; at the same time, select the number k of sub-storage with EP ≤ 90% * EM, reduce its storage space, obtain the storage idle value D of each sub-storage through the formula D = 90% * EM - EP, and sort the sub-storage from large to small according to the storage idle value D as D1, D2... D k , and reduce them in sequence according to the sorting order. The higher its ranking, the more sufficient its storage space, and the more storage space is reduced for this sub-storage. Through this method, the storage space in each sub-storage can be dynamically adjusted automatically according to the storage situation in each sub-storage, so as to use the storage space in the cloud server more reasonably, reduce the waste of storage resources, and save storage costs. It should be noted that the determination of the adjustment time point can be artificially determined independently. The preset coefficients τ1 and τ2, and the preset proportionality coefficients α1 and α2 can all be determined according to empirical data, and the standard business generation volume Q0 of the enterprise, the standard storage data change curve WR ax (t), WR ay (t), etc. between each adjustment time point can be determined according to the relevant historical data of the enterprise, and will not be elaborated here.

[0088] The above content is only an example and explanation of the concept of the present invention. Those skilled in the art of this technology make various modifications or supplements to the described specific embodiments or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by this claim book, they should all belong to the protection scope of the present invention.

Claims

1. An enterprise data storage management system based on cloud storage, characterized in that, The management system includes: A memory division module, which is used to divide the storage space in the cloud server into multiple sub - memories according to different storage contents, and a trained matching feature set after a neural network model is provided in each sub - memory. The matching feature set contains multiple matching features related to the storage content of the sub - memory; An allocation module, which is used to input the data input into the cloud server into the corresponding sub - memory for storage according to its deviation coefficient; A cleaning module, which is used to clean the stored data in the sub - memory irregularly; A space adjustment module, which is used to dynamically adjust the storage space according to the storage conditions of each sub - memory; The working method of the space adjustment module is as follows: Set multiple adjustment time points, and obtain the storage space size in the storage of the current adjustment time point ; Obtain the storage prediction value through the formula ;​ When occurs, the storage space of the sub-memory is adjusted; Wherein, is the first estimated value, is the second estimated value, and is the preset coefficient. The method for obtaining the first estimated value is: obtaining the curve of the stored data varying with time within the current adjustment time point to the next adjustment time point from the historical stored data, and the curve of the stored data varying with time within the previous adjustment time point to the current adjustment time point from the historical stored data; ; Through the formula Obtain the first estimated value ; Among them, is the preset current adjustment time point to the next adjustment time point the curve of the standard storage data changing with time within, is the preset previous adjustment time point to the current adjustment time point the curve of the standard storage data changing with time within; The second predicted value The acquisition method is: acquire the business generation volume Q, the frequency h of the enterprise's daily business activities, and the change amount P of the enterprise's internal basic data within the time period from the previous adjustment time point to the current adjustment time point; Thus, through the formula Derive the second estimated value ; Among them, and are preset proportionality coefficients, is the preset previous adjustment time point to the current adjustment time point the standard business generation volume of the enterprise within, is the curve of the change of the frequency of the enterprise's daily business activities over time set, is the frequency of the enterprise's daily business activities at time.

2. The enterprise data storage and management system based on cloud storage according to claim 1, characterized in that, The method for the allocation module to work is: The allocation is divided into a first stage and a second stage: The first stage: Extract multiple matching features of the data input into the cloud server to form an initial matching feature set, and compare the initial matching feature set with the matching feature sets in each sub - memory to screen out the sub - memories containing all the initial matching feature sets; Second stage: Obtain multiple matching features of the data input into the cloud service, and based on the number of times the same matching feature appears, obtain the occupancy ratio of each matching feature , and the occupancy ratio of each matching feature is compared with the standard occupancy ratio of each matching feature in each sub-storage , and the deviation value of each matching feature is obtained through the formula ; ; Furthermore, through the formula the deviation coefficients of each sub-memory are obtained ; Input the data input into the cloud service into the sub - memory with the smallest deviation coefficient; Among them, is the number of matching feature items of the data input into the cloud service obtained, is the deviation value of the first matching feature, and .

3. A cloud storage-based enterprise data storage management system according to claim 1, characterized in that, The working method of the cleaning module is: Obtain the time when the stored data in each sub - memory is entered into the sub - memory. When its storage time exceeds the preset storage duration, clean the storage time; The data cleaning method is: Remove the expired stored data from the sub - memory and convert the stored data into a cold storage mode.

4. A cloud storage-based enterprise data storage management system according to claim 1, characterized in that, The method for the space adjustment module to adjust the storage space of the sub - memory is: Select the number m of sub - memories, increase its storage space, and obtain the storage deviation values of each sub - memory through the formula , and sort the sub - memories from large to small according to the storage deviation value , and sort the sub - memories from large to small , …… , and increase them sequentially according to the sorting order; Select at the same time The number k of sub-storage devices, reduce its storage space, and obtain the storage free value of each sub-storage device through the formula And sort the sub-storage devices from large to small according to the storage free value , …… , and reduce them in turn according to the sorting order.​​

Images