Mass data distributed storage management method

Abstract

The invention discloses a mass data distributed storage management method, and relates to the technical field of data distributed storage management, and the method comprises the steps: based on mass data collection, analyzing data income overhead, utilizing an exponential smoothing method, predicting a data trend vector, and generating a structured data set; analyzing the hardware function of the distributed storage server according to the predicted data trend vector, and quantifying a storage capability vector; performing association matching based on the quantized data demand vector and the quantized storage capability vector, and generating a distributed storage server node meeting the data income overhead; and based on the data type, performing data fragmentation and the state of the distributed storage server meeting the data income overhead, establishing a minimum storage cost management model, and generating a mass data distributed storage management scheme. The method has the advantages that distributed storage server nodes can be efficiently matched, storage resource configuration is optimized, storage cost is reduced, and dynamic adjustment and efficient management are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage management, and in particular to a method for distributed storage management of massive data. Background Art

[0002] A distributed storage management method for massive data refers to a technology and method for managing and storing large amounts of data through a distributed system. This method is usually used to process massive, dispersed, and dynamically changing data to ensure data reliability, scalability, and efficiency. Specifically, a distributed storage system improves data storage and processing capabilities by distributing data across multiple different storage nodes to avoid the bottleneck problem of a single node.

[0003] The main problems faced by distributed storage management methods for massive data include: difficulties in ensuring data consistency, especially when the network fails or the node goes down; as the amount of data and the number of nodes grow, performance bottlenecks may become system bottlenecks, resulting in slower response speeds; during system expansion, data migration and resource scheduling difficulties may be encountered, affecting overall efficiency; the design of fault recovery and fault tolerance mechanisms is complex, which may lead to data loss or slow recovery; data access delays and security issues, especially when deployed across regions, may affect user experience and data protection; in addition, system management and maintenance also become more complex due to the expansion of scale, increasing the difficulty of operation and maintenance. Summary of the Invention

[0004] In order to solve the above technical problems, a method for distributed storage management of massive data is provided. This technical solution solves the above problems. The main problems faced by the method for distributed storage management of massive data include: difficulties in ensuring data consistency, especially when the network fails or the node goes down; as the amount of data and the number of nodes increase, performance bottlenecks may become bottlenecks of the system, resulting in reduced response speed; data migration and resource scheduling difficulties may be encountered during system expansion, affecting overall efficiency; fault recovery and fault tolerance mechanism design are complex, which may lead to data loss or slow recovery; data access delays and security issues, especially when deployed across regions, may affect user experience and data protection; in addition, system management and maintenance have become more complicated due to the expansion of scale, increasing the difficulty of operation and maintenance.

[0005] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0006] A method for distributed storage management of massive data, comprising:

[0007] Based on massive data collection, we analyze data input and expenditure, use exponential smoothing to predict data trend vectors, and generate structured data sets;

[0008] Based on the predicted data trend vector, the hardware functions of the distributed storage server are analyzed to quantify the storage capacity vector;

[0009] Based on the correlation and matching of the quantified data demand vector and the quantified storage capacity vector, a distributed storage server node is generated to meet the data input overhead.

[0010] Based on data type, data sharding is carried out and the status of the distributed storage server itself that meets the data input overhead is established to establish a minimum storage cost management model and generate a distributed storage management solution for massive data.

[0011] Preferably, based on massive data collection, the Min-Max normalization algorithm is used to scale the data and perform data preprocessing;

[0012] Based on data preprocessing, the ant colony algorithm is used to calculate the costs of data storage, processing, transmission, and processing in data preprocessing, and to construct a statistical table of data incoming expenses;

[0013] Based on the data income and expenditure statistics table, with historical data as input, the Holt-Winters model is used to calculate the predicted value of the data at each time point, predict the data trend vector, and generate a structured data set.

[0014] Preferably, a distributed storage node server hardware capability factor matrix is ​​constructed and standardized based on the predicted data trend vector and the structured data set;

[0015] Calculate the proportion of each server hardware capability factor of the distributed storage node to the overall server hardware capability, and obtain the proportion index of each server hardware capability factor of the distributed storage node;

[0016] Based on the information entropy formula, the discrete degree of the proportion of each server hardware capability factor of the distributed storage node to the overall server hardware capability is calculated to obtain the information entropy of each server hardware capability factor of the distributed storage node;

[0017] According to the weighted method, the information entropy of the hardware capability factor of each server in the distributed storage node is normalized, and a weight of the hardware capability factor of each server in the distributed storage node is assigned to the hardware capability factor of the distributed storage node server, and the hardware capability index of each server in the distributed storage node is calculated.

[0018] Preferably, according to the hardware capabilities of each server in the distributed storage node, the ideal solution and the negative ideal solution of the hardware capability of each server in the distributed storage node are determined, and the range normalization method is used to perform normalization processing on them, in the following manner:

[0019]

[0020] in, is the positive ideal solution distance of the hardware capability of the i-th server in the distributed storage node, is the negative ideal solution distance of the hardware capability of the i-th server in the distributed storage node, V ij is the jth hardware capability index of the i-th server in the distributed storage node, is the maximum value of the jth hardware capability index, is the minimum value of the jth hardware capability indicator;

[0021] According to the weight of each server hardware capability factor of the distributed storage node, a weighted standardized decision matrix is ​​constructed, the distance between the hardware capability of each server in the distributed storage node and the ideal solution and the negative ideal solution is calculated, and the storage capability vector is quantified.

[0022] Preferably, based on the quantified data demand vector and the quantified storage capacity vector, a constraint-based filtering algorithm is used to compare the hard constraints of the hardware capacity of the overall distributed storage node server with the hardware capacity of each server of the distributed storage node, and the hardware capacity of each server of the distributed storage node that does not meet the hard constraints is filtered out;

[0023] Using the TOPSIS algorithm, the hardware capabilities of each server in the distributed storage node are further optimized according to the hardware capabilities of each server in the distributed storage node.

[0024] Preferably, based on the optimization of the hardware capabilities of each server of the distributed storage node, data is avoided from being concentrated in a few storage nodes. A greedy algorithm is used to obtain the optimal hardware capabilities of the distributed storage node server, and a distributed storage server node that meets the data collection overhead is generated in the following manner:

[0025]

[0026] in, is the jth hardware capability vector of the i-th server of the optimal distributed storage node, w ij is the jth hardware capability weight of the i-th server of the distributed storage node, H ij is the jth hardware capability index of the i-th server of the distributed storage node, L ij is the jth hardware capacity load indicator of the i-th server of the distributed storage node.

[0027] Preferably, a suitable shard key that satisfies the data input overhead is selected based on the structured data set, a hash algorithm is applied to the shard key, the value of the shard key that satisfies the data input overhead is converted into a hash value, a modulo operation is performed on the number of shards using the hash value, the data is allocated to the corresponding shards, and a data shard list is constructed;

[0028] Based on the data shard list, the storage media cost, network transmission cost, and redundancy cost are modeled and the storage cost is quantified.

[0029] Preferably, based on the quantified storage cost, an initial population of data shards is randomly generated, the fitness of each data shard individual to satisfy the overall data receiving overhead is calculated, and the roulette wheel selection method is used to select the excellent individuals in the data shards. The excellent individuals in the data shards are selected, crossed, and mutated to generate a new generation of individuals for the data shards. Each data shard individual in the initial population of the data shards is replaced according to the fitness of the new generation of individuals for the data shards to find a better solution to satisfy the data distribution, obtain the data shard to the node with the lowest data receiving overhead, and generate a distributed storage management solution for massive data in the following manner:

[0030]

[0031] f(X q )=α·C storage (X q )+β·C cost (X q )

[0032] X q Cost(X q ) is the data item r stored in node X q The storage cost on the server, AccessCost is the access cost, C cost (X q ) is the data entry cost, that is, the cost incurred when accessing the storage node.

[0033] C storage (X q ) is the data storage cost, that is, the storage usage of each node, Cost is the cost,

[0034] f(X q ) is the fitness of each data shard to satisfy the overall data input cost, α is used to control the influence of storage cost in the fitness function, and β is used to control the influence of access cost in the fitness function;

[0035] X * =argmin(f(X))

[0036] X * Data is sharded to the node with the lowest data entry cost. Argmin is the optimal solution for data distribution, generating a distributed storage management solution for massive data.

[0037] Compared with the existing technology, the beneficial effects of the present invention are: by predicting data trends, quantifying storage capacity and demand, and reasonably sharding based on data types, it can efficiently match distributed storage server nodes, optimize storage resource configuration, reduce storage costs, achieve dynamic adjustment and efficient management, and ensure that the storage needs and revenue expenses of massive data are met.

[0038] The present invention proposes a method for distributed storage management of massive data.

[0039] The beneficial effect is that by combining data trend prediction with dynamic matching of storage resources, efficient distributed storage management of massive data is achieved: the exponential smoothing method is used to accurately predict data growth trends and quantify storage needs, and the server hardware capabilities are simultaneously evaluated. Node selection is optimized through intelligent correlation matching, and finally the optimal storage solution is generated based on the sharding strategy and cost model, which significantly improves storage resource utilization, reduces hardware costs, and ensures the scalability and real-time performance of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a flow chart of a method for distributed storage management of massive data. DETAILED DESCRIPTION

[0041] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0042] Reference Figure 1 As shown, a method for distributed storage management of massive data includes:

[0043] Step 1: Based on massive data collection, analyze data input and expenditure, use exponential smoothing to predict data trend vectors, and generate structured data sets;

[0044] The step 1 includes the following:

[0045] Step 101: Based on massive data collection, the Min-M (a massive data distributed storage management method) normalization algorithm is used to scale the data and perform data preprocessing.

[0046] Step 102: Based on data preprocessing, the ant colony algorithm is used to calculate the costs of data storage, processing, transmission, and processing in data preprocessing, and to construct a statistical table of data receipt and expenditure;

[0047] Step 103: Based on the data income and expenditure statistics table, using historical data as input, using the Holt-Winters model, calculate the predicted value of the data at each time point, predict the data trend vector, and generate a structured data set.

[0048] When using, combine the contents in steps 101 to 103.

[0049] By combining massive data collection with multiple data processing technologies, the accuracy and efficiency of data analysis and prediction are improved. First, the Min-M normalization algorithm (a method for distributed storage and management of massive data) is used to scale the data to a uniform scale, which helps eliminate bias in the data. Then, the ant colony algorithm is used to optimize the storage, processing, and transmission costs of the data to construct a detailed cost statistics table to help rationally allocate resources. Finally, based on the Holt-Winters model, time series prediction is combined with historical data to obtain data trend vectors, thereby generating a structured data set. This process not only effectively optimizes data processing costs but also improves the accuracy of data prediction, providing reliable support for decision-making.

[0050] Step 2: Analyze the hardware functions of the distributed storage server based on the predicted data trend vector and quantify the storage capacity vector;

[0051] The second step includes the following:

[0052] Step 201: Construct a distributed storage node server hardware capability factor matrix based on the predicted data trend vector and the structured data set and perform standardization processing on it;

[0053] Calculate the proportion of each server hardware capability factor of the distributed storage node to the overall server hardware capability, and obtain the proportion index of each server hardware capability factor of the distributed storage node;

[0054] Based on the information entropy formula, the discrete degree of the proportion of each server hardware capability factor of the distributed storage node to the overall server hardware capability is calculated to obtain the information entropy of each server hardware capability factor of the distributed storage node;

[0055] According to the weighted method, the information entropy of the hardware capability factor of each server in the distributed storage node is normalized, and a weight is assigned to the hardware capability factor of each server in the distributed storage node and the hardware capability factor of the distributed storage node server to calculate the hardware capability index of each server in the distributed storage node;

[0056] Step 202: Determine the ideal solution and negative ideal solution of the hardware capability of each server in the distributed storage node based on the hardware capability of each server in the distributed storage node, and perform normalization processing on them using the range normalization method, as follows:

[0057]

[0058] in, is the positive ideal solution distance of the hardware capability of the i-th server in the distributed storage node, is the negative ideal solution distance of the hardware capability of the i-th server in the distributed storage node, V ij is the jth hardware capability index of the i-th server in the distributed storage node, is the maximum value of the jth hardware capability index, is the minimum value of the jth hardware capability indicator;

[0059] According to the weight of each server hardware capability factor of the distributed storage node, a weighted standardized decision matrix is ​​constructed, the distance between the hardware capability of each server in the distributed storage node and the ideal solution and the negative ideal solution is calculated, and the storage capability vector is quantified.

[0060] When used, combine the contents in steps 201 to 202.

[0061] Through quantitative and standardized analysis of the hardware capabilities of each server in the distributed storage node, a capability matrix is ​​constructed based on the predicted data trend vector and structured data set, and the degree of discreteness of each hardware factor is evaluated through the information entropy formula; the weighted normalization method helps determine the weight of each hardware factor and the overall hardware capability of the distributed storage node; by calculating the distance between the ideal solution and the negative ideal solution, the storage capability vector is quantified; this method can accurately measure and optimize the hardware performance of the distributed storage system, improve the system's resource allocation efficiency and performance evaluation accuracy, ensure efficient utilization of hardware resources and support future expansion needs.

[0062] Step 3: Generate distributed storage server nodes that meet the data input overhead based on the correlation matching between the quantified data demand vector and the quantified storage capacity vector;

[0063] The step three includes the following:

[0064] Step 301: Based on the quantified data demand vector and the quantified storage capacity vector, a constraint-based filtering algorithm is used to compare the hard constraints of the overall distributed storage node server hardware capacity with the hardware capacity of each distributed storage node, and the hardware capacity of each distributed storage node server that does not meet the hard constraints is filtered out.

[0065] Step 302: Using the TOPSIS algorithm, further prioritize the hardware capabilities of each server in the distributed storage node according to the hardware capabilities of each server in the distributed storage node;

[0066] Step 303: Based on the optimal hardware capabilities of each distributed storage node, data is prevented from being concentrated in a small number of storage nodes. A greedy algorithm is used to obtain the optimal hardware capabilities of the distributed storage node servers, and a distributed storage server node is generated that meets the data collection overhead. The method is as follows:

[0067]

[0068] in, is the jth hardware capability vector of the i-th server of the optimal distributed storage node, w ij is the jth hardware capability weight of the i-th server of the distributed storage node, H ij is the jth hardware capability index of the i-th server of the distributed storage node, L ij is the jth hardware capacity load indicator of the i-th server of the distributed storage node.

[0069] When used, combine the contents in steps 301 to 303.

[0070] The implementation process of this step first associates and matches the quantified data demand vector with the quantified storage capacity vector to ensure that the data storage demand and hardware capacity match. The specific operations include: Step 301 screens the hardware capabilities of the distributed storage nodes through the constraint filtering algorithm to eliminate nodes that do not meet the hard constraint conditions; Step 302 uses the TOPSIS algorithm to select the best storage node among the hardware capabilities that meet the conditions; Step 303 further optimizes the selection of storage nodes through the greedy algorithm to avoid data concentration in a few nodes, thereby achieving load balancing; Finally, a distributed storage server node is generated that meets the data collection overhead, improves storage efficiency, optimizes hardware resource utilization, and ensures efficient operation of the system.

[0071] Step 4: Segment data based on data type and the state of the distributed storage server itself that meets the data input overhead, establish a minimum storage cost management model, and generate a distributed storage management solution for massive data;

[0072] The step 4 includes the following contents:

[0073] Step 401: Select a suitable shard key that meets the data input overhead based on the structured data set, apply a hash algorithm to it, convert the value of the shard key that meets the data input overhead into a hash value, perform a modulo operation on the number of shards using the hash value, assign the data to the corresponding shards, and construct a data shard list;

[0074] Step 402: Based on the data shard list, the storage medium cost, network transmission cost, and redundancy cost are modeled and the storage cost is quantified.

[0075] Step 403: Based on the quantified storage cost, randomly generate an initial population of data shards. Calculate the fitness of each individual in the data shard to meet the overall data acquisition cost. Use the roulette wheel selection method to select outstanding individuals in the data shards. Select, crossover, and mutate the outstanding individuals in the data shards to generate a new generation of individuals for the data shards. Replace each individual in the initial population of the data shards based on the fitness of the new generation of individuals for the data shards. Find the optimal solution for satisfying the data distribution, obtain the node with the lowest data acquisition cost for the data shards, and generate a distributed storage management solution for massive data. The method is as follows:

[0076]

[0077] f(X q )=α·C storage (X q )+β·C cost (X q )

[0078] X q Cost(X q ) is the data item r stored in node X q The storage cost on the server, AccessCost is the access cost, C cost (X q ) is the data entry cost, that is, the cost incurred when accessing the storage node.

[0079] C storage (X q ) is the data storage cost, that is, the storage usage of each node, Cost is the cost,

[0080] f(X q ) is the fitness of each data shard to satisfy the overall data input cost, α is used to control the influence of storage cost in the fitness function, and β is used to control the influence of access cost in the fitness function;

[0081] X * =argmin(f(X))

[0082] X * Data is sharded to the node with the lowest data entry cost. Argmin is the optimal solution for data distribution, generating a distributed storage management solution for massive data.

[0083] When used, combined with the contents in steps 401 to 403,

[0084] Through data sharding and storage cost modeling, the aim is to optimize the distributed storage management solution for massive data; the specific implementation principles include: first, selecting a suitable sharding key based on the structured data set, and distributing the data to different shards through a hash algorithm; then, quantifying the storage cost based on factors such as storage medium cost, network transmission cost, and redundancy cost; then, using a genetic algorithm to randomly generate an initial data shard population, and continuously optimizing the data sharding solution through operations such as selection, crossover, and mutation to find the distributed storage configuration with the lowest storage cost; ultimately, the optimized solution can effectively reduce storage, transmission, and redundancy costs, achieving efficient and economical data management; this process helps to improve the utilization efficiency of storage resources while reducing data collection overhead, and is suitable for the efficient management of massive data.

[0085] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for distributed storage management of massive data, characterized in that: include: S1. Based on massive data collection, analyze data input and expenditure, use exponential smoothing method to predict data trend vectors, and generate structured data sets; S2. Analyze the hardware functions of the distributed storage server based on the predicted data trend vector and quantify the storage capacity vector; S3. Generate distributed storage server nodes that meet the data input overhead based on the correlation and matching between the quantified data demand vector and the quantified storage capacity vector; S4. Data sharding is performed based on the data type and the state of the distributed storage server itself that meets the data input overhead, a minimum storage cost management model is established, and a distributed storage management solution for massive data is generated.

2. A method for distributed storage management of massive data according to claim 1, characterized in that: Said S1 comprises: Based on massive data collection, the Min-Max normalization algorithm is used to scale the data and perform data preprocessing; Based on data preprocessing, the ant colony algorithm is used to calculate the costs of data storage, processing, transmission, and processing in data preprocessing, and to construct a statistical table of data incoming expenses; Based on the data income and expenditure statistics table, with historical data as input, the Holt-Winters model is used to calculate the predicted value of the data at each time point, predict the data trend vector, and generate a structured data set.

3. A method for distributed storage management of massive data according to claim 2, characterized in that: The S2 includes: Based on the predicted data trend vector and structured data set, a distributed storage node server hardware capability factor matrix is ​​constructed and standardized; Calculate the proportion of each server hardware capability factor of the distributed storage node to the overall server hardware capability, and obtain the proportion index of each server hardware capability factor of the distributed storage node; Based on the information entropy formula, the discrete degree of the proportion of each server hardware capability factor of the distributed storage node to the overall server hardware capability is calculated to obtain the information entropy of each server hardware capability factor of the distributed storage node; According to the weighted method, the information entropy of the hardware capability factor of each server in the distributed storage node is normalized, and a weight of the hardware capability factor of each server in the distributed storage node is assigned to the hardware capability factor of the distributed storage node server, and the hardware capability index of each server in the distributed storage node is calculated.

4. A method for distributed storage management of massive data according to claim 3, characterized in that: Said S2 further comprises: According to the hardware capabilities of each server in the distributed storage node, the ideal solution and negative ideal solution of the hardware capabilities of each server in the distributed storage node are determined, and the range normalization method is used to standardize them as follows: in, is the positive ideal solution distance of the hardware capability of the i-th server in the distributed storage node, is the negative ideal solution distance of the hardware capability of the i-th server in the distributed storage node, V ij is the jth hardware capability index of the i-th server in the distributed storage node, is the maximum value of the jth hardware capability index, is the minimum value of the jth hardware capability indicator; According to the weight of each server hardware capability factor of the distributed storage node, a weighted standardized decision matrix is ​​constructed, the distance between the hardware capability of each server in the distributed storage node and the ideal solution and the negative ideal solution is calculated, and the storage capability vector is quantified.

5. A method for distributed storage management of massive data according to claim 1, characterized in that: The S3 includes: Based on the quantified data demand vector and the quantified storage capacity vector, a constraint-based filtering algorithm is used to compare the hard constraints of the overall distributed storage node server hardware capacity with the hardware capacity of each distributed storage node, and the hardware capacity of each distributed storage node that does not meet the hard constraints is filtered out; Using the TOPSIS algorithm, the hardware capabilities of each server in the distributed storage node are further optimized according to the hardware capabilities of each server in the distributed storage node.

6. A method for distributed storage management of massive data according to claim 5, characterized in that: Said S3 further comprises: Based on the optimization of the hardware capabilities of each server in the distributed storage node, data is prevented from being concentrated in a small number of storage nodes. A greedy algorithm is used to obtain the optimal hardware capabilities of the distributed storage node server and generate distributed storage server nodes that meet the data collection overhead. The method is as follows: in, is the jth hardware capability vector of the i-th server of the optimal distributed storage node, w ij is the jth hardware capability weight of the i-th server of the distributed storage node, H ij is the jth hardware capability index of the i-th server of the distributed storage node, L ij is the jth hardware capacity load indicator of the i-th server of the distributed storage node.

7. A method for distributed storage management of massive data according to claim 6, characterized in that: The S4 includes: Select a suitable shard key that meets the data input overhead based on the structured data set, apply a hash algorithm to it, convert the value of the shard key that meets the data input overhead into a hash value, perform a modulo operation on the number of shards using the hash value, assign the data to the corresponding shards, and construct a data shard list; Based on the data shard list, the storage media cost, network transmission cost, and redundancy cost are modeled and the storage cost is quantified.

8. A method for distributed storage management of massive data according to claim 1, characterized in that: Said S4 further comprises: Based on the quantified storage cost, an initial population of data shards is randomly generated. The fitness of each data shard individual to meet the overall data collection cost is calculated. The roulette wheel selection method is used to select the best individuals in the data shards. The best individuals in the data shards are selected, crossed, and mutated to generate a new generation of individuals for the data shards. Each data shard individual in the initial population of the data shards is replaced based on the fitness of the new generation of individuals for the data shards. The optimal solution for satisfying the data distribution is found, and the data shards are assigned to the node with the lowest data collection cost. This generates a distributed storage management solution for massive data as follows: X q Cost(X q ) is the data item r stored in node X q The storage cost on the server, AccessCost is the access cost, C cost (X q ) is the data entry cost, that is, the cost incurred when accessing the storage node. C storage (X q ) is the data storage cost, that is, the storage usage of each node, Cost is the cost, f(X q ) is the fitness of each data shard to satisfy the overall data input cost, α is used to control the influence of storage cost in the fitness function, and β is used to control the influence of access cost in the fitness function; X * =argmin(f(X)) X * Data is sharded to the node with the lowest data entry overhead, argmin is the optimal solution for data distribution, and a distributed storage management solution for massive data is generated.

Images