A distributed storage system data management method

Abstract

The present invention belongs to the technical field of data management, and provides a distributed storage system data management method, including: obtaining historical data storage information in a storage node and processing it to obtain a data key value; classifying the data into important data and ordinary data based on the data key value; obtaining historical security data of each storage node in the distributed storage system and processing it to obtain a node risk value of the storage node and sorting the storage nodes for security; allocating an expected allocation position for the important data based on the security sorting, if the current storage position is different from the expected allocation position, generating a position adjustment, and comparing the storage capacity required for the important data with the corresponding expected allocation position storage node limited remaining capacity, and adjusting the storage position of the important data, if there is no position adjustment space, comparing the data information of the ordinary data and processing it, so as to realize data management of the storage node.

Description

Technical Field

[0001] The invention belongs to the technical field of data management, and in particular to a distributed storage system data management method. Background Art

[0002] With the rapid development of information technology, the amount of data has exploded, and traditional centralized storage systems can no longer meet the needs of large-scale data storage and processing. Centralized storage systems usually rely on a single storage server, which not only becomes a bottleneck for system performance, but also limits the reliability and security of the system. In order to meet this challenge, distributed storage systems came into being.

[0003] In the actual application of distributed storage systems, with the continuous increase in the amount of stored data, the demand for data management and optimization has become increasingly prominent. Traditional data storage management methods often ignore the importance of data and the security of nodes, resulting in low storage efficiency, unreasonable resource allocation, and decreased data access performance. Information such as the access frequency, storage duration, and non-access duration of different data directly reflects the activity and value of the data. However, traditional storage management methods usually treat all data equally and do not classify and optimize them according to the criticality of the data, resulting in important data may not be able to obtain sufficient storage resources and protection; secondly, the security of storage nodes is also an important factor affecting system performance. Distributed storage systems contain multiple storage nodes, and the security of each node directly determines the reliability of the entire system. Traditional storage management methods often ignore the risk value of nodes and do not conduct effective security assessment and sorting; reasonable allocation and optimization of storage resources is the key to improving system performance. Traditional storage management methods are usually based on static resource configuration and do not make dynamic adjustments based on the importance of data and the security of nodes, resulting in unreasonable storage resource allocation. Some nodes may affect performance due to insufficient resources, while other nodes may be wasted due to excess resources.

[0004] To this end, the present invention provides a distributed storage system data management method. Summary of the invention

[0005] The object of the present invention is to provide a distributed storage system data management method to solve at least one of the above-mentioned prior art problems.

[0006] The present invention provides a distributed storage system data management method, comprising the following steps:

[0007] Obtaining historical data storage information in the storage node, wherein the historical data storage information includes the number of data accesses, the data storage duration, and the data non-access duration, processing the historical data storage information to obtain data key values, and classifying the data into important data and ordinary data based on the data key values;

[0008] Process the number of times each storage node in the distributed storage system is attacked and the number of times the attack interception fails when attacked, obtain the node risk value of the storage node, and sort the storage nodes by security based on the node risk value;

[0009] Allocate an expected allocation position for the important data based on the data key value of the important data and the security ranking of the storage node, compare the current storage position of the important data with the expected allocation position, and generate a position adjustment signal if the current storage position is different from the expected allocation position;

[0010] Based on the generated position adjustment signal, the required storage capacity of the important data is compared with the corresponding expected allocation position storage node limited remaining capacity, the storage position of the important data is adjusted according to the comparison result, and it is determined whether there is position adjustment space in the storage node. If not, a capacity optimization signal is generated;

[0011] Based on the generated capacity optimization signal and the marked ordinary data, data information of the ordinary data is obtained, wherein the data information includes the data key value of the ordinary data and the number of bytes contained in the redundant backup of the ordinary data. The data information is processed to obtain the optimized performance value, and data management of the storage node is performed according to the obtained optimized performance value.

[0012] The beneficial effects of the present invention are as follows:

[0013] 1. Traverse all storage paths in the target distributed storage system, and obtain historical data storage information in the storage nodes during the traversal process, where the historical data storage information includes the number of data accesses, the length of time the data was stored, and the length of time the data was not accessed. Process the historical data storage information to obtain data key values; obtain the data key values ​​of the historical storage data, and classify the data into important data and ordinary data based on the data key values, so as to facilitate subsequent optimization of the storage system;

[0014] 2. Obtain the historical security data of each storage node in the distributed storage system and perform data processing on it to obtain the node risk value of the storage node, and rank the storage nodes for security based on the node risk value; assign expected allocation positions to important data based on the data key value of the important data and the security ranking of the storage node, compare the current storage position of the important data with the expected allocation position, and generate a position adjustment signal if the current storage position is different from the expected allocation position; based on the generated position adjustment signal, obtain the storage capacity required for the important data, and compare it with the corresponding expected allocation position storage node limited remaining capacity, adjust the storage position of the important data according to the comparison result, and determine whether there is position adjustment space in the storage node, if not, generate a capacity optimization signal; based on the generated capacity optimization signal and the marked ordinary data, obtain the data information of the ordinary data and process it to obtain the optimized performance value, and manage the storage nodes according to the obtained optimized performance value. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention will be further described below in conjunction with the accompanying drawings.

[0016] Figure 1 is a flowchart of the steps of a distributed storage system data management method according to an embodiment of the present invention;

[0017] Figure 2 It is a flow chart of the steps for obtaining data key values ​​in a distributed storage system data management method described in Example 1 of the present invention. DETAILED DESCRIPTION

[0018] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0019] Example 1

[0020] like Figure 1 As shown, a distributed storage system data management method according to an embodiment of the present invention includes:

[0021] Step 1: traverse all storage paths in the target distributed storage system, and obtain historical data storage information in the storage node during the traversal process, wherein the historical data storage information includes the number of data accesses, the data storage duration, and the data non-access duration, and process the historical data storage information to obtain the data key value;

[0022] like Figure 2 As shown in the figure, the specific method of obtaining the key value of the data is:

[0023] Obtain the total data access times of all historical storage data in the distributed storage system and the time the distributed storage system has been in use, perform ratio processing on the total data access times and the time the distributed storage system has been in use, and obtain the total access frequency value;

[0024] Based on any historical storage data in the distributed storage system:

[0025] Obtain the number of data accesses and the data storage duration of historical storage data, perform ratio processing on the number of data accesses and the data storage duration to obtain an access frequency value, perform ratio processing on the obtained access frequency value and the total access frequency value to obtain an access frequency ratio, which is marked as XT;

[0026] Obtain the data non-access duration of historical storage data, perform a ratio processing on the data non-access duration and the time the distributed storage system has been put into use, and obtain the non-access duration ratio, which is marked as XL.

[0027] The data of access frequency ratio XT and non-access duration ratio XL are processed, and the formula The data key value YX is obtained, where s1 and s2 are both preset proportional coefficients, s1=3.171, s2=1.276;

[0028] It should be noted that the purpose of the data key value YX is to analyze the access frequency and non-access duration of historical storage data in the distributed storage system, and then evaluate the importance of historical storage data, so as to facilitate the subsequent optimization of the storage system;

[0029] Step 2: Obtain data key values ​​of historical storage data, and classify the data into important data and common data based on the data key values;

[0030] Specifically, the obtained data key value is compared with the data key threshold:

[0031] If the data key value of the historical stored data is greater than the data key threshold, the stored data is marked as important data;

[0032] If the data key value of the historical stored data is less than or equal to the data key threshold, the stored data will be marked as normal data;

[0033] The technical solution of an embodiment of the present invention is: traverse all storage paths in the target distributed storage system, and obtain historical data storage information in the storage nodes during the traversal process, wherein the historical data storage information includes the number of data accesses, the length of time the data is stored, and the length of time the data is not accessed, process the historical data storage information to obtain data key values; obtain the data key values ​​of the historical storage data, and classify the data into important data and ordinary data based on the data key values; and facilitate subsequent optimization of the storage system.

[0034] Example 2

[0035] like Figure 1 As shown, based on Example 1, a distributed storage system data management method according to an embodiment of the present invention includes:

[0036] Step 3: Obtain historical security data of each storage node in the distributed storage system, the historical security data including the number of times the storage node was attacked and the number of times the attack interception failed when the storage node was attacked, process the historical security data to obtain the node risk value of the storage node, and sort the storage nodes by security based on the node risk value;

[0037] Specifically, based on any storage node, the number of times the storage node is attacked in the historical information is obtained through the background, and the ratio is processed with the sum of the number of times all storage nodes are attacked to obtain the attack number ratio, which is marked as KA;

[0038] Obtain the number of failed interception attempts when the storage node is attacked, and compare it with the number of attacks on the storage node to obtain the interception failure ratio, marked as KB.

[0039] The attack frequency ratio KA and interception failure ratio KB are processed and the formula The node risk value CG is obtained, where a1 and a2 are both preset proportional coefficients, a1=2.079, a2=3.196;

[0040] It should be noted that the meaning of the node risk value is to judge the risk of the storage node by analyzing the number of times the storage node is attacked and the number of times the attack is not successfully intercepted. The higher the node risk value, the higher the risk of the storage node and the lower the security.

[0041] Compare the node risk values ​​of all storage nodes in the distributed storage system, sort the storage nodes by security from small to large according to the node risk value, and mark the storage nodes with serial numbers according to the sorting;

[0042] Step 4: Allocate an expected allocation position for the important data based on the data key value of the important data and the security ranking of the storage node, compare the current storage position of the important data with the expected allocation position, and generate a position adjustment signal if the current storage position is different from the expected allocation position;

[0043] Specifically, the data key values ​​of the important data are obtained, the important data are sorted from large to small according to the data key values, and serial numbers are marked for the important data according to the sorting;

[0044] Obtain the total number of important data copies and the total number of storage nodes in the distributed storage system and compare them;

[0045] If the total number of important data is less than or equal to the total number of storage nodes, the storage nodes with the same serial number will be used as the expected allocation locations of the important data;

[0046] If the total number of important data is greater than the total number of storage nodes, an average distribution signal is generated;

[0047] Based on the generated average allocation signal, the total number of important data in the distributed storage system is processed by ratio with the total number of storage nodes to obtain the number of storage node allocations. If the number of storage node allocations contains decimals, only integers are retained and the values ​​are incremented.

[0048] According to the order of important data and the security order of storage nodes, each storage node is sequentially allocated with a number of copies of important data equal to the number of storage nodes allocated, and the expected allocation position of the storage node is obtained;

[0049] For example, if there are 30 storage nodes in a distributed storage system, and 79 copies of important data are stored in the 30 storage nodes, then the number of storage node allocations is 3, that is, the expected allocation positions of the important data with serial numbers 1, 2, and 3 are the storage node with serial number 1, and the expected allocation positions of the important data with serial numbers 4, 5, and 6 are the storage node with serial number 2... and so on, until each important data has its expected allocation position;

[0050] Based on any important data, comparing the current storage location of the important data with the expected allocation location;

[0051] If the current storage location of important data is the same as the expected allocation location, no processing is done;

[0052] If the current storage location of the important data is different from the expected allocation location, a position adjustment signal is generated;

[0053] Step 5: Based on the generated position adjustment signal, the required storage capacity of the important data is obtained, and it is compared with the corresponding expected allocation position storage node limited remaining capacity, the storage position of the important data is adjusted according to the comparison result, and it is determined whether there is position adjustment space in the storage node. If not, a capacity optimization signal is generated;

[0054] Specifically, based on any important data, the storage capacity required for the important data and the remaining capacity limited by the corresponding expected allocation location storage node are obtained and compared and analyzed;

[0055] If the storage capacity required for the important data is less than or equal to the remaining capacity of the corresponding storage node at the expected allocation position, the important data is adjusted, and the important data is stored in the storage node at the expected allocation position, and the important data originally stored in the storage node is cleared;

[0056] If the storage capacity required for important data is greater than the remaining capacity of the storage node at the corresponding expected allocation location, obtain the storage node with the largest remaining capacity at this time, and compare the storage capacity required for important data with the maximum remaining capacity at this time;

[0057] If the storage capacity required for important data is less than or equal to the maximum remaining capacity at this time, the important data is temporarily stored in the storage node with the largest remaining capacity, and the important data originally stored in the storage node is cleared, a data temporary storage signal is generated, and the sequence number of the important data is recorded;

[0058] If the storage capacity required for important data is greater than the maximum remaining capacity at this time, it is determined that there is no position adjustment space in the storage node, and a capacity optimization signal is generated;

[0059] It should be noted that the purpose of generating the data temporary storage signal is that during the process of adjusting the important data to the expected allocation position, if the limited remaining capacity in the storage node at the expected allocation position does not meet the storage requirements of the allocated important data due to the storage of a large amount of important data to be adjusted, the important data is first temporarily stored in the storage node with the largest limited remaining capacity, and after a large amount of important data to be adjusted is adjusted, there is sufficient limited remaining capacity in the storage node at the expected allocation position, and then the temporarily stored important data is stored accordingly;

[0060] The operation is performed on all important data in the distributed storage system. If no capacity optimization signal is generated after the operation is performed on all important data, the required capacity for storing the important data that generates the data temporary storage signal is compared again with the remaining capacity limited by the storage node at the corresponding expected allocation position;

[0061] If the storage capacity required for the important data is less than or equal to the corresponding expected allocation location storage node limit remaining capacity, the location adjustment operation is performed;

[0062] If at this time there is still important data whose storage capacity is greater than the corresponding expected allocation location storage node limit remaining capacity, a capacity optimization signal is generated;

[0063] Step 6: Based on the generated capacity optimization signal and the marked common data, data information of the common data is obtained, wherein the data information includes the data key value of the common data and the number of bytes included in the redundant backup of the common data, the data information is processed to obtain the optimization performance value, and data management of the storage node is performed according to the obtained optimization performance value;

[0064] Specifically, the number of bytes included in the common data redundant backup is obtained based on the generated capacity optimization signal, and the ratio is processed with the number of bytes of the total capacity of the distributed storage system to obtain the redundant byte ratio, which is marked as YR;

[0065] Get the data key value YX of the common data, process the obtained redundant byte ratio YR and the data key value YX, and use the formula The optimized performance value CR is obtained, where b1 and b2 are both preset proportional coefficients, b1=2.134, b2=1.009;

[0066] It should be noted that the purpose of optimizing the performance value CR is to analyze the size and importance of common data, and then make more efficient planning for the capacity optimization of the distributed storage system;

[0067] Sort the common data from large to small according to the optimization performance value to generate an optimization sequence table, delete the redundant backups of the common data in the optimization sequence table in sequence, until the required storage capacity of all important data not stored in the expected allocation position is less than or equal to the corresponding expected allocation position storage node limit remaining capacity, then stop deleting, and perform the position adjustment operation on the important data;

[0068] If all redundant backups of common data have been deleted and there is still important data that cannot be stored in the expected allocation location, an adjustment incomplete signal is generated and staff are arranged to handle it;

[0069] The technical solution of the embodiment of the present invention is: obtaining historical security data of each storage node in a distributed storage system and performing data processing on the data to obtain the node risk value of the storage node, and ranking the storage nodes for security based on the node risk value; allocating expected allocation positions for important data based on the data key value of the important data and the security ranking of the storage node, comparing the current storage position of the important data with the expected allocation position, and generating a position adjustment signal if the current storage position is different from the expected allocation position; based on the generated position adjustment signal, obtaining the storage capacity required for the important data, and comparing it with the corresponding expected allocation position storage node limited remaining capacity, adjusting the storage position of the important data according to the comparison result, and judging whether there is position adjustment space in the storage node, if not, generating a capacity optimization signal; based on the generated capacity optimization signal and the marked ordinary data, obtaining data information of the ordinary data and processing it to obtain an optimized performance value, and performing data management on the storage node according to the obtained optimized performance value.

[0070] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A distributed storage system data management method, characterized in that: include: Obtain the historical data storage information in the storage node, where the historical data storage information includes the number of data accesses, the data storage duration, and the data non-access duration. Process the historical data storage information and calculate the access frequency ratio XT and the non-access duration ratio XL. The data key value YX is obtained, where s1 and s2 are preset proportional coefficients, and the data is classified into important data and ordinary data based on the data key value; Process the number of times each storage node in the distributed storage system is attacked and the number of times the attack interception fails when attacked, obtain the node risk value of the storage node, and sort the storage nodes by security based on the node risk value; The specific method of obtaining the node risk value is as follows: Based on any storage node, obtain the number of attacks on the storage node in the historical information through the background, and compare it with the sum of the number of attacks on all storage nodes to obtain the attack number ratio, which is marked as KA; Obtain the number of failed interception attempts when the storage node is attacked, and compare it with the number of attacks on the storage node to obtain the interception failure ratio, marked as KB. By formula Get the node risk value CG, where a1 and a2 are both preset proportional coefficients; The specific method of ranking storage nodes for security is as follows: Compare the node risk values ​​of all storage nodes in the distributed storage system, sort the storage nodes by security from small to large according to the node risk value, and mark the storage nodes with serial numbers according to the sorting; Allocate an expected allocation position for the important data based on the data key value of the important data and the security ranking of the storage node, compare the current storage position of the important data with the expected allocation position, and generate a position adjustment signal if the current storage position is different from the expected allocation position; Based on the generated position adjustment signal, the required storage capacity of the important data is compared with the corresponding expected allocation position storage node limited remaining capacity, the storage position of the important data is adjusted according to the comparison result, and it is determined whether there is position adjustment space in the storage node. If not, a capacity optimization signal is generated; Based on the generated capacity optimization signal and the marked ordinary data, data information of the ordinary data is obtained, wherein the data information includes the data key value of the ordinary data and the number of bytes contained in the redundant backup of the ordinary data, the data information is processed to obtain the optimized performance value, and data management of the storage node is performed according to the obtained optimized performance value; The specific method of data management for storage nodes is as follows: Sort the common data from large to small according to the optimization performance value to generate an optimization sequence table, and delete the redundant backups of the common data in the optimization sequence table in sequence until the required storage capacity of all important data that are not stored in the expected allocation position is less than or equal to the corresponding expected allocation position storage node limit remaining capacity, then stop deleting and perform position adjustment operations on the important data; If all redundant backups of common data have been deleted and there is still important data that cannot be stored in the expected allocation location, an adjustment incomplete signal is generated and staff are arranged to make adjustments; The specific method of obtaining the optimization performance value is as follows: Based on the generated capacity optimization signal, the number of bytes contained in the common data redundant backup is obtained, and the ratio is processed with the number of bytes of the total capacity of the distributed storage system to obtain the redundant byte ratio, which is marked as YR; By formula The optimized performance value CR is obtained, wherein b1 and b2 are both preset proportional coefficients.

2. A distributed storage system data management method according to claim 1, characterized in that: The specific method of obtaining the access frequency ratio XT and the non-access duration ratio XL is as follows: Obtain the total data access times of all historical storage data in the distributed storage system and the time the distributed storage system has been in use, perform ratio processing on the total data access times and the time the distributed storage system has been in use, and obtain the total access frequency value; Obtain the number of data accesses and the data storage duration of historical storage data, perform ratio processing on the number of data accesses and the data storage duration to obtain an access frequency value, perform ratio processing on the obtained access frequency value and the total access frequency value to obtain an access frequency ratio, which is marked as XT; The data non-access time of historical storage data is obtained, and the data non-access time is ratioed with the time when the distributed storage system is put into use to obtain the non-access time ratio, which is marked as XL.

3. A distributed storage system data management method according to claim 1, characterized in that: The specific method of classifying data into important data and ordinary data is as follows: Compare the obtained data key value with the data key threshold: If the data key value of the historical stored data is greater than the data key threshold, the stored data is marked as important data; If the data key value of the historical stored data is less than or equal to the data key threshold, the stored data will be marked as normal data.

4. A distributed storage system data management method according to claim 1, characterized in that: The specific method of generating the position adjustment signal is: Obtain data key values ​​of important data, sort the important data from large to small according to the data key values, and mark the important data with serial numbers according to the sorting; Obtain the total number of important data copies and the total number of storage nodes in the distributed storage system and compare them; If the total number of important data is less than or equal to the total number of storage nodes, the storage nodes with the same serial number will be used as the expected allocation locations of the important data; If the total number of important data is greater than the total number of storage nodes, an average distribution signal is generated; Based on the generated average allocation signal, the total number of important data in the distributed storage system is processed by ratio with the total number of storage nodes to obtain the number of storage node allocations. If the number of storage node allocations contains decimals, only integers are retained and the values ​​are incremented. According to the order of important data and the security order of storage nodes, each storage node is sequentially allocated with a number of copies of important data equal to the number of storage nodes allocated, and the expected allocation position of the storage node is obtained; Based on any important data, comparing the current storage location of the important data with the expected allocation location; If the current storage location of the important data is different from the expected allocation location, a location adjustment signal is generated.

5. A distributed storage system data management method according to claim 1, characterized in that: The specific method of generating the capacity optimization signal is: Based on any important data, obtain the storage capacity required for the important data and the remaining capacity limited by the corresponding expected allocation location storage node and perform comparative analysis; If the storage capacity required for the important data is less than or equal to the remaining capacity of the corresponding storage node at the expected allocation position, the important data is adjusted, and the important data is stored in the storage node at the expected allocation position, and the important data originally stored in the storage node is cleared; If the storage capacity required for important data is greater than the remaining capacity of the storage node at the corresponding expected allocation location, obtain the storage node with the largest remaining capacity at this time, and compare the storage capacity required for important data with the maximum remaining capacity at this time; If the storage capacity required for important data is less than or equal to the maximum remaining capacity at this time, the important data is temporarily stored in the storage node with the largest remaining capacity, and the important data originally stored in the storage node is cleared, a data temporary storage signal is generated, and the sequence number of the important data is recorded; If the storage capacity required for important data is greater than the maximum remaining capacity at this time, it is determined that there is no position adjustment space in the storage node, and a capacity optimization signal is generated; Perform the position adjustment operation on all important data in the distributed storage system. If no capacity optimization signal is generated after performing the position adjustment operation on all important data, compare the required capacity of the important data storage that generates the data temporary storage signal with the remaining capacity of the storage node corresponding to the expected allocation position again; If the storage capacity required for the important data is less than or equal to the corresponding expected allocation location storage node limit remaining capacity, the location adjustment operation is performed; If at this time there is still important data whose storage capacity is greater than the corresponding expected allocation location storage node limit remaining capacity, a capacity optimization signal is generated.

Images