Distributed storage system data management method, device, system and electronic equipment

Abstract

The application discloses a distributed storage system data management method, device, system and electronic equipment. The method comprises the following steps: traversing all storage paths in a target distributed storage cluster, and acquiring file storage information of the storage paths in the traversal process; traversing all storage paths in the target distributed storage cluster, and acquiring file storage information of the storage paths in the traversal process; calculating a redundancy strategy score of the storage paths according to the file storage information, and determining a storage strategy adapted to the storage paths according to the redundancy strategy score and a first redundancy strategy score threshold; in the case that the storage strategy adapted to the storage paths is a erasure code storage strategy, determining a data recovery mode of the storage paths according to architecture information of the storage paths; in the case that the storage strategy adapted to the storage paths is a multi-copy storage strategy, determining a copy quantity of the storage paths according to a business importance degree of the storage paths and the redundancy strategy score.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data storage, in particular to a distributed storage system data management method, device, system and electronic equipment. BACKGROUND

[0002] For a distributed storage system, various storage strategies are provided in the related art, but the related art cannot determine the most suitable storage strategy for each storage path in the system, but determines a storage strategy for the whole distributed storage system, which leads to the fact that the storage strategy actually used by the distributed storage system is not the best storage strategy, and further leads to the fact that the storage efficiency of the distributed storage system is poor and the storage overhead is large.

[0003] For the above problems, no effective solution has been proposed so far. SUMMARY

[0004] Embodiments of the present application provide a distributed storage system data management method, device, system and electronic equipment to at least solve the technical problem of low storage efficiency and large storage overhead of the distributed storage system due to the fact that the related art cannot determine the most suitable storage strategy for each storage path in the distributed storage system.

[0005] According to an aspect of an embodiment of the present application, a distributed storage system data management method is provided, comprising: traversing all storage paths in a target distributed storage cluster according to business classification information of the target distributed storage cluster, and obtaining file storage information of the storage paths in the traversing process; calculating a redundancy strategy score of the storage path according to the file storage information, and determining a storage strategy adapted to the storage path according to the redundancy strategy score and a first redundancy strategy score threshold, wherein the redundancy strategy score is used to reflect the adaptation degree between the storage path and a preset storage strategy, and the preset storage strategy includes an erasure coding storage strategy and a multi-copy storage strategy; in the case where the storage strategy adapted to the storage path is determined to be the erasure coding storage strategy, determining a data recovery mode of the storage path according to architecture information of the storage path; in the case where the storage strategy adapted to the storage path is determined to be the multi-copy storage strategy, determining a number of copies of the storage path according to the business importance degree of the storage path and the redundancy strategy score.

[0006] Optionally, the step of determining the storage strategy adapted to the storage path according to the redundancy strategy score and the first redundancy strategy score threshold comprises: in the case where the redundancy strategy score is greater than the first redundancy strategy score threshold, determining that the storage strategy adapted to the storage path is the erasure coding storage strategy; and in the case where the redundancy strategy score is not greater than the first redundancy strategy score threshold, determining that the storage strategy adapted to the storage path is the multi-copy storage strategy.

[0007] Optionally, the architecture information comprises topology information of the storage path; and the step of determining the data recovery mode of the storage path according to the architecture information of the storage path comprises: determining a data transmission index of the storage path according to the topology information of the storage path, wherein the data transmission index is a ratio of data transmitted across racks in the storage path for repairing a failed node in the storage path to all data stored in the storage path; comparing the data transmission index with a preset data transmission index; and determining the data recovery mode of the storage path according to the comparison result.

[0008] Optionally, the step of determining the data recovery mode of the storage path according to the comparison result comprises: in a case where the data transmission index is not greater than the preset data transmission index, determining that the data recovery mode adopted by the storage path is that, in a case where there is a failed node in the storage path, a non-failed node in the storage path directly sends saved data to a rack where the failed node is located; and in a case where the data transmission index is greater than the preset data transmission index, determining that the data recovery mode adopted by the storage path is that, in a case where there is a failed node in the storage path, a check block is generated according to data saved by a non-failed node in each rack of the storage path, and the check block is sent to the failed node.

[0009] Optionally, the step of determining the number of replicas of the storage path according to the service importance degree and the redundancy strategy score of the storage path comprises: determining a second redundancy strategy score threshold, wherein the second redundancy strategy score threshold is less than the first redundancy strategy score threshold; determining a service importance degree evaluation value of the storage path and a preset service importance degree evaluation threshold; comparing the redundancy strategy score with the second redundancy strategy score threshold and comparing the service importance degree evaluation value with the service importance degree evaluation threshold; and determining the number of replicas of the storage path according to the comparison result.

[0010] Optionally, the service importance degree evaluation threshold comprises a first service importance degree evaluation threshold and a second service importance degree evaluation threshold, wherein the first service importance degree evaluation threshold is greater than the second service importance degree evaluation threshold; and the step of determining the number of replicas of the storage path according to the comparison result comprises: in a case where the comparison result is that the redundancy strategy score is less than the second redundancy strategy score threshold or the service importance degree evaluation value is greater than the first service importance degree evaluation threshold, determining that the number of replicas of the storage path is three; in a case where the comparison result is that the service importance degree evaluation value is less than the second service importance degree evaluation threshold and the redundancy strategy score is greater than the second redundancy strategy score threshold and less than the first redundancy strategy score threshold, determining that the number of replicas of the storage path is zero; and in other cases, determining that the number of replicas of the storage path is two.

[0011] Optionally, the file storage information includes at least one of the following: library information, table information, domain information, and tenant information in the storage path; the duration of data not accessed under each access path of the target distributed storage cluster; the amount of stored data under each access path of the target distributed storage cluster; the number of files under each access path of the target distributed storage cluster; the average file size; the number of files; and the file storage format.

[0012] According to another aspect of the embodiments of this application, a data management device for a distributed storage system is also provided, comprising: a first processing module, configured to traverse all storage paths in a target distributed storage cluster based on business classification information of the target distributed storage cluster, and obtain file storage information of the storage paths during the traversal process; a second processing module, configured to calculate a redundancy policy score for the storage path based on the file storage information, and determine a storage policy adapted to the storage path based on the redundancy policy score and a first redundancy policy score threshold, wherein the redundancy policy score is used to reflect the adaptability between the storage path and a preset storage policy, and the preset storage policy includes erasure coding storage policy and multi-replica storage policy; a third processing module, configured to determine a data recovery method for the storage path based on the architecture information of the storage path when it is determined that the storage policy adapted to the storage path is an erasure coding storage policy; and a fourth processing module, configured to determine the number of replicas of the storage path based on the business importance of the storage path and the redundancy policy score when it is determined that the storage policy adapted to the storage path is a multi-replica storage policy.

[0013] According to another aspect of the embodiments of this application, a distributed storage system data management system is also provided, including a target distributed storage cluster and a storage management platform. The target distributed storage cluster is used for storing data. The storage management platform is used to traverse all storage paths in the target distributed storage cluster based on the business classification information of the target distributed storage cluster, and obtain file storage information of the storage paths during the traversal process. It calculates a redundancy policy score for the storage path based on the file storage information, and determines a storage policy suitable for the storage path based on the redundancy policy score and a first redundancy policy score threshold. The redundancy policy score reflects the compatibility between the storage path and a preset storage policy, which includes erasure coding storage policy and multi-replica storage policy. If the storage policy suitable for the storage path is determined to be an erasure coding storage policy, a data recovery method for the storage path is determined based on the architecture information of the storage path. If the storage policy suitable for the storage path is determined to be a multi-replica storage policy, the number of replicas for the storage path is determined based on the business importance of the storage path and the redundancy policy score.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute a distributed storage system data management method when it runs.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes a distributed storage system data management method during runtime.

[0016] In this embodiment, based on the service classification information of the target distributed storage cluster, all storage paths in the target distributed storage cluster are traversed, and file storage information of the storage paths is obtained during the traversal. A redundancy policy score for the storage path is calculated based on the file storage information, and a suitable storage policy is determined based on the redundancy policy score and a first redundancy policy score threshold. The redundancy policy score reflects the compatibility between the storage path and a preset storage policy, which includes erasure coding storage policy and multi-replica storage policy. If the suitable storage policy for the storage path is determined to be an erasure coding storage policy, data recovery for the storage path is determined based on the architecture information of the storage path. The method involves determining the number of replicas for a storage path based on its business importance and redundancy score, when the optimal storage strategy for that path is a multi-replica storage strategy. This is achieved by traversing the storage path to obtain its file storage information and then using that information to determine the appropriate storage strategy. This ensures the storage path operates with the most suitable strategy, thereby improving storage efficiency and reducing overhead. This solves the problem of low storage efficiency and high overhead in distributed storage systems caused by the inability to determine the most suitable storage strategy for each path. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a schematic diagram of the structure of a distributed storage system data management system according to an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a data management method for a distributed storage system according to an embodiment of this application;

[0020] Figure 3This is a schematic diagram of an analysis process for analyzing a distributed storage system according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of a data recovery process using erasure coding according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of a data recovery process using MSR erasure coding according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of a data recovery process using a dual-generated erasure code according to an embodiment of this application;

[0024] Figure 7 This is a flowchart illustrating a data storage strategy analysis process for a distributed storage system according to an embodiment of this application.

[0025] Figure 8 This is a schematic diagram of an erasure coding recovery strategy analysis process provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of a copy quantity analysis process provided according to an embodiment of this application;

[0027] Figure 10 This is a schematic diagram of the structure of a data management device for a distributed storage system according to an embodiment of this application;

[0028] Figure 11 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0032] Erasure coding (EC) is a data protection method that divides data into segments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations. Erasure coding creates a mathematical function to describe a set of numbers, allowing for verification of their accuracy and enabling recovery should a number be lost.

[0033] DCR (Double Regenerating Codes): This application provides an improved data protection method based on EC erasure coding. It is a storage strategy suitable for specific rack and cluster topologies. Compared with traditional EC erasure coding, it can reduce cross-rack traffic transmission to improve data recovery efficiency.

[0034] Currently, while various storage strategies are offered for distributed storage systems, such as erasure coding and multiple replicas, existing technologies cannot definitively determine the most suitable storage strategy for a specific distributed storage system. This means that the storage strategy employed during actual operation is likely not optimal, leading to poor storage efficiency and high overhead. Furthermore, while traditional erasure coding strategies offer high redundancy and storage efficiency, data recovery requires additional computing power and incurs additional network overhead. Traditional multiple replica strategies typically use the same default number of replicas for all data, resulting in additional storage overhead and potentially greater economic losses in production environments.

[0035] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0036] According to an embodiment of this application, a method embodiment for data management in a distributed storage system is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] The method embodiments provided in this application can be executed in a mobile terminal, computer terminal, or similar computing device, such as... Figure 1 This is executed within the data management system of the distributed storage system shown. From... Figure 1 As can be seen, the system includes a target distributed storage cluster 10 and a storage management platform 12, wherein,

[0038] The target is a distributed storage cluster of 10, used for storing data;

[0039] The storage management platform 12 is used to traverse all storage paths in the target distributed storage cluster based on the business classification information of the target distributed storage cluster, and obtain the file storage information of the storage path during the traversal; calculate the redundancy policy score of the storage path based on the file storage information, and determine the storage policy adapted to the storage path based on the redundancy policy score and a first redundancy policy score threshold. The redundancy policy score is used to reflect the adaptability between the storage path and the preset storage policy, which includes erasure coding storage policy and multi-replica storage policy; if the storage policy adapted to the storage path is determined to be erasure coding storage policy, the data recovery method of the storage path is determined based on the architecture information of the storage path; if the storage policy adapted to the storage path is determined to be multi-replica storage policy, the number of replicas of the storage path is determined based on the business importance of the storage path and the redundancy policy score.

[0040] In some embodiments of this application, users can utilize the query function of MySQL and the aforementioned storage management platform 12, combined with business display cluster storage status and visualization tools, to address various visualization and alarm needs in scenarios such as custom cold data standards, storage growth within a time range, and directory location with a large number of small files.

[0041] It should also be noted that the storage management platform 12 provided in this application embodiment has low requirements for the storage components and cluster load in the target distributed storage cluster 10. That is, it does not require the storage path 10 to contain a specific type of storage component, nor does it consume too much load on the target distributed storage cluster 10. For example, for a distributed file system storing 100 million files and a primary / standby namenode, assuming each file requires 200 bytes of metadata information, the metadata information occupies approximately 20GB of storage. The disk size required for the storage profile can then be estimated based on the actual situation of the cluster. If we ignore the changes in the size of the Fsimage file (image file), and denote the size of the Fsimage file obtained from the namenode daily as M_fsimage, taking a cluster with over a thousand nodes as an example, the daily generated Fsimage file size M_fsimage is approximately 60GB. After parsing, the file size expands approximately three times, so the data stored in the Hive database is approximately 200GB. Therefore, for the three common storage components, after adopting the recommended deployment method, the load situation after using the storage management platform 12 provided in this application embodiment is shown in the following table:

[0042]

[0043]

[0044] Under the above operating environment, embodiments of this application provide a data management method for a distributed storage system, such as... Figure 2 As shown, the method includes the following steps:

[0045] Step S202: Based on the business classification information of the target distributed storage cluster, traverse all storage paths in the target distributed storage cluster and obtain the file storage information of the storage paths during the traversal process.

[0046] In the technical solution provided in step S202, the file storage information includes at least one of the following: library information, table information, domain information and tenant information in the target distributed storage cluster, data inaccessibility duration under each access path of the target distributed storage cluster, amount of stored data under each access path of the target distributed storage cluster, number of files under each access path of the target distributed storage cluster, average file size, number of files and file storage format.

[0047] As an optional implementation, when traversing the target distributed storage cluster, Ansible automation scripts can be used to automatically collect and parse the Fsimage files (image files) of the standby named nodes in each cluster, and load the parsing results into the Hive database. After correctly configuring the Hive metadata database, the Hive metadata table can be further obtained during the analysis phase and correlated with the Fsimage files for analysis.

[0048] When analyzing Hive metadata tables in a Hive database and comparing them with Fsimage files, the analysis can be performed using the parsed results of the Fsimage files stored in the Hive database and the Hive metadata tables as input. The analysis can cover aspects such as small files, storage growth, cold data, temporary tables, and compression formats in the cluster storage. Output dimensions include table names, access times, partition information, number of files, and directory levels. The analysis results can be saved to the target distributed database. Furthermore, users can perform secondary analysis on the results according to business needs, or continue with subsequent steps to further process the analysis results.

[0049] Specifically, the process of traversing the target distributed cluster and performing further analysis and processing based on the traversal results is as follows: Figure 3 As shown. From Figure 3 As can be seen, the method provided in this application first pulls image files from each node (NS, NS2, NS3, NS4) and parses Hive metadata. After parsing, the parsing results are imported into the Hive database. Then, a preliminary analysis of the target distributed cluster can be performed based on the parsing results, including cluster storage, small file analysis, storage growth analysis, cold data analysis, Hive table analysis, and other user-defined SQL analysis statements. A secondary analysis is then performed on the preliminary analysis results, including MySQL deployment analysis, quota analysis, and system administrator alarm configuration. Finally, a visualization component can be used to display the results to the user.

[0050] Step S204: Calculate the redundancy strategy score of the storage path based on the file storage information, and determine the storage strategy that the storage path is adapted to based on the redundancy strategy score and the first redundancy strategy score threshold. The redundancy strategy score is used to reflect the compatibility between the storage path and the preset storage strategy. The preset storage strategy includes erasure coding storage strategy and multi-copy storage strategy.

[0051] In the technical solution provided in step S204, the step of determining the storage strategy adapted to the storage path based on the redundancy strategy score and the first redundancy strategy score threshold includes: if the redundancy strategy score is greater than the first redundancy strategy score threshold, the storage strategy adapted to the storage path is determined to be an erasure coding storage strategy; if the redundancy strategy score is not greater than the first redundancy strategy score threshold, the storage strategy adapted to the storage path is determined to be a multi-replica storage strategy.

[0052] Specifically, the parameters and definitions that may be used in calculating the redundancy strategy score of the storage path and in subsequently determining the data recovery method and the number of replicas are shown in the table below:

[0053] Parameter Meaning i Library, table, field, tenant information T Data under path not accessed duration m Storage size under path f File number under path m_avg Average file size under path m_t Small file definition threshold f_small Small file number s_t File storage format M_t Pre-set data transmission index s Redundancy strategy score T1 First redundancy strategy score threshold T2 Second redundancy strategy score threshold I Business importance evaluation value c Cold data coefficient

[0054] Among the parameters mentioned above, mt, T1, T2, and I can all be set by the user after evaluating the storage path. M_t is a preset data transmission index obtained by combining the distributed topology, used to evaluate whether the twin-code correction code data recovery method used in the storage path can effectively reduce the total amount of data transmitted across racks.

[0055] The higher the score of the redundancy strategy rating s, the longer the time since the last access to the data is now. In other words, the data in the storage path is closer to cold data, and the number of small files and the average file size are both small. This indicates that the storage path is more suitable for the erasure coding storage strategy. Conversely, it indicates that the storage path is more suitable for the multi-copy storage strategy or does not need to use redundant backup.

[0056] The above-mentioned importance evaluation value can be determined by the user to assess the storage path, and the value range is 0 < I < 100.

[0057] As an optional implementation, the formula for calculating the redundancy policy score of the storage path is as follows:

[0058]

[0059] Step S206: If the storage strategy adapted to the storage path is determined to be erasure coding storage strategy, the data recovery method of the storage path is determined based on the architecture information of the storage path.

[0060] In the technical solution provided in step S206, the architecture information includes the topology information of the storage path; the step of determining the data recovery method of the storage path based on the architecture information of the storage path includes: determining the data transmission index of the storage path based on the topology information of the storage path, wherein the data transmission index is the ratio of the data transmitted across racks in the storage path to repair the faulty node when there is a faulty node in the storage path to all the data stored in the storage path; comparing the data transmission index with the preset data transmission index; and determining the data recovery method of the storage path based on the comparison result.

[0061] Specifically, if the topology of the storage path is defined as G, then the data transmission index of the storage path is T(G), where T() represents a preset data transmission index calculation function, the input is the topology of the target cluster, and the output is the data transmission index.

[0062] Since the size of the data transmission index is mainly related to the number of racks in the topology and the number of parity blocks on each rack, G can be represented by a function that mainly includes the number of racks and the number of parity blocks. Generally speaking, the fewer the number of racks and the more parity blocks on each rack, the more suitable it is to use the double-generated code method, and the larger T(G) will be.

[0063] In some embodiments of this application, the step of determining the data recovery method of the storage path based on the comparison result includes: when the data transmission index is not greater than the preset data transmission index, the data recovery method adopted by the storage path is determined to be: when there is a faulty node in the storage path, the non-faulty node in the storage path sends the saved data to the rack where the faulty node is located; when the data transmission index is greater than the preset data transmission index, the data recovery method adopted by the storage path is determined to be: when there is a faulty node in the storage path, within each rack of the storage path, a check block is generated based on the data saved by the non-faulty node, and then the check block is sent to the faulty node.

[0064] Specifically, when the data transmission index T(G) is not greater than the preset data transmission index M_t, the recovery method used when recovering data from the faulty node is as follows: Figure 4 or Figure 5 As shown. It can be seen that... Figure 4 In this process, the node directly sends the stored data to the faulty node. Figure 5 The middle part first deduplicates the data stored in the node before sending it to the faulty node. It can be seen that... Figure 5 Data repair methods and Figure 4 Compared to the previous example, under otherwise identical conditions, the amount of data transmitted is 3 / 4.

[0065] When the data transmission index T(G) is greater than the preset data transmission index M_t, the repair method for repairing data in the faulty node is as follows: Figure 6 As shown. From Figure 6 As can be seen, each rack first calculates a check block based on the data of the healthy nodes in the rack, and then sends the check block to the faulty node, thereby reducing the amount of data transmitted across racks.

[0066] It can be seen that when using the dual-generated code data recovery method provided in this application, the bandwidth required for data recovery can be effectively reduced, thus ensuring the efficiency of data recovery.

[0067] Step S208: If the storage strategy adapted to the storage path is determined to be a multi-replica storage strategy, the number of replicas of the storage path is determined based on the business importance of the storage path and the redundancy strategy score.

[0068] In the technical solution provided in step S208, the step of determining the number of replicas of a storage path based on the business importance and redundancy policy score of the storage path includes: determining a second redundancy policy score threshold, wherein the second redundancy policy score threshold is less than a first redundancy policy score threshold; determining a business importance evaluation value for the storage path and a preset business importance evaluation threshold; comparing the redundancy policy score and the second redundancy policy score threshold, and comparing the business importance evaluation value and the business importance evaluation threshold; and determining the number of replicas of the storage path based on the comparison result.

[0069] As an optional implementation, the business importance evaluation threshold includes a first business importance evaluation threshold and a second business importance evaluation threshold, wherein the first business importance evaluation threshold is greater than the second business importance evaluation threshold. The step of determining the number of replicas of the storage path based on the comparison result includes: if the comparison result is that the redundancy policy score is less than the second redundancy policy score threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, the number of replicas of the storage path is determined to be three replicas; if the comparison result is that the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy policy score is greater than the second redundancy policy score threshold but less than the first redundancy policy score threshold, the number of replicas of the storage path is determined to be zero; in other cases, the number of replicas of the storage path is determined to be two replicas.

[0070] In summary, the process described in this application for analyzing the storage paths of the target distributed cluster and determining its appropriate storage strategy is as follows: Figure 7 As shown, it includes the following steps:

[0071] Step S702: Traverse the storage path according to the business category to obtain the file storage information of the storage path;

[0072] Step S704: Determine the redundancy strategy score of the storage path based on the file storage information of the storage path.

[0073] Step S706: Determine the first redundancy strategy scoring threshold, the second redundancy strategy scoring threshold, and the business importance evaluation value of the storage path.

[0074] Step S708: Compare the redundancy strategy score with the first redundancy strategy score threshold. If the comparison result shows that the redundancy strategy score is greater than the first redundancy strategy score threshold, it is confirmed that the storage path is suitable for the erasure coding storage strategy; otherwise, it is indicated that it is suitable for the multi-replica storage strategy.

[0075] For cases where erasure coding is applicable, such as Figure 8 As shown, the applicability of a twin code data recovery strategy can be further determined based on the storage path architecture information, specifically including the following steps:

[0076] Step S802: Obtain the architecture information of the storage path;

[0077] Step S804: Calculate the data transfer metrics of the storage path based on the architecture information;

[0078] Step S806: Compare the data transmission index with the preset data transmission index. If the comparison result shows that the data transmission index is greater than the preset data transmission index, a low-traffic cross-rack data recovery scheme with dual-generated codes is adopted. Otherwise, the following method is used: Figure 4 or Figure 5 The data recovery solution shown.

[0079] For situations where a multi-replica strategy is applicable, such as Figure 9 As shown, the specific number of replicas applicable to the storage path can be further determined, including the following steps:

[0080] Step S902: If the redundancy strategy score is less than the second redundancy strategy score threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, confirm that the storage path is suitable for three replicas; otherwise, proceed to step S904.

[0081] Step S904: If the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy strategy score is greater than the second redundancy strategy score threshold but less than the first redundancy strategy score threshold, confirm that the storage path does not need a copy or that the data in the storage path can be deleted; otherwise, proceed to step S906.

[0082] Step S906: Determine the storage path to be subject to a two-replica storage strategy.

[0083] By employing the business classification information of the target distributed storage cluster, all storage paths in the target distributed storage cluster are traversed, and file storage information of the storage paths is obtained during the traversal. A redundancy policy score for the storage path is calculated based on the file storage information, and a suitable storage policy is determined based on the redundancy policy score and a first redundancy policy score threshold. The redundancy policy score reflects the compatibility between the storage path and a preset storage policy, which includes erasure coding storage policy and multi-replica storage policy. If the suitable storage policy for the storage path is determined to be an erasure coding storage policy, data recovery for the storage path is determined based on the storage path's architecture information. The proposed method involves determining the number of replicas for a storage path based on its business importance and redundancy score, when the optimal storage strategy for that path is a multi-replica storage strategy. This method traverses the storage paths to obtain file storage information and then uses this information to determine the appropriate storage strategy. This ensures that the storage paths operate with the most suitable storage strategy, thereby improving storage efficiency and reducing storage overhead. Furthermore, it addresses the technical problem of low storage efficiency and high storage overhead in distributed storage systems caused by the inability to determine the most suitable storage strategy for each storage path in the system.

[0084] Furthermore, this application analyzes each storage path in the target distributed storage cluster to obtain a storage profile of the target distributed storage cluster for the previous year. The analysis results of the storage profile provide key indicators affecting data storage strategies, such as data storage location, number of small files, data inaccessibility time, and cluster topology. Combining the analysis results of cluster storage information with a series of thresholds set by business personnel for periodic management, specific and advanced redundancy strategies and memory layout strategies can be automatically generated for each storage path.

[0085] This application provides a data management device for a distributed storage system. Figure 10 This is a schematic diagram of the device. (For example...) Figure 10As shown, the device includes: a first processing module 100, used to traverse all storage paths in the target distributed storage cluster based on the business classification information of the target distributed storage cluster, and obtain file storage information of the storage paths during the traversal process; a second processing module 102, used to calculate the redundancy policy score of the storage path based on the file storage information, and determine the storage policy adapted to the storage path based on the redundancy policy score and a first redundancy policy score threshold, wherein the redundancy policy score is used to reflect the adaptability between the storage path and the preset storage policy, and the preset storage policy includes erasure coding storage policy and multi-replica storage policy; a third processing module 104, used to determine the data recovery method of the storage path based on the architecture information of the storage path when it is determined that the storage policy adapted to the storage path is erasure coding storage policy; and a fourth processing module 106, used to determine the number of replicas of the storage path based on the business importance of the storage path and the redundancy policy score when it is determined that the storage policy adapted to the storage path is multi-replica storage policy.

[0086] In some embodiments of this application, the file storage information includes at least one of the following: library information, table information, domain information and tenant information in the target distributed storage cluster, data inaccessibility duration under each access path of the target distributed storage cluster, amount of stored data under each access path of the target distributed storage cluster, number of files under each access path of the target distributed storage cluster, average file size, number of files and file storage format.

[0087] In some embodiments of this application, the step of the second processing module 102 determining the storage strategy adapted to the storage path based on the redundancy strategy score and the first redundancy strategy score threshold includes: if the redundancy strategy score is greater than the first redundancy strategy score threshold, determining that the storage strategy adapted to the storage path is an erasure coding storage strategy; if the redundancy strategy score is not greater than the first redundancy strategy score threshold, determining that the storage strategy adapted to the storage path is a multi-replica storage strategy.

[0088] In some embodiments of this application, the architecture information includes the topology information of the storage path; the step of the third processing module 104 determining the data recovery method of the storage path based on the architecture information of the storage path includes: determining the data transmission index of the storage path based on the topology information of the storage path, wherein the data transmission index is the ratio of the data transmitted across racks in the storage path to repair the faulty node when there is a faulty node in the storage path to all the data stored in the storage path; comparing the data transmission index with a preset data transmission index; and determining the data recovery method of the storage path based on the comparison result.

[0089] In some embodiments of this application, the step of the third processing module 104 determining the data recovery method of the storage path based on the comparison result includes: when the data transmission index is not greater than the preset data transmission index, the data recovery method adopted by the storage path is determined to be: when there is a faulty node in the storage path, the non-faulty node in the storage path directly sends the saved data to the rack where the faulty node is located; when the data transmission index is greater than the preset data transmission index, the data recovery method adopted by the storage path is determined to be: when there is a faulty node in the storage path, within each rack of the storage path, a check block is generated based on the data saved by the non-faulty node, and then the check block is sent to the faulty node.

[0090] In some embodiments of this application, the step of the fourth processing module 106 determining the number of replicas of a storage path based on the business importance and redundancy policy score of the storage path includes: determining a second redundancy policy score threshold, wherein the second redundancy policy score threshold is less than a first redundancy policy score threshold; determining a business importance evaluation value of the storage path and a preset business importance evaluation threshold; comparing the redundancy policy score and the second redundancy policy score threshold, and comparing the business importance evaluation value and the business importance evaluation threshold; and determining the number of replicas of the storage path based on the comparison result.

[0091] In some embodiments of this application, the business importance evaluation threshold includes a first business importance evaluation threshold and a second business importance evaluation threshold, wherein the first business importance evaluation threshold is greater than the second business importance evaluation threshold; the step of the fourth processing module 106 determining the number of replicas of the storage path based on the comparison result includes: if the comparison result is that the redundancy policy score is less than the second redundancy policy score threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, determining the number of replicas of the storage path to be three replicas; if the comparison result is that the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy policy score is greater than the second redundancy policy score threshold but less than the first redundancy policy score threshold, determining the number of replicas of the storage path to be zero; in other cases, determining the number of replicas of the storage path to be two replicas.

[0092] It should be noted that each module in the above-mentioned distributed storage system data management device can be a program module (for example, a set of program instructions that implements a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0093] Figure 11 A hardware block diagram of a computer terminal (or mobile device) for implementing a data management method in a distributed storage system is shown.Figure 11 As shown, the computer terminal 110 (or mobile device 110) may include one or more processors 1102 (shown as 1102a, 1102b, ..., 1102n in the figure) (processor 1102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1104 for storing data, and a transmission module 1106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 11 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the computer terminal 110 may also include... Figure 11 The more or fewer components shown, or having the same Figure 11 The different configurations shown.

[0094] It should be noted that the aforementioned one or more processors 1102 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 110 (or mobile device). As involved in the embodiments of this application, this data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0095] The memory 1104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the distributed storage system data management method in this embodiment. The processor 1102 executes various functional applications and data processing by running the software programs and modules stored in the memory 1104, thereby realizing the aforementioned distributed storage system data management method. The memory 1104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1104 may further include memory remotely located relative to the processor 1102, and these remote memories can be connected to the computer terminal 110 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0096] The transmission device 1106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 110. In one example, the transmission device 1106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 1106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0097] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 110 (or mobile device).

[0098] According to an embodiment of this application, a non-volatile storage medium is also provided, which stores a program. During program execution, the program controls the device containing the non-volatile storage medium to perform the following distributed storage system data management method: Based on the business classification information of the target distributed storage cluster, traverse all storage paths in the target distributed storage cluster, and obtain file storage information of the storage paths during the traversal; calculate the redundancy policy score of the storage path based on the file storage information, and determine the storage policy suitable for the storage path based on the redundancy policy score and a first redundancy policy score threshold. The redundancy policy score reflects the compatibility between the storage path and a preset storage policy, which includes erasure coding storage policy and multi-replica storage policy; if the storage policy suitable for the storage path is determined to be an erasure coding storage policy, determine the data recovery method of the storage path based on the architecture information of the storage path; if the storage policy suitable for the storage path is determined to be a multi-replica storage policy, determine the number of replicas of the storage path based on the business importance of the storage path and the redundancy policy score.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0102] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0103] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data management method for a distributed storage system, characterized in that, include: Based on the business classification information of the target distributed storage cluster, all storage paths in the target distributed storage cluster are traversed, and file storage information of the storage paths is obtained during the traversal. The redundancy policy score for each storage path is calculated based on the file storage information. The storage policy adapted to each storage path is then determined based on the redundancy policy score and a first redundancy policy score threshold. The redundancy policy score reflects the compatibility between the storage path and a preset storage policy, which includes erasure coding and multiple-copy storage policies. If the redundancy policy score is greater than the first redundancy policy score threshold, the storage policy adapted to the storage path is determined to be the erasure coding storage policy; if the redundancy policy score is not greater than the first redundancy policy score threshold, the storage policy adapted to the storage path is determined to be the multiple-copy storage policy. If the storage policy adapted to the storage path is determined to be the erasure coding storage policy, the data recovery method of the storage path is determined based on the architecture information of the storage path. If the storage policy adapted to the storage path is determined to be the multi-replica storage policy, the number of replicas of the storage path is determined based on the business importance of the storage path and the redundancy policy score.

2. The data management method for a distributed storage system according to claim 1, characterized in that, The architecture information includes the topology information of each storage path in the target distributed storage cluster; The step of determining the data recovery method for the storage path based on the architecture information of the storage path includes: Based on the topology information, a data transmission index for the storage path is determined, wherein the data transmission index is the ratio of the data transmitted across racks in the storage path to repair the faulty node to all the data stored in the storage path when a faulty node exists in the storage path. Compare the data transmission metrics with the preset data transmission metrics; The data recovery method for the storage path is determined based on the comparison results.

3. The data management method for a distributed storage system according to claim 2, characterized in that, The step of determining the data recovery method for the storage path based on the comparison results includes: If the data transmission index is not greater than the preset data transmission index, the data recovery method adopted by the storage path is determined as follows: if there is a faulty node in the storage path, the non-faulty node in the storage path sends the saved data to the rack where the faulty node is located. If the data transmission index is greater than the preset data transmission index, the data recovery method adopted by the storage path is determined to be as follows: if there is a faulty node in the storage path, a check block is generated in each rack of the storage path based on the data saved by the non-faulty node, and then the check block is sent to the faulty node.

4. The data management method for a distributed storage system according to claim 1, characterized in that, The step of determining the number of replicas for the storage path based on the business importance of the storage path and the redundancy policy score includes: A second redundancy strategy scoring threshold is determined, wherein the second redundancy strategy scoring threshold is less than the first redundancy strategy scoring threshold; Determine the business importance evaluation value of the storage path, and the preset business importance evaluation threshold; Compare the redundancy strategy score with the second redundancy strategy score threshold, and compare the business importance evaluation value with the business importance evaluation threshold; The number of copies of the storage path is determined based on the comparison results.

5. The data management method for a distributed storage system according to claim 4, characterized in that, The business importance evaluation threshold includes a first business importance evaluation threshold and a second business importance evaluation threshold, wherein the first business importance evaluation threshold is greater than the second business importance evaluation threshold; the step of determining the number of replicas of the storage path based on the comparison result includes: If the comparison result is that the redundancy strategy score is less than the second redundancy strategy score threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, the number of replicas of the storage path is determined to be three replicas. If the comparison result shows that the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy strategy score is greater than the second redundancy strategy score threshold but less than the first redundancy strategy score threshold, then the number of replicas of the storage path is determined to be zero. In all other cases, the number of replicas of the storage path is determined to be two replicas.

6. The data management method for a distributed storage system according to claim 1, characterized in that, The file storage information includes at least one of the following: library information, table information, domain information, and tenant information in the target distributed storage cluster; data inaccessibility duration under each access path of the target distributed storage cluster; the amount of stored data under each access path of the target distributed storage cluster; the number of files under each access path of the target distributed storage cluster; the average file size; the number of files; and the file storage format.

7. A data management device for a distributed storage system, characterized in that, include: The first processing module is used to traverse all storage paths in the target distributed storage cluster based on the business classification information of the target distributed storage cluster, and obtain the file storage information of the storage path during the traversal process. The second processing module is used to calculate the redundancy policy score of the storage path based on the file storage information, and to determine the storage policy adapted to the storage path based on the redundancy policy score and a first redundancy policy score threshold. The redundancy policy score reflects the compatibility between the storage path and a preset storage policy, which includes erasure coding storage policy and a multi-replica storage policy. If the redundancy policy score is greater than the first redundancy policy score threshold, the storage policy adapted to the storage path is determined to be the erasure coding storage policy; if the redundancy policy score is not greater than the first redundancy policy score threshold, the storage policy adapted to the storage path is determined to be the multi-replica storage policy. The third processing module is used to determine the data recovery method of the storage path based on the architecture information of the storage path when it is determined that the storage strategy adapted to the storage path is the erasure coding storage strategy. The fourth processing module is used to determine the number of replicas of the storage path based on the business importance of the storage path and the redundancy policy score when it is determined that the storage policy adapted to the storage path is the multi-replica storage policy.

8. A data management system for a distributed storage system, characterized in that, This includes the target distributed storage cluster and the storage management platform, among which, The target distributed storage cluster is used to store data; The storage management platform is used to traverse all storage paths in the target distributed storage cluster based on the business classification information of the target distributed storage cluster, and obtain file storage information of the storage paths during the traversal. It calculates a redundancy policy score for the storage path based on the file storage information, and determines the appropriate storage policy for the storage path based on the redundancy policy score and a first redundancy policy score threshold. The redundancy policy score reflects the compatibility between the storage path and a preset storage policy, which includes erasure coding and multi-replica storage policies. If the redundancy policy score is greater than the first redundancy policy score threshold, the storage policy appropriate for the storage path is the erasure coding storage policy; if the redundancy policy score is not greater than the first redundancy policy score threshold, the storage policy appropriate for the storage path is the multi-replica storage policy. If the storage policy appropriate for the storage path is the erasure coding storage policy, the platform determines the data recovery method for the storage path based on the architecture information of the storage path. If the storage policy appropriate for the storage path is the multi-replica storage policy, the platform determines the number of replicas for the storage path based on the business importance of the storage path and the redundancy policy score.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to execute the distributed storage system data management method according to any one of claims 1 to 6.

10. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the distributed storage system data management method according to any one of claims 1 to 6.

Images