Data management method, apparatus and system for distributed storage system, and electronic device

Abstract

The present application discloses a data management method, apparatus and system for a distributed storage system, and an electronic device. The method comprises: traversing all storage paths in a target distributed storage cluster, and acquiring file storage information of the storage paths during traversal; on the basis of the file storage information, calculating a redundancy policy score of each storage path, and on the basis of the redundancy policy score and a first redundancy policy score threshold, determining a storage policy adapted to the storage path; when it is determined that the storage policy adapted to the storage path is an erasure code storage policy, determining a data recovery mode of the storage path on the basis of the architecture information of the storage path; and when it is determined that the storage policy adapted to the storage path is a multi-copy storage policy, determining the number of copies of the storage path on the basis of a service importance degree of the storage path and the redundancy policy score.

Description

Distributed storage system data management method, device, system and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 29, 2023, with application number 2023116194449, and application name “Distributed Storage System Data Management Method, Device, System and Electronic Equipment”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data storage, and more specifically, to a distributed storage system data management method, device, system, and electronic equipment. Background Art

[0003] For distributed storage systems, various storage strategies are provided in related technologies. However, currently related technologies are unable to determine the most suitable storage strategy for each storage path in a distributed storage system. Instead, a storage strategy is determined for the entire distributed storage system. As a result, the storage strategy adopted during the actual operation of the distributed storage system is likely not the optimal storage strategy, which in turn leads to poor storage efficiency and high overhead of the distributed storage system.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a distributed storage system data management method, device, system and electronic device to at least solve the technical problem that the distributed storage system has low storage efficiency and high storage overhead due to the inability to determine the most appropriate storage strategy for each storage path in the distributed storage system in the related art.

[0007] According to one aspect of an embodiment of the present application, a distributed storage system data management method is provided, comprising: traversing all storage paths in a target distributed storage cluster based on business classification information of the target distributed storage cluster, and obtaining file storage information of the storage paths during the traversal process; calculating a redundancy policy score for each storage path based on the file storage information, and determining a storage policy adapted for the storage path based on the redundancy policy score and a first redundancy policy score threshold, wherein the redundancy policy score is used to reflect the degree of compatibility between the storage path and a preset storage policy, the preset storage policy including an erasure code storage policy and a multi-copy storage policy; when it is determined that the storage policy adapted for the storage path is the erasure code storage policy, determining a data recovery method for the storage path based on the architecture information of the storage path; when it is determined that the storage policy adapted for the storage path is the multi-copy storage policy, determining the number of copies of the storage path based on the business importance of the storage path and the redundancy policy score.

[0008] In some embodiments, the step of determining the storage strategy for storage path adaptation based on the redundancy policy score and the first redundancy policy score threshold includes: when the redundancy policy score is greater than the first redundancy policy score threshold, determining that the storage strategy for storage path adaptation is an erasure code storage strategy; when the redundancy policy score is not greater than the first redundancy policy score threshold, determining that the storage strategy for storage path adaptation is a multi-copy storage strategy.

[0009] In some embodiments, the architecture information includes topology information of each storage path of the target distributed storage cluster; the step of determining the data recovery method of the storage path based on the architecture information of the storage path includes: determining the data transmission index of the storage path based on the topology information, wherein the data transmission index is the ratio of the data transmitted across the rack in the storage path to repair the faulty node when there is a faulty node in the storage path to the total data stored in the storage path; comparing the data transmission index with the preset data transmission index; and determining the data recovery method of the storage path based on the comparison result.

[0010] In some embodiments, the step of determining the data recovery method of the storage path based on the comparison result includes: when the data transmission index is not greater than the preset data transmission index, determining that the data recovery method adopted by the storage path is: when there is a faulty node in the storage path, the non-faulty nodes in the storage path send the saved data to the rack where the faulty node is located; when the data transmission index is greater than the preset data transmission index, determining that the data recovery method adopted by the storage path is: when there is a faulty node in the storage path, within each rack of the storage path, a check block is generated based on the data saved by the non-faulty node, and then the check block is sent to the faulty node.

[0011] In some embodiments, the step of determining the number of copies of the storage path based on the business importance and redundancy policy score of the storage path includes: determining a second redundancy policy score threshold, wherein the second redundancy policy score threshold is less than the first redundancy policy score threshold; determining a business importance evaluation value of the storage path, and a preset business importance evaluation threshold; comparing the redundancy policy score and the second redundancy policy score threshold, and comparing the business importance evaluation value and the business importance evaluation threshold; and determining the number of copies of the storage path based on the comparison result.

[0012] In some embodiments, the business importance evaluation threshold includes a first business importance evaluation threshold and a second business importance evaluation threshold, wherein the first business importance evaluation threshold is greater than the second business importance evaluation threshold; the step of determining the number of copies of the storage path based on the comparison result includes: when the comparison result is that the redundancy policy score is less than the second redundancy policy score threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, determining the number of copies of the storage path to be three copies; when the comparison result is that the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy policy score is greater than the second redundancy policy score threshold and less than the first redundancy policy score threshold, determining the number of copies of the storage path to be zero; in other cases, determining the number of copies of the storage path to be two copies.

[0013] In some embodiments, the file storage information includes at least one of the following: library information, table information, domain information and tenant information in the target distributed storage cluster, the length of time that data under each access path of the target distributed storage cluster has not been accessed, the amount of stored data under each access path of the target distributed storage cluster, the number of files under each access path of the target distributed storage cluster, the average file size, the number of files and the file storage format.

[0014] According to another aspect of an embodiment of the present application, a distributed storage system data management device is also provided, including: a first processing module, used to traverse all storage paths in the target distributed storage cluster based on the business classification information of the target distributed storage cluster, and obtain file storage information of the storage path during the traversal process; a second processing module, used to calculate the redundancy policy score of the storage path based on the file storage information, and determine the storage policy adapted to the storage path based on the redundancy policy score and a first redundancy policy score threshold, wherein the redundancy policy score is used to reflect the degree of adaptability between the storage path and the preset storage policy, and the preset storage policy includes an erasure code storage policy and a multi-copy storage policy; a third processing module, used to determine the data recovery method of the storage path based on the architecture information of the storage path when it is determined that the storage policy adapted to the storage path is the erasure code storage policy; a fourth processing module, used to determine the number of copies of the storage path based on the business importance of the storage path and the redundancy policy score when it is determined that the storage policy adapted to the storage path is the multi-copy storage policy.

[0015] According to another aspect of an embodiment of the present application, a distributed storage system data management system is also provided, including a target distributed storage cluster and a storage management platform, wherein the target distributed storage cluster is used to store data; the storage management platform is used to traverse all storage paths in the target distributed storage cluster based on the business classification information of the target distributed storage cluster, and obtain file storage information of the storage path during the traversal process; calculate the redundancy policy score of the storage path based on the file storage information, and determine the storage policy adapted to the storage path based on the redundancy policy score and a first redundancy policy score threshold, wherein the redundancy policy score is used to reflect the degree of adaptability between the storage path and a preset storage policy, and the preset storage policy includes an erasure code storage policy and a multi-copy storage policy; when it is determined that the storage policy adapted to the storage path is the erasure code storage policy, determine the data recovery method of the storage path based on the architecture information of the storage path; when it is determined that the storage policy adapted to the storage path is the multi-copy storage policy, determine the number of copies of the storage path based on the business importance of the storage path and the redundancy policy score.

[0016] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored. When the program is running, the device where the non-volatile storage medium is located is controlled to execute a distributed storage system data management method.

[0017] According to another aspect of an embodiment of the present application, an electronic device is provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the distributed storage system data management method is executed when the program is run.

[0018] In an embodiment of the present application, all storage paths in the target distributed storage cluster are traversed based on the business classification information of the target distributed storage cluster, and the file storage information of the storage path is obtained during the traversal process; the redundancy strategy score of the storage path is calculated based on the file storage information, and the storage strategy adapted to the storage path is determined based on the redundancy strategy score and the first redundancy strategy score threshold, wherein the redundancy strategy score is used to reflect the adaptability between the storage path and the preset storage strategy, and the preset storage strategy includes an erasure code storage strategy and a multi-copy storage strategy; when it is determined that the storage strategy adapted to the storage path is the erasure code storage strategy, the data recovery strategy of the storage path is determined based on the architecture information of the storage path. duplication method; when it is determined that the storage strategy adapted by the storage path is a multi-copy storage strategy, the number of copies of the storage path is determined according to the business importance of the storage path and the redundancy strategy score, the file storage information of the storage path is obtained by traversing the storage path, and the storage strategy adapted by the storage path is determined according to the file storage information, so as to achieve the purpose of allowing the storage path to run with the most adapted storage strategy, thereby achieving the technical effect of improving the storage efficiency of the storage path and reducing the storage overhead, and further solving the technical problem of low storage efficiency and high storage overhead of the distributed storage system due to the inability to determine the most suitable storage strategy for each storage path in the distributed storage system in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] FIG1 is a schematic structural diagram of a distributed storage system data management system provided according to an embodiment of the present application;

[0021] FIG2 is a flow chart of a distributed storage system data management method provided according to an embodiment of the present application;

[0022] FIG3 is a schematic diagram of an analysis process for analyzing a distributed storage system according to an embodiment of the present application;

[0023] FIG4 is a schematic diagram of a data recovery process of an erasure code according to an embodiment of the present application;

[0024] FIG5 is a schematic diagram of a data recovery process of an MSR erasure code according to an embodiment of the present application;

[0025] FIG6 is a schematic diagram of a data recovery process of an erasure code of a dual-generated code according to an embodiment of the present application;

[0026] 7 is a schematic diagram of a process flow of a distributed storage system data storage strategy analysis process provided according to an embodiment of the present application;

[0027] FIG8 is a schematic diagram of an analysis process of an erasure code recovery strategy according to an embodiment of the present application;

[0028] FIG9 is a schematic diagram of a replica number analysis process according to an embodiment of the present application;

[0029] 10 is a schematic structural diagram of a distributed storage system data management device according to an embodiment of the present application;

[0030] FIG11 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0034] Erasure Code (EC): A data protection method that splits data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations. Erasure coding creates a mathematical function to describe a set of numbers so that their accuracy can be checked and restored if one of the numbers is lost.

[0035] DCR (Double Regenerating Codes) twin code: This application provides an improved data protection method based on EC erasure code. It is a storage strategy suitable for specific rack and cluster topologies. Compared with traditional EC erasure code, it can reduce cross-rack traffic transmission to improve data recovery efficiency.

[0036] While various storage strategies are currently available for distributed storage systems, such as erasure coding and multiple replicas, current technologies are unable to determine the most appropriate storage strategy for a given distributed storage system. This results in a high probability that the storage strategy employed during actual operation will not be optimal, leading to poor storage efficiency and high overhead for the distributed storage system. Furthermore, while traditional erasure coding strategies offer high redundancy and storage efficiency, they also require additional computing power for data recovery and incur additional network overhead. Traditional multiple replica strategies typically use the same default number of replicas for all data, resulting in additional storage overhead and, in production environments, greater financial losses.

[0037] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0038] According to an embodiment of the present application, a method embodiment of a distributed storage system data management method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device, for example, in a distributed storage system data management system as shown in FIG1 . As can be seen from FIG1 , the system includes a target distributed storage cluster 10 and a storage management platform 12, wherein:

[0040] The target distributed storage cluster 10 is used to store data;

[0041] The storage management platform 12 is used to traverse all storage paths in the target distributed storage cluster based on the business classification information of the target distributed storage cluster, and obtain file storage information of the storage path during the traversal process; calculate the redundancy strategy score of the storage path based on the file storage information, and determine the storage strategy adapted to the storage path based on the redundancy strategy score and the first redundancy strategy score threshold, wherein the redundancy strategy score is used to reflect the degree of compatibility between the storage path and the preset storage strategy, and the preset storage strategy includes an erasure code storage strategy and a multi-copy storage strategy; when it is determined that the storage strategy adapted to the storage path is the erasure code storage strategy, determine the data recovery method of the storage path based on the architecture information of the storage path; when it is determined that the storage strategy adapted to the storage path is the multi-copy storage strategy, determine the number of copies of the storage path based on the business importance of the storage path and the redundancy strategy score.

[0042] In some embodiments of the present application, users can utilize the query function of MySQL and the above-mentioned storage management platform 12, combined with the storage status of the business display cluster and visualization tools, to cope with various scenario visualization and alarm requirements such as customized cold data standards, storage growth within a time range, and directory location with a large number of small files.

[0043] It should also be noted that the storage management platform 12 provided in the embodiment of the present application has low requirements for the storage components and cluster load in the target distributed storage cluster 10, that is, it does not require that the storage path 10 must contain a specified type of storage component, nor will it occupy too much load on the target distributed storage cluster 10. For example, for a distributed file system that stores 100 million files and a primary and backup named node (namenode), assuming that each file requires 200 bytes of metadata information, the metadata information occupies about 20GB of storage. The disk size required for the storage image can then be estimated based on the actual situation of the cluster. If the change in the size of the Fsimage file (image file) is ignored, the size of the Fsimage file obtained from the named node every day is recorded as M_fsimage. Taking the cluster practice of a cluster with more than a thousand nodes as an example, the file size M_fsimage of the Fsimage generated every day is about 60GB. After parsing, the file size expands by about three times, so the size of the data stored in the Hive database is about 200GB. Therefore, for three common types of storage components, after adopting the recommended deployment method, the load situation after using the storage management platform 12 provided by the embodiment of the present application is as shown in the following table:

[0044] In the above operating environment, an embodiment of the present application provides a distributed storage system data management method, as shown in FIG2 , which includes the following steps:

[0045] Step S202: traverse all storage paths in the target distributed storage cluster according to the service classification information of the target distributed storage cluster, and obtain file storage information of the storage paths during the traversal process;

[0046] In the technical solution provided in step S202, the file storage information includes at least one of the following: library information, table information, domain information and tenant information in the target distributed storage cluster, the length of time that data under each access path of the target distributed storage cluster has not been accessed, the amount of stored data under each access path of the target distributed storage cluster, the number of files under each access path of the target distributed storage cluster, the average file size, the number of files and the file storage format.

[0047] As an optional implementation, when traversing the target distributed storage cluster, you can use Ansible automation scripts to automatically collect and parse the Fsimage files (image files) of the backup named nodes in each cluster and load the parsed results into the Hive database. After correctly configuring the Hive metadata database, you can further obtain the Hive metadata table during the analysis phase and perform correlation analysis with the Fsimage files.

[0048] When analyzing Hive metadata tables and Fsimage files in a Hive database, the analysis uses the parsed Fsimage files and Hive metadata tables stored in the Hive database as input to analyze cluster storage in areas such as small files, storage growth, cold data, temporary tables, and compression formats. Output analysis results include dimensions such as table name, access time, partition information, file count, and directory level. The analysis results can be saved to the target distributed database. Users can also perform secondary analysis based on business needs or continue with subsequent steps to further process the analysis results.

[0049] In some embodiments, the process of traversing the target distributed cluster and performing further analysis and processing based on the traversal results is shown in Figure 3. As can be seen from Figure 3, the method provided by the present application will first pull the image file from each node (NS, NS2, NS3, NS4), and obtain the Hive metadata for parsing. After parsing, the parsing results will be imported into the Hive database. After that, the target distributed cluster can be preliminarily analyzed according to the parsing results, including cluster storage, small file analysis, storage growth analysis, cold data analysis, Hive table analysis, and other user-defined sql analysis statements. Afterwards, the preliminary analysis results will be subjected to a secondary analysis, including landing MySQL analysis, quota analysis, and system manager alarm configuration. Finally, a visual component can be used to display it to the user.

[0050] Step S204: Calculate a redundancy strategy score for each storage path based on the file storage information, and determine a storage strategy that matches the storage path based on the redundancy strategy score and a first redundancy strategy score threshold. The redundancy strategy score reflects the compatibility between the storage path and a preset storage strategy, which includes an erasure code storage strategy and a multi-copy storage strategy.

[0051] In the technical solution provided in step S204, the step of determining the storage strategy for storage path adaptation based on the redundancy strategy score and the first redundancy strategy score threshold includes: when the redundancy strategy score is greater than the first redundancy strategy score threshold, determining that the storage strategy for storage path adaptation is the erasure code storage strategy; when the redundancy strategy score is not greater than the first redundancy strategy score threshold, determining that the storage strategy for storage path adaptation is the multi-copy storage strategy.

[0052] In some embodiments, the parameters and definitions that may be used in calculating the redundancy strategy score of a storage path and subsequently determining the data recovery method and the number of replicas are shown in the following table:

[0053] Among the above parameters, mt, T1, T2, and I can be set by the user after evaluating the storage path. M_t is a preset data transmission indicator derived from the distributed topology. It is used to determine whether the twin-code-based data recovery method used in the storage path can effectively reduce the total amount of data transmitted across racks.

[0054] A larger redundancy strategy score s indicates that the data was last accessed longer ago. This means that the data in the storage path is closer to cold data. Furthermore, if the number of small files and the average file size are both small, an erasure code storage strategy is more suitable for the storage path. Conversely, a smaller score indicates that a multiple-copy storage strategy is more suitable for the storage path or that redundant backup is not required.

[0055] The above-mentioned business importance evaluation value can be determined by the user through evaluation of the storage path, and the value range is 0<I<100.

[0056] As an optional implementation, the calculation formula of the redundancy policy score of the storage path is as follows:

[0057] Step S206: When it is determined that the storage strategy adapted to the storage path is the erasure coding storage strategy, a data recovery method for the storage path is determined based on the architecture information of the storage path.

[0058] In the technical solution provided in step S206, the architecture information includes the topology information of each storage path of the target distributed storage cluster; the step of determining the data recovery method of the storage path based on the architecture information of the storage path includes: determining the data transmission index of the storage path based on the topology information, wherein the data transmission index is the ratio of the data transmitted across the rack in the storage path to repair the faulty node when there is a faulty node in the storage path to the total data stored in the storage path; comparing the data transmission index with the preset data transmission index; and determining the data recovery method of the storage path based on the comparison result.

[0059] In some embodiments, the topology of the storage path is defined as G, and the data transmission index of the storage path is T(G), where T() represents a preset transmission index calculation function, the input is the topology of the target cluster, and the output is the data transmission index.

[0060] Since the data transmission index is primarily related to the number of racks in the topology and the number of parity blocks in each rack, G can be represented by a function that primarily includes the number of racks and the number of parity blocks. Generally speaking, the fewer the number of racks and the greater the number of parity blocks in each rack, the more suitable the dual-generating code approach is, and the larger T(G) will be.

[0061] In some embodiments of the present application, the step of determining the data recovery method of the storage path based on the comparison result includes: when the data transmission index is not greater than the preset data transmission index, determining that the data recovery method adopted by the storage path is: when there is a faulty node in the storage path, the non-faulty nodes in the storage path send the saved data to the rack where the faulty node is located; when the data transmission index is greater than the preset data transmission index, determining that the data recovery method adopted by the storage path is: when there is a faulty node in the storage path, within each rack of the storage path, a check block is generated based on the data saved by the non-faulty node, and then the check block is sent to the faulty node.

[0062] In some embodiments, when the data transmission indicator T(G) is not greater than the preset data transmission indicator M_t, the recovery method used to recover data from a failed node is as shown in Figure 4 or Figure 5. It can be seen that in Figure 4, the node directly sends the stored data to the failed node. In contrast, in Figure 5, the data stored in the node is first deduplicated before being sent to the failed node. It can be seen that the data recovery method in Figure 5, compared to that in Figure 4, transmits 3 / 4 of the data, assuming all other conditions are the same.

[0063] When the data transmission index T(G) is greater than the preset data transmission index M_t, the repair method for repairing data in the faulty node is shown in Figure 6. As can be seen from Figure 6, each rack first calculates a check block based on the data of healthy nodes in the rack, and then sends the check block to the faulty node, thereby reducing the amount of data transmitted across racks.

[0064] It can be seen that when the data recovery method of the dual generation code provided in this application is adopted, the bandwidth required for data recovery can be effectively reduced, thereby ensuring the efficiency of data recovery.

[0065] Step S208: When it is determined that the storage strategy adapted to the storage path is a multi-copy storage strategy, the number of copies of the storage path is determined according to the business importance of the storage path and the redundancy strategy score.

[0066] In the technical solution provided in step S208, the step of determining the number of copies of the storage path based on the business importance of the storage path and the redundancy policy score includes: determining a second redundancy policy score threshold, wherein the second redundancy policy score threshold is less than the first redundancy policy score threshold; determining a business importance evaluation value of the storage path, and a preset business importance evaluation threshold; comparing the redundancy policy score and the second redundancy policy score threshold, and comparing the business importance evaluation value and the business importance evaluation threshold; and determining the number of copies of the storage path based on the comparison result.

[0067] As an optional implementation method, the business importance evaluation threshold includes a first business importance evaluation threshold and a second business importance evaluation threshold, wherein the first business importance evaluation threshold is greater than the second business importance evaluation threshold; the step of determining the number of copies of the storage path based on the comparison result includes: when the comparison result is that the redundancy policy score is less than the second redundancy policy score threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, determining the number of copies of the storage path to be three copies; when the comparison result is that the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy policy score is greater than the second redundancy policy score threshold and less than the first redundancy policy score threshold, determining the number of copies of the storage path to be zero; in other cases, determining the number of copies of the storage path to be two copies.

[0068] In summary, the process of analyzing the storage path of the target distributed cluster and determining its appropriate storage strategy provided in this application is shown in FIG7 , and includes the following steps:

[0069] Step S702, traverse the storage path according to the business classification to obtain the file storage information of the storage path;

[0070] Step S704: determining a redundancy strategy score for the storage path based on the file storage information of the storage path;

[0071] Step S706, determining a first redundancy strategy scoring threshold, a second redundancy strategy scoring threshold, and a service importance evaluation value of the storage path;

[0072] Step S708: Compare the redundancy policy score with the first redundancy policy score threshold. If the comparison result shows that the redundancy policy score is greater than the first redundancy policy score threshold, confirm that the storage path is applicable to the erasure code storage strategy; otherwise, it indicates that the storage path is applicable to the multi-copy storage strategy.

[0073] For the case where erasure codes are applicable, as shown in FIG8 , it is also possible to further determine whether a data recovery strategy for twin codes is applicable based on the architecture information of the storage path. Specifically, the steps include:

[0074] Step S802, obtaining the architecture information of the storage path;

[0075] Step S804, calculating the data transmission index of the storage path according to the architecture information;

[0076] Step S806: Compare the data transmission index with the preset data transmission index. If the comparison result shows that the data transmission index is greater than the preset data transmission index, a low-flow cross-rack data recovery solution using dual generation codes is adopted. Otherwise, a data recovery solution as shown in FIG. 4 or FIG. 5 is adopted.

[0077] For a multi-copy strategy, as shown in Figure 9, the number of copies applicable to the storage path can be further determined, including the following steps:

[0078] Step S902: If the redundancy strategy score is less than the second redundancy strategy score threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, confirm that the storage path is suitable for three copies; otherwise, jump to step S904;

[0079] Step S904: If the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy strategy score is greater than the second redundancy strategy score threshold but less than the first redundancy strategy score threshold, it is determined that the storage path does not require a copy, or that the data in the storage path can be deleted. Otherwise, the process jumps to step S906.

[0080] Step S906: Determine whether the storage path is suitable for the storage strategy of two copies.

[0081] By adopting the business classification information of the target distributed storage cluster, all storage paths in the target distributed storage cluster are traversed, and the file storage information of the storage path is obtained during the traversal process; the redundancy strategy score of the storage path is calculated according to the file storage information, and the storage strategy adapted to the storage path is determined according to the redundancy strategy score and the first redundancy strategy score threshold, wherein the redundancy strategy score is used to reflect the adaptability between the storage path and the preset storage strategy, and the preset storage strategy includes an erasure code storage strategy and a multi-copy storage strategy; when it is determined that the storage strategy adapted to the storage path is the erasure code storage strategy, the data recovery of the storage path is determined according to the architecture information of the storage path. Method; when it is determined that the storage strategy adapted by the storage path is a multi-copy storage strategy, the number of copies of the storage path is determined according to the business importance of the storage path and the redundancy strategy score, the file storage information of the storage path is obtained by traversing the storage path, and the storage strategy adapted by the storage path is determined according to the file storage information, thereby achieving the purpose of allowing the storage path to run with the most adapted storage strategy, thereby achieving the technical effect of improving the storage efficiency of the storage path and reducing the storage overhead, and further solving the technical problem of low storage efficiency and high storage overhead of the distributed storage system due to the inability to determine the most suitable storage strategy for each storage path in the distributed storage system in the related technology.

[0082] Furthermore, this application analyzes each storage path in the target distributed storage cluster to generate a storage profile of the target distributed system. This profile analysis provides key indicators that influence data storage strategies, such as data storage location, number of small files, data inaccessibility time, and cluster topology. Combining the analysis of cluster storage information with a series of thresholds established by business personnel during lifecycle management, a specific and advanced redundancy strategy and memory layout strategy can be automatically generated for each storage path.

[0083] An embodiment of the present application provides a distributed storage system data management device, and FIG10 is a schematic diagram of the structure of the device. As shown in FIG10, the device includes: a first processing module 100, configured to traverse all storage paths in a target distributed storage cluster based on business classification information of the target distributed storage cluster, and obtain file storage information of the storage paths during the traversal process; a second processing module 102, configured to calculate a redundancy strategy score of the storage path based on the file storage information, and determine a storage strategy adapted to the storage path based on the redundancy strategy score and a first redundancy strategy score threshold, wherein the redundancy strategy score is used to reflect the degree of compatibility between the storage path and a preset storage strategy, wherein the preset storage strategy includes an erasure code storage strategy and a multi-copy storage strategy; a third processing module 104, configured to, if it is determined that the storage strategy adapted to the storage path is the erasure code storage strategy, determine a data recovery method for the storage path based on the architecture information of the storage path; and a fourth processing module 106, configured to, if it is determined that the storage strategy adapted to the storage path is the multi-copy storage strategy, determine the number of copies of the storage path based on the business importance of the storage path and the redundancy strategy score.

[0084] In some embodiments of the present application, the file storage information includes at least one of the following: library information, table information, domain information and tenant information in the target distributed storage cluster, the length of time that data has not been accessed under each access path of the target distributed storage cluster, the amount of stored data under each access path of the target distributed storage cluster, the number of files under each access path of the target distributed storage cluster, the average file size, the number of files and the file storage format.

[0085] In some embodiments of the present application, the step in which the second processing module 102 determines the storage strategy for storage path adaptation based on the redundancy strategy score and the first redundancy strategy score threshold includes: when the redundancy strategy score is greater than the first redundancy strategy score threshold, determining that the storage strategy for storage path adaptation is an erasure code storage strategy; when the redundancy strategy score is not greater than the first redundancy strategy score threshold, determining that the storage strategy for storage path adaptation is a multi-copy storage strategy.

[0086] In some embodiments of the present application, the architectural information includes topological structure information of the storage path; the step in which the third processing module 104 determines the data recovery method of the storage path based on the architectural information of the storage path includes: determining the data transmission index of the storage path based on the topological structure information of the storage path, wherein the data transmission index is the ratio of the data transmitted across the rack in the storage path to repair the faulty node when there is a faulty node in the storage path to the total data stored in the storage path; comparing the data transmission index with the preset data transmission index; and determining the data recovery method of the storage path based on the comparison result.

[0087] In some embodiments of the present application, the step of the third processing module 104 determining the data recovery method of the storage path based on the comparison result includes: when the data transmission index is not greater than the preset data transmission index, determining that the data recovery method adopted by the storage path is: when there is a faulty node in the storage path, the non-faulty nodes in the storage path directly send the saved data to the rack where the faulty node is located; when the data transmission index is greater than the preset data transmission index, determining that the data recovery method adopted by the storage path is: when there is a faulty node in the storage path, within each rack of the storage path, a check block is generated based on the data saved by the non-faulty node, and then the check block is sent to the faulty node.

[0088] In some embodiments of the present application, the step of the fourth processing module 106 determining the number of copies of the storage path based on the business importance and the redundancy policy score of the storage path includes: determining a second redundancy policy score threshold, wherein the second redundancy policy score threshold is less than the first redundancy policy score threshold; determining a business importance evaluation value of the storage path, and a preset business importance evaluation threshold; comparing the redundancy policy score and the second redundancy policy score threshold, and comparing the business importance evaluation value and the business importance evaluation threshold; and determining the number of copies of the storage path based on the comparison result.

[0089] In some embodiments of the present application, the business importance evaluation threshold includes a first business importance evaluation threshold and a second business importance evaluation threshold, wherein the first business importance evaluation threshold is greater than the second business importance evaluation threshold; the step in which the fourth processing module 106 determines the number of copies of the storage path based on the comparison result includes: when the comparison result is that the redundancy policy score is less than the second redundancy policy score threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, determining that the number of copies of the storage path is three copies; when the comparison result is that the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy policy score is greater than the second redundancy policy score threshold and less than the first redundancy policy score threshold, determining that the number of copies of the storage path is zero; in other cases, determining that the number of copies of the storage path is two copies.

[0090] It should be noted that the various modules in the above-mentioned distributed storage system data management device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0091] Figure 11 shows a hardware block diagram of a computer terminal (or mobile device) for implementing a distributed storage system data management method. As shown in Figure 11, the computer terminal 110 (or mobile device 110) may include one or more processors 1102 (illustrated in the figure as 1102a, 1102b, ..., 1102n) (the processor 1102 may include, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA), a memory 1104 for storing data, and a transmission module 1106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that the structure shown in Figure 11 is merely illustrative and does not limit the structure of the electronic device described above. For example, the computer terminal 110 may include more or fewer components than shown in Figure 11, or have a different configuration than that shown in Figure 11.

[0092] It should be noted that the one or more processors 1102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 110 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0093] The memory 1104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the distributed storage system data management method in the embodiment of the present application. The processor 1102 executes various functional applications and data processing by running the software programs and modules stored in the memory 1104, that is, implementing the above-mentioned distributed storage system data management method. The memory 1104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1104 may further include a memory remotely located relative to the processor 1102, and these remote memories may be connected to the computer terminal 110 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0094] Transmission device 1106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of computer terminal 110. In one embodiment, transmission device 1106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 1106 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0095] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 110 (or mobile device).

[0096] According to an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following distributed storage system data management method: based on the business classification information of the target distributed storage cluster, all storage paths in the target distributed storage cluster are traversed, and file storage information of the storage path is obtained during the traversal process; the redundancy policy score of the storage path is calculated based on the file storage information, and the storage strategy adapted to the storage path is determined based on the redundancy policy score and a first redundancy policy score threshold, wherein the redundancy policy score is used to reflect the adaptability between the storage path and the preset storage strategy, and the preset storage strategy includes an erasure code storage strategy and a multi-copy storage strategy; when it is determined that the storage strategy adapted to the storage path is the erasure code storage strategy, the data recovery method of the storage path is determined based on the architecture information of the storage path; when it is determined that the storage strategy adapted to the storage path is the multi-copy storage strategy, the number of copies of the storage path is determined based on the business importance of the storage path and the redundancy policy score.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0098] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0099] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0100] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0101] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for data management in a distributed storage system, comprising: Traversing all storage paths in the target distributed storage cluster according to the service classification information of the target distributed storage cluster, and obtaining the file storage information of the storage paths during the traversal; Calculating the redundancy policy score for each storage path according to the file storage information, and determining the storage policy adapted to the storage path according to the redundancy policy score and the first redundancy policy score threshold, wherein the redundancy policy score is used to reflect the adaptability between the storage path and the preset storage policy, and the preset storage policy includes an erasure code storage policy and a multi-copy storage policy; In the case where the storage policy adapted to the storage path is determined to be the erasure code storage policy, determining the data recovery method of the storage path according to the architecture information of the storage path; In the case where the storage policy adapted to the storage path is determined to be the multi-copy storage policy, determining the number of copies of the storage path according to the business importance degree of the storage path and the redundancy policy score.

2. The method for data management of the distributed storage system according to claim 1, wherein, The step of determining the storage policy adapted to the storage path according to the redundancy policy score and the first redundancy policy score threshold includes: In the case where the redundancy policy score is greater than the first redundancy policy score threshold, determining the storage policy adapted to the storage path to be the erasure code storage policy; In the case where the redundancy policy score is not greater than the first redundancy policy score threshold, determining the storage policy adapted to the storage path to be the multi-copy storage policy.

3. The data management method of the distributed storage system according to claim 1, wherein, The architecture information includes the topology structure information of each storage path of the target distributed storage cluster; The step of determining the data recovery method of the storage path according to the architecture information of the storage path includes: Determining the data transmission index of the storage path according to the topology structure information, wherein the data transmission index is the ratio of the data transmitted across racks to repair the faulty node in the storage path to all the data stored in the storage path when there is a faulty node in the storage path; Comparing the data transmission index with a preset data transmission index; Determining the data recovery method of the storage path according to the comparison result.

4. The method for managing data in a distributed storage system according to claim 3, wherein, The step of determining the data recovery method of the storage path according to the comparison result includes: In the case where the data transmission index is not greater than the preset data transmission index, determining the data recovery method adopted by the storage path as: when there is a faulty node in the storage path, the non-faulty nodes in the storage path send the saved data to the rack where the faulty node is located; In the case where the data transmission index is greater than the preset data transmission index, determining the data recovery method adopted by the storage path as: when there is a faulty node in the storage path, inside each rack of the storage path, generating a check block according to the data saved by the non-faulty nodes, and then sending the check block to the faulty node.

5. The data management method of the distributed storage system according to claim 1, wherein, The step of determining the number of copies of the storage path according to the business importance degree of the storage path and the redundancy policy score includes: Determine a second redundancy policy scoring threshold, where the second redundancy policy scoring threshold is less than the first redundancy policy scoring threshold; Determine the business importance evaluation value of the storage path and a preset business importance evaluation threshold; Compare the redundancy policy score with the second redundancy policy scoring threshold, and compare the business importance evaluation value with the business importance evaluation threshold; Determine the number of replicas of the storage path according to the comparison results.

6. The method for managing data in a distributed storage system according to claim 5, wherein, The business importance evaluation threshold includes a first business importance evaluation threshold and a second business importance evaluation threshold, where the first business importance evaluation threshold is greater than the second business importance evaluation threshold; the step of determining the number of replicas of the storage path according to the comparison results includes: When the comparison result is that the redundancy policy score is less than the second redundancy policy scoring threshold or the business importance evaluation value is greater than the first business importance evaluation threshold, determine that the number of replicas of the storage path is three replicas; When the comparison result is that the business importance evaluation value is less than the second business importance evaluation threshold, and the redundancy policy score is greater than the second redundancy policy scoring threshold and less than the first redundancy policy scoring threshold, determine that the number of replicas of the storage path is zero; In other cases, determine that the number of replicas of the storage path is two replicas.

7. The data management method of the distributed storage system according to claim 1, wherein, The file storage information includes at least one of the following: library information, table information, domain information, and tenant information in the target distributed storage cluster, the unaccessed duration of data under each access path of the target distributed storage cluster, the stored data volume under each access path of the target distributed storage cluster, the number of files under each access path of the target distributed storage cluster, the average file size, the number of files, and the file storage format.

8. The data management method of the distributed storage system according to claim 1, wherein The traversing the target distributed storage cluster includes: Using an automated script to collect and parse the mirror file of the standby namenode in the target distributed storage cluster; Loading the parsing result into the database; Obtaining the metadata of the database; Performing an association analysis on the obtained metadata table and the mirror file.

9. A data management device for a distributed storage system, including: A first processing module, configured to traverse all storage paths in the target distributed storage cluster according to the business classification information of the target distributed storage cluster, and obtain the file storage information of the storage path during the traversing process; A second processing module, configured to calculate the redundancy policy score of the storage path according to the file storage information, and determine the storage policy adapted to the storage path according to the redundancy policy score and the first redundancy policy scoring threshold, where the redundancy policy score is used to reflect the adaptability between the storage path and the preset storage policy, and the preset storage policy includes an erasure code storage policy and a multi-replica storage policy; A third processing module, configured to determine the data recovery method of the storage path according to the architecture information of the storage path when the determined storage policy adapted to the storage path is the erasure code storage policy; A fourth processing module, configured to, when it is determined that the storage policy adapted to the storage path is the multi-copy storage policy, determine the number of copies of the storage path according to the business importance degree of the storage path and the redundancy policy score.

10. A distributed storage system data management system, including a target distributed storage cluster and a storage management platform, where the target distributed storage cluster is configured to store data; the storage management platform is configured to traverse all storage paths in the target distributed storage cluster according to the service classification information of the target distributed storage cluster, and obtain the file storage information of the storage paths during the traversal process; calculate the redundancy policy score of the storage paths according to the file storage information, and determine the storage policy adapted to the storage paths according to the redundancy policy score and a first redundancy policy score threshold, where the redundancy policy score is used to reflect the adaptability between the storage paths and a preset storage policy, and the preset storage policy includes an erasure code storage policy and a multi-copy storage policy; when it is determined that the storage policy adapted to the storage paths is the erasure code storage policy, determine the data recovery method of the storage paths according to the architecture information of the storage paths; when it is determined that the storage policy adapted to the storage paths is the multi-copy storage policy, determine the number of copies of the storage paths according to the business importance degree of the storage paths and the redundancy policy score.

11. A non-volatile storage medium stores a program, wherein, When the program runs, control the device where the non-volatile storage medium is located to execute the distributed storage system data management method according to any one of claims 1 to 8.

12. An electronic device, comprising: A memory and a processor, where the processor is configured to run a program stored in the memory, and when the program runs, execute the distributed storage system data management method according to any one of claims 1 to 8.

Images