Metadata addressing method and apparatus, and medium and product

Abstract

The present application relates to the technical field of distributed storage. Disclosed are a metadata addressing method and apparatus, and a medium and a product. A target number of metadata service paths are obtained by filtering metadata feature parameters within a preset time, such that local management is performed on metadata in a large-scale distributed storage file system, and path information of the metadata is recorded and counted to improve the locality management of the metadata; moreover, a first path result is obtained by means of granularity calculation, and granularity selection is performed on the metadata paths to improve the temporal locality of the metadata; in addition, a second path result is obtained by evaluating and calculating the first path result on the basis of a path optimization strategy, such that locality processing is performed on the second path result on the basis of a locality function. Thus, the locality data structure of metadata is changed, thereby ensuring the spatial locality of metadata storage paths in a distributed file system, optimizing the paths for metadata operations in the distributed storage file system and improving the I / O performance of a storage system.

Description

Metadata addressing method, device, medium and product

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 202410834385.5, filed on June 26, 2024, and entitled "Metadata addressing method, device, medium and product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of distributed storage, in particular to a metadata addressing method, device, non-volatile readable storage medium and product. BACKGROUND

[0004] Metadata locality plays an important role in distributed storage systems, which can effectively organize and access metadata in the storage system, and is crucial for improving the performance scalability of the system.

[0005] Traditional metadata is managed in the following three ways, the first is based on data tables, as metadata and data tables continue to increase, there are problems of dynamic expansion and moving data, which cannot guarantee the locality performance of metadata; the second is based on hash partitioning to achieve business load balancing, which causes frequent distributed locks and transactions, thereby introducing lock overhead costs and increasing metadata request processing time; the third uses static and dynamic sub-tree partitioning, which cannot accurately migrate and has diverse and frequent business loads, resulting in increased metadata migration overhead. In summary, in the metadata management process, the overhead cost increases and the time-consuming time of frequent requests is longer, which further reduces the locality performance of metadata. SUMMARY

[0006] The purpose of the present application is to provide a metadata addressing method, device, non-volatile readable storage medium and product to solve the problem of increased overhead cost and longer time-consuming time of frequent requests in the metadata management process, which further reduces the locality performance of metadata.

[0007] To solve the above technical problems, the present application provides a metadata addressing method applied to a distributed storage system, comprising:

[0008] Obtaining a target number of metadata service paths in the distributed storage system; wherein the target number of metadata service paths is obtained by filtering all metadata service paths in the distributed storage system using metadata characteristic parameters within a preset time;

[0009] Performing granularity calculation based on the target number of metadata service paths to obtain a first path result;

[0010] According to the path optimization strategy, the first path result is evaluated and calculated to obtain a second path result;

[0011] According to the locality function, the second path result is locally processed to determine the locality of the metadata corresponding to the source metadata service and the target metadata service.

[0012] Optionally, the metadata characteristic parameters at least include a timestamp parameter; the target number of metadata service paths is obtained by filtering all metadata service paths in the distributed storage system using the metadata characteristic parameters within a preset time, including:

[0013] All metadata service paths in the distributed storage system are pre-added with a timestamp parameter;

[0014] Within the preset time and in the case that the distributed storage system has request information, the metadata service paths corresponding to the request information are filtered according to the continuous timestamp parameters of the preset time to obtain a first number of metadata service paths;

[0015] The first number of metadata service paths is taken as the target number of metadata service paths.

[0016] Optionally, after the first number of metadata service paths is filtered, it further includes:

[0017] A preset number is obtained;

[0018] It is judged whether the first number is greater than the preset number;

[0019] If the first number is greater than the preset number, the metadata service paths corresponding to the first number are sorted from large to small according to the time degree of the timestamp parameter occupying the request information;

[0020] Taking the metadata service path corresponding to the maximum time degree of occupying the request information as a reference, the metadata service paths corresponding to the preset target number in front are intercepted as the target number of metadata service paths;

[0021] If the first number is less than or equal to the preset number, the first number of metadata service paths is taken as the target number of metadata service paths.

[0022] Optionally, the metadata characteristic parameters at least include request access permission information; the target number of metadata service paths is obtained by filtering all metadata service paths in the distributed storage system using the metadata characteristic parameters within a preset time, including:

[0023] Within the preset time and in the case that the distributed storage system has request information, the metadata service paths corresponding to the request access permission information of the request information are filtered to obtain a second number of metadata service paths;

[0024] The second number of metadata service paths is taken as a target number of metadata service paths.

[0025] Optionally, the metadata service path is obtained from any one metadata service of a source node of the distributed storage system to any one metadata service of a target node; wherein each node of the distributed storage system comprises at least one metadata service; if each node of the distributed storage system comprises multiple metadata services, the multiple metadata services comprise a primary metadata service and a backup metadata service.

[0026] Optionally, the first path result is obtained by performing granularity calculation based on the target number of metadata service paths, comprising:

[0027] Obtaining time parameters corresponding to request data transmitted from the source node to the target node and feedback information under each metadata service path in the target number;

[0028] Determining corresponding time delay information according to the time parameters;

[0029] Obtaining path lengths corresponding to each metadata service path in the target number;

[0030] Establishing a mapping relationship between the path lengths and the time delay information of each metadata service path;

[0031] According to the mapping relationship, the target number of metadata service paths is sorted to obtain the first path result.

[0032] Optionally, the second path result is obtained by performing evaluation calculation on the first path result according to a path optimization strategy, comprising:

[0033] Taking the first path result as an initial output vector;

[0034] Obtaining an iteration number and a preset layer number;

[0035] According to a recursive algorithm, the initial output vector is subjected to feature calculation to obtain an addressing matrix and an addressing feature matrix corresponding to the iteration number and the preset layer number;

[0036] The second path result is determined according to the addressing matrix and the addressing feature matrix.

[0037] Optionally, according to the recursive algorithm, the initial output vector is subjected to feature calculation to obtain an addressing matrix and an addressing feature matrix corresponding to the iteration number and the preset layer number, comprising:

[0038] Taking path addressing information corresponding to the initial output vector as a first layer addressing matrix;

[0039] According to the first layer addressing matrix, the iteration number and the initial output vector, a first layer addressing feature matrix is determined.

[0040] The addressing feature matrix of the first layer is taken as an initial output vector of the second layer;

[0041] The path addressing information corresponding to the initial output vector of the second layer is taken as an addressing matrix of the second layer, and the addressing feature matrix of the third layer is determined according to the addressing matrix of the second layer, the number of iterations and the initial output vector of the second layer; and the process is repeated until the current number of iterations reaches the preset number of iterations and the current layer number reaches the preset layer number, and the addressing matrix of the kth layer and the addressing feature matrix of the kth layer are output; wherein k is a positive integer greater than or equal to 1.

[0042] Optionally, the first path result is evaluated and calculated according to a path optimization strategy to obtain a second path result, including:

[0043] The historical path result corresponding to the first path result and the corresponding path access information are obtained;

[0044] The historical path result and the path access information are processed by calling a machine learning algorithm to obtain the second path result.

[0045] Optionally, the second path result is locally processed according to a locality function to determine the locality of the metadata corresponding to the source metadata service and the target metadata service, including:

[0046] The path result of the metadata corresponding to the source metadata service and the target metadata service is obtained in the second path result;

[0047] The request traversal number of the metadata corresponding to the source metadata service and the target metadata service is obtained;

[0048] The locality of the metadata is determined according to the path result, the request traversal number, the marked path information corresponding to the source metadata service and the target metadata service, and the target number.

[0049] Optionally, the determination process of the request traversal number includes:

[0050] The preset request information is obtained;

[0051] The preset request information is sent from the source node where the source metadata service is located to the remaining nodes except the source node, and the remaining nodes are taken as the target nodes where the target metadata services are located;

[0052] The target nodes where the target metadata services are located are traversed to obtain the preset request information and the corresponding request traversal number.

[0053] Optionally, the locality of the metadata is determined according to the path result, the request traversal number, the marked path information corresponding to the source metadata service and the target metadata service, and the target number, including:

[0054] acquiring a number of times of traversing each request corresponding to the source metadata service to the target metadata service;

[0055] subtracting the target number from the mark path information of the source metadata service to obtain first data;

[0056] subtracting 1 from the first data to obtain second data;

[0057] multiplying the target number and the mark path information of the target metadata service to obtain third data;

[0058] multiplying the number of times of traversing the request corresponding to the current target metadata service and the second data to obtain fourth data;

[0059] dividing the fourth data by the third data to obtain fifth data;

[0060] multiplying the fifth data and the second path result corresponding to the current target metadata service to obtain locality corresponding to the current target metadata service;

[0061] adding the localities corresponding to each target metadata service to obtain locality of the metadata.

[0062] To solve the above technical problems, the application further provides a metadata addressing device, comprising:

[0063] a memory configured to store a computer program;

[0064] a processor configured to implement the steps of the metadata addressing method as described above when executing the computer program.

[0065] To solve the above technical problems, the application further provides a computer non-volatile readable storage medium, the computer non-volatile readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the metadata addressing method as described above.

[0066] To solve the above technical problems, the application further provides a computer program product, comprising a computer program / instruction, which is executed by a processor to implement the steps of the metadata addressing method.

[0067] The application provides a metadata addressing method applied to a distributed storage system, comprising: obtaining a target number of metadata service paths in the distributed storage system; wherein the target number of metadata service paths is obtained by filtering all metadata service paths in the distributed storage system using metadata characteristic parameters within a preset time; performing granularity calculation based on the target number of metadata service paths to obtain a first path result; performing evaluation calculation on the first path result according to a path optimization strategy to obtain a second path result; and performing locality processing on the second path result according to a locality function to determine the locality of metadata corresponding to a source metadata service and a target metadata service.

[0068] The application has the beneficial effect that the target number of metadata service paths obtained by filtering using metadata characteristic parameters within a preset time enables local management of metadata of a large-scale distributed storage file system, records and counts path information of the metadata, and improves the locality management of the metadata. Meanwhile, the first path result is obtained by using a granularity calculation method, the metadata path is selected in granularity, and the time locality of the metadata is improved. Meanwhile, the second path result is obtained by performing evaluation calculation on the first path result based on a path optimization strategy, the locality processing is performed on the second path result based on a locality function, the metadata locality data structure is changed, the spatial locality of the metadata storage path of the distributed file system is ensured, and the optimal path of metadata operation in the distributed storage file system and the IO performance of the storage system are improved.

[0069] Secondly, the metadata service paths of the continuous timestamp parameter within the preset time are recorded, so as to improve the time locality of the metadata; after the first number of metadata service paths are filtered, the preset number is compared with the first number, so as to ensure that the metadata service path finally filtered is in the locality management among all the metadata service paths, and the metadata service path is further optimized; the metadata service paths corresponding to the request access permission information of the request information within the preset time are recorded, so as to improve the time locality of the metadata; the path process of access determination between the two provided metadata services is provided, the flexibility and diversity of metadata access addressing are improved, so as to ensure the synchronization and consistency check of data communication; the first path result is the metadata service path of the optimal shortest path, so as to improve the locality and timeliness of the metadata of the distributed storage system; the recursive formula is used to calculate the shortest path, and in the construction and optimization of the metadata service path, the recursive idea realizes the evaluation and selection of the optimal path of the actual problem node by node.

[0070] In addition, the application also provides a metadata addressing device, a nonvolatile readable storage medium and a product, which have the same beneficial effects as the above-mentioned metadata addressing method. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to make the application more clearly, the following will be a brief introduction to the drawings needed in the embodiments, obviously, the following description of the drawings is only some embodiments of the application, for those skilled in the art, without creative labor, can also be obtained from these drawings other drawings.

[0072] Fig. 1 is a flow chart of a metadata addressing method according to an embodiment of the application;

[0073] Fig. 2 is a structural diagram of a distributed storage system according to an embodiment of the application;

[0074] Fig. 3 is a structural diagram of a metadata addressing device according to an embodiment of the application. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0076] The core of the application is to provide a metadata addressing method, device, non-volatile readable storage medium and product, to solve the problem of increasing overhead cost in metadata management process and frequent request time-consuming time, which further leads to the problem of reducing the locality performance of metadata.

[0077] In order to make the person skilled in the art better understand the application scheme, the following will be further described in detail in combination with the drawings and specific embodiments.

[0078] Distributed storage system has the characteristics of scalability, high availability, easy management, high performance, data security and low cost. Distributed storage system can meet different data scale and functional requirements, especially in large-scale data storage and processing, distributed storage system has obvious advantages. However, there are some challenges and limitations in distributed storage system, such as metadata performance, data security and management problems.

[0079] Metadata is data that describes the attributes of data, including file system directory tree structure, identification number, location information and data size, permission and other attribute information. Generally speaking, in a distributed storage file system, metadata operations account for about 75% of data access operations, which means that metadata plays an important role in the file system. Distributed storage systems provide file system services through metadata servers (Metadata Server, MDS). MDS services are usually deployed on distributed storage nodes with multiple masters and multiple backups. The master MDS provides metadata services externally, and the standby MDS does not provide services externally and serves as a failover service when a failure occurs. Distributed metadata servers MDS have scalability, high availability and consistency. In intensive input output (Input Output, IO) storage access, the storage path of metadata and the client access addressing process constrain the overall performance of distributed storage. Therefore, an efficient metadata storage addressing management method is a crucial research object for the performance of distributed storage systems.

[0080] Metadata locality is the proximity of metadata and actual data in physical or logical storage systems. Locality principle has a great impact on the design and performance of hardware and software systems. Locality is usually divided into temporal locality and spatial locality. Temporal locality refers to the fact that a memory location that has been referenced once is likely to be referenced multiple times in the near future. Spatial locality refers to the fact that if a memory location is referenced once, the program is likely to reference a nearby memory location in the near future. In a distributed storage file system, metadata locality is represented as the access path of the distributed file system metadata, i.e. the addressing path of the metadata. Frequent access of metadata will reference a neighboring metadata in the near future to improve the performance of distributed storage IO.

[0081] In traditional distributed storage file systems, metadata is stored through multiple partitions of MDS to improve metadata performance, scalability and high availability. Metadata service MDS manages metadata operations, including metadata path lookup and operations, i.e. metadata lookup, getattr, creat, setattr, etc. The locality of metadata is a difficult problem in distributed storage research due to its high complexity and difficulty in accurately predicting metadata access path rules. In general, in the metadata management of a distributed file system, data tables, Hash partitioning, static or dynamic subtree partitioning are usually used. The metadata storage path is determined by the metadata partitioning and location logical structure of the distributed storage MDS service, which mainly includes:

[0082] 1. Metadata management method based on data table: it needs to manage metadata in the order of data table structure, and the distributed metadata service MDS maps the metadata process, that is, when the storage system is just running, the table structure has performance advantages, but in a large-scale file system, as the metadata increases, the data table will also increase, and there are disadvantages of dynamic expansion performance consumption (frequent call of mal loc will consume memory) and insertion of data to move data, which cannot guarantee the locality performance of metadata.

[0083] 2. Metadata management method based on Hash partitioning: the distribution of Hash partitioning data is almost random, so the distribution is relatively uniform, which can avoid hot spot problem to a certain extent and can realize load balancing of business, but the locality of metadata is lost, which causes frequent distributed locks and transactions, introduces large lock overhead to prevent metadata conflict, reduces the performance of distributed storage, and increases the time consumption of metadata request processing.

[0084] 3. Based on static or dynamic sub-tree partitioning: static sub-tree partitioning, metadata is divided into different distributed storage metadata MDS services (MDS1, MDS2, MDSn), because the regularity of data business IO cannot be judged, static sub-tree partitioning will exist MDS service imbalance phenomenon, for example, MDS1 has a large number of metadata operations, and MDS2 is in a dormant state, so the performance and scalability of distributed storage are restricted. Dynamic sub-tree partitioning, according to the load capacity of the file system, dynamically migrates and adjusts the corresponding sub-tree, the decision of sub-tree partitioning and migration is a synchronous process, and at present, it is impossible to achieve precise migration and optimal decision mechanism of sub-tree partitioning, when the business load is various and changes frequently, it will suffer from high overhead of frequent metadata migration, which affects the overall performance of distributed storage.

[0085] The metadata addressing method provided in the application can solve the above technical problems.

[0086] Fig. 1 is a flowchart of a metadata addressing method provided by an embodiment of the application, as shown in Fig. 1, the method is applied to a distributed storage system, and includes:

[0087] S11: obtaining a target number of metadata service paths in the distributed storage system;

[0088] Among them, the target number of metadata service paths is obtained by filtering all metadata service paths in the distributed storage system using metadata characteristic parameters within a preset time;

[0089] S12: performing granularity calculation based on the target number of metadata service paths to obtain a first path result;

[0090] S13: evaluating and calculating the first path result according to the path optimization strategy to obtain a second path result;

[0091] S14: performing local processing on the second path result according to the locality function to determine the locality of the metadata corresponding to the source metadata service and the target metadata service.

[0092] Optionally, the target number in step S11 is less than the number of all metadata service paths in the distributed storage system; that is, the embodiment based on the metadata of the large-scale distributed storage file system performs locality management, and needs to filter all metadata service paths based on the metadata characteristic parameters to obtain the target number of metadata service paths, so as to improve the temporal locality of the metadata.

[0093] The specified time period is limited by a preset time in the embodiment, and the specific time of the preset time is not limited in the embodiment and can be set according to actual conditions. The metadata characteristic parameter is mainly based on the information (attribute information of the file or object) of the metadata, such as the file name, file size, file permission, creation and modification time, etc. Here, the metadata characteristic parameter can be based on at least one characteristic parameter, that is, it can be one characteristic parameter or a combination of multiple characteristic parameters for filtering. The combination of multiple characteristic parameters for filtering can be based on multiple characteristic parameters combined into a hybrid characteristic parameter for filtering, or can be based on a priority order to filter all metadata service paths.

[0094] In some embodiments, the metadata characteristic parameter at least includes a timestamp parameter; the target number of metadata service paths is obtained by filtering all metadata service paths in the distributed storage system using the metadata characteristic parameter within the preset time, including:

[0095] The timestamp parameter is added to all metadata service paths in the distributed storage system in advance;

[0096] In the case that the distributed storage system appears a request information within the preset time, the metadata service path corresponding to the request information is filtered according to the continuous timestamp parameter of the preset time to obtain a first number of metadata service paths;

[0097] The first number of metadata service paths is taken as the target number of metadata service paths.

[0098] Optionally, when the metadata characteristic parameters at least include the timestamp parameter, the metadata of the large-scale distributed storage file system is managed locally, the path information of the metadata is recorded and counted, and the metadata with the timestamp parameter is recorded in a specified time period. The timestamp parameter needs to be added to all metadata service paths in advance.

[0099] In a preset time, and in the case that the distributed storage system appears request information, the metadata service path corresponding to the request information is screened based on the continuous timestamp parameter of the preset time to obtain a first number of metadata service paths. Specifically, in a certain preset time, it is checked whether the timestamp parameter of the metadata service path for the request information is continuous. If the timestamp parameter of the metadata service path for the request information is continuous, the metadata service path to which the continuous timestamp parameter belongs is screened out.

[0100] The embodiment records the metadata service path of the continuous timestamp parameter in a preset time, thereby improving the temporal locality of the metadata.

[0101] On the basis of the above-mentioned embodiments, in some embodiments, after the first number of metadata service paths is screened out, the following steps are further included:

[0102] A preset number is obtained.

[0103] It is judged whether the first number is greater than the preset number.

[0104] If the first number is greater than the preset number, the metadata service paths corresponding to the first number are sorted from large to small according to the time degree of the timestamp parameter occupying the request information.

[0105] The metadata service path corresponding to the maximum time degree of occupying the request information is taken as a reference, and the metadata service paths corresponding to the first preset target number are intercepted as the target number of metadata service paths.

[0106] If the first number is less than or equal to the preset number, the first number of metadata service paths is taken as the target number of metadata service paths.

[0107] Optionally, if the number of the first number of metadata service paths is still relatively large, which is not conducive to local management, it is necessary to judge whether the first number is greater than the preset number. If the first number is greater than the preset number, it means that the current first number of metadata service paths is relatively large, and therefore, screening is still needed.

[0108] The time degrees of the timestamp parameters occupying the request information are sorted, that is, the timestamp parameters in the preset time in the above embodiment are continuous, and the number of the continuous timestamp parameters is not limited, as long as the preset number of continuous timestamp parameters is met. In addition, the number of continuous timestamp parameters in each metadata service path is different, and in the case of meeting the preset number of continuous timestamp parameters, the time degrees of the timestamp parameters occupying the request information are sorted, that is, the number of continuous timestamp parameters is sorted from large to small in the case of the request information, and the metadata service path corresponding to the maximum time degree of occupying the request information is taken as a reference, and the metadata service paths corresponding to the preset target number are intercepted as the target number of metadata service paths. The preset target number, the preset number and the first number are all different.

[0109] If the first number is less than or equal to the preset number, the first number of metadata service paths can be taken as the target number of metadata service paths.

[0110] In the embodiment, after the first number of metadata service paths are screened, the preset number is compared with the first number to ensure that the finally screened metadata service paths are locally managed in all metadata service paths, and the metadata service paths are further optimized.

[0111] In some other embodiments, the metadata characteristic parameters at least include request access permission information, and the target number of metadata service paths are obtained by screening all metadata service paths in the distributed storage system by using the metadata characteristic parameters within the preset time, including:

[0112] In the case that the request information appears in the distributed storage system within the preset time, the metadata service paths corresponding to the request access permission information of the request information are screened to obtain the second number of metadata service paths.

[0113] The second number of metadata service paths are taken as the target number of metadata service paths.

[0114] Optionally, in the case that the request information appears in the distributed storage system within the preset time, the metadata service paths corresponding to the request access permission information of the request information are screened, that is, the metadata service paths with the request access permission information of the request information are screened out for the request information to obtain the second number of metadata service paths, so as to be taken as the target number of metadata service paths.

[0115] In addition, the second number can be compared with the preset number to determine whether the second number is too large.

[0116] The embodiment records the metadata service path corresponding to the request access permission information of the request information within a preset time, so as to improve the temporal locality of the metadata.

[0117] The metadata service path herein is for any one metadata service corresponding to each storage node. In some embodiments, the metadata service path is obtained from any one metadata service of a source node of a distributed storage system to any one metadata service of a target node; wherein at least one metadata service is included in each node of the distributed storage system; if multiple metadata services are included in each node of the distributed storage system, the multiple metadata services include a primary metadata service and a backup metadata service.

[0118] FIG. 2 is a structural schematic diagram of a distributed storage system provided by an embodiment of the application. As shown in FIG. 2, the distributed storage system includes multiple distributed storage nodes, each of which includes multiple MDS service processes, one of which is a primary MDS mainly responsible for providing services externally, and the remaining MDS service processes are backup MDSs used to provide metadata services in case of failure switching. The distributed storage nodes are physically connected through double-gigabit network links, and the MDSs are connected through double-gigabit links for data communication synchronization and consistency checking.

[0119] Based on the addressing method corresponding to each storage node, the steps are as follows:

[0120] S21: coarse-grained calculation is performed on a small amount of metadata service paths;

[0121] S22: the evaluation result of the metadata service path is obtained;

[0122] S23: the evaluation result is trained by the metadata path optimization calculation unit, and feedback is provided;

[0123] S24: path recommendation addressing is performed through the metadata addressing optimization evaluation algorithm;

[0124] S25: the metadata locality is determined through addressing evaluation optimization.

[0125] The metadata service path is a path formed from any one metadata service in a distributed storage node of a distributed storage system to any one metadata service of another distributed storage node, which can be a primary metadata service in a distributed storage node to a backup metadata service of another distributed storage node, a primary metadata service in a distributed storage node to a primary metadata service of another distributed storage node, or a backup metadata service in a distributed storage node to a backup metadata service of another distributed storage node, which is not limited herein and can be set according to actual conditions.

[0126] The path process of access determination between the two metadata services provided in the embodiment improves the flexibility and diversity of metadata access addressing, thereby ensuring synchronization and consistency verification of data communication.

[0127] The granularity calculation based on the target number of metadata service paths in step S12 obtains a first path result. The granularity calculation here designs information processing of different granularity levels. The data granularity is based on the subdivision degree of data and can be based on time series analysis to aggregate according to different granularities such as years, months, days, or hours. The calculation granularity is the size of the task assigned to a single processing unit in parallel computing and distributed computing. Coarse-grained calculation involves larger task units, and fine-grained calculation involves smaller task units.

[0128] The specific size of the granularity calculation in the embodiment is not limited, and the main purpose is to determine regional addressing through coarse-grained calculation to find the shortest and optimal path. The coarse granularity of the present application can be based on the quantization process between the timestamp and the path to calculate the shortest path.

[0129] In coarse-grained calculation, data mapping is the process of allocating data to different processing units in the system to optimize performance and resource utilization. In a coarse-grained reconfigurable structure, existing routing paths are reused to reduce routing overhead and improve efficiency.

[0130] In some embodiments, the granularity calculation based on the target number of metadata service paths obtains a first path result, including:

[0131] Obtain the time parameter corresponding to the request data transmitted from the source node to the target node and the feedback information under each metadata service path under the target number;

[0132] Determine the corresponding time delay information according to the time parameter;

[0133] Obtain the path length corresponding to each metadata service path under the target number;

[0134] Establish a mapping relationship between the path length and the time delay information of each metadata service path;

[0135] Sort the metadata service paths of the target number according to the mapping relationship to obtain a first path result.

[0136] Optionally, according to the request data of the distributed storage client user in FIG. 2, the time value required for IO path from one end (source node) to the other end (target node) of the distributed storage system to receive the confirmation information (feedback information) is calculated as the time delay, that is, the time delay information determined based on the time parameter corresponding to the feedback information. The calculation of the corresponding time parameter in this embodiment can determine the corresponding IO path delay according to the time value corresponding to the conventional feedback information, or can calculate the feedback data of each actual IO path. The conventional calculation method can measure the time delay through the precision time protocol (PTP), can measure the time delay through the software-defined network (SDN), or can measure the time delay of each path through the path management software in the multi-path design, and the like, which is not limited here.

[0137] The path length corresponding to each metadata service path under the target number is obtained. It should be noted that the path length in this embodiment can be the number of routers or switches required for the request data to be transmitted from one node to another node, the complexity of the path (such as the number of different types of network devices or services on the path), the number of path entries recorded in the routing table from one node to another node, the logical distance or the logical steps of the response, and the path overhead on the path, and the like, which is not limited here.

[0138] The mapping relationship between the path length and the time delay information is established. The mapping relationship here can be established in the form of a table or a matrix, which can be set according to the actual situation. The metadata service paths of the target number are sorted based on the mapping relationship to determine the first path result.

[0139] The first path result of this embodiment is the metadata service path with the optimal shortest path, so as to improve the metadata locality and timeliness of the distributed storage system.

[0140] In some embodiments, the mapping relationship between the path length and the time delay information of each metadata service path is established, comprising:

[0141] The path length is taken as the first priority information;

[0142] The first priority information is sorted from large to small according to each metadata service path;

[0143] According to the sorted each metadata service path, it is determined whether the corresponding time delay information exists in the large-to-small sorting manner;

[0144] If the corresponding time delay information exists in the large-to-small sorting manner, the mapping relationship between the path length and the time delay information is established.

[0145] Correspondingly, the target number of metadata service paths are sorted according to the mapping relationship to obtain a first path result, including:

[0146] The sorted metadata service paths are sequentially set to a decimal threshold less than 1 gradually increasing from small to large;

[0147] The data obtained by subtracting each decimal threshold less than 1 from 1 are respectively taken as the first path result corresponding to each sorted metadata service path.

[0148] Specifically, the path length is taken as the first priority, and each metadata service path is sorted in size based on the first priority information, while checking whether the corresponding latency information is also sorted in size, if the corresponding latency information is sorted in descending order, the criterion of the lower the latency, the shorter the path is basically maintained, and the mapping relationship between the path length and the latency information is established.

[0149] Correspondingly, the sorted metadata service paths are sequentially set to a decimal threshold less than 1 gradually increasing from small to large, for example, there are 5 metadata service paths sorted in descending order (A1, A2, A3, A4 and A5), corresponding to the decimal thresholds (0.1, 0.2, 0.3, 0.4 and 0.5) respectively, the data obtained by subtracting each decimal threshold less than 1 from 1 are (0.9, 0.8, 0.7, 0.6, 0.5) respectively, as the first path result, thereby it can be determined that the A1 path has the longest latency and the A5 has the shortest latency.

[0150] The first path result is represented by Ls, Ls = [1-n], in the value of Ls, n represents the lowest latency and the shortest path, and 1 represents the highest latency and the longest path. Here, n corresponds to identify each path, the data closest to 1 has the highest latency and the longest path, and the data farther away from 1 has the shorter latency and the shorter path.

[0151] The embodiment provides a specific process of mapping relationship between the suggested path length and the latency information and the corresponding first path result, and the path length and the latency information are quantified, which is convenient for finding the shortest path to improve the metadata locality and timeliness of the distributed storage file system.

[0152] The second path result is calculated by evaluating the first path result in step S13 through a path optimization strategy. The optimization strategy can be based on a recursive calculation method, and can also use a machine learning algorithm, and is not limited herein. The optimization strategy of the present embodiment is a fine-grained calculation method, which decomposes the problem into smaller problems to find the shortest path or the optimal path between two nodes. Alternatively, the recursive calculation method can be a depth-first search (DFS), a breadth-first search (BFS), or the like, and is not limited herein. Based on the path optimization strategy, the second path result can be obtained by refining the evaluation based on the first path result.

[0153] The locality function in step S14 performs locality processing on the basis of the second path result to obtain the distribution of the metadata service path and the corresponding locality. The main scheme is to quantitatively describe the locality of the distributed storage file system metadata. The locality function is used to measure the locality in the data access mode, and mainly describes the characteristics of the program accessing data and tending to repeatedly access the same or adjacent data.

[0154] It can be understood that the locality function can solve the time locality, space locality, logical locality, sequential locality, etc. Before using the locality function, the space locality is achieved by changing the data structure (obtaining the second path result), and the space locality and the time locality are combined herein to improve the IO processing capability of the system, and improve the locality of the metadata and the total throughput of the metadata operations per second.

[0155] The application provides a metadata addressing method applied to a distributed storage system, including: obtaining a target number of metadata service paths in the distributed storage system; wherein the target number of metadata service paths is obtained by filtering all metadata service paths in the distributed storage system using metadata characteristic parameters within a preset time; performing granularity calculation based on the target number of metadata service paths to obtain a first path result; performing evaluation calculation on the first path result according to a path optimization strategy to obtain a second path result; and performing locality processing on the second path result according to a locality function to determine the locality of metadata corresponding to the source metadata service and the target metadata service. The target number of metadata service paths obtained by filtering using the metadata characteristic parameters within the preset time enables local management of metadata of a large-scale distributed storage file system, records and counts path information of the metadata, and improves the locality management of the metadata. Meanwhile, the first path result is obtained by using the granularity calculation method to select the metadata path in granularity, thereby improving the temporal locality of the metadata. Meanwhile, the second path result is obtained by performing evaluation calculation on the first path result based on the path optimization strategy, and the locality processing is performed on the second path result based on the locality function, so that the metadata locality data structure is changed, the spatial locality of the metadata storage path of the distributed file system is ensured, and the optimal path of the metadata operation in the distributed storage file system and the IO performance of the storage system are improved.

[0156] In some embodiments, the second path result is obtained by performing evaluation calculation on the first path result according to a path optimization strategy, including:

[0157] The first path result is taken as an initial output vector;

[0158] The number of iterations and the preset number of layers are obtained;

[0159] The initial output vector is calculated according to a recursive algorithm to obtain an addressing matrix and an addressing feature matrix corresponding to the number of iterations and the preset number of layers;

[0160] The second path result is determined according to the addressing matrix and the addressing feature matrix.

[0161] Optionally, the first path result is taken as an initial output vector Z0; meanwhile, the number of iterations m and the preset number of layers k are obtained, the initial output vector is calculated according to a recursive algorithm to obtain an addressing matrix and an addressing feature matrix corresponding to the number of iterations and the preset number of layers; and the second path result is determined according to the addressing matrix and the addressing feature matrix. In the recursive algorithm, the output vector of the bottom layer is the input vector of the last layer, and the distributed metadata service path is first described. The input parameters include a metadata service path addressing training feature matrix set X; the output parameters include an addressing matrix W of the metadata service path i ; wherein 1≤i≤k, Z kThe kth layer calculation result is represented, and the final output is an addressing evaluation result (second path result).

[0162] In some embodiments, the initial output vector is subjected to feature calculation according to a recursive algorithm to obtain an addressing matrix and an addressing feature matrix corresponding to an iteration number and a preset layer number, including:

[0163] The path addressing information corresponding to the initial output vector is taken as the addressing matrix of the first layer;

[0164] The addressing feature matrix of the first layer is determined according to the addressing matrix of the first layer, the iteration number and the initial output vector;

[0165] The addressing feature matrix of the first layer is taken as the initial output vector of the second layer;

[0166] The path addressing information corresponding to the initial output vector of the second layer is taken as the addressing matrix of the second layer, and the addressing feature matrix of the third layer is determined according to the addressing matrix of the second layer, the iteration number and the initial output vector of the second layer; and the process is repeated until the current iteration number reaches the preset iteration number and the current layer number reaches the preset layer number, and the addressing matrix of the kth layer and the addressing feature matrix of the kth layer are output; wherein k is a positive integer greater than or equal to 1.

[0167] Optionally, the path addressing information corresponding to the initial output vector is taken as the addressing matrix W1 of the first layer; the addressing feature matrix of the first layer is determined according to the addressing matrix of the first layer, the iteration number and the initial output vector, and the addressing feature matrix of the first layer is taken as the initial output vector of the second layer; the path addressing information corresponding to the initial output vector of the second layer is taken as the addressing matrix of the second layer, and the addressing feature matrix of the third layer is determined according to the addressing matrix of the second layer, the iteration number and the initial output vector of the second layer; and the process is repeated until the current iteration number reaches the preset iteration number and the current layer number reaches the preset layer number, and the addressing matrix of the kth layer and the addressing feature matrix of the kth layer are output; and the formula is as follows:

[0168] Wherein Z0 is the initial output vector, Z k is the addressing matrix of the kth layer, W i is the addressing feature matrix of the ith layer; m is the iteration number.

[0169] According to the initialization Z1 as the input of the second layer addressing calculation, W2 and Z2 are obtained; the iteration number m and the preset layer number k are calculated to obtain Z k and W k .

[0170] At this time, the optimal mds addressing path Ls is obtained through the improved Logistic classification function, and the formula is as follows:

[0171] In this embodiment, the recursive formula is analogous to the calculation of the metadata path, which can regard Z k as the cumulative cost or cumulative weight of reaching a certain metadata item, and Z0 is the initial cost, W i represents the cost or weight of accessing each metadata item, and m is the fixed cost or weight of each item.

[0172] Z0 is the initial cost, which can represent the cost of accessing the starting point of the metadata path, such as the initialization cost when requesting metadata from the client. For each node or service i in the metadata path, W i may represent the cost of accessing the node, which may include network delay, processing time, storage access time, etc. m is the fixed cost or weight of each item, which may represent the general cost of each node or service on the metadata path, such as the overhead of security checks, log recording, data replication, etc. Z k represents the cumulative cost of reaching the kth node in the path, which can be used to evaluate the performance of the entire metadata service path. By calculating the cumulative cost of different metadata paths, this recursive formula can be used to compare the efficiency of different paths and select the path with the lowest cost.

[0173] The embodiment provided in this embodiment calculates the shortest path using a recursive formula, which realizes the evaluation and selection of the optimal path for each node in the construction and optimization of the metadata service path.

[0174] In some other embodiments, the first path result is evaluated and calculated according to the path optimization strategy to obtain a second path result, including:

[0175] Obtain the historical path result corresponding to the first path result and the corresponding path access information;

[0176] Call a machine learning algorithm to process the historical path result and the path access information to obtain the second path result.

[0177] It can be understood that the information of the access pattern of the metadata, the storage system structure, the access delay, the network topology and the historical access data, that is, the historical path result of the first path result and the corresponding path access information, is collected. Based on the two parameters, feature selection is performed, and the corresponding feature parameters can be file size, access frequency, file type, node load and network congestion, etc. In the definition process, the shortest path of accessing the metadata, the most reliable path and / or the path that optimizes a specific performance indicator can be predicted. According to the nature of the problem, a suitable machine learning model is selected. Possible models include decision trees, random forests, gradient boosting machines, neural networks, graph neural networks, etc. The model is trained using historical data so that it can predict the optimal path of metadata access according to the input features. The performance of the model is evaluated using a validation set or test set, using appropriate evaluation metrics such as accuracy, F1 score, mean absolute error, etc. It may be necessary to use ensemble learning methods to combine the predictions of multiple models to improve accuracy and robustness. If the metadata access pattern changes over time, an online learning algorithm is used to update the model to adapt to the new pattern. Different metadata path planning strategies are tested in a simulated environment using the model to evaluate their effectiveness.

[0178] The machine learning algorithm provided by the embodiment obtains the second path result. In the implementation process, the explainability, real-time performance, accuracy and dynamic changes of the system of the model need to be considered. In addition, it is also necessary to ensure that the prediction of the model does not introduce too much additional overhead, and can adapt to changes in system load and network conditions. The diversity and flexibility of the optimization strategy are improved, and the accuracy of the path is further improved.

[0179] In some embodiments, the locality of the metadata corresponding to the source metadata service and the target metadata service is determined according to the locality function on the second path result, including:

[0180] Obtaining the path result of the metadata corresponding to the source metadata service and the target metadata service in the second path result;

[0181] Obtaining the request traversal times of the metadata corresponding to the source metadata service and the target metadata service;

[0182] Determining the locality of the metadata according to the path result, the request traversal times, the labeled path information corresponding to the source metadata service and the target metadata service, and the target quantity.

[0183] Optionally, the path result of the metadata corresponding to the source metadata service and the target metadata service in the second path result and the corresponding request traversal times are obtained, and the locality of the metadata is determined based on the path result, the request traversal times, the labeled path information corresponding to the source metadata service and the target metadata service, and the target quantity.

[0184] In some embodiments, the request traversal number determination process comprises:

[0185] Obtaining preset request information;

[0186] Sending the preset request information from a source node where the source metadata service is located to the remaining nodes other than the source node, and taking the remaining nodes as target nodes where target metadata services are located;

[0187] Traversing the target nodes where the target metadata services are located to obtain the preset request information and corresponding request traversal numbers.

[0188] Optionally, the traversal numbers obtained by traversing each target node from a source node, for example, there are n nodes, each storage node includes 2 MDS services, a primary MDS service and a backup MDS service, wherein the primary MDS service provides service and the backup MDS service does not provide service, thereby realizing high availability of the MDS service, and it is stipulated that the n storage nodes include n metadata MDS services, and the distributed storage has at least 3 MDS services, which are represented by a set M i , wherein M i ∈{3,n}; defining the traversal number h of the MDS metadata path; the request metadata operation of the distributed client processes the path traversal search number h∈{1,n}, that is, the request information can be obtained by traversing once, or the request information can be found by traversing n times. For the process of finding the optimal path of the distributed metadata MDS, the locality of the distributed storage file system metadata can be quantitatively described.

[0189] In some embodiments, the locality of the metadata is determined according to the path result, the request traversal number, the corresponding marked path information of the source metadata service to the target metadata service, and the target number, comprising:

[0190] Obtaining each request traversal number corresponding to the source metadata service to the target metadata service;

[0191] Subtracting the target number from the marked path information of the source metadata service to obtain first data;

[0192] Subtracting 1 from the first data to obtain second data;

[0193] Multiplying the target number and the marked path information of the target metadata service to obtain third data;

[0194] Multiplying the request traversal number corresponding to the current target metadata service and the second data to obtain fourth data;

[0195] Dividing the fourth data by the third data to obtain fifth data;

[0196] The fifth data is multiplied with the second path result corresponding to the current target metadata service to obtain the locality corresponding to the current target metadata service;

[0197] The localities corresponding to the target metadata services are added to obtain the locality of the metadata.

[0198] Optionally, the locality function adopts D l (i) is expressed as follows:

[0199] wherein M max is the maximum MDS number n; at this time, n here represents the target number in the embodiment, which is quantitatively counted based on each path of the metadata service path, for example, 5 metadata service paths, the corresponding target number is 5, and the maximum MDS number here is the maximum number selected. r s is the marked path information of the source metadata service, r d is the marked path information of the target metadata service; Ls(i) represents the optimal path corresponding to the i-th request traversal time, and the distributed clusters MDS are interconnected through double-gigabit networks. D l (i) represents that the increase of the traversal time h is reduced and monotonically decreased, and the smaller the value, the better the locality of the distributed cluster metadata.

[0200] wherein the first data is r s -M max ; the second data is r s -M max -1; the third data is M max x r d ; the fourth data is i x (r s -M max -1); and the fifth data is The locality corresponding to the current target metadata service is

[0201] In a large-scale distributed storage system, when the MDS service number is 8 to 16, the storage IO throughput of the distributed storage system can be improved by more than 35% using the metadata management method based on the application, and with the increase of the MDS service scale number, the metadata locality of the client hot data request is obviously improved through the addressing optimization algorithm.

[0202] The embodiment provided in the application determines the locality of the distributed cluster metadata based on the locality function, improves the locality of the metadata and the total throughput of the metadata operation per second, continuously optimizes and iteratively improves the locality of the metadata, and improves the time effectiveness and expansibility of the IO performance of the distributed storage system.

[0203] The metadata path and latency are found through an addressing algorithm, and the optimal path method is quantified to improve the metadata distribution structure and the optimal path of metadata operation. An optimal evaluation algorithm of metadata addressing and a locality method of metadata operation are proposed to select the optimal metadata path. With the increase of the size of the distributed storage file system, compared with the existing metadata distribution technologies, such as data table, Hash partition, static or dynamic sub-tree partition, the metadata management method of the present application can improve the total IO throughput of the distributed storage file system and improve the locality of metadata.

[0204] The above detailed description of the metadata addressing method corresponds to various embodiments, and on this basis, the present application also discloses a metadata addressing device corresponding to the above method, which comprises:

[0205] The first acquisition module is configured to acquire a target number of metadata service paths in the distributed storage system; wherein the target number of metadata service paths is obtained by filtering all metadata service paths in the distributed storage system using metadata characteristic parameters within a preset time;

[0206] The first calculation module is configured to perform granularity calculation based on the target number of metadata service paths to obtain a first path result;

[0207] The second calculation module is configured to perform evaluation calculation on the first path result according to a path optimization strategy to obtain a second path result;

[0208] The first processing module is configured to perform locality processing on the second path result according to a locality function to determine the locality of the metadata corresponding to the source metadata service to the target metadata service.

[0209] In some embodiments, after the first number of metadata service paths is filtered, it further comprises:

[0210] The first acquisition submodule is configured to acquire a preset number;

[0211] The first judgment submodule is configured to judge whether the first number is greater than the preset number; if the first number is greater than the preset number, the first sorting submodule is triggered, and if the first number is less than or equal to the preset number, the first serving submodule is triggered;

[0212] The first sorting submodule is configured to sort the metadata service paths corresponding to the first number from large to small according to the time degree of the timestamp parameter occupying the request information;

[0213] The second serving submodule is configured to take the metadata service path corresponding to the maximum time degree of occupying the request information as a reference, and intercept the metadata service paths corresponding to the preset target number as the target number of metadata service paths.

[0214] The first sub-module is configured to take the first number of metadata service paths as the target number of metadata service paths.

[0215] Since the embodiments of the device part correspond to the embodiments of the method part described above, the embodiments of the device part are described with reference to the embodiments of the method part described above, and will not be described here again.

[0216] For the metadata addressing device provided in the present application, refer to the method embodiments described above. The present application will not be described here again, and has the same beneficial effects as the metadata addressing method described above.

[0217] FIG. 3 is a structural diagram of a metadata addressing device provided in an embodiment of the present application. As shown in FIG. 3, the device includes:

[0218] The memory 21 is configured to store a computer program.

[0219] The processor 22 is configured to implement the steps of the metadata addressing method when executing the computer program.

[0220] The metadata addressing device provided in the present embodiment can include but is not limited to a smart phone, a tablet computer, a notebook computer, or a desktop computer, etc.

[0221] The processor 22 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 22 can be implemented in at least one of a hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 22 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 22 can be integrated with a graphics processor (GPU) that is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 22 can also include an artificial intelligence (AI) processor for processing machine learning-related computing operations.

[0222] The memory 21 can include one or more computer non-volatile readable storage media that can be non-transitory. The memory 21 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In the embodiment, the memory 21 is at least configured to store the following computer program 211, wherein the computer program is loaded and executed by the processor 22 and can implement the related steps of the metadata addressing method disclosed in any of the foregoing embodiments. In addition, the resources stored by the memory 21 can also include an operating system 212 and data 213, etc., and the storage mode can be temporary storage or permanent storage. The operating system 212 can include Windows, Unix, Linux, etc. The data 213 can include but is not limited to the data involved in the metadata addressing method, etc.

[0223] In some embodiments, the metadata addressing apparatus can further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0224] As understood by those skilled in the art, the structure shown in FIG. 3 does not constitute a limitation on the metadata addressing apparatus, and can include more or fewer components than shown.

[0225] The processor 22 implements the metadata addressing method provided in any of the foregoing embodiments by invoking instructions stored in the memory 21.

[0226] For the metadata addressing apparatus provided in the present application, please refer to the method embodiments described above, which has the same beneficial effects as the metadata addressing method described above.

[0227] Further, the present application also provides a computer non-volatile readable storage medium, and the computer non-volatile readable storage medium stores a computer program, and the computer program is executed by the processor 22 to implement the steps of the metadata addressing method as described above.

[0228] It can be understood that if the method in the above embodiment is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer nonvolatile readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and performs all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0229] For the introduction of a computer nonvolatile readable storage medium provided by the present application, please refer to the above method embodiment, and the present application will not be repeated here, which has the same beneficial effects of the above metadata addressing method.

[0230] Further, the present application also provides a computer program product, comprising computer programs / instructions, which are executed by a processor to realize the steps of the metadata addressing method.

[0231] For the introduction of a computer program product provided by the present application, please refer to the above method embodiment, and the present application will not be repeated here, which has the same beneficial effects of the above metadata addressing method.

[0232] The above provides a detailed introduction of the metadata addressing method, device, nonvolatile readable storage medium and product provided by the present application. The embodiments in the specification are described in a progressive manner, and each embodiment emphasizes the difference from other embodiments. The same or similar parts of each embodiment can be referred to. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the protection scope of the present application.

[0233] It also needs to be explained that in the present specification, the relational terms such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A method for addressing metadata, characterized in that, Applications in distributed storage systems include: Obtain a target number of metadata service paths within the distributed storage system; wherein, the target number of metadata service paths is obtained by filtering all metadata service paths within the distributed storage system using metadata feature parameters within a preset time period; The first path result is obtained by granular calculation based on the target number of metadata service paths; The second path result is obtained by evaluating and calculating the first path result based on the path optimization strategy; The second path result is processed using a locality function to determine the locality of the corresponding metadata from the source metadata service to the target metadata service.

2. The metadata addressing method according to claim 1, characterized in that, The metadata feature parameters include at least a timestamp parameter; the target number of metadata service paths is obtained by filtering all metadata service paths within the distributed storage system using the metadata feature parameters within a preset time period, including: A timestamp parameter is pre-added to all metadata service paths within the distributed storage system; If a request message appears in the distributed storage system within a preset time period, the metadata service paths corresponding to the request message are filtered according to the continuous timestamp parameter of the preset time to obtain a first number of metadata service paths. The first number of metadata service paths shall be used as the target number of metadata service paths.

3. The metadata addressing method according to claim 2, characterized in that, After obtaining the first number of metadata service paths through filtering, the following are also included: Get the preset quantity; Determine if the first quantity is greater than the preset quantity; If the first quantity is greater than the preset quantity, then the metadata service paths corresponding to the first quantity are sorted from largest to smallest according to the time the timestamp parameter occupies the request information. Based on the metadata service path corresponding to the time level of the maximum occupied request information, the metadata service path corresponding to the preset target quantity is extracted as the metadata service path of the target quantity. If the first quantity is less than or equal to the preset quantity, then the metadata service path of the first quantity is used as the metadata service path of the target quantity.

4. The metadata addressing method according to claim 1, characterized in that, The metadata feature parameters include at least request access permission information; the target number of metadata service paths is obtained by filtering all metadata service paths within the distributed storage system using the metadata feature parameters within a preset time period, including: If a request message is generated in the distributed storage system within a preset time, the metadata service path corresponding to the request access permission information of the request message is filtered to obtain a second number of metadata service paths. The second number of metadata service paths shall be used as the target number of metadata service paths.

5. The metadata addressing method according to any one of claims 1 to 4, characterized in that, The metadata service path is obtained from any metadata service of the source node corresponding to the distributed storage system to any metadata service of the target node; wherein, each node of the distributed storage system includes at least one metadata service; if each node of the distributed storage system has multiple metadata services, it includes a primary metadata service and a backup metadata service.

6. The metadata addressing method according to claim 5, characterized in that, The first path result is obtained by granular calculation based on the target number of metadata service paths, including: The time parameters corresponding to the transmission of request data from the source node to the target node and the feedback information under each metadata service path under the target quantity are obtained. Determine the corresponding delay information based on the time parameters; Obtain the path length corresponding to each metadata service path under the target quantity; Establish a mapping relationship between path length and latency information for each metadata service path; The metadata service paths for the target number are sorted according to the mapping relationship to obtain the first path result.

7. The metadata addressing method according to claim 6, characterized in that, The second path result is obtained by evaluating and calculating the first path result according to the path optimization strategy, including: Use the result of the first path as the initial output vector; Get the number of iterations and the preset number of layers; The initial output vector is subjected to feature calculations using a recursive algorithm to obtain the addressing matrix and addressing feature matrix corresponding to the number of iterations and the preset number of layers. The second path result is determined based on the addressing matrix and the addressing feature matrix.

8. The metadata addressing method according to claim 7, characterized in that, The initial output vector is processed using a recursive algorithm to obtain the addressing matrix and addressing feature matrix corresponding to the number of iterations and the preset number of layers, including: Use the path addressing information corresponding to the initial output vector as the addressing matrix of the first layer; The addressing feature matrix of the first layer is determined based on the addressing matrix of the first layer, the number of iterations, and the initial output vector. Use the addressing feature matrix of the first layer as the initial output vector of the second layer; The path addressing information corresponding to the initial output vector of the second layer is used as the addressing matrix of the second layer. The addressing feature matrix of the third layer is determined based on the addressing matrix of the second layer, the number of iterations, and the initial output vector of the second layer. This process continues until the current number of iterations reaches the preset number of iterations and the current number of layers reaches the preset number of layers. At this point, the addressing matrix of the kth layer and the addressing feature matrix of the kth layer are output. Here, k is a positive integer greater than or equal to 1.

9. The metadata addressing method according to claim 8, characterized in that, The addressing matrix of the k-th layer is output using the following formula: Where Z0 is the initial output vector, Z k Let W be the addressing matrix of the k-th layer. i Let m be the addressing feature matrix of the i-th layer; m is the iteration number.

10. The metadata addressing method according to claim 6, characterized in that, The second path result is obtained by evaluating and calculating the first path result according to the path optimization strategy, including: Obtain the historical path results and corresponding path access information corresponding to the first path result; The historical path results and path access information are processed by a machine learning algorithm to obtain the second path result.

11. The metadata addressing method according to claim 8, characterized in that, The second path result is subjected to locality processing based on a locality function to determine the locality of the corresponding metadata from the source metadata service to the target metadata service, including: Obtain the path result from the source metadata service to the target metadata service in the second path result; The number of request traversals from the source metadata service to the corresponding metadata of the target metadata service is obtained; The locality of the metadata is determined based on the path result, the number of request traversals, the marked path information corresponding to the source metadata service to the target metadata service, and the number of targets.

12. The metadata addressing method according to claim 11, characterized in that, The process of determining the number of requests to be traversed includes: Get the preset request information; The preset request information is sent from the source node where the source metadata service is located to the other nodes besides the source node, and the other nodes are regarded as the target nodes where the target metadata service is located; The target node containing the target metadata service is traversed to obtain the preset request information and the corresponding number of request traversals.

13. The metadata addressing method according to claim 12, characterized in that, The locality of the metadata is determined based on the path results, the number of request traversals, the tagged path information corresponding to the source metadata service to the target metadata service, and the number of targets, including: Obtain the number of traversals for each request from the source metadata service to the target metadata service; Subtract the target quantity from the marked path information of the source metadata service to obtain the first data; Subtract 1 from the first data to obtain the second data; The third data is obtained by multiplying the target quantity with the tag path information of the target metadata service; The fourth data is obtained by multiplying the number of request traversals corresponding to the current target metadata service by the second data. The fifth data is obtained by dividing the fourth data by the third data; The locality corresponding to the current target metadata service is obtained by multiplying the fifth data with the second path result corresponding to the current target metadata service. The locality of the metadata is obtained by adding the localities corresponding to each target metadata service.

14. The metadata addressing method according to claim 6, characterized in that, Establish a mapping relationship between path length and latency information for each metadata service path, including: The path length is used as the first priority information; Sort the metadata service paths from largest to smallest according to the first priority information; Based on the sorted metadata service paths, determine whether the corresponding latency information is sorted from largest to smallest; If the corresponding latency information is sorted from largest to smallest, then a mapping relationship between path length and latency information is established.

15. The metadata addressing method according to claim 13, characterized in that, Sort the metadata service paths of the target quantity according to the mapping relationship to obtain the first path result, including: Set the decimal threshold less than 1 for each sorted metadata service path in sequence, gradually increasing from small to large. The data obtained by subtracting each decimal threshold less than 1 from 1 are used as the first path result corresponding to each sorted metadata service path.

16. The metadata addressing method according to claim 2, characterized in that, The metadata service paths corresponding to the request information are filtered based on a preset continuous timestamp parameter to obtain a first number of metadata service paths, including: Check whether the timestamp parameter of the metadata service path for the requested information is continuous; If the timestamp parameters of the metadata service paths for the requested information are consecutive, then the metadata service paths to which the consecutive timestamp parameters belong are filtered out to obtain the first number of metadata service paths.

17. The metadata addressing method according to claim 1, characterized in that, The distributed storage system includes multiple distributed storage nodes, and each distributed storage node includes multiple metadata server service processes. One of them serves as the primary metadata server service process, which is mainly responsible for providing services to the outside world. The other metadata server service processes besides the primary metadata server service process serve as backup metadata server service processes, which are used to provide metadata services during failover. The distributed storage nodes are physically connected through dual 10 Gigabit network links, and the metadata server service processes communicate, synchronize, and verify data consistency through dual 10 Gigabit links.

18. A metadata addressing device, characterized in that, include: Memory, configured to store computer programs; The processor, configured to execute the computer program, implements the steps of the metadata addressing method as described in any one of claims 1 to 17.

19. A computer non-volatile readable storage medium, characterized in that, The computer non-volatile readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the metadata addressing method as described in any one of claims 1 to 17.

20. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the addressing method for metadata as described in any one of claims 1 to 17.

Images