Metadata processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a metadata processing method and device, electronic equipment and a storage medium, and the method comprises the steps: dividing storage resources of each storage node in a distributed storage system into a plurality of data objects, and obtaining metadata information of each data object; for any data object, when metadata information of the data object is stored to a local node, a copy node corresponding to the data object is selected in the distributed storage system, and the metadata information of the data object serves as a metadata copy to be synchronously stored to the copy node; and under the condition that any replica node is offline, recovering the metadata replica stored in the offline replica node to the online storage node. By adopting a multi-copy storage mode, the metadata information of the data object is stored to the local node and the copy node, so that the situation that the concurrent access of multi-node data is limited by the management of the system on the metadata is avoided, and the overall performance of the distributed storage system is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a metadata processing method, device, electronic device and storage medium. Background Art

[0002] With the development of information technology, distributed storage systems have been widely used due to their advantages such as high reading performance and multi-copy storage mechanism.

[0003] In related technologies, a metadata management service is usually deployed in a distributed storage system to record data copy information of each storage node in the system based on the metadata management service. When any storage node fails, the data copy to be restored is read from other normal storage nodes based on the data copy information recorded by the metadata management service.

[0004] However, when the scale of the distributed storage system is large, since the metadata management service needs to participate in the data access of each storage node, the metadata management service will limit the concurrent access to the data of multiple nodes, thereby reducing the overall performance of the distributed storage system. Summary of the invention

[0005] The present application provides a metadata processing method, device, electronic device and storage medium to solve the defects in related technologies such as data management services restricting concurrent access to multi-node data, reducing the overall performance of distributed storage systems, etc.

[0006] The first aspect of the present application provides a metadata processing method, comprising:

[0007] Divide the storage resources of each storage node in the distributed storage system into a plurality of data objects, and obtain metadata information of each of the data objects;

[0008] For any of the data objects, while storing the metadata information of the data object in the local node, selecting a replica node corresponding to the data object in the distributed storage system, and synchronously storing the metadata information of the data object as a metadata replica in the replica node;

[0009] In the event that any of the replica nodes goes offline, the metadata copy stored in the offline replica node is restored to the online storage node.

[0010] In an optional implementation, the selecting a replica node corresponding to the data object in the distributed storage system includes:

[0011] Obtaining the current version number of the distributed storage system;

[0012] Determine the current data distribution logic according to the current version number of the distributed storage system;

[0013] According to the current data distribution logic, the replica node corresponding to the storage object is determined based on the unique identifier of the data object corresponding to the storage object.

[0014] In an optional implementation, restoring the metadata copy stored in the offline replica node to the online storage node includes:

[0015] For any of the online storage nodes, according to the source node index and the latest version number, a metadata request is sent to each of the online storage nodes to obtain a metadata information list fed back by each of the online storage nodes;

[0016] Determine the node to be restored list according to the metadata information list fed back by each of the online storage nodes;

[0017] Performing metadata copy recovery on each online storage node according to the node to-be-recovered list obtained by each online storage node;

[0018] The latest version number is updated when a node goes online or offline in the distributed storage system, and the node to be restored list includes a metadata copy stored in the offline copy node.

[0019] In an optional embodiment, the method further includes:

[0020] In the case where any of the online storage nodes receives the metadata request, if the latest version number carried by the metadata request is greater than the local cache version number, local metadata information of the local data object is traversed according to the latest version number carried by the metadata request and the source node index, so as to filter the local metadata information matching the source node index in the local cache;

[0021] Summarize the local metadata information that matches the source node index to obtain a metadata information list to be fed back;

[0022] According to the source node index carried in the metadata request, the metadata information list is fed back to the corresponding online storage node.

[0023] In an optional implementation, traversing the local metadata information of the local data object according to the latest version number and the source node index carried by the metadata request to filter the local metadata information matching the source node index includes:

[0024] Determine the latest data distribution logic according to the latest version number carried by the metadata request; wherein the latest version number at least records the total number of online storage nodes of the distributed storage system and the storage capacity of each of the online storage nodes;

[0025] Traverse the metadata information of the local data object to determine the unique identifier of the data object corresponding to each metadata information;

[0026] According to the latest data distribution logic, and based on the unique identifier of the data object corresponding to each metadata cache in the local cache, determine the target storage node corresponding to each local metadata information;

[0027] According to the source node index carried in the metadata request and the target storage node corresponding to each metadata cache, local metadata information matching the source node index is screened.

[0028] In an optional embodiment, the method further includes:

[0029] If the latest version number carried in the metadata request is smaller than the local cache version number, error information is fed back to the corresponding online storage node according to the source node index carried in the metadata request;

[0030] If the latest version number carried by the metadata request is equal to the local cache version number, execute the step of filtering local metadata information that matches the source node index based on the source node index carried by the metadata request and the target storage nodes corresponding to each metadata cache.

[0031] In an optional embodiment, the method further includes:

[0032] In the case where a new storage node is newly online in the distributed storage system, data redistribution processing is performed on the distributed storage system to allocate part of the metadata copies cached on the previously online storage node to the new storage node.

[0033] A second aspect of the present application provides a metadata processing device, comprising:

[0034] An acquisition module is used to divide the storage resources of each storage node in the distributed storage system into a plurality of data objects and acquire metadata information of each of the data objects;

[0035] A backup module is used for storing metadata information of any of the data objects in a local node, selecting a replica node corresponding to the data object in the distributed storage system, and synchronously storing the metadata information of the data object as a metadata replica in the replica node;

[0036] The processing module is used to restore the metadata copy stored in the offline replica node to the online storage node when any of the replica nodes goes offline.

[0037] A third aspect of the present application provides an electronic device, comprising: at least one processor and a memory;

[0038] The memory stores computer-executable instructions;

[0039] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0040] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0041] A fifth aspect of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the method described in the first aspect and various possible designs of the first aspect.

[0042] The technical solution of this application has the following advantages:

[0043] The present application provides a metadata processing method, device, electronic device and storage medium, the method comprising: dividing the storage resources of each storage node in a distributed storage system into a number of data objects, and obtaining metadata information of each data object; for any data object, storing the metadata information of the data object to the local node, selecting a replica node corresponding to the data object in the distributed storage system, and synchronously storing the metadata information of the data object as a metadata copy to the replica node; in the event that any replica node goes offline, restoring the metadata copy stored in the offline replica node to the online storage node. The method provided by the above scheme stores the metadata information of the data object to the local node and the replica node by adopting a multi-copy storage method, thereby realizing node decentralized management of the metadata information, avoiding the system's management of metadata restricting concurrent access to multi-node data, and improving the overall performance of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0045] Figure 1 A schematic diagram of the structure of a distributed storage system based on an embodiment of the present application;

[0046] Figure 2 A flowchart of a metadata processing method provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the application process of a client node provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the application process of the server node provided in the embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of a metadata processing device provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0051] The above drawings have shown clear embodiments of the present application, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0054] With the development of information technology, storage technology is constantly evolving around how to ensure high performance and high reliability of data storage. Centralized storage concentrates storage resources on a single storage device or server, and data access performance is limited by a single point bottleneck. In addition, since data is stored according to a single copy strategy, a single point failure may cause data loss or unavailability; distributed storage connects multiple independent storage devices or servers through a network and uses software technology to provide storage services to the outside world as a whole. On the one hand, it can access data from multiple nodes concurrently to improve read and write performance. On the other hand, data is usually stored using a multi-copy mechanism (storing multiple copies of the same data), which has higher reliability. With these advantages, distributed storage has developed rapidly and the market size continues to expand.

[0055] In a distributed storage system, some data copies are lost after a node fails. In order to meet the number of data copies, the lost data copies need to be restored. The independent metadata management service records the data copies of each node, obtains the data copies lost by the failed node, reads the data of the remaining data copies from the normal nodes, and restores the data copies to the normal nodes in the system according to the data distribution of each node in the system.

[0056] When a node fails, an independent metadata management service can help efficiently identify the data copies to be recovered and improve data recovery efficiency. However, in large-scale system scenarios, data access requires the participation of metadata management services. At this time, the management service often becomes a single point bottleneck, which is contrary to concurrent access to multi-node data, resulting in reduced system performance.

[0057] In response to the above problems, an embodiment of the present application provides a metadata processing method, device, electronic device and storage medium, the method comprising: dividing the storage resources of each storage node in a distributed storage system into a number of data objects, and obtaining metadata information of each data object; for any data object, while storing the metadata information of the data object to the local node, selecting a replica node corresponding to the data object in the distributed storage system, and synchronously storing the metadata information of the data object as a metadata copy to the replica node; in the event that any replica node goes offline, restoring the metadata copy stored in the offline replica node to the online storage node. The method provided by the above scheme stores the metadata information of the data object to the local node and the replica node by adopting a multi-copy storage method, thereby realizing node decentralized management of the metadata information, avoiding the system's management of metadata restricting concurrent access to multi-node data, and improving the overall performance of the distributed storage system.

[0058] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0059] First, the structure of the distributed storage system on which this application is based is described:

[0060] The metadata processing method, device, electronic device and storage medium provided in the embodiments of the present application are suitable for managing metadata information of data copies of each storage node in a distributed storage system. Figure 1 As shown, it is a structural diagram of the distributed storage system based on the embodiment of the present application. The distributed storage system includes multiple storage nodes, and each storage node is deployed with a metadata processing device so that each storage node can store and manage multiple copies of metadata information based on the metadata processing device.

[0061] The embodiment of the present application provides a metadata processing method for managing metadata information of data replicas of each storage node in a distributed storage system. The execution subject of the embodiment of the present application is an electronic device, such as a server, a desktop computer, a laptop computer, a tablet computer, and other electronic devices that can be used for metadata information management.

[0062] like Figure 2 FIG. 1 is a flow chart of a metadata processing method provided in an embodiment of the present application, the method comprising:

[0063] Step 201: divide the storage resources of each storage node in the distributed storage system into a plurality of data objects, and obtain metadata information of each data object.

[0064] Specifically, the storage resources provided by the storage node can be divided into multiple data objects according to a certain unit size. Each node in the distributed storage system only manages the data object metadata information of the node. The metadata information includes at least the data object ID (data object unique identifier), version, and whether it has been written.

[0065] Step 202, for any data object, while storing the metadata information of the data object in the local node, select a replica node corresponding to the data object in the distributed storage system, and synchronously store the metadata information of the data object as a metadata replica in the replica node.

[0066] Specifically, after obtaining the metadata information of the data object of the local node, the storage node saves the metadata information persistently to the local node disk, and also selects a replica node corresponding to each data object in the distributed storage system, and synchronously stores the metadata information of the data object as a metadata copy to the replica node. The embodiment of the present application cancels the unified metadata management service and adopts a multi-copy metadata information management method to achieve complete decentralization and optimize the performance bottleneck in large-scale scenarios.

[0067] Step 203: When any replica node goes offline, the metadata copy stored in the offline replica node is restored to the online storage node.

[0068] Specifically, when any replica node goes offline, in order to avoid the loss of metadata copies stored in the offline replica node, metadata copies can be redistributed to restore the metadata copies stored in the offline replica node to the online storage node.

[0069] Based on the above embodiment, as an implementable manner, in one embodiment, selecting a replica node corresponding to the data object in a distributed storage system includes:

[0070] Step 2021, obtaining the current version number of the distributed storage system;

[0071] Step 2022, determining the current data distribution logic according to the current version number of the distributed storage system;

[0072] Step 2023: According to the current data distribution logic and based on the unique identifier of the data object corresponding to the storage object, determine the replica node corresponding to the storage object.

[0073] Among them, the current version number of the distributed storage system is used to characterize the current state of the distributed storage system, which can actually reflect the evolution of the data distribution state of the system at different time points due to factors such as node joining, leaving, and storage capacity changes. By obtaining the current version number, the current state of the distributed storage system can be accurately identified, providing a basic basis for the subsequent determination of data distribution logic.

[0074] Specifically, multiple data distribution logics can be set in advance to achieve the goals of balanced data storage, high availability, and efficient access. After determining the current version number of the distributed storage system, the specific current data distribution logic can be mapped according to the current version number. For example, based on a certain consistent hashing algorithm (such as a hash ring built based on factors such as node storage capacity), the structure, capacity, or calculation method of the hash ring may be different under different version numbers, resulting in changes in the distribution results of data objects on the nodes. The system searches for the corresponding rule set or algorithm parameters based on the current version number to determine the current data distribution logic and clarify how the metadata copies of the data objects should be distributed and stored among the nodes.

[0075] Furthermore, after determining the current data distribution logic, the unique identifier of the data object can be used as input for calculation according to the current data distribution logic (such as an algorithm based on a hash ring). For example, by performing a hash operation on the data object ID and combining the distribution of nodes on the hash ring (determining the location of the node on the hash ring based on factors such as the node storage capacity), the node location to which the metadata copy of the data object should be distributed under the current system state is calculated. That is, based on its unique identifier and the current data distribution logic, its replica node is accurately found to achieve redundant storage and high availability of data, and to ensure the reliability and recoverability of data in the event of node failure or other abnormal conditions.

[0076] Based on the above embodiment, as an implementable manner, in one embodiment, restoring the metadata copy stored in the offline replica node to the online storage node includes:

[0077] Step 2031: for any online storage node, send a metadata request to each online storage node according to the source node index and the latest version number, so as to obtain a metadata information list fed back by each online storage node;

[0078] Step 2032, determining the node to-be-restored list according to the metadata information list fed back by each online storage node;

[0079] Step 2033: Perform metadata copy recovery on each online storage node according to the node to-be-recovered list obtained by each online storage node.

[0080] Among them, the latest version number is updated when a node goes online or offline in the distributed storage system. The source node index is used to mark the location of the online storage node that sends the metadata request in the system. The node to be restored list includes the metadata copies stored in the offline replica nodes.

[0081] Specifically, when a node failure occurs in a distributed storage system and the system goes offline, the system broadcasts the event through the network. After receiving the event, each storage node in the system generates a list of nodes in the system, starting with the current node, and obtains the required metadata information node by node. Each node acts as a client to send requests to other nodes, and also acts as a server to process node requests. After obtaining the data objects distributed by this node according to the data distribution algorithm, the data objects of the remote nodes (the list to be restored by this node) and the metadata information of the local data objects (the stored metadata copies) are compared, and the data object copies that do not currently exist in this node or the data objects with smaller version numbers are restored.

[0082] Specifically, when a node in the distributed storage system goes online or offline, the system will update the version number to reflect changes in data distribution and node status. When any online storage node detects that the version number has been updated, the online storage node will perform the above steps 2031 to 2032 as a client node, that is, send the metadata request to each storage node in the distributed storage system in the form of a broadcast, and the storage node that receives the metadata request will respond to the metadata request as a server node to feedback a metadata information list to the online storage node (client node), and the metadata information list actually includes the metadata copy to be backed up and stored by the server node to the client node.

[0083] Specifically, the client node determines the list to be restored of this node by summarizing the metadata information lists fed back by each server node. The list to be restored of this node actually includes the metadata copies of each server node to be backed up and stored in the client node. The list to be restored of this node must include the metadata copies stored in the offline replica nodes, because some of the metadata copies previously backed up in the offline replica nodes will be backed up by the server node to the client node.

[0084] For example, Figure 3 As shown, it is a schematic diagram of the application process of the client node provided by the embodiment of the present application. As a client node, obtain the list of all node information corresponding to the latest version number to determine the information of each storage node in the distributed storage system and record the number of nodes. The list obtained by each node in the system is consistent. Each node starts from this node, traverses the list to find the index of this node, assembles and sends metadata requests according to the IP (index of this node) and the latest version number, so as to obtain the list of metadata information of the data object from the server node, that is, obtain the list of metadata information fed back by each server node. After receiving the result returned by this node, temporarily record it. If it is the result of other nodes in the system, search for the list of data objects to be restored. If there is no addition of data objects, there is a comparison and update of metadata information version. Traverse the node list and send a request to the next node. If this is the last index of the list, start from the first index of the list again until the number of completions is equal to the total number of nodes corresponding to the version. That is, since each storage node in the system sends metadata requests concurrently, in order to avoid metadata request congestion, each storage node sends metadata requests in the same direction in the embodiment of the present application. For example, the system includes 10 storage nodes, and the serial number of the current storage node is 5. The current storage node first sends a metadata request to the storage node No. 6, and then sends a metadata request to the storage node No. 7, and so on, until the metadata request for the storage node No. 10 is sent, and then the metadata request is sent from the storage node No. 1 until the obtained metadata information list reaches the total number of nodes (10). After the above steps, two metadata lists are obtained: the metadata list corresponding to the existing data objects of this node, and the list of data objects that should be merged with the data objects of this node (the list to be restored at this node) calculated by each remote node. The two metadata lists are compared, and the data objects that do not exist at this node, or the data objects that exist at this node but have a lower version number need to copy data from the remote node (server node) for recovery.

[0085] Furthermore, when the data copy of the last node in the system is restored, the network broadcasts the event to all nodes in the system. Each node assembles a request to clean up the metadata list cache with a version number and submits it to the local for processing. First, the request version number is compared with the metadata cache version number. If the cleanup request version number is lower, the request is invalid and will not be processed. Otherwise, if the metadata cache is not empty, it is necessary to traverse all nodes in the cache and clean them one by one, that is, clear the list of nodes to be restored in each storage node to free up storage space.

[0086] Based on the above embodiment, as an implementable manner, in one embodiment, the method further includes:

[0087] Step 301: When any online storage node receives a metadata request, if the latest version number carried in the metadata request is greater than the local cache version number, local metadata information of the local data object is traversed according to the latest version number carried in the metadata request and the source node index, so as to select local metadata information matching the source node index in the local cache;

[0088] Step 302, summarizing the local metadata information that matches the source node index to obtain a metadata information list to be fed back;

[0089] Step 303: Feedback the metadata information list to the corresponding online storage node according to the source node index carried in the metadata request.

[0090] It should be noted that if the latest version number is greater than the local cache version number, this indicates that the client node's request is based on an updated system status. The server node needs to adjust its response behavior based on this new version number to provide metadata information consistent with the latest system status.

[0091] Specifically, in one embodiment, the latest data distribution logic can be determined based on the latest version number carried by the metadata request; wherein the latest version number records at least the total number of online storage nodes of the distributed storage system and the storage capacity of each online storage node; the metadata information of the local data object is traversed to determine the unique identifier of the data object corresponding to each metadata information; according to the latest data distribution logic, the target storage node corresponding to each local metadata information is determined based on the unique identifier of the data object corresponding to each metadata cache in the local cache; based on the source node index carried by the metadata request and the target storage node corresponding to each metadata cache, the local metadata information matching the source node index is screened.

[0092] Specifically, when the server node determines that the latest version number carried by the metadata request is greater than the local cache version number, the server determines the latest data distribution logic based on the latest version number. Then, the server node begins to traverse the metadata information of the local data object. During the traversal process, the unique identifier of the data object corresponding to each metadata information (such as the data object ID) is determined. This unique identifier is the key to accurately distinguish and locate data objects in the distributed storage system. It plays a core role in subsequent data distribution calculations and metadata information matching, ensuring that the server can accurately find metadata information related to the source node index according to the data distribution logic. According to the determined latest data distribution logic, the target storage node corresponding to each metadata information is calculated according to the unique identifier of the data object corresponding to each metadata cache in the local cache. Ensure that the server node only returns metadata information of data objects related to the requesting node (source node) and in line with the latest system status, avoid providing irrelevant data, ensure the accuracy and effectiveness of metadata information, and provide a reliable basis for subsequent operations such as data copy recovery.

[0093] Accordingly, in one embodiment, if the latest version number carried by the metadata request is smaller than the local cache version number, error information is fed back to the corresponding online storage node based on the source node index carried by the metadata request; if the latest version number carried by the metadata request is equal to the local cache version number, the step of screening local metadata information that matches the source node index is executed based on the source node index carried by the metadata request and the target storage nodes corresponding to each metadata cache.

[0094] For example, Figure 4As shown, it is a schematic diagram of the application process of the server node provided by the embodiment of the present application. As a server node, after receiving the metadata request, it verifies according to the version number in the request. If the version number in the metadata information cache is higher, that is, the latest version number carried by the metadata request is smaller than the local cache version number, it returns an error message without processing. Otherwise, it further checks whether there is data in the local cache. If there is no data, it first updates the version number in the metadata cache to the latest version number carried by the request. The corresponding node and data distribution information are obtained from the version number. A certain amount of memory is pre-applied for each node to save the corresponding metadata information list, and then the metadata information list of all local data objects is obtained. The traversal list calculates the node (replica node) where the data object is located under the latest version number according to the data object ID, and copies the data object to the node metadata list. That is, when processing the first node request, the metadata list corresponding to all nodes under this version is constructed and cached; if there is data in the cache, it is necessary to verify the version number in the cache. When the version number in the cache is small, the data in the cache is unavailable, and it is necessary to rebuild the metadata list of all nodes with the requested version number; if the version numbers are the same, the cache is searched with the request IP, and the node metadata list is returned, recording that the cache of the node has been used. After the request is processed, it is checked that all nodes in the cache have been used, and the cache is cleaned up uniformly.

[0095] Specifically, in one embodiment, when a new storage node is newly online in the distributed storage system, data redistribution processing is performed on the distributed storage system to allocate part of the metadata copies cached on the previously online storage node to the new storage node.

[0096] Among them, the specific process of allocating part of the metadata copies cached on the previously online storage node to the new storage node is the same as the above process, that is, all storage nodes in the system including the new storage node respond to the above process both as client nodes and as server nodes to achieve the redistribution of metadata copies to balance the metadata cache load of each storage node in the system.

[0097] Specifically, in one embodiment, in order to further improve the performance of the distributed storage system and achieve load balancing of the distributed storage system, the performance fluctuations of each storage node of the distributed storage system (such as changes in the read and write speed of the storage device, temporary limitation of the network bandwidth, etc.), the dynamic evolution of the data access mode (such as a sudden increase or decrease in the access frequency of certain data objects), etc. can be monitored in real time. Through comprehensive monitoring of these multi-dimensional change factors, the system can dynamically adjust the data distribution logic and metadata management strategy. For example, when it is detected that the read and write speed of the storage device of a certain node has dropped, the system can automatically adjust the data distribution, migrate some high-frequency access data objects to other nodes with better performance, and update the metadata information accordingly to ensure that efficient access to data is not affected, so that the system can always maintain good performance and reliability in a complex and changing environment.

[0098] The metadata processing method provided by the embodiment of the present application divides the storage resources of each storage node in the distributed storage system into several data objects, and obtains the metadata information of each data object; for any data object, the metadata information of the data object is stored in the local node, and the replica node corresponding to the data object is selected in the distributed storage system, and the metadata information of the data object is synchronously stored as a metadata copy in the replica node; in the case of any replica node going offline, the metadata copy stored in the offline replica node is restored to the online storage node. The method provided by the above scheme stores the metadata information of the data object in the local node and the replica node by adopting a multi-copy storage method, realizes the node decentralized management of the metadata information, avoids the system's management of metadata limiting the concurrent access of multi-node data, and improves the overall performance of the distributed storage system. In addition, when each node obtains the metadata information list from each other, each node only needs to traverse the local data object metadata list once, reducing repeated calculations, and accurately identifies the data copy that needs to be restored through the comparison of the metadata information list, ensuring the efficiency of data copy recovery.

[0099] An embodiment of the present application provides a metadata processing device, which is used to execute the metadata processing method provided in the above embodiment.

[0100] like Figure 5 , which is a schematic diagram of the structure of a metadata processing device provided in an embodiment of the present application. The metadata processing device 50 comprises: an acquisition module 501 , a backup module 502 and a processing module 503 .

[0101] Among them, the acquisition module is used to divide the storage resources of each storage node in the distributed storage system into several data objects, and obtain the metadata information of each data object; the backup module is used to store the metadata information of any data object to the local node, select the replica node corresponding to the data object in the distributed storage system, and synchronously store the metadata information of the data object as a metadata copy to the replica node; the processing module is used to restore the metadata copy stored in the offline replica node to the online storage node when any replica node goes offline.

[0102] Regarding the metadata processing device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0103] The metadata processing device provided in the embodiment of the present application is used to execute the metadata processing method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0104] An embodiment of the present application provides an electronic device for executing the metadata processing method provided in the above embodiment.

[0105] like Figure 6 FIG. 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 60 includes: at least one processor 61 and a memory 62 .

[0106] The memory stores computer-executable instructions; at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the metadata processing method provided in the above embodiment.

[0107] The electronic device provided in the embodiment of the present application is used to execute the metadata processing method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0108] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the metadata processing method provided in any of the above embodiments is implemented.

[0109] The storage medium containing computer executable instructions provided in the embodiments of the present application can be used to store computer executable instructions of the metadata processing method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0110] An embodiment of the present application provides a computer program product, including computer instructions, wherein the computer instructions are used to enable a computer to execute the metadata processing method provided in the aforementioned embodiment.

[0111] The computer program product provided in the embodiments of the present application can be used to execute computer instructions of the metadata processing method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0112] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0115] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0116] A part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0117] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A metadata processing method, characterized in that: include: Divide the storage resources of each storage node in the distributed storage system into a plurality of data objects, and obtain metadata information of each of the data objects; For any of the data objects, while storing the metadata information of the data object in the local node, selecting a replica node corresponding to the data object in the distributed storage system, and synchronously storing the metadata information of the data object as a metadata replica in the replica node; In the event that any of the replica nodes goes offline, the metadata copy stored in the offline replica node is restored to the online storage node.

2. The method according to claim 1, characterized in that The selecting a replica node corresponding to the data object in the distributed storage system includes: Obtaining the current version number of the distributed storage system; Determine the current data distribution logic according to the current version number of the distributed storage system; According to the current data distribution logic, the replica node corresponding to the storage object is determined based on the unique identifier of the data object corresponding to the storage object.

3. The method according to claim 1, characterized in that The step of restoring the metadata copy stored in the offline replica node to the online storage node includes: For any of the online storage nodes, according to the source node index and the latest version number, a metadata request is sent to each of the online storage nodes to obtain a metadata information list fed back by each of the online storage nodes; Determine the node to be restored list according to the metadata information list fed back by each of the online storage nodes; Performing metadata copy recovery on each online storage node according to the node to-be-recovered list obtained by each online storage node; The latest version number is updated when a node goes online or offline in the distributed storage system, and the node to be restored list includes a metadata copy stored in the offline copy node.

4. The method according to claim 3, characterized in that The method further comprises: In the case where any of the online storage nodes receives the metadata request, if the latest version number carried by the metadata request is greater than the local cache version number, local metadata information of the local data object is traversed according to the latest version number carried by the metadata request and the source node index, so as to filter the local metadata information matching the source node index in the local cache; Summarize the local metadata information that matches the source node index to obtain a metadata information list to be fed back; According to the source node index carried in the metadata request, the metadata information list is fed back to the corresponding online storage node.

5. The method according to claim 4, characterized in that The traversing the local metadata information of the local data object according to the latest version number and the source node index carried by the metadata request to filter the local metadata information matching the source node index includes: Determine the latest data distribution logic according to the latest version number carried by the metadata request; wherein the latest version number at least records the total number of online storage nodes of the distributed storage system and the storage capacity of each of the online storage nodes; Traverse the metadata information of the local data object to determine the unique identifier of the data object corresponding to each metadata information; According to the latest data distribution logic, and based on the unique identifier of the data object corresponding to each metadata cache in the local cache, determine the target storage node corresponding to each local metadata information; According to the source node index carried in the metadata request and the target storage node corresponding to each metadata cache, local metadata information matching the source node index is screened.

6. The method according to claim 5, characterized in that The method further comprises: If the latest version number carried in the metadata request is smaller than the local cache version number, error information is fed back to the corresponding online storage node according to the source node index carried in the metadata request; If the latest version number carried by the metadata request is equal to the local cache version number, execute the step of filtering local metadata information that matches the source node index based on the source node index carried by the metadata request and the target storage nodes corresponding to each metadata cache.

7. The method according to claim 1, characterized in that The method further comprises: In the case where a new storage node is newly online in the distributed storage system, data redistribution processing is performed on the distributed storage system to allocate part of the metadata copies cached on the previously online storage node to the new storage node.

8. A metadata processing device, characterized in that: include: An acquisition module is used to divide the storage resources of each storage node in the distributed storage system into a plurality of data objects and acquire metadata information of each of the data objects; A backup module is used for storing metadata information of any of the data objects in a local node, selecting a replica node corresponding to the data object in the distributed storage system, and synchronously storing the metadata information of the data object as a metadata replica in the replica node; The processing module is used to restore the metadata copy stored in the offline replica node to the online storage node when any of the replica nodes goes offline.

9. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 6 is implemented.

Images