Data storage system, method and device

Abstract

The invention relates to a data storage system, method and device. The system comprises a terminal, at least one metadata storage node and a metadata control node, wherein the terminal is used for sending a first metadata operation request to the metadata control node; the metadata control node is used for determining a metadata storage identifier, pre-estimating an expected execution time length of the metadata control node, and sending the metadata storage identifier to the terminal when the expected execution time length is greater than or equal to a time length threshold value; the terminal is further used for determining a first metadata storage node corresponding to the metadata storage identifier and sending a second metadata operation request to the first metadata storage node; the first metadata storage node is further used for responding to the second metadata operation request and executing target operation on the target metadata corresponding to the metadata storage identifier. By adopting the method, the request processing efficiency of the control node can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data storage system, method and device. Background Art

[0002] With the development of computer technology, distributed storage clusters are becoming more and more common. When users need to access data in a distributed storage cluster, they often need to obtain the metadata of the data first, and determine which storage node in the cluster the data is stored on based on the metadata, before they can access the storage node and obtain the data.

[0003] Related technologies usually set a control node to manage the process of users obtaining metadata. The terminal's request is first sent to the control node, which determines which storage node the metadata is located on, and reads the metadata from the storage node and returns it to the terminal.

[0004] Since the metadata is read frequently, the control node often needs to process a large number of terminal requests in a short period of time. If the control node takes too long to process certain requests, the terminal will not be able to obtain metadata in time. When a large number of unfinished requests occupy the thread resources of the control node, it may even cause the distributed storage cluster to respond slowly or be temporarily unavailable. Therefore, it is necessary to provide a method that can improve the efficiency of request processing by the control node. Summary of the invention

[0005] Based on this, it is necessary to provide a data storage system, method and device to address the above technical problems.

[0006] In a first aspect, the present application provides a data storage system. The system includes a terminal, at least one metadata storage node and a metadata control node, wherein:

[0007] The terminal is used to send a first metadata operation request to the metadata control node, where the first metadata operation request carries feature information of the metadata and an operation identifier corresponding to a target operation;

[0008] The metadata control node is used to determine the metadata storage identifier based on the characteristic information, and estimate the expected execution time of the metadata control node corresponding to the target operation, and send the metadata storage identifier to the terminal when the expected execution time is greater than or equal to a time threshold;

[0009] The terminal is further used to determine, based on the metadata storage identifier, a first metadata storage node corresponding to the metadata storage identifier, and send a second metadata operation request to the first metadata storage node, wherein the second metadata operation request carries the metadata storage identifier and the operation identifier;

[0010] The first metadata storage node is further configured to execute the target operation on the target metadata corresponding to the metadata storage identifier in response to the second metadata operation request.

[0011] In one embodiment, the metadata control node is further used to determine each target metadata storage node involved in the target operation, and estimate the expected response time of each target metadata storage node according to the load information and network information of each target metadata storage node;

[0012] The metadata control node is further used to determine the expected execution duration according to the expected response duration of each target metadata storage node.

[0013] In one of the embodiments, the metadata storage node is further used to periodically report load information and network information to the metadata control node, and after performing any operation, report the response time corresponding to the operation to the metadata control node;

[0014] The metadata control node is also used to estimate the expected response time of each target metadata storage node based on the load information and network information of each target metadata storage node, as well as the load information, network information and response time reported by each target metadata storage node during a target statistical period.

[0015] In one of the embodiments, the system further comprises at least one business data storage node;

[0016] In the case where the target operation is a read operation, the first metadata storage node is further configured to send the target metadata corresponding to the metadata storage identifier to the terminal in response to the second metadata operation request;

[0017] The terminal is further used to determine, based on the target metadata, a first business data storage node where the business data corresponding to the target metadata is located, and send a business data acquisition request to the first business data storage node, wherein the business data acquisition request carries the target metadata;

[0018] The first business data storage node is further configured to send the business data corresponding to the target metadata to the terminal in response to the business data acquisition request.

[0019] In one of the embodiments, when the target operation is the read operation, the metadata control node is further used to send a metadata read request to the first metadata storage node when the expected execution time is less than the time threshold, and the metadata read request carries the metadata storage identifier;

[0020] The first metadata storage node is further configured to send the target metadata corresponding to the metadata storage identifier to the metadata control node in response to the metadata read request;

[0021] The metadata control node is further used to send the target metadata to the terminal.

[0022] In one of the embodiments, when the target operation is a move operation, the second metadata operation request also carries node information of a second metadata storage node, and the second metadata storage node is the destination node to which the target metadata needs to be moved;

[0023] The first metadata storage node is further configured to send the target metadata corresponding to the metadata storage identifier to the second metadata storage node in response to the second metadata operation request;

[0024] The second metadata storage node is further used to store the target metadata and send a metadata storage identifier modification request to the metadata control node;

[0025] The metadata control node is further used to update the metadata storage identifier of the target metadata based on the metadata storage identifier modification request.

[0026] In one of the embodiments, the terminal is further used to send an idle metadata storage node acquisition request to the metadata control node, wherein the idle metadata storage node acquisition request carries read requirement information for the target metadata, and the read requirement information at least includes a target response time;

[0027] The metadata control node is further used to determine candidate metadata storage nodes according to the read demand information, and send node information of each candidate metadata storage node to the terminal, wherein the node information includes at least one of remaining storage space, expected response time, and location information of the candidate metadata storage node;

[0028] The terminal is further configured to determine the second metadata storage node from among the candidate metadata storage nodes according to the node information.

[0029] In one of the embodiments, the metadata control node is also used to determine the expected response time of each metadata storage node based on the load information and network information of each metadata storage node, and select the metadata storage node whose expected response time is less than or equal to the target response time as the candidate metadata storage node.

[0030] In one of the embodiments, when the target operation is a deletion operation, the first metadata storage node is further used to delete the target metadata corresponding to the metadata storage identifier in response to the second metadata operation request, and send a metadata storage identifier deletion request to the metadata control node;

[0031] The metadata control node is further used to delete the metadata storage identifier of the target metadata based on the metadata storage identifier deletion request.

[0032] In a second aspect, the present application further provides a data storage method, which is applied to a metadata control node and includes:

[0033] Receiving a first metadata operation request sent by a terminal, wherein the first metadata operation request carries characteristic information of the metadata and an operation identifier corresponding to a target operation;

[0034] A metadata storage identifier is determined based on the characteristic information, and an expected execution duration of the metadata control node corresponding to the target operation is estimated. When the expected execution duration is greater than or equal to a duration threshold, the metadata storage identifier is sent to the terminal. The metadata storage identifier is used to indicate the metadata storage node where the metadata is located.

[0035] In a third aspect, the present application further provides a data storage device. The device is applied to a metadata control node, comprising:

[0036] A receiving module, configured to receive a first metadata operation request sent by a terminal, wherein the first metadata operation request carries feature information of the metadata and an operation identifier corresponding to a target operation;

[0037] A determination module is used to determine a metadata storage identifier based on the characteristic information, and estimate the expected execution time of the metadata control node corresponding to the target operation. When the expected execution time is greater than or equal to a time threshold, the metadata storage identifier is sent to the terminal, and the metadata storage identifier is used to indicate the metadata storage node where the metadata is located.

[0038] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any one of the above methods is implemented.

[0039] In a fifth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, any of the above methods is implemented.

[0040] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0041] The above data storage system, method and device enable the metadata control node to estimate the expected execution time of the target operation by the metadata control node after receiving the first metadata operation request, and when the expected execution time is greater than or equal to the time threshold, it is no longer responsible for executing the target operation, but after determining the metadata required by the user according to the user's request, it returns the identifier of the storage node where the metadata is located to the terminal, and the terminal initiates a request to the storage node to read the metadata. Therefore, the target operation with too long processing time can be transferred to the terminal for self-processing, reducing the average time required for the control node to complete the request execution, thereby improving the efficiency of the control node in processing requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of signaling interaction among nodes in a data storage system in one embodiment;

[0043] Figure 2 A signaling interaction flow chart when a terminal obtains service data in an embodiment;

[0044] Figure 3 is a flowchart of signaling interactions of various nodes when a terminal reads metadata when the expected execution duration is less than a duration threshold in an embodiment;

[0045] Figure 4 It is a flowchart of signaling interaction of various nodes when a terminal moves metadata in one embodiment;

[0046] Figure 5 A flowchart of signaling interactions among various nodes when a terminal determines a second metadata storage node in an embodiment;

[0047] Figure 6 A flowchart of signaling interactions among various nodes when a terminal deletes metadata in an embodiment;

[0048] Figure 7A schematic diagram of a flow chart of a data storage method in one embodiment;

[0049] Figure 8 is a structural block diagram of a data storage device in one embodiment;

[0050] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In one embodiment, Figure 1 As shown, a data storage system is provided. The data storage system includes a terminal, at least one metadata storage node and a metadata control node. The terminal is used to send a first metadata operation request to the metadata control node, the first metadata operation request carries feature information of metadata and an operation identifier corresponding to a target operation; the metadata control node is used to determine the metadata storage identifier based on the feature information, and estimate the expected execution time of the target operation corresponding to the metadata control node, and send the metadata storage identifier to the terminal when the expected execution time is greater than or equal to the duration threshold; the terminal is also used to determine the first metadata storage node corresponding to the metadata storage identifier based on the metadata storage identifier, and send a second metadata operation request to the first metadata storage node, the second metadata operation request carries the metadata storage identifier and the operation identifier; the first metadata storage node is also used to perform the target operation on the target metadata corresponding to the metadata storage identifier in response to the second metadata operation request.

[0053] In an embodiment of the present application, the data storage system includes a terminal and a data storage cluster, and the data storage cluster is composed of a metadata storage node and a metadata control node. Metadata is stored in the metadata storage node. The metadata storage node can store only metadata or store business data in addition to metadata. The metadata control node stores information of all metadata storage nodes.

[0054] When the terminal needs to operate the metadata (including but not limited to reading, moving, deleting, and modifying), it sends a first metadata operation request to the metadata control node. The first metadata operation request carries at least the characteristic information of the metadata and the operation identifier corresponding to the target operation. The characteristic information is any information that can identify unique metadata, such as the name of the business data corresponding to the metadata, the unique number of the metadata, etc.

[0055] The terminal can display a console page on the interface so that users can operate metadata through the console page. The console page can display each metadata storage node in the current data storage cluster. Users can enter metadata feature information and specify the operations to be performed on the metadata represented by the metadata feature information on the console page.

[0056] After receiving the first metadata operation request, the metadata control node determines the metadata storage identifier based on the characteristic information. Each metadata corresponds to a unique metadata storage identifier. The metadata storage identifier can be used to characterize the first metadata storage node where the metadata is located (for example, some bits in the metadata storage identifier represent the unique number of the metadata storage node). A metadata storage identifier library can be set up in the system (which can be located on the same device as the metadata control node), and a data table recording the correspondence between metadata characteristic information and metadata storage identifiers is stored in the metadata storage identifier library. The metadata control node can query the metadata storage identifier in the metadata storage identifier library according to the characteristic information of the metadata, and return the metadata storage identifier to the metadata control node.

[0057] The metadata control node then estimates the expected execution time of the target operation corresponding to the metadata control node. The expected execution time can be counted through historical execution time, or estimated through the current network status and load status of the metadata control node and each metadata storage node. For example, each time the metadata control node interacts with a metadata storage node to perform a certain operation, it can record the execution time from the time the metadata control node sends a request to the metadata storage node to start the interaction to the time the metadata control node returns a response to the terminal indicating that the operation is completed. When estimating the expected execution time, the metadata control node can determine the first metadata storage node corresponding to the metadata storage identifier, and obtain the recorded execution times of the target operation corresponding to the first metadata storage node, and determine the expected execution time based on the average value of each execution time.

[0058] The duration threshold can be set by those skilled in the art according to actual needs. The duration threshold can be a fixed duration, or it can be a duration calculated in real time. For example, the duration threshold can be set to be inversely proportional to the current load of the metadata control node, so that the metadata control node can terminate more first metadata operation requests more quickly when the load is high. In another implementation, the metadata control node can estimate the duration of sending the metadata storage identifier back to the terminal, allowing the terminal to interact with each metadata storage node on its own to complete the execution of the target operation (hereinafter referred to as the terminal execution duration), and use the terminal execution duration as the duration threshold. In this case, the metadata control node allows the terminal to interact with each metadata storage node on its own when the duration condition is met, which is conducive to improving the efficiency of the terminal in obtaining the target metadata.

[0059] If the expected execution time is greater than or equal to the time threshold, it means that it will take a long time for the metadata control node to execute the target operation by itself, that is, it will take a long time to complete the processing of the first metadata operation request. In this case, the metadata control node can send the metadata storage identifier to the terminal. After the metadata control node sends the metadata storage identifier to the terminal, it is deemed that the first metadata operation request has been processed.

[0060] After receiving the metadata storage identifier, the terminal can parse the metadata storage identifier to obtain the relevant information of the first metadata storage node. In one example, the terminal can further obtain the IP address (Internet Protocol) of the first metadata storage node from the relevant node of the data storage cluster (for example, the node storing the relevant information and the IP address correspondence table) according to the relevant information, and send the second metadata operation request to the first metadata storage node according to the IP address. In another example, when the terminal communicates with the node in the data storage cluster, it needs to be routed through the gateway node of the cluster. The terminal can carry the relevant information in the second metadata operation request, and first send the second metadata operation request to the gateway node, so that the gateway node can route the second metadata operation request to the corresponding first metadata storage node according to the relevant information. The second metadata operation request carries the metadata storage identifier and the operation identifier to indicate to the first metadata storage node which target metadata the terminal needs to operate on and which specific operation to perform.

[0061] The first metadata storage node may retrieve the target metadata based on the metadata storage identifier, and perform a target operation on the target metadata.

[0062] The data storage system provided by the embodiment of the present application enables the metadata control node to estimate the expected execution time of the target operation by the metadata control node after receiving the first metadata operation request, and when the expected execution time is greater than or equal to the time threshold, it is no longer responsible for executing the target operation, but after determining the metadata required by the user according to the user's request, it returns the identifier of the storage node where the metadata is located to the terminal, and the terminal initiates a request to the storage node to read the metadata. Therefore, the target operation that takes too long to be processed can be transferred to the terminal for self-processing, reducing the average time required for the control node to complete the request execution, thereby improving the efficiency of the control node in processing requests.

[0063] In one embodiment, the metadata control node is also used to determine the target metadata storage nodes involved in the target operation, and to estimate the expected response time of each target metadata storage node based on the load information and network information of each target metadata storage node; the metadata control node is also used to determine the expected execution time based on the expected response time of each target metadata storage node.

[0064] In an embodiment of the present application, the metadata control node may estimate the expected execution time of the target operation based on the expected response time of each target metadata storage node involved in the target operation.

[0065] The target metadata storage node is the metadata storage node that needs to execute at least one step in the target operation. For example, when the target operation is a read operation, the process of executing the target operation is that the metadata control node sends a metadata read request to the first metadata storage node, the first metadata storage node obtains the metadata, and returns the metadata to the metadata control node. Therefore, the target metadata storage node is the first metadata storage node. When the target operation is a move operation, the process of executing the target operation is that the metadata control node sends a metadata move request to the first metadata storage node, the first metadata storage node sends the metadata to the second metadata storage node, and after the second metadata storage node stores the metadata, it instructs the metadata control node to modify the metadata storage identifier. Therefore, the target metadata storage nodes are the first metadata storage node and the second metadata storage node.

[0066] In one embodiment, the metadata storage node is also used to periodically report load information and network information to the metadata control node, and after performing any operation, report the response time corresponding to the operation to the metadata control node; the metadata control node is also used to estimate the expected response time of each target metadata storage node based on the load information and network information of each target metadata storage node, and the load information, network information and response time reported by each target metadata storage node within the target statistical period.

[0067] In an embodiment of the present application, each metadata storage node may periodically report its current load information (including but not limited to CPU (Central Processing Unit) usage, memory usage, etc.) and network information (including but not limited to the available bandwidth of each metadata storage node, the latency and packet loss rate when communicating with other metadata storage nodes and metadata control nodes, etc.) to the metadata control node. The metadata control node estimates the expected response time of each target metadata storage node based on the load information and network information of each target metadata storage node.

[0068] The metadata control node can calculate a load score to characterize the load of the target metadata storage node based on the load information, and a higher score indicates a higher load. And a network score to characterize the quality of the network of the target metadata storage node can also be calculated based on the network information, and a higher score indicates a better network.

[0069] When executing the target operation, each metadata storage node can record the time it takes from receiving the request to sending the request to the next node, and report the above time as its own response time for the target operation to the metadata control node. The metadata control node can fit the relationship between the load score, network score and response time of each metadata storage node for each statistical period. The metadata control node can then estimate the estimated response time of the target metadata storage node based on the fitted relationship, the load score and network score of the target metadata storage node.

[0070] The metadata control node may add the estimated response times of the target metadata storage nodes to obtain the expected execution time, or the metadata control node may further add a preset expected error value to the result obtained by the above addition to obtain the expected execution time.

[0071] The data storage system provided in the embodiment of the present application estimates the expected response time of each target metadata storage node based on the load information, network information and response time required for executing an operation reported by each target metadata storage node within a cycle. The current response time performance of the target metadata storage node can be estimated based on the response time performance of the target metadata storage node in the past period of time, thereby improving the estimation accuracy of the expected response time.

[0072] In one embodiment, Figure 2 As shown, the system also includes at least one business data storage node; when the target operation is a read operation, the first metadata storage node is also used to send the target metadata corresponding to the metadata storage identifier to the terminal in response to the second metadata operation request; the terminal is also used to determine the first business data storage node where the business data corresponding to the target metadata is located based on the target metadata, and send a business data acquisition request to the first business data storage node, the business data acquisition request carrying the target metadata; the first business data storage node is also used to send the business data corresponding to the target metadata to the terminal in response to the business data acquisition request.

[0073] In the embodiment of the present application, when the target operation is a read operation, the first metadata storage node sends the target metadata corresponding to the metadata storage identifier to the terminal. If the ultimate purpose of the terminal reading the target metadata is to obtain the business data corresponding to the target metadata, after obtaining the target metadata, the terminal can determine on which business data storage node the business data corresponding to the target metadata is specifically stored according to the information recorded in the target metadata.

[0074] After the terminal determines the first business data storage node where the business data is located (since the business data may be fragmented and stored on multiple business data storage nodes, there may be one or more first business data storage nodes), it sends a business data acquisition request to the first business data storage node. The business data acquisition request may carry target metadata to indicate to the first business data storage node the business data that the terminal needs to acquire.

[0075] After the first business data storage node retrieves the business data according to the target metadata, it returns the business data to the terminal. If the business data is divided into multiple pieces and stored in different business data storage nodes, after the terminal receives all the business data pieces returned by the first business data storage node, it also needs to combine the business data pieces to obtain the complete business data.

[0076] In one embodiment, Figure 3 As shown, when the target operation is a read operation, the metadata control node is also used to send a metadata read request to the first metadata storage node when the expected execution time is less than a time threshold, and the metadata read request carries a metadata storage identifier; the first metadata storage node is also used to respond to the metadata read request and send the target metadata corresponding to the metadata storage identifier to the metadata control node; the metadata control node is also used to send the target metadata to the terminal.

[0077] In an embodiment of the present application, if the target operation is a read operation and the expected execution time is less than the time threshold, the metadata control node can obtain the target metadata from the first metadata storage node by itself and send the target metadata to the terminal.

[0078] After determining that the expected execution time is less than the time threshold, the metadata control node sends a metadata read request to the first metadata storage node, and carries the metadata storage identifier in the metadata read request. After the first metadata storage node retrieves the target metadata according to the metadata storage identifier, it can send the target metadata to the metadata control node. The metadata control node can then send the target metadata to the terminal to complete the processing of the first metadata operation request.

[0079] In one embodiment, Figure 4As shown, in the case where the target operation is a move operation, the second metadata operation request also carries the node information of the second metadata storage node, and the second metadata storage node is the destination node to which the target metadata needs to be moved; the first metadata storage node is also used to respond to the second metadata operation request and send the target metadata corresponding to the metadata storage identifier to the second metadata storage node; the second metadata storage node is also used to store the target metadata and send a metadata storage identifier modification request to the metadata control node; the metadata control node is also used to update the metadata storage identifier of the target metadata based on the metadata storage identifier modification request.

[0080] In an embodiment of the present application, the terminal may specify a second metadata storage node to which the target metadata needs to be moved. The node information of the second metadata storage node may be a unique identifier of the second metadata storage node, an IP address of the second metadata storage node, etc. The node information of the second metadata storage node may be obtained by the terminal interacting with the metadata control node before sending the first metadata operation request.

[0081] If the target operation is a move operation, the terminal also needs to carry the node information of the second metadata storage node in the second metadata operation request. After receiving the second metadata operation request, the first metadata storage node obtains the target metadata and determines the IP address of the second metadata storage node based on the node information of the second metadata storage node. The first metadata storage node then sends the target metadata to the IP address of the second metadata storage node.

[0082] After receiving the target metadata, the second metadata storage node performs storage processing on the target metadata. After the storage processing is completed, since the storage location of the target metadata has changed, the metadata storage identifier of the target metadata needs to be updated.

[0083] After the second metadata storage node generates the metadata storage identifier by itself, it can carry the metadata storage identifier in the metadata storage identifier modification request and send it to the metadata control node for updating. For example, the second metadata storage node can assign a random number that is unique in the second metadata storage node to the target metadata, and then generate the part of the metadata storage identifier related to the second metadata storage node based on its own information, and concatenate the random number with the part to obtain the metadata storage identifier.

[0084] Alternatively, the second metadata storage node may also indicate in the metadata storage identifier modification request that the metadata control node needs to generate a new metadata storage identifier. The metadata control node may generate a metadata storage identifier that is different from all current metadata in the data storage cluster and can characterize the second metadata storage node based on the information of the second metadata storage node. The metadata control node may then update the data table in the metadata storage identifier library.

[0085] In one embodiment, Figure 5 As shown, the terminal is also used to send a free metadata storage node acquisition request to the metadata control node, and the free metadata storage node acquisition request carries read requirement information for the target metadata, and the read requirement information at least includes the target response time; the metadata control node is also used to determine the candidate metadata storage nodes according to the read requirement information, and send the node information of each candidate metadata storage node to the terminal, and the node information includes at least one of the remaining storage space, expected response time, and location information of the candidate metadata storage node; the terminal is also used to determine the second metadata storage node from each candidate metadata storage node according to each node information.

[0086] In an embodiment of the present application, when the terminal needs to move the target metadata, it may request the metadata control node to return to the terminal all current candidate metadata storage nodes that meet the terminal's requirements, so that the terminal can determine the second metadata storage node to which the target metadata is to be moved from the candidate metadata storage nodes.

[0087] The terminal can indicate its own needs to the metadata control node through the read demand information. Since the terminal's read demand for metadata is generally related to the speed at which the data storage cluster returns metadata to the terminal, the terminal can carry at least the target response time required by the terminal in the read demand information. The target response time refers to the time it takes for the metadata storage node to retrieve the metadata required by the terminal from the various metadata stored in itself, that is, when the target operation is a read operation, the time from the metadata storage node receiving the second metadata operation request from the terminal to the metadata storage node sending the metadata to the terminal. The read demand information may also include the location of the candidate metadata storage node required by the terminal, the remaining storage space, etc.

[0088] The metadata control node can determine the candidate metadata storage nodes that meet the requirements from the metadata storage nodes according to the read demand information. For example, the metadata control node can determine the expected response time of each metadata storage node according to the load information and network information of each metadata storage node, and use the metadata storage node whose expected response time is less than or equal to the target response time as the candidate metadata storage node. The specific calculation method of the expected response time can refer to the description of the above embodiment, and the embodiment of this application will not be repeated.

[0089] If the metadata control node does not find any candidate metadata storage node, the metadata control node returns a message to the terminal indicating that the candidate metadata storage node retrieval failed. If the metadata control node finds at least one candidate metadata storage node, the metadata control node sends the node information of each candidate metadata storage node to the terminal. The node information includes at least one of the remaining storage space, expected response time, and location information of the candidate metadata storage node, so that the terminal can select the best candidate metadata storage node as the second metadata storage node according to the node information. Moving the metadata to the best second metadata storage node can improve the efficiency of the subsequent terminal in acquiring metadata.

[0090] The terminal can then select a second metadata storage node according to actual needs. For example, when the terminal needs the fastest response efficiency, the candidate metadata storage node with the shortest expected response time can be selected as the second metadata storage node. When the fastest transmission speed is required, the candidate metadata storage node that is closer to itself can be selected as the second metadata storage node based on the location information. Since the more data stored in the metadata storage node, the more requests the metadata storage node may need to process simultaneously, and the response time may be unstable, so when the most stable response time performance is required, the candidate metadata storage node with the most remaining storage space can be selected as the second metadata storage node. Alternatively, the terminal can also calculate a score for each candidate metadata storage node based on the above information, and select the candidate metadata storage node with the highest score as the second metadata storage node. This embodiment of the present application does not specifically limit this.

[0091] After the terminal determines the second metadata storage node, it may execute the process of moving the target metadata in the aforementioned embodiment.

[0092] The data storage system provided by the embodiment of the present application enables the metadata control node to determine the candidate metadata storage node that meets the requirements based on the read demand information sent by the terminal and return it to the terminal, so that the terminal can select the second metadata storage node that meets the requirements from each candidate metadata storage node as the node to which the target metadata needs to be moved. The terminal can be allowed to freely move the storage location of the metadata in the data storage cluster according to the requirements.

[0093] In one embodiment, Figure 6As shown, in the case where the target operation is a deletion operation, the first metadata storage node is also used to respond to the second metadata operation request, delete the target metadata corresponding to the metadata storage identifier, and send a metadata storage identifier deletion request to the metadata control node; the metadata control node is also used to delete the metadata storage identifier of the target metadata based on the metadata storage identifier deletion request.

[0094] In the embodiment of the present application, if the terminal needs to delete the target metadata, the first metadata storage node needs to send a metadata storage identifier deletion request to the metadata control node in addition to deleting the target metadata itself from its own storage device, so that the metadata control node deletes the metadata storage identifier of the target metadata. The metadata storage identifier deletion request may carry feature information of the target metadata, and may also carry the metadata storage identifier itself.

[0095] After receiving the metadata storage identifier deletion request, the metadata control node searches for the record of the metadata storage identifier in the metadata storage identifier library according to the characteristic information of the target metadata or the metadata storage identifier itself, and deletes the metadata storage identifier.

[0096] In one embodiment, Figure 7 As shown, a data storage method is provided, which is applied to a metadata control node and includes:

[0097] Step 702: receiving a first metadata operation request sent by a terminal, wherein the first metadata operation request carries feature information of the metadata and an operation identifier corresponding to a target operation;

[0098] Step 704, determine the metadata storage identifier based on the characteristic information, and estimate the expected execution time of the target operation corresponding to the metadata control node. When the expected execution time is greater than or equal to the time threshold, send the metadata storage identifier to the terminal. The metadata storage identifier is used to indicate the metadata storage node where the metadata is located.

[0099] In an embodiment of the present application, when the terminal needs to operate the metadata, it sends a first metadata operation request to the metadata control node. After receiving the first metadata operation request, the metadata control node determines the metadata storage identifier based on the characteristic information. The metadata control node further determines the expected execution time of the target operation.

[0100] If the expected execution time is greater than or equal to the time threshold, the metadata control node sends the metadata storage identifier to the terminal to speed up the efficiency of the metadata control node in processing the first metadata operation request. If the expected execution time is less than the time threshold, the metadata control node determines the first metadata storage node according to the metadata storage identifier, obtains the target metadata from the first metadata storage node, and returns the target metadata to the terminal. Since the communication rate between the metadata control node and the first metadata storage node is generally higher than the communication rate between the terminal and the first metadata storage node, the speed at which the terminal obtains the target metadata can be increased.

[0101] Among them, the way in which the metadata control node determines the metadata storage identifier and the expected execution time can be found in the aforementioned embodiment, and the embodiments of the present application will not be repeated here.

[0102] The data storage method provided by the embodiment of the present application enables the metadata control node to estimate the expected execution time of the target operation by the metadata control node after receiving the first metadata operation request, and when the expected execution time is greater than or equal to the time threshold, it is no longer responsible for executing the target operation, but after determining the metadata required by the user according to the user's request, it returns the identifier of the storage node where the metadata is located to the terminal, and the terminal initiates a request to the storage node to read the metadata. Therefore, the target operation that takes too long to be processed can be transferred to the terminal for self-processing, reducing the average time required for the control node to complete the request execution, thereby improving the efficiency of the control node in processing requests.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, the embodiment of the present application also provides a data storage device for implementing the data storage method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data storage device embodiments provided below can refer to the limitations on the data storage method above, and will not be repeated here.

[0105] In one embodiment, Figure 8 As shown, a data storage device 800 is provided, including: a receiving module 802, a determining module 804, wherein:

[0106] The receiving module 802 is configured to receive a first metadata operation request sent by a terminal, wherein the first metadata operation request carries feature information of the metadata and an operation identifier corresponding to a target operation;

[0107] The determination module 804 is used to determine the metadata storage identifier based on the characteristic information, and estimate the expected execution time of the metadata control node corresponding to the target operation. When the expected execution time is greater than or equal to the time threshold, the metadata storage identifier is sent to the terminal. The metadata storage identifier is used to indicate the metadata storage node where the metadata is located.

[0108] The data storage device provided by the embodiment of the present application enables the metadata control node to estimate the expected execution time of the target operation by the metadata control node after receiving the first metadata operation request, and when the expected execution time is greater than or equal to the time threshold, it is no longer responsible for executing the target operation, but after determining the metadata required by the user according to the user's request, it returns the identifier of the storage node where the metadata is located to the terminal, and the terminal initiates a request to the storage node to read the metadata. Therefore, the target operation that takes too long to be processed can be transferred to the terminal for self-processing, reducing the average time required for the control node to complete the request execution, thereby improving the efficiency of the control node in processing requests.

[0109] Each module in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0110] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data storage method is implemented.

[0111] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0113] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0114] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0115] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0116] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0117] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data storage system, characterized in that: The system comprises a terminal, at least one metadata storage node and a metadata control node, wherein: The terminal is used to send a first metadata operation request to the metadata control node, where the first metadata operation request carries feature information of the metadata and an operation identifier corresponding to a target operation; The metadata control node is used to determine the metadata storage identifier based on the characteristic information, and estimate the expected execution time of the metadata control node corresponding to the target operation, and send the metadata storage identifier to the terminal when the expected execution time is greater than or equal to a time threshold; The terminal is further used to determine, based on the metadata storage identifier, a first metadata storage node corresponding to the metadata storage identifier, and send a second metadata operation request to the first metadata storage node, wherein the second metadata operation request carries the metadata storage identifier and the operation identifier; The first metadata storage node is further configured to execute the target operation on the target metadata corresponding to the metadata storage identifier in response to the second metadata operation request.

2. The data storage system according to claim 1, characterized in that: The metadata control node is further used to determine each target metadata storage node involved in the target operation, and estimate the expected response time of each target metadata storage node according to the load information and network information of each target metadata storage node; The metadata control node is further used to determine the expected execution duration according to the expected response duration of each target metadata storage node.

3. The data storage system according to claim 2, characterized in that: The metadata storage node is further used to periodically report load information and network information to the metadata control node, and after performing any operation, report the response time corresponding to the operation to the metadata control node; The metadata control node is also used to estimate the expected response time of each target metadata storage node based on the load information and network information of each target metadata storage node, as well as the load information, network information and response time reported by each target metadata storage node during a target statistical period.

4. The data storage system according to claim 1, characterized in that: The system also includes at least one business data storage node; In the case where the target operation is a read operation, the first metadata storage node is further configured to send the target metadata corresponding to the metadata storage identifier to the terminal in response to the second metadata operation request; The terminal is further used to determine, based on the target metadata, a first business data storage node where the business data corresponding to the target metadata is located, and send a business data acquisition request to the first business data storage node, wherein the business data acquisition request carries the target metadata; The first business data storage node is further configured to send the business data corresponding to the target metadata to the terminal in response to the business data acquisition request.

5. The data storage system according to claim 4, characterized in that: When the target operation is the read operation, the metadata control node is further configured to send a metadata read request to the first metadata storage node when the expected execution time is less than a time threshold, wherein the metadata read request carries the metadata storage identifier; The first metadata storage node is further configured to send the target metadata corresponding to the metadata storage identifier to the metadata control node in response to the metadata read request; The metadata control node is further used to send the target metadata to the terminal.

6. The data storage system according to claim 1, characterized in that: In the case where the target operation is a move operation, the second metadata operation request further carries node information of a second metadata storage node, and the second metadata storage node is a destination node to which the target metadata needs to be moved; The first metadata storage node is further configured to send the target metadata corresponding to the metadata storage identifier to the second metadata storage node in response to the second metadata operation request; The second metadata storage node is further used to store the target metadata and send a metadata storage identifier modification request to the metadata control node; The metadata control node is further used to update the metadata storage identifier of the target metadata based on the metadata storage identifier modification request.

7. The data storage system according to claim 6, characterized in that: The terminal is further configured to send an idle metadata storage node acquisition request to the metadata control node, wherein the idle metadata storage node acquisition request carries read requirement information for the target metadata, and the read requirement information at least includes a target response time; The metadata control node is further used to determine candidate metadata storage nodes according to the read demand information, and send node information of each candidate metadata storage node to the terminal, wherein the node information includes at least one of remaining storage space, expected response time, and location information of the candidate metadata storage node; The terminal is further configured to determine the second metadata storage node from among the candidate metadata storage nodes according to the node information.

8. The data storage system according to claim 7, characterized in that: The metadata control node is also used to determine the expected response time of each metadata storage node based on the load information and network information of each metadata storage node, and use the metadata storage node whose expected response time is less than or equal to the target response time as the candidate metadata storage node.

9. The data storage system according to claim 1, characterized in that: In the case where the target operation is a deletion operation, the first metadata storage node is further configured to delete the target metadata corresponding to the metadata storage identifier in response to the second metadata operation request, and send a metadata storage identifier deletion request to the metadata control node; The metadata control node is further used to delete the metadata storage identifier of the target metadata based on the metadata storage identifier deletion request.

10. A data storage method, characterized in that: The method is applied to a metadata control node, comprising: Receiving a first metadata operation request sent by a terminal, wherein the first metadata operation request carries characteristic information of the metadata and an operation identifier corresponding to a target operation; A metadata storage identifier is determined based on the characteristic information, and an expected execution duration of the metadata control node corresponding to the target operation is estimated. When the expected execution duration is greater than or equal to a duration threshold, the metadata storage identifier is sent to the terminal. The metadata storage identifier is used to indicate the metadata storage node where the metadata is located.

11. A data storage device, characterized in that: The device is applied to a metadata control node, and comprises: A receiving module, configured to receive a first metadata operation request sent by a terminal, wherein the first metadata operation request carries feature information of the metadata and an operation identifier corresponding to a target operation; A determination module is used to determine a metadata storage identifier based on the characteristic information, and estimate the expected execution time of the metadata control node corresponding to the target operation. When the expected execution time is greater than or equal to a time threshold, the metadata storage identifier is sent to the terminal, and the metadata storage identifier is used to indicate the metadata storage node where the metadata is located.

Images