Distributed storage method, device and system

Abstract

The invention relates to a distributed storage method, device and system. The method is applied to a storage node and comprises the following steps: after a data reading request initiated by an application client is received, acquiring cluster real-time information of the distributed storage system; based on the cluster real-time information, determining a target data acquisition mode corresponding to the data reading request; and returning data information corresponding to the data reading request to the application client according to the target data acquisition mode. By adopting the method, the data reading flexibility can be improved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a distributed storage method, device and system. Background Art

[0002] A distributed storage system is an architecture that stores data in multiple physical locations, connecting different storage nodes through a network to achieve data redundancy, fault tolerance, and high availability. In this system, data is divided into multiple small blocks and stored in different nodes. The nodes work together through a consistency protocol to ensure the correct access and update of data, thereby ensuring data consistency and reliability.

[0003] However, in traditional technologies, when using distributed storage systems to access data, there is often a problem of poor data reading flexibility. Summary of the invention

[0004] Based on this, it is necessary to provide a distributed storage method, device and system that can improve data reading flexibility in response to the above technical problems.

[0005] In a first aspect, the present application provides a distributed storage method, applied to a storage node, comprising:

[0006] After receiving the data read request initiated by the application client, obtain the real-time cluster information of the distributed storage system;

[0007] Based on the real-time information of the cluster, determine the target data acquisition method corresponding to the read data request;

[0008] According to the target data acquisition method, the data information corresponding to the read data request is returned to the application client.

[0009] In one embodiment, the cluster real-time information includes at least input and output data volume information and cluster bandwidth information. The method of determining the target data acquisition method corresponding to the read data request based on the cluster real-time information includes:

[0010] When the input / output data volume information is less than the first preset threshold, or when the input / output data volume information is greater than the first preset threshold and the cluster bandwidth information is less than the second preset threshold, determining the target data acquisition mode as the transit data acquisition mode;

[0011] When the input and output data volume information is greater than a first preset threshold and the cluster bandwidth information is greater than a second preset threshold, it is determined that the target data acquisition mode is a direct read data acquisition mode.

[0012] In one embodiment, the storage node includes an index server and a data block server; according to the target data acquisition method, the storage node returns data information corresponding to the data read request to the application client, including:

[0013] When the target data acquisition mode is a direct-read data acquisition mode, the index server returns metadata to the application client to instruct the application client to acquire data information corresponding to the read data request from the data block server.

[0014] In one embodiment, the metadata includes location information where the data information is stored in the data block server; the index server returns the metadata to the application client to instruct the application client to obtain the data information corresponding to the read data request from the data block server, including:

[0015] Acquire location information from a location mapping relationship table pre-stored in the index server based on a read data request;

[0016] The location information is sent to the application client, so that the application client obtains data information from the data block server based on the location information.

[0017] In one embodiment, the storage node includes an index server and a data block server; according to the target data acquisition method, the storage node returns data information corresponding to the data read request to the application client, including:

[0018] When the target data acquisition mode is the transit data acquisition mode, the index server obtains data information from the data block server and returns the data information to the application client.

[0019] In one embodiment, the method further comprises:

[0020] Based on the recorded information of the historical stage, the execution results of multiple candidate data acquisition methods are predicted to obtain the success probability corresponding to each candidate data acquisition method;

[0021] Correcting the target data acquisition method based on the success probability corresponding to each candidate data acquisition method to obtain a corrected target data acquisition method;

[0022] Correspondingly, according to the target data acquisition method, the data information corresponding to the read data request is returned to the application client, including:

[0023] According to the modified target data acquisition method, data information corresponding to the read data request is returned to the application client.

[0024] In one embodiment, the method further comprises:

[0025] The execution result of the target data acquisition method is recorded to obtain record information.

[0026] In a second aspect, the present application also provides a distributed storage device, including:

[0027] The acquisition module is used to obtain the real-time cluster information of the distributed storage system after receiving the data read request initiated by the application client;

[0028] A determination module, used to determine a target data acquisition method corresponding to a read data request based on cluster real-time information;

[0029] The return module is used to return the data information corresponding to the read data request to the application client according to the target data acquisition method.

[0030] In a third aspect, the present application further provides a distributed storage system, the system comprising a computing node and a storage node; an application client is deployed on the computing node;

[0031] Application client, used to send read data requests to storage nodes;

[0032] The storage node is used to obtain the real-time cluster information of the distributed storage system after receiving the read data request initiated by the application client; based on the real-time cluster information, determine the target data acquisition method corresponding to the read data request; and return the data information corresponding to the read data request to the application client according to the target data acquisition method.

[0033] In a fourth aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] After receiving the data read request initiated by the application client, obtain the real-time cluster information of the distributed storage system;

[0035] Based on the real-time information of the cluster, determine the target data acquisition method corresponding to the read data request;

[0036] According to the target data acquisition method, the data information corresponding to the read data request is returned to the application client.

[0037] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0038] After receiving the data read request initiated by the application client, obtain the real-time cluster information of the distributed storage system;

[0039] Based on the real-time information of the cluster, determine the target data acquisition method corresponding to the read data request;

[0040] According to the target data acquisition method, the data information corresponding to the read data request is returned to the application client.

[0041] In a sixth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0042] After receiving the data read request initiated by the application client, obtain the real-time cluster information of the distributed storage system;

[0043] Based on the real-time information of the cluster, determine the target data acquisition method corresponding to the read data request;

[0044] According to the target data acquisition method, the data information corresponding to the read data request is returned to the application client.

[0045] In the above-mentioned distributed storage method, device and system, after receiving the data read request initiated by the application client, the storage node obtains the real-time cluster information of the distributed storage system; based on the real-time cluster information, the target data acquisition method corresponding to the data read request is determined; according to the target data acquisition method, the data information corresponding to the data read request is returned to the application client; in this method, by obtaining the real-time cluster information of the distributed storage system, the load of the cluster can be understood in real time. Under different load conditions, the most suitable target data acquisition method can be flexibly determined according to the real-time cluster information. This mechanism of dynamically adjusting the data acquisition method according to the real-time cluster information greatly improves the adaptability of data reading to different distributed storage system environments, thereby enhancing flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is an internal structure diagram of a distributed storage system in one embodiment;

[0048] Figure 2 A schematic diagram of a distributed storage method in one embodiment;

[0049] Figure 3 is a flow chart of a distributed storage method in another embodiment;

[0050] Figure 4 is a flow chart of a distributed storage method in another embodiment;

[0051] Figure 5 is a schematic diagram of a distributed storage method in another embodiment;

[0052] Figure 6 is a structural block diagram of a distributed storage device in one embodiment;

[0053] Figure 7 is an internal structure diagram of a distributed storage system in another embodiment;

[0054] Figure 8 is an internal structure diagram of a computer device in one embodiment;

[0055] Description of reference numerals:

[0056] Computing node 10, application client 101;

[0057] Storage node 11; index server 111;

[0058] Data block server 112. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The distributed storage method provided in the embodiment of the present application can be applied to Figure 1 The distributed storage system shown in the figure includes a computing node 10 and a storage node 11; the computing node 10 is used to send a read data request to the storage node 11; the storage node 11 is used to obtain the cluster real-time information of the distributed storage system after receiving the read data request initiated by the computing node 10; based on the cluster real-time information, determine the target data acquisition method corresponding to the read data request; according to the target data acquisition method, return the data information corresponding to the read data request to the computing node 10. Among them, the storage node 11 is a basic component unit in the distributed storage system. It is a physical device or logical entity that actually stores data, which can usually be a server or storage device. In the distributed storage system, multiple storage nodes 11 work together to store and manage data. For example, in a large data center, there may be hundreds of storage nodes 11, which are connected together through a network to form a huge storage resource pool. Each storage node 11 has a certain storage capacity, computing power and network communication capability, and is responsible for storing data blocks, processing data read and write requests, and interacting and collaborating with other nodes.

[0061] In an exemplary embodiment, Figure 2As shown, a distributed storage method is provided, which is applied to Figure 1 The storage node in FIG. 1 is taken as an example to illustrate, including the following steps 201 to 203. Among them:

[0062] Step 201: After receiving a data read request initiated by an application client, obtain cluster real-time information of a distributed storage system.

[0063] Among them, the distributed storage system is a storage architecture that stores data in multiple storage nodes. It uses distributed algorithms and network communication technologies to achieve data storage, management, retrieval and access. Compared with traditional centralized storage systems, distributed storage systems have better scalability, reliability and performance.

[0064] The application client App Client refers to an application running on a user device (such as a computer, mobile phone, tablet computer, etc.), which communicates with the distributed storage system through a network and sends various operation requests to the storage node 11, including data read requests.

[0065] Cluster real-time information refers to the various states and attribute information of the cluster in the distributed storage system at the current moment.

[0066] In the embodiment of the present application, when the storage node receives a data read request initiated by the App Client, it first obtains the real-time cluster information of the distributed storage system. This includes but is not limited to the following aspects:

[0067] (1) Storage node status information: Check the node's own hardware resource usage, such as CPU usage, memory usage, disk I / O load, etc., to determine whether it has enough resources to process data read requests. For example, if the CPU usage is already very high, it may affect the data reading speed.

[0068] (2) Other storage node status information: Communicate with other storage nodes in the cluster to obtain their operating status, including whether the node is online, whether it is busy, etc. For example, if a node is performing a large amount of data writing operations, its reading performance may be affected, so you need to consider avoiding this node when selecting a data acquisition node.

[0069] (3) Network connection status information: Evaluate the quality of the network connection between the storage node and other nodes, such as network bandwidth utilization, network latency, packet loss rate, etc. The network connection status will affect the data transmission speed. If the network latency is high, it may take a long time to obtain data from the remote node.

[0070] (4) Data distribution information: Determine the storage location of the required data in the cluster, including the distribution of data blocks on each node, the number of data copies and their distribution locations, etc. Understanding data distribution helps to select the optimal data acquisition path. For example, if the data has a copy on the local node and the local node is in good condition, it may be faster to obtain data from the local node first.

[0071] Step 202: Determine a target data acquisition method corresponding to a data read request based on cluster real-time information.

[0072] The target data acquisition method is a specific method or strategy for acquiring data requested by the application client, which is determined based on the real-time information of the cluster.

[0073] In the embodiment of the present application, first check whether the data requested by the application client is stored locally. If a complete and latest copy exists locally, and the local node resources are sufficient and the network condition is good, then the target data acquisition method may be to read the data directly from the local, which can reduce data transmission time and network overhead.

[0074] If there is no data locally or the local storage node is overloaded, and you need to obtain data from other storage nodes, you must consider the load of other nodes. Select a storage node with a lighter load to obtain data to ensure efficient data reading. For example, if the CPU usage of storage node A is 30% and the CPU usage of storage node B is 80%, when other conditions are the same, storage node A is preferred for data acquisition.

[0075] In some embodiments, when multiple copies of data are distributed on different storage nodes, the best copy is selected for reading based on network conditions and node performance. If the network connection of the storage node where a copy is located is stable and has a high bandwidth, and the storage node is lightly loaded, then the copy will become the preferred target.

[0076] In some embodiments, if the data is distributed on multiple storage nodes and the network and storage node resources allow, a parallel data reading method can be used. Data is requested from multiple suitable storage nodes at the same time, and data merging and integration processing is performed on the storage node side to increase the speed of data reading. For example, for data reading of a large file, data can be obtained in parallel from multiple storage nodes storing different parts of the file at the same time, and then the data can be combined into complete file data.

[0077] Step 203: Return data information corresponding to the data read request to the application client according to the target data acquisition method.

[0078] The data information refers to the specific data content corresponding to the data read request of the application client, which can be various types of data, such as text content in a text file, image data in a picture file, video stream data in a video file, records in a database, etc.

[0079] In an embodiment of the present application, data information is read from a corresponding storage node (local or remote) according to a determined acquisition method. During the reading process, some pre-processing operations need to be performed on the data information, such as data decompression (if the data is stored in a compressed form), data format conversion (converting the data into a format that the application client can recognize), etc. For example, if the stored data information is a compressed image file, it is necessary to first perform a decompression operation when reading, and then convert the decompressed image data into a format that can be displayed by the application client, such as JPEG format.

[0080] Before returning the data to the application client, the read data is checked for integrity and accuracy. Check whether the data is complete and whether there is any data loss or damage. The integrity of the data can be verified by calculating the checksum of the data or comparing it with a pre-stored checksum.

[0081] The verified data information is transmitted back to the application client through the network. The data information is encapsulated according to the communication protocol between the application client and the storage node.

[0082] In the above-mentioned distributed storage method, after receiving the read data request initiated by the application client, the storage node obtains the real-time cluster information of the distributed storage system; based on the real-time cluster information, the target data acquisition method corresponding to the read data request is determined; according to the target data acquisition method, the data information corresponding to the read data request is returned to the application client; in this method, by obtaining the real-time cluster information of the distributed storage system, the load of the cluster can be understood in real time. Under different load conditions, the most suitable target data acquisition method can be flexibly determined according to the real-time cluster information. This mechanism of dynamically adjusting the data acquisition method according to the real-time cluster information greatly improves the adaptability of data reading to different distributed storage system environments, thereby enhancing flexibility.

[0083] In an exemplary embodiment, the cluster real-time information includes at least input and output data volume information and cluster bandwidth information. On this basis, the above “determining the target data acquisition method corresponding to the read data request based on the cluster real-time information” includes:

[0084] In case 1, when the input and output data volume information is less than the first preset threshold, or when the input and output data volume information is greater than the first preset threshold and the cluster bandwidth information is less than the second preset threshold, the target data acquisition mode is determined to be the transit data acquisition mode.

[0085] Among them, the input and output data volume information mainly refers to the amount of data input (such as write operations) and data output (such as read operations) currently being performed in the distributed storage system. It can be expressed in many forms, such as the number of bytes per unit time, the number of data blocks, etc. If the input and output data volume is too large, it means that the cluster is under high work pressure, which will affect the speed and efficiency of data reading and writing.

[0086] Cluster bandwidth information refers to data related to the data transmission capacity between nodes within the cluster and between the cluster and the external network in a distributed storage system. It is usually measured in terms of the amount of data that can be transmitted per unit time, such as the number of bytes transmitted per second. The cluster bandwidth determines the upper limit of the speed at which data can be transmitted in the cluster. If the cluster bandwidth is low, even if the storage nodes have sufficient processing power, data transmission will be limited, resulting in increased latency in data reading or writing.

[0087] The first preset threshold is a pre-set value used as a limit to determine whether the input and output data volume is within a normal or acceptable range. The setting of the first preset threshold is usually determined based on factors such as the hardware configuration, performance requirements, and past operating experience of the distributed storage system. When the input and output data volume information is less than this threshold, it means that the cluster load is relatively low and the impact of data read and write operations on the system is small; when the input and output data volume information is greater than this threshold, it may indicate that the cluster is facing a high load pressure and needs to adopt different data acquisition strategies to ensure the stability and efficient operation of the system.

[0088] The second preset threshold is also a pre-set value, which is used to determine whether the cluster bandwidth can meet the current data processing requirements. The determination of this threshold also needs to consider factors such as the overall architecture of the distributed storage system and the performance of network devices. When the cluster bandwidth information is greater than this threshold, it means that the network transmission capacity is strong and can support faster data transmission. At this time, the direct reading data acquisition method can be used to directly obtain data from the data source; when the cluster bandwidth information is less than this threshold, the network may become a bottleneck for data transmission, and it is necessary to adopt a transit data acquisition method to reduce the dependence on bandwidth through intermediate nodes or other optimization strategies to improve the efficiency of data acquisition.

[0089] The transit data acquisition method refers to a method of acquiring data through one or more intermediate nodes instead of directly acquiring data from the data source under specific cluster real-time information conditions (such as when the input and output data volume is large and the cluster bandwidth is low). The purpose of this method is to alleviate the load pressure of the cluster and avoid excessive impact on the network bandwidth caused by direct transmission of large amounts of data. At the same time, the efficiency of data acquisition can also be improved by using the cache and other functions of the intermediate nodes.

[0090] In case 2, when the input and output data volume information is greater than the first preset threshold and the cluster bandwidth information is greater than the second preset threshold, it is determined that the target data acquisition mode is the direct reading data acquisition mode.

[0091] Among them, the direct reading data acquisition method refers to the method in which the storage node directly reads data from the data source (such as the original node storing the data) and returns it to the application client when the real-time information of the cluster meets certain conditions (such as the input and output data volume is relatively small and the cluster bandwidth is sufficient).

[0092] In an embodiment of the present application, the storage node first needs to obtain input and output data volume information and cluster bandwidth information of the distributed storage system.

[0093] Compare the acquired input and output data volume information with the pre-set first preset threshold. If the input and output data volume information is less than the first preset threshold, it means that the current cluster load is relatively low, and the data read and write operations have little impact on the system. At this time, proceed to the next step to determine the relationship between the cluster bandwidth information and the second preset threshold; if the input and output data volume information is greater than the first preset threshold, proceed directly to the next step to determine the relationship between the cluster bandwidth information and the second preset threshold to determine whether the conditions for the direct read data acquisition method are met. For example, for a distributed storage system with a first preset threshold set to 100MB of data per second, if the currently detected input and output data volume is 80MB per second, it is less than the threshold, and the bandwidth situation continues to be determined; if the input and output data volume is 120MB per second, it is greater than the threshold, and the bandwidth determination step is directly entered.

[0094] Determine the relationship between the cluster bandwidth information and the second preset threshold, and when the input and output data volume information is greater than the first preset threshold, compare the cluster bandwidth information with the second preset threshold. If the cluster bandwidth information is less than the second preset threshold, it means that the network transmission capacity may become a bottleneck for data transmission. Even if the storage node has sufficient processing power, direct transmission of a large amount of data may cause increased delays. At this time, the target data acquisition method is determined to be the transit data acquisition method; if the cluster bandwidth information is greater than the second preset threshold, it means that the network transmission capacity is strong and can support faster data transmission, and the target data acquisition method is determined to be the direct reading data acquisition method. For example, if the second preset threshold is set to a bandwidth of 1Gbps per second, when it is detected that the current cluster bandwidth is 800Mbps per second and the input and output data volume is large, the transit data acquisition method is selected; if the cluster bandwidth is 1.2Gbps per second and the input and output data volume is large, the direct reading data acquisition method is selected.

[0095] When the input and output data volume information is less than the first preset threshold, the relationship between the cluster bandwidth information and the second preset threshold is determined again. If the cluster bandwidth information is less than the second preset threshold, the target data acquisition method is also determined to be the transit data acquisition method to avoid the problem of slow data transmission due to insufficient bandwidth; if the cluster bandwidth information is greater than the second preset threshold, the target data acquisition method is determined to be the direct reading data acquisition method. At this time, due to the low cluster load and sufficient bandwidth, directly reading data from the data source can provide the fastest data acquisition speed. For example, when the input and output data volume is small (such as 50MB per second), if the cluster bandwidth is 500Mbps per second (less than the second preset threshold), the transit data acquisition method is adopted; if the cluster bandwidth is 1.5Gbps per second (greater than the second preset threshold), the direct reading data acquisition method is adopted.

[0096] Through the above embodiments, according to the input and output data volume information and cluster bandwidth information in the cluster real-time information, as well as the pre-set threshold, the target data acquisition method corresponding to the read data request can be accurately determined, thereby optimizing the data reading process and improving the performance and efficiency of the distributed storage system.

[0097] In an exemplary embodiment, the storage node includes an index server and a data block server. On this basis, the above “returning data information corresponding to the read data request to the application client according to the target data acquisition method” includes:

[0098] When the target data acquisition mode is a direct-read data acquisition mode, the index server returns metadata to the application client to instruct the application client to acquire data information corresponding to the read data request from the data block server.

[0099] The index server is responsible for managing and maintaining the index information of the data. By recording metadata such as the storage location and structure of the data, the system can quickly locate and obtain the required data.

[0100] Metadata describes the basic properties, characteristics, and storage location of data. For example, metadata may include the data's file name, file size, creation time, modification time, owner, and the data block server on which it is stored. When the index server returns metadata to the application client, the application client can learn about the required data based on the metadata, and thus know where to obtain the actual data content.

[0101] The data block server is the server node that actually stores data blocks in the distributed storage system. It is responsible for storing and managing data blocks and providing data block read and write services based on requests.

[0102] In the embodiment of the present application, when the target data acquisition method is determined to be a direct read data acquisition method, the index server first receives a data read request from the application client. Then, based on the key information contained in the request (such as data identification, file name, data characteristics, etc.), a query operation is performed in the metadata database it manages. For example, if the application client requests to read the data of a specific file, the index server will search for records related to the file name in the metadata and obtain metadata such as the location information of the file data stored on the data block server and the data block distribution.

[0103] The index server organizes and encapsulates the queried metadata and returns it to the application client in a suitable format. After receiving the metadata returned by the index server, the application client parses it and extracts key information such as the address and storage path of the data block server. Then, based on this information, a network connection is established with the corresponding data block server.

[0104] After the connection is established, the application client obtains data blocks from each data block server in the order indicated in the metadata. It sends a data read request to the data block server. After receiving the request, the data block server reads the corresponding data block from the local storage according to the data block number and other information in the request, and returns it to the application client. For example, the application client first obtains data block 1 from server A, then obtains data block 2 from server B, and finally obtains data block 3 from server C. After receiving the data blocks, the application client will combine and integrate them in order, restore them to the data information corresponding to the complete read data request, and finally complete the data read operation.

[0105] In an exemplary embodiment, the metadata includes location information of the data information stored in the data block server, based on which, Figure 3As shown, the above “the index server returns metadata to the application client to instruct the application client to obtain data information corresponding to the read data request from the data block server” includes:

[0106] Step 301: acquiring location information from a location mapping relationship table pre-stored in an index server based on a data read request.

[0107] The location mapping relationship table is a data structure stored in the index server, which is used to record the correspondence between data information and the storage location in the data block server. Each record in the table associates a specific data identifier (such as a file name, data block number, etc.) with the storage location information of the data in the data block server (such as the server address, storage path, offset of the data block in the server, etc.). By querying the location mapping relationship table, the index server can quickly find the storage location of the data information related to the read data request.

[0108] In the embodiment of the present application, the index server first receives a data read request from the application client, and then parses the request to extract key information that can be used to locate the data, such as the file name, file identifier, data type, or specific keywords related to the data.

[0109] The parsed key information is used as an index to perform a search operation in the location mapping relationship table pre-stored in the index server. Detailed location information is extracted from the found location information record, and is sorted and formatted so that it can be easily understood and used by the application client.

[0110] Step 302: Send the location information to the application client, so that the application client obtains data information from the data block server based on the location information.

[0111] In an embodiment of the present application, the sorted location information is sent to the application client via network communication.

[0112] The application client receives the location information data packet from the index server, and then parses it to extract key content such as the server address, storage path, etc. Based on this information, the application client prepares to establish a connection with the data block server to obtain data information.

[0113] The application client establishes a network connection with the corresponding data block server based on the parsed server address and storage path information. Then, it sends a data read request to the data block server. After receiving the request, the data block server retrieves and reads the corresponding data block from the local storage and sends the data block back to the application client through the network. The application client repeats this process, obtains the remaining data blocks from other related data block servers, and then combines and integrates these data blocks in a certain order to restore them to the complete data information corresponding to the read data request, and finally completes the data read operation.

[0114] In an exemplary embodiment, the storage node includes an index server and a data block server. On this basis, the above “returning data information corresponding to the read data request to the application client according to the target data acquisition method” includes:

[0115] When the target data acquisition mode is the transit data acquisition mode, the index server obtains data information from the data block server and returns the data information to the application client.

[0116] In the embodiment of the present application, when the target data acquisition mode is the transit data acquisition mode, the index server sends a data acquisition request to the corresponding data block server. The data acquisition request contains the identification information of the data required for the read data request (such as the file name, data block number, etc.), so that the data block server can accurately locate and extract the data.

[0117] After receiving the request from the index server, the data block server searches and reads the corresponding data block in the local storage according to the identification information in the request. Then, the processed data is transmitted back to the index server through the network. The index server receives the data information returned from the data block server. The index server organizes and encapsulates the acquired data information so that it conforms to the format that the application client can receive and process. Then, the data information is sent to the application client through network communication.

[0118] In an exemplary embodiment, Figure 4 As shown, the above method also includes:

[0119] Step 401 : predicting the execution results of multiple candidate data acquisition methods based on the record information of the historical stage, and obtaining the success probability corresponding to each candidate data acquisition method.

[0120] The record information of the historical stage refers to the data records related to data acquisition accumulated by the distributed storage system during its past operation.

[0121] The candidate data acquisition mode refers to different methods or strategies that can be selected in the distributed storage system for acquiring the data requested by the application client.

[0122] The success probability refers to the possibility of successfully acquiring data for each candidate data acquisition method, predicted based on historical record information and current system status.

[0123] In the embodiment of the present application, the historical record information is first obtained from the storage device (such as a hard disk, a database, etc.) of the distributed storage system. For each candidate data acquisition method, analysis is performed based on the collated historical record information. The ratio of the number of successful attempts to the total number of attempts of the data acquisition method under a similar current system state (such as the current input and output data volume, cluster bandwidth, etc. are similar to certain time periods in the historical records) is calculated as a preliminary estimate of its success probability.

[0124] Step 402 , modifying the target data acquisition method based on the success probabilities corresponding to the candidate data acquisition methods to obtain a modified target data acquisition method.

[0125] Correspondingly, according to the target data acquisition method, the data information corresponding to the read data request is returned to the application client, including:

[0126] According to the modified target data acquisition method, data information corresponding to the read data request is returned to the application client.

[0127] The correction process refers to the process of adjusting and optimizing the initially determined target data acquisition method after predicting the success probability of each candidate data acquisition method based on historical record information.

[0128] In the embodiment of the present application, the predicted success probabilities of the candidate data acquisition methods are compared. If the initially determined target data acquisition method has a lower success probability than other candidate methods, and the difference between the two probabilities exceeds a certain threshold, the target data acquisition method is corrected to a candidate method with a higher success probability.

[0129] The data acquisition operation is performed according to the modified target data acquisition method. If the modified target data acquisition method is the direct reading data acquisition method, the operation is performed according to the steps of the direct reading data acquisition method described above, that is, the index server returns metadata to the application client, instructing the application client to obtain data information from the data block server; if it is the transit data acquisition method, the index server obtains data information from the data block server and returns the data information to the application client. During the data acquisition process, the data is processed as necessary, such as data verification (ensuring the integrity and accuracy of the data), format conversion (converting the data into a format that the application client can recognize), etc., and then the processed data information is transmitted back to the application client through the network to complete the entire data acquisition and return process. For example, if the modified target data acquisition method is the transit data acquisition method, after the index server obtains the data from the data block server, it performs verification and format conversion, and then sends the data to the application client so that the application client can correctly receive and use the data.

[0130] In an exemplary embodiment, the method further includes:

[0131] The execution result of the target data acquisition method is recorded to obtain record information.

[0132] In the embodiment of the present application, the execution result of the target data acquisition method is recorded to obtain record information.

[0133] In some embodiments, the following may be recorded:

[0134] (1) Record the relevant identifiers of the data acquisition request, such as the request number, request time, requested data type or file name, so that the data acquisition operation can be accurately identified and traced later. Record the specific target data acquisition method used, whether it is direct reading data acquisition method or transit data acquisition method, or other possible data acquisition strategies. For example, if the direct reading data acquisition method is used, record it as "direct reading".

[0135] (2) Record whether the final result of the data acquisition operation is success or failure. If successful, further record the amount of data acquired and the data integrity check result (such as whether the hash check is passed, etc.); if failed, record the reason for the failure, such as data block server connection timeout, data does not exist, insufficient permissions, and other detailed information. For example, if the data acquisition is successful, record "success, the amount of data acquired is 10MB, and the hash check is passed"; if it fails, record "failure, reason: data block server A connection timeout".

[0136] According to some embodiments of the present application, a distributed storage method is provided, such as Figure 5 As shown, in Figure 5In the example, AppClient represents an application client, Index Server represents an index server, Chunk Server represents a data block server, direct read data represents a direct read data acquisition method, transit data represents a transit data acquisition method, and IO size represents input and output data volume information. This method is described by taking the application of the method to a storage node as an example, and may include the following steps:

[0137] Step 1: After receiving a data read request initiated by an application client, obtain the cluster real-time information of the distributed storage system. The cluster real-time information includes at least input and output data volume information and cluster bandwidth information.

[0138] Step 2: when the input / output data volume information is less than the first preset threshold, or when the input / output data volume information is greater than the first preset threshold and the cluster bandwidth information is less than the second preset threshold, determine that the target data acquisition mode is the transit data acquisition mode.

[0139] Step 3: When the target data acquisition method is the transit data acquisition method, the index server obtains data information from the data block server and returns the data information to the application client.

[0140] Step 4: Record the execution result of the target data acquisition method to obtain record information.

[0141] According to some embodiments of the present application, a distributed storage method is provided, which is described by taking the application of the method to a storage node as an example, and may include the following steps:

[0142] Step 1: After receiving a data read request initiated by an application client, obtain the cluster real-time information of the distributed storage system. The cluster real-time information includes at least input and output data volume information and cluster bandwidth information.

[0143] Step 2: when the input and output data volume information is greater than a first preset threshold and the cluster bandwidth information is greater than a second preset threshold, determining that the target data acquisition mode is a direct read data acquisition mode.

[0144] Step 3: When the target data acquisition mode is a direct-read data acquisition mode, the location information is acquired from a location mapping relationship table pre-stored in the index server based on the data read request.

[0145] Step 4: Send the location information to the application client, so that the application client obtains data information from the data block server based on the location information.

[0146] Step 5: When the target data acquisition mode is the transit data acquisition mode, the index server acquires data information from the data block server and returns the data information to the application client.

[0147] Step 6: Record the execution result of the target data acquisition method to obtain record information.

[0148] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0149] Based on the same inventive concept, the embodiment of the present application also provides a distributed storage device for implementing the above-mentioned distributed storage method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more distributed storage device embodiments provided below can refer to the limitations on the distributed storage method above, and will not be repeated here.

[0150] In an exemplary embodiment, Figure 6 As shown, a distributed storage device is provided, including: an acquisition module 501, a determination module 502 and a return module 503, wherein:

[0151] The acquisition module 501 is used to acquire the real-time cluster information of the distributed storage system after receiving the data read request initiated by the application client;

[0152] A determination module 502 is used to determine a target data acquisition method corresponding to a read data request based on cluster real-time information;

[0153] The return module 503 is used to return data information corresponding to the data read request to the application client according to the target data acquisition method.

[0154] In an exemplary embodiment, the cluster real-time information includes at least input and output data volume information and cluster bandwidth information. The above-mentioned determination module 502 is specifically used to determine that the target data acquisition method is a transit data acquisition method when the input and output data volume information is less than a first preset threshold, or when the input and output data volume information is greater than the first preset threshold and the cluster bandwidth information is less than a second preset threshold; when the input and output data volume information is greater than the first preset threshold and the cluster bandwidth information is greater than the second preset threshold, determine that the target data acquisition method is a direct reading data acquisition method.

[0155] In an exemplary embodiment, the storage node includes an index server and a data block server. The above-mentioned return module 503 is specifically used for the index server to return metadata to the application client when the target data acquisition method is a direct read data acquisition method, so as to instruct the application client to obtain data information corresponding to the read data request from the data block server.

[0156] In an exemplary embodiment, the metadata includes location information of the data information stored in the data block server. The above-mentioned return module 503 is specifically used to obtain the location information from the location mapping relationship table pre-stored in the index server based on the read data request; and send the location information to the application client so that the application client obtains the data information from the data block server based on the location information.

[0157] In an exemplary embodiment, the storage node includes an index server and a data block server. The above-mentioned return module 503 is specifically used for the index server to obtain data information from the data block server and return the data information to the application client when the target data acquisition method is the transit data acquisition method.

[0158] In an exemplary embodiment, the above device further includes:

[0159] The prediction module 504 is used to predict the execution results of multiple candidate data acquisition methods based on the record information of the historical stage, and obtain the success probability corresponding to each candidate data acquisition method;

[0160] A correction module 505 is used to correct the target data acquisition method based on the success probability corresponding to each candidate data acquisition method to obtain a corrected target data acquisition method;

[0161] Correspondingly, according to the target data acquisition method, data information corresponding to the read data request is returned to the application client, including: according to the modified target data acquisition method, data information corresponding to the read data request is returned to the application client.

[0162] In an exemplary embodiment, the above device further includes:

[0163] The recording module 506 is used to record the execution result of the target data acquisition method to obtain recording information.

[0164] Each module in the above distributed storage device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0165] In an exemplary embodiment, a distributed storage system is provided. Figure 1 , the system includes a computing node 10 and a storage node 11; the computing node 10 is used to send a read data request to the storage node 11; the storage node 11 is used to obtain the cluster real-time information of the distributed storage system after receiving the read data request initiated by the computing node 10; based on the cluster real-time information, determine the target data acquisition method corresponding to the read data request; according to the target data acquisition method, return the data information corresponding to the read data request to the computing node 10. Among them, the storage node 11 is the basic component unit in the distributed storage system. It is a physical device or logical entity that actually stores data, which can usually be a server or storage device. In the distributed storage system, multiple storage nodes 11 work together to store and manage data. For example, in a large data center, there may be hundreds of storage nodes 11, which are connected together through a network to form a huge storage resource pool. Each storage node 11 has a certain storage capacity, computing power and network communication capability, and is responsible for storing data blocks, processing data read and write requests, and interacting and collaborating with other nodes.

[0166] In an exemplary embodiment, Figure 7 As shown, in Figure 7 In the figure, App Client refers to the application client, Index Server refers to the index server, Chunk Server refers to the data block server, and chunk client refers to the client program or module that interacts with the data block chunk. Its functions involve requesting data blocks from the data block server, receiving data blocks, preprocessing data blocks, or locally managing data block-related operations. The storage node 11 is deployed with an index server 111 and a data block server 112;

[0167] When the target data acquisition method is a direct read data acquisition method, the index server 111 is used to return metadata to the application client 101 to instruct the application client 101 to obtain data information corresponding to the read data request from the data block server 112; when the target data acquisition method is a transit data acquisition method, the index server 111 is used to obtain data information from the data block server 112 and return the data information to the application client 101.

[0168] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in a distributed storage process. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a distributed storage method is implemented.

[0169] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0170] According to some embodiments of the present application, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0171] According to some embodiments of the present application, a computer program product is also provided, and when the computer program is executed by a processor, the above method can be implemented. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, part or all of the above method can be implemented in whole or in part according to the process or function described in the embodiment of the present application.

[0172] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0173] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0174] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A distributed storage method, characterized in that: Applied to a storage node, the method comprises: After receiving the data read request initiated by the application client, obtain the real-time cluster information of the distributed storage system; Determine, based on the cluster real-time information, a target data acquisition method corresponding to the read data request; According to the target data acquisition method, data information corresponding to the data read request is returned to the application client.

2. The method according to claim 1, characterized in that The cluster real-time information includes at least input and output data volume information and cluster bandwidth information. The determining, based on the cluster real-time information, a target data acquisition method corresponding to the read data request includes: When the input / output data volume information is less than a first preset threshold, or when the input / output data volume information is greater than the first preset threshold and the cluster bandwidth information is less than a second preset threshold, determining that the target data acquisition mode is a transit data acquisition mode; When the input and output data volume information is greater than the first preset threshold and the cluster bandwidth information is greater than the second preset threshold, it is determined that the target data acquisition mode is a direct read data acquisition mode.

3. The method according to claim 2, characterized in that The storage node includes an index server and a data block server; and returning data information corresponding to the read data request to the application client according to the target data acquisition method includes: In the case where the target data acquisition mode is the direct-read data acquisition mode, the index server returns metadata to the application client to instruct the application client to acquire data information corresponding to the read data request from the data block server.

4. The method according to claim 3, characterized in that The metadata includes location information of the data information stored in the data block server; The index server returns metadata to the application client to instruct the application client to obtain data information corresponding to the read data request from the data block server, including: Acquire the location information from a location mapping relationship table pre-stored in the index server based on the data read request; The location information is sent to the application client, so that the application client obtains the data information from the data block server based on the location information.

5. The method according to claim 2, characterized in that: The storage node includes an index server and a data block server; according to the target data acquisition method, returning data information corresponding to the read data request to the application client, including: In the case where the target data acquisition method is the transit data acquisition method, the index server acquires the data information from the data block server and returns the data information to the application client.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Based on the recorded information of the historical stage, the execution results of multiple candidate data acquisition methods are predicted to obtain the success probability corresponding to each of the candidate data acquisition methods; Correcting the target data acquisition method based on the success probability corresponding to each of the candidate data acquisition methods to obtain a corrected target data acquisition method; Correspondingly, according to the target data acquisition method, returning data information corresponding to the data read request to the application client includes: According to the modified target data acquisition method, data information corresponding to the data read request is returned to the application client.

7. The method according to any one of claim 6, characterized in that: The method further comprises: The execution result of the target data acquisition method is recorded to obtain record information.

8. A distributed storage device, characterized in that: Applied to a storage node, the device comprises: The acquisition module is used to obtain the real-time cluster information of the distributed storage system after receiving the data read request initiated by the application client; A determination module, used to determine a target data acquisition method corresponding to the read data request based on the cluster real-time information; The return module is used to return the data information corresponding to the read data request to the application client according to the target data acquisition method.

9. A distributed storage system, characterized in that: The system includes a computing node and a storage node; an application client is deployed on the computing node; The application client is used to send a read data request to the storage node; The storage node is used to obtain the real-time cluster information of the distributed storage system after receiving a data read request initiated by the application client; Determine, based on the cluster real-time information, a target data acquisition method corresponding to the read data request; According to the target data acquisition method, data information corresponding to the data read request is returned to the application client.

10. The system according to claim 9, characterized in that An index server and a data block server are deployed on the storage node; In the case where the target data acquisition mode is a direct read data acquisition mode, the index server is used to return metadata to the application client to instruct the application client to obtain data information corresponding to the read data request from the data block server; In the case where the target data acquisition method is a transit data acquisition method, the index server is used to obtain the data information from the data block server and return the data information to the application client.

Images