Data processing method and equipment based on distributed storage system

Abstract

The embodiment of the invention provides a data processing method and device based on a distributed storage system. The method comprises the steps that whether the access heat corresponding to data maintained by the distributed storage system exceeds a first threshold value or not is periodically monitored through a coordination node in the distributed storage system; when the access popularity of the monitored target data exceeds a first threshold value, a storage node used for constructing a temporary copy of the target data is selected from the distributed storage system to serve as a copy node of the target data, and the temporary copy of the target data is constructed through the copy node of the target data according to data fragment position information of the target data maintained by the coordination node; and reading the corresponding data fragment from the corresponding storage node and constructing a copy of the target data, wherein a copy node of the target data is used for responding to an access request for the target data based on the copy of the target data. According to the method, the data reading performance can be improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of distributed storage technology, and more particularly to a data processing method and device based on a distributed storage system. Background Art

[0002] In distributed storage systems, in order to ensure data security, the erasure code (EC) mode is usually used to store and manage data. In this way, the data is divided into multiple data fragments and multiple data fragments for verification are generated to be stored in multiple storage nodes. When responding to data access requests, the distributed storage system reads the corresponding data fragments from multiple storage nodes to assemble the required data. In actual applications, when the distributed storage system uses the erasure code mode to manage data, it needs to read multiple data fragments for assembly, which is time-consuming. Especially for large-scale data requests, the performance of the distributed storage system is low and it is difficult to meet business needs. Summary of the invention

[0003] The embodiments of the present disclosure provide a data processing method and device based on a distributed storage system, which can improve the data reading performance of the distributed storage system.

[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method based on a distributed storage system, comprising:

[0005] Periodically monitoring, by a coordination node in a distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system includes a plurality of storage nodes, the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, the data shard location information being used to identify the storage locations in a plurality of storage nodes of a plurality of data shards into which the data is divided when the data is stored in an erasure coding mode;

[0006] When the access popularity of the monitored target data exceeds a first threshold, a storage node used to construct a temporary copy of the target data is selected from the distributed storage system as a replica node of the target data, and the replica node of the target data reads the corresponding data shard from the corresponding storage node and constructs a copy of the target data according to the data shard location information of the target data maintained by the coordination node through the replica node of the target data. The replica node of the target data is used to respond to access requests for the target data based on the copy of the target data.

[0007] In a second aspect, an embodiment of the present disclosure provides a data reading device based on a distributed storage system, including:

[0008] A monitoring unit, configured to periodically monitor, through a coordination node in a distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system includes a plurality of storage nodes, the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, the data shard location information being used to identify storage locations in a plurality of storage nodes of a plurality of data shards into which the data is divided when the data is stored in an erasure coding mode;

[0009] A data processing unit is used to select a storage node for building a temporary copy of the target data from the distributed storage system as a copy node of the target data when the access popularity of the monitored target data exceeds a first threshold, and read the corresponding data shard from the corresponding storage node and build a copy of the target data through the copy node of the target data according to the data shard location information of the target data maintained by the coordination node, and the copy node of the target data is used to respond to access requests for the target data based on the copy of the target data.

[0010] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0011] The memory stores computer-executable instructions;

[0012] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data processing method based on the distributed storage system as described in the first aspect and various possible designs of the first aspect.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the data processing method based on a distributed storage system as described in the first aspect and various possible designs of the first aspect is implemented.

[0014] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method based on a distributed storage system as described in the first aspect and various possible designs of the first aspect.

[0015] The present embodiment provides a data processing method and device based on a distributed storage system, the method comprising: periodically monitoring, through a coordination node in the distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system comprises a plurality of storage nodes, the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, the data shard location information being used to identify the storage locations of a plurality of data shards into which the data is divided when the data is stored in an erasure code mode in a plurality of storage nodes; when the access heat of the monitored target data exceeds the first threshold, a storage node for constructing a temporary copy of the target data is selected from the distributed storage system as a copy node of the target data, and through the copy node of the target data, according to the data shard location information of the target data maintained by the coordination node, the corresponding data shard is read from the corresponding storage node and a copy of the target data is constructed, the copy node of the target data being used to respond to access requests for the target data based on the copy of the target data. In this technical solution, when the access popularity of the target data exceeds the first threshold, the replica node of the target data reads the corresponding data shard from the corresponding storage node and builds a copy of the target data according to the data shard location information of the target data maintained by the coordination node. In this way, when accessing the above-mentioned target data, a data copy can be obtained from the replica node without obtaining data shards from each storage node and merging data shards, thereby improving the data reading performance of the distributed storage system and thus improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 Schematic diagram of reading data in a multi-copy mode provided by an embodiment of the present disclosure Figure 1 ;

[0018] Figure 2 Schematic diagram of reading data through erasure coding mode provided by the embodiment of the present disclosure Figure 1 ;

[0019] Figure 3 The process of the data processing method based on the distributed storage system provided in the embodiment of the present disclosure is Figure 1 ;

[0020] Figure 4A schematic diagram of a data processing method based on a distributed storage system provided in an embodiment of the present disclosure Figure 1 ;

[0021] Figure 5 A schematic diagram of a data processing method based on a distributed storage system provided in an embodiment of the present disclosure Figure 2 ;

[0022] Figure 6 A schematic diagram of a data processing method based on a distributed storage system provided in an embodiment of the present disclosure Figure 3 ;

[0023] Figure 7 A schematic diagram of a data processing method based on a distributed storage system provided in an embodiment of the present disclosure Figure 4 ;

[0024] Figure 8 A schematic diagram of a data processing method based on a distributed storage system provided in an embodiment of the present disclosure Figure 5 ;

[0025] Fig. 9 A schematic diagram of the structure of a data reading device based on a distributed storage system provided in an embodiment of the present disclosure;

[0026] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0029] In a distributed storage system, data is usually backed up to ensure data security. Currently, there are two main backup methods: multi-copy mode and erasure code (EC) mode.

[0030] The multi-copy mode is to completely copy the data into multiple identical copies of the data, which are stored on multiple different data storage server nodes (hereinafter referred to as storage nodes). When the user wants to read the data, he can read it from any of the storage nodes. For the N-copy mode, up to N-1 storage nodes can be tolerated to fail at the same time. As long as there is one storage node that is normal, the user can access the data normally, and copy a new copy from the normal storage node and store it on the new storage node, keeping the total number of N copies unchanged.

[0031] like Figure 1 In the 3-copy mode shown, a data including (abcd) is copied into 3 copies and stored on 3 data storage server nodes, namely, node A, node B, and node C. At most 2 nodes can be tolerated to fail at the same time. When no node fails, when the user wants to read the data, he only needs to read the data from any one of the nodes, such as reading the data from node 2.

[0032] Among them, the erasure code mode is to divide the data into N data shards, and calculate M different check shards according to a certain mathematical method, and place these N+M shards on N+M different data storage server nodes. Any N shards among these N+M shards, whether they are data shards or check shards, can be assembled into complete data. When the user wants to read the data, it is necessary to read all the data shards from the N storage nodes and then assemble them into complete data. When E (E≤M) of the nodes fail, it is necessary to read E check shards from the E check nodes, calculate them first and then assemble them into complete data, and store the recalculated data shards on the new nodes, keeping the total number of N+M shards unchanged. For the N+M erasure code mode, up to M nodes can be tolerated to fail at the same time. As long as there are N nodes that are normal, users can access the data correctly.

[0033] If we want to achieve the same tolerance of two nodes failing at the same time, M must be 2. Figure 2 In the 4+2 erasure code mode shown, a copy of data (abcd) is divided into four data shards: (a), (b), (c), and (d). Data shard (a) is stored in storage node A, data shard (b) is stored in storage node B, data shard (c) is stored in storage node C, and data shard (d) is stored in storage node D. Two check shards are calculated, including check data (x) stored in check node X, and check data (y) stored in check node Y. In the case that no node fails, when the user wants to read the data, the data needs to be read from nodes A, B, C, and D where all four data shards are located.

[0034] The multi-copy mode and the erasure code mode each have very obvious advantages and disadvantages. For the multi-copy mode, the disadvantage is that one copy of the original data needs to be stored in N copies, N times write amplification, and three copies means three times write amplification, and the storage space utilization rate is not high. But the advantage is that reading and writing are simple, no calculation is required, and the CPU overhead is low; especially for read operations, under normal circumstances, a read operation only requires one network read request, and the reading performance is high. For the erasure code mode, the advantage is that one copy of the original data is divided into N copies and then stored in N+M shards, (N+M) / N times write amplification, and the 4+2 erasure code mode means only 1.5 times write amplification. Under the same data reliability (tolerating the simultaneous failure of the same number of nodes), the storage space utilization rate of the erasure code mode is higher. But the disadvantage is that reading and writing are complicated, and when writing, the shards need to be split and the shards need to be calculated and verified, and the CPU overhead is high. Especially for read operations in erasure code mode, under normal circumstances, a read operation also requires N network read requests. The time taken for a read operation is from issuing the first network read request, then receiving the results of all N network read requests, and finally assembling the N received results, which is time-consuming and has low reading performance. If node failures are taken into account, the time required to re-issue the read request to the verification node, wait for the results, and verify and calculate the original data must be added, and the delay of the read request will be even higher.

[0035] It can be seen that when the erasure code (EC) mode is used to store and manage data, one data is divided into multiple data fragments and multiple data fragments for verification are generated to be stored in multiple storage nodes. When the distributed storage system responds to data access requests, it reads the corresponding data fragments from multiple storage nodes to assemble the required data. In practical applications, when the distributed storage system uses the erasure code mode to manage data, it is time-consuming to read multiple data fragments for assembly processing. Especially for a large number of data requests, the performance of the distributed storage system is low and it is difficult to meet business needs. In response to the technical problems in the prior art, the inventor's technical conception is as follows: select a storage node for building a temporary copy from the distributed storage system as a copy node, and temporarily upgrade the hot data with an access heat exceeding the first threshold to a complete data copy through the copy node. When accessing the above hot data, the data copy can be directly read without obtaining data fragments from each storage node and merging data fragments, thereby improving the data reading performance.

[0036] Accordingly, the specific steps may include: first, periodically monitoring the access heat corresponding to each data maintained by the distributed storage system through the coordination node in the distributed storage system to see if it exceeds a first threshold; wherein the distributed storage system includes multiple storage nodes, and the coordination node is a storage node designated in the distributed storage system for maintaining the data shard location information corresponding to each data in the distributed storage system, and the data shard location information is used to identify the storage locations of multiple data shards into which the data is divided when the data is stored in the erasure code mode in multiple storage nodes. Secondly, when the access heat of the monitored target data exceeds the first threshold, a storage node used to construct a temporary copy of the target data is selected from the distributed storage system as a copy node of the target data, and the copy node of the target data reads the corresponding data shard from the corresponding storage node and constructs a copy of the target data according to the data shard location information of the target data maintained by the coordination node through the copy node of the target data, and the copy node of the target data is used to respond to access requests for the target data based on the copy of the target data.

[0037] In this technical solution, when the access popularity of the target data exceeds the first threshold, the replica node of the target data reads the corresponding data shard from the corresponding storage node and builds a copy of the target data according to the data shard location information of the target data maintained by the coordination node. In this way, when accessing the above-mentioned target data, a data copy can be obtained from the replica node without obtaining data shards from each storage node and merging data shards, thereby improving the data reading performance of the distributed storage system and thus improving the user experience.

[0038] The following is a specific implementation process of the data processing method and device based on the distributed storage system involved in the embodiments of the present disclosure. Some examples are only for illustrative purposes and are not intended to be limiting. The execution subject of the data processing method based on the distributed storage system involved in the embodiments of the present disclosure is an electronic device, which can be a terminal, a server, etc.

[0039] Figure 3 The process of the data processing method based on the distributed storage system provided in the embodiment of the present disclosure is Figure 1 ,like Figure 3 As shown, the data processing method may include:

[0040] S301. Periodically monitor, through a coordination node in a distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system includes multiple storage nodes, and the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, and the data shard location information is used to identify the storage locations of multiple data shards into which the data is divided when the data is stored in an erasure code mode in multiple storage nodes.

[0041] In the disclosed embodiment, the target data may be hot data with high access popularity. For example, data with access popularity exceeding a first threshold. The access popularity may be access frequency and / or access times, etc. The first threshold may be a preset access frequency and / or a preset access times. In the present embodiment, the value of the first threshold is not specifically limited.

[0042] In some embodiments, the distributed storage system includes multiple storage nodes, and a coordination node can be selected from the multiple storage nodes according to the principle of load balancing. Accordingly, the step of selecting the coordination node may include: preferentially selecting a storage node with the lowest load or the largest free space from the storage nodes in the distributed storage system as the coordination node.

[0043] For example, Figure 4 The 4+2 erasure code mode shown in the figure, the distributed storage system includes 4 storage nodes and 2 check nodes. A data including (abcd) is divided into 4 data shards: (a), (b), (c), (d), (x), (y). Data shard (a) is stored in storage node A, data shard (b) is stored in storage node B, data shard (c) is stored in storage node C, data shard (d) is stored in storage node D, and two check shards are calculated, including check data (x) stored in check node X, and check data (y) stored in check node Y.

[0044] A storage node B with the smallest load is selected from the four storage nodes as a coordination node. The storage node is designated in the distributed storage system to maintain the data shard location information corresponding to each data in the distributed storage system. At this time, the storage node B stores: the data shard location information of data shard (a), the data shard location information of data shard (b), the data shard location information of data shard (c), the data shard location information of data shard (d), the data shard location information of data shard (x), and the data shard location information of data shard (y).

[0045] In the embodiment of the present disclosure, the data shard location information may be information related to the storage location of the data shard. Optionally, the data shard location information may include: the IP address of the storage node where the data shard is located, the storage path of the data shard in the storage node, etc. Figure 4 As shown, in the present application, the data shard location information may be represented as PeerInfo.

[0046] In other embodiments, the number of coordination nodes may be one or more. When the number of coordination nodes is multiple, it is necessary to set the primary and secondary of the multiple coordination nodes. Accordingly, the step of selecting coordination nodes may include: according to the load and free space of each storage node in the distributed storage system, multiple storage nodes with the smallest load or the largest free space are preferentially selected as coordination nodes according to the load balancing principle, and the multiple coordination nodes are configured to provide services in a primary-backup manner.

[0047] Optionally, the coordination node includes a first coordination node and a second coordination node. Accordingly, the steps of providing services in a master-slave manner are as follows: if the first coordination node operates normally, the data shard location information corresponding to each data in the distributed storage system is maintained through the first coordination node; if the first coordination node fails, the data shard location information corresponding to each data in the distributed storage system is maintained through the second coordination node.

[0048] In the embodiments of the present disclosure, when storing certain data, the distributed storage system can divide the data into multiple data shards according to the erasure code mode and store them in each storage node. When the distributed storage system stores the data shards in each storage node, the data shard location information of the data shard can be obtained, and the data shard location information of the data shard can be stored in the coordination node, and the data shard location information of the multiple data shards corresponding to the data can be maintained through the coordination node.

[0049] S302. When the access popularity of the monitored target data exceeds a first threshold, a storage node used to construct a temporary copy of the target data is selected from the distributed storage system as a replica node of the target data, and the replica node of the target data reads the corresponding data shard from the corresponding storage node and constructs a copy of the target data according to the data shard location information of the target data maintained by the coordination node through the replica node of the target data. The replica node of the target data is used to respond to access requests for the target data based on the copy of the target data.

[0050] In the embodiment of the present disclosure, a replica node may be selected from multiple storage nodes according to the principle of load balancing. Accordingly, the step of selecting a replica node may include: preferentially selecting a storage node with the lowest load or the largest free space from each storage node in the distributed storage system as a replica node for the target data.

[0051] It should be noted that the replica node and the coordination node can be the same storage node or two different storage nodes. When the replica node is not the coordination node, the data shard location information of the target data needs to be sent to the replica node through the coordination node.

[0052] For example, Figure 5As shown, the replica node and the coordination node are not the same storage node. Storage node B is the coordination node, and storage node C is the replica node; the data shard location information of multiple data shards of the target data (abcd) is forwarded to storage node C through storage node B.

[0053] In this step, the replica node quickly reads other data shards from other storage nodes and assembles them according to the node location information of each storage node, and quickly builds a temporary replica. When building a temporary replica, if one of the storage nodes fails, the verification shard can be read from the verification node, and the data shards on the failed node can be calculated and then assembled.

[0054] For example, Figure 6 As shown, the replica node (storage node C) reads multiple data shards corresponding to the target data from each storage node, and assembles them to create a data replica (abcd) of the target data.

[0055] It should be noted that in order to build a temporary copy as quickly as possible, when the replica node reads data from each storage node, the length of the data read each time can be set as large as possible. Accordingly, the replica node of the target data reads the corresponding data shard from the corresponding storage node and builds a copy of the target data according to the data shard location information of the target data maintained by the coordination node, including: the replica node of the target data reads the corresponding data shard from the corresponding storage node according to the data shard location information of the target data maintained by the coordination node, and builds a copy of the target data according to the data reading length of the preset data size. Among them, the data reading length of the preset data size is: N times the data reading length when reading data from each storage node based on the distributed storage system. Among them, N is greater than 1. In this application, the numerical value of the preset data size is not specifically limited, and can be set and modified according to the device performance of the replica node and whether there is a jam during reading.

[0056] In the disclosed embodiment, after the data copy of the target data in the replica node is constructed, when the distributed storage system reads the target data again, the replica node, in addition to returning the data shard of the node to the distributed storage system, also carries notification information of the creation of the data copy of the target data in the response returned to the distributed storage system, informing the distributed storage system that the replica node already has a complete copy of the target data. Accordingly, this step may include: receiving a data read request for the target data sent by the client, responding to the data read request through the replica node of the target data, and carrying replica information in the response, wherein the replica information is used to inform the client that the replica node has a complete copy of the target data.

[0057] For example, Figure 7As shown, when the distributed storage system reads the target data (abcd) again, it only needs to read the data from the replica node, and does not need to read each data shard from the storage nodes where other data shards are located. In other words, the number of network requests for a read operation is reduced from 4 to 1. In addition, there is no need to assemble data shards, thereby improving the data reading performance. It should be further explained that when a storage node fails, as long as the replica node is not faulty, there is no need to read and calculate the verification shard, and it can still be read from the replica node.

[0058] The disclosed embodiment provides a data processing method based on a distributed storage system: the access heat corresponding to each data maintained by the distributed storage system is periodically monitored by a coordination node in the distributed storage system to see whether it exceeds a first threshold value; wherein the distributed storage system includes multiple storage nodes, and the coordination node is a storage node designated in the distributed storage system for maintaining the data shard location information corresponding to each data in the distributed storage system, and the data shard location information is used to identify the storage locations of multiple data shards into which the data is divided when the data is stored in an erasure code mode in multiple storage nodes. Secondly, when the access heat of the monitored target data exceeds the first threshold value, a storage node used to construct a temporary copy of the target data is selected from the distributed storage system as a copy node of the target data, and the corresponding data shard is read from the corresponding storage node and a copy of the target data is constructed through the copy node of the target data according to the data shard location information of the target data maintained by the coordination node, and the copy node of the target data is used to respond to access requests for the target data based on the copy of the target data. In this technical solution, when the access popularity of the target data exceeds the first threshold, the replica node of the target data reads the corresponding data shard from the corresponding storage node and builds a copy of the target data according to the data shard location information of the target data maintained by the coordination node. In this way, when accessing the above-mentioned target data, a data copy can be obtained from the replica node without obtaining data shards from each storage node and merging data shards, thereby improving the data reading performance of the distributed storage system and thus improving the user experience.

[0059] In some embodiments, when other storage nodes fail and need to be repaired, it is not necessary to read the verification shards and calculate, but the data originally stored in the failed node can be directly split from the complete copy in the replica node and copied to the new storage node, keeping the total number of N+M shards unchanged. Accordingly, the method also includes: splitting the shard data originally stored in the failed storage node from the complete copy of the target data maintained by the replica node of the target data, and copying it to the new storage node used to replace the failed storage node.

[0060] For example, Figure 8As shown, if storage node A fails, the data a originally stored in storage node A can be split from the copy of the replica node and copied to the new storage node E.

[0061] It should be noted that if the data access heat continues to increase and reaches a new threshold, the replica node can cache the replica data in memory; if it reaches a higher threshold, the replica node can copy the replica to more storage nodes or verification nodes to balance the access load.

[0062] In some embodiments, the replica node includes hard disk storage and memory storage; accordingly, after creating a data copy of the target data through multiple data shards, the method also includes: storing the data copy of the target data in the hard disk storage of the replica node; when monitoring the access heat of the target data exceeds a second threshold, caching the replica data in the replica node of the target data to the memory, and responding to data access requests through the target data stored in the memory; when monitoring the access heat of the target data exceeds a third threshold, copying the replica to other newly added replica nodes through the replica node of the target data, and realizing data access load balancing through multiple replica nodes.

[0063] In some embodiments, when it is monitored that the access popularity of the target data is lower than a first threshold, the copy of the target data in the copy node of the target data is deleted.

[0064] It should be noted that when the hot data becomes cold, the replicas in the replica nodes can be deleted to save storage space. As for when and how to select the replicas to be deleted in order, there can be different strategies, such as regularly retrieving the coldest replica in the system and deleting the coldest replica in the replica nodes.

[0065] Fig. 9 A schematic diagram of the structure of a data reading device based on a distributed storage system provided in an embodiment of the present disclosure, such as Fig. 9 As shown, the data reading device based on the distributed storage system includes:

[0066] A monitoring unit 901 is used to periodically monitor, through a coordination node in a distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system includes multiple storage nodes, and the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, and the data shard location information is used to identify the storage locations of multiple data shards into which the data is divided when the data is stored in an erasure code mode in multiple storage nodes;

[0067] The data processing unit 902 is used to select a storage node for building a temporary copy of the target data from the distributed storage system as a copy node of the target data when the access popularity of the monitored target data exceeds a first threshold, and read the corresponding data shard from the corresponding storage node and build a copy of the target data through the copy node of the target data according to the data shard location information of the target data maintained by the coordination node. The copy node of the target data is used to respond to access requests for the target data based on the copy of the target data.

[0068] According to one or more embodiments of the present disclosure, the data reading device also includes: a request processing unit, which is used to receive a data reading request for target data sent by a client, respond to the data reading request through a replica node of the target data, and carry replica information in the response, wherein the replica information is used to inform the client that the replica node has a complete copy of the target data.

[0069] According to one or more embodiments of the present disclosure, the data reading device also includes: a data repair unit, which is used to split the shard data originally stored by the failed storage node from the complete copy of the target data maintained by the replica node of the target data, and copy it to a new storage node used to replace the failed storage node.

[0070] According to one or more embodiments of the present disclosure, the monitoring unit 901 is also used to cache the replica data in the replica node of the target data to the memory when the access heat of the monitored target data exceeds a second threshold, and respond to data access requests through the target data stored in the memory; and / or, when the access heat of the monitored target data exceeds a third threshold, copy the replica to other newly added replica nodes through the replica node of the target data, and realize data access load balancing through multiple replica nodes.

[0071] According to one or more embodiments of the present disclosure, the monitoring unit 901 is further configured to delete the copy of the target data in the copy node of the target data when it is monitored that the access heat of the target data is lower than a first threshold.

[0072] According to one or more embodiments of the present disclosure, the data reading device further includes: a node selection unit, configured to preferentially select a storage node with the lowest load or the largest free space from among the storage nodes in the distributed storage system as the coordination node.

[0073] According to one or more embodiments of the present disclosure, the node selection unit is also used to preferentially select multiple storage nodes with the smallest load or the largest free space as the coordination nodes according to the load and free space conditions of each storage node in the distributed storage system and in accordance with the load balancing principle. The multiple coordination nodes are configured to provide services in a master-slave manner.

[0074] According to one or more embodiments of the present disclosure, the node selection unit is also used to preferentially select a storage node with the lowest load or the largest free space from among the storage nodes in the distributed storage system as a replica node for the target data; when the replica node is not the coordination node, the data shard location information of the target data is sent to the replica node via the coordination node.

[0075] According to one or more embodiments of the present disclosure, the data processing unit 902, through the replica node of the target data according to the data shard location information of the target data maintained by the coordination node, reads the corresponding data shard from the corresponding storage node and builds a copy of the target data, including: through the replica node of the target data according to the data shard location information of the target data maintained by the coordination node, reads the corresponding data shard from the corresponding storage node according to the data reading length of a preset data size and builds a copy of the target data.

[0076] refer to Fig.10 , which shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing the embodiment of the present disclosure, and the electronic device 1000 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0077] like Fig.10As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 to a random access memory (RAM) 1003. Various programs and data required for the operation of the electronic device 1000 are also stored in the RAM 1003. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0078] Typically, the following devices may be connected to the I / O interface 1005: input devices 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1008 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0079] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0080] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0081] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0082] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0083] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0085] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0086] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0087] In a first aspect, according to one or more embodiments of the present disclosure, a data processing method based on a distributed storage system is provided, comprising:

[0088] Periodically monitoring, by a coordination node in a distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system includes a plurality of storage nodes, the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, the data shard location information being used to identify the storage locations in a plurality of storage nodes of a plurality of data shards into which the data is divided when the data is stored in an erasure coding mode;

[0089] When the access popularity of the monitored target data exceeds a first threshold, a storage node used to construct a temporary copy of the target data is selected from the distributed storage system as a replica node of the target data, and the replica node of the target data reads the corresponding data shard from the corresponding storage node and constructs a copy of the target data according to the data shard location information of the target data maintained by the coordination node through the replica node of the target data. The replica node of the target data is used to respond to access requests for the target data based on the copy of the target data.

[0090] According to one or more embodiments of the present disclosure, it also includes: receiving a data read request for target data sent by a client, responding to the data read request through a replica node of the target data, and carrying replica information in the response, wherein the replica information is used to inform the client that the replica node has a complete copy of the target data.

[0091] According to one or more embodiments of the present disclosure, it also includes: splitting the shard data originally stored in the failed storage node from the complete copy of the target data maintained by the replica node of the target data, and copying it to a new storage node used to replace the failed storage node.

[0092] According to one or more embodiments of the present disclosure, it also includes: when the access heat of the monitored target data exceeds a second threshold, the replica data in the replica node of the target data is cached to the memory, and the data access request is responded to through the target data stored in the memory; and / or, when the access heat of the monitored target data exceeds a third threshold, the replica is copied to other newly added replica nodes through the replica node of the target data, and data access load balancing is achieved through multiple replica nodes.

[0093] According to one or more embodiments of the present disclosure, the method further includes: when it is monitored that the access heat of the target data is lower than a first threshold, deleting the copy of the target data in the copy node of the target data.

[0094] According to one or more embodiments of the present disclosure, it also includes: preferentially selecting a storage node with the lowest load or the largest free space from among the storage nodes in the distributed storage system as the coordination node.

[0095] According to one or more embodiments of the present disclosure, it also includes: based on the load and free space conditions of each storage node in the distributed storage system, multiple storage nodes with the smallest load or the largest free space are preferentially selected as the coordination nodes in accordance with the load balancing principle, and the multiple coordination nodes are configured to provide services in a master-slave manner.

[0096] According to one or more embodiments of the present disclosure, it also includes: preferentially selecting a storage node with the lowest load or the largest free space from each storage node in the distributed storage system as a replica node for the target data; when the replica node is not the coordination node, sending the data shard location information of the target data to the replica node via the coordination node.

[0097] According to one or more embodiments of the present disclosure, the target data replica node reads the corresponding data shard from the corresponding storage node and constructs a copy of the target data according to the data shard location information of the target data maintained by the coordinating node, including: the target data replica node reads the corresponding data shard from the corresponding storage node according to the data reading length of a preset data size and constructs a copy of the target data according to the data shard location information of the target data maintained by the coordinating node.

[0098] In a second aspect, according to one or more embodiments of the present disclosure, a data reading device based on a distributed storage system is provided, comprising:

[0099] A monitoring unit, configured to periodically monitor, through a coordination node in a distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system includes a plurality of storage nodes, the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, the data shard location information being used to identify storage locations in a plurality of storage nodes of a plurality of data shards into which the data is divided when the data is stored in an erasure coding mode;

[0100] A data processing unit is used to select a storage node for building a temporary copy of the target data from the distributed storage system as a copy node of the target data when the access popularity of the monitored target data exceeds a first threshold, and read the corresponding data shard from the corresponding storage node and build a copy of the target data through the copy node of the target data according to the data shard location information of the target data maintained by the coordination node, and the copy node of the target data is used to respond to access requests for the target data based on the copy of the target data.

[0101] According to one or more embodiments of the present disclosure, the data reading device also includes: a request processing unit, which is used to receive a data reading request for target data sent by a client, respond to the data reading request through a replica node of the target data, and carry replica information in the response, wherein the replica information is used to inform the client that the replica node has a complete copy of the target data.

[0102] According to one or more embodiments of the present disclosure, the data reading device also includes: a data repair unit, which is used to split the shard data originally stored by the failed storage node from the complete copy of the target data maintained by the replica node of the target data, and copy it to a new storage node used to replace the failed storage node.

[0103] According to one or more embodiments of the present disclosure, the monitoring unit is also used to cache the replica data in the replica node of the target data to the memory when the access heat of the monitored target data exceeds a second threshold, and respond to data access requests through the target data stored in the memory; and / or, when the access heat of the monitored target data exceeds a third threshold, copy the replica to other newly added replica nodes through the replica node of the target data, and realize data access load balancing through multiple replica nodes.

[0104] According to one or more embodiments of the present disclosure, the monitoring unit is further configured to delete the copy of the target data in the copy node of the target data when it is monitored that the access heat of the target data is lower than a first threshold.

[0105] According to one or more embodiments of the present disclosure, the data reading device further includes: a node selection unit, configured to preferentially select a storage node with the lowest load or the largest free space from among the storage nodes in the distributed storage system as the coordination node.

[0106] According to one or more embodiments of the present disclosure, the node selection unit is also used to preferentially select multiple storage nodes with the smallest load or the largest free space as the coordination nodes according to the load and free space conditions of each storage node in the distributed storage system and in accordance with the load balancing principle. The multiple coordination nodes are configured to provide services in a master-slave manner.

[0107] According to one or more embodiments of the present disclosure, the node selection unit is also used to preferentially select a storage node with the lowest load or the largest free space from among the storage nodes in the distributed storage system as a replica node for the target data; when the replica node is not the coordination node, the data shard location information of the target data is sent to the replica node via the coordination node.

[0108] According to one or more embodiments of the present disclosure, the data processing unit, through the replica node of the target data according to the data shard location information of the target data maintained by the coordination node, reads the corresponding data shard from the corresponding storage node and builds a copy of the target data, including: through the replica node of the target data according to the data shard location information of the target data maintained by the coordination node, reads the corresponding data shard from the corresponding storage node according to the data reading length of a preset data size and builds a copy of the target data.

[0109] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0110] The memory stores computer-executable instructions;

[0111] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data processing method based on the distributed storage system as described in the first aspect and various possible designs of the first aspect above.

[0112] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the data processing method based on a distributed storage system as described in the first aspect and various possible designs of the first aspect is implemented.

[0113] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the data processing method based on a distributed storage system as described in the first aspect and various possible designs of the first aspect.

[0114] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0115] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0116] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data processing method based on a distributed storage system, characterized in that: The method comprises: Periodically monitoring, by a coordination node in a distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system includes a plurality of storage nodes, the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, the data shard location information being used to identify the storage locations in a plurality of storage nodes of a plurality of data shards into which the data is divided when the data is stored in an erasure coding mode; When the access popularity of the monitored target data exceeds a first threshold, a storage node used to construct a temporary copy of the target data is selected from the distributed storage system as a replica node of the target data, and the replica node of the target data reads the corresponding data shard from the corresponding storage node and constructs a copy of the target data according to the data shard location information of the target data maintained by the coordination node through the replica node of the target data. The replica node of the target data is used to respond to access requests for the target data based on the copy of the target data.

2. The data processing method according to claim 1, characterized in that: The method further comprises: Receive a data read request for target data sent by a client, respond to the data read request through a replica node of the target data, and carry replica information in the response, wherein the replica information is used to inform the client that the replica node has a complete copy of the target data.

3. The data processing method according to claim 1, characterized in that: The method further comprises: The shard data originally stored in the failed storage node is split out from the complete copy of the target data maintained by the replica node of the target data, and is copied to a new storage node used to replace the failed storage node.

4. The data processing method according to claim 1, characterized in that: The method further comprises: When the access heat of the monitored target data exceeds a second threshold, cache the copy data in the copy node of the target data to the memory, and respond to the data access request through the target data stored in the memory; and / or, When the access popularity of the monitored target data exceeds a third threshold, the replica node of the target data is copied to other newly added replica nodes, and data access load balancing is achieved through multiple replica nodes.

5. The data processing method according to claim 1, characterized in that: The method further comprises: When it is monitored that the access heat of the target data is lower than a first threshold, the copy of the target data in the copy node of the target data is deleted.

6. The data processing method according to claim 1, characterized in that: The method further comprises: A storage node with the lowest load or the largest free space is preferentially selected from among the storage nodes in the distributed storage system as the coordination node.

7. The data processing method according to claim 1, characterized in that: The method further comprises: According to the load and free space conditions of each storage node in the distributed storage system, multiple storage nodes with the smallest load or the largest free space are preferentially selected as the coordination nodes in accordance with the load balancing principle, wherein the multiple coordination nodes are configured to provide services in a master-slave mode.

8. The data processing method according to claim 1, characterized in that: The method further comprises: Preferentially selecting a storage node with the lowest load or the largest free space from each storage node in the distributed storage system as a replica node of the target data; When the replica node is not the coordination node, the data shard location information of the target data is sent to the replica node through the coordination node.

9. A data reading device based on a distributed storage system, characterized in that: The device comprises: A monitoring unit, configured to periodically monitor, through a coordination node in a distributed storage system, whether the access heat corresponding to each data maintained by the distributed storage system exceeds a first threshold; wherein the distributed storage system includes a plurality of storage nodes, the coordination node is a storage node designated in the distributed storage system for maintaining data shard location information corresponding to each data in the distributed storage system, the data shard location information being used to identify storage locations in a plurality of storage nodes of a plurality of data shards into which the data is divided when the data is stored in an erasure coding mode; A data processing unit is used to select a storage node for building a temporary copy of the target data from the distributed storage system as a copy node of the target data when the access popularity of the monitored target data exceeds a first threshold, and read the corresponding data shard from the corresponding storage node and build a copy of the target data through the copy node of the target data according to the data shard location information of the target data maintained by the coordination node, and the copy node of the target data is used to respond to access requests for the target data based on the copy of the target data.

10. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the data processing method based on the distributed storage system according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the data processing method based on the distributed storage system as described in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method based on a distributed storage system as described in any one of claims 1 to 8 is implemented.

Images