Data processing method and device based on distributed storage system, equipment and medium

Abstract

The embodiment of the invention provides a data processing method and device based on a distributed storage system, equipment and a medium, and the method comprises the steps: obtaining the load data of a cluster in response to a data obtaining request, determining a first target node in a plurality of first nodes and a plurality of second nodes according to the load data, and obtaining the load data of the cluster through the first target node; acquiring a plurality of first block data aiming at the data acquisition request, performing integrity verification on the plurality of first block data, if the integrity verification is passed, generating target data by adopting the plurality of first block data, and if the integrity verification is not passed, determining missing data corresponding to the plurality of first block data, and obtaining target backup data according to the missing data, generating target data by adopting the multiple pieces of first block data and the target backup data, and feeding back the target data. Through the embodiment of the invention, the first target node needing to be accessed can be flexibly adjusted according to the load condition of the cluster, the data acquisition efficiency is improved, and the stability of the cluster is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device, equipment and medium based on a distributed storage system. Background Art

[0002] As the amount of data continues to grow, the demand for high availability and high fault tolerance of data storage continues to increase, and distributed storage systems have become one of the key focus and research topics in the industry.

[0003] Erasure coding is a data redundancy technology that is widely used in distributed storage systems. The principle is to divide the original data into multiple data blocks, encode the multiple data blocks into multiple check blocks as backup, store the data blocks in different storage nodes, and store the check blocks in different check nodes. When some storage nodes lose data or cannot read data due to failure, the original data can be restored and read by requesting the check nodes to decode the check blocks.

[0004] In the related art, when a request to read data from a storage node fails, a new request needs to be sent to a verification node to recover the data, that is, at least two rounds of requests are required to complete the data reading. It can be seen that this method increases the response time of the system and affects the processing efficiency of related data services. Summary of the invention

[0005] In view of the above problems, a data processing method, device, equipment, and medium based on a distributed storage system are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] A data processing method based on a distributed storage system, the distributed storage system comprising at least one cluster, the cluster comprising a plurality of first nodes and a plurality of second nodes, the plurality of first nodes being used to store data in blocks, the plurality of second nodes being used to back up data stored in the first nodes, the method comprising:

[0007] In response to a data acquisition request, acquiring load data of the cluster, and determining a first target node among the plurality of first nodes and the plurality of second nodes according to the load data;

[0008] Acquire, through the first target node, a plurality of first block data for the data acquisition request;

[0009] Performing integrity check on the plurality of first block data;

[0010] If the integrity check passes, the target data is generated using the plurality of first block data;

[0011] If the integrity check fails, determining missing data corresponding to the plurality of first block data;

[0012] Acquire target backup data according to the missing data, and generate target data using the plurality of first block data and the target backup data;

[0013] The target data is fed back.

[0014] Optionally, determining a first target node among the plurality of first nodes and the plurality of second nodes according to the load data includes:

[0015] Determine a load index value of the cluster according to the load data;

[0016] Comparing the load index value with a preset first threshold and a preset second threshold to obtain a comparison result; wherein the preset first threshold is greater than the preset second threshold;

[0017] A first target node is determined among the plurality of first nodes and the plurality of second nodes according to the comparison result.

[0018] Optionally, determining a first target node from among the plurality of first nodes and the plurality of second nodes according to the comparison result includes:

[0019] If the load index value is less than or equal to the preset second threshold, all the first nodes and all the second nodes are determined as the first target nodes;

[0020] If the load index value is less than the preset first threshold value and greater than the preset second threshold value, all the first nodes and a preset number of the second nodes are determined as the first target nodes;

[0021] If the load indicator value is greater than or equal to the preset first threshold, all the first nodes are determined as the first target nodes.

[0022] Optionally, when the first target node is all the first nodes and all the second nodes, acquiring the plurality of first block data for the data acquisition request through the first target node includes:

[0023] Concurrently initiating multiple first requests to the first target node to obtain multiple first return results;

[0024] Acquire the plurality of first block data from the first return results corresponding to all the first nodes;

[0025] The acquiring target backup data according to the missing data comprises:

[0026] According to the missing data, the target backup data is obtained from the first returned results corresponding to all the second nodes.

[0027] Optionally, when the first target node is all the first nodes and a preset number of the second nodes, acquiring the plurality of first block data for the data acquisition request through the first target node further includes:

[0028] Concurrently initiating multiple second requests to the first target node to obtain multiple second return results;

[0029] Acquire the plurality of first block data from the second returned results corresponding to all the first nodes;

[0030] The acquiring target backup data according to the missing data further comprises:

[0031] Acquire the first backup data from the second returned results corresponding to the preset number of second nodes;

[0032] Determining whether the first backup data can restore the missing data;

[0033] If the first backup data can restore the missing data, determining the first backup data as the target backup data;

[0034] If the first backup data cannot restore the missing data, determining a second target node from among the plurality of second nodes according to the first backup data and the missing data;

[0035] The second backup data is acquired through the second target node, and the first backup data and the second backup data are determined as the target backup data.

[0036] Optionally, when the first target nodes are all the first nodes, acquiring the plurality of first block data for the data acquisition request through the first target nodes further includes:

[0037] Concurrently initiate multiple third requests to the first target node to obtain the multiple first block data;

[0038] The acquiring target backup data according to the missing data further comprises:

[0039] Determine a third target node among the plurality of second nodes according to the missing data;

[0040] The target backup data is obtained through the third target node.

[0041] Optionally, the load data includes one or more of central processing unit load data, memory load data and disk load data.

[0042] A data processing device based on a distributed storage system, the distributed storage system comprising at least one cluster, the cluster comprising a plurality of first nodes and a plurality of second nodes, the plurality of first nodes being used to store data in blocks, the plurality of second nodes being used to back up data stored in the first nodes, the device comprising:

[0043] a first target node determination module, configured to obtain load data of the cluster in response to a data acquisition request, and determine a first target node among the plurality of first nodes and the plurality of second nodes according to the load data;

[0044] A first block data acquisition module, used for acquiring a plurality of first block data corresponding to the data acquisition request through the first target node;

[0045] An integrity check module, used for performing integrity check on the plurality of first block data;

[0046] A first target data generating module, configured to generate target data using the plurality of first block data if the integrity check passes;

[0047] A missing data acquisition module, configured to determine the missing data corresponding to the plurality of first block data if the integrity check fails;

[0048] a second target data generating module, configured to obtain target backup data according to the missing data, and generate target data using the plurality of first block data and the target backup data;

[0049] The target data feedback module is used to feed back the target data.

[0050] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the data processing method based on the distributed storage system as described above is implemented.

[0051] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data processing method based on the distributed storage system as described above is implemented.

[0052] The embodiments of the present invention have the following advantages: by responding to a data acquisition request, the load data of the cluster is acquired, and a first target node is determined among multiple first nodes and multiple second nodes according to the load data, multiple first block data for the data acquisition request are acquired through the first target node, and integrity verification is performed on the multiple first block data. If the integrity verification passes, the multiple first block data are used to generate target data. If the integrity verification fails, the missing data corresponding to the multiple first block data is determined, and target backup data is acquired according to the missing data. The target data is generated by using multiple first block data and the target backup data, and the target data is fed back, so that the first target node that needs to be accessed can be flexibly adjusted according to the load situation of the cluster, thereby generating the required target data. In the case of low load, the complete target data can be restored through one round of requests, which effectively improves the efficiency of data acquisition, reduces the system response time, and ensures the stability of cluster operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0054] Figure 1 is a schematic diagram of a process of obtaining erasure coded data by a related technology;

[0055] Figure 2 is a flowchart of a data processing method based on a distributed storage system provided by an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of a cluster structure provided by an embodiment of the present invention;

[0057] Figure 4 It is a schematic diagram of a process of reading cluster node data to generate target data provided by an embodiment of the present invention;

[0058] Figure 5 It is a structural block diagram of a data device based on a distributed storage system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In distributed storage systems, erasure coding technology is widely used for data redundancy backup and improving fault tolerance. Erasure coding technology encodes the original data into n data blocks and m check blocks, and then stores them in the corresponding data nodes and check nodes respectively. As long as no more than m nodes fail (for example, data is unreadable), the missing data can be restored to the original data by performing erasure coding calculations on the check blocks.

[0061] In the related art, the implementation of erasure code data reading is mainly as follows: Figure 1 As shown:

[0062] 1. Request all data nodes through the proxy server. If all data nodes return successfully, the complete data can be pieced together and returned immediately;

[0063] 2. If a data node request fails, then request the verification node;

[0064] 3. When n nodes return successfully, the original data can be restored and returned through erasure code calculation. When more than m nodes fail to request, the read failure is returned.

[0065] When reading data in a distributed environment, the ideal situation is that all data nodes can be read successfully, and then the complete data can be directly spliced ​​out and returned. However, in actual operation, some problems may occur that cause some data nodes to fail to read successfully. For example, a data node may be offline, or a majority write strategy is used when writing data, but the data fails to be successfully written to a data node for some reason. In these cases, the data reading process may be affected because it is necessary to wait for the failed data node to return the read failure or fault information, and then add read requests to the verification node until a sufficient number of nodes return success. This results in the reading process requiring at least two rounds of requests to complete.

[0066] Although, in order to reduce the reading time as much as possible, we can consider sending requests to both the data node and the check node at the same time in the first round of requests, so that complete data can be obtained in the first round of requests, but this will introduce the problem of read amplification, because the total data size read each time will increase (including the data of the check node). Especially when the node load is high, read amplification will further increase the node load and may cause the system to avalanche.

[0067] Based on this, the present invention proposes a data processing method and device, equipment, and medium based on a distributed storage system, which evaluates the load of the system by monitoring multiple load data. Based on these load data, the data reading strategy can be intelligently adjusted to adapt to different load environments. When the load is low, in order to improve the reading efficiency, the read amplification can be appropriately increased to improve the reading efficiency and quickly respond to user requests. When the load is relatively high, the system will adopt a conservative data reading strategy to avoid read amplification further increasing the system load and ensure the stability and reliability of the system.

[0068] The embodiments of the present invention are described in detail below.

[0069] Reference Figure 2 , shows a flow chart of steps of a data processing method based on a distributed storage system provided by an embodiment of the present invention, wherein the distributed storage system includes at least one cluster, the cluster includes a plurality of first nodes and a plurality of second nodes, the plurality of first nodes are used to store data in blocks, and the plurality of second nodes are used to back up the data stored in the first nodes, and specifically may include the following steps:

[0070] Step 201, in response to a data acquisition request, acquiring load data of the cluster, and determining a first target node among the plurality of first nodes and the plurality of second nodes according to the load data;

[0071] In step 201, the load data of the cluster is obtained to determine the current load condition of the cluster, and then the first target node to be accessed is determined according to the load request. According to different load conditions, the first target node may include the first node and the second node, or may only include the first node. The more nodes that need to be accessed, the greater the pressure on the system load.

[0072] In practical applications, the first node may be a data node that stores the original data in blocks, and the second node may be a verification node that performs data backup after erasure coding the original data.

[0073] In some embodiments of the present invention, the load data includes one or more of central processing unit load data, memory load data and disk load data.

[0074] In a specific implementation, the central processing unit (CPU) load data may be CPU utilization, CPU temperature, CPU frequency and other load data; the memory load data may be memory utilization, memory read and write rate and other load data; the disk load data may be disk read and write rate, hard disk temperature and other load data.

[0075] In practical applications, other types of load data may also be included, such as the time consumption of cluster node requests, etc., which can be flexibly selected according to specific needs.

[0076] In some embodiments of the present invention, step 201 includes:

[0077] Sub-step 2011, determining a load index value of the cluster according to the load data;

[0078] Sub-step 2012, comparing the load index value with a preset first threshold and a preset second threshold to obtain a comparison result; wherein the preset first threshold is greater than the preset second threshold;

[0079] Sub-step 2013: determining a first target node among the plurality of first nodes and the plurality of second nodes according to the comparison result.

[0080] In this embodiment, after the load data is acquired, the current load index value of the cluster can be determined according to the load data through data analysis, importing a model for calculation, etc., so as to quantify the current load condition of the cluster.

[0081] Furthermore, the load conditions of the cluster are divided by using two levels of preset first thresholds and preset second thresholds.

[0082] As an example, assume that the preset first threshold is a, the preset second threshold is b, and a is greater than b. When the load index value is less than or equal to b, the cluster can be considered to be in a low load state; when the load index value is greater than b and less than a, the cluster can be considered to be in a medium load state; when the load index value is less than or equal to a, the cluster can be considered to be in a high load state.

[0083] In some embodiments of the present invention, sub-step 2013 specifically includes:

[0084] If the load index value is less than or equal to the preset second threshold, all the first nodes and all the second nodes are determined as the first target nodes;

[0085] If the load index value is less than the preset first threshold value and greater than the preset second threshold value, all the first nodes and a preset number of the second nodes are determined as the first target nodes;

[0086] If the load indicator value is greater than or equal to the preset first threshold, all the first nodes are determined as the first target nodes.

[0087] In this embodiment, if the load index value is less than or equal to the preset second threshold value, it can be considered that the cluster is in a low load state, so all the first nodes and the second nodes can be accessed at the same time to obtain all the multiple first block data and the backup data. Even if some of the first nodes cannot respond and the multiple first block data cannot be combined into complete data, the complete data can be restored through the backup data returned by the second node;

[0088] If the load index value is less than the preset first threshold and greater than the preset second threshold, the cluster is considered to be in a medium load state. In addition to accessing all first nodes, some second nodes can also be accessed to improve fault tolerance. For example, there are currently n first nodes and m second nodes. When it is confirmed that the load index value is less than the preset first threshold and greater than the preset second threshold, the first block data is obtained through n first nodes, and the backup data is obtained through k (1 < k < m) second nodes. As long as no more than k first nodes fail, the complete data can be restored through the backup data obtained by the second nodes.

[0089] If the load index value is greater than or equal to the preset first threshold, the cluster is considered to be in a high load state. To avoid affecting the stability and operation efficiency of the cluster, at this time, only all the first nodes are determined as the first target nodes.

[0090] Step 202: acquiring, through the first target node, a plurality of first block data corresponding to the data acquisition request;

[0091] In step 202, a request may be initiated to a first target node to obtain a plurality of first block data in response to a data acquisition request.

[0092] Step 203, performing integrity check on the plurality of first block data;

[0093] In step 203, since the plurality of first block data are stored in blocks by the plurality of first nodes respectively, and some of the first nodes may fail to read data, integrity check needs to be performed on the plurality of first block data.

[0094] Step 204: if the integrity check passes, the target data is generated using the plurality of first block data;

[0095] In step 204, if the integrity check passes, the multiple first block data are complete and no additional backup data is needed for recovery. For example, when all first nodes successfully return the first block data, the multiple first block data can be considered complete, and then the multiple first block data are merged to generate target data.

[0096] Step 205: if the integrity check fails, determining missing data corresponding to the plurality of first block data;

[0097] In step 205, if the integrity check fails, the multiple first block data are incomplete and need to be restored using additional backup data, and the missing data corresponding to the multiple first block data need to be determined. For example, among the n first nodes, only the first block data stored in n-2 first nodes are returned, then it can be determined that the missing data is the first block data stored in the two first nodes that are not returned.

[0098] Step 206, acquiring target backup data according to the missing data, and generating target data by using the plurality of first block data and the target backup data;

[0099] In step 206, after the missing data is determined, it is necessary to obtain target backup data based on the missing data, where the target backup data is backup data that can restore the missing data, and the obtained incomplete multiple first block data and the target backup data are used to generate complete target data.

[0100] In a specific implementation, if the first target node includes the second node, the target backup data can be obtained from the data returned by the second node in the first target node; if the first target node does not include the second node, the second node can be additionally accessed to obtain the target backup data.

[0101] In some embodiments of the present invention, when the first target nodes are all the first nodes and all the second nodes, step 202 includes:

[0102] Concurrently initiating multiple first requests to the first target node to obtain multiple first return results;

[0103] Acquire the plurality of first block data from the first return results corresponding to all the first nodes;

[0104] Step 206 includes:

[0105] According to the missing data, the target backup data is obtained from the first returned results corresponding to all the second nodes.

[0106] In this embodiment, when the first target node is all first nodes and all second nodes, it is determined that the cluster is in a low load state, and multiple first requests are initiated to all first nodes and all second nodes. The first request is a data read request to obtain all first block data and all backup data corresponding to the first block data within one round of requests.

[0107] As an example, there are n first nodes storing original data in blocks, and m nodes storing original data encoded with erasure codes. When it is confirmed that the cluster is in a low-load state, n+m first requests are initiated to all first nodes and second nodes, and multiple first return results are obtained.

[0108] If the multiple first returned results include the first block data of n first nodes, the integrity check passes, and the target data can be generated by merging the multiple first block data;

[0109] If there are less than n first block data of the first nodes in the multiple first return results, the integrity check fails, and it is determined that the data is missing, and the target backup data is obtained from the first return result corresponding to the second node according to the missing data. For example, if the first block data of two first nodes is missing, the target backup data is obtained from the first return results returned by the two second nodes, and the target data can be generated by performing erasure code calculation on the incomplete first block data and the target backup data.

[0110] In some embodiments of the present invention, when the first target nodes are all the first nodes and a preset number of the second nodes, step 202 further includes:

[0111] Concurrently initiating multiple second requests to the first target node to obtain multiple second return results;

[0112] Acquire the plurality of first block data from the second returned results corresponding to all the first nodes;

[0113] Step 206 also includes:

[0114] Acquire the first backup data from the second returned results corresponding to the preset number of second nodes;

[0115] Determining whether the first backup data can restore the missing data;

[0116] If the first backup data can restore the missing data, determining the first backup data as the target backup data;

[0117] If the first backup data cannot restore the missing data, determining a second target node from among the plurality of second nodes according to the first backup data and the missing data;

[0118] The second backup data is acquired through the second target node, and the first backup data and the second backup data are determined as the target backup data.

[0119] In this embodiment, when the first target node is all first nodes and all second nodes, it is determined that the cluster is in a medium load state, and multiple second requests are initiated to all first nodes and all second nodes. The second requests are data read requests to obtain all first block data and part of the backup data corresponding to the first block data as redundant data within one round of requests.

[0120] As an example, there are n first nodes storing original data in blocks, and m nodes storing original data encoded with erasure coding. When it is confirmed that the cluster is in a medium load state, multiple second requests are initiated to all first nodes and k second nodes, and multiple second return results are obtained.

[0121] If the multiple second returned results include the first block data of n first nodes, the integrity check passes, and the target data can be generated by merging the multiple first block data;

[0122] If there are fewer than n first block data of the first nodes in the multiple second returned results, the integrity check fails, and the first backup data is obtained from the returned results corresponding to the k second nodes.

[0123] Then, determine whether the first backup data can restore the missing data. For example, when k=3, the first backup data is the backup data stored by the three second nodes. If the missing data is the first block data stored by the three first nodes, the missing data can be restored by the first backup data, and the first backup data can be confirmed as the target backup data. By performing erasure code calculation on n-3 first block data and 3 first backup data, the complete original data, i.e., the target data, can be generated.

[0124] For another example, when k=3, the first backup data is the backup data stored by 3 second nodes. If the missing data is the first block data stored by 4 first nodes, the missing data cannot be restored by the first backup data, and the backup data stored by 1 second node is also needed. Then, a second target node is determined among the remaining mk second nodes, and the second backup data is obtained from the second target node. At this time, the 3 first backup data and 1 second backup data are the target backup data. By performing erasure code calculation on n-4 first block data, 3 first backup data, and 1 second backup data, the complete original data, i.e., the target data, can be generated.

[0125] In some embodiments of the present invention, when the first target nodes are all the first nodes, step 202 further includes:

[0126] Concurrently initiate multiple third requests to the first target node to obtain the multiple first block data;

[0127] Step 206 also includes:

[0128] Concurrently initiate multiple third requests to the first target node to obtain the multiple first block data;

[0129] The acquiring target backup data according to the missing data further comprises:

[0130] Determine a third target node among the plurality of second nodes according to the missing data;

[0131] The target backup data is obtained through the third target node.

[0132] In this embodiment, when the first target nodes are all first nodes, it is confirmed that the cluster is in a high load state. At this time, multiple first block data are obtained by only concurrently initiating data read requests from all first nodes to avoid additional load pressure on the cluster.

[0133] As an example, there are n first nodes storing original data in blocks, and m nodes storing erasure coded original data. When it is confirmed that the cluster is in a high load state, multiple third requests are concurrently initiated only to the n first nodes to obtain multiple first block data.

[0134] If the first block data of n first nodes are obtained, the integrity check passes, and the target data can be generated by merging the multiple first block data;

[0135] If less than n first nodes’ first block data are obtained, the integrity check fails, and it is necessary to determine the third target node from multiple second nodes based on the missing data. For example, if only n-2 first nodes return the first block data, the two second nodes are determined as the third target nodes, and a data read request is initiated to the third target node to obtain the target backup data. Then, the erasure code calculation is performed on the n-2 first block data and the target backup data to obtain the complete original data, i.e., the target data.

[0136] Step 207: Feedback the target data.

[0137] In step 207, after the target data is generated, the target data is fed back to the initiator of the data acquisition request, for example, the target data is fed back to the client, the proxy server, etc.

[0138] In some examples, for ease of understanding, the implementation process of generating target data will be described in conjunction with specific scenarios in a distributed storage system.

[0139] As Figure 3 shown, in a group of clusters, there are n data nodes (i.e., the first nodes) for storing raw data in chunks, and m parity nodes (i.e., the second nodes) for backing up the data stored in the data nodes after encoding by erasure code. When the proxy receives a data reading request from the upper-layer service, it will forward it to the leader node of the corresponding group of nodes, and determine the number of concurrent read requests to the data nodes or parity nodes according to the set load threshold.

[0140] As Figure 4 shown, the specific implementation process of generating the target data based on the data nodes and parity nodes is as follows:

[0141] 1. The proxy receives a data reading request;

[0142] 2. The proxy forwards it to the leader node of the corresponding group of clusters, which includes n data nodes and m parity nodes;

[0143] 3. The leader node makes a judgment according to the current load situation. If it is in a low-load state, it will concurrently send read requests to all data nodes and parity nodes (i.e., n + m read requests), take the data of the first n nodes that successfully return, and calculate through erasure code to restore the original complete data (i.e., the target data), otherwise it returns a read failure;

[0144] 4. If it is currently between high-load and low-load states, it will concurrently send read requests to all data nodes and k parity nodes (i.e., n + k read requests, where 1 < k < m). Even if a data node fails or has no data, as long as no more than k data nodes have problems, we can obtain the data of n nodes in this round of requests, and then calculate to restore the complete data. If there are still less than n node data, it will concurrently request the remaining parity nodes;

[0145] 5. If it is currently in a high-load state, it will only concurrently send read requests to all data nodes (i.e., n read requests). If a data node fails, it will continue to increase the read requests to the parity nodes until n read requests have been successful, otherwise it returns a read failure;

[0146] 6. After the reading is completed, it returns the corresponding target data and the data reading result.

[0147] The above example has the following advantages:

[0148] 1. Dynamically adjust the erasure code data reading strategy according to the load status:

[0149] Dynamically adjust the erasure code data reading strategy according to the system load status. Compared with related technologies, it can more accurately adjust the data reading strategy according to the system load to improve reading efficiency and avoid unnecessary burden on the system.

[0150] 2. Increase read amplification when the system is under low load:

[0151] When the entire system is under low load, read amplification is used to improve the efficiency of data reading. Compared with traditional methods, this method can obtain data more quickly and reduce request waiting time.

[0152] 3. Reduce the number of requests when reading data when the system is under high load:

[0153] When the entire system is under high load, the number of read requests can be appropriately reduced, and read requests are initiated only when necessary to avoid read amplification and prevent system avalanche. Compared with related technologies, this adaptive adjustment can better protect system stability.

[0154] 4. Real-time adjustment:

[0155] Adaptive adjustment is real-time and can be adjusted according to the instant changes in system load. This is more flexible and efficient than periodic or static adjustment methods.

[0156] The embodiments of the present invention have the following advantages: by responding to a data acquisition request, the load data of the cluster is acquired, and a first target node is determined among multiple first nodes and multiple second nodes according to the load data, multiple first block data for the data acquisition request are acquired through the first target node, and integrity verification is performed on the multiple first block data. If the integrity verification passes, the multiple first block data are used to generate target data. If the integrity verification fails, the missing data corresponding to the multiple first block data is determined, and target backup data is acquired according to the missing data. The target data is generated by using multiple first block data and the target backup data, and the target data is fed back, so that the first target node that needs to be accessed can be flexibly adjusted according to the load situation of the cluster, thereby generating the required target data. In the case of low load, the complete target data can be restored through one round of requests, which effectively improves the efficiency of data acquisition, reduces the system response time, and ensures the stability of cluster operation.

[0157] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0158] Reference Figure 5 , shows a schematic diagram of the structure of a data processing device based on a distributed storage system provided by an embodiment of the present invention, the distributed storage system includes at least one cluster, the cluster includes multiple first nodes and multiple second nodes, the multiple first nodes are used to store data in blocks, the multiple second nodes are used to back up the data stored in the first nodes, and the data processing device may specifically include the following modules:

[0159] A first target node determination module 501 is used to obtain the load data of the cluster in response to a data acquisition request, and determine a first target node among the plurality of first nodes and the plurality of second nodes according to the load data;

[0160] A first block data acquisition module 502, configured to acquire, through the first target node, a plurality of first block data corresponding to the data acquisition request;

[0161] An integrity check module 503, configured to perform integrity check on the plurality of first block data;

[0162] A first target data generating module 504, configured to generate target data using the plurality of first block data if the integrity check passes;

[0163] A missing data acquisition module 505 is used to determine the missing data corresponding to the plurality of first block data if the integrity check fails;

[0164] A target data second generating module 506, configured to obtain target backup data according to the missing data, and generate target data using the plurality of first block data and the target backup data;

[0165] The target data feedback module 507 is used to feed back the target data.

[0166] In some embodiments of the present invention, the first target node determination module 501 includes:

[0167] A load index value determination submodule, used to determine the load index value of the cluster according to the load data;

[0168] A load index value comparison submodule, used to compare the load index value with a preset first threshold value and a preset second threshold value to obtain a comparison result; wherein the preset first threshold value is greater than the preset second threshold value;

[0169] The first target node determination submodule is used to determine a first target node among the plurality of first nodes and the plurality of second nodes according to the comparison result.

[0170] In some embodiments of the present invention, the first target node determination submodule further includes:

[0171] A first target node first determination unit, configured to determine all the first nodes and all the second nodes as the first target nodes if the load index value is less than or equal to the preset second threshold;

[0172] a first target node second determination unit, configured to determine all the first nodes and a preset number of the second nodes as the first target nodes if the load index value is less than the preset first threshold and greater than the preset second threshold;

[0173] The first target node third determination unit is configured to determine all the first nodes as the first target nodes if the load indicator value is greater than or equal to the preset first threshold.

[0174] In some embodiments of the present invention, the first block data acquisition module 502 includes:

[0175] The first block data acquisition submodule is used to:

[0176] When the first target node is all the first nodes and all the second nodes, multiple first requests are initiated to the first target node in parallel to obtain multiple first return results;

[0177] Acquire the plurality of first block data from the first return results corresponding to all the first nodes;

[0178] The target data second generation module 506 includes:

[0179] The first acquisition submodule of target backup data is used to:

[0180] When the first target nodes are all the first nodes and all the second nodes, the target backup data is acquired from the first return results corresponding to all the second nodes according to the missing data.

[0181] In some embodiments of the present invention, the first block data acquisition module 502 further includes:

[0182] A second block data acquisition submodule, configured to, when the first target node is all the first nodes and a preset number of the second nodes, initiate multiple second requests to the first target node in parallel to obtain multiple second return results;

[0183] The multiple first block data are obtained from the second returned results corresponding to all the first nodes.

[0184] The target data second generation module 406 further includes:

[0185] The second acquisition submodule of target backup data is used to:

[0186] When the first target nodes are all the first nodes and a preset number of the second nodes, obtaining the first backup data from the second return results corresponding to the preset number of second nodes;

[0187] Determining whether the first backup data can restore the missing data;

[0188] If the first backup data can restore the missing data, determining the first backup data as the target backup data;

[0189] If the first backup data cannot restore the missing data, determining a second target node from among the plurality of second nodes according to the first backup data and the missing data;

[0190] The second backup data is acquired through the second target node, and the first backup data and the second backup data are determined as the target backup data.

[0191] In some embodiments of the present invention, the first block data acquisition module 502 further includes:

[0192] The third block data acquisition submodule is used to:

[0193] When the first target nodes are all the first nodes, a plurality of third requests are concurrently initiated to the first target nodes to obtain the plurality of first block data;

[0194] The target data second generation module 506 also includes:

[0195] The third acquisition submodule of target backup data is used to:

[0196] When the first target nodes are all the first nodes, determining a third target node from the plurality of second nodes according to the missing data;

[0197] The target backup data is obtained through the third target node.

[0198] Some embodiments of the present invention also provide an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the method of data processing based on a distributed storage system as described above is implemented.

[0199] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for data processing based on a distributed storage system as described above is implemented.

[0200] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above method for data processing based on a distributed storage system when executed by a processor.

[0201] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0202] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0203] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0204] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0205] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0206] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0207] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0208] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0209] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0210] The data processing method, device, equipment, and medium based on the distributed storage system are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method based on a distributed storage system, characterized in that: The distributed storage system includes at least one cluster, the cluster includes a plurality of first nodes and a plurality of second nodes, the plurality of first nodes are used to store data in blocks, and the plurality of second nodes are used to back up data stored in the first nodes, and the method includes: In response to a data acquisition request, acquiring load data of the cluster, and determining a first target node among the plurality of first nodes and the plurality of second nodes according to the load data; Acquire, through the first target node, a plurality of first block data for the data acquisition request; Performing integrity check on the plurality of first block data; If the integrity check passes, the target data is generated using the plurality of first block data; If the integrity check fails, determining missing data corresponding to the plurality of first block data; Acquire target backup data according to the missing data, and generate target data using the plurality of first block data and the target backup data; The target data is fed back.

2. The method according to claim 1, characterized in that The determining a first target node among the plurality of first nodes and the plurality of second nodes according to the load data comprises: Determine a load index value of the cluster according to the load data; Comparing the load index value with a preset first threshold and a preset second threshold to obtain a comparison result; wherein the preset first threshold is greater than the preset second threshold; A first target node is determined among the plurality of first nodes and the plurality of second nodes according to the comparison result.

3. The method according to claim 2, characterized in that The determining a first target node from among the plurality of first nodes and the plurality of second nodes according to the comparison result includes: If the load index value is less than or equal to the preset second threshold, all the first nodes and all the second nodes are determined as the first target nodes; If the load index value is less than the preset first threshold value and greater than the preset second threshold value, all the first nodes and a preset number of the second nodes are determined as the first target nodes; If the load indicator value is greater than or equal to the preset first threshold, all the first nodes are determined as the first target nodes.

4. The method according to claim 3, characterized in that When the first target node is all the first nodes and all the second nodes, acquiring a plurality of first block data for the data acquisition request through the first target node includes: Concurrently initiating multiple first requests to the first target node to obtain multiple first return results; Acquire the plurality of first block data from the first return results corresponding to all the first nodes; The acquiring target backup data according to the missing data comprises: According to the missing data, the target backup data is obtained from the first returned results corresponding to all the second nodes.

5. The method according to claim 3, characterized in that: When the first target nodes are all the first nodes and a preset number of the second nodes, the acquiring, through the first target nodes, a plurality of first block data for the data acquisition request further includes: Concurrently initiating multiple second requests to the first target node to obtain multiple second return results; Acquire the plurality of first block data from the second returned results corresponding to all the first nodes; The acquiring target backup data according to the missing data further comprises: Acquire the first backup data from the second returned results corresponding to the preset number of second nodes; Determining whether the first backup data can restore the missing data; If the first backup data can restore the missing data, determining the first backup data as the target backup data; If the first backup data cannot restore the missing data, determining a second target node from among the plurality of second nodes according to the first backup data and the missing data; The second backup data is acquired through the second target node, and the first backup data and the second backup data are determined as the target backup data.

6. The method according to claim 3, characterized in that When the first target nodes are all the first nodes, acquiring the plurality of first block data for the data acquisition request through the first target nodes further includes: Concurrently initiate multiple third requests to the first target node to obtain the multiple first block data; The acquiring target backup data according to the missing data further comprises: Determine a third target node among the plurality of second nodes according to the missing data; The target backup data is obtained through the third target node.

7. The method according to any one of claims 1 to 6, characterized in that: The load data includes one or more of central processing unit load data, memory load data and disk load data.

8. A data processing device based on a distributed storage system, characterized in that: The distributed storage system includes at least one cluster, the cluster includes a plurality of first nodes and a plurality of second nodes, the plurality of first nodes are used to store data in blocks, and the plurality of second nodes are used to back up the data stored in the first nodes, and the device includes: a first target node determination module, configured to obtain load data of the cluster in response to a data acquisition request, and determine a first target node among the plurality of first nodes and the plurality of second nodes according to the load data; A first block data acquisition module, used for acquiring a plurality of first block data corresponding to the data acquisition request through the first target node; An integrity check module, used for performing integrity check on the plurality of first block data; A first target data generating module, configured to generate target data using the plurality of first block data if the integrity check passes; A missing data acquisition module, configured to determine the missing data corresponding to the plurality of first block data if the integrity check fails; a second target data generating module, configured to obtain target backup data according to the missing data, and generate target data using the plurality of first block data and the target backup data; The target data feedback module is used to feed back the target data.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for processing data based on a distributed storage system as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method based on a distributed storage system according to any one of claims 1 to 7 is implemented.

Images