A data storage method and apparatus

Abstract

Embodiments of the present application provide a data storage method and device, a control center receives a data storage request; the control center determines a first storage node storing a master copy from each node array of an optimal node array set, and determines N second storage nodes storing a copy from each node array having the first storage node; wherein each storage node in the same node array is located in different racks, and the optimal node array set is a grouping result of array pair statistical values being minimum in T times of random grouping of each storage node; the array pair statistical value is used to count the number of array pairs in which the number of same storage nodes in each grouping is greater than a set threshold; and the control center stores data corresponding to the data storage request to the first storage node and N second storage nodes respectively. The above method can reduce the data loss rate while ensuring the data repair speed.

Description

Technical Field

[0001] This application relates to the fields of network technology and computer technology in financial technology (Fintech), and in particular to a data storage method and apparatus. Background Technology

[0002] In recent years, with the development of computer technology, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology. For example, to accelerate the processing speed of related business transactions, increase the convenience of business processing, and enhance the security of data storage, large amounts of business data are generally stored in computers. However, if the computer malfunctions, it can lead to the complete loss of data, causing huge losses.

[0003] To address the potential for total data loss when storing data on a single computer, existing technologies employ two approaches. Taking the example of each data point having a primary and a replica stored across four storage nodes: One approach involves backing up each piece of data, placing the primary copy on one computer and the replica on another. This ensures data integrity even if one computer fails, as long as the computer storing the primary or replica remains operational, data loss is minimized. However, because this approach stores the primary copy on one computer and the replica on a separate computer, simultaneous failures of both will inevitably result in total data loss. Therefore, while this approach has a low data loss rate, it suffers from significant data loss. Furthermore, even if only one computer fails without data loss, centralized data recovery leads to slow recovery speeds. The other approach involves backing up each piece of data, with the primary and replica stored randomly across four computers. Thus, when a computer malfunctions, although the two malfunctioning computers are likely to cause data loss, the probability of multiple data being lost simultaneously on the primary and replica computers is low. Therefore, although this method has a high data loss rate, the amount of data lost is small, and because the data is not concentrated, the repair speed is faster.

[0004] Therefore, there is an urgent need for a data storage method and device that can reduce the data loss rate while ensuring data recovery speed. Summary of the Invention

[0005] This invention provides a data storage method and apparatus that can reduce data loss rate while ensuring data repair speed.

[0006] In a first aspect, embodiments of the present invention provide a data storage method, the method comprising:

[0007] The control center receives data storage requests;

[0008] The control center determines the first storage node for storing the primary copy from each node array in the optimal node array set, and determines N different second storage nodes for storing the replica from each node array containing the first storage node; wherein, the storage nodes in the same node array are located in different racks, and the optimal node array set is the grouping result with the smallest array pair statistical value in T random groupings for each storage node; the array pair statistical value is used to count the number of array pairs in each group where the number of identical storage nodes is greater than a set threshold; the control center stores the data corresponding to the data storage request to the first storage node and the N second storage nodes respectively.

[0009] In the above method, storage nodes located in the same node array are placed in different racks. This effectively prevents the situation where all storage nodes in the first subarray belong to the same rack, which could lead to the simultaneous loss of both the primary and replica data should the rack's routing be damaged. Therefore, placing storage nodes in the same node array in different racks effectively reduces data loss. The optimal node array set is determined by the statistical value of array pairs where the number of identical storage nodes between any two node arrays exceeds a set threshold. Since the statistical value of array pairs in the optimal node array is the smallest among T random groupings for each storage node, this means that for any two node arrays in the optimal node array set, the number of selectable storage nodes for the replica is increased; that is, the optimal node array set is the node array set with the highest dispersion. Thus, the control center randomly selects a node number from each node array in the optimal node array set to store the primary data; the storage node corresponding to this node number is the first storage node. The replica of this data is then randomly stored in the second storage node corresponding to another node number in each node array of the first storage node, and the number of other node numbers is equal to the highest dispersion. Therefore, while ensuring the dispersion of replicas, i.e., ensuring the repair speed, the number of combinations of N different second storage nodes randomly selected for each replica is reduced, i.e., the loss rate is reduced.

[0010] Optionally, the optimal node array set is the grouping result with the smallest array pair statistical value among T random groupings for each storage node, including:

[0011] For any random grouping of storage nodes, the number P of random node source arrays is determined based on the number R of storage nodes storing the same data; wherein, the random node source arrays contain the node numbers of each storage node, and the order of the node numbers in any two random node source arrays is different.

[0012] For any random node source array, the random node source array is divided into multiple node arrays, wherein each node array has R storage nodes and each storage node in each node array is located in a different rack;

[0013] For each node array in the P random node source arrays, determine the number of identical storage nodes between any two node arrays; if the number of identical storage nodes is greater than the set threshold, increment the array count by 1.

[0014] The optimal node array set is determined from the P random node source arrays that have the smallest array pair statistical value in the T random groupings.

[0015] In the above method, during the process of obtaining the optimal node array set, each node array contains R storage nodes, and the storage nodes in each node array are located in different racks. This reduces the number of simultaneous losses of the primary and replica data due to rack routing failures. For any random grouping of storage nodes, the number P of random node source arrays is determined based on the number R of storage nodes storing the same data. This effectively limits the number of second storage nodes corresponding to the first storage node in the obtained optimal node array set, balancing the relationship between the number of second storage nodes and the number of primary and replica data (R) (or the number of storage nodes storing the same data). This balance ensures both data recovery speed and reduced loss rate. The random node source arrays contain the node numbers of each storage node, and the order of the node numbers in any two random node source arrays is different. This effectively reduces the duplication of storage nodes between node arrays, ensuring the dispersion of replicas. The node arrays among the P random node source arrays with the smallest array pair statistics in T random groupings are determined as the optimal node array set. This ensures that the optimal set of nodes used for data storage has the highest data dispersion, which means that the data loss rate is reduced while ensuring data dispersion. Here, dispersion can be used to measure the speed of data repair.

[0016] Optionally, the racks where each storage node is located are obtained through the following methods:

[0017] The control center receives startup success notifications from each storage node, and the startup success notifications contain the rack number corresponding to the storage node.

[0018] The control center assigns a node number to each storage node and records the node number and rack number of each storage node.

[0019] In the above method, the successful startup notification includes the rack number corresponding to the storage node. This facilitates the control center's subsequent verification of whether storage nodes in the same node array belong to the same rack, effectively preventing a situation where all storage nodes in the node array belong to the same rack, which could lead to the simultaneous loss of both the primary and replica data if the rack's routing is damaged. Therefore, it can effectively reduce the amount of data loss.

[0020] Optionally, it also includes: if at least two storage nodes in the node array of the random node source array are located in the same rack, the control center randomly sorts the node numbers according to a random algorithm to obtain a new random node source array, until the storage nodes of each node array in the random node source array are located in different racks.

[0021] In the above method, the master is stored in the first storage node, and the replica is stored in the second storage node within the node array corresponding to the first storage node. By comparing the rack numbers of each storage node in each node array, if at least two storage nodes in the node array have the same rack number, the node array is re-acquired until the rack numbers of the storage nodes in each node array are different. This effectively prevents situations where at least two storage nodes in the node array belong to the same rack, which could lead to the simultaneous loss of both the master and replica of the same data if the rack routing is damaged. Therefore, it can effectively reduce the amount of data loss.

[0022] Optionally, the random node source array is divided into multiple node arrays, including: if the number of node numbers in the first node array is less than R, then a node number is randomly selected from the second node array and added to the first node array to make up the number of node numbers in the first node array, wherein the second node array is the other node arrays among the multiple node arrays divided from the random node source array.

[0023] In the above method, if the number of node numbers in the first node array is less than R, then a node number is randomly selected from the second node array and added to the first node array to make up the number of node numbers in the first node array. This ensures the number of storage nodes corresponding to the node numbers in the first node array, further guaranteeing the dispersion, i.e., ensuring the data repair speed.

[0024] Optionally, when a new storage node is added, the method further includes: the control center assigning a node number to the new storage node; the control center creating a new node array for each random node source group in the optimal node array set, adding the node number of the new storage node to each new node array respectively, and selecting node numbers from the random node source array to fill the number of node numbers in the new node array, wherein the number of the same storage nodes in the new node array and any node array in the optimal node array set is not greater than the set threshold.

[0025] In the above method, when a new storage node is added, the number of identical storage nodes between the newly added node array and any node array in the optimal node array set is kept below a preset threshold. This ensures fast data repair and reduces the data loss rate.

[0026] Optionally, the method further includes: the control center obtaining the data to be copied corresponding to the node number of the storage node to be deleted from the data storage location record, wherein the data storage location record contains the correspondence between the node number of each storage node and the data identifier of the data stored in each storage node; the control center copying the data to be copied from the storage node corresponding to the data to be copied; and the control center determining a third storage node from each node array of the optimal node array set other than the node array where the storage node to be deleted is located, and storing the data to be copied in the third storage node.

[0027] In the above method, when a storage node is deleted, the control center copies the data to be copied from the storage node corresponding to the data to be copied; the control center then stores the data to be copied in a randomly selected third storage node. This ensures fast data recovery and reduces the data loss rate.

[0028] In a second aspect, embodiments of the present invention provide a data storage device, the device comprising:

[0029] The receiving module is used to receive data storage requests;

[0030] The processing unit is configured to determine a first storage node to be used as primary storage from each node array in the optimal node array set, and to determine N different second storage nodes to be used as replica storage from each node array containing the first storage node; wherein the storage nodes located in the same node array are located in different racks, and the optimal node array set is the grouping result with the smallest array pair statistical value in T random groupings for each storage node; the array pair statistical value is used to count the number of array pairs in each group where the number of identical storage nodes is greater than a set threshold;

[0031] The processing unit is further configured to store the data corresponding to the data storage request to the first storage node and N second storage nodes respectively.

[0032] Thirdly, embodiments of this application also provide a computing device, including: a memory for storing a program; and a processor for calling the program stored in the memory and executing the method described in various possible designs of the first aspect according to the obtained program.

[0033] Fourthly, embodiments of this application also provide a computer-readable non-volatile storage medium including a computer-readable program that, when read and executed by a computer, causes the computer to perform the method described in various possible designs of the first aspect.

[0034] These or other implementations of this application will become clearer and easier to understand in the following description of the embodiments. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A schematic diagram of a data storage architecture provided for an embodiment of the present invention;

[0037] Figure 2 A flowchart illustrating a data storage method provided in an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a node array structure provided in an embodiment of the present invention;

[0039] Figure 4 A flowchart illustrating a data storage method provided in an embodiment of the present invention;

[0040] Figure 5 A flowchart illustrating a data storage method provided in an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram of a data storage device provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0043] Figure 1 In a data storage system architecture provided by an embodiment of the present invention, application 101 sends a data storage request to control center 102. Upon receiving the data storage request, control center 102 determines a first storage node from the node arrays of the optimal node array set, and stores the copy corresponding to the data storage request in the first storage node. Control center 102 then obtains the node arrays corresponding to the first storage node, determines N different second storage nodes from each node array, and randomly stores the copy corresponding to the data storage request in the N different second storage nodes. The storage node cluster 103 contains multiple storage nodes, and the node numbers of each storage node in the storage node cluster 103 form the node arrays of the optimal node array set.

[0044] Based on this, embodiments of this application provide a data storage method flow, such as... Figure 2 As shown, it includes:

[0045] Step 201: The control center receives a data storage request;

[0046] Here, data storage requests can be automatically triggered by applications or application systems, such as data storage requests for storing logs; or they can be generated based on user-triggered file uploads, image uploads, etc. Data storage requests can be generated in any way; there are no restrictions on how data storage requests are generated here.

[0047] Step 202: The control center determines the first storage node for storing the primary machine from each node array in the optimal node array set, and determines N different second storage nodes for storing replicas from each node array containing the first storage node; wherein, the storage nodes in the same node array are located in different racks, and the optimal node array set is the grouping result with the smallest array pair statistical value in T random groupings for each storage node; the array pair statistical value is used to count the number of array pairs in each group where the number of identical storage nodes is greater than a set threshold;

[0048] Step 203: The control center stores the data corresponding to the data storage request to the first storage node and N second storage nodes respectively.

[0049] In the above method, storage nodes located in the same node array are placed in different racks. This effectively prevents the situation where all storage nodes in the first subarray belong to the same rack, which could lead to the simultaneous loss of both the primary and replica data should the rack's routing be damaged. Therefore, placing storage nodes in the same node array in different racks effectively reduces data loss. The optimal node array set is determined by the statistical value of array pairs where the number of identical storage nodes between any two node arrays exceeds a set threshold. Since the statistical value of array pairs in the optimal node array is the smallest among T random groupings for each storage node, this means that for any two node arrays in the optimal node array set, the number of selectable storage nodes for the replica is increased; that is, the optimal node array set is the node array set with the highest dispersion. Thus, the control center randomly selects a node number from each node array in the optimal node array set to store the primary data; the storage node corresponding to this node number is the first storage node. The replica of this data is then randomly stored in the second storage node corresponding to another node number in each node array of the first storage node, and the number of other node numbers is equal to the highest dispersion. Therefore, while ensuring the dispersion of replicas, i.e., ensuring the repair speed, the number of combinations of N different second storage nodes randomly selected for each replica is reduced, i.e., the loss rate is reduced.

[0050] This application provides a method for obtaining an optimal node array set, wherein the optimal node array set is the grouping result with the smallest statistical value of array pairs among T random groupings of each storage node, including:

[0051] For any random grouping of storage nodes, the number P of random node source arrays is determined based on the number R of storage nodes storing the same data; wherein the random node source arrays contain the node numbers of each storage node, and the order of the node numbers in any two random node source arrays is different; for any random node source array, the random node source array is divided into multiple node arrays, wherein each node array has R storage nodes and the storage nodes in each node array are located in different racks; for each of the P random node source arrays, the number of identical storage nodes between any two node arrays is determined; if the number of identical storage nodes is greater than the set threshold, the array pair statistics are incremented by 1; the node arrays of the P random node source arrays with the smallest array pair statistics in T random groupings are determined as the optimal node array set.

[0052] Here, based on Figure 3The diagram showing the node array structure provides an example. If the node numbers of the storage nodes corresponding to the control center are 1, 2, 3, 4, 5, 6, 7, 8, and 9; where storage nodes 1 and 5 correspond to rack r1, storage nodes 2 and 6 to rack r2, storage nodes 3 and 7 to rack r3, storage nodes 4 and 8 to rack r4, and storage node 9 to rack r5; and the number of storage nodes storing the same data is R = 3 (i.e., the sum of the number of primary and replica data sets is 3), and the number of random groupings is T = 3, with a threshold of 0, then the number of random node source arrays can be determined using the formula: P = N / (R-1). Here, N represents the dispersion, which is set to 4, so P equals 2.

[0053] Further, during the T=1 random grouping process, the following random node source array 1 = {1-r1, 3-r3, 4-r4, 2-r2, 5-r1, 7-r3, 6-r2, 8-r4, 9-r5} is obtained. This array is then split into R node arrays, i.e., 3 node arrays: node array 1-{1-r1, 3-r3, 4-r4}, node array...

[0054] 2-{2-r2,5-r1,7-r3}, node array 3-{6-r2,8-r4,9-r5}; random node source array 2 = {4-r4,5-r1,6-r2,1-r1,8-r4,9-r5,2-r2,3-r3,7-r3}, split into 3 node arrays: node array 1-{4-r4,5-r1,6-r2}, node array 2-{1-r1,8-r4,9-r5}, node array 3-{2-r2,3-r3,7-r3}. However, storage nodes 3 and 7 in the random node source array 2 and node array 3 belong to the same rack r3. Therefore, they need to be randomly grouped again. Furthermore, the number of identical storage nodes in the node array 3-{6-r2,8-r4,9-r5} in the random node source array 1 and the node array 2-{1-r1,8-r4,9-r5} in the random node source array 2 is greater than the set threshold 0. That is, the array pair consists of node array 3 in the random node source array 1 and node array 2 in the random node source array 2. Therefore, the array pair statistics are incremented by 1. The array pair statistics for T=1 random groupings are recorded as 1.

[0055] The random node source array 3 = {1-r1, 3-r3, 4-r4, 2-r2, 5-r1, 7-r3, 6-r2, 8-r4, 9-r5} is obtained during the T=2 random grouping process. This array is then split into R node arrays, i.e., 3 node arrays: node array 1-{1-r1, 3-r3, 4-r4}, node array...

[0056] 2-{2-r2,5-r1,7-r3}, node array 3-{6-r2,8-r4,9-r5}; random node source array 4 = {1-r1,8-r4,2-r2,3-r3,5-r1,4-r4,6-r2,7-r3,9-r5}, split into 3 node arrays: node array 1-{1-r1,8-r4,2-r2}, node array 2-{3-r3,5-r1,4-r4}, node array 3-{6-r2,7-r3,9-r5}. If no node array in random node source array 3 contains storage nodes with the same rack number; and no node array in random node source array 4 contains storage nodes with the same rack number; and the number of identical storage nodes in node arrays 3-{6-r2,8-r4,9-r5} in random node source array 3 and node arrays 3-{6-r2,7-r3,9-r5} in random node source array 4 is greater than the set threshold 0, that is, the array pair is defined as node array 3 in random node source array 3 and node array 3 in random node source array 2; then the array pair statistical value is incremented by 1. Record the array pair statistical value as 1 for T = 2 random groupings.

[0057] The random node source array 5 = {1-r1, 3-r3, 4-r4, 2-r2, 5-r1, 7-r3, 6-r2, 8-r4, 9-r5} is obtained during the T=3 random grouping process. This is then split into R node arrays, i.e., 3 node arrays: node array 1-{1-r1, 3-r3, 4-r4}, node array...

[0058] 2-{2-r2,5-r1,7-r3}, node array 3-{6-r2,8-r4,9-r5}; random node source array 6 = {1-r1,8-r4,2-r2,3-r3,5-r1,9-r5,6-r2,7-r3,4-r4}, split into 3 node arrays: node array 1-{1-r1,8-r4,2-r2}, node array 2-{3-r3,5-r1,9-r5}, node array 3-{6-r2,7-r3,4-r4}. If no node array in random node source array 5 contains storage nodes with the same rack number; if no node array in random node source array 6 contains storage nodes with the same rack number; and if no node array in random node source array 5 has the same storage node as the node array in random node source array 2, then the number of identical storage nodes is greater than the set threshold 0, and the array pair statistics are incremented by 0. The statistics for array pairs with T = 3 random groupings are recorded as 0.

[0059] Based on the above three random groupings T=3, the following statistics are obtained: the array pair statistics for T=1 random groupings are 1, the array pair statistics for T=2 random groupings are 1, and the array pair statistics for T=3 random groupings are 0. In the T=1 random groupings, if different storage nodes within the node arrays have the same rack number, they are not used; otherwise, if that rack fails, storage nodes 3 and 7 in the node array 3 of the random node source array 2 of the T=1 random groupings will fail simultaneously, increasing the data loss rate. Since the array pair statistics for T=3 random groupings are less than those for T=2 random groupings, the node arrays of random node source array 1 and random node source array 2 obtained from the T=3 random groupings are selected as the optimal node array set. In the process of generating random node source arrays to obtain the optimal node array set, the optimal node source array is used where any two storage nodes in each node array do not belong to the same rack. This prevents data loss due to rack failure causing simultaneous failure of storage nodes in that rack, resulting in the loss of both the primary and replica copies of the same data. This effectively reduces the amount of data loss. Furthermore, it ensures that the number of identical storage nodes between any two node arrays in the optimal node source array does not exceed a set threshold, guaranteeing data dispersion and thus ensuring data recovery speed.

[0060] This application provides a method for obtaining the node number and rack number of a storage node. The rack number of each storage node is obtained through the following steps: the control center receives a startup success notification from each storage node, the startup success notification containing the rack number corresponding to the storage node; the control center assigns a node number to each storage node and records the node number and rack number of each storage node. In other words, the storage node sends a startup success notification to the control center carrying the rack number. After receiving the startup success notification, the control center assigns a node number to the storage node and stores the node number and rack number of the storage node accordingly. In one example, the storage node contains a running process and a configuration file; after the process starts, it obtains and applies the rack number from the configuration file and sends a startup success notification containing the rack number to the control center. This example is only one implementation of sending a startup success notification and does not limit the method of sending startup success notifications.

[0061] This application provides a method for locating storage nodes in different racks, further comprising: if at least two storage nodes in a node array of the random node source array are located in the same rack, the control center randomly sorts the node numbers according to a random algorithm to obtain a new random node source array, until the storage nodes of each node array in the random node source array are located in different racks. In the above example, if at least two storage nodes in a node array of the random node source array 2 obtained in T=1 times are located in the same rack, the control center randomly sorts the node numbers according to a random algorithm to obtain new random node source arrays 3 and 4, and determines that the storage nodes of each node array in random node source arrays 3 and 4 are located in different racks. This ensures that no two node arrays contain storage nodes located in the same rack, improving data security and preventing the loss of both the primary and replica copies of a single piece of data due to rack failure, thus preventing an increase in data loss.

[0062] This application provides a method for padding the number of node numbers in a node array. The method divides the random node source array into multiple node arrays, including: if the number of node numbers in a first node array is less than R, then randomly selecting node numbers from a second node array and adding them to the first node array to pad the number of node numbers in the first node array. The second node array is any of the multiple node arrays divided from the random node source array excluding the first node array. In one example, the node numbers of the storage nodes corresponding to the control center are 1, 2, 3, 4, 5, 6, 7, and 8; R = 3, then node arrays 1-{1, 2, 3}, 2-{4, 5, 6}, and 3-{7, 8} are obtained; node array 3 with a number of node numbers less than 3 is the first node array, and node arrays 1 and 2 with a number of node numbers equal to 3 are the second node arrays. Then, a node number is randomly selected from the second node array and added to the first node array to complete the number of node numbers in the first node array, resulting in a completed node array 3-{7, 8, 6} or 3-{7, 8, 4}, etc. In this way, by completing the node array as described above, each node array has R storage nodes, ensuring the number of second storage nodes for subsequent replica storage. That is, increasing the dispersion effectively guarantees the repair speed and reduces the loss rate.

[0063] This application provides a method for adding a new storage node. When adding a new storage node, the method further includes: the control center assigning a node number to the new storage node; the control center creating a new node array for each random node source group in the optimal node array set, adding the node number of the new storage node to each new node array, and selecting node numbers from the random node source array to fill the number of node numbers in the new node array. The number of storage nodes in the new node array that are the same as any node array in the optimal node array set is not greater than a set threshold. Based on the above example, the node numbers of the storage nodes corresponding to the control center are 1, 2, 3, 4, 5, 6, 7, 8, and 9, respectively; where storage node 1 and storage node 5 correspond to rack r1, storage node 2 and storage node 6 correspond to rack r2, storage node 3 and storage node 7 correspond to rack r3, storage node 4 and storage node 8 correspond to rack r4, and storage node 9 corresponds to rack r5. The number of storage nodes storing the same data is R = 3. The set threshold is 0. Then the control center assigns a node number of 10 to the new storage node, and the corresponding rack number is r5. The control center creates a new node array for random node source groups 5 and 6 in the optimal node array set. During this process, it must be ensured that the number of identical storage nodes between the new node array and any node array in the optimal node array set is no greater than 0, and that the storage nodes within the new node array do not belong to the same rack. Therefore, a new node array 4-{10-r5,1-r1,7-r3} is created for random node source group 5 in the optimal node array set; and a new node array 4-{10-r5,2-r2,8-r3} is created for random node source group 6 in the optimal node array set. The final optimal set of node arrays is the random node source array 5: node array 1-{1-r1,3-r3,4-r4}, node array 2-{2-r2,5-r1,7-r3}, node array 3-{6-r2,8-r4,9-r5}, node array 4-{10-r5,1-r1,7-r3}; and the random node source array 6: node array 1-{1-r1,8-r4,2-r2}, node array 2-{3-r3,5-r1,9-r5}, node array 3-{6-r2,7-r3,4-r4}, and 4-{10-r5,2-r2,8-r3}. Specifically, each node array in the random node source array 5 still does not contain storage nodes with the same rack number; each node array in the random node source array 6 still does not contain storage nodes with the same rack number; and the node arrays in the random node source array 5 and the node arrays in the random node source array 2 still do not contain the same storage nodes.

[0064] This application provides a method for deleting nodes, including:

[0065] The control center obtains the data to be copied corresponding to the node number of the storage node to be deleted from the data storage location record. The data storage location record contains the correspondence between the node number of each storage node and the data identifier of the data stored in each storage node. The control center copies the data to be copied from the storage node corresponding to the data to be copied. The control center determines a third storage node from each node array in the optimal node array set other than the node array where the storage node to be deleted is located, and stores the data to be copied in the third storage node. Based on the above example, if the node numbers of the storage nodes corresponding to the control center are 1, 2, 3, 4, 5, 6, 7, 8, and 9 respectively, and storage node 9 is the storage node to be deleted, then the control center obtains the data to be copied A and data to be copied B corresponding to the node number of the storage node to be deleted from the data storage location record; and determines that data to be copied A is still stored in storage node 7 and data to be copied B is still stored in storage node 8; then the control center copies data to be copied A and data to be copied B from storage node 7 and storage node 8 respectively; the control center can store data to be copied A and data to be copied B in any one of the third storage nodes 1-6.

[0066] Based on the above method flow, embodiments of this application provide a data storage method flow, such as... Figure 4 As shown, it includes:

[0067] Step 401: The process in the storage node uses the rack number in the configuration file to start the storage node.

[0068] Step 402: The process generates a startup success notification based on the rack number.

[0069] Step 403: The process will send a successful startup notification to the control center.

[0070] Step 404: The control center receives a startup success notification from the process.

[0071] Step 405: The control center determines and acquires all corresponding storage nodes, assigns a node number to each storage node, and stores the node number and rack number of each storage node in a corresponding manner. The node number can be an integer, and the control center can assign numbers to each storage node in ascending order; alternatively, it can be in descending order. Furthermore, the order in which the storage nodes are assigned numbers can be determined by the time sequence of the received successful startup notifications, or it can be randomly determined. The specific order of the assigned numbers is not limited here.

[0072] Step 406: The control center receives configuration parameters: set the threshold, R: the number of storage nodes storing the same data, the number of cycles T, and the dispersion N, which means that the replica is randomly stored in N different second storage nodes.

[0073] Step 407: The control center executes the initialization command.

[0074] Step 408: The control center calculates the number P of the random node source array obtained, that is, P = N / (R-1).

[0075] Step 409: The control center obtains a source array of P random nodes with different node numbering orders based on the random algorithm and the node number of each storage node.

[0076] Step 410: Divide each random node source array into units of R node numbers, ensuring that each node array in the random node source array contains R node numbers. If a first node array contains fewer than R node numbers, randomly select node numbers from the second node array and add them to the first node array to make up the number of node numbers in the first node array. The second node array is the array of nodes other than the first node array among the multiple node arrays divided from the random node source array.

[0077] Step 411: For each node array in the source array of each random node, the control center determines whether each node array contains a storage node with the same rack number. If it does, proceed to step 409; otherwise, proceed to step 412.

[0078] Step 412: For each node array in the P random node source arrays, determine the number of identical storage nodes between any two node arrays; if the number of identical storage nodes is greater than a set threshold, increment the array pair statistics by 1. For example, if the final number of array pairs with the number of identical storage nodes greater than the set threshold is 3, then the array pair statistics are incremented to 3.

[0079] Step 413: Store the node array set and array pair statistics obtained in this loop.

[0080] Step 414: Increment T by 1, and determine whether T is equal to the preset value. If it is less than the preset value, proceed to step 409; otherwise, proceed to step 415.

[0081] Step 415: Obtain the array pair statistics for each loop, and select the node array set corresponding to the smallest array pair statistics as the optimal node array set, and apply this optimal node array set.

[0082] Based on the above method flow, embodiments of this application provide a data storage method flow, such as... Figure 5As shown, it includes:

[0083] Step 501: The control center receives a request to add a new storage node.

[0084] Step 502: The control center assigns a node number to the newly added storage node.

[0085] Step 503: The control center creates a new node array for each random node source group in the optimal node array set, and adds the node number of the newly added storage node to each new node array.

[0086] Step 504: The control center selects node numbers from the random node source array to supplement the number of node numbers in the newly added node array; wherein, the number of identical storage nodes in the newly added node array and any node array in the optimal node array set is not greater than a set threshold.

[0087] Step 505: The control center receives a request to delete a storage node.

[0088] Step 506: The control center obtains the data to be copied corresponding to the node number of the storage node to be deleted from the data storage location record; wherein, the data storage location record contains the correspondence between the node number of each storage node and the data identifier of the data stored in each storage node.

[0089] Step 507: The control center copies the data to be copied from the storage node corresponding to the data to be copied.

[0090] Step 508: The control center determines the third storage node from all node arrays in the optimal node array set except for the node array containing the storage node to be deleted, and stores the data to be copied in the third storage node.

[0091] It should be noted that the above process steps are not unique. For example, steps 501 to 504 and steps 505 to 508 are respectively the process of adding and deleting storage nodes. Steps 501 to 504 and steps 505 to 508 may not occur simultaneously; that is, only one of them may occur.

[0092] Based on the same concept, embodiments of the present invention provide a data storage device. Figure 6 A schematic diagram of a data storage device provided in an embodiment of this application is shown below. Figure 6 The following are examples:

[0093] Receiver module 601 is used to receive data storage requests;

[0094] Processing unit 602 is configured to determine a first storage node as primary storage from each node array in the optimal node array set, and to determine N different second storage nodes as replica storage from each node array containing the first storage node; wherein, the storage nodes in the same node array are located in different racks, and the optimal node array set is the grouping result with the smallest array pair statistical value in T random groupings for each storage node; the array pair statistical value is used to count the number of array pairs in each group where the number of identical storage nodes is greater than a set threshold;

[0095] The processing unit 602 is further configured to store the data corresponding to the data storage request to the first storage node and N second storage nodes respectively.

[0096] Optionally, the processing unit 602 is specifically configured to: for any random grouping of storage nodes, determine the number P of random node source arrays based on the number R of storage nodes storing the same data; wherein the random node source arrays contain the node numbers of each storage node, and the order of the node numbers in any two random node source arrays is different; for any random node source array, divide the random node source array into multiple node arrays, wherein each node array has R storage nodes and each storage node in each node array is located in a different rack; for each node array in the P random node source arrays, determine the number of identical storage nodes between any two node arrays; if the number of identical storage nodes is greater than the set threshold, increment the array pair statistics by 1; and determine the node arrays in the P random node source arrays with the smallest array pair statistics in the T random groupings as the optimal node array set.

[0097] Optionally, the receiving module 601 is specifically used to: receive a startup success notification sent by each storage node, the startup success notification containing the rack number corresponding to the storage node; and set a node number for each storage node and record the node number and rack number of each storage node through the processing unit 602.

[0098] Optionally, the processing unit 602 is further configured to: if at least two storage nodes in the node array of the random node source array are located in the same rack, the control center randomly sorts the node numbers according to a random algorithm to obtain a new random node source array, until the storage nodes of each node array in the random node source array are located in different racks.

[0099] Optionally, the processing unit 602 is specifically used to: if the number of node numbers in the first node array is less than R, then randomly select node numbers from the second node array and add them to the first node array to make up the number of node numbers in the first node array, wherein the second node array is the other node arrays among the multiple node arrays divided by the random node source array.

[0100] Optionally, the processing unit 602 is further configured to: assign node numbers to the newly added storage nodes; create a new node array for each random node source group in the optimal node array set, add the node numbers of the newly added storage nodes to each new node array respectively, and select node numbers from the random node source array to make up the number of node numbers in the new node array, wherein the number of the same storage nodes in the new node array and any node array in the optimal node array set is not greater than the set threshold.

[0101] Optionally, the processing unit 602 is further configured to: obtain the data to be copied corresponding to the node number of the storage node to be deleted from the data storage location record, wherein the data storage location record contains the correspondence between the node number of each storage node and the data identifier of the data stored in each storage node; copy the data to be copied from the storage node corresponding to the data to be copied; determine a third storage node from each node array of the optimal node array set other than the node array where the storage node to be deleted is located, and store the data to be copied in the third storage node.

[0102] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data storage method, characterized in that, The method includes: The control center receives data storage requests; The control center determines the first storage node for storing the primary copy from each node array in the optimal node array set, and determines N different second storage nodes for storing replicas from each node array containing the first storage node; wherein, the storage nodes in the same node array are located in different racks, and the optimal node array set is the grouping result with the smallest array pair statistical value in T random groupings for each storage node, including: for any random grouping of each storage node, determining the number P of random node source arrays based on the number R of storage nodes storing the same data; wherein, the random node source array contains the node number of each storage node, and the order of the node numbers in any two random node source arrays is different; for any random node source array, the random node source array is divided into multiple node arrays, wherein each node array has R storage nodes and each storage node in each node array is located in a different rack; If at least two storage nodes in the node array of the random node source array are located in the same rack, the control center randomly sorts the node numbers according to the random algorithm to obtain a new random node source array, until the storage nodes of each node array in the random node source array are located in different racks. For each node array in the P random node source arrays, determine the number of identical storage nodes between any two node arrays; if the number of identical storage nodes is greater than a set threshold, increment the array pair statistics by 1; determine the optimal node array set from the P random node source arrays with the smallest array pair statistics in T random groupings; the array pair statistics are used to count the number of array pairs in each grouping where the number of identical storage nodes is greater than a set threshold. The control center stores the data corresponding to the data storage request to the first storage node and N second storage nodes respectively.

2. The method as described in claim 1, characterized in that, The racks where each storage node is located are obtained in the following ways: The control center receives startup success notifications from each storage node, and the startup success notifications contain the rack number corresponding to the storage node. The control center assigns a node number to each storage node and records the node number and rack number of each storage node.

3. The method as described in claim 1, characterized in that, The random node source array is divided into multiple node arrays, including: If the number of node numbers in the first node array is less than R, then a node number is randomly selected from the second node array and added to the first node array to make up the number of node numbers in the first node array. The second node array is the other node arrays among the multiple node arrays divided from the random node source array.

4. The method as described in claim 1, characterized in that, When adding a new storage node, it also includes: The control center assigns a node number to the newly added storage node; The control center creates a new node array for each random node source group in the optimal node array set, adds the node number of the new storage node to each new node array, and selects node numbers from the random node source array to make up the number of node numbers in the new node array. The number of the same storage nodes in the new node array and any node array in the optimal node array set is not greater than the set threshold.

5. The method as described in claim 1, characterized in that, Also includes: The control center obtains the data to be copied corresponding to the node number of the storage node to be deleted from the data storage location record. The data storage location record contains the correspondence between the node number of each storage node and the data identifier of the data stored in each storage node. The control center copies the data to be copied from the storage node corresponding to the data to be copied; The control center determines a third storage node from all node arrays in the optimal node array set except for the node array containing the storage node to be deleted, and stores the data to be copied in the third storage node.

6. A data storage device, characterized in that, The device includes: The receiving module is used to receive data storage requests; A processing unit is configured to determine a first storage node as primary storage from each node array in an optimal node array set, and to determine N different second storage nodes as replica storage from each node array containing the first storage node; wherein the storage nodes in the same node array are located in different racks, and the optimal node array set is the grouping result with the smallest array pair statistical value in T random groupings for each storage node, including: for any random grouping of each storage node, determining the number P of random node source arrays based on the number R of storage nodes storing the same data; wherein the random node source array contains the node number of each storage node, and the order of the node numbers in any two random node source arrays is different; for any random node source array, dividing the random node source array into multiple node arrays, wherein each node array has R storage nodes and each storage node in each node array is located in a different rack; If at least two storage nodes in the node array of the random node source array are located in the same rack, the control center will randomly sort the node numbers according to the random algorithm to obtain a new random node source array, until the storage nodes of each node array in the random node source array are located in different racks. For each node array in the P random node source arrays, determine the number of identical storage nodes between any two node arrays; if the number of identical storage nodes is greater than a set threshold, increment the array pair statistics by 1; determine the optimal node array set from the P random node source arrays with the smallest array pair statistics in T random groupings; the array pair statistics are used to count the number of array pairs in each grouping where the number of identical storage nodes is greater than a set threshold. The processing unit is further configured to store the data corresponding to the data storage request to the first storage node and N second storage nodes respectively.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a computer, causes the computer to perform the method of any one of claims 1 to 5.

8. A computer device, characterized in that, include: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method as described in any one of claims 1 to 5 according to the obtained program.

Images