A data writing method and related device

Abstract

The embodiment of the application discloses a data writing method and related equipment, and the method comprises the following steps: in the case that a computing device needs to write to-be-stored data into a target storage pool, the computing device performs fragmentation on the to-be-stored data according to a redundancy ratio of the target storage pool, to obtain a first quantity of data fragments and a second quantity of check fragments; if the disk types corresponding to a plurality of LUNs in the target storage pool are multi-arm disks, then a third quantity of target LUNs are selected from the plurality of LUNs in the target storage pool based on a multi-arm disk selection strategy; wherein the third quantity of target LUNs respectively correspond to a third quantity of different disks; and the first quantity of data fragments and the second quantity of check fragments are stored into the third quantity of target LUNs. According to the embodiment of the application, the third quantity of target LUNs selected by the computing device respectively correspond to the third quantity of different disks, so that the reliability of data storage can be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a data writing method and related equipment. BACKGROUND

[0002] In a distributed storage system, in order to improve the reliability of the storage system, an erasure coding (EC) algorithm / technology can be used to guarantee the reliability of disk data and node data.

[0003] EC is a data shard redundancy protection mechanism, which realizes data redundancy protection by calculating check shards. When writing data, the data is usually divided into N data shards, and then M check shards are calculated based on the N data shards through an EC encoding algorithm. Then, the N data shards and the M check shards can be distributedly stored. Subsequently, in the case of shard loss, if the number of lost shards is less than or equal to M, the original data can be restored according to the shards that are not lost.

[0004] In actual scenarios, users can configure different EC redundancy ratios according to the number of storage nodes. When the number of storage nodes is greater than or equal to (N+M), an N+M redundancy ratio can be used. When the N+M redundancy ratio is used, the requirement that M disks or M nodes fail without data loss needs to be met. When the number of storage nodes is greater than or equal to ((N+M) / M) and less than (N+M), an N+M:K redundancy ratio can be used. When the N+M:K redundancy ratio is used, the requirement that M disks or K nodes fail without data loss needs to be met.

[0005] Currently, when N+M shards are stored, the storage is generally based on the LUN of the node. For single-arm disks and multi-arm disks, the number of LUNs presented to the upper-layer operating system is different. The failure of a single-arm disk will affect one LUN, and the failure of a multi-arm disk will affect multiple LUNs. Therefore, in different scenarios (such as a multi-arm disk scenario), how to meet the reliability requirement of M disk failures is a problem that technicians are concerned about. SUMMARY

[0006] Embodiments of the present application disclose a data writing method and related equipment, which can guarantee the reliability of data storage.

[0007] The first aspect discloses a data writing method, which can be applied to a computing device, a module (for example, a chip) in the computing device, a logic module or software capable of realizing all or part of the functions of the computing device, and is described below by taking the application to the computing device as an example. The data writing method can include: in the case that the computing device needs to write to-be-stored data into a target storage pool, the computing device shards the to-be-stored data according to a redundancy ratio of the target storage pool, to obtain a first number of data shards and a second number of check shards; if the disk types corresponding to a plurality of LUNs in the target storage pool are multi-arm disks, a third number of target LUNs are selected from the plurality of LUNs in the target storage pool based on a multi-arm disk selection strategy; wherein the third number of target LUNs correspond to a third number of different disks respectively; and the first number of data shards and the second number of check shards are stored into the third number of target LUNs.

[0008] In the embodiments of the present application, in the case that the disk types corresponding to a plurality of LUNs in the target storage pool are multi-arm disks, the third number of target LUNs selected based on the multi-arm disk selection strategy correspond to a third number of different disks respectively when the computing device stores the first number of data shards and the second number of check shards. In this way, since each target LUN corresponds to a different disk, the requirement of failing a second number of disks can be met, and the reliability of data storage can be ensured.

[0009] As a possible implementation, if the disk types corresponding to a plurality of LUNs in the target storage pool are multi-arm disks, a third number of target LUNs are selected from the plurality of LUNs in the target storage pool based on a multi-arm disk selection strategy, which includes: if the disk types corresponding to a plurality of LUNs in the target storage pool are multi-arm disks, a fourth number of candidate LUNs are selected from the plurality of LUNs in the target storage pool based on a multi-arm disk selection strategy; and a third number of target LUNs are selected from the fourth number of candidate LUNs based on the redundancy ratio of the target storage pool.

[0010] In the embodiments of the present application, the computing device can first select a fourth number of candidate LUNs from the plurality of LUNs in the target storage pool based on a multi-arm disk selection strategy, and then select a third number of target LUNs from the fourth number of candidate LUNs based on the redundancy ratio of the target storage pool. In this way, the disks meeting the conditions can be filtered out by the multi-arm disk selection strategy first, and then further selected based on the redundancy ratio, so that the requirement of the redundancy ratio can be met, and the reliability of data storage can be ensured.

[0011] As a possible implementation, the method can further include: if the disk types corresponding to the plurality of LUNs in the target storage pool are single-arm disks, selecting a fourth number of candidate LUNs from the plurality of LUNs in the target storage pool based on a single-arm disk selection policy; if the disk types corresponding to the plurality of LUNs in the target storage pool include single-arm disks and multi-arm disks, selecting a fourth number of candidate LUNs from the plurality of LUNs in the target storage pool based on a single-arm disk selection policy and a multi-arm disk selection policy; and selecting a third number of target LUNs from the fourth number of candidate LUNs based on the redundancy matching of the target storage pool.

[0012] In the embodiments of the present application, for the single-arm scenario and the mixed single-arm and multi-arm scenario, a fourth number of candidate LUNs can be selected from the plurality of LUNs in the target storage pool based on the single-arm disk selection policy, the single-arm disk selection policy and the multi-arm disk selection policy respectively, and then a third number of target LUNs can be selected from the fourth number of candidate LUNs based on the redundancy matching of the target storage pool. In this way, different selection policies can be used for different scenarios, so that the requirements of the redundancy matching in various scenarios can be met, and the reliability of data storage can be ensured.

[0013] As a possible implementation, the multi-arm disk selection policy can include that the fourth number of candidate LUNs respectively correspond to a fourth number of different disks.

[0014] In the embodiments of the present application, for the multi-arm disk, at most one LUN of the same multi-arm disk is selected as a candidate LUN when the candidate LUN is selected. In this way, the requirement of failing a second number of disks can be met.

[0015] As a possible implementation, the multi-arm disk selection policy can further include one or more of the following: a current capacity usage percentage of the fourth number of candidate LUNs is less than a first threshold value, a current central processing unit (CPU) utilization of a node corresponding to the fourth number of candidate LUNs is less than a second threshold value, a current memory load of the node corresponding to the fourth number of candidate LUNs is less than a third threshold value, a current bandwidth of a disk corresponding to the fourth number of candidate LUNs is less than a fourth threshold value, and a current input / output per second (IOPS) of the disk corresponding to the fourth number of candidate LUNs is less than a fifth threshold value.

[0016] In the embodiments of the present application, when the candidate LUN is selected, the computing device can also consider capacity balancing and performance balancing (i.e., the current CPU utilization, memory load, disk bandwidth, etc. of the node), so that the processor resources, memory resources, disk resources, etc. of each node can be fully utilized, thereby improving the writing efficiency of data and achieving load balancing.

[0017] As a possible implementation, the single-magnetic-arm disk selection strategy includes one or more of the following: the current capacity usage percentage of the fourth number of candidate LUNs is less than a first threshold value, the current central processing unit (CPU) utilization of the nodes corresponding to the fourth number of candidate LUNs is less than a second threshold value, the current memory load of the nodes corresponding to the fourth number of candidate LUNs is less than a third threshold value, the current bandwidth of the disks corresponding to the fourth number of candidate LUNs is less than a fourth threshold value, and the current input / output per second (IOPS) of the disks corresponding to the fourth number of candidate LUNs is less than a fifth threshold value.

[0018] As a possible implementation, the selecting the third number of target LUNs from the fourth number of candidate LUNs based on the redundancy ratio of the target storage pool includes: when the redundancy ratio of the target storage pool is the first number + the second number, selecting the third number of target LUNs from the fourth number of candidate LUNs according to the first number + the second number redundancy ratio, the third number of target LUNs corresponding to the third number of different nodes respectively; and when the redundancy ratio of the target storage pool is the first number + the second number: the sixth number, selecting the third number of target LUNs from the fourth number of candidate LUNs according to the first number + the second number: the sixth number redundancy ratio, the third number of target LUNs corresponding to the fifth number of different nodes respectively.

[0019] In the embodiments of the present application, the scenario of using different redundancy ratios for the target storage pool can be processed specifically, thereby meeting the requirements of the redundancy ratio and ensuring the reliability of data storage.

[0020] As a possible implementation, the method can further include: determining a plurality of LUNs belonging to the same multi-magnetic-arm disk in the target storage pool; and marking the plurality of LUNs through the serial number of the multi-magnetic-arm disk.

[0021] In the embodiments of the present application, the computing device can mark a plurality of LUNs belonging to the same multi-magnetic-arm disk in the target storage pool through the serial number of the multi-magnetic-arm disk. In this way, when selecting candidate LUNs, the candidate LUNs can be selected based on the marking of each LUN, and at most one LUN of the same multi-magnetic-arm disk can be selected as a candidate LUN, thereby meeting the requirement of failing the second number of disks.

[0022] As a possible implementation, the storing the first quantity of data shards and the second quantity of check shards into the third quantity of target LUNs comprises: sending a data write request to each node corresponding to the third quantity of target LUNs, each data write request comprising an identification of one or more LUNs in the third quantity of target LUNs and one or more shards of the first quantity of data shards and the second quantity of check shards that need to be written into the one or more LUNs.

[0023] In the embodiments of the present application, the computing node can send a data write request to the node corresponding to the selected target LUN, and carry the identification of the corresponding target LUN and the data that needs to be stored in the target LUN in the data write request, so as to facilitate the corresponding node to write data into the target LUN, and realize the redundant protection of data.

[0024] The second aspect discloses a computing device, comprising a processor, a memory and a communication interface, the communication interface is used to receive information from other electronic devices outside the computing device and output information to other electronic devices outside the computing device, the processor invokes the computer program stored in the memory to realize the data write method provided in the first aspect and any possible implementation manner of the first aspect.

[0025] The third aspect discloses a data storage system, comprising a computing device and a distributed storage system, the computing device is connected with the distributed storage system, the distributed storage system comprises a plurality of storage nodes, each of the storage nodes comprises at least one disk, and the computing device is used to execute the data write method provided in the first aspect and any possible implementation manner of the first aspect.

[0026] The fourth aspect discloses a computer readable storage medium, the computer readable storage medium stores a computer program or computer instructions, when the computer program or computer instructions are run, the data write method disclosed in the above aspects is realized.

[0027] The fifth aspect discloses a chip, comprising a processor, used to execute the program stored in the memory, when the program is executed, the chip executes the data write method disclosed in the above aspects.

[0028] As a possible implementation, the memory is located outside the chip.

[0029] The sixth aspect discloses a computer program product, the computer program product comprises computer program code, when the computer program code is run, the data write method disclosed in the above aspects is executed.

[0030] It can be understood that the computing device provided by the second aspect, the computer readable storage medium provided by the fourth aspect, the chip provided by the fifth aspect and the computer program product provided by the sixth aspect can be used to execute the data writing method provided by the first aspect and any possible implementation manner in the first aspect. Therefore, the beneficial effects achieved thereby can refer to the beneficial effects in the corresponding method, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0032] Figure 1 is a scene diagram of data storage disclosed by an embodiment of the present application;

[0033] Figure 2 is another scene diagram of data storage disclosed by an embodiment of the present application;

[0034] Figure 3 is still another scene diagram of data storage disclosed by an embodiment of the present application;

[0035] Figure 4 is still another scene diagram of data storage disclosed by an embodiment of the present application;

[0036] Figure 5 is a system architecture diagram of data storage disclosed by an embodiment of the present application;

[0037] Figure 6 is still another scene diagram of data storage disclosed by an embodiment of the present application;

[0038] Figure 7 is a flow diagram of a data writing method disclosed by an embodiment of the present application;

[0039] Figure 8 is a scene diagram of marking a multi-magnetic-arm disk disclosed by an embodiment of the present application;

[0040] Figure 9 is still another scene diagram of data storage disclosed by an embodiment of the present application;

[0041] Figure 10 is still another scene diagram of data storage disclosed by an embodiment of the present application;

[0042] Figure 11 is a flow diagram of another data writing method disclosed by an embodiment of the present application;

[0043] Figure 12 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present application disclose a data writing method and related equipment, which can guarantee the reliability of data storage. The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

[0045] In order to better understand the embodiments of the present application, the related technologies of the embodiments of the present application will be described first.

[0046] With the rapid development of high performance computing (HPC), big data, artificial intelligence (AI) and the like, massive data is impacting various industries with an unprecedented growth trend, in order to effectively manage, maintain and process the data, users have increasingly high requirements for data storage capacity and data processing efficiency. Under this background, disk technology develops towards large capacity and high performance, and a multi-actuator disk can balance disk capacity and performance, and the data read-write efficiency is multiple times that of a traditional single-actuator disk.

[0047] A single-actuator disk (i.e. a traditional mechanical hard disk) has one magnetic arm, one or more disks and one or more magnetic heads inside. The upper and lower surfaces of one disk can store data, and the upper and lower surfaces can have one magnetic head for reading and writing respectively, and one disk can be divided into multiple sectors, and the magnetic head can be positioned to a target sector (i.e. a sector that needs to write or read data) on the disk through swinging of the magnetic arm, and then the magnetic head can read or write data from the target sector. Compared with the single-actuator disk, the multi-actuator disk (i.e. multi-actuator physical disk) has multiple independent magnetic arms, multiple disks and multiple magnetic heads inside. The disks inside the multi-actuator disk can be divided into multiple groups, and each magnetic arm can be responsible for data reading and writing of a group of disks. It can be seen that, since the multi-actuator disk can have multiple magnetic arms to concurrently read and write data, and the single-actuator disk only has one magnetic arm to read and write data, the read-write efficiency of the multi-actuator disk is much higher than that of the single-actuator disk, and therefore the multi-actuator disk is widely used in distributed storage systems.

[0048] It can be understood that a single magnetic arm disk, due to only one magnetic arm reading and writing data, generally presents one LUN (logical unit number) to the upper layer operating system, that is, one logical disk (Disk), and correspondingly, single magnetic arm disk: LUN / Disk = 1:1. The multi-magnetic arm disk has multiple (such as 2) magnetic arms in the disk body, and the multiple magnetic arms can work simultaneously, so it generally presents multiple LUNs to the upper layer operating system (operating system, OS), that is, multiple logical disks, and correspondingly, multi-magnetic arm disk: LUN / Disk = 1:Q (Q≥2). It should be understood that the multi-magnetic arm disk has how many magnetic arms, and it can present how many LUNs to the upper layer operating system. For example, for a multi-magnetic arm disk with 2 magnetic arms, it can present 2 LUNs to the upper layer operating system. For another example, for a multi-magnetic arm disk with 3 magnetic arms, it can present 3 LUNs to the upper layer operating system.

[0049] In a distributed storage system, a storage node is usually built based on a general server. In order to improve the reliability of the distributed storage system, an erasure coding (EC) algorithm / technology can be used to guarantee the reliability of disk data and node data. Compared with the replica technology, the EC technology has higher capacity utilization. For example, the EC uses N+2:1, and the capacity utilization is 2 / n+2. When n is 1, 2 / n+2 is about 66%, and when 2 replicas are used (that is, each piece of data is stored twice), the capacity utilization is 1 / 2, that is, 50%.

[0050] EC is a data shard redundancy protection mechanism, which realizes data redundancy protection by calculating check shards. It is commonly used in traditional single magnetic arm distributed storage systems to realize data reliability redundancy protection. When writing data in a distributed storage system, the data is usually divided into N data shards, and then M check shards are calculated based on the N data shards through an EC encoding algorithm. Then, the N data shards and the M check shards can be distributed stored. It should be understood that in the case of loss of a shard (that is, a data shard or a check shard), the original data (that is, the original data shard and the check shard) can be restored by any N shards of the N+M shards. Therefore, if the number of lost shards is less than or equal to M, the original data can be restored according to the non-lost shards. N and M are integers greater than or equal to 1.

[0051] In actual scenarios, a user can configure different EC redundancy ratios according to the number of storage nodes. When the number of storage nodes is greater than or equal to (N+M), an N+M redundancy ratio can generally be used. When the N+M redundancy ratio is used, N+M shards (N data shards and M check shards) can be stored on the disks of N different storage nodes, and therefore, the N+M redundancy ratio can allow M node failures or M disk failures without data loss. When the number of storage nodes is greater than or equal to ((N+M) / M) and less than (N+M), an N+M:1 or N+M:2 redundancy ratio can generally be used. When the N+M:1 or N+M:2 redundancy ratio is used, N+M shards can be stored on all storage nodes, and each storage node stores at least one shard. It should be understood that the N+M:1 redundancy ratio can allow M disk failures or 1 node failure without data loss, and the N+M:2 redundancy ratio can allow M disk failures or 2 node failures without data loss. Similarly, the N+M:K redundancy ratio can allow M disk failures or K node failures without data loss. K is an integer greater than or equal to 1.

[0052] The following describes a scenario in which the number of storage nodes is greater than or equal to (N+M) in a single-arm disk scenario with N being 4, M being 2, and the number of storage nodes being 6. Please refer to Figure 1 , Figure 1 is a schematic diagram of a data storage scenario disclosed by an embodiment of the present application. As shown in Figure 1 , each storage node can include 4 single-arm disks, and each single-arm disk can correspond to a LUN. When data is written, the data can be divided into 4 data shards, and then 2 check shards can be obtained based on the 4 data shards. The 4 data shards and 2 check shards can generally be referred to as data shards or strips. Since the number of shards is equal to the number of nodes, an N+M (4+2) redundancy ratio can be used, that is, one shard can be stored in each node, and at this time, one LUN can be selected in each node to store the shard. For example, the 4 data shards can be stored in a LUN of node 2, node 3, node 4, and node 6, and the 2 check shards can be stored in a LUN of node 1 and node 5. In this way, the storage system can be guaranteed to fail 2 nodes (such as node 1 and node 3) or 2 disks at the same time without data loss.

[0053] The following describes a scenario in which the number of storage nodes is greater than or equal to ((N+M) / M) and less than (N+M) in a single-arm disk scenario with N being 4, M being 2, and the number of storage nodes being 3. Please refer to Figure 2 , Figure 2 is another schematic diagram of a data storage scenario disclosed by an embodiment of the present application. As shown in Figure 2As shown, each storage node can include four single-arm disks, and each single-arm disk can correspond to one LUN. When writing data, the data can be divided into four data fragments, and then two parity fragments can be obtained based on these four data fragments. Since the number of fragments is greater than the number of nodes and less than ((N+M) / M), an N+M:1 (4+2:1) redundancy ratio can be adopted. In this case, the EC algorithm can be used to randomly distribute the four data fragments and two parity fragments across all nodes, and two LUNs can be selected on each node to store the fragments. For example, one data fragment can be stored in one LUN on node 1, one data fragment in one LUN on node 3, two data fragments can be distributed across two LUNs on node 2, and the two parity fragments can be distributed across one LUN on nodes 1 and 3. In this way, it can be ensured that if one node or two disks fail in the storage system, no data will be lost.

[0054] In the data storage scenarios described above, single-arm disks are used in the nodes. The following section introduces data storage in multi-arm disk scenarios.

[0055] The following explanation focuses on scenarios with N = 4, M = 2, and 6 storage nodes, illustrating the case where the number of storage nodes is greater than or equal to (N + M) in a multi-arm disk setup. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram illustrating another data storage scenario disclosed in an embodiment of this application. For example... Figure 3 As shown, each storage node can include two dual-arm disks, and each dual-arm disk can correspond to two LUNs. When writing data, the data can be divided into four data fragments, and then two parity fragments can be obtained based on these four data fragments. Since the number of fragments is equal to the number of nodes, an N+M (4+2) redundancy ratio can be adopted, meaning that one fragment can be stored on each node. In this case, one LUN can be selected on each node to store the fragment. For example, the four data fragments can be distributed and stored in one LUN on nodes 2, 3, 4, and 5, and the two parity fragments can be distributed and stored in one LUN on nodes 1 and 6. In this way, since each node only stores one fragment, it can be ensured that no data is lost even if two nodes or two disks fail simultaneously.

[0056] The following explanation focuses on a scenario with N = 4, M = 2, and 3 storage nodes, where the number of storage nodes is greater than or equal to ((N+M) / M) but less than (N+M). Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram illustrating another data storage scenario disclosed in an embodiment of this application. For example... Figure 4As shown, each storage node can include multiple dual-armed magnetic disks, and each dual-armed magnetic disk can correspond to 2 LUNs. When writing data, the data can be split into 4 data shards, and then 2 check shards can be obtained based on the 4 data shards. Since the number of shards is greater than the number of nodes and less than ((N+M) / M), N+M: 1 (4+2: 1) redundancy ratio can be used, at this time, the 4 data shards and 2 check shards can be randomly scattered and stored in all nodes by EC algorithm, and two LUNs can be selected in each node to store the shards. For example, 1 data shard in the 4 data shards can be stored in one LUN of node 1, 1 data shard can be stored in one LUN of node 3, 2 data shards can be stored in two LUNs of node 2, and 2 check shards can be stored in one LUN of node 1 and node 3. In this way, the storage system can be guaranteed to fail 1 node without losing data.

[0057] However, since the LUNs where the check shards and data shards stored in node 1 belong to the same dual-armed magnetic disk, and the LUNs where the two data shards stored in node 2 also belong to the same dual-armed magnetic disk, the failure of any one of the two dual-armed magnetic disks will result in the loss of 2 shards, and under the N+M: 1 redundancy ratio, the maximum number of shards that can be lost is 2, in this case, it is not possible to guarantee that the storage system fails 2 disks at the same time without losing data. For example, when the two dual-armed magnetic disks storing shards in the above node 1 and node 2 fail, 4 shards will be lost, at this time, the original data (i.e., the original 6 shards) cannot be restored according to the remaining two shards. It should be understood that the above Figures 1-4 The LUNs of each node (such as node 1-node 6 in Figure 1 may belong to the same storage pool.

[0058] It should be understood that the data shard redundancy protection algorithm is mainly designed for traditional single-armed magnetic disks, so when storing N+M shards, the above storage is generally based on the LUN of the node, because N+M (such as the above 4+2) redundancy ratio and N+M: K (such as the above 4+2: 1) redundancy ratio both need to meet the requirement that M disks fail without losing data, so at most only 1 shard of 1 data shard can be stored on one LUN, and then when storing data, (N+M) different LUNs need to be selected to store (N+M) shards of data shards.

[0059] In the case where the number of storage nodes is greater than or equal to (N+M) and N+M (such as 4+2 described above) redundancy matching is adopted, since one node stores at most one fragment of one data stripe, the single-arm disk or multi-arm disk in the node also stores at most one fragment of one data stripe, so that the data is not lost when M disks or M nodes of the storage system fail at the same time.

[0060] In the case where the number of storage nodes is greater than or equal to ((N+M) / M) and less than (N+M) and N+M:K (such as 4+2:1 described above) redundancy matching is adopted, some or all nodes need to store multiple fragments (such as two fragments) of one data stripe, that is, multiple LUNs need to be selected in these nodes to store different fragments. In this case, for the single-arm disk, since the single-arm disk presents one LUN to the upper operating system, when all the disks in the node are single-arm disks, the multiple LUNs selected from the node belong to different single-arm disks, so that the data is not lost when M disks of the storage system fail at the same time. For the multi-arm disk, since the multi-arm disk presents multiple LUNs to the upper operating system, when the disks in the node include multi-arm disks, the multiple LUNs selected from the node may belong to the same multi-arm disk, so that the data is lost when M disks of the storage system fail at the same time, that is, the reliability requirement of M disk failure cannot be met. In addition, in this case, multiple fragments of the same data stripe may be stored in the same multi-arm disk, which causes local hot disk / multi-arm disk load imbalance and reduces reliability.

[0061] To solve the above problems, in the data writing process, the judgment process of the multi-arm disk is increased and optimized in the embodiments of the present application, so that the data writing method (i.e. EC data fragment redundancy protection algorithm) of the present application can support traditional single-arm disks and multi-arm disks. For the multi-arm disk, at most one LUN of the same multi-arm disk is selected as the candidate LUN when the candidate LUN is selected. In this way, when N+M:K (such as 4+2:1 described above) redundancy matching is adopted, the reliability requirement of M disk failure can be met.

[0062] In order to better understand the embodiments of the present application, the system architecture used in the embodiments of the present application will be described first.

[0063] Please refer to Figure 5 , Figure 5 is a data storage system architecture diagram disclosed by the embodiments of the present application. As Figure 5 shown, the data storage system architecture can include a distributed storage system 501, a network / switch 502, and a computing device 503. The distributed storage system 501 can include a plurality of storage nodes. For example, as shown inFigure 5 The illustrated nodes 1 (5011), node 2 (5012), node 3 (5013), …, node H (5014), H is an integer greater than or equal to 2. In some embodiments, each node can be one or more servers, such as a blade server, a high-density server, a rack server, a cabinet server, etc.

[0064] Each node (i.e., node 1 - node H) and the computing device 503 of the distributed storage system 501 can be connected to the network / switch 502, through which the communication (i.e., data interaction) between each node and the computing device 503 can be achieved. The communication between each node of the distributed storage system 501 can also be achieved through the network / switch 502.

[0065] It can be understood that the distributed storage system 501 can be regarded as a storage system composed of the above-mentioned multiple nodes (i.e., node 1 - node H), which can be used to initialize disks, obtain disk types, etc., and can configure data EC redundancy ratio (i.e., protection level), etc. In some embodiments, each node in the distributed storage system 501 can include distributed storage software, which can obtain data of each node (such as disk information of each node), and the distributed storage software can maintain the entire storage system in a healthy state. In addition, one or more storage pools can be configured through the distributed storage software of each node, and each storage pool can include one or more LUNs of one or more nodes. Each node (i.e., node 1 - node H) in the distributed storage system 501 is mainly used for storing data (i.e., data shards and check shards), and each node can include one or more disks.

[0066] The computing device 503 can include a client (i.e., installed with storage client software) 5031, which can be responsible for reading and writing data. For example, when writing data, the client can split the data into N (i.e., a first number) data shards, and then can obtain M (i.e., a second number) check shards based on the N data shards. Then, the client 5031 can store the N data shards and the M check shards on the nodes in the distributed storage system 501 through the EC algorithm. It should be noted that in some embodiments, one or more storage pools of the distributed storage system 501 can be mounted to the client 5031 (i.e., one or more storage pools are configured for the client 5031). Then, when the client 5031 stores data, it needs to select a target LUN for storing data from a mounted storage pool, and then can store the data in the selected target LUN.

[0067] It should be noted that, Figure 5The system architecture shown is only illustrative and does not constitute a limitation. In other embodiments of the present application, Figure 5 The system architecture shown can include more or fewer devices or modules than illustrated, and is not limited to including only Figure 5 The distributed storage system 501, network / switch 502, and computing device 503 shown in FIG. 1.

[0068] The following describes a scenario of data storage of the present application. Please refer to Figure 6 , Figure 6 is another scenario of data storage disclosed by embodiments of the present application. As shown in FIG. 2, when the client 5031 needs to write data to the distributed storage system 501, in order to achieve EC data protection, the client 5031 can split the data into N (e.g., 4) data shards based on the EC redundancy ratio reliability requirement, and then obtain M (e.g., 2) check shards based on the N data shards. Figure 6

[0069] After that, the client 5031 can read the initialized disk information (including disk serial number, disk type, etc.), determine whether the disk corresponding to the LUN in the storage pool is a single-arm disk or a multi-arm disk, if it is a single-arm disk, select candidate LUNs from the storage pool according to the LUN capacity and performance balancing principle, if it is a multi-arm disk, mark the LUNs belonging to the same multi-arm disk based on the disk information, then select candidate LUNs from the storage pool according to the LUN capacity and performance balancing principle, and any two LUNs in the selected candidate LUNs do not belong to the same multi-arm disk, that is, at most one LUN of a multi-arm disk is selected as a candidate disk. After the client 5031 selects the candidate disks (i.e., candidate LUNs) according to the disk type, it can select N+M (i.e., the third number) target LUNs from the candidate disks according to the EC redundancy ratio, and then distribute the N+M shards to the corresponding nodes through the network / switch 502 and write them into the selected target LUNs.

[0070] ​It can be understood that the above is the generation of data sharding and check shard by the client 5031, and the selection of candidate disks and the distribution of data are also performed by the client 5031. However, in some other embodiments of the present application, the operations performed by the client 5031 can be performed by the nodes in the distributed storage system 501 (such as node 1 5011). For example, the client 5031 can directly write (i.e., send) data to a node (such as node 1 5011) in the distributed storage system 501, and then the node can split the data into N data shards, and can obtain M check shards based on the N data shards, and then the node can select candidate disks based on the above disk selection strategy, and then can select N+M target LUNs from the candidate disks according to the EC redundancy ratio, and finally the node can distribute the N+M shards to the corresponding nodes and write them into the selected target LUNs.

[0071] In some embodiments, the computing device 503 can be a tablet computer, a notebook computer, a smart car, a desktop computer, a server, etc., which is not limited herein.

[0072] It should be understood that the data writing method provided by the embodiments of the present application can be applied to the data protection scene of the distributed storage system, including but not limited to the scene of HPC, big data analysis, etc. It should be noted that, Figure 6 The data storage scene shown is only illustrative and does not limit the present application.

[0073] Based on the above system architecture, please refer to Figure 7 , Figure 7 is a flowchart of a data writing method disclosed by the embodiments of the present application. As Figure 7 shown, the data writing method can include but is not limited to the following steps:

[0074] 701. The storage system configures at least one storage pool according to the obtained disk information, and each storage pool is configured with a corresponding redundancy ratio.

[0075] When each node of the storage system (i.e., the above distributed storage system) is powered on, the storage system can initialize the disks of each node and obtain the disk information of each node.

[0076] Specifically, when each node of the storage system is powered on, the disk of each node will perform an initialization operation. During the initialization process, the disk will report disk-related information (such as disk type, etc.) to the upper-layer operating system. The operating system of each node can generate a LUN corresponding to the disk. Since each node belongs to the same distributed storage system, each node can obtain or synchronize the relevant LUN information of other nodes. In addition, the distributed storage system (i.e., each node) can also call each disk interface to obtain or read the information of all disks of the distributed storage system. The information of the disk (i.e., disk information) can include a serial number (SN) of the disk, a type of the disk (single-armed disk / two-armed disk / three-armed disk, etc.), LUNs included in the disk, a manufacturer of the disk, and the like.

[0077] After obtaining the disk information of each node, the storage system can automatically create or configure one or more storage pools for the storage system, can create or configure one or more storage pools for the storage system through manual selection by a user (i.e., receiving user input), or can configure one or more storage pools for the storage system in other manners. In a possible implementation, any node (such as node 1) of the storage system can receive a storage pool configuration request sent by a user terminal or other device, or any node of the storage system can receive a storage pool configuration request input by a user, etc. The storage pool configuration request is used to request configuration of one or more storage pools, and can include the ID or number of one or more LUNs. For example, the user can input the storage pool configuration request on the display interface of the storage node (such as node 1) through distributed storage software or a command line.

[0078] It should be understood that one storage pool can include one or more LUNs of one or more nodes in the storage system. In addition, for a multi-armed disk, the multiple LUNs of the same multi-armed disk can be configured in the same storage pool. In addition, the disks corresponding to the LUNs in one storage pool can be the same type of physical disks, for example, can all be single-armed disks, or can all be two-armed disks, or can all be three-armed disks.

[0079] After one or more storage pools are configured for the storage system, in order to implement EC redundancy protection, an EC redundancy ratio (i.e., a protection level) can be configured for each storage pool. It should be understood that the number of LUNs included in different storage pools can be different, and the number of nodes corresponding to each storage pool (i.e., the number of nodes corresponding to the LUNs in the storage pool) and the storage capacity (i.e., the total storage capacity corresponding to the LUNs in the storage pool) can also be different, and thus different EC redundancy ratios can be configured for different storage pools. For example, assuming there are 2 storage pools, storage pool 1 can include 24 LUNs, and the 24 LUNs belong to 6 nodes respectively, and each node has 4 LUNs, i.e., 2 double magnetic arms. Storage pool 2 can include 12 LUNs, and the 12 LUNs belong to 3 nodes respectively, and each node has 4 LUNs, i.e., 2 double magnetic arms. Thus, for storage pool 1, an EC redundancy ratio of 4+2 can be configured, and for storage pool 2, an EC redundancy ratio of 4+2:1 can be configured.

[0080] It should be noted that in order for the storage client or the computing device to write data to the storage system, one or more storage pools of the storage system can be mounted to the client, i.e., one or more storage pools are assigned to the client for use. Then, the client or the computing device can store data based on the assigned storage pool.

[0081] It should also be noted that in the case of configuring multiple storage pools for the client, the client stores data (i.e., the computing device stores data) based on one of the storage pools, and thus for ease of understanding, the following describes one storage pool (i.e., a target storage pool) of the client. Alternatively, in some embodiments, when the client or the computing device stores data, it can first select a storage pool based on a selection strategy (such as selecting the storage pool with the largest remaining capacity), and the following describes the storage pool selected by the client or the computing device (i.e., the target storage pool).

[0082] 702. In the case where the computing device needs to write to-be-stored data to the target storage pool, the computing device shards the to-be-stored data according to the redundancy ratio of the target storage pool.

[0083] After the storage pool is configured for the client of the computing device, the computing device can store data through the storage pool, and the following describes the target storage pool. Specifically, when the computing device needs to write data (such as a file), the computing device can first shard the data according to the configured redundancy ratio (such as N+M, N+M:K, etc.) of the target storage pool, which can be divided into N (i.e., a first number) data shards, and then M (i.e., a second number) check shards can be generated based on the N data shards. N and M are integers greater than or equal to 1, and the target storage pool can be one of one or more storage pools mounted to the client of the computing device.

[0084] The data written by the computing device can come from other user terminals, servers, electronic devices, etc., or from user input, which is not limited here.

[0085] It can be understood that the computing device can read or obtain the disk information of each node in the storage system, and the related information of the configured storage pool in the storage system (such as the remaining capacity of each LUN in the storage pool, the capacity utilization, etc.). In addition, the computing device can also obtain the memory information (such as memory utilization) of each node in the storage system, the central processing unit (CPU) information (such as CPU utilization), the disk bandwidth (i.e. disk throughput), the input / output per second (IOPS) of the disk, etc., which are not limited here. And since the above data can change over time, the computing device can periodically obtain these data, or can obtain them when needed, so as to ensure the accuracy of the data.

[0086] 703. The computing device determines whether the disk type corresponding to the LUN in the target storage pool is a multi-armed disk. If it is a multi-armed disk, step 704 is performed, and if it is not a multi-armed disk, step 706 is performed.

[0087] Since one multi-armed disk can correspond to multiple LUNs, and one single-armed disk can correspond to one LUN, in order to meet the redundancy matching requirement, the computing device can use different disk selection strategies to select candidate disks for different types of storage pools (such as a storage pool composed of LUNs all of which are single-armed disks or a storage pool composed of LUNs all of which are double-armed disks). Specifically, when the computing device selects candidate disks from the target storage pool, it can first determine whether the disk type corresponding to the LUN in the target storage pool is a multi-armed disk or a single-armed disk. If it is a multi-armed disk (such as a double-armed disk), step 704 can be performed, and if it is not a multi-armed disk, step 706 can be performed.

[0088] It should be understood that since the disk type corresponding to the LUN in the default storage pool here is of the same type (such as all single-armed disks or all multi-armed disks), the computing device can only determine whether the disk corresponding to one LUN is a multi-armed disk.

[0089] 704. The computing device obtains the LUNs in the target storage pool that belong to the same multi-armed disk.

[0090] If it is determined that the disk type corresponding to the LUN in the target storage pool is a multi-arm disk, the computing device can label LUNs belonging to the same multi-arm disk in the target storage pool (i.e., obtain LUNs belonging to the same multi-arm disk in the target storage pool). In some embodiments, the computing device can label LUNs belonging to the same multi-arm disk using the SN of the multi-arm disk corresponding to each LUN. Alternatively, the computing device can also label LUNs belonging to the same multi-arm disk using the WWN (worldwide name), number, or name of the multi-arm disk corresponding to each LUN, etc., without limitation.

[0091] For example, such as Figure 8 As shown, assuming the storage pool includes n dual-arm disks (i.e., multi-arm disk 1 to multi-arm disk n), the computing device can number the LUNs for each dual-arm disk. The two LUNs of multi-arm disk 1 can be numbered LUN-1a and LUN-1b, the two LUNs of multi-arm disk 2 can be numbered LUN-2a and LUN-2b, and similarly, the two LUNs of multi-arm disk n can be numbered LUN-na and LUN-nb. Thus, by using the LUN numbers, it can be determined whether two LUNs belong to the same dual-arm disk. For example, it can be determined that LUN-4a and LUN-4b belong to the same dual-arm disk, while LUN-3a and LUN-4b do not.

[0092] It should be understood that in some embodiments, step 704 may be performed in advance, for example, before step 702, and this application embodiment is not limited thereto.

[0093] 705. The computing device selects a fourth number of candidate LUNs from the target storage pool based on a multi-arm disk selection strategy.

[0094] Specifically, based on all the LUNs in the target storage pool, the computing device can select a fourth number of candidate disks (i.e., candidate LUNs) from the target storage pool according to a multi-armed disk selection policy, the fourth number can be L, L is an integer greater than or equal to N+M (i.e., the first number + the second number, that is, the third number). Wherein, the selection policy can be that any two LUNs selected as candidate LUNs do not belong to the same multi-armed disk, that is, for the same multi-armed disk, at most one LUN of it can be selected as a candidate LUN. For example, for the two LUNs (i.e., LUN-1a and LUN-1b) of the multi-armed disk 1 described above, if LUN-1a is selected as a candidate LUN, then LUN-1b cannot be a candidate LUN, and if LUN-1b is selected as a candidate LUN, then LUN-1a cannot be a candidate LUN. In this way, it can be ensured that multiple shards of the same data stripe will not be stored in the LUNs of the same multi-armed disk. It should be understood that the number of LUNs included in the target storage pool can be greater than or equal to L.

[0095] In some embodiments, based on the above selection of candidate LUNs (i.e., any two LUNs selected as candidate LUNs do not belong to the same multi-armed disk), the multi-armed disk selection policy can include more selection conditions (i.e., selection principles), for example, when selecting candidate LUNs, the computing device can select candidate LUNs from the target storage pool based on the capacity balancing principle. In one possible implementation, when selecting candidate LUNs, the computing device can consider the capacity usage percentage (i.e., utilization) or the remaining capacity of each LUN in the target storage pool, and then can preferentially select LUNs with a relatively small current capacity usage percentage (e.g., less than a first threshold) or a relatively large remaining capacity as candidate LUNs. In this way, it can be ensured that each LUN in the target storage pool is load balanced. It should be understood that the first threshold can be set according to actual conditions, for example, when the capacity usage percentage of each LUN in the target storage pool is relatively large, the first threshold can be set to be relatively large (e.g., 70%), and when the capacity usage percentage of each LUN in the target storage pool is relatively small, the first threshold can be set to be relatively small (e.g., 30%), and for example, the first threshold can be set to the median or average of the capacity usage percentage of each LUN in the target storage pool, etc.

[0096] For another example, on the basis that any two LUNs in the selected candidate LUNs do not belong to the same multi-platter disk, the computing device can select the candidate LUN from the target storage pool based on the principle of performance balancing when selecting the candidate LUN. In a possible implementation, the computing device can consider the CPU utilization, memory load (i.e., memory usage), and disk bandwidth, IOPS, etc. of the node where each LUN in the target storage pool is located when selecting the candidate LUN, and then preferentially select a LUN with relatively small current CPU utilization (e.g., less than a second threshold value), relatively low current memory load (e.g., less than a third threshold value), and relatively small current disk bandwidth (e.g., less than a fourth threshold value) and relatively small current IOPS (e.g., less than a fifth threshold value) as the candidate LUN. In this way, the CPU resources, memory resources, disk resources, etc. of each node can be fully utilized, thereby improving the data write efficiency.

[0097] It should be understood that the second threshold value, the third threshold value, the fourth threshold value, and the fifth threshold value can be set according to actual conditions. For example, the second threshold value can be set according to the current CPU utilization of the node corresponding to each LUN in the target storage pool. When the current CPU utilization of each node is relatively large, the second threshold value can be set to be relatively large (e.g., 70%). When the current CPU utilization of each node is relatively small, the second threshold value can be set to be relatively small (e.g., 30%). Similarly, the third threshold value, the fourth threshold value, and the fifth threshold value can also be set according to the current memory load, the current disk bandwidth, and the current disk IOPS of the node corresponding to each LUN in the target storage pool, respectively. For another example, the second threshold value, the third threshold value, the fourth threshold value, and the fifth threshold value can be set to the median or average of the current CPU utilization, the current memory load, the current disk bandwidth, and the current disk IOPS of the node corresponding to each LUN in the target storage pool.

[0098] It should be understood that in some embodiments, on the basis that any two LUNs in the selected candidate LUNs do not belong to the same multi-platter disk, the computing device can select the candidate LUN from the target storage pool based on the principle of capacity balancing and performance balancing.

[0099] It should also be understood that in some embodiments, when the computing device selects the candidate LUNs from the target storage pool, the computing device can first select the candidate nodes, i.e., select the candidate nodes from the nodes corresponding to the LUNs in the target storage pool, and then select the candidate LUNs from the LUNs corresponding to the candidate nodes, where the candidate LUNs belong to the target storage pool. When selecting the candidate nodes, the computing device can consider the average capacity usage percentage or the average remaining capacity of all LUNs corresponding to each node in the target storage pool, and then select the node with a relatively small average capacity usage percentage or a relatively large average remaining capacity as the candidate node. Similarly, when selecting the candidate nodes, the computing device can also consider the CPU utilization, memory load of each node, and the average disk bandwidth, average IOPS, etc. of all LUNs corresponding to each node in the target storage pool, and then select the node with a relatively small CPU utilization, a relatively low memory load, and a relatively small average disk bandwidth, a relatively small average IOPS as the candidate node. The computing device selects the candidate LUNs from the LUNs corresponding to the candidate nodes in the same way as described above, and reference can be made to the above description.

[0100] It should be noted that in some embodiments, when the computing device selects the candidate LUNs from the target storage pool, the computing device also needs to consider the redundancy matching (e.g., needs to ensure that the number of selected candidate LUNs is greater than or equal to N+M) on the basis that any two LUNs in the selected candidate LUNs do not belong to the same multi-arm disk, so as to ensure that the computing device can successfully select the target LUN from the candidate LUNs later. For example, assuming that the target storage pool includes 24 LUNs (i.e., LUN1-LUN24), where LUN1-LUN4 belong to node a, LUN5-LUN8 belong to node b, LUN9-LUN12 belong to node c, LUN13-LUN16 belong to node d, LUN17-LUN20 belong to node e, and LUN21-LUN24 belong to node f. Wherein, the redundancy matching corresponding to the target storage pool is 4+2, then when the computing device selects the candidate LUNs from the target storage pool, it needs to select at least one LUN from LUN1-LUN4, and select at least one LUN from LUN5-LUN8. Similarly, at least one LUN needs to be selected from LUN9-LUN12, LUN13-LUN16, LUN17-LUN20, and LUN21-LUN24. Therefore, in the case that LUN1-LUN4 does not satisfy the above capacity balancing or performance balancing principle (e.g., in the case that the current capacity usage percentage of LUN1, LUN2, LUN3, and LUN4 is greater than or equal to the first threshold), the computing device still needs to select one LUN from LUN1-LUN4 as a candidate LUN, at this time, the LUN with a relatively small or the smallest current capacity usage percentage in LUN1-LUN4 can be selected.

[0101] 706. The computing device selects a fourth number of candidate LUNs from the target storage pool according to the single magnetic arm disk selection policy.

[0102] Specifically, based on all the LUNs in the target storage pool, the computing device can select a fourth number of candidate LUNs (i.e., L candidate LUNs) from the target storage pool according to the single magnetic arm disk selection policy. The selection policy can include a capacity balancing and / or performance balancing selection policy, and specific details can be referred to the related description in step 705. It should be understood that in some embodiments, when the disk type corresponding to the LUN in the target storage pool is a single magnetic arm disk, the computing device can directly select the candidate disk from the target storage pool without using the single magnetic arm disk selection policy.

[0103] It should also be understood that in some embodiments, when the computing device selects the candidate LUN from the target storage pool, it can first select the node, i.e., select the candidate node from the nodes corresponding to the LUNs in the target storage pool, and then select the candidate LUN from the LUNs corresponding to the candidate node. Specific details can be referred to the related description in step 705.

[0104] It should be noted that in some embodiments, when the computing device selects the candidate LUN from the target storage pool, it also needs to consider the redundancy ratio in order to ensure that the computing device can successfully select the target LUN from the candidate LUN later. Specific details can be referred to the related description in step 705.

[0105] 707. The computing device determines whether the redundancy ratio is the first number + the second number (i.e., N+M), and in the case of the first number + the second number, step 709 is performed, and in the case of not being the first number + the second number, step 708 is performed.

[0106] Since the selection of N+M (i.e., the third number) target LUNs from the candidate LUNs by the computing device is different in the case of N+M redundancy ratio and N+M:K (such as N+M:1, N+M:2, etc.) redundancy ratio. Therefore, the computing device can first determine whether the redundancy ratio corresponding to the target storage pool is N+M or N+M:K, and if it is N+M, step 709 can be performed, and if it is N+M:K, step 708 can be performed. It should be understood that 4+2, 8+4, etc. are N+M redundancy ratios, and 4+2:1, 8+4:1, 8+4:2, etc. are N+M:K redundancy ratios. K is an integer greater than or equal to 1, and K is less than M.

[0107] 708.The computing device selects a third number of target LUNs from the fourth number of candidate LUNs according to a first number+second number: sixth number (i.e., N+M:K) redundancy ratio, and writes the third number of shards (i.e., the first number of data shards and the second number of check shards) into the third number of target LUNs.

[0108] Specifically, in the case that the redundancy ratio corresponding to the target storage pool is N+M:K, the computing device can write data according to the intra / inter-node EC reliability, and accordingly, the computing device can select N+M target LUNs from L candidate LUNs according to the redundancy ratio corresponding to the target storage pool, and then can write N+M shards into the N+M target LUNs, i.e., can write N data shards and M check shards to N+M target LUNs (i.e., the disks corresponding to the target LUNs) corresponding to the fifth number of nodes (i.e., Y nodes) through the network. Wherein, (N+M) / M≤Y<N+M (such as 4+2:1, then Y can be 3).

[0109] Wherein, under the redundancy ratio of N+M:K, it is necessary to ensure that M disk failures or K node failures can be allowed without data loss. Wherein, since any two candidate LUNs in the L candidate LUNs obtained by the above selection do not belong to the same disk, the computing device can ensure that data is not lost when any N+M target LUNs are selected from the L candidate LUNs.

[0110] In order to meet the requirement that data is not lost when K nodes fail, the computing device selects N+M target LUNs from L candidate LUNs, and any [M+1, M+N] LUNs in the N+M target LUNs cannot belong to K nodes. And, for the candidate LUNs of the same node, at most LUNs in the candidate LUNs of the same node can be selected as target LUNs. Wherein, [M+1, M+N] represents any number in the range of (M+1) to (M+N), which can be (M+1) or (M+N). represents the ceiling of M / K, for example, is 3, is 2.

[0111] For example, please refer to Figure 9 , Figure 9 is another data storage scenario disclosed by the embodiments of the present application. As Figure 9As shown, the redundancy ratio corresponding to the target storage pool is 4+2:1, the target storage pool can include LUN1-LUN12, etc., and the candidate LUNs selected by the computing device from the target storage pool can include LUN1, LUN4, LUN6, LUN8, LUN9, LUN12, etc. It can be seen that for the two LUNs of the same dual-arm disk, the computing device can select one of the LUNs as a candidate LUN. After the computing device obtains the candidate LUNs, the computing device can select 4+2 LUNs from all the candidate LUNs as target LUNs, Figure 6 The six target LUNs selected by the computing device are LUN1, LUN4, LUN6, LUN8, LUN9, and LUN12. After that, the computing device can store the 4 data shards and 2 check shards in the selected six target LUNs.

[0112] It should be understood that in some embodiments, the computing device can write the N+M shards into the target LUNs through the internet protocol (IP) address of the node corresponding to the target LUN and the number, index, or name of the target LUN, etc. (i.e., the identification of the LUN), which can identify the information of the target LUN. Among them, the IP address of the node corresponding to the target LUN can be used for communication between the computing device and the corresponding node. For example, the computing device can send a data write request to the corresponding node through the IP address of each target LUN corresponding node, and the data write request can include the shards that need to be written and the information that can identify the target LUN, such as the number of the target LUN on the node. In this way, when the node receives the data write request from the computing device, it can determine the target LUN according to the number of the target LUN, etc. information, and then write the corresponding shards into the target LUN. For example, for node 1, data shard 1 and check shard 1 need to be written into LUN1 and LUN4 of node 1. The computing device can send a data write request to node 1, which can include data shard 1, check shard 1, and the identification of LUN1 and LUN4. When node 1 receives the data write request from the computing device, it can write data shard 1 and check shard 1 into LUN1 and LUN4 of node 1, respectively, according to the data write request.

[0113] 709. The computing device selects a third number of target LUNs from the fourth number of candidate LUNs according to the first number+second number (i.e., N+M) redundancy ratio, and writes the third number of shards into the third number of target LUNs.

[0114] Specifically, in the case that the redundancy ratio corresponding to the target storage pool is N+M, the computing device can write data according to the inter-node EC reliability, and accordingly, the computing device can select N+M target LUNs from the L candidate LUNs according to the redundancy ratio corresponding to the target storage pool, and then can write N+M shards into the N+M target LUNs, that is, can write N data shards and M check shards into N+M target LUNs corresponding to N+M nodes through the network.

[0115] It should be understood that in the redundancy ratio of N+M, it is necessary to ensure that M disk failures or M node failures can be allowed without data loss. Among them, since any two candidate LUNs in the L candidate LUNs obtained by the above selection do not belong to the same disk, the computing device can ensure that the data of the failed M disks is not lost by randomly selecting N+M target LUNs from the L candidate LUNs.

[0116] In order to meet the requirement that the data of K failed nodes is not lost, when the computing device selects N+M target LUNs from the L candidate LUNs, any two LUNs in the selected N+M target LUNs cannot belong to the same node, that is, the N+M target LUNs selected from the L candidate LUNs belong to N+M different nodes, that is, any node in the N+M nodes only includes one target LUN. Therefore, when selecting N+M target LUNs from the L candidate LUNs, for the candidate LUNs of the same node, the computing device can only select one candidate LUN as the target LUN.

[0117] For example, please refer to Figure 10 , Figure 10 is another data storage scenario disclosed by the embodiments of the present application. As shown in Figure 10 , the redundancy ratio corresponding to the target storage pool is 4+2, the target storage pool can include LUN1-LUN24, etc., the candidate LUNs obtained by the computing device from the target storage pool can include LUN1, LUN4, LUN6, LUN8, LUN9, LUN12, LUN13, LUN16, LUN18, LUN19, LUN21, LUN23, etc. It can be seen that for the two LUNs of the same double-arm disk, the computing device can select one LUN as a candidate LUN. After obtaining the candidate LUNs, the computing device can select 4+2 LUNs as target LUNs from all candidate LUNs, and the 6 target LUNs belong to different nodes, Figure 6 The 6 target LUNs selected in the above embodiment are LUN4, LUN6, LUN9, LUN12, LUN16, LUN18, LUN21, and then the computing device can store 4 data shards and 2 check shards into the selected 6 target LUNs.

[0118] In some embodiments, the computing device can first select N+M target LUNs, and then split the data into N data shards and generate M check shards. After obtaining the N data shards and the M check shards, the computing device can store the N data shards and the M check shards in the selected N+M target LUNs. Alternatively, in some other embodiments, the computing device can select the N+M target LUNs, split the data, and generate the check shards at the same time, which is not limited in the embodiments of the present application.

[0119] It should be noted that, in some embodiments, there can be LUNs belonging to the same multi-platter disk in the L candidate LUNs selected by the computing device. Then, when the computing device selects N+M target LUNs from the L candidate LUNs according to the redundancy ratio (such as N+M or N+M:K redundancy ratio) corresponding to the target storage pool, for the multiple LUNs belonging to the same multi-platter disk in the L candidate LUNs, the computing device can select at most one LUN as a target LUN, so that the N+M target LUNs can belong to N+M different disks respectively.

[0120] In the above method flow, for the multi-platter disk, the existing EC data shard redundancy protection algorithm (i.e., the improved disk selection strategy and data writing strategy) is improved. In the process of selecting candidate disks, the judgment process of multi-platter disks and single-platter disks is added. For the multi-platter disk, when selecting candidate LUNs from the storage pool, any two LUNs in the selected candidate LUNs do not belong to the same multi-platter disk under the condition of ensuring the EC redundancy ratio, so that the problem of local hot disk / load imbalance in the stripe (i.e., local hot disk / load imbalance of the multi-platter disk) can be solved. In this way, the N+M shards of the same data stripe can be evenly distributed on different multi-platter disks, so that the requirement of M failed disks can be met, and the reliability of the storage system can be improved.

[0121] In addition, when (N+M) / M≤the number of storage nodes<N+M, and the redundancy ratio is N+M:K (such as N+M:1 or N+M:2), the data writing method provided by the present application can make the storage system fail any K nodes or any M disks, and the EC redundancy protection level does not fail, that is, the data is not lost.

[0122] It should be noted that, Figure 7 In the method embodiment shown, the disk types corresponding to the LUNs in the default storage pool are of the same type (such as all single-platter disks or all multi-platter disks). The following describes the case where the disk types corresponding to the LUNs in the storage are not of the same type, for example, the LUNs include single-platter disks and multi-platter disks.

[0123] Based on the above system architecture, please refer to Figure 11 , Figure 11 is a flow diagram of another data writing method disclosed in the embodiments of the present application. As shown in Figure 11 , the data writing method can include but is not limited to the following steps:

[0124] 1101. The storage system configures at least one storage pool according to the obtained disk information, and each storage pool is configured with a corresponding redundancy ratio.

[0125] Step 1101 is the same as step 701, and the related description in the above step 701 can be referred to.

[0126] 1102. In the case that the computing device needs to write the to-be-stored data to the target storage pool, the computing device shards the to-be-stored data according to the redundancy ratio of the target storage pool.

[0127] Step 1102 is similar to step 702, and the related description in the above step 702 can be referred to.

[0128] 1103. The computing device obtains the LUNs belonging to the same multi-platter disk in the target storage pool.

[0129] Step 1103 is similar to step 704, and the related description in the above step 704 can be referred to.

[0130] 1104. The computing device selects a fourth number of candidate LUNs from the target storage pool according to the single-platter disk selection strategy and the multi-platter disk selection strategy.

[0131] Specifically, based on all the LUNs in the target storage pool, the computing device can select L candidate LUNs from the target storage pool according to the multi-platter disk selection strategy and the single-platter disk selection strategy, L being an integer greater than or equal to N+M. Among them, at most one LUN of the same multi-platter disk can be selected as a candidate LUN. And the multi-platter disk selection strategy and the single-platter disk selection strategy can include capacity balancing and / or performance balancing selection strategies, and the specific description can be referred to in step 705.

[0132] For more detailed description of step 1104, the related description in the above steps 705 and 706 can be referred to.

[0133] 1105. The computing device judges whether the redundancy ratio is the first number+the second number (i.e. N+M), and in the case of N+M, step 1107 is executed, and in the case of not N+M, step 1106 is executed.

[0134] Step 1105 is similar to step 707, and the related description in the above step 707 can be referred to.

[0135] 1106. The computing device selects the third number of target LUNs from the fourth number of candidate LUNs according to the first number + the second number: the sixth number (i.e., N+M:K) redundancy ratio, and writes the third number of shards (i.e., the first number of data shards and the second number of check shards) into the third number of target LUNs.

[0136] The step 1106 is similar to the step 708, and the related description in the step 708 can be referred to.

[0137] 1107. The computing device selects the third number of target LUNs from the fourth number of candidate LUNs according to the first number + the second number (i.e., N+M) redundancy ratio, and writes the third number of shards into the third number of target LUNs.

[0138] The step 1107 is similar to the step 709, and the related description in the step 709 can be referred to.

[0139] It should be understood that the operations performed by the computing device in the method processes of the above Figure 7 and Figure 11 may be the operations performed by the client of the computing device. It should also be understood that in some cases, a node (such as node 1) in the storage system can receive unsplit data (such as unsplit data sent by the computing device), and then the node can perform the above steps (i.e., steps 702-709 or steps 1102-1107) performed by the computing device to achieve the redundancy protection of the data.

[0140] It should be noted that the related information (i.e., the same information or similar information) and the related description in the above different embodiments can be referred to each other.

[0141] It should be understood that the above Figure 7 and Figure 11 take the computing device as an example of the execution subject of the interaction to illustrate the above processing flow, but the application does not limit the execution subject of the interaction. For example, Figure 7 and Figure 11 The computing device in the above may also be a chip, a chip system, or a processor supporting the implementation of the method by the computing device, and can also be a logic module or software capable of realizing all or part of the functions of the computing device.

[0142] Based on the above system architecture, please refer to Figure 12 , Figure 12 is a structural schematic diagram of an electronic device disclosed by the embodiments of the application. The electronic device 1200 can include a processor 1201, a communication interface 1202, and a memory 1203. The processor 1201, the communication interface 1202, and the memory 1203 can be connected to each other or connected to each other through a bus 1204.

[0143] The memory 1203 is configured to store computer programs and data of the electronic device 1200. The memory 1203 can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), and the like. The communication interface 1202 is configured to support the electronic device 1200 to communicate, for example, to receive or send data.

[0144] The processor 1201 can be a CPU, a complex programmable logic device, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like.

[0145] In an embodiment, the electronic device 1200 can be a computing device, and the processor 1201 can be configured to read programs stored in the memory 1203 described above, and execute the programs described above. Figure 7 Or Figure 11 The operations performed by the computing device in the method embodiments shown above can be referred to the related descriptions above, and will not be described in detail here.

[0146] In an embodiment, the electronic device 1200 can be a first node (i.e., any node in the storage system described above), and the processor 1201 can be configured to read programs stored in the memory 1203 described above, and execute the programs described above. Figure 7 Or Figure 11 The operations performed by the computing device in the method embodiments shown above can be referred to the related descriptions above, and will not be described in detail here.

[0147] It should be noted that, Figure 12 The electronic device 1200 shown above is only one implementation of the embodiments of the present application, and in actual applications, the electronic device 1200 can further include more or fewer components, which are not limited here.

[0148] The embodiments of the present application also disclose a computer readable storage medium, which stores instructions, and the instructions are executed to perform the method in the method embodiments described above.

[0149] The embodiments of the present application also disclose a computer program product including instructions, and the instructions are executed to perform the method in the method embodiments described above.

[0150] It should be apparent that the foregoing described embodiments are merely descriptive of some embodiments of the present application and are not meant to be limiting. Reference herein to "an embodiment" indicates that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all referring to a common embodiment, or an embodiment that is preferred over other embodiments. As will be apparent to those of ordinary skill in the art, embodiments described herein can be combined with other embodiments. All combinations of embodiments described herein are within the scope of the application. The terms "first", "second", "third", etc. are used herein to distinguish one element from another, and are not necessarily used to describe a particular sequential order, unless explicitly stated. Furthermore, the terms "comprise", "include", "contain", and "have" and variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements, but can include additional steps or elements not expressly listed or inherent to such process, method, article, or apparatus. It will be understood that the equality of the above-mentioned conditional judgment can be greater than one end or less than one end, for example, the above-mentioned conditional judgment of greater than, less than or equal to a threshold value can be changed to the conditional judgment of greater than or equal to, less than the threshold value, which is not limited herein. It should be understood that the above-mentioned communication mode can be indirect communication and direct communication, and direct communication can be understood as communication between two devices without other devices, and indirect communication can be understood as communication between two devices through other devices.

[0151] It should be noted that the steps in the above-mentioned method flow can be flexibly adjusted in some cases, for example, some steps in the above-mentioned method flow can be executed in advance, again for example, some steps in the above-mentioned method flow can be split or combined, or can be deleted, which is not limited herein.

[0152] It should be understood that only some of the parts related to the present application are shown in the drawings, and not all. It should be understood that some example embodiments are described as processes or methods depicted as flow diagrams. Although the operations (or steps) of the flow diagrams are depicted in a particular order, many of the operations can be performed concurrently, in parallel, or simultaneously. In addition, the order of the operations can be re-arranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the figure. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0153] The terms "component," "module," "system," "unit," and the like are used in the present specification to represent a computer-related entity, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. Also, the units can be executed from various computer-readable media having various data structures stored thereon. The units can communicate, for example, by way of local and / or remote processes, according to a signal having one or more data packets (e.g., from a second unit and to a second unit by way of a local system, distributed system, and / or across the Internet, e.g., via a signal between other systems).

[0154] The above detailed description has been made for the purpose of explaining the purpose, technical solutions, and beneficial effects of the present application. It should be understood that the above description is only a specific implementation of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application shall be included in the protection scope of the present application.

Claims

1. A data write method, characterized by, The method comprises: In the case that the computing device needs to write to-be-stored data into a target storage pool, the computing device shards the to-be-stored data according to a redundancy ratio of the target storage pool, to obtain a first number N of data shards and a second number M of check shards; The number of storage nodes is greater than or equal to ((N+M) / M) and less than (N+M), and the N+M:K redundancy ratio is adopted, K being an integer greater than or equal to 1 and less than M; If the disk types corresponding to a plurality of LUNs in the target storage pool are multi-arm disks, a fourth number of candidate LUNs is selected from the plurality of LUNs in the target storage pool based on a multi-arm disk selection strategy, and a third number of target LUNs is selected from the fourth number of candidate LUNs based on the redundancy ratio of the target storage pool; wherein the third number of target LUNs respectively correspond to a fifth number of different disks; The first number of data shards and the second number of check shards are stored in the third number of target LUNs; The method further comprises:

2. The method of claim 1, wherein, If the disk types corresponding to a plurality of LUNs in the target storage pool are single-arm disks, a fourth number of candidate LUNs is selected from the plurality of LUNs in the target storage pool based on a single-arm disk selection strategy; If the disk types corresponding to a plurality of LUNs in the target storage pool include single-arm disks and multi-arm disks, a fourth number of candidate LUNs is selected from the plurality of LUNs in the target storage pool based on a single-arm disk selection strategy and a multi-arm disk selection strategy; The fourth number of candidate LUNs is selected from the fourth number of candidate LUNs based on the redundancy ratio of the target storage pool. The multi-arm disk selection strategy comprises that the fourth number of candidate LUNs respectively correspond to a fourth number of different disks.

3. The method of claim 2, wherein, The multi-arm disk selection strategy further comprises one or more of the following: the current capacity usage percentage of the fourth number of candidate LUNs is less than a first threshold value, the current CPU utilization of the nodes corresponding to the fourth number of candidate LUNs is less than a second threshold value, the current memory load of the nodes corresponding to the fourth number of candidate LUNs is less than a third threshold value, the current bandwidth of the disks corresponding to the fourth number of candidate LUNs is less than a fourth threshold value, and the current IOPS of the disks corresponding to the fourth number of candidate LUNs is less than a fifth threshold value.

4. The method of claim 3, wherein, ​ 5. The method according to any one of claims 2-4, characterized in that, The single-magnetic-arm disk selection strategy includes one or more of the following: the current capacity usage percentage of the fourth number of candidate LUNs is less than a first threshold value, the current central processing unit (CPU) utilization of the nodes corresponding to the fourth number of candidate LUNs is less than a second threshold value, the current memory load of the nodes corresponding to the fourth number of candidate LUNs is less than a third threshold value, the current bandwidth of the disks corresponding to the fourth number of candidate LUNs is less than a fourth threshold value, and the current input / output per second (IOPS) of the disks corresponding to the fourth number of candidate LUNs is less than a fifth threshold value.

6. The method of claim 1 or 2, wherein, The selecting the third number of target LUNs from the fourth number of candidate LUNs based on the redundancy ratio of the target storage pool includes: In a case where the redundancy ratio of the target storage pool is a first number+second number, selecting the third number of target LUNs from the fourth number of candidate LUNs according to the first number+second number redundancy ratio, the third number of target LUNs corresponding to a third number of different nodes respectively.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Determining a plurality of LUNs belonging to the same multi-magnetic-arm disk in the target storage pool; Labeling the plurality of LUNs by a serial number (SN) of the multi-magnetic-arm disk.

8. The method according to any one of claims 1 to 4, characterized in that, The storing the first number of data shards and the second number of check shards into the third number of target LUNs includes: Sending a data write request to each node corresponding to the third number of target LUNs, each data write request including an identification of one or more LUNs in the third number of target LUNs, and one or more shards of the first number of data shards and the second number of check shards that need to be written into the one or more LUNs.

9. A data storage system, characterized by The method includes: A computing device and a distributed storage system, the computing device being connected to the distributed storage system, the distributed storage system including a plurality of storage nodes, each of the storage nodes including at least one disk, the computing device being configured to perform the method according to any one of claims 1 to 8.

Images