A dual-copy file storage method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a dual-copy file storage method, apparatus, device, and storage medium. The method includes: obtaining a first replica identifier set within each fault domain in a file storage system; evenly distributing replica identifiers in each first replica identifier set to a primary replica container pool and a secondary replica container pool; selecting an i-th primary replica identifier from the primary replica container pool in order from the 1st to the Nth; selecting an i-th secondary replica identifier from the secondary replica container pool based on the fault domain in which the i-th primary replica identifier is located; wherein the fault domain in which the i-th secondary replica identifier is located is different from the fault domain in which the i-th primary replica identifier is located; determining the i-th primary replica identifier and the i-th secondary replica identifier as an i-th dual-copy group; and storing a dual-copy file based on each replica identifier in the i-th dual-copy group.

Description

Technical Field

[0001] The present application relates to cloud computing file storage technology, and in particular to a dual-copy file storage method, apparatus, device, and computer-readable storage medium. Background Art

[0002] In related technologies, when placing dual-copy files, people generally only focus on determining the fault domain and consider it only from the perspective of capacity. The master and slave copies are placed randomly, resulting in low network resource utilization and low data reliability. Summary of the Invention

[0003] The embodiments of the present application are intended to provide a dual-copy file storage method, apparatus, device, and computer storage medium.

[0004] In a first aspect, an embodiment of the present application provides a dual-copy file storage method, the method comprising:

[0005] Obtain a set of first replica identifiers in each fault domain in the file storage system;

[0006] Evenly distributing the replica identifiers in each of the first replica identifier sets to the master replica container pool and the slave replica container pool;

[0007] Selecting the i-th primary replica identifier from the primary replica container pool in the order of the first to the N-th, wherein N is the number of dual replica identifiers contained in the file storage system, and i is an integer from 1 to N;

[0008] Selecting an i-th slave replica identifier from the slave replica container pool according to the fault domain where the i-th master replica identifier is located; wherein the fault domain where the i-th slave replica identifier is located is different from the fault domain where the i-th master replica identifier is located; and determining the i-th master replica identifier and the i-th slave replica identifier as an i-th dual replica group;

[0009] The dual copy files are stored according to each copy identifier in the i-th dual copy group.

[0010] In a second aspect, an embodiment of the present application provides a dual-copy file storage device, the device comprising:

[0011] An acquisition device, configured to acquire a first replica identifier set in each fault domain in the file storage system;

[0012] A first balanced distribution module, configured to evenly distribute the replica identifiers in each of the first replica identifier sets to the primary replica container pool and the secondary replica container pool;

[0013] A first selection module is configured to select an i-th primary replica identifier from the primary replica container pool in the order of the first to the N-th, where N is the number of dual replica identifiers included in the file storage system, and i is an integer from 1 to N;

[0014] A second selection module is configured to select an i-th slave replica identifier from the slave replica container pool according to the fault domain where the i-th master replica identifier is located; wherein the fault domain where the i-th slave replica identifier is located is different from the fault domain where the i-th master replica identifier is located;

[0015] a determination module, configured to determine the i-th master replica identifier and the i-th slave replica identifier as an i-th dual replica group;

[0016] The storage module is configured to store the dual copy file according to each copy identifier in the i-th dual copy group.

[0017] In a third aspect, the embodiment disclosed in this application also provides a dual copy file storage device, including: a memory and a processor

[0018] The memory is used to store executable instructions;

[0019] The processor is configured to implement any of the above-mentioned dual-copy file storage methods when executing the executable instructions stored in the memory.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing executable instructions for implementing any of the above-described dual-copy file storage methods when executed by a processor.

[0021] In an embodiment of the present application, the replica identifiers in the first replica identifier set within each fault domain are evenly distributed to the primary replica container pool and the secondary replica container pool. Then, the i-th primary replica identifier is selected from the primary replica container pool in the order of the 1st to the Nth. And, based on the fault domain where the i-th primary replica identifier is located, the i-th secondary replica identifier is selected from the secondary replica container pool to obtain the i-th dual replica group. Therefore, the replica identifiers can be evenly distributed to all storage fault domains as much as possible, thereby fully utilizing network resources. At the same time, because the formed dual replica groups are located in different fault domains, the reliability of file storage is high.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0024] Figure 1 This is a structural framework diagram of a distributed file storage system in related technology;

[0025] Figure 2 A flowchart of a dual-copy file storage method provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of another dual-copy file storage method provided in an embodiment of the present application;

[0027] Figure 4 A flowchart of another network dual copy file storage method provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of an implementation method for allocating replica locations in each fault domain in a file storage system to a primary replica container pool and a secondary replica location container pool, provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of an implementation method for allocating replica locations in each fault domain in a file storage system to a primary replica container pool and a secondary replica location container pool, provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of implementing replica group pairing provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of the structure of a dual-copy file storage device provided in an embodiment of the present application;

[0032] Figure 9 This is a structural diagram of a dual-copy file storage device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The present application will be further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely intended to explain the present application and are not intended to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions described in the embodiments of the present application may be implemented in any combination.

[0034] It should be noted that, in the embodiments of the present application, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus comprising a series of elements includes not only the elements explicitly stated, but also other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other related elements (such as steps in the method or units in the apparatus, for example, a unit may be a portion of a circuit, a portion of a processor, a portion of a program or software, etc.) in the method or apparatus comprising the element.

[0035] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three possible relationships exist. For example, "U and / or W" can represent three situations: the existence of U alone, the existence of both U and W, and the existence of W alone. Furthermore, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of items. For example, "at least one of U, W, and V" can represent any one or more elements selected from the set consisting of U, W, and V.

[0036] A distributed file storage system generally consists of a client, a system management service, a metadata management service, and a file data storage service. The client exposes standard file and metadata input and output (IO) interfaces through a file management system. Users can directly access the distributed file storage through these IO interfaces.

[0037] Figure 1 This is a structural framework diagram of a distributed file storage system in related technologies, such as Figure 1 As shown, the distributed file storage system includes: client 101, client 102, system management service 103, metadata management service 104, metadata management service 105, file data storage service 106, and file data storage service 107. Clients 101 and 102 are connected to and communicate with the backend metadata management service 104 and storage nodes where file data storage service 106 is located via a storage network. IO requests reach metadata management service 104, metadata management service 105, file data storage service 106, and file data storage service 107 via the storage network. System management service 103 may be implemented by one or more computing nodes.

[0038] To improve the reliability of metadata and file data, distributed file storage systems generally adopt a dual-copy approach to achieve high availability of metadata and file data. For a dual-copy group, it is generally necessary to select one as the primary copy and the other as the secondary copy. The primary copy stores the latest and most reliable data and directly carries the client's read and write IO requests. The secondary copy generally only passively synchronizes the data of the primary copy to ensure high data availability. Many existing distributed file storage systems, such as glusterfs, generally use mature algorithms, such as the Distributed Hash Table (DHT) method, to implement dual-copy placement. First, the file storage fault domain is determined, and then the primary and secondary replica locations within the fault domain are determined. Generally, both copies of data can be ensured to be located in different storage fault domains, thereby achieving high availability of files and metadata.

[0039] In actual engineering production, the above-mentioned distributed file storage system has the following deficiencies and shortcomings when implementing dual data copies:

[0040] (1) The means of setting the master and slave replicas are not perfect: In the related technologies, when setting up the dual-copy file placement, generally only the determination of the fault domain is focused on, and the master and slave replicas are determined randomly, or even manually.

[0041] (2) Failure to fully consider performance optimization at the network level: Distributed file storage system clients and back-end storage nodes are generally interconnected through the network. When the storage network performance is high, such as using Remote Direct Memory Access (RDMA) technology, making full use of the storage network performance can further improve the performance of the entire distributed file storage. Therefore, it is necessary to consider distributing the fault domains where the primary and secondary copies are located on different network paths as much as possible.

[0042] Based on the above technical problems, the embodiment of the present application provides a dual copy file storage method, such as Figure 2 As shown, the method includes:

[0043] Step S201: Obtain a first copy identifier set in each fault domain in the file storage system.

[0044] Here, the fault domain refers to a storage fault domain. In one example, a fault domain refers to a group of servers, storage, and / or network connectivity components that are collectively affected by a fault. For example, a fault domain can be at the storage server level, switch level, or storage server rack level. Accordingly, a fault domain can be two different storage servers, two different switches, or two different storage server racks.

[0045] In one example, the first replica identifier set can be a set consisting of multiple replica identifiers; here, the replica identifier can be a replica location; the identifier of one of the dual replicas, or the primary replica identifier, or the secondary replica identifier. Generally, a fault domain contains multiple replica locations; the first replica identifier set in each fault domain can be a set consisting of multiple replica identifiers contained in each fault domain. For example, for fault domain 11, fault domain 12, and fault domain 13, the replica identifiers in fault domain 11 include: 1101 and 1102, the replica identifiers in fault domain 12 include: 1201 and 1202; the replica identifiers in fault domain 13 include: 1301, 1302, 1303, and 11304; correspondingly, the first replica identifier set in fault domain 11 is {1101, 1102}; the first replica identifier set in fault domain 12 is {1201, 1202}; and the first replica identifier set in fault domain 13 is {1301, 1302, 1303, 1304}.

[0046] In a possible implementation, obtaining the first replica identifier set in each fault domain in the file storage system may be that the dual-replica file storage device obtains a replica identifier set consisting of multiple replica identifiers contained in each fault domain in the file storage system.

[0047] Here, the dual copy file storage device may be a device for implementing the above-mentioned system management service.

[0048] Step S202: Evenly distribute the replica identifiers in each of the first replica identifier sets to the primary replica container pool and the secondary replica container pool.

[0049] Here, the master copy container pool and the slave copy container pool refer to storage resource pools for storing master copy identifiers and slave copy identifiers, respectively.

[0050] In a possible implementation, the replica identifiers in each of the first replica identifier sets are evenly distributed to the primary replica container pool and the secondary replica container pool. This may be the case where the number of replica identifiers in the first replica identifier set is even, where half of the replica identifiers in the first replica identifier set are distributed to the primary replica container pool, and the other half of the replica identifiers in the first replica identifier set are distributed to the secondary replica container pool.

[0051] In another possible implementation, the replica identifiers in each of the first replica identifier sets are evenly distributed to the master replica container pool and the slave replica container pool. This can be done by, if the number of replica identifiers in the first replica identifier set is odd, allocating half of the sum of the odd number plus 1 to the master replica container pool and the remaining replica identifiers in the first replica identifier set to the slave replica container pool. Alternatively, if the number of replica identifiers in the first replica identifier set is odd, allocating half of the sum of the odd number plus 1 to the slave replica container pool and the remaining replica identifiers in the first replica identifier set to the master replica container pool. For example, if the 14th replica set is {1401, 1402, 1403}, either 1401 and 1402 can be allocated to the master replica container pool and 1403 to the slave replica container pool, or 1401 and 1402 can be allocated to the slave replica container pool and 1403 to the slave replica container pool.

[0052] Step S203: Select the i-th primary copy identifier from the primary copy container pool in the order of the 1st to the Nth; N is the number of dual copy identifiers contained in the file storage system; i is an integer from 1 to N.

[0053] Here, N is a positive integer greater than 1, and is determined based on the number of dual copy identifiers included in the file storage system. In one example, if file storage system 1 includes an even number of replica identifiers, for example, file storage system 1 includes 20 replica identifiers, and file storage system 2 includes an odd number of replica identifiers, for example, file storage system 2 includes 21 replica identifiers, the number of dual copy identifiers included in file storage system 1 and file storage system 2 is the same, namely, 10.

[0054] In one possible implementation, the i-th primary replica identifier is selected from the primary replica container pool in the order from the 1st to the Nth. This can be accomplished by first randomly selecting a replica identifier from the primary replica container pool as the first primary replica identifier, then selecting a replica identifier from the remaining replica identifiers in the primary replica container pool as the second primary replica identifier, and so on until the N-th primary replica identifier is selected. It is understood that if the total number of replica identifiers in the file storage system is an even number, the number of replica identifiers contained in the primary replica container pool and the number of replica identifiers contained in the secondary replica container pool are the same, i.e., N. Therefore, after the N-th primary replica identifier is selected, the primary replica container pool is empty. Of course, if the total number of replica identifiers in the file storage system is an odd number, 2N+1, the number of dual replicas contained is N, and the primary replica container pool contains N+1 replica identifiers, then if the secondary replica container pool contains N replica identifiers, after the N-th primary replica identifier is selected, one replica identifier will remain in the primary replica container pool. For example, for a file storage system including 4 dual replicas, the first replica identifier set in fault domain A is {1001, 1002}; the first replica identifier set in fault domain B is {2001, 2002}; the first replica identifier set in fault domain C is {3001, 3002, 3003, 3004}, and 1001 and 1002 in the replica identifier set in fault domain A are respectively assigned to the primary replica container pool and the secondary replica container pool; 2001 and 2002 in the replica identifier set in fault domain B are respectively assigned to the primary replica container pool and the secondary replica container pool; 3001 and 3002 in the replica identifier set in fault domain C are assigned to the primary replica container pool, and 3003 and 3004 are assigned to the secondary replica container pool. In the case where 2001 is selected from the primary replica container pool as the first primary replica identifier in the order of the first to the fourth.

[0055] Step S204: Selecting an i-th slave replica identifier from the slave replica container pool according to the fault domain where the i-th master replica identifier is located; wherein the fault domain where the i-th slave replica identifier is located is different from the fault domain where the i-th master replica identifier is located.

[0056] In some possible implementations, the i-th slave replica identifier is selected from the slave replica container pool based on the failure domain where the i-th master replica identifier is located. This may involve selecting a replica identifier from the slave replica container pool that is different from the failure domain where the i-th master replica identifier is located, and using the selected replica identifier as the i-th slave replica identifier. For example, based on failure domain B of 2001, any one of 1002, 3003, or 3004 may be selected from the slave replica container pool as the first slave replica identifier.

[0057] Step S205: Determine the i-th master replica identifier and the i-th slave replica identifier as the i-th dual replica group.

[0058] In some possible implementations, determining the i-th master replica identifier and the i-th slave replica identifier as the i-th dual replica group may be by forming the i-th dual replica group using the i-th master replica identifier and the i-th slave replica identifier. For example, the first dual replica group (2001, 1002) is obtained by using 2001 and 1002; or the first dual replica group (2001, 3003) is obtained by using 2001 and 3003; or the first dual replica (2001, 3004) is obtained by using 2001 and 3004.

[0059] Step S206: storing the dual copy file according to each copy identifier in the i-th dual copy group.

[0060] In some possible implementations, storing a dual-copy file according to each replica identifier in the i-th dual-copy group may involve storing replica 1 in the dual-copy file at one storage identifier in the i-th dual-copy group, and storing replica 2 in the dual-copy file at another storage identifier in the i-th dual-copy group. For example, when the i-th dual-copy group is (2001, 1002) or (2001, 3003) or (2001, 3004), one of the dual-copy files is stored at identifier 2001, and the other is stored at identifier 1002, 3003, or 3004.

[0061] In practical applications, steps S201 to S206 can be implemented using a processor in a dual-copy file storage device, and the processor can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0062] In the embodiment of the present application, the replica identifiers in the first replica identifier set in each fault domain are evenly distributed to the primary replica container pool and the secondary replica container pool. Then, the i-th primary replica identifier is selected from the primary replica container pool in the order from the 1st to the Nth. And based on the fault domain where the i-th primary replica identifier is located, the i-th secondary replica identifier is selected from the secondary replica container pool to obtain the i-th dual replica group. Therefore, the replica identifiers can be evenly distributed to all storage fault domains as much as possible, making full use of network resources. At the same time, because the formed dual replica groups are located in different fault domains, the reliability of file storage is high.

[0063] The present application also provides another dual copy file storage method, such as Figure 3 As shown, the method includes:

[0064] Step S301: Obtain a first copy identifier set in each fault domain in the file storage system.

[0065] Step S302: Evenly distribute the replica identifiers in each replica identifier set to the corresponding master replica identifier queue and the corresponding slave replica identifier queue.

[0066] Here, the replica identification queue can be a storage queue that can reflect the order of the selected replica identifications. Correspondingly, the master replica identification queue is used to store the selected master replica identifications in sequence; the slave replica identification queue is used to store the selected slave replica identifications in sequence.

[0067] In some possible implementations, one replica identifier corresponds to one master replica identifier queue and one slave replica identifier queue, that is, one fault domain corresponds to one master replica identifier queue and one slave replica identifier queue.

[0068] As for the implementation method of evenly distributing the replica identifiers in each of the replica identifier sets to the corresponding master replica identifier queue and the corresponding slave replica identifier queue, exemplarily, when the number of replica identifiers in the first replica identifier set is an even number, half of the replica identifiers in the first replica identifier set are distributed to the corresponding master replica identifier queue, and the other half of the replica identifiers in the first replica identifier set are distributed to the corresponding slave replica identifier queue. For example, the first replica identification set corresponding to fault domain 5 is replica identification set 5, replica identification set 5 corresponds to master replica identification queue 51 and slave replica identification queue 52, the first replica identification set corresponding to fault domain 6 is replica identification set 6, replica identification set 6 corresponds to master replica identification queue 61 and slave replica identification queue 62, and replica identification set 5 is {5001, 5002}; if replica identification set 6 is {6001, 6002}, then 5001 can be assigned to master replica identification queue 51, 5002 can be assigned to slave replica identification queue 52; 6001 can be assigned to master replica identification queue 61 and 6002 can be assigned to slave replica identification queue 62.

[0069] In another possible embodiment, the replica identifiers in each replica identifier set are evenly distributed to the corresponding master replica identifier queue and the corresponding slave replica identifier queue. For example, when the number of replica identifiers in the first replica identifier set is an odd number, half of the replica identifiers of the sum of the odd number plus 1 are distributed to the corresponding master and slave identifier queues, and the remaining replica identifiers in the first replica identifier set are distributed to the slave replica identifier queue; or when the number of replica identifiers in the first replica identifier set is an odd number, half of the replica identifiers of the sum of the odd number plus 1 are distributed to the slave replica identifier queue, and the remaining replica identifiers in the first replica identifier set are distributed to the master replica identifier queue. For example, for the replica set 7 corresponding to fault domain 7, the replica identification set 7 corresponds to the master replica identification queue 71 and the slave replica identification queue 72. When the replica set 7 is {7001, 7002, 7003}, 7001 and 7002 can be assigned to the corresponding master replica identification queue 71, and 7003 can be assigned to the slave replica identification queue 72, or 7001 and 7002 can be assigned to the slave replica identification queue 72, and 7003 can be assigned to the slave master replica identification queue 71.

[0070] Step S303: constructing a master copy container pool according to the plurality of master copy identification queues.

[0071] In some possible implementations, building a master replica container pool according to the multiple master replica identifier queues may involve storing the replica identifiers stored in the multiple master replica identifier queues in corresponding master replica container pools.

[0072] Step S304: construct a secondary replica container pool according to the plurality of secondary replica identifier queues.

[0073] In some possible implementations, constructing a slave replica container pool according to the multiple slave replica identifier queues may involve storing the replica identifiers stored in the multiple slave replica identifier queues in corresponding slave replica container pools.

[0074] Step S305: Select the i-th primary copy identifier from the primary copy container pool in the order of the 1st to the Nth; N is the number of dual copy identifiers contained in the file storage system; i is an integer from 1 to N.

[0075] Step S306: Determine a second replica identifier set in the slave replica container pool that is different from the failure domain where the i-th master replica identifier is located.

[0076] It can be understood that each replica identifier in the second replica identifier set is in a different fault domain from the i-th primary replica identifier.

[0077] In one possible implementation, determining the second set of replica identifiers in the slave replica container pool that are in a different fault domain than the i-th master replica identifier is located in can include determining multiple replica identifiers from the slave replica container that are in a different fault domain than the i-th master replica identifier is located in, and determining the set consisting of these multiple replica identifiers as the second set of replica identifiers. For example, if the slave replica container pool includes 7004, 6006, 5005, and 4005, which are located in fault domains 7, 6, 5, and 4, respectively; if the i-th master replica identifier is 4004 and the fault domain is 4, then the determined second set of replica identifiers includes 7004, 6006, and 5005.

[0078] Step S307: Determine any replica identifier selected from the second replica set as the i-th slave replica identifier; wherein the fault domain where the i-th slave replica identifier is located is different from the fault domain where the i-th master replica identifier is located.

[0079] In some possible implementations, any replica identifier selected from the second replica set is determined as the i-th slave replica identifier. When the second replica identifier set includes 7004, 6006 and 5005, 7004 or 6006 or 5005 is determined as the i-th slave replica identifier.

[0080] Step S308: Determine the i-th master replica identifier and the i-th slave replica identifier as the i-th dual replica group.

[0081] Step S309: storing the dual copy file according to each copy identifier in the i-th dual copy group.

[0082] In an embodiment of the present application, the replica identifiers in each replica identifier set are evenly distributed to the corresponding master replica identifier queue and the corresponding slave replica identifier queue; a master replica container pool is constructed based on the multiple master replica identifier queues; and a slave replica container pool is constructed based on the multiple slave replica identifier queues. The master replica container pool and the slave replica container pool can be constructed in the order in which the replica identifiers are selected, thereby avoiding omission of replica identifiers; a second replica identifier set in the slave replica container pool is determined, which is different from the failure domain where the i-th master replica identifier is located; and any replica identifier selected from the second replica set is determined as the i-th slave replica identifier. The i-th replica identifier can be selected from the second replica identifier set, which provides a certain degree of selectivity.

[0083] The present application embodiment further provides a dual copy file storage method, such as Figure 4 As shown, the method includes:

[0084] Step S401: Obtain a first copy identifier set in each fault domain in the file storage system.

[0085] Step S402: Evenly distribute the replica identifiers in each of the first replica identifier sets to the primary replica container pool and the secondary replica container pool.

[0086] Step S403: Select the i-th primary copy identifier from the primary copy container pool in the order of the 1st to the Nth; N is the number of dual copy identifiers contained in the file storage system; i is an integer from 1 to N.

[0087] Step S404: Select an i-th slave replica identifier from the slave replica container pool according to the fault domain where the i-th master replica identifier is located; wherein the fault domain where the i-th slave replica identifier is located is different from the fault domain where the i-th master replica identifier is located.

[0088] Step S405: Determine the i-th master replica identifier and the i-th slave replica identifier as the i-th dual replica group.

[0089] Step S406: storing the dual copy file according to each copy identifier in the i-th dual copy group.

[0090] Step S407: when the slave replica container pool does not have a replica identifier different from the fault domain where the i-th master replica identifier is located, obtain the fault domain where the replica identifier in each replica group from the 1st to the i-1th replica groups is located.

[0091] Here, the case where the slave replica container pool does not have a replica ID that is different from the failure domain where the i-th master replica ID is located may be when the remaining replica IDs in the master replica container pool and the slave replica container pool are the same. For example, when the remaining replica IDs in the master replica container pool are 1303 and 1304, and the remaining replica IDs in the slave replica container pool are 1305 and 1306.

[0092] In the embodiment of the present application, replica identifiers are selected from the primary replica container pool and the secondary replica container pool in the order of the 1st to the Nth. Therefore, if the secondary replica container pool does not contain a replica identifier that is different from the failure domain where the i-th primary replica identifier is located, the 1st to the (i-1)th replica groups are already constructed replica groups whose replica identifiers are in different failure domains.

[0093] In some possible implementations, obtaining the failure domain where the replica identifiers in each replica group from the 1st to the i-1th replica identifier groups reside may involve obtaining the failure domain where each replica identifier in the 1st replica identifier group resides, then obtaining the failure domain where each replica identifier in the 2nd replica identifier group resides, and so on until the failure domain where each replica identifier in the i-1th replica group resides is obtained. For example, when i is 3, the first replica group is (1002, 6002), and the second replica group is (1003, 5002), the failure domain where 1002 and the failure domain where 6002 are located in the first replica group are obtained, and then the failure domain where 1003 and the failure domain where 5002 are located in the second replica group are obtained.

[0094] Step S408: Select a target replica group from the 1st to i-1th replica groups based on the fault domain where the replica identifier in each replica group in the 1st to i-1th replica groups is located; the fault domain where the replica identifier in the target replica group is located is different from the fault domain where the i-th master replica identifier is located.

[0095] It is understood that the target replica group is the replica group containing the replica identifier that needs to be exchanged with the i-th primary replica identifier. In one example, the target replica group can be any target replica group in the target replica group set. Here, the fault domain containing the replica identifier in any target replica group in the target replica group set is different from the fault domain containing the i-th primary replica identifier.

[0096] In some possible implementations, the target replica group is selected from the first to the i-1th replica groups based on the failure domain where the replica identifier in each of the first to the i-1th replica groups is located. This may be based on the failure domain where the replica identifier in each of the first to the i-1th replica groups is located, and the failure domain where the replica identifier selected from the first to the i-1th replica groups is located is different from the failure domain where the i-th primary replica identifier is located. For example, when i is 3, the first replica group is (1002, 6002), the second replica group is (1003, 5002), and the third primary replica identifier is 3005, based on the failure domains where 1002, 6002, 1003, and 5002 are located, the first replica group (1002, 6002) or the second replica group (1003, 5002) is determined to be the target replica group.

[0097] Step S409: Exchange the i-th primary replica identifier with the target replica identifier to obtain the exchanged i-th primary replica identifier and the exchanged target replica group; the target replica identifier is any replica identifier in the target replica group.

[0098] In some possible implementations, the i-th master copy identifier is exchanged with the target copy identifier to obtain the exchanged i-th master copy identifier and the exchanged target copy group. The i-th master copy identifier can be exchanged with any copy identifier in the target copy group to obtain the exchanged i-th master copy identifier and the exchanged target copy identifier.

[0099] Here, when i is 3, the target replica group is (1003, 5002), the target replica identifier is 1003, and the third master replica identifier is 3005, 3005 and 1003 can be exchanged, and the third master replica identifier after the exchange is 1003, and the target replica group after the exchange is (3005, 5002).

[0100] Step S410: Select any replica identifier in the slave replica container pool as the i-th slave replica identifier.

[0101] In one possible implementation, selecting any replica identifier in the slave replica container pool as the i-th slave replica identifier may be selecting any remaining replica identifier in the slave replica container pool as the i-th slave replica identifier. For example, if the slave replica container pool includes 3006 and 3007, either 3006 or 3007 may be selected as the third slave replica identifier.

[0102] Step S411: Determine the exchanged i-th master replica identifier and the i-th slave replica identifier as the i-th dual replica group.

[0103] In some possible implementations, determining the swapped i-th master replica identifier and the i-th slave replica identifier as the i-th dual replica group may be performed by forming the i-th dual replica group with the swapped i-th master replica identifier and the i-th slave replica identifier. For example, the swapped third master replica identifier 1003 and the third slave replica identifier 3006 or 3007 may form the third replica group (1003, 3006) or (1003, 3007).

[0104] In an embodiment of the present application, when there is no replica identifier in the slave replica container pool that is different from the fault domain where the i-th primary replica identifier is located, the fault domain where the replica identifier in each replica group from the 1st to i-1th replica groups is located is obtained; based on the fault domain where the replica identifier in each replica group from the 1st to i-1th replica groups is located, the target replica group is selected from the 1st to i-1th replica groups, and the i-th primary replica identifier and the target replica identifier are exchanged to obtain the exchanged i-th primary replica identifier and the exchanged target replica group, and the i-th dual replica group is formed by any replica identifier in the slave replica container pool and the exchanged i-th primary replica identifier. The replica identifiers in the obtained i-th dual replica group are located in different fault domains, and although they are exchanged with the target replica identifier, the replica identifiers in the exchanged target replica group are still located in different fault domains. Therefore, it is not only beneficial to evenly distribute the replica identifiers to the storage fault domains as much as possible and fully utilize network resources, but also to improve the reliability of file storage.

[0105] In an embodiment of the present application, a dual-copy file storage method is proposed, comprising: obtaining all replica locations in all fault domains included in a file storage system, and evenly dividing all replica locations in each fault domain into a corresponding master replica location queue and a corresponding slave replica location queue; constructing a master replica container pool using all the master replica location queues; constructing a slave replica container pool using all the slave replica location queues; and respectively taking one replica location each from the master and slave replica container pools to form the dual-copy group until the master and slave replica container pools are empty.

[0106] It can be understood that the total number of replica positions of all the above-mentioned fault domains must meet the even number condition to ensure that the total number of master and slave replicas in the master-slave replica container pool constructed by the above-mentioned master-slave replica position queues is equal; and each of the above-mentioned fault domains corresponds to a master replica position queue and a slave replica position queue, that is, each master replica position queue and each slave replica position queue have fault domain attributes.

[0107] In an embodiment of the present application, the dual replica group is composed of a master replica location in a master replica container pool and a slave replica location in a slave replica container pool. During the pairing process of the dual replica group, a replica location is randomly selected from the master replica pool and the slave replica container pool respectively. The replica location must meet the following conditions: the failure domains of the master replica location and the slave replica location cannot be the same. When only one replica location queue remains in each of the master replica container pool and the slave replica container pool, and the failure domains of the replica locations in the master replica location queue and the slave replica location queue are the same, the remaining unpaired replica locations in the master replica location queue and / or the slave replica location queue are exchanged one by one with the successfully paired dual replica group. Here, the exchange must meet the following condition: the failure domains of the remaining unpaired replica locations must be different from the failure domains of the target master and slave replica locations to be exchanged.

[0108] The embodiment of the present application provides a method for allocating replica locations in each fault domain in a file storage system to a primary replica container pool and a secondary replica location container pool, such as Figure 5 As shown, the file storage system includes fault domains 1 to 3. The replica locations included in fault domain 1 are: 101 and 102; the replica locations included in fault domain 2 are: 201 and 202; the replica locations included in fault domain 3 are: 301, 302, 303 and 304; the replica locations included in the primary replica container pool are: 201, 101, 301 and 302; the replica locations included in the secondary replica container pool are: 102, 202, 303 and 304; according to Figure 5 As you can see, each fault domain contains an even number of replica locations, so the total number of replica locations in the file storage system must also be an even number. The primary replica container pool contains half of the replica locations in each fault domain, and the secondary replica container pool contains half of the replica locations in each fault domain.

[0109] The embodiment of the present application also provides a method for allocating replica locations in each fault domain in a file storage system to a primary replica container pool and a secondary replica location container pool, such as Figure 6 As shown, the file storage system includes fault domains 1 to 3. The replica locations included in fault domain 1 are: 101; the replica locations included in fault domain 2 are: 201 and 202; the replica locations included in fault domain 3 are: 301, 302, and 303; the replica locations included in the primary replica container pool are: 201, 101, and 301; the replica locations included in the secondary replica container pool are: 202, 302, and 303; wherein, Figure 6 and Figure 5 The difference is: Figure 6In the example, fault domain 1 and fault domain 3 contain an odd number of replica locations. However, the total number of replica locations in all fault domains must be an even number. For fault domains containing an odd number of replica locations, the corresponding replica container pool is divided according to the divisor of the number of replica locations by 2 plus 1 or the divisor of the number of replica locations by 2. Figure 6 It can be seen that the primary replica container pool includes the replica location of fault domain 1 and one replica location of fault domain 3. The secondary replica container pool does not include the replica location of fault domain 1, but includes two replica locations of fault domain 3.

[0110] Figure 7 The embodiment of the present application also provides a schematic diagram for implementing replica group pairing, such as Figure 7 As shown, the master replica container pool includes replica locations 201, 101, 301, and 302; the slave replica container pool includes replica locations 102, 202, 304, and 304. When selecting reset positions in the primary replica container pool and the secondary replica container pool to pair replica groups, 101 is first selected from the primary replica container pool and 202 is selected from the secondary replica container pool to form the first replica group (101, 202); then, 201 is selected from the primary replica container pool and 102 is selected from the secondary replica container pool to form the second replica group (201, 102); at this time, the remaining replica positions in the primary replica container pool are: 301 and 302, and the remaining replica positions in the secondary replica container pool are: 304 and 304. Since the fault domains of the remaining replica positions in the primary replica container pool and the remaining replica positions in the secondary replica container pool are both fault domain 3, which is the same fault domain, for the remaining unpaired replica positions, no matter how they are paired, it cannot be guaranteed that the replica positions in the third replica group and / or the fourth replica group obtained by pairing are in the same fault domain, and the requirement that the replica positions in the replica group are in different fault domains cannot be met.

[0111] In this embodiment of the present application, the replica positions in the third replica group that were not successfully paired are: 301 and 302; the replica positions in the fourth replica group that were successfully paired are: 303 and 304. Therefore, it is necessary to swap 301 or 301 in the third replica group that were not successfully paired with any replica position in the first replica group or the second replica group that were successfully paired, to obtain a successfully paired third replica group and a successfully paired fourth replica group. For example, swapping 301 and 101 results in a new first replica group (301, 202) and a successfully paired third replica group (101, 302); swapping 303 and 201 results in a new second replica group (303, 102) and a successfully paired fourth replica group (201, 304).

[0112] In the embodiment of the present application, the process of implementing replica group pairing is to select one replica position from each master-slave replica container pool each time to perform replica group pairing, but the two replicas must not be in the same fault domain. Since the replica positions are randomly selected, it is possible that the replica positions of fault domains 1 and 2 are paired, and the replica position of fault domain 3 remains unpaired. Therefore, an interactive step is required to exchange the unpaired fault domain replica position with the replica position of the paired replica group. The fault domain of the unpaired replica position must be different from the fault domain of the two replica positions of the paired replica group. After the exchange step, the replica group that meets the requirements can be finally completed.

[0113] The dual-copy file storage method provided in the embodiment of the present application optimizes the placement method and enriches the means of replica group placement. Especially when the number of replicas is very large, there is no need to manually specify the replica group, which greatly improves the actual engineering efficiency; and the master and slave replica positions of the replica placement group can be spread as evenly as possible to all storage fault domains. On the one hand, it fully utilizes network resources and improves performance. On the other hand, it avoids the occurrence of too many master replicas in certain fault domains, resulting in network performance bottlenecks.

[0114] Figure 8 A schematic diagram of the structure of the dual copy file storage device provided in the embodiment of the present application is shown as follows: Figure 8 As shown, the dual copy file storage device 800 includes:

[0115] An acquisition module 801 is configured to acquire a first replica identifier set in each fault domain in the file storage system;

[0116] A balanced distribution module 802 is configured to evenly distribute the replica identifiers in each of the first replica identifier sets to the primary replica container pool and the secondary replica container pool;

[0117] The first selection module 803 is configured to select the i-th primary replica identifier from the primary replica container pool in the order of the first to the N-th, where N is the number of dual replica identifiers included in the file storage system, and i is an integer from 1 to N.

[0118] A second selection module 804 is configured to select an i-th slave replica identifier from the slave replica container pool according to the fault domain where the i-th master replica identifier is located; wherein the fault domain where the i-th slave replica identifier is located is different from the fault domain where the i-th master replica identifier is located;

[0119] A determination module 805 is configured to determine the i-th master replica identifier and the i-th slave replica identifier as an i-th dual replica group;

[0120] The storage module 806 is configured to store the dual copy file according to each copy identifier in the i-th dual copy group.

[0121] In some embodiments, the balanced distribution module 802 is used to evenly distribute the replica identifiers in each replica identifier set to the corresponding master replica identifier queue and the corresponding slave replica identifier queue; build a master replica container pool based on multiple master replica identifier queues; and build a slave replica container pool based on multiple slave replica identifier queues.

[0122] In some embodiments, the second selection module 804 is configured to determine a set of second replica identifiers in the slave replica container pool that is different from the failure domain where the i-th master replica identifier is located; and determine any replica identifier selected from the second replica set as the i-th slave replica identifier.

[0123] In some embodiments, the dual-copy file storage device 800 also includes: an exchange module 807, which is used to exchange the i-th master copy identifier with the target copy identifier when there is no replica identifier in the slave copy container pool that is different from the fault domain where the i-th master copy identifier is located, to obtain the exchanged i-th master copy identifier and the exchanged target copy group; the target copy identifier is any replica identifier in the target copy group; select any replica identifier in the slave copy container pool as the i-th slave copy identifier; and determine the exchanged i-th master copy identifier and the i-th slave copy identifier as the i-th dual copy group.

[0124] In some embodiments, the dual-copy file storage device 800 also includes: a third selection module 808, which is used to obtain the fault domain where the replica identifier in each replica group from the 1st to i-1th replica groups is located when there is no replica identifier different from the fault domain where the i-th primary replica identifier is located in the slave replica container pool; based on the fault domain where the replica identifier in each replica group from the 1st to i-1th replica groups is located, select the target replica group from the 1st to i-1th replica groups; the fault domains where the replica identifiers in the target replica group are located are all different from the fault domain where the i-th primary replica identifier is located.

[0125] In actual applications, the acquisition module 801, the second acquisition module 802, the first selection module 803, the second selection module 804, the determination module 805, the storage module 806, the exchange module 807 and the third selection module 808 can be implemented using the processor in the dual-copy file storage device, and the above-mentioned processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0126] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0127] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0128] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a dual-copy file storage device to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0129] Generally speaking, the computer program instructions corresponding to a dual-copy file storage method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the computer program instructions corresponding to a dual-copy file storage method in the storage medium are read or executed by a dual-copy file storage device, any dual-copy file storage method in the aforementioned embodiments is implemented.

[0130] Based on the same technical concept as the above embodiment, see Figure 9 , which shows a schematic structural diagram of a dual-copy file storage device 900 provided in an embodiment of the present application, which may include: a memory 901 and a processor 902; wherein,

[0131] The memory 901 is used to store computer programs and data;

[0132] The processor 902 is configured to execute the computer program stored in the memory to implement any one of the dual-copy file storage methods of the aforementioned embodiments.

[0133] In one embodiment, the dual copy file storage device 900 may be a device for implementing the above-mentioned system management service.

[0134] In practical applications, the memory 901 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 902.

[0135] The processor 902 may be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor. It is understood that for different augmented reality cloud platforms, the electronic device used to implement the above processor functions may also be other, and this embodiment of the application is not limited thereto.

[0136] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0137] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0138] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0139] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0140] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course, by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a dual copy file storage device to execute the methods described in each embodiment of the present application.

[0141] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned implementation methods. The above-mentioned implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A dual copy file storage method, characterized in that: include: Obtain a set of first replica identifiers in each fault domain in the file storage system; Evenly distributing the replica identifiers in each of the first replica identifier sets to the master replica container pool and the slave replica container pool; Selecting the i-th primary replica identifier from the primary replica container pool in the order of the first to the N-th, wherein N is the number of dual replica identifiers contained in the file storage system, and i is an integer from 1 to N; Selecting an i-th slave replica identifier from the slave replica container pool according to the fault domain where the i-th master replica identifier is located; wherein the fault domain where the i-th slave replica identifier is located is different from the fault domain where the i-th master replica identifier is located; Determine the i-th master replica identifier and the i-th slave replica identifier as the i-th dual replica group; The dual copy files are stored according to each copy identifier in the i-th dual copy group.

2. The method according to claim 1, characterized in that The step of evenly distributing the replica identifiers in each replica identifier set to the master replica container pool and the slave replica container pool includes: Evenly distribute the replica identifiers in each replica identifier set to the corresponding master replica identifier queue and the corresponding slave replica identifier queue; Building a master copy container pool according to the plurality of master copy identification queues; A slave replica container pool is constructed according to the plurality of slave replica identification queues.

3. The method according to claim 1, characterized in that The selecting the i-th slave replica identifier from the slave replica container pool according to the fault domain where the i-th master replica identifier is located includes: Determine a second replica identifier set in the slave replica container pool that is different from the fault domain where the i-th master replica identifier is located; Any replica identifier selected from the second replica set is determined as the i-th slave replica identifier.

4. The method according to claim 1, wherein The method further comprises: If the slave replica container pool does not have a replica identifier that is different from the fault domain where the i-th master replica identifier is located, swapping the i-th master replica identifier with the target replica identifier to obtain a swapped i-th master replica identifier and a swapped target replica group; the target replica identifier is any replica identifier in the target replica group; Select any replica identifier in the slave replica container pool as the i-th slave replica identifier; The exchanged i-th master replica identifier and the i-th slave replica identifier are determined as the i-th dual replica group.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: If there is no replica identifier in the slave replica container pool that is different from the fault domain where the i-th master replica identifier is located, obtain the fault domain where the replica identifier in each replica group from the 1st to the i-1th replica groups is located; According to the fault domain where the replica identifier in each replica group in the 1st to i-1th replica groups is located, a target replica group is selected from the 1st to i-1th replica groups; the fault domain where the replica identifier in the target replica group is located is different from the fault domain where the i-th master replica identifier is located.

6. A dual copy file storage device, characterized in that: include: An acquisition module, configured to acquire a first replica identifier set in each fault domain in the file storage system; a balanced distribution module, configured to evenly distribute the replica identifiers in each of the first replica identifier sets to the primary replica container pool and the secondary replica container pool; A first selection module is configured to select an i-th primary replica identifier from the primary replica container pool in the order of the first to the N-th, where N is the number of dual replica identifiers included in the file storage system, and i is an integer from 1 to N; A second selection module is configured to select an i-th slave replica identifier from the slave replica container pool according to the fault domain where the i-th master replica identifier is located; wherein the fault domain where the i-th slave replica identifier is located is different from the fault domain where the i-th master replica identifier is located; a determination module, configured to determine the i-th master replica identifier and the i-th slave replica identifier as an i-th dual replica group; The storage module is configured to store the dual copy file according to each copy identifier in the i-th dual copy group.

7. The device according to claim 6, characterized in that The balanced distribution module is used to evenly distribute the replica identifiers in each replica identifier set to the corresponding master replica identifier queue and the corresponding slave replica identifier queue; build a master replica container pool based on multiple master replica identifier queues; and build a slave replica container pool based on multiple slave replica identifier queues.

8. The device according to claim 6, characterized in that The second selection module is configured to determine a set of second replica identifiers in the slave replica container pool that is different from the fault domain where the i-th master replica identifier is located; and determine any replica identifier selected from the second replica set as the i-th slave replica identifier.

9. A dual copy file storage device, characterized in that: include: Memory and processor The memory is used to store executable instructions; The processor is used to implement the dual-copy file storage method according to any one of claims 1 to 5 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that Executable instructions are stored for implementing the dual-copy file storage method according to any one of claims 1 to 5 when executed by a processor.

Images