Method, device, electronic device and storage medium for configuring a copy

Abstract

An embodiment of the present invention provides a method, device, electronic device and storage medium for configuring a replica. The method includes: configuring at least two replicas of the same partition of a table in a distributed search engine so that the at least two replicas share the same file data path, which is a path for accessing file data stored in a distributed file system.

Description

Technical Field

[0001] The present application relates to distributed search engine implementation technology, and in particular to a replica configuration method, device, electronic device and storage medium. Background Art

[0002] Currently, the database HBase stores the underlying file data based on the Hadoop Distributed File System (HDFS). For example, a distributed search engine such as Solr is used to provide full-text indexing for HBase, and the underlying distributed file system such as HDFS is used as the storage of file data (such as index data).

[0003] In a distributed search engine such as Solr, a table (collection) is configured, which is a collection of data. The table has at least one partition (shard), and each partition has at least one replica (replica). In the case of multiple replicas, the multiple replicas may include a leader replica and a slave replica. The leader replica plays the main role, and there can only be one of them. The slave replica plays a supplementary role, and there can be one or more. The slave replica will synchronize the updates of the leader replica to ensure the consistency between the replicas.

[0004] Currently, Solr has only three replication mechanisms: TLOG / NRT / PULL. In these three replication mechanisms, for each replica, a copy of the same file data is written to the file system, thereby providing redundancy of replicas and file data to ensure high reliability.

[0005] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present invention and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because these solutions are described in the background technology section of the present invention. Summary of the invention

[0006] The inventors found that the three replication mechanisms of TLOG / NRT / PULL are all abstractly implemented based on the local file system, and no appropriate replication mechanism is redefined for a distributed file system such as HDFS.

[0007] Since the redundancy of file data has been achieved through the mechanism of the distributed file system itself, in the Solr copy mechanism based on the distributed file system, if the above three copy mechanisms are still used, that is, if the same file data is still written to each copy in the distributed file system when there are multiple copies, the file data stored in the distributed file system will have redundancy increased through the mechanisms of both the distributed file system and the distributed search engine. In this way, the redundancy of file data is unnecessarily increased, resulting in higher storage costs and lower write performance.

[0008] In addition, in the above three copy mechanisms, when there are multiple copies, synchronization between multiple copies is achieved by writing copies to the file system one by one so that multiple slave copies are synchronized with the master copy. In this way, if the write operation of a copy fails, other copies that have performed write operations before the copy cannot be rolled back to the state before the write operation, thus causing inconsistencies between the copies.

[0009] In order to solve the above problems or similar problems, the embodiments of the present invention provide a method, device, electronic device and storage medium for configuring a replica, which can avoid excessive redundancy of file data, thereby reducing storage costs, improving write performance, and easily ensuring consistency between replicas.

[0010] According to a first aspect of an embodiment of the present invention, a method for configuring a replica is provided, wherein the method comprises: configuring at least two replicas of the same partition of a table in a distributed search engine so that the at least two replicas share the same file data path, wherein the file data path is a path for accessing file data stored on a distributed file system.

[0011] According to a second aspect of an embodiment of the present invention, a device for configuring a replica is provided, wherein the device includes: a path configuration unit, which configures at least two replicas of the same partition of a table in a distributed search engine so that the at least two replicas share the same file data path, which is a path for accessing file data stored on a distributed file system.

[0012] According to a third aspect of an embodiment of the present invention, an electronic device is provided, and the electronic device includes the apparatus according to the second aspect.

[0013] According to a fourth aspect of an embodiment of the present invention, a storage medium storing a processor-readable program is provided, wherein the program enables the processor to execute the following steps: configuring at least two copies of the same partition of a table in a distributed search engine so that the at least two copies share the same file data path, wherein the file data path is a path for accessing file data stored on a distributed file system.

[0014] One beneficial effect of an embodiment of the present invention is that by making at least two copies of the same partition of a table in a distributed search engine share the same file data path, it is possible to avoid excessive redundancy of file data, thereby reducing storage costs, improving write performance, and easily ensuring consistency between the copies.

[0015] With reference to the following description and accompanying drawings, specific embodiments of the present invention are disclosed in detail, indicating the manner in which the principles of the present invention can be adopted. It should be understood that the embodiments of the present invention are not limited in scope. Within the spirit and scope of the appended claims, the embodiments of the present invention include many changes, modifications and equivalents.

[0016] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0017] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the implementation of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 It is a schematic diagram of the configuration method of embodiment 1 of the present invention.

[0020] Figure 2 It is a schematic diagram of the relationship between the copy and the file data in Example 1 of the present invention.

[0021] Figure 3 It is a schematic diagram of the relationship between the copy and the file data of the comparative example of the embodiment 1 of the present invention.

[0022] Figure 4 It is another schematic diagram of the configuration method of embodiment 1 of the present invention.

[0023] Figure 5 It is a schematic diagram of a configuration device of embodiment 2 of the present invention.

[0024] Figure 6It is another schematic diagram of the configuration device of embodiment 2 of the present invention.

[0025] Figure 7 It is a schematic diagram of an electronic device according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0026] With reference to the accompanying drawings, the above and other features of the present invention will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present invention are specifically disclosed, which show some embodiments in which the principles of the present invention can be adopted, and it should be understood that the present invention is not limited to the described embodiments, on the contrary, the present invention includes all modifications, variations and equivalents falling within the scope of the appended claims.

[0027] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0028] In the embodiments of the present invention, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or time sequence of these elements, etc. These elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms.

[0029] In the embodiments of the present invention, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.

[0030] It should also be noted that, in some alternative implementations, the functions / actions mentioned may occur in a different order than that indicated in the accompanying drawings. For example, depending on the functions / actions involved, two figures shown in succession may actually be performed substantially simultaneously or may sometimes be performed in the reverse order.

[0031] Various implementations of the embodiments of the present invention are described below in conjunction with the accompanying drawings. These implementations are only exemplary and are not intended to limit the embodiments of the present invention.

[0032] Example 1

[0033] This embodiment 1 provides a method for configuring a replica.

[0034] Figure 1 FIG. 1 is a schematic diagram of the configuration method of this embodiment. Figure 1 As shown, the method includes:

[0035] Step 101: configure at least two copies of the same partition of a table in a distributed search engine so that the at least two copies share the same file data path, where the file data path is a path for accessing file data stored on a distributed file system.

[0036] Through the above embodiment, by making at least two copies of the same partition of the table in the distributed search engine share the same file data path, the file data can be written only once in the distributed file system for the at least two copies, without having to repeatedly write the file data in the distributed file system for each copy. In this way, it is possible to avoid excessive redundancy of file data, reduce the amount of file data stored in the distributed file system, and also reduce the number of times file data is written to the distributed file system, thereby reducing storage costs and improving write performance; and, since each copy accesses the same file data according to the same file data path, there is no need to synchronize between the copies, so it is easy to ensure consistency between the copies.

[0037] In this embodiment, the distributed search engine may be, for example, Solr, but this embodiment is not limited thereto and may also be other distributed search engines.

[0038] In the distributed search engine, a table (collection) is configured, which is a collection of data. The table has at least one partition (shard), and each partition has at least one replica (replica). In the case of multiple replicas, the multiple replicas may include a leader replica and a slave replica. The leader replica plays a major role, and there can only be one of them. The slave replica plays a supplementary role, and there may be one or more of them.

[0039] The distributed file system may be HDFS, but the present embodiment is not limited thereto and may also be other distributed file systems.

[0040] File data can be the entire file or a file block formed by splitting a file. File data can be one or more files or one or more file blocks. In a distributed file system, file data can include at least two copies of the same data, thereby achieving redundancy of file data and improving the reliability of the distributed file system.

[0041] The file data path is a path used to access file data stored on a distributed file system. In other words, the file data path provides a "link" used by each replica to access file data in a distributed file system. Each replica obtains the location or identifier of the file data stored in the distributed file system based on the file data path, thereby being able to access the file data.

[0042] In this embodiment, at least two copies share the same file data path, for example, at least two copies respectively record the same data directory, and the same data directory corresponds to a certain file data path. Alternatively, at least two copies respectively record the same identifier, and the same identifier corresponds to a certain file data path. This embodiment is not limited to the above manner.

[0043] Figure 2 FIG. 1 is a schematic diagram of the relationship between the replica and the file data of this embodiment. Figure 2 In , file data is stored in the form of multiple data blocks. Figure 2 As shown, A, B, C, ... are, for example, multiple distributed search engine nodes, a, b, c, ... are, for example, multiple data nodes (datanodes) in a distributed file system, 21 are multiple locations for recording copies in the distributed search engine server, and 22 are multiple locations for storing data blocks in the distributed file system.

[0044] Assume that there is a partition shard1 in the table of a distributed search engine, and that shard1 has two copies, namely copy 1 and copy 1'. Distributed file systems a, b, c, ... perform redundant backups on each file data stored thereon, for example, data block 1 and data block 2 back up each other, and data blocks 1' and 2' back up each other. Data blocks 1 and data blocks 2 constitute one file data, and data blocks 1' and data blocks 2' constitute another identical file data, such as Figure 2 As shown, replica 1 and replica 1' share the paths of data block 1, data block 2, data block 1', and data block 2'. In this way, replica 1 and replica 1' can access the same data block 1, data block 2, data block 1', and data block 2' according to the same path.

[0045] Figure 3 FIG. 1 is a schematic diagram of the relationship between the replica and the file data of the comparative example of this embodiment. In this comparative example, the existing replica mechanism of "writing a copy of the same file data to each replica in the distributed file system when there are multiple replicas" is adopted. Figure 3 In, with Figure 2The difference is that for replica 1, data block 1 and data block 2 of replica 1 constitute one file data, and data block 1' and data block 2' of replica 1 constitute another identical file data; for replica 1', data block 1 and data block 2 of replica 1' constitute one file data, and data block 1' and data block 2' of replica 1' constitute another identical file data. Figure 3 As shown, replica 1 records the paths of data block 1, data block 2, data block 1', and data block 2' of replica 1, and replica 1' records the paths of data block 1, data block 2, data block 1', and data block 2' of replica 1'. In this way, replica 1 and replica 1' can both access data block 1, data block 2, data block 1', and data block 2' according to the paths they record respectively.

[0046] It can be seen that in the comparative example, when there are two copies for the same partition shard1, 8 redundant data blocks are stored in the distributed file system; in contrast, in the above embodiment of the present application, under the same number of copies, only 4 redundant data blocks are stored in the distributed file system. Through the embodiment of the present application, while ensuring the reliability of file data through the redundancy mechanism of the distributed file system itself, the number of file data stored in the distributed file system can be greatly reduced, and the number of times file data is written to the distributed file system can also be reduced, thereby reducing storage costs and improving write performance; and, since each copy accesses the same file data according to the same file data path, there is no need to synchronize between the copies, so the consistency between the copies can be easily guaranteed.

[0047] In this embodiment, each replica can be configured to perform various operations such as read operations, write operations, master-replica switching operations, merge operations, split operations, add operations, delete operations, and move operations.

[0048] In this embodiment, for a read operation, at least two replicas may be configured to access file data according to a shared file data path.

[0049] In this embodiment, for write operations, the master replica can be configured to perform write operations on file data, and the slave replica can be configured not to perform write operations on file data. In other words, all replicas can be configured to perform only one write operation on file data (i.e., the write operation of the master replica). However, this embodiment is not limited to this, and each replica can also be configured to be able to perform write operations on file data. When receiving a request for a write operation, each replica contends for the write lock, and the replica that successfully contends performs the write operation.

[0050] In this embodiment, for the master copy switching operation when the master copy is abnormal, any slave copy can be switched to the master copy. Here, the master copy abnormality refers to the situation that the master copy cannot work normally, such as the node where the master copy is located loses power or fails.

[0051] The master copy switching operation requires that the file data of the slave copy that is switched to the master copy is the latest version. If the file data of a slave copy is not the latest version, the switching operation of switching the slave copy to the master copy will fail. In this embodiment, since the file data corresponding to the slave copy and the master copy are the same, the file data must be the latest file data. Therefore, in this embodiment, the switching operation of switching any slave copy to the master copy will not fail. In the comparative example corresponding to the prior art, since the file data corresponding to each slave copy may not be synchronized, some slave copies may not be able to switch to the master copy (i.e., the switching fails). If the file data corresponding to all the slave copies are not synchronized with the latest file data when the switching occurs, the master copy switching operation cannot be performed.

[0052] In this embodiment, for the merge operation of merging at least two partitions, all copies of the at least two partitions can be closed, and at least two new copies of the merged partition can be started. The at least two new copies share the same file data path, which is used to access all file data corresponding to the original at least two partitions.

[0053] In this embodiment, for a split operation in which a partition is split into at least a first partition and a second partition, all copies of the partition before the split can be closed, the file data can be split into first file data and second file data, and, for the first partition, at least two new first copies are started, and the at least two new first copies share the file data path of the first file data, and for the second partition, at least two new second copies are started, and the at least two new second copies share the file data path of the second file data.

[0054] In this embodiment, for the adding operation of adding a replica, corresponding metadata information may be added for the newly added replica at the corresponding node, and the newly added replica may be configured to share the same file data path with other replicas.

[0055] In this embodiment, for the deletion operation of deleting a copy, the information related to the copy can be directly deleted. In addition, if the deleted copy is a master copy, one of the slave copies can be selected as a new master copy. The specific selection method can refer to the prior art.

[0056] In this embodiment, for a move operation of moving a replica between nodes, the replica may be deleted at the original node where the replica was located before the move, and the replica may be added at the new node after the move.

[0057] This embodiment is not limited to the above operations. Based on the same principle and idea, other operations of the replica may also be adjusted accordingly based on the existing technology.

[0058] In this embodiment, multiple replica modes (i.e., replica configuration modes) can also be compatible at the same time, such as a regular replica mode and a shadow replica mode, wherein the regular replica mode can be a currently existing replica mode, such as TLOG, NRT or PULL mode, and the shadow replica mode is the mode described in step 101 above.

[0059] Figure 4 FIG. 4 is another schematic diagram of the configuration method of this embodiment. Figure 4 As shown, the method includes:

[0060] Step 401, determine whether the copy mode is the shadow copy mode; if the determination result is yes, execute step 403, otherwise execute step 405;

[0061] Step 403: configure the at least two replicas so that the at least two replicas share the same file data path, where the file data path is a path for accessing file data stored on the distributed file system;

[0062] Step 405, configure the replica according to the normal replica mode.

[0063] Step 403 may be the same as the above-mentioned step 101 .

[0064] In addition, each replica can be configured to perform the above-mentioned read operations, write operations, master-replica switching operations, merge operations, split operations, add operations, delete operations, move operations, etc. only when the judgment result of step 401 is yes. If the judgment result of step 401 is no, these operations are performed according to the normal replica mode.

[0065] Through the replica configuration method of this embodiment, at least two replicas of the same partition of a table in a distributed search engine share the same file data path, which can avoid excessive redundancy of file data, thereby reducing storage costs, improving write performance, and easily ensuring consistency between replicas.

[0066] Example 2

[0067] This embodiment 2 provides a device for configuring a replica. The same contents as those in the embodiment 1 are not described in detail, and the following mainly describes the differences.

[0068] Figure 5 FIG. 1 is a schematic diagram of the configuration device of this embodiment. Figure 5 As shown, the configuration device 500 includes a path configuration unit 501, which configures at least two copies of the same partition of a table in a distributed search engine so that the at least two copies share the same file data path, which is a path for accessing file data stored on a distributed file system.

[0069] In this embodiment, the distributed search engine, table, partition, and replica are the same as those in Embodiment 1 and will not be described in detail here.

[0070] In this embodiment, the distributed file system, file data, and file data path are the same as those in Embodiment 1 and are not described in detail here.

[0071] In this embodiment, the manner in which at least two copies share the same file data path is the same as in Embodiment 1, and will not be described in detail here.

[0072] Figure 6 FIG. 1 is another schematic diagram of the configuration device of this embodiment. Figure 6 As shown, the configuration device 600 includes a path configuration unit 601 and an access configuration unit 602, wherein the path configuration unit 601 is the same as the above-mentioned path configuration unit 501, and the access configuration unit 602 configures at least two copies so that the at least two copies access the file data according to the shared file data path. The access configuration unit 602 is an optional component.

[0073] In this embodiment, if Figure 6 As shown, the configuration device 600 may further include a write configuration unit 603, which configures at least two copies so that the master copy performs a write operation on the file data and the slave copy does not perform a write operation on the file data. The write configuration unit 603 is an optional component.

[0074] In this embodiment, if Figure 6 As shown, the configuration device 600 may further include a switching configuration unit 604. When an abnormality occurs in the master copy, the switching configuration unit 604 configures at least one slave copy so that one of the at least one slave copies switches to the master copy. The switching configuration unit 604 is an optional component.

[0075] In this embodiment, if Figure 6As shown, the configuration device 600 may further include a merge configuration unit 605. When at least two partitions are merged, the merge configuration unit 605 closes all copies of the at least two partitions and starts at least two new copies for the merged partition. The at least two new copies share the same file data path, and the file data path is used to access all file data corresponding to the original at least two partitions. The merge configuration unit 605 is an optional component.

[0076] In this embodiment, if Figure 6 As shown, the configuration device 600 may further include a split configuration unit 606. When a partition is split into at least a first partition and a second partition, the split configuration unit 606 closes all copies of the partition, splits the file data into first file data and second file data, and, for the first partition, starts at least two new copies, the at least two new copies share the file data path of the first file data, and for the second partition, starts at least two new copies, the at least two new copies share the file data path of the second file data. The split configuration unit 606 is an optional component.

[0077] In this embodiment, if Figure 6 As shown, the configuration device 600 may further include an adding configuration unit 607, which adds corresponding metadata information for the newly added replica at the corresponding node and configures the newly added replica to share the same file data path with other replicas. The adding configuration unit 607 is an optional component.

[0078] In this embodiment, if Figure 6 As shown, the configuration device 600 may further include a deletion configuration unit 608, which directly deletes the information related to the replica. In addition, if the deleted replica is a master replica, the configuration device 600 may further include a selection unit 609, which selects one from each slave replica as a new master replica, and the switching configuration unit 604 switches the selected slave replica to the new master replica. The deletion configuration unit 608 and the selection unit 609 are optional components.

[0079] In this embodiment, if Figure 6 As shown, the configuration device 600 may further include a mobile configuration unit 610, which may delete the replica at the original node where the replica is located before the move, and add the replica at the new node after the move. The mobile configuration unit 610 is an optional component.

[0080] In this embodiment, if Figure 6As shown, the configuration device 600 may also include a judgment unit 611, which judges whether the copy mode is a shadow copy mode, and the path configuration unit 601 configures at least two copies when the copy mode is a shadow copy mode so that the at least two copies share the same file data path.

[0081] The above-mentioned access configuration unit 602, write configuration unit 603, switch configuration unit 604, merge configuration unit 605, split configuration unit 606, add configuration unit 607, delete configuration unit 608, selection unit 609, and move configuration unit can also perform the above-mentioned functions when the judgment result of the judgment unit 611 is yes.

[0082] The determination unit 611 is an optional component.

[0083] Through the replica configuration device of this embodiment, at least two replicas of the same partition of a table in a distributed search engine share the same file data path, which can avoid excessive redundancy of file data, thereby reducing storage costs, improving write performance, and easily ensuring consistency between replicas.

[0084] Example 3

[0085] This embodiment 3 provides an electronic device. The same contents as those of the embodiment 1 or 2 are not described in detail, and the following mainly describes the differences.

[0086] Figure 7 FIG. 1 is a schematic diagram of an electronic device of this embodiment. Figure 7 As shown, the electronic device 700 may include: a processor 701 and a memory 702 , where the memory 702 is coupled to the processor 701 .

[0087] Among them, the memory 702 can store programs for implementing certain functions, such as storing programs for implementing the configuration method of Example 1, and the program is executed under the control of the processor 701; in addition, the memory 702 can also store various data, such as tables and their partitions, replica-related data, etc.

[0088] In one embodiment, the functions in the configuration device may be integrated into the processor 701 for execution.

[0089] In this embodiment, the processor 701 may be configured to:

[0090] At least two replicas of the same partition of a table in a distributed search engine are configured so that the at least two replicas share a same file data path, where the file data path is a path for accessing file data stored on a distributed file system.

[0091] In this embodiment, the processor 701 may also be configured to:

[0092] Determine whether the copy mode is shadow copy mode,

[0093] Furthermore, when the copy mode is a shadow copy mode, the at least two copies are configured so that the at least two copies share the same file data path.

[0094] In this embodiment, the processor 701 may also be configured to:

[0095] In the event that an abnormality occurs to the master replica, the at least one slave replica is configured so that one of the at least one slave replica is switched to be the master replica.

[0096] In this embodiment, the processor 701 may also be configured to:

[0097] When at least two partitions are merged, all copies of the at least two partitions are closed, and at least two new copies of the merged partition are started, and the at least two new copies share the same file data path, and the file data path is used to access all file data corresponding to the original at least two partitions.

[0098] In this embodiment, the processor 701 may also be configured to:

[0099] In the case where a partition is split into at least a first partition and a second partition, all copies of the partition before the split are closed, the file data is split into first file data and second file data, and, for the first partition, at least two new first copies are started, and the at least two new first copies share the file data path of the first file data, and for the second partition, at least two new second copies are started, and the at least two new second copies share the file data path of the second file data.

[0100] In this embodiment, the processor 701 may also be configured to execute other method steps described in Embodiment 1, which will not be described in detail here.

[0101] like Figure 7 As shown, the electronic device 700 may further include a communication unit 703, which may send or receive information via a bus or a network, for example, may send or receive various control commands, request messages, and the like.

[0102] It is worth noting that the electronic device 700 does not necessarily have to include Figure 7 All components shown in the figure may be omitted as needed, for example, the communication unit 703 may be omitted; in addition, the electronic device 700 may also include Figure 7 For components not shown, reference may be made to the prior art.

[0103] In this embodiment, the term "electronic device" includes user devices and network devices. The network devices include but are not limited to a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of computers or network servers based on cloud computing (Cloud Computing), wherein cloud computing is a type of distributed computing, a super virtual computer consisting of a group of loosely coupled computer sets. The computer device can be run alone to implement the present application, or it can be connected to the network and implement the present application through interactive operations with other computer devices in the network. The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, VPN network, etc.

[0104] It should be noted that the user equipment, network equipment and network are only examples. Other existing or future computer equipment or networks that are applicable to the present application should also be included in the scope of protection of the present application and included here by reference.

[0105] As an example, the electronic device 700 may be a node device implementing a distributed search engine, such as a Solr server.

[0106] Through the electronic device of this embodiment, at least two copies of the same partition of a table in a distributed search engine can share the same file data path, which can avoid excessive redundancy of file data, thereby reducing storage costs, improving write performance, and easily ensuring consistency between copies.

[0107] The embodiment of the present invention further provides a processor-readable program, which enables the processor to execute the method described in Embodiment 1.

[0108] An embodiment of the present invention further provides a storage medium storing a processor-readable program, wherein the program enables the processor to execute the method described in Embodiment 1.

[0109] The above method / device of the present invention can be implemented by hardware, or by hardware combined with software. The present invention relates to such a computer-readable program, which, when executed by a logic component, enables the logic component to implement the above-mentioned device or component, or enables the logic component to implement the various methods or steps described above. The logic component is, for example, a field programmable logic component, a microprocessor, a processor used in a computer, etc. The present invention also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0110] The method / device described in conjunction with the embodiments of the present invention may be directly embodied as hardware, a software unit executed by a processor, or a combination of the two. Figure 5 One or more of the functional block diagrams shown in and / or one or more combinations of the functional block diagrams may correspond to various software units of the computer program flow or to various hardware units. These software units may correspond to Figure 1 These hardware units can be implemented by solidifying these software units using, for example, a field programmable gate array (FPGA).

[0111] The software unit may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software unit may be stored in the memory of the device or in a memory card that can be inserted into the device. For example, if the device uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software unit may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0112] For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can be implemented as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any appropriate combination thereof for performing the functions described in the present invention. For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0113] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles of the present application, and these modifications and variations are also within the scope of the present application.

Claims

1. A method for configuring a replica, wherein: The method comprises: At least two replicas of the same partition of a table in a distributed search engine are configured so that the at least two replicas share the same file data path, which is a path for accessing file data stored on a distributed file system; the at least two replicas include a master replica and at least one slave replica; wherein the at least two replicas share the same file data path, including: the at least two replicas respectively record the same data directory or identifier, and the same data directory or identifier has a corresponding file data path; In the case where a partition is split into at least a first partition and a second partition, all copies of the partition before the split are closed, the file data is split into first file data and second file data, and, for the first partition, at least two new first copies are started, and the at least two new first copies share the file data path of the first file data; for the second partition, at least two new second copies are started, and the at least two new second copies share the file data path of the second file data.

2. The method according to claim 1, wherein: The file data includes at least two identical data.

3. The method according to claim 1, wherein: The method further comprises: Determine whether the copy mode is shadow copy mode, Furthermore, when the copy mode is a shadow copy mode, the at least two copies are configured so that the at least two copies share the same file data path.

4. The method according to claim 1, wherein: The method further comprises: The at least two copies are configured so that the at least two copies access the file data according to the shared file data path.

5. The method according to claim 1, wherein: The method further comprises: The at least two replicas are configured so that the master replica performs a write operation on the file data and the slave replica does not perform a write operation on the file data.

6. The method according to claim 1, wherein: The method further comprises: In the event that an abnormality occurs to the master replica, the at least one slave replica is configured so that one of the at least one slave replica is switched to be the master replica.

7. The method according to claim 6, wherein: The method further comprises: When at least two partitions are merged, all copies of the at least two partitions are closed, and at least two new copies are started for the merged partition. The at least two new copies share the same file data path, and the file data path is used to access all file data corresponding to the original at least two partitions.

8. A replica configuration device, wherein: The device comprises: A path configuration unit, which configures at least two replicas of the same partition of a table in a distributed search engine so that the at least two replicas share the same file data path, where the file data path is a path for accessing file data stored on a distributed file system; the at least two replicas include a master replica and at least one slave replica; wherein the at least two replicas share the same file data path, including: the at least two replicas respectively record the same data directory or identifier, and the same data directory or identifier has a corresponding file data path; The device also includes: A split configuration unit, which, when a partition is split into at least a first partition and a second partition, closes all copies of the partition before the split, splits the file data into first file data and second file data, and, for the first partition, starts at least two new first copies, the at least two new first copies share the file data path of the first file data, and for the second partition, starts at least two new second copies, the at least two new second copies share the file data path of the second file data.

9. The device according to claim 8, wherein: The file data includes at least two identical data.

10. The device according to claim 8, wherein: The device also includes: a judging unit, which judges whether the copy mode is a shadow copy mode, Furthermore, when the copy mode is a shadow copy mode, the path configuration unit configures the at least two copies so that the at least two copies share the same file data path.

11. The device according to claim 8, wherein: The device also includes: An access configuration unit configures the at least two copies so that the at least two copies access the file data according to the shared file data path.

12. The device according to claim 8, wherein: The device also includes: A write configuration unit is configured to configure the at least two replicas so that the primary replica performs a write operation on the file data and the secondary replica does not perform a write operation on the file data.

13. The device according to claim 8, wherein: The device also includes: A switching configuration unit is configured to configure the at least one slave copy so that one of the at least one slave copies is switched to the master copy when an abnormality occurs in the master copy.

14. The device according to claim 8, wherein: The device also includes: A merge configuration unit, which, when at least two partitions are merged, closes all copies of the at least two partitions and starts at least two new copies for the merged partition, wherein the at least two new copies share the same file data path, and the file data path is used to access all file data corresponding to the original at least two partitions.

15. An electronic device, wherein: The electronic device comprises the apparatus according to any one of claims 8-14.

16. A storage medium storing a processor-readable program, wherein the program causes the processor to execute the following steps: At least two replicas of the same partition of a table in a distributed search engine are configured so that the at least two replicas share the same file data path, where the file data path is a path for accessing file data stored on a distributed file system; the at least two replicas include a master replica and at least one slave replica; wherein, The at least two copies share the same file data path, including: the at least two copies respectively record the same data directory or identifier, and the same data directory or identifier has a corresponding file data path; In the case where a partition is split into at least a first partition and a second partition, all copies of the partition before the split are closed, the file data is split into first file data and second file data, and, for the first partition, at least two new first copies are started, and the at least two new first copies share the file data path of the first file data; for the second partition, at least two new second copies are started, and the at least two new second copies share the file data path of the second file data.

Images