Index fragment distribution method and device based on Elasticsearch

Abstract

The invention discloses an index fragmentation distribution method based on Elasticsearch. The index fragmentation distribution method comprises the following steps of: obtaining an index fragment; the method comprises the steps that configuration information of a user for distribution parameters of index fragments is received, and the distribution parameters of the index fragments comprise serial numbers of servers in a cluster and serial numbers of Leaf nodes corresponding to the servers; and distributing the index fragments of the target index according to the Leaf node numbers corresponding to the servers in the cluster, and distributing the index fragments of the target index and the backup fragments corresponding to the index fragments to the servers with different Leaf node numbers. The index fragment of the target index and the backup fragment corresponding to the index fragment are distributed in different Leaf nodes, the servers where the index fragment and the backup fragment are located are not in unified power supply and network supply, meanwhile, the fault probability is low, and when the server where the index fragment is located breaks down, the failure rate of the target index is increased. The server where the backup fragment is located can still work normally, and data recovery can be completed from the corresponding backup fragment, so that the availability of the data is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an index shard allocation method, device, electronic device and computer-readable storage medium based on Elasticsearch. Background Art

[0002] In the current era of big data, there are many data warehouses for storing massive data, and the open source distributed search engine Elasticsearch (search engine, ES) is one of them. Elasticsearch can not only store and manage large-scale data, but also realize near real-time full-text retrieval of its stored data.

[0003] In Elasticsearch, one server corresponds to one data node. Each data node runs a program. The programs communicate with each other. The communicating programs form a cluster and record data and data locations through indexes. To achieve distributed storage, the existing index shard allocation method divides a complete index into multiple shards and distributes them to different servers in the cluster, and ensures that the index shard and its corresponding backup shard are not allocated to the same server.

[0004] However, when multiple servers belonging to the same Leaf node fail in the cluster, the existing index shard allocation method will cause the index shard and its corresponding backup shard to be unavailable, and the data availability cannot be guaranteed. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide an index shard allocation method, device, electronic device and computer-readable storage medium based on Elasticsearch that overcome the above problems or at least partially solve the above problems.

[0006] The present application embodiment adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides an index shard allocation method based on Elasticsearch, which is applied to a server and includes:

[0008] Receive configuration information of allocation parameters of the index shards from the user, wherein the allocation parameters of the index shards include the numbers of the servers in the cluster and the numbers of the corresponding leaf nodes;

[0009] The index shards of the target index are allocated according to the Leaf node numbers corresponding to the servers in the cluster, and the index shards of the target index and their corresponding backup shards are allocated to servers with different Leaf node numbers.

[0010] Optionally, the servers in the cluster belong to different logical subclusters;

[0011] The allocation parameters of the index shards also include: logical sub-cluster numbers corresponding to each server in the cluster;

[0012] The method further comprises:

[0013] The index shards of the target index are allocated according to the logical sub-cluster numbers corresponding to the servers in the cluster, and the index shards of different target indexes are allocated to servers belonging to different logical sub-clusters.

[0014] Optionally, the method further includes: determining whether the target index is important data;

[0015] When the target index is important data, the index slices of the target index are allocated according to the logical sub-cluster numbers corresponding to the servers in the cluster, and index slices of different target indexes are allocated to servers belonging to different logical sub-clusters.

[0016] Optionally, the important data includes: log data and core business data.

[0017] Optionally, allocating index shards of different target indexes to servers belonging to different logical subclusters includes:

[0018] The index shards of the first target index and the second target index are allocated according to the logical subcluster numbers corresponding to the servers in the cluster, and the index shards of the first target index are allocated to the servers in the first logical subcluster, and the index shards of the second target index are allocated to the servers in the second logical subcluster.

[0019] Optionally, the first logical sub-cluster and the second logical sub-cluster correspond to different Leaf nodes respectively.

[0020] Optionally, the method further comprises:

[0021] Evaluating the performance of each server in the cluster;

[0022] When allocating index shards for the target index, servers with high performance are given priority.

[0023] In a second aspect, an embodiment of the present application further provides an index shard allocation device based on Elasticsearch, which is applied to a server and includes:

[0024] A configuration information receiving unit, configured to receive configuration information of a user for allocation parameters of an index shard, wherein the allocation parameters of the index shard include numbers of each server in the cluster and numbers of its corresponding leaf nodes;

[0025] The index shard allocation unit is used to allocate the index shards of the target index according to the Leaf node numbers corresponding to the servers in the cluster, and allocate the index shards of the target index and their corresponding backup shards to servers with different Leaf node numbers.

[0026] Optionally, the servers in the cluster belong to different logical subclusters;

[0027] The allocation parameters of the index shards also include: logical sub-cluster numbers corresponding to each server in the cluster;

[0028] The index fragment allocation unit is also used for:

[0029] The index shards of the target index are allocated according to the logical sub-cluster numbers corresponding to the servers in the cluster, and the index shards of different target indexes are allocated to servers belonging to different logical sub-clusters.

[0030] Optionally, the device further comprises: a judging unit, configured to judge whether the target index is important data;

[0031] The index shard allocation unit is further used to allocate index shards of the target index according to the logical subcluster numbers corresponding to the servers in the cluster when the target index is important data, and allocate index shards of different target indexes to servers belonging to different logical subclusters.

[0032] Optionally, the important data includes: log data and core business data.

[0033] Optionally, allocating index shards of different target indexes to servers belonging to different logical subclusters includes:

[0034] The index shards of the first target index and the second target index are allocated according to the logical subcluster numbers corresponding to the servers in the cluster, and the index shards of the first target index are allocated to the servers in the first logical subcluster, and the index shards of the second target index are allocated to the servers in the second logical subcluster.

[0035] Optionally, the first logical sub-cluster and the second logical sub-cluster correspond to different Leaf nodes respectively.

[0036] Optionally, the device further comprises:

[0037] An evaluation unit, used to evaluate the performance of each server in the cluster;

[0038] The index shard allocation unit is further used to: in the process of allocating index shards of the target index, give priority to selecting servers with high performance.

[0039] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0040] Processor; and

[0041] A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the above-mentioned Elasticsearch-based index shard allocation method.

[0042] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the above-mentioned Elasticsearch-based index shard allocation method.

[0043] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0044] In the process of allocating index shards, the embodiment of the present application adds the Leaf node number as an allocation parameter to ensure that the index shard of the target index and its corresponding backup shard are allocated to servers with different Leaf node numbers. In this way, the index shard of the target index and its corresponding backup shard are distributed on different Leaf nodes, and the servers where the index shard and its backup shard are located are not uniformly powered and networked, and the probability of failure is low. When the server where the index shard is located fails, the server where its backup shard is located can still work normally, and data recovery can be completed from its corresponding backup shard, thereby improving data availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 A schematic diagram of the principle of the existing index shard allocation method;

[0047] Figure 2 A flowchart of an index shard allocation method based on Elasticsearch according to an embodiment of the present application;

[0048] Figure 3This is a schematic diagram of the principle of an index shard allocation method based on Elasticsearch according to an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the structure of an index shard allocation device based on Elasticsearch according to an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0052] Figure 1 The following is a schematic diagram of the principle of the existing index shard allocation method. Figure 1 As shown, servers 1 to 15 form a cluster. To achieve distributed storage of index A, the existing index shard allocation method divides one index A into five index shards A. 1 , A 2 , A 3 , A 4 , A 5 To improve data availability, each index shard has a corresponding backup shard A. 1 '、A 2 '、A 3 '、A 4 '、A 5 ', when an index shard is unavailable, it can be restored from the corresponding backup shard. Index shard A 1 , A 2 , A 3 , A 4 , A 5 Distributed to server 1, server 2, server 3, server 4, and server 5 respectively, backup shard A 1 '、A 2 '、A 3 '、A 4 '、A 5 ' are distributed to server 5, server 3, server 4, server 2, and server 1 respectively. Similarly, to achieve distributed storage of index B, the existing index shard allocation method divides an index B into 5 index shards B 1 , B 2, B 3 , B 4 , B 5 To improve data availability, each index shard has a corresponding backup shard B. 1 ', B 2 ', B 3 ', B 4 ', B 5 ', when an index shard is unavailable, it can be restored from the corresponding backup shard. Index shard B 1 , B 2 , B 3 , B 4 , B 5 Distributed to server 1, server 2, server 3, server 4, and server 5 respectively, backup shard B 1 ', B 2 ', B 3 ', B 4 ', B 5 'Distributed to server 5, server 3, server 4, server 2, and server 1 respectively.

[0053] It is understandable that the existing index shard allocation method simply randomly distributes the index shards of index A and index B to different servers in the cluster, and ensures that the index shard and its corresponding backup shard are not allocated to the same server.

[0054] A group of servers in a cluster that share power and network supply constitute a Leaf node. Since Leaf node parameters are not considered in the existing index shard allocation process, it is very likely that an index shard and its corresponding backup shard will be allocated to servers belonging to the same Leaf node.

[0055] If servers 1 to 5 belong to the same Leaf node, when one of the servers fails, the other four servers will also fail, causing the index shard and its corresponding backup shard to be unavailable, and data availability cannot be guaranteed.

[0056] Figure 2 1 is a flow chart of an index shard allocation method based on Elasticsearch according to an embodiment of the present application. The index shard allocation method based on Elasticsearch according to an embodiment of the present application is applied to a server, such as Figure 2 As shown, the method includes:

[0057] S210: receiving configuration information of allocation parameters of the index shards from the user, wherein the allocation parameters of the index shards include the numbers of the servers in the cluster and the numbers of the corresponding leaf nodes;

[0058] It is understandable that a group of servers in a cluster that share power and network supply constitute a Leaf node. Leaf-id can be used to represent the Leaf node number, which is used to identify different Leaf nodes.

[0059] In actual applications, users can input configuration information to each server in the cluster through software to complete parameter configuration.

[0060] S220: Allocate index shards of the target index according to the Leaf node numbers corresponding to the servers in the cluster, and allocate the index shards of the target index and their corresponding backup shards to servers with different Leaf node numbers.

[0061] It is understandable that the prior art does not consider Leaf node parameters in the allocation process of index shards, and it is very likely that index shards and their corresponding backup shards will be allocated to servers belonging to the same Leaf node. Since the servers in the Leaf node are powered and supplied by the same network, it is easy for multiple servers in the same Leaf node to fail, resulting in data unavailability.

[0062] The index shard allocation method based on Elasticsearch in the embodiment of the present application adds the Leaf node number as an allocation parameter in the index shard allocation process, ensuring that the index shard of the target index itself and its corresponding backup shard are allocated to servers with different Leaf node numbers. In this way, the index shard of the target index itself and its corresponding backup shard are distributed on different Leaf nodes, and the servers where the index shard itself and its backup shard are located are not uniformly powered and networked, and the probability of failure is low. When the server where the index shard is located fails, the server where its backup shard is located can still work normally, and data recovery can be completed from its corresponding backup shard, thereby improving data availability.

[0063] In an optional implementation of the embodiment of the present application, the servers in the cluster belong to different logical subclusters;

[0064] The allocation parameters of the index shards also include: logical sub-cluster numbers corresponding to each server in the cluster;

[0065] The method further comprises:

[0066] The index shards of the target index are allocated according to the logical sub-cluster numbers corresponding to the servers in the cluster, and the index shards of different target indexes are allocated to servers belonging to different logical sub-clusters.

[0067] It should be noted that a logical subcluster refers to a logical group consisting of several servers in a cluster. Each logical subcluster is logically isolated from each other and will not interfere with each other.

[0068] In practical applications, logical_cluster can be used to represent the logical subcluster number to identify different logical subclusters.

[0069] The index shard allocation method based on Elasticsearch in the embodiment of the present application not only adds the Leaf node number as an allocation parameter in the index shard allocation process, but also adds the logical subcluster number as an allocation parameter, and allocates index shards of different target indexes to servers belonging to different logical subclusters. When a logical subcluster fails and the data distributed in the logical subcluster is unavailable, other logical subclusters are still working normally, and the data distributed in other logical subclusters is still available, which further improves the data availability.

[0070] Furthermore, to improve the allocation efficiency of index shards, the method further includes: determining whether the target index is important data;

[0071] When the target index is important data, the index slices of the target index are allocated according to the logical sub-cluster numbers corresponding to the servers in the cluster, and index slices of different target indexes are allocated to servers belonging to different logical sub-clusters.

[0072] In practical applications, the important data includes: log data and core business data.

[0073] It is understandable that in order to take into account both data availability and index shard allocation efficiency, the embodiment of the present application only allocates index shards based on logical sub-cluster isolation for important data such as log data and core business data, and only allocates index shards based on Leaf node isolation for general data.

[0074] In practical applications, the importance identification field can be used to distinguish important data from general data, for example, 1 is used to represent important data and 0 is used to represent general data.

[0075] As an optional implementation of the embodiment of the present application, the allocating index shards of different target indexes to servers belonging to different logical subclusters includes:

[0076] The index shards of the first target index and the second target index are allocated according to the logical subcluster numbers corresponding to the servers in the cluster, and the index shards of the first target index are allocated to the servers in the first logical subcluster, and the index shards of the second target index are allocated to the servers in the second logical subcluster.

[0077] For example, the first target index is A, and its index fragments include A 1 '、A 2 '、A 3 '、A 4 '、A 5 '; The second target index is B, and its index fragments include B 1 ', B 2 ', B 3 ', B 4 ', B 5 '. To further improve data availability, the index shard A of the first target index A 1 '、A 2 '、A 3 '、A 4 '、A 5 'Shard to the server of the first logical sub-cluster, and split the index shard B of the second target index B 1 ', B 2 ', B 3 ', B 4 ', B 5 'Assigned to the server of the second logical subcluster. When the first logical subcluster fails and index A distributed in the logical subcluster becomes unavailable, but the second logical subcluster is still working normally, index B distributed in the second logical subcluster is still available.

[0078] Furthermore, the first logical sub-cluster and the second logical sub-cluster correspond to different Leaf nodes respectively.

[0079] It should be noted that if the first logical sub-cluster and the second logical sub-cluster both correspond to the same Leaf node, although the data unavailability problem caused by cluster logic errors can be overcome, the data unavailability problem caused by Leaf node failures still exists. Therefore, the first logical sub-cluster and the second logical sub-cluster of the embodiment of the present application correspond to different Leaf nodes respectively, which can simultaneously overcome the data unavailability problem caused by cluster logic errors and Leaf node failures.

[0080] Furthermore, the method further comprises:

[0081] Evaluating the performance of each server in the cluster;

[0082] When allocating index shards for the target index, servers with high performance are given priority.

[0083] It is understandable that if the allocation process of index shards is based only on the principle of decentralized placement of shards without considering the performance of cluster servers, when there is a large concurrent access to hot data in the Elasticsearch cluster, some servers may be overloaded, thereby affecting the retrieval performance. In the allocation process of index shards, the embodiment of the present application adds the Leaf node number and the logical subcluster number as allocation parameters, while evaluating the performance of each server in the cluster, and preferentially selects servers with high performance for the distribution of each index shard, thereby improving the performance of index shards and cluster load balancing while ensuring data availability.

[0084] Figure 3 This is a schematic diagram of the principle of an index shard allocation method based on Elasticsearch in one embodiment of the present application. Figure 3 As shown, servers 1 to 20 form a cluster. To implement index A and distributed storage of indexes, the index shard allocation method of the embodiment of the present application divides index A into 5 index shards A. 1 , A 2 , A 3 , A 4 , A 5 , divide index B into 5 index shards B 1 , B 2 , B 3 , B 4 , B 5 To improve data availability, each index shard has a corresponding backup shard A. 1 '、A 2 '、A 3 '、A 4 '、A 5 ', B 1 ', B 2 ', B 3 ', B 4 ', B 5 ', when an index shard is unavailable, it can be restored from the corresponding backup shard.

[0085] Servers 1 to 5 belong to the Leaf 1 node, servers 6 to 10 belong to the Leaf 2 node, servers 11 to 15 belong to the Leaf 3 node, and servers 16 to 20 belong to the Leaf 4 node.

[0086] Index Shard A 1 , A 2 , A 3 , A 4 , A 5 Distributed to server 1, server 2, server 3, server 4, and server 5 respectively, backup shard A1 '、A 2 '、A 3 '、A 4 '、A 5 'Distributed to server 6, server 7, server 8, server 9, and server 10 respectively. Index shard B 1 , B 2 , B 3 , B 4 , B 5 Distributed to servers 11, 12, 13, 14, and 15 respectively, backup shard B 1 ', B 2 ', B 3 ', B 4 ', B 5 'Distributed to server 16, server 17, server 18, server 19, and server 20 respectively.

[0087] It is understandable that the embodiment of the present application ensures that the index shards of index A and index B and their corresponding backup shards are allocated to servers with different Leaf node numbers. In this way, the index shards of index A and index B and their corresponding backup shards are distributed on different Leaf nodes, and the servers where the index shards themselves and their backup shards are located are not uniformly powered and networked, and the probability of failure is low. When the server where the index shard is located fails, the server where its backup shard is located can still work normally, and data recovery can be completed from its corresponding backup shard, thereby improving data availability.

[0088] In addition, servers 1 to 10 belong to logical subcluster 1, and servers 16 to 20 belong to logical subcluster 2. 1 , A 2 , A 3 , A 4 , A 5 Distributed on the servers of logical subcluster 1, index shard B of index B 1 , B 2 , B 3 , B 4 , B 5 Distributed on the servers of logical subcluster 2. When logical subcluster 1 fails, index A distributed in the logical subcluster becomes unavailable, but logical subcluster 2 is still working normally, and index B distributed in logical subcluster 2 is still available, further improving data availability.

[0089] Understandably, Figure 3 This is only a specific example of the index shard allocation method of this application and is not intended to limit this application.

[0090] Figure 4 Schematic diagram of the structure of an index shard allocation device based on Elasticsearch in one embodiment of the present application. The index shard allocation device based on Elasticsearch in the embodiment of the present application is applied to a server such as Figure 4 As shown, the device comprises:

[0091] The configuration information receiving unit 410 is used to receive the configuration information of the allocation parameters of the index shards from the user, wherein the allocation parameters of the index shards include the numbers of the servers in the cluster and the corresponding Leaf node numbers;

[0092] The index shard allocation unit 420 is used to allocate the index shards of the target index according to the Leaf node numbers corresponding to the servers in the cluster, and allocate the index shards of the target index and their corresponding backup shards to servers with different Leaf node numbers.

[0093] The index shard allocation device based on Elasticsearch in the embodiment of the present application adds the Leaf node number as an allocation parameter in the index shard allocation process, ensuring that the index shard of the target index itself and its corresponding backup shard are allocated to servers with different Leaf node numbers. In this way, the index shard of the target index itself and its corresponding backup shard are distributed on different Leaf nodes, and the servers where the index shard itself and its backup shard are located are not uniformly powered and networked, and the probability of failure is low. When the server where the index shard is located fails, the server where its backup shard is located can still work normally, and data recovery can be completed from its corresponding backup shard, thereby improving data availability.

[0094] In an optional implementation of the embodiment of the present application, the servers in the cluster belong to different logical subclusters;

[0095] The allocation parameters of the index shards also include: logical sub-cluster numbers corresponding to each server in the cluster;

[0096] The index fragment allocation unit 420 is further configured to:

[0097] The index shards of the target index are allocated according to the logical sub-cluster numbers corresponding to the servers in the cluster, and the index shards of different target indexes are allocated to servers belonging to different logical sub-clusters.

[0098] Furthermore, the device further comprises: a judging unit, configured to judge whether the target index is important data;

[0099] The index shard allocation unit 420 is further configured to allocate index shards of the target index according to the logical sub-cluster numbers corresponding to the servers in the cluster when the target index is important data, and allocate index shards of different target indexes to servers belonging to different logical sub-clusters.

[0100] Specifically, the important data includes: log data and core business data.

[0101] Furthermore, the allocating index shards of different target indexes to servers belonging to different logical subclusters includes:

[0102] The index shards of the first target index and the second target index are allocated according to the logical subcluster numbers corresponding to the servers in the cluster, and the index shards of the first target index are allocated to the servers in the first logical subcluster, and the index shards of the second target index are allocated to the servers in the second logical subcluster.

[0103] Preferably, the first logical sub-cluster and the second logical sub-cluster correspond to different Leaf nodes respectively.

[0104] Furthermore, the device also includes:

[0105] An evaluation unit, used to evaluate the performance of each server in the cluster;

[0106] The index shard allocation unit 420 is further configured to: during the allocation of index shards of the target index, give priority to selecting a server with high performance.

[0107] It is understandable that the above-mentioned index shard allocation device based on Elasticsearch can implement each step of the method provided in the aforementioned embodiment, and the relevant explanations about the method are applicable to the device and will not be repeated here.

[0108] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0109] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0110] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0111] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a reserve fund management device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0112] Receive configuration information of allocation parameters of the index shards from the user, wherein the allocation parameters of the index shards include the numbers of the servers in the cluster and the numbers of the corresponding leaf nodes;

[0113] The index shards of the target index are allocated according to the Leaf node numbers corresponding to the servers in the cluster, and the index shards of the target index and their corresponding backup shards are allocated to servers with different Leaf node numbers.

[0114] The above application Figure 2The method disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0115] The electronic device may also perform Figure 2 The method in this application will not be described in detail here.

[0116] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 2 The method in the embodiment shown is specifically used to perform:

[0117] Receive configuration information of allocation parameters of the index shards from the user, wherein the allocation parameters of the index shards include the numbers of the servers in the cluster and the numbers of the corresponding leaf nodes;

[0118] The index shards of the target index are allocated according to the Leaf node numbers corresponding to the servers in the cluster, and the index shards of the target index and their corresponding backup shards are allocated to servers with different Leaf node numbers.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0120] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0123] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0124] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0125] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0128] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An index shard allocation method based on Elasticsearch, applied to a server, characterized in that: include: Receive configuration information of allocation parameters of the index shards from the user, wherein the allocation parameters of the index shards include the numbers of the servers in the cluster and the numbers of the corresponding leaf nodes; The index shards of the target index are allocated according to the Leaf node numbers corresponding to the servers in the cluster, and the index shards of the target index and their corresponding backup shards are allocated to servers with different Leaf node numbers.

2. The method according to claim 1, characterized in that: The servers in the cluster belong to different logical subclusters; The allocation parameters of the index shards also include: logical sub-cluster numbers corresponding to each server in the cluster; The method further comprises: The index shards of the target index are allocated according to the logical sub-cluster numbers corresponding to the servers in the cluster, and the index shards of different target indexes are allocated to servers belonging to different logical sub-clusters.

3. The method according to claim 2, characterized in that: The method further includes: determining whether the target index is important data; When the target index is important data, the index slices of the target index are allocated according to the logical sub-cluster numbers corresponding to the servers in the cluster, and index slices of different target indexes are allocated to servers belonging to different logical sub-clusters.

4. The method according to claim 3, characterized in that: The important data includes: log data and core business data.

5. The method according to claim 2, characterized in that: The allocating index shards of different target indexes to servers belonging to different logical subclusters includes: The index shards of the first target index and the second target index are allocated according to the logical subcluster numbers corresponding to the servers in the cluster, and the index shards of the first target index are allocated to the servers in the first logical subcluster, and the index shards of the second target index are allocated to the servers in the second logical subcluster.

6. The method according to claim 5, characterized in that The first logical sub-cluster and the second logical sub-cluster correspond to different Leaf nodes respectively.

7. The method according to claim 1, characterized in that: The method further comprises: Evaluating the performance of each server in the cluster; When allocating index shards for the target index, servers with high performance are given priority.

8. An index shard allocation device based on Elasticsearch, applied to a server, characterized in that: include: A configuration information receiving unit, configured to receive configuration information of a user for allocation parameters of an index shard, wherein the allocation parameters of the index shard include numbers of each server in the cluster and numbers of its corresponding leaf nodes; The index shard allocation unit is used to allocate the index shards of the target index according to the Leaf node numbers corresponding to the servers in the cluster, and allocate the index shards of the target index and their corresponding backup shards to servers with different Leaf node numbers.

9. An electronic device, characterized in that: include: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.

Images