Inter-node fragment balancing method and device, equipment and storage medium

Abstract

The invention discloses an inter-node fragment balancing method and device, equipment and a storage medium, and relates to the technical field of computers. The method is used in a cluster comprising m nodes, the m nodes process a fragments in a distributed mode, each fragment in the a fragments corresponds to an index, different fragments correspond to the same index, m is a positive integer larger than 1, and a is a positive integer larger than 1; based on one or more pieces of attribute information of the index, determining a weight value of each fragment in the a fragments; based on the weight value of each fragment, determining the total weight value of the existing fragments of each node in the m nodes; and performing load balancing on the m nodes based on the total weight value of the existing fragments of each node so as to improve the utilization rate of cluster resources and the balance of the balanced fragments among the nodes.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for sharding balance between nodes. Background Technology

[0002] Elasticsearch (ES) cluster is an open-source, distributed, RESTful search engine, typically used in cloud computing, offering real-time search capabilities, stability, reliability, speed, and ease of installation and use.

[0003] The current Elasticsearch cluster uses a load balancing strategy based on the number of shards, which divides the index into a fixed number of shards and distributes the shards evenly across the cluster nodes.

[0004] Because the shards are different from each other, the above method will result in uneven resource allocation among cluster nodes. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for sharding balance between nodes. The technical solution provided by this application includes the following aspects.

[0006] According to one aspect of the embodiments of this application, a method for sharding load balancing among nodes is provided. The method is used in a cluster comprising m nodes, wherein the m nodes distribute and process a shards, each of the a shards corresponds to an index, and different shards may correspond to the same index, where m is a positive integer greater than 1 and a is a positive integer greater than 1.

[0007] Based on one or more attribute information of the index, determine the weight value of each of the a fragments;

[0008] Based on the weight value of each fragment, determine the total weight value of the existing fragments of each of the m nodes;

[0009] Based on the total weight value of the existing shards of each node, load balancing is performed on the m nodes.

[0010] According to one aspect of the embodiments of this application, a sharding load balancing device is provided among nodes. The device includes a cluster of m nodes, wherein the m nodes distribute and process a shards, each of the a shards corresponds to an index, and different shards may correspond to the same index. Here, m is a positive integer greater than 1, and a is a positive integer greater than 1. The device includes:

[0011] The first determining module is used to determine the weight value of each of the a fragments based on one or more attribute information of the index.

[0012] The second determining module is used to determine the total weight value of the existing fragments of each of the m nodes based on the weight value of each fragment.

[0013] The load balancing module is used to perform load balancing on the m nodes based on the total weight value of the existing shards of each node.

[0014] In one possible implementation, the first determining module is configured to determine the weight value of each of the a fragments based on one or more attribute information of the index and the weight coefficients corresponding to the one or more attribute information.

[0015] In one possible implementation, the one or more attribute information includes at least one of the following:

[0016] The index size is the total amount of data occupied by one or more shards corresponding to the index.

[0017] The index importance is the degree of importance of the index among the multiple indexes that the cluster has been split into.

[0018] The number of index shards is the number of shards corresponding to one or more shards of the index.

[0019] In one possible implementation, the attribute information includes the index size, and the weight coefficient corresponding to the index size is a first coefficient;

[0020] The first determining module is used to determine the weight value of each of the a partitions based on the product of the first ratio and the first coefficient, wherein the first ratio is the ratio of the index size to the maximum index size;

[0021] The maximum index size is the amount of data occupied by the shard corresponding to the maximum index, and the maximum index is the index with the largest data volume among all the indexes split from the cluster.

[0022] In one possible implementation, the attribute information includes the index importance, and the weight coefficient corresponding to the index importance is a second coefficient;

[0023] The first determining module is used to determine the weight value of each of the a fragments based on the product of the index importance and the second coefficient.

[0024] In one possible implementation, the attribute information includes the number of index shards, and the weight coefficient corresponding to the number of index shards is a third coefficient;

[0025] The first determining module is used to determine the weight value of each of the a fragments based on the product of the second ratio and the third coefficient, wherein the second ratio is the ratio of the number of indexed fragments to a.

[0026] In one possible implementation, the importance of the index is determined based on the index's operational data over a preset time period;

[0027] The operational data includes at least one of access frequency and read / write operation volume.

[0028] In one possible implementation, the second determining module is used to determine the sum of the weight values ​​of each fragment as the total weight value of the existing fragments of each of the m nodes.

[0029] In one possible implementation, the balancing module is used to determine the balancing weight value of each node based on the total weight value of the existing shards of each node.

[0030] Based on the aforementioned load balancing weight values, load balancing is performed on the m nodes.

[0031] In one possible implementation, the balancing module is used to determine overloaded nodes and underloaded nodes among the m nodes based on the balancing weight value, wherein the overloaded node is a node whose total weight value is greater than the balancing weight value, and the underloaded node is a node whose total weight value is less than the balancing weight value.

[0032] Migration fragments are determined in the overloaded nodes;

[0033] The migration shards are migrated to the underloaded node.

[0034] In one possible implementation, the overloaded node distributes the processing of b shards out of the a shards, where b is a positive integer;

[0035] The balancing module is used to identify the shards among the b shards whose read / write request frequency is less than a preset threshold as the migration shards.

[0036] According to one aspect of the embodiments of this application, a terminal device is provided, the terminal device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described inter-node sharding equalization method.

[0037] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described inter-node sharding balance method.

[0038] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described inter-node sharding equalization method.

[0039] The technical solution provided in this application can bring the following beneficial effects:

[0040] This application first determines the index information corresponding to each shard in the cluster. Based on the load of the shard itself, the shards in the cluster are evenly distributed to m nodes. This avoids placing too many core and important index shards and shards with heavy loads on the same node, thereby improving the cluster resource utilization and the balance of shard distribution among nodes. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a computer system provided in one embodiment of this application;

[0042] Figure 2 This is a flowchart of a node sharding balance method provided in one embodiment of this application;

[0043] Figure 3 This is an overall flowchart of a node sharding balance method provided in one embodiment of this application;

[0044] Figure 4 This is a block diagram of an inter-node sharding equalization device provided in one embodiment of this application;

[0045] Figure 5 This is a structural block diagram of a terminal device provided in one embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0047] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0048] First, the terms used in this application will be explained.

[0049] Elasticsearch (ES), an open-source distributed search engine based on Lucene, supports real-time search and analysis and can efficiently store and retrieve massive amounts of structured, semi-structured and unstructured data, such as log files and e-commerce product data.

[0050] Index: Used to store a collection of structured, similar documents, similar to a "table" in a relational database.

[0051] A node is a single server instance in an Elasticsearch cluster, responsible for storing data and participating in cluster operations.

[0052] Sharding: In order to achieve distributed storage and processing, Elasticsearch divides the index into multiple parts and stores them on different nodes.

[0053] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the control device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0054] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0055] Figure 1 A schematic diagram of a computer system provided in an embodiment of this application is shown. The computer system includes a computer device 101 and a server 102.

[0056] In one possible implementation, computer device 101 is any electronic product capable of human-computer interaction with an interactive object through one or more means such as a keyboard, touchpad, touchscreen, remote control, voice interaction, or handwriting device. Examples include PCs (Personal Computers), mobile phones, smartphones, PDAs (Personal Digital Assistants), wearable devices, handheld portable gaming devices, PPCs (Pocket PCs), tablets, laptops, desktop computers, smart car systems, smart TVs, smart speakers, smartwatches, and in-vehicle terminals, but it is not limited to these.

[0057] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This application embodiment does not limit this. Server 102 communicates directly or indirectly with computer device 101 via wired or wireless communication methods, which is not limited here. Server 102 has data receiving, data processing, and data sending functions. Of course, server 102 may also have other functions, which are not limited in this application embodiment.

[0058] Server 102 provides background services to the clients installed on computer device 101. In one possible implementation, server 102 undertakes the primary computing task, and computer device 101 undertakes the secondary computing task. Alternatively, server 102 undertakes the secondary computing task, and computer device 101 undertakes the primary computing task. Or, computer device 101 and server 102 collaborate on computing using a distributed computing architecture.

[0059] Computer device 101 can refer to one of a plurality of computer devices. This embodiment uses computer device 101 as an example only. Those skilled in the art will know that the number of computer devices 101 can be more or less. For example, there may be only one computer device 101, or there may be dozens or hundreds of computer devices 101, or more. This application embodiment does not limit the number or type of computer devices 101.

[0060] The task processing method based on load information provided in this application embodiment can be executed by computer device 101, server 102, or interactively by computer device 101 and server 102. This application embodiment does not limit the execution of these methods. In some embodiments, computer device 101 can send uplink synchronization data to server 102. The uplink synchronization data includes the current sharding information of m nodes. Server 102 processes the sharding information of the m nodes sent by computer device 101 using the sharding balancing method provided in this application embodiment to obtain the final load balancing method, and then sends downlink synchronization data to computer device 101. The downlink synchronization data includes the load balancing method.

[0061] For example, such as Figure 1As shown, this method is used in a cluster consisting of three nodes: node 1 (103), node 2 (104), and node 3 (105). The three nodes distribute the processing across eight shards: shard 1, shard 2, shard 3, shard 4, shard 5, shard 6, shard 7, and shard 8. Shards 1, shard 3, and shard 5 correspond to index 1; shards 2, shard 4, and shard 7 correspond to index 2; and shards 6 and shard 8 correspond to index 3.

[0062] For index 1, the index size (106), index importance (107), and number of index shards (108) are determined. Based on the product of a first ratio and a first coefficient, the first weight values ​​corresponding to shard 1, shard 3, and shard 5 are determined respectively. The first ratio is the ratio of the index size to the maximum index size; the index size is the total amount of data occupied by shard 1, shard 3, and shard 5; the maximum index size is the amount of data occupied by the shard corresponding to the maximum index, which is the index with the largest data volume among all the indexes split from the cluster. Based on the product of the index importance (107) and a second coefficient, the second weight values ​​of shard 1, shard 3, and shard 5 are determined; the index importance is the degree of importance of index 1 among the multiple indexes split from the cluster. The third weight value for shard 1, shard 3, and shard 5 is determined based on the product of the second ratio and the third coefficient. The second ratio is the ratio of the number of index shards (shards 1, shard 3, and shard 5 are the three index shards) to 8 (the total number of shards). The number of index shards is the number of shards 1, shard 3, and shard 5 corresponding to index 1. The sum of the first weight value, the second weight value, and the third weight value is the weight value of 1 for each shard corresponding to index 1.

[0063] The weight value 2 corresponding to shard 2 is determined by the above method. The sum of the weight value 1 corresponding to shard 1, the weight value 2 corresponding to shard 2, and the weight value 1 corresponding to shard 3 is determined as the total weight value 1 of the existing shards of node 1.

[0064] Based on the above method, the total weight value 2 of the existing fragments of node 2 and the total weight value 3 of the existing fragments of node 3 are determined. The sum of the total weight value 1, the total weight value 2 and the total weight value 3 are determined as the balanced weight value of each node 103, node 204 and node 305.

[0065] Based on the balanced weight value, overloaded nodes and underloaded nodes are identified among the three nodes. Overloaded nodes are those with a total weight value greater than the balanced weight value, and underloaded nodes are those with a total weight value less than the balanced weight value. Migration shards are then determined from the overloaded nodes.

[0066] Those skilled in the art should understand that the computer device 101 and server 102 described above are merely illustrative examples. Other existing or future computer devices or servers that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0067] Based on the above Figure 1 The computer system shown in this application provides a method for sharding load balancing between nodes. This method can be executed by computer device 101, server 102, or by interaction between computer device 101 and server 102. This application does not limit the specific implementation of this method.

[0068] Current Elasticsearch clusters employ a load balancing strategy based on the number of shards. This involves dividing the index into a fixed number of shards and distributing them evenly across the cluster nodes. The core algorithm uses only the difference in the number of shards between nodes as the basis for load balancing.

[0069] In related technologies, data leveling is triggered when the difference in the number of shards between nodes exceeds a threshold, migrating shards from nodes with more shards to nodes with fewer shards. However, this approach does not consider the actual data volume of each shard, leading to the equivalent treatment of large and small shards, resulting in uneven allocation of node storage and computing resources. Furthermore, it fails to differentiate between high-priority indexes, potentially causing service delays due to hot nodes.

[0070] This application embodiment illustrates the method by example, where the method is executed by a computer device, which can be either a computer or a server. Figure 2 As shown, the sharding balance method between nodes provided in this application embodiment may include at least one of the following steps 201 to 203. The method is used in a cluster including m nodes, where the m nodes distribute and process a shards, each of the a shards corresponds to an index, and there are different shards corresponding to the same index, where m is a positive integer greater than 1 and a is a positive integer greater than 1.

[0071] In step 201, the weight value of each of the a shards is determined based on one or more attribute information of the index.

[0072] The cluster consists of m nodes, which distribute the processing across a shards. Each shard has a corresponding index, and the indexes carry the data within the cluster. In other words, the cluster constructs different indexes, which can be understood as collections of documents with similar structures. The data in the cluster belongs to different indexes, and the index data is split into a shards, distributed across m nodes. Through distributed processing by these m nodes, load balancing is achieved.

[0073] A 'a' shard is a large dataset (cluster) divided into multiple smaller parts according to rules. Each part is called a shard. In the cluster, 'a' shards are distributed across 'm' nodes for storage and processing.

[0074] For example, a cluster containing 1 billion user transaction records divides the data into 100 shards, each containing approximately 10 million records. Each of the m nodes only needs to process a portion of the data, avoiding performance degradation of a single node due to excessive data volume.

[0075] In the cluster, each of the m nodes is responsible for processing shards. Each node is responsible for the storage, read / write operations, and data management of one or more shards.

[0076] The idea that each shard in a shard corresponds to an index can be understood as creating an index for each shard to ensure efficient data retrieval.

[0077] For example, taking order data from an e-commerce platform, in the shards processed by node A with order identifiers ranging from 1 to 100,000, indexes will be created for the order identifier, user identifier, and order time. When a query is executed for this shard, whether it's searching for orders based on the order identifier or counting the number of orders for a user based on the user identifier, the index can be used to quickly locate the data. Each shard corresponds to one index, ensuring that the index is closely integrated with the shard data and reducing the overhead of cross-shard queries.

[0078] The existence of different shards corresponding to the same index can be understood as follows: although each shard has its own index, there may be situations where different shards correspond to the same index. This is because data in different shards needs to be queried using the same logic.

[0079] For example, document data is sharded and stored across multiple nodes. To enable full-text search, all documents in all shards need to have the same full-text index. When a user account initiates a full-text search request, the cluster sends the request to all nodes containing the relevant shards. Each node performs the search based on the full-text index of its local shard, and finally aggregates the results and returns them to the user.

[0080] One or more attribute information of an index is a set of configuration parameters that control the index's behavior, performance, and storage method.

[0081] Based on one or more attribute information of the index, the weight value of each shard in 'a' shards is determined. This can be understood as determining the importance of each shard in the cluster among the 'a' shards after combining one or more attribute information of its corresponding index.

[0082] Optionally, the weight value of each of the a shards is determined based on one or more attribute information of the index, including: determining the weight value of each of the a shards based on one attribute information of the index.

[0083] For example, based on the index size, the weight value of each of the 'a' shards is determined, where the index size is the total amount of data occupied by one or more shards corresponding to the index. That is, the weight value of each of the 'a' shards is measured from the perspective of the index size.

[0084] Optionally, the weight value of each of the a shards is determined based on one or more attribute information of the index, including: determining the weight value of each of the a shards based on multiple attribute information of the index.

[0085] It should be noted that multiple attribute information can be understood as at least two attribute information.

[0086] Optionally, one or more attribute information includes at least one of the following: the index size of the index, which is the total amount of data occupied by one or more shards corresponding to the index; the index importance, which is the importance of the index among the multiple indexes split into in the cluster; and the number of index shards, which is the number of shards of one or more shards corresponding to the index.

[0087] The index size can be understood as the disk usage information of the index.

[0088] For example, the weight value of each shard in a 'a' shards is determined based on the index size; or, the weight value of each shard in a 'a' shards is determined based on the index importance; or, the weight value of each shard in a 'a' shards is determined based on the number of index shards; or, the weight value of each shard in a 'a' shards is determined based on both the index size and the index importance; or, the weight value of each shard in a 'a' shards is determined based on both the index size and the number of index shards; or, the weight value of each shard in a 'a' shards is determined based on both the index importance and the number of index shards; or, the weight value of each shard in a 'a' shards is determined based on both the index size, the index importance, and the number of index shards.

[0089] Optionally, the importance of an index is determined based on the index's operational data over a preset period of time; wherein, the operational data includes at least one of access frequency and read / write operation volume.

[0090] The preset time is a pre-defined time used to collect and analyze index data.

[0091] Optionally, the preset time can be set in one of the following ways, including but not limited to: determining it based on historical running data of the index; or, setting it based on a user account.

[0092] The access frequency is the frequency of accessing the index within a preset time period, and the read / write operation volume is the total number of read / write requests to the index within a preset time period.

[0093] Optionally, determining index importance based on index operation data within a preset time period includes: determining index importance based on the access frequency of the index within the preset time period.

[0094] Optionally, determining index importance based on the index's operational data within a preset time period includes: determining index importance based on the amount of read and write operations performed on the index within the preset time period.

[0095] Optionally, the importance of an index is determined based on the index's operational data over a preset period, including: determining the index importance based on the index's access frequency and the amount of read and write operations over the preset period.

[0096] For example, taking a seven-day week as a preset timeframe, the total number of read and write requests for each index over the past week is statistically analyzed, and then converted into index importance using a preset algorithm. This method can reflect the actual importance of an index in more real-time.

[0097] Optionally, index importance is set by the system administrator for each index based on business needs.

[0098] Optionally, the setting method includes, but is not limited to: using a 1-5 point rating system, with higher scores indicating higher index importance.

[0099] For example, the index with the highest index importance is given a score of 5, and the index with the lowest index importance is given a score of 1.

[0100] The weight value for each of the 'a' shards represents its importance to the other 'a' shards in the cluster. This weight value is determined by comprehensively considering one or more attribute information of the index corresponding to each shard.

[0101] In step 202, based on the weight value of each shard, the total weight value of the existing shards of each of the m nodes is determined.

[0102] Since each of the m nodes includes at least one shard, the total weight value of the distributed storage shards on each node is determined by the weight value of each shard. That is, the m nodes in the cluster are traversed, and the sum of the balanced weights (total weight value) of all shards currently hosted on each node is calculated.

[0103] Optionally, based on the weight value of each shard, the total weight value of the existing shards of each of the m nodes is determined, including: determining at least one shard for distributed processing by each of the m nodes; determining the sub-weight values ​​corresponding to each of the at least one shard; and summing the sub-weight values ​​corresponding to each of the at least one shards to obtain the total weight value.

[0104] In step 203, load balancing is performed on m nodes based on the total weight value of the existing shards of each node.

[0105] Given the total weight value of each node's existing shards, the differences in the total weight value corresponding to each node can be identified. Based on the differences in the total weight value corresponding to each node, load balancing is performed on m nodes.

[0106] Regarding the total weight value of existing shards on each node, it can be determined that if the total weight value is too large, the node may be overloaded, meaning that the load is distributed across too many shards. It's important to clarify that overloading the load across shards includes not only distributing an excessive number of shards, but also distributing the load across shards that handle an excessive amount of data. Conversely, if the total weight value is too small, the node may be underutilized, meaning that some of the node's resources are idle and being wasted.

[0107] Based on the total weight of the existing shards of each node, load balancing is performed on m nodes, including migrating shards from nodes with excessively high total weights to nodes with excessively low total weights.

[0108] It should be noted that since the amount of data and the number of shards in the cluster are dynamically adjusted, load balancing is also a continuous process.

[0109] Based on the total weight value of the existing shards of each node, load balancing is performed on m nodes. This comprehensively considers one or more attribute information of the index, improving the efficiency and balance of load balancing.

[0110] This application first determines the index information corresponding to each shard in the cluster. Based on the load of the shard itself, the shards in the cluster are evenly distributed to m nodes. This avoids placing too many core and important index shards and shards with heavy loads on the same node, thereby improving the cluster resource utilization and the balance of shard distribution among nodes.

[0111] In some embodiments, determining the weight value of each of the a shards based on one or more attribute information of the index includes: determining the weight value of each of the a shards based on one or more attribute information of the index and the weight coefficients corresponding to the one or more attribute information respectively.

[0112] Each attribute corresponds to its own weight coefficient, which is a value used to measure the relative importance of one or more attribute information in the whole. It reflects the proportion or degree of influence of one or more attribute information on the whole.

[0113] Optionally, based on one or more attribute information of the index and the weight coefficients corresponding to one or more attribute information, the weight value of each of the a shards is determined, including: determining the product corresponding to each attribute information of the index, the product being the product of the attribute information and its corresponding weight coefficient; and the sum of the products corresponding to one or more attribute information is the weight value of each shard.

[0114] Optionally, the attribute information includes the index size, and the weight coefficient corresponding to the index size is the first coefficient; based on one or more attribute information of the index, and the weight coefficients corresponding to one or more attribute information respectively, the weight value of each shard in the a shards is determined, including: based on the product of the first ratio and the first coefficient, the weight value of each shard in the a shards is determined, where the first ratio is the ratio of the index size to the maximum index size; wherein, the maximum index size is the amount of data occupied by the shard corresponding to the maximum index, and the maximum index is the index with the largest amount of data among all the indexes split from the cluster.

[0115] The first coefficient is the weight coefficient corresponding to the index size. The first coefficient can be set according to at least one of the following: the business scenario and the cluster running status.

[0116] For example, when the business scenario has high requirements for response speed to the large total data volume occupied by shards, the first coefficient is increased.

[0117] The maximum index size is the index with the largest total data volume among all the split indexes built in the cluster, corresponding to one or more shards.

[0118] The first ratio is the ratio of the current index size to the maximum index size. The weight value of each of the 'a' shards is determined by multiplying the first ratio by the first coefficient. This can be understood as determining the weight score of each shard in the current index by multiplying the first ratio by the first coefficient.

[0119] For example,

[0120] In formula (1), α is the first coefficient; Size is the index size; MaxSize is the maximum index size.

[0121] Optionally, the attribute information includes index importance, and the weight coefficient corresponding to the index importance is the second coefficient; based on one or more attribute information of the index, and the weight coefficients corresponding to one or more attribute information respectively, the weight value of each of the a shards is determined, including: based on the product of the index importance and the second coefficient, the weight value of each of the a shards is determined.

[0122] The second coefficient is the weighting coefficient corresponding to the index importance. The second coefficient can be set based on at least one of the following: the business scenario and the cluster's operating status.

[0123] For example, when the business has extremely high requirements for the response speed of important indexes, the second coefficient is increased.

[0124] For example, the weight value of each of the a fragments is β × Importance, as shown in formula (2).

[0125] In formula (2), β is the second coefficient; Importance is the index importance.

[0126] Optionally, the attribute information includes the number of index shards, and the weight coefficient corresponding to the number of index shards is the third coefficient; based on one or more attribute information of the index, and the weight coefficients corresponding to one or more attribute information respectively, the weight value of each shard in the 'a' shards is determined, including: based on the product of the second ratio and the third coefficient, the weight value of each shard in the 'a' shards is determined, where the second ratio is the ratio of the number of index shards to 'a'.

[0127] The third coefficient is the weighting coefficient corresponding to the number of index shards. The third coefficient can be set according to at least one of the following: the business scenario and the cluster operating status.

[0128] For example, when the business has extremely high requirements for response speed to a large number of index shards, the third coefficient is increased.

[0129] The second ratio is the ratio of the number of index shards into which the current index is split to the total number of index shards (a) into which all indexes in the cluster are split.

[0130] For example,

[0131] In formula (3), γ is the third coefficient; ShardNum is the number of index shards of the current index; TotalShardNumde takes the value a.

[0132] Optionally, when the attribute information includes the index size, index importance, and number of index shards, the weight value of each shard in the 'a' shards is determined based on one or more attribute information of the index and the weight coefficients corresponding to the one or more attribute information, including: determining the first product of a first ratio and a first coefficient, where the first ratio is the ratio of the index size to the maximum index size; determining the second product of the index importance and a second coefficient; determining the third product of the second ratio and a third coefficient, where the second ratio is the ratio of the number of index shards to 'a'; and determining the weight value of each shard in the 'a' shards based on the sum of the first product, the second product, and the third product.

[0133] For example,

[0134] In formula (4), Weight is the weight value of each of the a-parts; The first product is β; β × Importance is the second product. The third product is α + β + γ = 1.

[0135] It should be noted that the first, second, and third coefficients can be dynamically adjusted according to the actual business scenario and cluster operation status.

[0136] In some embodiments, determining the total weight value of existing shards for each of the m nodes based on the weight value of each shard includes: determining the sum of the weight values ​​of each shard as the total weight value of existing shards for each of the m nodes.

[0137] After calculating the weight value of each shard, since the existing shards in each node may correspond to different weight values, the weight values ​​of each shard are the same when the existing shards correspond to the same index, and different when the existing shards correspond to different indices.

[0138] The sum of the weights of each shard is determined as the total weight of the existing shards of each of the m nodes. This can be understood as traversing the m nodes in the cluster and calculating the sum of the weights of all the shards currently carried by each node.

[0139] In some embodiments, load balancing is performed on m nodes based on the total weight value of the existing shards of each node, including: determining the balance weight value of each node based on the total weight value of the existing shards of each node; and performing load balancing on the m nodes based on the balance weight value.

[0140] Optionally, based on the total weight value of the existing shards of each node, the equilibrium weight value of each node is determined, including: based on the total weight value of the existing shards of each node, the total weight value of all shards corresponding to m nodes is determined; and the average of the total weight values ​​of all shards is determined as the equilibrium weight value of each node.

[0141] Optionally, load balancing is performed on m nodes based on the balance weight value, including: determining overloaded nodes and underloaded nodes among the m nodes based on the balance weight value, where an overloaded node is a node whose total weight value is greater than the balance weight value and an underloaded node is a node whose total weight value is less than the balance weight value; determining migration shards among the overloaded nodes; and migrating the migration shards to the underloaded nodes.

[0142] By migrating shards of overloaded nodes, service degradation caused by resource exhaustion of individual nodes is avoided, ensuring uniform overall cluster response speed; the idle resources of underloaded nodes are fully utilized, improving cluster resource utilization; after load balancing, the request processing pressure of each node is evenly distributed, avoiding request queuing or timeouts caused by overload; in addition, when a node fails, because the load has been evenly distributed, other nodes can take over the shards of the failed node more quickly, reducing fault recovery time.

[0143] An overloaded node is a node in the cluster whose resource usage (i.e., total weight value) exceeds the balanced weight value. In other words, it can be understood as a node in the cluster whose workload exceeds its processing capacity limit. "Load" includes indicators such as CPU utilization, memory usage, network traffic, or request processing volume.

[0144] An underutilized node is a node in the cluster whose resource utilization (i.e., total weight value) is lower than the balanced weight value, and its resources are in an idle state.

[0145] Optionally, migrating the migration shards to the underloaded node includes: determining the target underloaded node for migration based on the remaining resources of the underloaded node, network topology (selecting nodes with low network latency to the overloaded node), and hardware configuration (such as storage capacity and CPU performance); and migrating the migration shards to the target underloaded node.

[0146] Optionally, the method further includes: when the number of migration fragments exceeds a preset threshold, migrating the migration fragments to underloaded nodes in batches to avoid network congestion or instantaneous overload of the target underloaded node caused by a one-time migration.

[0147] Optionally, the method further includes: confirming, through at least one of hash verification and reconciliation tools, that the data after the migration shards are completely consistent with the data before the migration, so as to avoid data loss or damage during the migration process.

[0148] Optionally, the method also includes: if an anomaly such as network interruption or underloaded node failure occurs during the migration process, automatically triggering a rollback mechanism (such as terminating the migration and restoring the source shard service), and recording anomaly logs for subsequent optimization.

[0149] Optionally, the overloaded node can be determined by identifying the node whose total weight value is greater than the equilibrium weight value among the m nodes as the overloaded node.

[0150] Optionally, the underloaded node can be determined by identifying the node whose total weight value is less than the equilibrium weight value among the m nodes as the underloaded node.

[0151] Migrating shards are shards selected from overloaded nodes and migrated to underloaded nodes.

[0152] In some embodiments, the overloaded node distributes the processing of b shards out of a shards, where b is a positive integer; determining the migration shards in the overloaded node includes: determining the shards among the b shards whose read / write request frequency is less than a preset threshold as migration shards.

[0153] The preset threshold is a pre-defined boundary value for the frequency of fragment read and write requests.

[0154] Optionally, the preset threshold may be determined in a way that includes, but is not limited to, at least one of the following: determining the preset threshold based on the historical read / write request frequency of the shard; or determining the preset threshold based on the user account's custom settings.

[0155] It should be noted that by comparing the total weight value of each node with its balanced weight value, nodes with a total weight value higher than the balanced weight value are identified as overloaded nodes, and nodes with a total weight value lower than the balanced weight value are identified as underloaded nodes. High-weight shards are selected from the overloaded nodes (i.e., migrated shards) and moved to the underloaded nodes. When selecting shards for migration, priority is given to shards that will have a smaller impact on the overloaded and underloaded nodes after migration. For example, by monitoring the read and write request frequency of shards, shards that are being frequently read and written are avoided from being migrated.

[0156] By continuously repeating the above steps, the shards of the ES cluster are dynamically balanced to adapt to changes in index data and business needs.

[0157] For example, taking a cluster environment with 3 nodes (Node1, Node2, Node3) and 3 indices (Index A, Index B, Index C), and weight coefficients set as α (first coefficient) = 0.4; β (second coefficient) = 0.5; γ (third coefficient) = 0.1, the sharding balance method among nodes in this application will be described.

[0158] See Table 1 for the basic parameters of the index in the cluster.

[0159]

[0160] Table 1

[0161] Based on Table 1, we determine that MaxSize = 10GB and TotalShardNum = 5 + 3 + 2 = 10 in the above formula (4).

[0162] Calculate the weight value of each of the a-parts.

[0163] The weight values ​​of each of the 5 shards corresponding to index A:

[0164] 0.4*(10 / 10)+0.5*5+0.1*(5 / 10)=0.4*1+2.5+0.1*0.5=2.95.

[0165] The weight values ​​of each of the three shards corresponding to index B:

[0166] 0.4*(5 / 10)+0.5*3+0.1*(3 / 10)=0.4*0.5+1.5+0.1*0.3=1.65.

[0167] The weight values ​​of each of the two shards corresponding to index C:

[0168] 0.4*(1 / 10)+0.5*1+0.1*(2 / 10)=0.4*0.1+0.5+0.1*0.5=0.35.

[0169] See Table 2 for the distribution of distributed processing shards among the top 3 load balancers in the cluster.

[0170]

[0171]

[0172] Table 2

[0173] Calculate the total weight of the existing fragments for each of the m (3) nodes.

[0174] Total weight of Node1: 3*2.95+1*1.65=8.85+1.65=10.5.

[0175] Total weight of Node2: 2*2.95+2*1.65+1*0.35=5.9+3.3+0.35=9.55.

[0176] Total weight of Node3: 1 * 0.35 = 0.35.

[0177] Based on the total weight value of the existing shards of each node, determine the balanced weight value of each node.

[0178] The equilibrium weight value is (10.5 + 9.55 + 0.35) / 3 = 6.8.

[0179] Based on the balanced weight values, overloaded nodes and underloaded nodes are identified among the three nodes.

[0180] Overloaded nodes: Node1 (10.5>6.8), Node2 (9.55>6.8).

[0181] Underloaded node: Node3 (0.35 < 6.8).

[0182] Identify migration shards in overloaded nodes; migrate the migration shards to underloaded nodes.

[0183] Step 1: Migrate a low-weight shard from Node1 to Node3.

[0184] The shard with the lowest total weight value in Node1 is index B (total weight value is 1.65), so migrate one index B shard.

[0185] The total weight of Node1 after migration is 10.5 - 1.65 = 8.85.

[0186] Node3 total weight: 0.35 + 1.65 = 2.0.

[0187] Step 2: Migrate one low-weight shard from Node2 to Node3.

[0188] In Node2, the shard with the lowest total weight value is index C (weight 0.35), but it only has 1 shard. If availability needs to be maintained, the index B shard (weight 1.65) should be migrated first.

[0189] The total weight of Node2 after migration is 9.55 - 1.65 = 7.9.

[0190] Total weight of Node3: 2.0 + 1.65 = 3.65.

[0191] Step 3: Continue migrating one index B shard from Node1 to Node3 (if load balancing is still required).

[0192] The total weight of Node1 after migration is 8.85 - 1.65 = 7.2.

[0193] Total weight of Node3: 3.65 + 1.65 = 5.3.

[0194] For example, in large e-commerce platforms, product search systems need to handle massive amounts of product data and high-concurrency query requests. For instance, a product database might contain 1 billion product listings, covering over 200 categories such as clothing, electronics, and home furnishings. Each category can be considered an independent "index," and user account search requests need to quickly locate relevant products from this data in real time.

[0195] Taking a cluster consisting of three nodes (node ​​1, node 2, and node 3), with 200 indexes, and weighting coefficients set as follows: α (first coefficient) = 0.4; β (second coefficient) = 0.5; γ (third coefficient) = 0.1, this paper illustrates the sharding balance method among nodes in this application.

[0196] First, based on the above formula (4), calculate the weight value of each shard in the sharding. That is, calculate the weight value of each shard corresponding to each of the 200 indices. Then, calculate the total weight value of the existing shards of each node; based on the total weight value of the existing shards of each node, determine the balanced weight value of each node. Finally, based on the balanced weight value, determine the overloaded nodes and underloaded nodes in the node; determine the shards to migrate in the overloaded nodes; and migrate the shards to the underloaded nodes. Thus, shard load balancing among nodes in a large e-commerce platform is achieved.

[0197] By continuously repeating the above steps, the sharding of the ES cluster is dynamically balanced to adapt to changes in index data and business needs, so as to achieve continuous sharding load balancing among nodes in a large e-commerce platform.

[0198] See Figure 3 The diagram shows the overall flowchart of the sharding balance method between nodes.

[0199] The method is used in a cluster consisting of m nodes, where the m nodes distribute the processing of a shards. Each shard in the a shards corresponds to an index, and there are instances where different shards correspond to the same index. m is a positive integer greater than 1, and a is a positive integer greater than 1.

[0200] Step 1: Determine the first product of the first ratio and the first coefficient.

[0201] It should be noted that the index size is the total amount of data occupied by one or more shards corresponding to the index; the first ratio is the ratio of the index size to the maximum index size; the maximum index size is the amount of data occupied by the shard corresponding to the maximum index, and the maximum index is the index with the largest amount of data among all the indexes split into the cluster.

[0202] Step 2: Determine the second product of the index importance and the second coefficient.

[0203] It should be noted that index importance refers to the degree of importance of an index among the multiple indexes split into in the cluster; index importance is determined based on the index's operational data over a preset period of time; where operational data includes at least one of access frequency and read / write operation volume.

[0204] Step 3: Determine the third product of the second ratio and the third coefficient.

[0205] It should be noted that the number of index shards is the number of shards of one or more shards corresponding to the index; the second ratio is the ratio of the number of index shards to 'a'.

[0206] Furthermore, this application does not restrict the execution order of steps 1, 2, and 3 mentioned above.

[0207] Step 4: Sum the first, second, and third products corresponding to the existing fragments of each of the m nodes to determine the total weight value of the existing fragments of each of the m nodes.

[0208] Step 5: Determine the balanced weight value of each node based on the total weight value of the existing shards of each node.

[0209] Step 6: Based on the balanced weight value, determine the overloaded nodes and underloaded nodes among the m nodes.

[0210] It should be noted that an overloaded node is a node whose total weight value is greater than the equilibrium weight value, and an underloaded node is a node whose total weight value is less than the equilibrium weight value.

[0211] Step 7: Determine the shards with a read / write request frequency less than a preset threshold among the b shards as migration shards.

[0212] It should be noted that the shards with a read / write request frequency less than a preset threshold among the b shards are identified as migration shards.

[0213] Step 8: Migrate the migration shards to the underloaded nodes.

[0214] This application comprehensively considers various index attributes and optimizes the shard allocation strategy among cluster nodes, achieving efficient utilization of cluster resources and a significant improvement in service quality. It comprehensively evaluates three key factors—index size, index importance, and the number of index shards—to construct a comprehensive and reasonable shard balance weight calculation model. Furthermore, a shard affinity strategy ensures the service quality of high-priority indexes.

[0215] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0216] Please refer to Figure 4This diagram illustrates a block diagram of a node sharding load balancing device according to an embodiment of this application. The device includes a cluster of m nodes, where the m nodes distribute and process a shards. Each of the a shards corresponds to an index, and different shards may correspond to the same index. Here, m is a positive integer greater than 1, and a is a positive integer greater than 1. The device includes:

[0217] The first determining module 401 is used to determine the weight value of each of the a fragments based on one or more attribute information of the index.

[0218] The second determining module 402 is used to determine the total weight value of the existing fragments of each of the m nodes based on the weight value of each fragment.

[0219] The load balancing module 403 is used to perform load balancing on m nodes based on the total weight value of the existing shards of each node.

[0220] In one possible implementation, the first determining module 401 is used to determine the weight value of each of the a fragments based on one or more attribute information of the index and the weight coefficients corresponding to the one or more attribute information respectively.

[0221] In one possible implementation, one or more attribute information includes at least one of the following:

[0222] The index size is the total amount of data occupied by one or more shards corresponding to the index.

[0223] Index importance refers to the degree of importance of an index among the multiple indexes that have been split into in the cluster.

[0224] The number of index shards, which is the number of shards corresponding to one or more shards of the index.

[0225] In one possible implementation, the attribute information includes the index size, and the weight coefficient corresponding to the index size is the first coefficient.

[0226] The first determining module 401 is used to determine the weight value of each of the a partitions based on the product of the first ratio and the first coefficient. The first ratio is the ratio of the index size to the maximum index size.

[0227] The maximum index size is the amount of data occupied by the shard corresponding to the maximum index, and the maximum index is the index with the largest data volume among all the indexes split in the cluster.

[0228] In one possible implementation, the attribute information includes index importance, and the weight coefficient corresponding to the index importance is the second coefficient.

[0229] The first determining module 401 is used to determine the weight value of each of the a fragments based on the product of the index importance and the second coefficient.

[0230] In one possible implementation, the attribute information includes the number of index shards, and the weight coefficient corresponding to the number of index shards is a third coefficient.

[0231] The first determining module 401 is used to determine the weight value of each of the a fragments based on the product of the second ratio and the third coefficient, wherein the second ratio is the ratio of the number of indexed fragments to a.

[0232] In one possible implementation, index importance is determined based on the index's operational data over a preset time period;

[0233] The operational data includes at least one of the following: access frequency and read / write operation volume.

[0234] In one possible implementation, the second determining module 402 is used to determine the sum of the weight values ​​of each fragment as the total weight value of the existing fragments of each of the m nodes.

[0235] In one possible implementation, the balancing module 403 is used to determine the balanced weight value of each node based on the total weight value of the existing shards of each node.

[0236] Load balancing is performed on m nodes based on the balanced weight values.

[0237] In one possible implementation, the balancing module 403 is used to determine overloaded nodes and underloaded nodes among m nodes based on the balancing weight value. Overloaded nodes are nodes whose total weight value is greater than the balancing weight value, and underloaded nodes are nodes whose total weight value is less than the balancing weight value.

[0238] Determine migration fragments in overloaded nodes;

[0239] Migrate the shards to the underloaded nodes.

[0240] In one possible implementation, the overloaded nodes distribute the processing of b shards out of a shards, where b is a positive integer;

[0241] The load balancing module 403 is used to identify the shards among b shards whose read / write request frequency is less than a preset threshold as migration shards.

[0242] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0243] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the inter-node sharding equalization method provided in the above-described method embodiments.

[0244] For example, Figure 5 This is a structural block diagram of a computer device 1000 provided in an exemplary embodiment of this application. The computer device 1000 may be... Figure 1 The computer device shown is used to implement the inter-node sharding balance method provided in the above embodiments.

[0245] The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including random access memory (RAM) 1002 and read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 to facilitate information transfer between various components within the computer device, and a mass storage device 1007 for storing the operating system 1013, application programs 1014, and other program modules 1015.

[0246] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1010 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0247] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable storage media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include computer-readable storage media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0248] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable storage instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1004 and mass storage device 1007 described above can be collectively referred to as memory.

[0249] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1001. The one or more programs contain instructions for implementing the above method embodiments. The central processing unit 1001 executes the one or more programs to implement the inter-node sharding balance method provided by the above method embodiments.

[0250] According to various embodiments of this application, the computer device 1000 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1012 via the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer device systems (not shown).

[0251] The memory further includes one or more programs stored in the memory, and the one or more programs include steps executed by a computer device in the inter-node sharding equalization method provided in the embodiments of this application.

[0252] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. When the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor of a computer device, the inter-node sharding balance method provided in the above-described method embodiments is implemented.

[0253] This application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the inter-node sharding equalization method provided in the above-described method embodiments.

[0254] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0255] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for sharding and load balancing among nodes, characterized in that, The method is used in a cluster consisting of m nodes, where the m nodes distribute and process a shards, each of the a shards corresponds to an index, and different shards may correspond to the same index, where m is a positive integer greater than 1 and a is a positive integer greater than 1. Based on one or more attribute information of the index, determine the weight value of each of the a fragments; Based on the weight value of each fragment, determine the total weight value of the existing fragments of each of the m nodes; Based on the total weight value of the existing shards of each node, load balancing is performed on the m nodes.

2. The method according to claim 1, characterized in that, The determination of the weight value of each of the a shards based on one or more attribute information of the index includes: Based on one or more attribute information of the index, and the weight coefficients corresponding to the one or more attribute information, the weight value of each of the a fragments is determined.

3. The method according to claim 2, characterized in that, The one or more attribute information includes at least one of the following: The index size is the total amount of data occupied by one or more shards corresponding to the index. The index importance is the degree of importance of the index among the multiple indexes that the cluster has been split into. The number of index shards is the number of shards corresponding to one or more shards of the index.

4. The method according to claim 3, characterized in that, The attribute information includes the index size, and the weight coefficient corresponding to the index size is the first coefficient; The determination of the weight value of each of the a shards based on one or more attribute information of the index and the weight coefficients corresponding to the one or more attribute information includes: Based on the product of the first ratio and the first coefficient, the weight value of each of the a fragments is determined, where the first ratio is the ratio of the index size to the maximum index size. The maximum index size is the amount of data occupied by the shard corresponding to the maximum index, and the maximum index is the index with the largest data volume among all the indexes split from the cluster.

5. The method according to claim 3, characterized in that, The attribute information includes the index importance, and the weight coefficient corresponding to the index importance is the second coefficient; The determination of the weight value of each of the a shards based on one or more attribute information of the index and the weight coefficients corresponding to the one or more attribute information includes: The weight value of each of the a fragments is determined based on the product of the index importance and the second coefficient.

6. The method according to claim 3, characterized in that, The attribute information includes the number of index shards, and the weight coefficient corresponding to the number of index shards is the third coefficient; The determination of the weight value of each of the a shards based on one or more attribute information of the index and the weight coefficients corresponding to the one or more attribute information includes: The weight value of each of the a fragments is determined based on the product of the second ratio and the third coefficient, where the second ratio is the ratio of the number of indexed fragments to a.

7. The method according to any one of claims 3 to 6, characterized in that, The importance of the index is determined based on the index's operational data over a preset time period; The operational data includes at least one of access frequency and read / write operation volume.

8. The method according to any one of claims 1 to 7, characterized in that, Determining the total weight value of the existing shards of each of the m nodes based on the weight values of each of the shards includes: Determining the sum of the weight values of each of the shards as the total weight value of the existing shards of each of the m nodes.

9. The method according to any one of claims 1 to 8, characterized in that, Performing load balancing on the m nodes based on the total weight value of the existing shards of each of the nodes includes: Determining the balanced weight value of each of the nodes based on the total weight value of the existing shards of each of the nodes; Performing load balancing on the m nodes based on the balanced weight value.

10. The method according to claim 9, characterized in that, Performing load balancing on the m nodes based on the balanced weight value includes: Based on the balanced weight value, determining overloaded nodes and underloaded nodes among the m nodes, where the overloaded nodes are the nodes with the total weight value greater than the balanced weight value, and the underloaded nodes are the nodes with the total weight value less than the balanced weight value; Determining shards to be migrated among the overloaded nodes; Migrating the shards to be migrated to the underloaded nodes.

11. The method according to claim 10, characterized in that, The overloaded nodes distributively process b of the a shards, where b is a positive integer; Determining shards to be migrated among the overloaded nodes includes: Determining the shards with the read-write request frequency less than a preset threshold among the b shards as the shards to be migrated.

12. A terminal device, characterized in that, The terminal device includes a processor and a memory, and a computer program is stored in the memory. The computer program is loaded and executed by the processor to implement the shard balancing method between nodes according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium. The computer program is loaded and executed by the processor to implement the shard balancing method between nodes according to any one of claims 1 to 11.

14. A computer program product, characterized in that, The computer program product includes a computer program. The computer program is stored in the computer-readable storage medium. The processor reads and executes the computer program from the computer-readable storage medium to implement the shard balancing between nodes according to any one of claims 1 to 11.

Images