Policy adjustment method, device and storage medium

Abstract

The present disclosure discloses a policy adjustment method, device, and storage medium. The policy adjustment method includes: obtaining access statistics of a shard; determining whether a policy adjustment condition is hit based on the access statistics; determining a target policy to be adjusted based on the hit policy adjustment condition; the target policy includes: at least one of a sharding policy and a routing policy; the sharding policy is used to control at least the number of shards, shard content, and / or distribution location of the shards; the routing policy is used to determine the routing path of the access request; adjusting the target policy; the target policy is used to control shard access within the cluster.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of network storage technology, and more particularly to a policy adjustment method, device, and storage medium. Background Art

[0002] Distributed storage technology distributes data across different nodes. For example, a single piece of data is split into multiple shards and deployed on nodes in different locations to ensure fast access. High-performance clustered search engine technology offers high availability, scalability, and powerful search capabilities. Combined with distributed storage technology, it can meet the demands of business expansion and explosive data growth. However, with the continued growth of data volumes and the increasing number of access requests, clusters are facing increasingly severe performance and stability challenges. Summary of the Invention

[0003] In view of this, embodiments of the present disclosure provide a policy adjustment method, device, and storage medium.

[0004] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0005] A first aspect of an embodiment of the present disclosure provides a policy adjustment method, comprising: obtaining access statistics of a shard; determining whether a policy adjustment condition is hit based on the access statistics; determining a target policy to be adjusted based on the hit policy adjustment condition; the target policy comprises: at least one of a sharding policy and a routing policy; the sharding policy is at least used to control the number of shards, shard content and / or distribution location of the shard; the routing policy is used to respond to the routing path of an access request; adjusting the target policy; the target policy is used to control shard access within a cluster.

[0006] Based on the above solution, determining whether a policy adjustment condition is hit according to the access statistical data includes: determining the hit policy adjustment condition according to the node load status and shard heat determined according to the access statistical data.

[0007] Based on the above scheme, the policy adjustment conditions that are hit are determined according to the node load status and shard heat determined according to the access statistical data, including at least one of the following: when the node load status indicates that the first node and the second node exist in the cluster and the first shard and the second shard do not exist in the cluster, it is determined that the first adjustment condition is hit; when the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, it is determined that the second adjustment condition or the third adjustment condition is hit; wherein, the difference between the load rate of the first node and the load rate of the second node is greater than the first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold; and / or, the difference between the data volume of the first shard and the data volume of the second shard is greater than the third threshold.

[0008] Based on the above solution, the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold, including: the difference in request volume between the access request volume of the first shard and the access request volume of the second shard is greater than the request volume threshold.

[0009] Based on the above scheme, when the node load condition indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, it is determined that the second adjustment condition or the third adjustment condition is hit, including: when the node load condition indicates that the first node and the second node exist in the cluster and the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold and less than the third threshold, it is determined that the second adjustment condition is hit; when the node load condition indicates that the first node and the second node exist in the cluster and the difference between the heat of the first shard and the heat of the second shard is greater than the third threshold, it is determined that the third adjustment condition is hit.

[0010] Based on the above scheme, the sharding strategy includes at least one of the following: a first sharding strategy, the first sharding strategy is used to migrate the first shard and / or the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or, increase the number of copies of the first shard; the first shard is an already set shard; the second sharding strategy is used to re-divide the data shard to obtain a second shard, and the second shard is used to replace the first shard.

[0011] Based on the above scheme, the target policy to be adjusted is determined according to the hit policy adjustment condition, including: when the access statistics indicate that the first adjustment condition is hit, determining that the target policy includes the routing policy; or, when the access statistics indicate that the second adjustment condition is hit, determining that the target policy includes the shard migration policy and the routing policy; when the access statistics indicate that the third adjustment condition is hit, determining that the target policy includes the shard regeneration policy and the routing policy.

[0012] Based on the above scheme, adjusting the target policy includes at least one of the following: adjusting the routing policy; wherein the adjusted routing policy is used to route the access request routed to the first node to the second node based on the routing policy before adjustment; adjusting the first sharding policy and adjusting the routing policy according to the adjusted first sharding policy; adjusting the second sharding policy and adjusting the routing policy according to the adjusted second sharding policy; wherein the second sharding policy is used to generate a second shard and determine the storage location of the second shard.

[0013] Based on the above solution, the method further includes: after generating the second fragment and adjusting the routing policy based on the second fragment, deleting the first fragment corresponding to the second fragment.

[0014] The second aspect of the embodiments of the present disclosure provides a policy adjustment device, including: an acquisition module for acquiring access statistics of shards; a first determination module for determining whether a policy adjustment condition is hit based on the access statistics; a second determination module for determining a target policy to be adjusted based on the hit policy adjustment condition; the target policy includes: at least one of a sharding strategy and a routing strategy; the sharding strategy is at least used to control the number of shards, shard content and / or distribution location of the shards; the routing strategy is used to respond to the routing path of the access request; an adjustment module for adjusting the target policy; the target policy is used to control shard access within the cluster.

[0015] A third aspect provides a computer storage medium storing computer-readable instructions, which can execute the method described in any technical solution of the first aspect after being executed.

[0016] The fourth aspect provides a computer-readable instruction, characterized in that after the computer-readable instruction is executed, it can execute the method described in any technical solution of the first aspect of claim.

[0017] A policy adjustment method, communication equipment and storage medium provided by the embodiments of the present disclosure will determine the hit adjustment policy based on access statistical data, and adjust the routing policy and / or sharding policy in a targeted manner based on the hit adjustment policy to minimize the problem of load imbalance, reduce system instability, large response delay and other problems caused by load imbalance, and improve system stability and service quality of responding to access requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a policy adjustment method provided in some embodiments of the present disclosure;

[0019] Figure 2 A flowchart of a policy adjustment method provided in some embodiments of the present disclosure;

[0020] Figure 3 A flowchart of a policy adjustment method provided in some embodiments of the present disclosure;

[0021] Figure 4 A schematic diagram of the structure of the strategic adjustment decoration provided in some embodiments of the present disclosure;

[0022] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Through research, we found that in distributed storage clusters, as data expands and grows, data skew and / or node load imbalance problems are likely to occur, becoming key factors affecting cluster performance.

[0024] Data skew primarily refers to the situation in which some shards in a distributed storage cluster carry significantly higher volumes of data or queries than other shards, causing excessive load on the nodes containing these shards, impacting the performance and responsiveness of the entire cluster (e.g., load imbalance between nodes). Load imbalance between data nodes (hereafter referred to as nodes) refers to the situation in which some nodes in a distributed storage cluster (hereafter referred to as the cluster) have significantly higher loads than others. This uneven resource utilization not only impacts overall system performance but can also lead to stability issues for heavily loaded nodes.

[0025] For example, by adopting the hash mapping method, data backups are evenly routed to different shards, and these shards are evenly distributed on each node as much as possible, which can achieve a balanced distribution of data and load to a certain extent. However, in actual applications, due to the unevenness of data access patterns, some nodes may still store more hot data, resulting in the load of these nodes being significantly higher than other nodes. In view of this, shards can be redistributed by restarting the cluster or manually migrating shards in order to temporarily reduce the pressure on high-load nodes; however, restarting the cluster will cause the cluster to restart frequently, affecting the access service provision during the restart period, and manual migration of shards is prone to errors and inefficient.

[0026] In view of this, if Figure 1 As shown, the embodiment of the present disclosure provides a policy adjustment method, which may include:

[0027] S1101: Obtain access statistics of the shard;

[0028] S1102: Determine whether a policy adjustment condition is met based on the access statistics data;

[0029] S1103: Determine a target policy to be adjusted based on the hit policy adjustment condition; the target policy includes: at least one of a sharding policy and a routing policy; the sharding policy is used to control at least the number of shards, shard content, and / or distribution location of the shards; the routing policy is used to respond to the routing path of the access request;

[0030] S1104: Adjust the target policy; the target policy is used to control shard access within the cluster.

[0031] The policy adjustment method may be executed by a search engine or an electronic device connected to the search engine.

[0032] In some embodiments, the number of access requests to each shard and / or the number of visits (queries) to each node are counted based on the search engine log, thereby understanding the load status of each node and / or the popularity of each shard. In some embodiments, the number of visits to each shard in each node and / or the data traffic of each node are counted to obtain access statistics.

[0033] In some embodiments, the access statistics may include but are not limited to at least one of the following:

[0034] The load of each node;

[0035] The load rate of each node;

[0036] The number of visits to the shard;

[0037] The access frequency of the shard, etc.

[0038] In short, based on the access statistics, we can determine the high-load nodes and low-load nodes in the distributed cluster, and / or distinguish the high-heat shards and low-heat shards in the cluster.

[0039] In some embodiments, determining whether a policy adjustment condition is met based on the access statistics data may include at least one of the following:

[0040] When the access statistics show that the load conditions between the nodes meet the load balancing conditions, the miss strategy adjustment conditions can be determined;

[0041] When the access statistics show that the heat between no single shards meets the heat balance condition, the miss strategy adjustment condition can be determined;

[0042] If the access statistics show that the load conditions between nodes do not meet the load balancing conditions, it can be determined that there are conditions for hitting the strategy adjustment;

[0043] When the access statistics indicate that the heat between no single shards does not meet the heat balance condition, it can be determined that there are hit strategy adjustment conditions.

[0044] In some embodiments, if the load condition meets the load balancing condition, it means that the difference in load between nodes in the cluster is within a threshold range, and / or the difference in load rate between nodes in the cluster is within a threshold range.

[0045] In some embodiments, if the heat of a shard meets the heat balance condition, it means that the number of visits to a single shard per unit time is basically the same. For example, if the difference in the number of visits to a single shard per unit time is within a threshold range, it can be considered to meet the heat balance condition.

[0046] Of course, the above are merely examples of load balancing conditions and / or heat balancing conditions, and are not limited to the above examples.

[0047] In the embodiment of the present disclosure, the search engine or the control device will periodically or continuously automatically determine whether the strategy adjustment strategy is hit based on the access statistics, so as to perform adaptive adjustment of the strategy.

[0048] In some embodiments, the aforementioned target policy may be a routing policy and / or a sharding policy. The routing policy may be used to determine the routing path of an access request, thereby determining the node and / or data replica (e.g., shard) that the access request ultimately accesses.

[0049] In some embodiments, the sharding strategy may be used to determine the division of a data shard, for example, the amount of data in a single shard, the number of shards, the storage location of the shards, and / or the content contained in a single shard.

[0050] In some embodiments, to achieve targeted policy adjustments, multiple policy adjustment conditions may be present in the disclosed embodiments. A hit on any of these multiple policy adjustment conditions indicates a need for policy adjustment. In some embodiments, different policy adjustment conditions may correspond to different policies to be adjusted. Thus, policies can be implemented based on the hit policy adjustment condition to better achieve load balancing, improve system stability, and enhance service quality.

[0051] In some embodiments, determining whether a policy adjustment condition is met based on the access statistics includes determining the met policy adjustment condition based on the node load and shard heat determined from the access statistics. In this way, the policy adjustment condition met by the current situation is considered from both the node dimension and the shard dimension.

[0052] In some embodiments, determining a hit policy adjustment condition based on the node load status and shard heat determined from the access statistics data includes at least one of the following:

[0053] When the node load status indicates that the first node and the second node exist in the cluster and the first shard and the second shard do not exist in the cluster, determining that a first adjustment condition is hit;

[0054] When the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determining that the second adjustment condition or the third adjustment condition is hit;

[0055] Among them, the difference between the load rate of the first node and the load rate of the second node is greater than a first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or the difference between the data volume of the first shard and the data volume of the second shard is greater than a third threshold.

[0056] In some embodiments, the load conditions between nodes are used to determine whether the first adjustment condition is met. In some cases, the first adjustment condition may serve as the basis for policy adjustment. In this case, only after the first adjustment condition is met will the shard's popularity be used to determine whether the first adjustment condition alone, the first adjustment condition and the second adjustment condition, or the first adjustment condition and the third adjustment condition, be met.

[0057] In some embodiments, to avoid overloading a single node, the popularity of the shards and / or the amount of data in a single shard are monitored, and the sharding strategy is adjusted in advance based on the popularity of the shards, thereby reducing node overload. After the shard overload is detected, the routing strategy is adaptively adjusted to ultimately achieve system load balancing.

[0058] In some embodiments, if the load rate of the first node is higher than the load rate of the second node, and the difference between the load rates is greater than a first threshold, the first adjustment condition may be met. For example, the difference between the load rates is greater than a specified percentage.

[0059] In some embodiments, if the capacities of multiple nodes are similar, whether the first adjustment condition is met can be determined directly based on the load.

[0060] In some embodiments, the load of any node in the cluster exceeding the overload warning line may be directly determined. Nodes with loads exceeding the overload warning line are designated as first nodes, while other nodes with loads exceeding the overload warning line are designated as second nodes. If the first node exists, the first adjustment condition is considered satisfied.

[0061] The popularity of the first and second shards can be reflected in access popularity. Access popularity is related to access frequency. Due to distributed storage, the higher the access popularity of a piece of data, the more shards are likely to be used to quickly respond to access requests. Therefore, the amount of data sharded can also reflect its popularity.

[0062] When dividing data into shards (shards for short), the shard size can be determined based on the type of data being accessed. For example, it's best to group less-accessed data into a larger shard to reduce fragmentation. However, grouping heavily accessed data into a larger shard can lead to over-access to that shard, which in turn can cause node overload.

[0063] In some embodiments, the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold, including at least one of the following: the difference in request volume between the access request volume of the first shard and the access request volume of the second shard is greater than the request volume threshold.

[0064] By dividing the first shard and the second shard of different popularity according to the difference in the number of access requests, shards of different access popularity can be defined simply and accurately.

[0065] In some embodiments, the sharding strategy includes at least one of the following:

[0066] A first sharding strategy, wherein the first sharding strategy is used to migrate the first shard and / or the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or, to increase the number of replicas of the first shard; the first shard is an already set shard;

[0067] The second sharding strategy is used to re-divide the data shards to obtain second shards, and the second shards are used to replace the first shards.

[0068] The first sharding strategy here is equivalent to adding data replicas to the most popular shards, or migrating the most popular shards. Using this sharding strategy in conjunction with the routing strategy can achieve load balancing between nodes with minimal overhead and without restarting the system.

[0069] The second sharding strategy is equivalent to a new shard generation strategy. In this way, the data contained in a single shard needs to be split, etc., which is a strategy with a larger adjustment range.

[0070] In some embodiments, determining the target policy to be adjusted based on the hit policy adjustment condition includes:

[0071] When the access statistics data indicates that the first adjustment condition is hit, determining that the target policy includes the routing policy; or,

[0072] When the access statistics data indicate that the second adjustment condition is met, determining that the target policy includes the shard migration policy and the routing policy; or

[0073] When the access statistics data indicates that a third adjustment condition is met, it is determined that the target policy includes the shard regeneration policy and the routing policy.

[0074] Based on the different conditions of different hits, the target strategy that needs to be adjusted is determined, and targeted adjustments can be made and / or the most efficient adjustments can be made with minimal overhead.

[0075] Adjusting the target strategy includes at least one of the following:

[0076] Adjusting the routing policy; wherein the adjusted routing policy is used to route the access request that was routed to the first node based on the routing policy before the adjustment to the second node;

[0077] adjusting the first sharding strategy and adjusting the routing strategy according to the adjusted first sharding strategy,

[0078] Adjust the second sharding strategy and adjust the routing strategy according to the adjusted second sharding strategy; wherein the second sharding strategy is used to generate a second shard and determine a storage location of the second shard.

[0079] In some cases, changing the routing policy can avoid significant load imbalances between nodes. This allows you to quickly achieve load balancing and system stability between nodes. In this case, the cluster does not need to be restarted, and there is no service interruption during the restart period.

[0080] For example, after identifying a high-popularity first shard, and determining whether the access popularity of each data replica of the same first shard is the same, if the first shard has a data replica with low access popularity, routing policy optimization can be used to achieve data balance among multiple data distributions within the first shard. After the routing policy is updated, upon receiving a query request, routing matching items can be added between different data replicas of the first shard, so that subsequent access requests to the first shard are transferred to the replica with low access popularity.

[0081] In some embodiments, the routing strategy is changed to achieve load balancing between nodes and system stability through shard migration and / or increase of data copies of shards. Similarly, in this case, the cluster does not need to be restarted, and there will be no service interruption during the restart period; however, in this case, node A with a high load on the shard may be migrated to node B; or, a separate data copy that is frequently accessed in node A is added to node B to enable node B to share the load of node A. Node A here can be an example of the first node mentioned above, and node B can be an example of the second node mentioned above. Both shard migration and increase of data copies require corresponding updates to the routing rules corresponding to these shards in the routing strategy to ultimately achieve load balancing between nodes. However, in this case, if the node of the shard is migrated, the access address will change, and if the shard increases the data copy, the access address of a shard will increase. In this case, the matching items of the routing rules contained in the routing strategy can be changed to achieve load balancing.

[0082] The adjusted first sharding strategy may adjust the distribution position of the existing first shards and / or add data replicas of some first shards. The adjusted first sharding strategy may reduce the heat difference between different first shards to less than or equal to the first threshold.

[0083] In some embodiments, the second sharding strategy is adjusted and the routing strategy is adjusted according to the adjusted second sharding strategy; wherein the second sharding strategy is used to generate a second shard and determine the storage location of the second shard. If the second sharding strategy is adjusted, it is equivalent to re-dividing the shards. Exemplarily, the shards are re-adjusted according to the access popularity of different contents. In this way, the data content of the re-divided second shard may be different from the shard content of the first shard. By re-dividing the shards, the heat between different shards is greatly and efficiently reduced, and / or load balancing between nodes is achieved.

[0084] In some embodiments, the routing policy and / or sharding policy are adjusted online without the need for a cluster restart. After the adjusted policy is generated and put online, the adjusted policy is used and the pre-adjusted policy is deprecated, thereby achieving uninterrupted service.

[0085] In some embodiments, when migrating a first shard, deletion of the first shard on the first node is stopped after the first shard is migrated to the second node. For example, after a data replica of the first shard is added, while the first shard on the first node continues to be accessed, a data replica of the first shard is added to the second node, and after the data replica of the first shard is added to the second node, the routing policy is updated.

[0086] In some embodiments, when the second shard is regenerated, after the second shard is generated and brought online, the corresponding first shard is stopped from being used to accept access or is deleted.

[0087] like Figure 2 As shown, the method further includes:

[0088] S1105: After generating the second fragment and adjusting the routing policy based on the second fragment, delete the first fragment corresponding to the second fragment.

[0089] By deleting the old first shard, corresponding data access errors can be reduced and the storage resources occupied by the old first shard can be reduced.

[0090] When the shard is re-divided to obtain the second shard, it is necessary to obtain the division rule of the second shard and re-divide the second shard according to the obtained division rule. The division rule may be the adjusted second shard strategy.

[0091] In some embodiments, the method further comprises:

[0092] Based on the access statistics obtained through monitoring, the recommended information for the partitioning rules of the second shard is regenerated;

[0093] The division rule is determined according to the user operation on the suggestion information.

[0094] In some embodiments, the suggestion information may be used to provide data information of the N most popular data and suggest separate sharding;

[0095] In some embodiments, the suggestion information can also be used to provide data information of the M data with the lowest popularity, and suggest merging them into a new shard or maintaining them in the current shard.

[0096] In some embodiments, after the second shard determines the storage location, the routing policy may be updated, and the updated routing policy takes effect after the second shard is brought online into the cluster.

[0097] The disclosed embodiments propose a method that dynamically adjusts data and access request routing policies by collecting and analyzing statistical information on data volume and service traffic, thereby achieving uniform data distribution within the cluster and reasonable allocation of query traffic. This approach effectively avoids data skew and node load imbalance, while significantly improving cluster search efficiency and stability, thereby providing high-quality search services and / or data access services. The method may include:

[0098] Step 1: Data volume and traffic statistics. First, we regularly collect and analyze data volume and traffic information for each shard. For example, the collected and statistical data includes metrics such as document count, query count, and query response time, to accurately understand the load on each shard.

[0099] Step 2: Dynamic routing strategy generation. Based on the collected statistical information, the system uses intelligent algorithms to dynamically generate routing strategies. Based on changes in data volume and query traffic, the system adjusts document storage locations and access request routing strategies to achieve load balancing.

[0100] Step 3: Shard redistribution. Under the new routing policy, the system may need to redistribute existing shards to ensure that data and query traffic are evenly distributed across the cluster and to avoid overloading certain nodes or shards.

[0101] Step 4: Intelligent routing of access requests. For access requests entering the system, the system intelligently routes the requests to the most appropriate node for processing based on the current routing strategy, thereby optimizing query response time and improving overall system performance.

[0102] Step 5: Continuous Monitoring and Adaptive Adjustment: The system will continuously monitor changes in data distribution and query traffic, and adaptively adjust routing strategies based on real-time conditions to ensure long-term system performance and stability.

[0103] Through the above technical solution, the disclosed embodiment can effectively solve the data skew and node load imbalance problems in cluster search engines, improving search efficiency and system stability. In addition, the solution has good scalability and adaptability, and can cope with ever-changing data and business needs.

[0104] The technical solution of the disclosed embodiment addresses the data skew and node load imbalance issues in cluster search engines by implementing a custom routing strategy, and has the following advantages:

[0105] High query efficiency: Through intelligent routing strategies, access requests are evenly distributed across nodes, reducing the query response time of high-load nodes, thereby improving overall query efficiency.

[0106] Strong system stability: Dynamically adjust routing strategies and shard distribution to avoid node overload and system bottlenecks, and enhance system stability and reliability.

[0107] High resource utilization: By evenly distributing data and query traffic, optimal resource utilization is achieved, improving the overall performance of the cluster and avoiding resource waste.

[0108] Strong adaptability: The system can automatically adjust routing strategies according to real-time data volume and business traffic changes. It has strong adaptability and flexibility and can cope with changing business needs.

[0109] Low maintenance cost: Compared with traditional manual intervention methods, such as manual migration of shards, the automated solution provided by the embodiments of the present disclosure greatly reduces maintenance costs and workload, and improves operation and maintenance efficiency.

[0110] High user experience: By optimizing query efficiency and system stability, the user experience is ultimately improved, which is particularly important for enterprises that rely on efficient search services.

[0111] In summary, the technical solutions created by the embodiments of the present disclosure not only solve the problems existing in the prior art but also provide additional benefits, and are of great practical value to modern enterprises that need to manage and search massive amounts of documents. In other words, the solutions provided by the embodiments of the present disclosure are particularly suitable for enterprise scenarios.

[0112] The disclosed embodiment uses an efficient data collection and analysis mechanism to monitor the data volume and service flow of each shard and node in real time, providing accurate data support for subsequent routing policy adjustments. The uniqueness and efficiency of this statistical method are a key point of the disclosed embodiment.

[0113] Dynamic routing strategy generation can dynamically generate and adjust routing strategies based on statistical data to achieve a balanced distribution of data and query traffic. Shard redistribution mechanism: In order to achieve load balancing, the embodiment of the present disclosure proposes an efficient shard redistribution mechanism, which can dynamically adjust the distribution of shards without affecting the normal services of the cluster. Intelligent routing processing of access requests: The embodiment of the present disclosure optimizes the distribution of access requests through intelligent routing strategies to ensure that each request can be quickly processed by the most appropriate node. The efficiency and innovation of this access request processing method is a protection point of the embodiment of the present disclosure. The embodiment of the present disclosure also includes a framework for continuous monitoring and adaptive adjustment, which can ensure that the system automatically optimizes and adjusts when facing changes in data and business traffic. The design concept and implementation technology of this framework are an important protection point of the embodiment of the present disclosure.

[0114] In summary, the problem of load imbalance is solved and the performance and stability of the system are improved through dynamic routing strategy generation, shard redistribution, access request processing, continuous monitoring and adaptive adjustment.

[0115] The following is a specific example. When initially creating an index, a sufficiently large number of shards can be pre-set, such as 1000. When writing data for the first time, the number of shards actually used is divided according to the data business, for example, only 10 shards are used. As business traffic changes and data volume increases, the system will dynamically migrate data to new shards and update routing policies, and finally offline and delete the old shards. Specific examples include:

[0116] Initial sharding settings: When creating an index, a larger number of shards is pre-set, such as 1000. When writing data for the first time, the number of shards actually used is divided according to the data business, for example, only 10 shards are used.

[0117] Statistical Information Collection and Evaluation: The system regularly collects statistical information on each shard, including data volume, access request volume, query response time, and node load. Based on this information, the system evaluates the current load and determines whether there is data skew or imbalanced node load.

[0118] Trigger conditions: Resharding and shard migration are triggered only when statistical information indicates a significant imbalance. Specific trigger conditions include, but are not limited to, at least one of the following: a shard's data volume, access request volume, query response time, or node load is significantly higher than the average of other shards or nodes.

[0119] Data service segmentation and request traffic analysis: Detailed segmentation of data services and analysis of request traffic distribution are performed. Using intelligent algorithms, the system can identify which data and request traffic are frequently accessed hot data and which are infrequently accessed cold data.

[0120] Resharding strategy generation: Based on the results of data business division and request traffic analysis, the system generates a new sharding strategy. The new sharding strategy will ensure that data and query traffic are evenly distributed in the cluster, avoiding overloading of certain nodes or shards.

[0121] Shard migration scheduling: Under the new sharding strategy, the system starts the shard migration scheduling process. During the shard migration process, the system will reallocate shards according to the new load balancing strategy to ensure even distribution of data and query traffic across the cluster.

[0122] Routing policy update: After shard migration is complete, the system will update the routing policy based on the new shard distribution. The new routing policy will ensure that access requests are intelligently routed to the most appropriate node for processing, thereby optimizing query response time and overall system performance.

[0123] Deleting and deleting old shards: Once the new sharding and routing policies are running stably, the system will gradually decommission old shards. After ensuring that the data in the old shards has been successfully migrated and is no longer being accessed, the system will delete the old shards to free up resources.

[0124] Data imbalance: Data imbalance occurs when some shards have significantly more data than others. Technical statistics: If the data volume of a shard exceeds twice the average data volume of all shards in the cluster (or some other set threshold), resharding and shard migration are triggered. For example, if there are 10 shards in the cluster, with an average data volume of 100GB per shard, resharding and migration are triggered if the data volume of a shard exceeds 200GB.

[0125] Unbalanced access requests: Unbalanced access requests occurs when the number of access requests to some shards is significantly higher than that to other shards.

[0126] Technical statistics: If the number of access requests to a shard exceeds twice the average number of access requests for all shards in the cluster (or other set thresholds), resharding and shard migration will be triggered. For example, if there are 10 shards in the cluster and the average number of access requests for each shard is 1,000 per minute, if the number of access requests to a shard exceeds 2,000 per minute, adjustment will be triggered.

[0127] Through the above-mentioned orderly process, the embodiment of the present disclosure can trigger re-sharding and shard migration in a timely manner when there is a significant imbalance in data volume and business traffic, thereby achieving load balancing and performance optimization of the cluster.

[0128] Here is a specific example:

[0129] Initial sharding settings: When creating an index, 1,000 shards are pre-set. When writing data for the first time, more than 20 shards are actually used based on subject divisions, with each subject occupying one shard.

[0130] Statistical Information Collection and Evaluation: The system regularly collects statistics on each shard, including data volume, access request volume, query response time, and node load. It was found that the Chinese shard had a data volume of 300GB, while the average data volume for other shards was 50GB. The Chinese shard had an access request volume of 5,000 per minute, while the average for other shards was 1,000 per minute.

[0131] Trigger condition judgment: The data volume and access request volume of the language shard are more than twice that of other shards, triggering re-sharding and shard migration operations.

[0132] Data segmentation and request traffic analysis: The system divides data on the Chinese language subject in detail and finds that the Chinese language sections (e.g., elementary, middle, and high school) have the highest traffic. Using intelligent algorithms, we identify the data for each section as frequently accessed hotspots.

[0133] Resharding strategy generation: Based on the analysis results, the system generates a new sharding strategy, further sharding the Chinese language data by grade level (e.g., elementary, middle, and high school). The new sharding strategy ensures a balanced distribution of data and query traffic across the cluster.

[0134] Shard migration scheduling: The system initiates the shard migration scheduling process, re-sharding the Chinese language data by grade level. For example, data from different grades, such as Chinese language for elementary school, Chinese language for junior high school, and Chinese language for senior high school, will be migrated to new shards.

[0135] Routing policy update: After the shard migration is completed, the system updates the routing policy to ensure that access requests can be intelligently routed to the most appropriate node for processing.

[0136] Old shards are taken offline and deleted: After the new shards and routing strategies are running stably, the system will gradually take the old language shards offline and delete them to free up resources.

[0137] Through the above implementation process, the data and query traffic of the Chinese language subject have been evenly distributed, the query efficiency and stability of the system have been significantly improved, and the user experience has been enhanced.

[0138] Figure 3 An embodiment of the present application is shown, including:

[0139] 1.1: Based on shard routing scheduling, query and search logs.

[0140] 1.2: Calculate the data volume and traffic by searching the log index;

[0141] 1.3: Write data volume / traffic statistics into the statistical data. This statistical data is a type of access statistics mentioned above;

[0142] 1.4: Based on data / traffic statistics, the shard routing map is calculated based on the statistical data. For example, based on the shard routing algorithm, a decision is made whether to adjust the shards. If a shard adjustment is made, the map needs to be updated to update the routing policy.

[0143] 1.5: Automatically or manually determine the resource index. If the shard is adjusted, the resource index needs to be updated.

[0144] 2.1: Obtaining data routing according to search parameters, illustratively, obtaining a search keyword input by a user, and obtaining a routing strategy based on the search keyword;

[0145] 2.2: Perform query mapping using the parameter mapping algorithm. The query here is the aforementioned access request;

[0146] 2.3: Determine whether the mapping object exists, that is, whether the target shard to be searched exists;

[0147] 2.3: If not stored, the sharding is adjusted based on the shard routing algorithm to determine whether sharding is required and then update the mapping. If so, the routing mapping is performed. The routing strategy is used to determine the routing path of the search request, thereby determining the destination shard and destination address of the access request. The destination address can be the address of the node.

[0148] 2.4: Query the specified route, for example, route the access request to the destination node and destination shard, and search the data.

[0149] 2.5: Query resources based on parameters, i.e. routes.

[0150] 2.6: Get the retrieved data based on the resource index and obtain the data list.

[0151] 2.7: Write the search parameters, results, etc. into the search log to facilitate subsequent data volume and / or traffic statistics.

[0152] In some embodiments, the method provided may include sharding strategy adjustment.

[0153] Exemplarily, the method may be used for adjusting the sharding strategy of an Elasticsearch cluster, particularly for optimizing the management of hot shards and cold nodes, including but not limited to one or more of the following steps:

[0154] a. Obtain and analyze shard access statistics to distinguish between hot and cold shards;

[0155] b. Evaluate the load status of nodes in the cluster based on access statistics and distinguish between hot and cold nodes;

[0156] c. Determine the hot shards that need to be migrated and select the target cold node as the migration destination;

[0157] d. Perform migration operations on hot shards to cold nodes to optimize cluster resource usage and improve query efficiency;

[0158] e. Based on the migration results, adjust the routing policy to ensure that access requests to the migrated shards are correctly routed to their new locations.

[0159] In some embodiments, hot shard migration strategies may include, but are not limited to, at least one of the following:

[0160] a. When determining hot shards to migrate, consider one or more of the shard's access frequency, access pattern, and overall load on the node where the shard resides;

[0161] b. When selecting a target cold node, evaluate the resource availability of the target node. The availability evaluation may include, but is not limited to, one or more of the node's storage capacity, memory usage, and CPU load to ensure load balancing after migration.

[0162] In some embodiments, routing policy adjustments may include but are not limited to at least one of the following:

[0163] a. Update the shard routing configuration of the Elasticsearch cluster to reflect the new distribution status after the hot shard migration;

[0164] b. Ensure that the new routing configuration can improve the efficiency of query and indexing operations and reduce the overhead of cross-node communication.

[0165] like Figure 4 As shown, an embodiment of the present disclosure provides a policy adjustment device, including:

[0166] An acquisition module 1101 is used to obtain access statistics of a shard;

[0167] A first determining module 1201 is configured to determine whether a policy adjustment condition is met based on the access statistics data;

[0168] The second determination module 1301 is configured to determine a target policy to be adjusted based on the hit policy adjustment condition; the target policy includes at least one of a sharding policy and a routing policy; the sharding policy is configured to control at least the number of shards, shard content, and / or distribution location of the shards; and the routing policy is configured to respond to a routing path for the access request.

[0169] The adjustment module 1401 is used to adjust the target policy; the target policy is used to control shard access within the cluster.

[0170] In some embodiments, the aforementioned acquisition module, first determination module, second determination module, and adjustment module may correspond to a processor.

[0171] In some embodiments, the first determination module is specifically used to determine the hit policy adjustment condition based on the node load status and shard heat determined by the access statistics data.

[0172] In some embodiments, the second determination module is specifically used to perform at least one of the following: when the node load status indicates that the first node and the second node exist in the cluster and the first shard and the second shard do not exist in the cluster, determine that the first adjustment condition is hit; when the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determine that the second adjustment condition or the third adjustment condition is hit; wherein the difference between the load rate of the first node and the load rate of the second node is greater than the first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold; and / or the difference between the data volume of the first shard and the data volume of the second shard is greater than the third threshold.

[0173] In some embodiments, the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold, including: the difference in request volume between the access request volume of the first shard and the access request volume of the second shard is greater than the request volume threshold.

[0174] In some embodiments, the second determining module is specifically configured to determine that the second adjustment condition is hit when the node load status indicates that a first node and a second node exist in the cluster and a difference between the heat of the first shard and the heat of the second shard is greater than the second threshold and less than a third threshold;

[0175] When the node load status indicates that a first node and a second node exist in the cluster and a difference between the heat of the first shard and the heat of the second shard is greater than the third threshold, it is determined that the third adjustment condition is met.

[0176] In some embodiments, the sharding strategy includes at least one of the following:

[0177] A first sharding strategy, wherein the first sharding strategy is used to migrate the first shard and / or the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or, to increase the number of replicas of the first shard; the first shard is an already set shard;

[0178] The second sharding strategy is used to re-divide the data shards to obtain second shards, and the second shards are used to replace the first shards.

[0179] In some embodiments, the second determination module is used to determine that the target policy includes the routing policy when the access statistics indicate that the first adjustment condition is hit; or, when the access statistics indicate that the second adjustment condition is hit, determine that the target policy includes the shard migration policy and the routing policy; or, when the access statistics indicate that the third adjustment condition is hit, determine that the target policy includes the shard regeneration policy and the routing policy.

[0180] In some embodiments, the adjustment module is used to perform at least one of the following: adjusting the routing policy; wherein the adjusted routing policy is used to route the access request routed to the first node to the second node based on the routing policy before the adjustment; adjusting the first sharding policy and adjusting the routing policy according to the adjusted first sharding policy; adjusting the second sharding policy and adjusting the routing policy according to the adjusted second sharding policy; wherein the second sharding policy is used to generate a second shard and determine the storage location of the second shard.

[0181] In some embodiments, the apparatus further comprises:

[0182] A deletion module is used to delete the first fragment corresponding to the second fragment after generating the second fragment and adjusting the routing policy based on the second fragment.

[0183] Optional, such as Figure 5 As shown, an embodiment of the present disclosure provides an electronic device. The electronic device may be a control device for a search engine or a cluster. Figure 5As shown, the electronic device may include: a processor 501 and a storage 502. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. Here, the program executed by the processor 501 may be stored in a storage 502 connected to the processor 501 via a communication bus 503. The storage 502 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface storage, optical disk, or compact disc read-only memory (CD-ROM); magnetic surface storage can be magnetic disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache.By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The memory 502 described in the embodiments of the present disclosure is intended to include, but is not limited to, these and any other suitable types of memory 502. The memory 502 in the embodiments of the present disclosure is used to store various types of data to support the operation of the processor 501. Examples of such data include: any computer program operated by the processor 501, such as an operating system and application programs; contact data; phone book data; messages; images; videos, etc. The operating system includes various system programs, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and process hardware-based tasks.

[0184] In some embodiments, the memory 502 in the disclosed embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 502 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0185] The processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 501 or by software instructions. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of this disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 502, and the processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method. In some embodiments, the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0186] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0187] Another embodiment of the present disclosure provides a computer storage medium, which stores an executable program. When the executable program is executed by the processor 501, the steps of the policy adjustment method applied to the endpoint device can be implemented. Figures 1 to 2 One or more of the methods shown.

[0188] In some embodiments, the computer storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program codes.

[0189] It should be noted that the technical solutions described in the embodiments of the present disclosure can be arbitrarily combined without conflict.

[0190] The above description is merely a preferred embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure.

Claims

1. A strategy adjustment method, characterized in that: include: Get access statistics of the shard; Determining whether a policy adjustment condition is met based on the access statistics data; Determining whether a policy adjustment condition is hit according to the access statistics data includes at least one of the following: determining that a first adjustment condition is hit when a node load condition indicates that a first node and a second node exist in the cluster and a shard heat indicates that the first shard and the second shard do not exist in the cluster; When the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determining that the second adjustment condition or the third adjustment condition is hit; wherein the difference between the load rate of the first node and the load rate of the second node is greater than a first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; and / or the difference between the data volume of the first shard and the data volume of the second shard is greater than a third threshold; Determining a target policy to be adjusted based on the hit policy adjustment condition; the target policy includes: at least one of a sharding policy and a routing policy; the sharding policy is used to control at least the number of shards, shard content, and / or distribution location of the shards; the routing policy is used to respond to the routing path of the access request; Adjust the target policy; the target policy is used to control shard access within the cluster.

2. The method according to claim 1, wherein The difference between the first shard heat and the second shard heat is greater than a second threshold, including: A request volume difference between the access request volume of the first shard and the access request volume of the second shard is greater than a request volume threshold.

3. The method according to claim 1, wherein When the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determining that the second adjustment condition or the third adjustment condition is hit includes: When the node load status indicates that a first node and a second node exist in the cluster and a difference between the first shard heat and the second shard heat is greater than the second threshold and less than a third threshold, determining that the second adjustment condition is met; When the node load status indicates that the first node and the second node exist in the cluster and the difference between the first shard heat and the second shard heat is greater than the third threshold, it is determined that the third adjustment condition is met.

4. The method according to claim 1, wherein The sharding strategy includes at least one of the following: A first sharding strategy, the first sharding strategy being used to migrate a first shard and / or increase the number of replicas of the first shard when a difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; the first shard being an already configured shard; The second sharding strategy is used to re-divide the data shards to obtain second shards, and the second shards are used to replace the first shards.

5. The method according to claim 4, characterized in that The step of determining the target policy to be adjusted according to the hit policy adjustment condition includes: When the access statistics data indicates that the first adjustment condition is hit, determining that the target policy includes the routing policy; or, When the access statistics data indicate that a second adjustment condition is met, determining that the target policy includes the first sharding policy and the routing policy; or When the access statistics data indicate that a third adjustment condition is met, it is determined that the target policy includes the second sharding policy and the routing policy.

6. The method according to claim 4, characterized in that Adjusting the target strategy includes at least one of the following: Adjusting the routing policy; wherein the adjusted routing policy is used to route the access request that was routed to the first node based on the routing policy before the adjustment to the second node; Adjusting the first sharding strategy and adjusting the routing strategy according to the adjusted first sharding strategy; Adjust the second sharding strategy and adjust the routing strategy according to the adjusted second sharding strategy; wherein the second sharding strategy is used to generate a second shard and determine a storage location of the second shard.

7. The method according to claim 5, characterized in that The method further comprises: After generating the second fragment and adjusting the routing policy based on the second fragment, deleting the first fragment corresponding to the second fragment.

8. A strategy adjustment device, characterized in that: include: The acquisition module is used to obtain access statistics of the shards; a first determining module, configured to determine whether a policy adjustment condition is hit based on the access statistics data; the first determining module is specifically configured to perform at least one of the following: determining that a first adjustment condition is hit when a node load condition indicates that a first node and a second node exist in the cluster and a shard heat condition indicates that a first shard and a second shard do not exist in the cluster; When the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determining that the second adjustment condition or the third adjustment condition is hit; wherein the difference between the load rate of the first node and the load rate of the second node is greater than a first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; and / or the difference between the data volume of the first shard and the data volume of the second shard is greater than a third threshold; A second determination module is configured to determine a target policy to be adjusted based on the hit policy adjustment condition; the target policy includes at least one of a sharding policy and a routing policy; the sharding policy is configured to control at least the number of shards, shard content, and / or distribution location of the shards; and the routing policy is configured to respond to a routing path for the access request. The adjustment module is used to adjust the target policy; the target policy is used to control shard access within the cluster.

9. A computer storage medium, characterized in that The computer storage medium stores computer-readable instructions, and the computer-readable instructions are capable of executing the method according to any one of claims 1 to 7 after being executed by a processor.

10. A computer-readable instruction product, characterized in that After being executed by a processor, the computer-readable instructions can perform the method according to any one of claims 1 to 7.

Images