Strategy adjustment method and device and storage medium

Abstract

The embodiment of the invention discloses a strategy adjustment method and device and a storage medium. The policy adjustment method comprises the following steps: acquiring access statistical data of a fragment; determining whether a strategy adjustment condition is hit according to the access statistical data; determining a to-be-adjusted target strategy according to the hit strategy adjustment condition; the target strategy comprises at least one of a fragmentation strategy and a routing strategy; the fragmentation strategy is at least used for controlling the fragmentation number, the fragmentation content and / or the distribution position of the fragmentation; the routing strategy is used for accessing a routing path of the request; adjusting the target strategy; the target strategy is used for controlling fragment access in the cluster.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of network storage technology, and more particularly to a policy adjustment method, device, and storage medium. Background Art

[0002] Distributed storage technology uses data to be distributed on different nodes. For example, a piece of data is divided into multiple shards and deployed on nodes in different locations to meet the fast access requirements of the corresponding location nodes. At the same time, high-performance clustered search engine technology has high availability, high scalability and powerful search capabilities, and is used in combination with distributed storage technology to meet the needs of business expansion and explosive growth in data volume. However, with the continuous growth of data volume and the continuous increase in access requests, the performance and stability challenges faced by the cluster are becoming increasingly severe. Summary of the invention

[0003] In view of this, embodiments of the present disclosure provide a policy adjustment method, device, and storage medium.

[0004] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0005] A first aspect of an embodiment of the present disclosure provides a policy adjustment method, comprising: obtaining access statistics data of a shard; determining whether a policy adjustment condition is hit based on the access statistics data; determining a target policy to be adjusted based on the hit policy adjustment condition; the target policy comprises: at least one of a sharding policy and a routing policy; the sharding policy is at least used to control the number of shards, shard content and / or distribution location of the shards; the routing policy is used to respond to a routing path of an access request; adjusting the target policy; the target policy is used to control shard access within a cluster.

[0006] Based on the above scheme, determining whether a policy adjustment condition is hit according to the access statistical data includes: determining the hit policy adjustment condition according to the node load status and shard heat determined according to the access statistical data.

[0007] Based on the above scheme, according to the node load status and shard heat determined by the access statistical data, the hit policy adjustment condition is determined, including at least one of the following: when the node load status indicates that the first node and the second node exist in the cluster and the first shard and the second shard do not exist in the cluster, it is determined that the first adjustment condition is hit; when the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, it is determined that the second adjustment condition or the third adjustment condition is hit; wherein the difference between the load rate of the first node and the load rate of the second node is greater than the first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold; and / or the difference between the data volume of the first shard and the data volume of the second shard is greater than the third threshold.

[0008] Based on the above scheme, the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold, including: the difference in request volume between the access request volume of the first shard and the access request volume of the second shard is greater than the request volume threshold.

[0009] Based on the above scheme, when the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, it is determined that the second adjustment condition or the third adjustment condition is hit, including: when the node load status indicates that the first node and the second node exist in the cluster and the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold and less than the third threshold, it is determined that the second adjustment condition is hit; when the node load status indicates that the first node and the second node exist in the cluster and the difference between the heat of the first shard and the heat of the second shard is greater than the third threshold, it is determined that the third adjustment condition is hit.

[0010] Based on the above scheme, the sharding strategy includes at least one of the following: a first sharding strategy, the first sharding strategy is used to migrate the first shard and / or the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or, increase the number of copies of the first shard; the first shard is an already set shard; a second sharding strategy, the second sharding strategy is used to re-divide the data shards to obtain a second shard, and the second shard is used to replace the first shard.

[0011] Based on the above scheme, the target policy to be adjusted is determined according to the hit policy adjustment condition, including: when the access statistics data indicate that the first adjustment condition is hit, determining that the target policy includes the routing policy; or, when the access statistics data indicate that the second adjustment condition is hit, determining that the target policy includes the shard migration policy and the routing policy; when the access statistics data indicate that the third adjustment condition is hit, determining that the target policy includes the shard rebirth policy and the routing policy.

[0012] Based on the above scheme, adjusting the target policy includes at least one of the following: adjusting the routing policy; wherein the adjusted routing policy is used to route the access request routed to the first node to the second node based on the routing policy before the adjustment; adjusting the first sharding policy and adjusting the routing policy according to the adjusted first sharding policy; adjusting the second sharding policy and adjusting the routing policy according to the adjusted second sharding policy; wherein the second sharding policy is used to generate a second shard and determine the storage location of the second shard.

[0013] Based on the above solution, the method further includes: after generating the second fragment and adjusting the routing policy based on the second fragment, deleting the first fragment corresponding to the second fragment.

[0014] The second aspect of the disclosed embodiment provides a policy adjustment device, including: an acquisition module, used to acquire access statistics of shards; a first determination module, used to determine whether a policy adjustment condition is hit based on the access statistics; a second determination module, used to determine a target policy to be adjusted based on the hit policy adjustment condition; the target policy includes: at least one of a sharding strategy and a routing strategy; the sharding strategy is at least used to control the number of shards, shard content and / or distribution location of the shards; the routing strategy is used to respond to the routing path of an access request; an adjustment module, used to adjust the target policy; the target policy is used to control shard access within a cluster.

[0015] A third aspect provides a computer storage medium storing computer-readable instructions, which can execute the method described in any technical solution of the first aspect after being executed.

[0016] The fourth aspect provides a computer-readable instruction, characterized in that after the computer-readable instruction is executed, it can execute the method described in any technical solution of the first aspect of claim.

[0017] A policy adjustment method, communication device and storage medium provided by the embodiments of the present disclosure will determine the hit adjustment policy based on access statistical data, and adjust the routing policy and / or sharding policy in a targeted manner based on the hit adjustment policy to minimize the problem of load imbalance, reduce system instability and large response delay caused by load imbalance, and improve system stability and service quality in responding to access requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a strategy adjustment method provided in some embodiments of the present disclosure;

[0019] Figure 2 A flowchart of a strategy adjustment method provided in some embodiments of the present disclosure;

[0020] Figure 3 A flowchart of a strategy adjustment method provided in some embodiments of the present disclosure;

[0021] Figure 4 A schematic diagram of a structure of a strategic adjustment decoration provided by some embodiments of the present disclosure;

[0022] Figure 5 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Through research, it is found that in a distributed storage cluster, as data expands and grows, data skew and / or node load imbalance problems are prone to occur, becoming a key factor affecting cluster performance.

[0024] Data skew mainly refers to: in a distributed storage cluster, the amount of data or query volume carried by some shards is significantly higher than that of other shards, resulting in excessive load on the nodes where these shards are located, thus affecting the performance of the entire cluster (for example, load imbalance between nodes) and response speed. Load imbalance between data nodes (hereinafter referred to as nodes) means: the load of some nodes in a distributed storage cluster (hereinafter referred to as cluster) is much higher than that of other nodes. This unbalanced resource utilization not only affects the overall performance of the system, but may also cause stability problems for high-load nodes.

[0025] For example, by adopting the hash mapping method, the data backup is evenly routed to different shards, and these shards are evenly distributed on each node as much as possible, which can achieve balanced distribution of data and load to a certain extent. However, in actual applications, due to the unevenness of data access patterns, some nodes may still store more hot data, resulting in the load of these nodes being significantly higher than other nodes. In view of this, shards can be redistributed by restarting the cluster or manually migrating shards in order to temporarily reduce the pressure on high-load nodes; however, restarting the cluster frequently affects the access service provision during the restart period, and manual migration of shards is prone to errors and inefficient.

[0026] In view of this, if Figure 1 As shown, the embodiment of the present disclosure provides a strategy adjustment method, which may include:

[0027] S1101: Obtain access statistics of the shards;

[0028] S1102: Determine whether a policy adjustment condition is met according to the access statistical data;

[0029] S1103: Determine a target policy to be adjusted according to the hit policy adjustment condition; the target policy includes: at least one of a sharding policy and a routing policy; the sharding policy is at least used to control the number of shards, shard content and / or distribution location of the shards; the routing policy is used to respond to the routing path of the access request;

[0030] S1104: Adjust the target policy; the target policy is used to control shard access within the cluster.

[0031] The policy adjustment method may be executed by a search engine or an electronic device connected to the search engine.

[0032] In some embodiments, the access request volume of each shard and / or the access volume (query volume) of each node are counted according to the log of the search engine, so as to know the load status of each node and / or the popularity of each shard. In some embodiments, the access volume of each shard in each node and / or the data flow of each node are counted, so as to obtain access statistics.

[0033] In some embodiments, the access statistics may include but are not limited to at least one of the following:

[0034] The load of each node;

[0035] The load rate of each node;

[0036] The number of visits to the shard;

[0037] The access frequency of the shard, etc.

[0038] In short, based on the access statistics, high-load nodes and low-load nodes in the distributed cluster can be determined, and / or high-heat shards and low-heat shards in the cluster can be distinguished.

[0039] In some embodiments, determining whether a policy adjustment condition is hit according to the access statistics data may include at least one of the following:

[0040] When the access statistics show that the load conditions between the nodes meet the load balancing conditions, the miss strategy adjustment conditions can be determined;

[0041] When the access statistics show that the heat between no single shards meets the heat balance condition, the miss strategy adjustment condition can be determined;

[0042] When the access statistics show that the load conditions between the nodes do not meet the load balancing conditions, it can be determined that there are hit strategy adjustment conditions;

[0043] When the access statistics show that the heat between no single shards does not meet the heat balance condition, it can be determined that there is a hit strategy adjustment condition.

[0044] In some embodiments, if the load condition meets the load balancing condition, it means that the difference in load between nodes in the cluster is within a threshold range, and / or the difference in load rate between nodes in the cluster is within a threshold range.

[0045] In some embodiments, if the heat of a shard meets a heat balance condition, it means that the number of visits to a single shard per unit time is basically the same. For example, if the difference in the number of visits to a single shard per unit time is within a threshold range, it can be considered to meet the heat balance condition.

[0046] Of course, the above are merely examples of load balancing conditions and / or heat balancing conditions, and are not limited to the above examples.

[0047] In the disclosed embodiment, the search engine or control device will periodically or continuously automatically determine whether to hit the strategy adjustment strategy based on the access statistics data, so as to perform adaptive adjustment of the strategy.

[0048] In some embodiments, the aforementioned target policy may be a routing policy and / or a sharding policy. The routing policy may be used to determine the routing path of an access request, thereby determining the node and / or data replica (eg, shard) that the access request ultimately accesses.

[0049] In some embodiments, the sharding strategy may be used to determine the division of a data shard, for example, the amount of data in a single shard, the number of shards, the storage location of the shards, and / or the content contained in a single shard.

[0050] In some embodiments, in order to achieve targeted policy adjustment, in the embodiments of the present disclosure, there may be multiple policy adjustment conditions, and if any one of the multiple policy adjustment conditions is hit, it means that the policy needs to be adjusted. In some embodiments, different policy adjustment conditions may correspond to different policies to be adjusted, so that corresponding policies can be implemented according to the hit policy adjustment conditions to better achieve load balancing, improve system stability and service quality.

[0051] In some embodiments, determining whether a policy adjustment condition is hit according to the access statistics data includes: determining the hit policy adjustment condition according to the node load status and the shard heat determined by the access statistics data. In this way, the policy adjustment condition hit by the current status is considered from the node dimension and the shard dimension respectively.

[0052] In some embodiments, determining the hit policy adjustment condition according to the node load status and the shard heat determined by the access statistics data includes at least one of the following:

[0053] When the node load status indicates that the first node and the second node exist in the cluster and the first shard and the second shard do not exist in the cluster, determining that a first adjustment condition is hit;

[0054] When the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determining that the second adjustment condition or the third adjustment condition is hit;

[0055] Among them, the difference between the load rate of the first node and the load rate of the second node is greater than a first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or the difference between the data volume of the first shard and the data volume of the second shard is greater than a third threshold.

[0056] In some embodiments, whether the first adjustment condition is met is determined based on the load conditions between nodes. In some cases, the first adjustment condition may be the basic condition for policy adjustment. In this case, it is equivalent to further determining whether the first adjustment condition is met alone, or the first adjustment condition and the second adjustment condition are met at the same time, or the first adjustment condition and the third adjustment condition are met at the same time, only when the first adjustment condition is met.

[0057] In some embodiments, in order to avoid overloading a single node, the heat of the shard and / or the amount of data in a single shard is monitored, and the sharding strategy is adjusted in advance according to the heat of the shard, thereby reducing the overload of the node. After the shard is overloaded, the routing strategy is adaptively adjusted to ultimately achieve load balancing of the system.

[0058] In some embodiments, if the load rate of the first node is higher than the load rate of the second node and the difference between the load rates is greater than a first threshold, it can be considered that the first adjustment condition is met. For example, the difference between the load rates is greater than a specified percentage.

[0059] In some embodiments, if the capacities of multiple nodes are similar, whether the first adjustment condition is satisfied can be determined directly based on the load.

[0060] In some embodiments, it is also possible to directly determine whether there is a node in the cluster whose load exceeds the overload alarm line, and the node whose load exceeds the overload alarm line is the first node, and other nodes whose load exceeds the overload alarm line are the second nodes. If the first node exists, it can be considered that the first adjustment condition is met.

[0061] The popularity of the first shard and the second shard can be reflected by the access popularity. Access popularity is related to access frequency. Due to distributed storage, the higher the access popularity of a piece of data, the more shards are likely to be used to quickly respond to access requests. Therefore, the amount of data that is sharded can also reflect the popularity.

[0062] When dividing data into shards (shards for short), the size of the shards can be determined according to the shards of the corresponding type of data. For example, it is better to divide data with low access volume into a larger shard, so as to reduce the phenomenon of fragmented shards. However, dividing data with high access volume into a larger shard may cause the access heat of the shard to be too high, which in turn causes the node heat to be too high.

[0063] In some embodiments, the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold, including at least one of the following: the difference in request volume between the access request volume of the first shard and the access request volume of the second shard is greater than a request volume threshold.

[0064] By dividing the first shard and the second shard of different popularity according to the difference in the number of access requests, the shards of different access popularity can be simply and accurately defined.

[0065] In some embodiments, the sharding strategy includes at least one of the following:

[0066] A first sharding strategy, wherein the first sharding strategy is used to migrate the first shard and / or the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or, to increase the number of replicas of the first shard; the first shard is a set shard;

[0067] The second sharding strategy is used to re-divide the data shards to obtain second shards, and the second shards are used to replace the first shards.

[0068] Here, the first sharding strategy is equivalent to the strategy of adding data replicas of the hot shards, or the migration strategy of the hot shards. Using this sharding strategy and the routing strategy can achieve load balancing between nodes with the smallest possible overhead and without restarting the system.

[0069] The second sharding strategy is equivalent to a new sharding generation strategy. In this case, the data contained in a single shard needs to be split, etc. This is a strategy with a larger adjustment range.

[0070] In some embodiments, determining the target policy to be adjusted according to the hit policy adjustment condition includes:

[0071] When the access statistics data indicate that the first adjustment condition is hit, determining that the target policy includes the routing policy; or,

[0072] When the access statistics data indicate that the second adjustment condition is hit, determining that the target policy includes the shard migration policy and the routing policy; or,

[0073] When the access statistics data indicate that the third adjustment condition is hit, it is determined that the target policy includes the shard regeneration policy and the routing policy.

[0074] According to different conditions of different hits, the target strategy that needs to be adjusted is determined, and targeted adjustments can be made and / or the most efficient adjustments can be made with minimal overhead.

[0075] Adjusting the target strategy includes at least one of the following:

[0076] Adjusting the routing policy; wherein the adjusted routing policy is used to route the access request routed to the first node to the second node based on the routing policy before the adjustment;

[0077] adjusting the first sharding strategy and adjusting the routing strategy according to the adjusted first sharding strategy,

[0078] The second sharding strategy is adjusted and the routing strategy is adjusted according to the adjusted second sharding strategy; wherein the second sharding strategy is used to generate a second shard and determine a storage location of the second shard.

[0079] In some cases, changing the routing policy can avoid the problem of large load imbalance between nodes. You can quickly achieve load balancing between nodes and system stability by modifying the routing policy. In this case, the cluster does not need to be restarted, and there is no service interruption during the restart period.

[0080] Exemplarily, after determining the high-heat first shard, and determining whether the access heat of each data copy of the same first shard is the same, if the first shard has a data copy with low access heat, the data balance between multiple data distributions in the first shard can be achieved through routing strategy optimization. After the routing strategy is updated, after receiving a query request, the routing matching items between different data copies of the first shard can be added, so that subsequent access requests in the first shard are transferred to the copy with low heat.

[0081] In some embodiments, the routing strategy is changed to achieve load balancing between nodes and system stability through shard migration and / or data copy addition of shards. Similarly, in this case, the cluster does not need to be restarted, and there is no service interruption during the restart period; however, in this case, there may be a high-loaded node A in the shard that is migrated to node B; or, a separate data copy that is frequently accessed in node A is added to node B to achieve node B sharing the load of node A. Node A here can be an example of the first node mentioned above, and node B can be an example of the second node mentioned above. Both shard migration and data copy addition require corresponding updates to the routing rules corresponding to these shards in the routing strategy to ultimately achieve load balancing between nodes. However, in this case, if the node of the shard is migrated, the access address will change. If the shard adds a data copy, the access address of a shard will increase. In this case, the matching items of the routing rules contained in the routing strategy can be changed to achieve load balancing.

[0082] The adjusted first sharding strategy may adjust the distribution position of the existing first shards and / or increase the data replicas of some first shards. The adjusted first sharding strategy may reduce the heat difference between different first shards to less than or equal to the first threshold.

[0083] In some embodiments, the second sharding strategy is adjusted and the routing strategy is adjusted according to the adjusted second sharding strategy; wherein the second sharding strategy is used to generate a second shard and determine the storage location of the second shard. If the second sharding strategy is adjusted, it is equivalent to re-dividing the shards. Exemplarily, the shards are re-adjusted according to the access heat of different contents. In this way, the data content of the re-divided second shard may be different from the shard content of the first shard. By re-dividing the shards, the heat between different shards is greatly and efficiently reduced, and / or load balancing between nodes is achieved.

[0084] In some embodiments, the routing strategy and / or the sharding strategy are adjusted online, and the cluster does not need to be restarted. After the adjusted strategy is generated and put online, the adjusted strategy is used and the pre-adjusted strategy is abolished, thereby achieving uninterrupted service.

[0085] In some embodiments, when performing migration of the first shard, after the first shard is migrated to the second node, the deletion of the first shard on the first node is stopped. For example, after the data copy of the first shard is added, while the first shard on the first node continues to be accessed, the data copy of the first shard is added to the second node, and after the data copy of the first shard is added to the second node, the routing policy is updated.

[0086] In some embodiments, when the second shard is regenerated, after the second shard is generated and brought online, the corresponding first shard is stopped from being used to accept access or is deleted.

[0087] like Figure 2 As shown, the method also includes:

[0088] S1105: After generating the second fragment and adjusting the routing policy based on the second fragment, delete the first fragment corresponding to the second fragment.

[0089] By deleting the old first shard, corresponding data access errors can be reduced and storage resources occupied by the old first shard can be reduced.

[0090] When the shard is re-divided to obtain the second shard, it is necessary to obtain the division rule of the second shard, and re-divide the second shard according to the obtained division rule. The division rule may be the adjusted second shard strategy.

[0091] In some embodiments, the method further comprises:

[0092] According to the access statistics information obtained through monitoring, the recommended information of the division rule of the second shard is regenerated;

[0093] The division rule is determined according to the user operation acting on the suggestion information.

[0094] In some embodiments, the suggestion information may be used to provide data information of the N most popular data and suggest separate sharding;

[0095] In some embodiments, the suggestion information can also be used to provide data information of the M data with the lowest heat, and suggest merging them into a new shard or maintaining the current shard.

[0096] In some embodiments, after the second shard determines the storage location, the routing policy may be updated, and after the second shard is brought online to the cluster, the updated routing policy takes effect.

[0097] The disclosed embodiment proposes a method that can dynamically adjust the routing strategy of data and access requests by collecting and analyzing statistical information on data volume and business traffic, so as to achieve uniform distribution of data in the cluster and reasonable distribution of query traffic. In this way, on the one hand, data skew and node load imbalance problems can be effectively avoided, and the search efficiency and stability of the cluster can be significantly improved to provide high-quality search services and / or data access services. The method may include:

[0098] Step 1: Data volume and business traffic statistics. First, the data volume and business traffic information on each shard will be collected and counted regularly. For example, the collected and counted data include indicators such as the number of documents, number of queries, and query response time, so as to accurately understand the load of each shard.

[0099] Step 2: Dynamic routing strategy generation. Based on the collected statistical information, the system will use intelligent algorithms to dynamically generate routing strategies. According to the changes in data volume and query traffic, the storage location of documents and the routing strategy of access requests are adjusted to achieve load balancing.

[0100] Step 3: Shard redistribution. Under the new routing strategy, the system may need to redistribute existing shards to ensure that data and query traffic are evenly distributed in the cluster and to avoid overloading certain nodes or shards.

[0101] Step 4: Intelligent routing of access requests. For access requests entering the system, according to the current routing strategy, intelligently route the request to the most appropriate node for processing to optimize query response time and improve the overall system performance.

[0102] Step 5: Continuous monitoring and adaptive adjustment. The system will continuously monitor changes in data distribution and query traffic, and adaptively adjust routing strategies based on real-time conditions to ensure long-term system performance and stability.

[0103] Through the above technical solution, the disclosed embodiment can effectively solve the data skew and node load imbalance problems in the cluster search engine, improve search efficiency and system stability. In addition, the solution also has good scalability and adaptability, and can cope with ever-changing data and business needs.

[0104] The technical solution of the embodiment of the present disclosure implements a custom routing strategy to address the data skew and node load imbalance problems in a cluster search engine, and has the following advantages:

[0105] High query efficiency: Through intelligent routing strategies, access requests are evenly distributed across nodes, reducing the query response time of high-load nodes, thereby improving overall query efficiency.

[0106] Strong system stability: Dynamically adjust routing strategies and shard distribution to avoid node overload and system bottlenecks, and enhance system stability and reliability.

[0107] High resource utilization: By evenly distributing data and query traffic, optimal resource utilization is achieved, the overall performance of the cluster is improved, and resource waste is avoided.

[0108] Strong adaptability: The system can automatically adjust routing strategies according to real-time data volume and business traffic changes. It has strong adaptability and flexibility and can cope with ever-changing business needs.

[0109] Low maintenance cost: Compared with traditional manual intervention methods, such as manual migration of shards, the automated solution provided by the embodiments of the present disclosure greatly reduces maintenance costs and workload, and improves operation and maintenance efficiency.

[0110] High user experience: By optimizing query efficiency and system stability, the user experience is ultimately improved, which is especially important for companies that rely on efficient search services.

[0111] In summary, the technical solution created by the embodiments of the present disclosure not only solves the problems existing in the prior art, but also provides additional beneficial effects, and has important practical value for modern enterprises that need to manage and search massive documents. That is, the solution provided by the embodiments of the present disclosure is particularly suitable for enterprise scenarios.

[0112] The embodiment of the present disclosure monitors the data volume and service flow of each shard and node in real time through an efficient data collection and analysis mechanism, and provides accurate data support for subsequent routing strategy adjustment. The uniqueness and efficiency of this statistical method are a key point of the embodiment of the present disclosure.

[0113] Dynamic routing strategy generation can dynamically generate and adjust routing strategies based on statistical data to achieve balanced distribution of data and query traffic. Shard redistribution mechanism: In order to achieve load balancing, the embodiment of the present disclosure proposes an efficient shard redistribution mechanism, which can dynamically adjust the distribution of shards without affecting the normal service of the cluster. Intelligent routing processing of access requests: The embodiment of the present disclosure optimizes the distribution of access requests through intelligent routing strategies to ensure that each request can be quickly processed by the most appropriate node. The efficiency and innovation of this access request processing method is a protection point of the embodiment of the present disclosure. The embodiment of the present disclosure also includes a framework for continuous monitoring and adaptive adjustment, which can ensure that the system automatically optimizes and adjusts when facing changes in data and business traffic. The design concept and implementation technology of the framework are an important protection point of the embodiment of the present disclosure.

[0114] In summary, the problem of load imbalance is solved and the performance and stability of the system are improved through dynamic routing strategy generation, shard redistribution, access request processing, continuous monitoring and adaptive adjustment.

[0115] The following is a specific example. When the index is initially created, a sufficiently large number of shards can be pre-set, such as 1,000. When data is first written, the number of shards actually used is divided according to the data business, for example, only 10 shards are used. As the business traffic changes and the amount of data increases, the system will dynamically migrate data to new shards and update the routing strategy, and finally go offline and delete the old shards. Specifically, it may include:

[0116] Initial shard setting: When creating an index, a larger number of shards is pre-set, such as 1000. When writing data for the first time, the number of shards actually used is divided according to the data business, for example, only 10 shards are used.

[0117] Statistical information collection and evaluation: The system regularly collects statistical information such as the amount of data, access request volume, query response time, node load, etc. of each shard. Based on the collected statistical information, the system evaluates the current load situation and determines whether there is a problem of data skew or node load imbalance.

[0118] Trigger condition judgment: Resharding and shard migration operations will only be triggered when statistical information indicates a significant imbalance. Specific trigger conditions include but are not limited to at least one of the following: the data volume, access request volume, query response time or node load of a shard is significantly higher than the average of other shards or nodes.

[0119] Data service division and request traffic analysis: Detailed division of data services and analysis of the distribution of request traffic. Through intelligent algorithms, the system can identify which data and request traffic are frequently accessed hot data and which are infrequently accessed cold data.

[0120] Re-sharding strategy generation: Based on the results of data business division and request traffic analysis, the system generates a new sharding strategy. The new sharding strategy will ensure that data and query traffic are evenly distributed in the cluster and avoid overloading certain nodes or shards.

[0121] Shard migration scheduling: Under the new sharding strategy, the system starts the shard migration scheduling process. During the shard migration process, the system will reallocate the shards according to the new load balancing strategy to ensure that data and query traffic are evenly distributed in the cluster.

[0122] Routing policy update: After the shard migration is completed, the system will update the routing policy based on the new shard distribution. The new routing policy will ensure that access requests can be intelligently routed to the most appropriate node for processing, thereby optimizing query response time and overall system performance.

[0123] Deleting and deleting old shards: After the new shards and routing policies are running stably, the system will gradually deleting old shards. After ensuring that the data in the old shards has been successfully migrated and is no longer accessed, the system will delete these old shards to free up resources.

[0124] Data volume imbalance: The condition for data volume imbalance is that the data volume of some shards is significantly higher than that of other shards. Technical statistics: If the data volume of a shard exceeds twice the average data volume of all shards in the cluster (or other set thresholds), resharding and shard migration are triggered. For example, assuming there are 10 shards in the cluster and the average data volume of each shard is 100GB, if the data volume of a shard exceeds 200GB, adjustment is triggered.

[0125] Unbalanced access requests: Unbalanced access requests occur when the number of access requests to some shards is significantly higher than that to other shards.

[0126] Technical statistics: If the number of access requests to a shard exceeds twice the average number of access requests to all shards in the cluster (or other set thresholds), resharding and shard migration are triggered. For example, assuming there are 10 shards in the cluster and the average number of access requests to each shard is 1,000 times / minute, if the number of access requests to a shard exceeds 2,000 times / minute, adjustment is triggered.

[0127] Through the above-mentioned orderly process, the embodiment of the present disclosure can trigger re-sharding and shard migration in a timely manner when there is a significant imbalance in data volume and business traffic, thereby achieving load balancing and performance optimization of the cluster.

[0128] Here is a specific example:

[0129] Initial shard setting: When creating an index, 1,000 shards are preset. When writing data for the first time, more than 20 shards are actually used based on subject division, with each subject occupying one shard.

[0130] Statistical information collection and evaluation: The system regularly collects statistical information such as data volume, access request volume, query response time, node load, etc. for each shard. It was found that the data volume of the Chinese shard reached 300GB, while the average data volume of other shards was 50GB; the access request volume of the Chinese shard was 5,000 times / minute, while the average of other shards was 1,000 times / minute.

[0131] Trigger condition judgment: The data volume and access request volume of the language shard are more than twice that of other shards, triggering re-sharding and shard migration operations.

[0132] Data business division and request traffic analysis: The system divides the data of Chinese subjects in detail and finds that the access volume of Chinese subjects in each stage (such as elementary school, junior high school, and high school) is relatively high. Through intelligent algorithms, the data of Chinese subjects in each stage are identified as hot data with high frequency of access.

[0133] Re-sharding strategy generation: Based on the analysis results, the system generates a new sharding strategy to further shard the Chinese data by grade level (such as elementary school, junior high school, and high school). The new sharding strategy ensures balanced distribution of data and query traffic in the cluster.

[0134] Shard migration scheduling: The system starts the shard migration scheduling process and re-shards the Chinese data by grade. For example, data from different grades such as Chinese-primary school, Chinese-junior high school, and Chinese-senior high school are migrated to new shards.

[0135] Routing policy update: After the shard migration is completed, the system updates the routing policy to ensure that access requests can be intelligently routed to the most appropriate node for processing.

[0136] Old shards are taken offline and deleted: After the new shards and routing strategies are running stably, the system gradually takes old language shards offline and deletes them to release resources.

[0137] Through the above implementation process, the data and query traffic of the Chinese language subject have been evenly distributed, the query efficiency and stability of the system have been significantly improved, and the user experience has been enhanced.

[0138] Figure 3 An embodiment of the present application is shown, comprising:

[0139] 1.1: Based on shard routing scheduling, query and search logs.

[0140] 1.2: Calculate the data volume and traffic by searching the log index;

[0141] 1.3: Write the data volume / traffic statistics into the statistical data. This statistical data is a type of the aforementioned access statistical data;

[0142] 1.4: Based on data / traffic statistics, calculate the shard routing mapping based on the statistical data. For example, based on the shard routing algorithm, determine whether to adjust the shard. If the shard is adjusted, the mapping needs to be updated to update the routing policy.

[0143] 1.5: Automatically or manually determine the resource index. If the shard is adjusted, the resource index needs to be updated.

[0144] 2.1: Obtaining data routing according to search parameters, illustratively, obtaining search keywords input by the user, and obtaining routing strategies based on the search keywords, etc.;

[0145] 2.2: Perform query mapping through parameter mapping algorithm. The query here is the access request mentioned above;

[0146] 2.3: Determine whether a mapping object exists, that is, whether the target shard to be searched exists;

[0147] 2.3: If not stored, perform shard adjustment based on the shard routing algorithm to determine whether to perform shard adjustment and enter the update mapping; if it exists, perform routing mapping. Determine the routing path of the search request based on the routing strategy, thereby determining the destination shard and destination address of the access request. The destination address can be the address of the node.

[0148] 2.4: Query the specified route, for example, route the access request to the destination node and destination shard, and perform data search.

[0149] 2.5: Query resources based on parameters, i.e. routes.

[0150] 2.6: Get the retrieved data based on the resource index and obtain the data list.

[0151] 2.7: Write the search parameters, results, etc. into the search log to facilitate subsequent data volume and / or traffic statistics.

[0152] The method provided in some embodiments may include sharding strategy adjustment.

[0153] Exemplarily, the method may be used for a sharding strategy adjustment method of an Elasticsearch cluster, particularly for optimizing management of hot shards and cold nodes, including but not limited to one or more of the following steps:

[0154] a. Obtain and analyze shard access statistics to distinguish between hot and cold shards;

[0155] b. Evaluate the load status of nodes in the cluster based on access statistics and distinguish between hot nodes and cold nodes;

[0156] c. Determine the hot shards that need to be migrated and select the target cold node as the migration destination;

[0157] d. Perform migration operations from hot shards to cold nodes to optimize cluster resource usage and improve query efficiency;

[0158] e. Based on the migration results, adjust the routing policy to ensure that access requests to the migrated shards can be correctly routed to their new locations.

[0159] In some embodiments, the hot shard migration strategy may include but is not limited to at least one of the following:

[0160] a. When determining the hot shards to be migrated, consider one or more of the shard access frequency, access pattern, and overall load status of the node where the relevant shard is located;

[0161] b. When selecting a target cold node, evaluate the resource availability of the target node, which may include but is not limited to one or more of the node's storage capacity, memory usage, and CPU load to ensure load balancing after migration.

[0162] In some embodiments, routing policy adjustment may include but is not limited to at least one of the following:

[0163] a. Update the shard routing configuration of the Elasticsearch cluster to reflect the new distribution status after the hot shard migration;

[0164] b. Ensure that the new routing configuration can improve the efficiency of query and index operations and reduce the overhead of cross-node communication.

[0165] like Figure 4 As shown, the embodiment of the present disclosure provides a strategy adjustment device, including:

[0166] An acquisition module 1101 is used to acquire access statistics of a shard;

[0167] A first determination module 1201 is used to determine whether a policy adjustment condition is hit according to the access statistics data;

[0168] The second determination module 1301 is used to determine the target policy to be adjusted according to the hit policy adjustment condition; the target policy includes: at least one of a sharding policy and a routing policy; the sharding policy is at least used to control the number of shards, shard content and / or distribution location of the shards; the routing policy is used to respond to the routing path of the access request;

[0169] The adjustment module 1401 is used to adjust the target policy; the target policy is used to control the shard access within the cluster.

[0170] In some embodiments, the aforementioned acquisition module, first determination module, second determination module, and adjustment module may correspond to a processor.

[0171] In some embodiments, the first determination module is specifically used to determine the hit policy adjustment condition based on the node load status and shard heat determined by the access statistics data.

[0172] In some embodiments, the second determination module is specifically used to perform at least one of the following: when the node load status indicates that the first node and the second node exist in the cluster and the first shard and the second shard do not exist in the cluster, determine that the first adjustment condition is hit; when the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determine that the second adjustment condition or the third adjustment condition is hit; wherein the difference between the load rate of the first node and the load rate of the second node is greater than a first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; and / or the difference between the data volume of the first shard and the data volume of the second shard is greater than a third threshold.

[0173] In some embodiments, the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold, including: the difference in request volume between the access request volume of the first shard and the access request volume of the second shard is greater than the request volume threshold.

[0174] In some embodiments, the second determination module is specifically configured to determine that the second adjustment condition is hit when the node load status indicates that the first node and the second node exist in the cluster and the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold and less than a third threshold;

[0175] When the node load status indicates that the first node and the second node exist in the cluster and the difference between the heat of the first shard and the heat of the second shard is greater than the third threshold, it is determined that the third adjustment condition is hit.

[0176] In some embodiments, the sharding strategy includes at least one of the following:

[0177] A first sharding strategy, wherein the first sharding strategy is used to migrate the first shard and / or the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or, to increase the number of replicas of the first shard; the first shard is a set shard;

[0178] The second sharding strategy is used to re-divide the data shards to obtain second shards, and the second shards are used to replace the first shards.

[0179] In some embodiments, the second determination module is used to determine that the target policy includes the routing policy when the access statistics indicate that the first adjustment condition is hit; or, when the access statistics indicate that the second adjustment condition is hit, determine that the target policy includes the shard migration policy and the routing policy; or, when the access statistics indicate that the third adjustment condition is hit, determine that the target policy includes the shard rebirth policy and the routing policy.

[0180] In some embodiments, the adjustment module is used to perform at least one of the following: adjusting the routing policy; wherein the adjusted routing policy is used to route the access request routed to the first node to the second node based on the routing policy before the adjustment; adjusting the first sharding policy and adjusting the routing policy according to the adjusted first sharding policy; adjusting the second sharding policy and adjusting the routing policy according to the adjusted second sharding policy; wherein the second sharding policy is used to generate a second shard and determine the storage location of the second shard.

[0181] In some embodiments, the apparatus further comprises:

[0182] A deletion module is used to delete a first fragment corresponding to the second fragment after generating the second fragment and adjusting the routing policy based on the second fragment.

[0183] Optional, such as Figure 5 As shown, an embodiment of the present disclosure provides an electronic device. The electronic device may be a control device of a search engine or a cluster. Figure 5As shown, the electronic device may include: a processor 501 and a storage 502. The processor 501 may be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Here, the program executed by the processor 501 may be stored in a storage 502 connected to the processor 501 via a communication bus 503. The storage 502 may be a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache.By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory 502 described in the embodiments of the present disclosure is intended to include but is not limited to these and any other suitable types of memory 502. The memory 502 in the embodiments of the present disclosure is used to store various types of data to support the operation of the processor 501. Examples of such data include: any computer program operated by the processor 501, such as an operating system and application programs; contact data; phone book data; messages; pictures; videos, etc. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks.

[0184] In some embodiments, the memory 502 in the disclosed embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 502 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0185] The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 501. The above processor 501 may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware. In some embodiments, the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0186] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0187] Another embodiment of the present disclosure provides a computer storage medium, which stores an executable program. When the executable program is executed by the processor 501, the steps of the policy adjustment method applied to the endpoint device can be implemented. Figure 1 to Figure 2 One or more of the methods shown.

[0188] In some embodiments, the computer storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0189] It should be noted that the technical solutions described in the embodiments of the present disclosure can be combined arbitrarily without conflict.

[0190] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure.

Claims

1. A strategy adjustment method, characterized in that: include: Get access statistics of the shards; Determining whether a policy adjustment condition is met according to the access statistics data; According to the hit policy adjustment condition, determine the target policy to be adjusted; the target policy includes: at least one of a sharding policy and a routing policy; the sharding policy is at least used to control the number of shards, shard content and / or distribution location of the shards; the routing policy is used to respond to the routing path of the access request; The target policy is adjusted; the target policy is used to control shard access within the cluster.

2. The method according to claim 1, characterized in that Determining whether a policy adjustment condition is hit according to the access statistical data includes: determining the hit policy adjustment condition according to the node load status and shard heat determined according to the access statistical data.

3. The method according to claim 2, characterized in that Determine the hit policy adjustment condition according to the node load status and the shard heat determined by the access statistics data, including at least one of the following: When the node load status indicates that the first node and the second node exist in the cluster and the first shard and the second shard do not exist in the cluster, determining that a first adjustment condition is hit; When the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determining that the second adjustment condition or the third adjustment condition is hit; Among them, the difference between the load rate of the first node and the load rate of the second node is greater than a first threshold; the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; and / or the difference between the data volume of the first shard and the data volume of the second shard is greater than a third threshold.

4. The method according to claim 3, wherein: The difference between the heat of the first shard and the heat of the second shard is greater than a second threshold, including: A request quantity difference between the access request quantity of the first shard and the access request quantity of the second shard is greater than a request quantity threshold.

5. The method according to claim 3, wherein: When the node load status indicates that the first node and the second node exist in the cluster and the shard heat indicates that the first shard and the second shard exist in the cluster, determining that the second adjustment condition or the third adjustment condition is hit includes: When the node load status indicates that the first node and the second node exist in the cluster and the difference between the heat of the first shard and the heat of the second shard is greater than the second threshold and less than a third threshold, determining that the second adjustment condition is hit; When the node load status indicates that the first node and the second node exist in the cluster and the difference between the heat of the first shard and the heat of the second shard is greater than the third threshold, it is determined that the third adjustment condition is hit.

6. The method according to claim 2 or 3, wherein: The sharding strategy includes at least one of the following: A first sharding strategy, wherein the first sharding strategy is used to migrate the first shard and / or the difference between the heat of the first shard and the heat of the second shard is greater than a second threshold; or, to increase the number of replicas of the first shard; the first shard is a set shard; The second sharding strategy is used to re-divide the data shards to obtain second shards, and the second shards are used to replace the first shards.

7. The method according to claim 6, characterized in that The step of determining the target strategy to be adjusted according to the hit strategy adjustment condition includes: When the access statistics data indicate that the first adjustment condition is hit, determining that the target policy includes the routing policy; or, When the access statistics data indicate that the second adjustment condition is hit, determining that the target policy includes the shard migration policy and the routing policy; or, When the access statistics data indicate that the third adjustment condition is hit, it is determined that the target policy includes the shard regeneration policy and the routing policy.

8. The method according to claim 6, characterized in that Adjusting the target strategy includes at least one of the following: Adjusting the routing policy; wherein the adjusted routing policy is used to route the access request routed to the first node to the second node based on the routing policy before the adjustment; Adjusting the first sharding strategy and adjusting the routing strategy according to the adjusted first sharding strategy; The second sharding strategy is adjusted and the routing strategy is adjusted according to the adjusted second sharding strategy; wherein the second sharding strategy is used to generate a second shard and determine a storage location of the second shard.

9. The method according to claim 8, characterized in that The method further comprises: After generating the second fragment and adjusting the routing policy based on the second fragment, deleting the first fragment corresponding to the second fragment.

10. A strategy adjustment device, characterized in that: include: The acquisition module is used to obtain the access statistics of the shards; A first determination module, used to determine whether a policy adjustment condition is hit according to the access statistical data; The second determination module is used to determine the target policy to be adjusted according to the hit policy adjustment condition; the target policy includes: at least one of a sharding policy and a routing policy; the sharding policy is used to at least control the number of shards, shard content and / or distribution location of the shards; the routing policy is used to respond to the routing path of the access request; The adjustment module is used to adjust the target strategy; the target strategy is used to control the shard access within the cluster.

11. A computer storage medium, characterized in that: The computer storage medium stores computer-readable instructions, and the computer-readable instructions can execute the method according to any one of claims 1 to 9 after being executed.

12. A computer-readable instruction, characterized in that: After being executed, the computer readable instructions can perform the method according to any one of claims 1 to 9.

Images