System optimization apparatus, method, equipment and medium based on distributed retrieval system

Abstract

This application discloses a system optimization device, method, equipment, and medium based on a distributed retrieval system, relating to the field of computer technology. The device includes: a skew balancing strategy module for setting monitoring rules, monitoring scope, and monitoring index thresholds for the current distributed retrieval cluster; a dynamic balancing monitoring module for monitoring the skewness of the distributed retrieval cluster according to the monitoring rules, monitoring scope, and monitoring index thresholds; and a skew balancing management module for dynamically balancing and partitioning / layering management of the resource distribution and request distribution of the distributed retrieval cluster based on cluster monitoring information. This application achieves balanced optimization of the current distributed retrieval cluster by adding a skew monitoring mechanism, a skew balancing management mechanism, and a request partitioning / layering control mechanism to the sharded storage and retrieval process of the distributed retrieval system, maintaining a balance between resources and requests, thereby avoiding request anomalies and resource waste caused by skewness.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a system optimization device, method, equipment, and medium based on a distributed retrieval system. Background Technology

[0002] In current distributed retrieval systems, the significant independent control each retrieval dataset possesses, particularly in the flexibility to control the size of each index shard and its data replicas, as well as the location distribution of shards, can lead to shard distribution and request distribution issues within the entire distributed retrieval cluster. For example, data storage capacity skew occurs, meaning data is consistently distributed to a minority of nodes in the cluster; similarly, request skew occurs, where requests consistently fall to a minority of nodes. This results in unbalanced performance utilization and resource waste. Especially when the data storage capacity of the entire distributed retrieval cluster reaches a critical level, a large number of abnormal retrieval requests may occur, reducing the performance utilization of the distributed retrieval system and leading to cluster request anomalies and resource waste. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a system optimization device, method, equipment, and medium based on a distributed retrieval system, which can achieve balanced optimization of the current distributed retrieval cluster, maintain the balance between resources and requests, and thus avoid the problems of request anomalies and resource waste caused by skew. The specific solution is as follows:

[0004] In a first aspect, this application discloses a system optimization device based on a distributed retrieval system, comprising:

[0005] The tilt balancing strategy module is used to set the monitoring rules, monitoring scope, and monitoring indicator thresholds for the current distributed retrieval cluster;

[0006] The dynamic balancing monitoring module is used to monitor the tilt of each retrieval node in the distributed retrieval cluster according to the monitoring rules, monitoring range and monitoring index threshold set by the tilt balancing strategy module to obtain corresponding cluster monitoring information.

[0007] The tilt balance management module is used to perform dynamic balance management of the resource distribution of the distributed retrieval cluster based on the cluster monitoring information reported by the dynamic balance monitoring module, and to perform partitioned and hierarchical management of the distribution of requests of the distributed retrieval cluster based on the cluster monitoring information to achieve balance optimization of the current distributed retrieval cluster.

[0008] Optionally, the system optimization device based on the distributed retrieval system further includes:

[0009] The cluster state protection module is used to circuit break the retrieval service of the retrieval node when a large number of requests in the distributed retrieval cluster are abnormal, and to transfer subsequent requests to other retrieval nodes to protect the stable operation of the distributed retrieval cluster.

[0010] Optionally, the tilt balancing strategy module runs on the main retrieval node of the distributed retrieval cluster, the dynamic balancing monitoring module runs on the data storage retrieval node of the distributed retrieval cluster, and the tilt balancing management module runs on both the data storage retrieval node and the main retrieval node of the distributed retrieval cluster.

[0011] Optionally, the dynamic equilibrium monitoring module is specifically used for:

[0012] Based on the monitoring rules and monitoring scope set by the tilt balancing strategy module, the corresponding node information is obtained by monitoring each retrieval node in the distributed retrieval cluster from the perspective of sharded storage and request distribution.

[0013] Based on the monitoring index thresholds set by the tilt balance strategy module, alarm analysis is performed on the relevant index data in the node information to obtain corresponding alarm analysis information;

[0014] The node information and alarm analysis information are summarized to obtain the corresponding cluster monitoring information.

[0015] Optionally, the tilt balance management module includes:

[0016] The data skew control unit is used to determine the data skew of each retrieval node in the current distributed retrieval cluster based on the cluster monitoring information, and to perform dynamic balance management of the resource distribution of the distributed retrieval cluster based on the data skew, until resource balance is achieved.

[0017] The request layering control unit is used to determine the request skewness of each retrieval node in the current distributed retrieval cluster based on the cluster monitoring information, and to perform partitioned and layered management of the request distribution of the distributed retrieval cluster based on the request skewness until request balance is achieved.

[0018] Optionally, the data tilt control unit is specifically used for:

[0019] Based on the data skew, the retrieval nodes with abnormal resource storage differences in the distributed retrieval cluster are identified;

[0020] Data balancing and receiving operations are performed on the resource distribution of the retrieval nodes.

[0021] Optionally, the request hierarchical control unit is specifically used for:

[0022] The historical and current requests of each retrieval node in the cluster monitoring information are summarized and statistically analyzed to determine the request frequency distribution of each retrieval node;

[0023] Based on the request frequency distribution, each retrieval node is partitioned into hot, warm, and cold zones to obtain hot, warm, and cold zones; the hot, warm, and cold zones represent nodes with different request activity levels.

[0024] For a busy hot zone node, the location of the shard replica corresponding to the hot zone node is retrieved, and the request is transferred to the cold zone node or the warm zone node corresponding to that location.

[0025] Secondly, this application discloses a system optimization method based on a distributed retrieval system, applied to the aforementioned system optimization device based on a distributed retrieval system, comprising:

[0026] Configure the monitoring rules, monitoring scope, and monitoring indicator thresholds for the current distributed retrieval cluster;

[0027] Based on the monitoring rules, the monitoring scope, and the monitoring index thresholds, the tilt of each retrieval node in the distributed retrieval cluster is monitored to obtain corresponding cluster monitoring information;

[0028] Based on the cluster monitoring information, the resource distribution of the distributed retrieval cluster is dynamically balanced and managed, and the distribution of requests in the distributed retrieval cluster is partitioned and layered based on the cluster monitoring information to achieve balanced optimization of the current distributed retrieval cluster.

[0029] Thirdly, this application discloses an electronic device, including:

[0030] Memory, used to store computer programs;

[0031] A processor is used to execute the computer program to implement the steps of the aforementioned system optimization method based on a distributed retrieval system.

[0032] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned system optimization method based on a distributed retrieval system.

[0033] As can be seen, this application provides a system optimization device based on a distributed retrieval system, comprising: a skew balancing strategy module, used to set monitoring rules, monitoring scope, and monitoring indicator thresholds for the current distributed retrieval cluster; a dynamic balancing monitoring module, used to monitor the skewness of each retrieval node in the distributed retrieval cluster according to the monitoring rules, monitoring scope, and monitoring indicator thresholds set by the skew balancing strategy module to obtain corresponding cluster monitoring information; and a skew balancing management module, used to perform dynamic balancing management of the resource distribution of the distributed retrieval cluster according to the cluster monitoring information reported by the dynamic balancing monitoring module, and to perform partitioned and hierarchical management of the request distribution of the distributed retrieval cluster according to the cluster monitoring information to achieve balanced optimization of the current distributed retrieval cluster. Therefore, this application, by adding a skew monitoring mechanism, a skew balancing management mechanism, and a request partitioned and hierarchical control mechanism to the sharded storage and retrieval process of the distributed retrieval system, can achieve balanced optimization of the current distributed retrieval cluster, maintain the balance of resources and requests, and thus avoid the problems of request anomalies and resource waste caused by skewness. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of a system optimization device based on a distributed retrieval system disclosed in this application;

[0036] Figure 2 This is a schematic diagram of a specific system optimization device based on a distributed retrieval system disclosed in this application;

[0037] Figure 3 This is a flowchart of a system optimization method based on a distributed retrieval system disclosed in this application;

[0038] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Currently, in the operation of distributed retrieval systems, the significant independent control each retrieval dataset possesses, particularly in the flexibility regarding the size of each index shard and its data replicas, as well as the location distribution of shards, can easily lead to shard distribution and request distribution issues within the entire distributed retrieval cluster. For example, data storage capacity skew occurs, meaning data is consistently distributed among a minority of nodes in the cluster; similarly, request skew occurs, where requests consistently fall to a minority of nodes. This results in unbalanced performance utilization and resource waste. Especially when the data storage capacity of the entire distributed retrieval cluster reaches a critical level, a large number of abnormal retrieval requests may occur, reducing the performance utilization of the distributed retrieval system and leading to cluster request anomalies and resource waste. To address this, this application provides a system optimization system based on distributed retrieval systems that can achieve balanced optimization of the current distributed retrieval cluster, maintaining a balance between resources and requests, thereby avoiding the request anomalies and resource waste caused by skew.

[0041] It should be noted that a distributed retrieval system is a fast retrieval system framework. Its working mode means that it can distribute its data processing work across all component nodes of the entire retrieval system, ensuring that each component node can participate in the data storage and read / write request work. This ensures that the entire retrieval system can operate in a balanced, fast, and efficient manner, and quickly respond to retrieval requests issued by clients.

[0042] This application also discloses a system optimization device based on a distributed retrieval system, see [link to relevant documentation]. Figure 1 As shown, the device includes:

[0043] The tilt balance strategy module 11 is used to set the monitoring rules, monitoring scope and monitoring indicator thresholds for the current distributed retrieval cluster.

[0044] It is understood that the tilt balancing strategy module 11 runs on the main retrieval node of the distributed retrieval cluster, and the tilt balancing strategy module 11 is open to users, that is, it is used to provide users with a custom strategy configuration entry, which is used to set the monitoring indicator thresholds, monitoring rules, monitoring ranges, etc. of various indicators of the current distributed retrieval cluster, and is also used to configure the baseline strategy when managing shard balance. In the balancing strategy of this embodiment, all the threshold settings involved have system default thresholds, and when users have special cluster settings for the distributed retrieval cluster or have user-defined requirements, they can also automatically set and take effect through the configuration window.

[0045] It should be noted that some special role nodes, such as coordinator nodes and master nodes, are mainly used to receive and distribute requests and implement cluster-level management. These nodes do not need to be included in subsequent load balancing management. Ordinary data nodes are the main players in data storage and retrieval request response. However, due to hardware configuration and cluster partitioning mechanisms, different data nodes have different capacity thresholds. Therefore, relevant parameters or thresholds need to be configured in the skew load balancing strategy module 11. This means configuring specific dynamic data load balancing thresholds, request load balancing transfer thresholds, etc., based on the actual cluster situation to provide a basis for load balancing. In other words, the skew load balancing strategy module is mainly responsible for providing the configuration entry point for custom monitoring and decision parameters, which serve as the triggering basis for actions in different modules. Reasonable parameter definitions can ensure the cluster maintains optimal load balancing efficiency, and custom-defined parameters have higher priority than default parameters.

[0046] For example, the monitoring rules can be set such that the monitoring reporting frequency is set to 1 minute / time, the statistical period for the average node request is 30 minutes or 1 hour, and the acquisition period for the peak node request is 30 minutes, that is, the number of peak times is counted within 30 minutes; the monitoring scope can be set such that the monitoring scope includes segment cache, shard storage, request distribution, etc.; the detailed settings of the monitoring indicator thresholds can be set such that the peak query threshold can be set to 1000, the average query threshold can be set to 800, the peak shard storage capacity threshold can be set to 75%, the average shard storage capacity threshold can be set to 70%, the peak segment memory usage threshold can be set to 80%, and the average segment memory usage threshold can be set to 70%. When alarms occur frequently, the dynamic balancing monitoring module 12 can directly skip the 1 minute / time reporting frequency and directly enable circuit breaker reporting, triggering the skew balancing management module 13 to handle cluster shard balancing.

[0047] The dynamic equilibrium monitoring module 12 is used to monitor the tilt of each retrieval node in the distributed retrieval cluster according to the monitoring rules, monitoring range and monitoring index threshold set by the tilt equilibrium strategy module to obtain corresponding cluster monitoring information.

[0048] It is understood that the dynamic balancing monitoring module 12 runs on the data storage and retrieval nodes of the distributed retrieval cluster. The dynamic balancing monitoring module monitors the overall resource and request situation to collect the decision information required by the tilt balancing management module, and provides triggering and analysis conditions for the tilt balancing management module. That is, from the dimensions of sharded storage and request distribution, the tilt situation of each retrieval node in the distributed retrieval cluster is monitored according to the monitoring rules, monitoring scope and monitoring index thresholds set by the tilt balancing strategy module 11 to obtain the corresponding cluster monitoring information.

[0049] As a specific embodiment, the dynamic balance monitoring module 12 is specifically used to monitor each retrieval node in the distributed retrieval cluster from the perspective of sharded storage and request distribution, according to the monitoring rules and monitoring range set by the tilt balance strategy module 11, to obtain corresponding node information; to perform alarm analysis on the relevant indicator data in the node information based on the monitoring indicator thresholds set by the tilt balance strategy module 11, to obtain corresponding alarm analysis information; and to summarize the node information and the alarm analysis information to obtain corresponding cluster monitoring information. It can be understood that the dynamic balance monitoring module 12 mainly monitors the tilt of each retrieval node in the distributed retrieval cluster from the dimensions of sharded storage and request distribution, obtaining cluster monitoring information including shard segment cache, shard data capacity, node data capacity, average node request and peak node request over the past 30 minutes, and current node request status. Then, the cluster monitoring information is summarized to the tilt balance management module 13 for dynamic balance analysis, thereby performing balanced management of node data and request distribution.

[0050] It is understood that the dynamic balancing monitoring module 12 runs on the data storage and retrieval nodes of the distributed retrieval cluster. From the perspective of sharded storage and request distribution, according to the monitoring rules and monitoring scope set by the skew balancing strategy module 11, it dynamically monitors the sharded storage, request distribution, segment caching, etc. of each retrieval node. It also performs alarm analysis on the monitoring information in combination with various indicator thresholds defined by the balancing strategy to obtain performance and resource usage issues of the monitored indicators in a timely manner. Then, it summarizes the historical monitoring status and alarm analysis information of sharded storage, request distribution, segment caching, etc. This information will be used as the basis for analysis and processing of shard balancing operations, and will be encapsulated and passed to the shard balancing module 13 for operation and processing.

[0051] For example, the dynamic balancing monitoring module 12 reads and parses the monitoring rules and monitoring scope defined by the tilt balancing strategy module 11, namely, the monitoring reporting frequency is 1 minute / time, the statistical period for the average node request is 30 minutes or 1 hour, the acquisition period for the peak node request is 30 minutes, the monitoring scope includes segment cache / shard storage / request distribution, the query peak threshold is set to 1000, the query average threshold is set to 800, the peak threshold for shard storage capacity is set to 75%, the average threshold for shard storage capacity is set to 70%, the peak threshold for segment memory usage is set to 80%, and the average threshold for segment memory usage is set to 70%, etc., and then monitors... The system collects the average resource consumption of the distributed retrieval cluster over the past 30 minutes, such as the average query value, average shard storage capacity, and average segment memory usage of node 1 over the past 30 minutes. If the average resource consumption exceeds the threshold defined by the skew balancing strategy module 11, it is directly marked as an average limit exceeding alarm. The system also monitors the peak query value, peak shard storage capacity, and peak segment memory usage of the distributed retrieval cluster over the past 30 minutes. If any peak value exceeds the threshold defined by the skew balancing strategy module 11, it is directly marked as a peak value exceeding alarm. Finally, the system summarizes all node monitoring information and alarm analysis information, encapsulates it, and reports it to the skew balancing management module 13.

[0052] The tilt balance management module 13 is used to perform dynamic balance management of the resource distribution of the distributed retrieval cluster according to the cluster monitoring information reported by the dynamic balance monitoring module, and to perform partitioned and hierarchical management of the distribution of requests of the distributed retrieval cluster according to the cluster monitoring information to achieve balance optimization of the current distributed retrieval cluster.

[0053] It is understood that the tilt balance management module 13 runs on the data storage retrieval nodes and the main retrieval node of the distributed retrieval cluster. It receives cluster monitoring information sent by the dynamic balance monitoring module 12 to perceive the real-time balance status of the distributed retrieval cluster, and performs dynamic balance management on the resource distribution of the distributed retrieval cluster and partitioned and hierarchical management on the distribution of requests to the distributed retrieval cluster. For example, it receives cluster monitoring information transmitted by the dynamic balance monitoring module 12, parses the node information in the cluster monitoring information, such as node 1's average query value of 900, peak query value of 1100, average shard storage capacity of 75%, peak shard storage capacity of 85%, average segment memory usage of 80%, peak segment memory usage of 90%, and alarm information such as the average query value and average segment memory usage exceeding thresholds. Then, based on the cluster monitoring information received and parsed, the node storage status is analyzed. For example, if the average shard storage capacity of node 1 reaches 75% in the past 30 minutes, the peak shard storage capacity exceeds 85%, or the peak segment memory usage reaches 90% and the average segment memory usage reaches 80%, and the average and peak values ​​both exceed the warning threshold, and an over-limit alarm is received directly from the dynamic balance monitoring module 12, it is confirmed that there is a risk of resource imbalance in the distributed retrieval cluster. Then, the shard migration action is initiated until the migration is completed and balance is achieved.

[0054] As a specific embodiment, the skew balance management module 13 includes a data skew control unit and a request layering control unit. The data skew control unit is used to determine the data skew status of each retrieval node in the current distributed retrieval cluster based on the cluster monitoring information, and to perform dynamic skew balance management on the resource distribution of the distributed retrieval cluster based on the data skew status until resource balance is achieved. The request layering control unit is used to determine the request skew status of each retrieval node in the current distributed retrieval cluster based on the cluster monitoring information, and to perform partitioned and layered management on the request distribution of the distributed retrieval cluster based on the request skew status until request balance is achieved.

[0055] As a specific embodiment, the data skew control unit is specifically used to determine the retrieval nodes with abnormal resource storage differences in the distributed retrieval cluster based on the data skew situation; and to perform data balancing transfer and reception operations on the resource distribution of the retrieval nodes. It can be understood that the data skew control unit performs balancing management on nodes with abnormal resource storage differences in the cluster, that is, it performs priority comparison migration on nodes with large data balance differences. For example, based on the dynamic balance monitoring module 12 monitoring the distribution of shards in the distributed retrieval cluster, dynamic balance management is performed on the data, that is, data balancing transfer and reception is performed on nodes with the same average as the abnormal nodes. Specifically, data migration is performed between the node with the most data storage and the node with the least data storage in the distributed retrieval cluster, and between the node with the second most data storage and the node with the second least data storage, and so on for other nodes. It should be noted that during the data balancing transfer process, nodes with higher dynamic sharding are prioritized for transfer, and it is also necessary to evaluate whether the new balance after the data balancing transfer achieves data balance between the transferring node and the receiving node, so as to avoid new balance problems caused by the migration of large shards.

[0056] As a specific embodiment, the request hierarchical control unit is specifically used to: summarize and statistically analyze the historical and current requests of each retrieval node in the cluster monitoring information to determine the request frequency distribution of each retrieval node; perform hot-warm-cold partitioning management on each retrieval node based on the request frequency distribution to obtain hot zone nodes, warm zone nodes, and cold zone nodes after partitioning; the hot zone nodes, warm zone nodes, and cold zone nodes represent nodes with different request busy levels; for the hot zone nodes with busy requests, the location of the shard replica corresponding to the hot zone node is retrieved, and the request is transferred to the cold zone node or the warm zone node corresponding to the location. It can be understood that in this embodiment, node requests are hierarchically managed, and node requests are redistributed to nodes with different busy levels to optimize the cluster request load. For example, based on the cluster monitoring information transmitted by the dynamic balance monitoring module 12, the historical and current requests of each node in the distributed retrieval cluster are summarized and statistically analyzed to obtain the request frequency distribution of each retrieval node. Then, based on the combined request frequency distribution of all nodes, each retrieval node is divided into three layers: cold, warm, and hot. Subsequent requests are actively allocated. That is, for hot nodes with heavy requests, the location of their corresponding shard replicas is retrieved. If the replica's location is a cold or warm node, the request is transferred to that cold or warm node for execution, avoiding a large number of abnormal requests from busy nodes. For example, the query distribution of all nodes transmitted by the dynamic balance monitoring module 12 is summarized and statistically analyzed. All nodes are managed as cold, warm, and hot zones according to the dynamic query average. When an abnormal request frequency is detected on a node, such as node 1's average query value reaching 1000 and its peak query value reaching 1200, a partition request transfer is triggered. A shard replica is found in the cold or warm zone, and the requests from nodes reaching the peak value are transferred, prioritizing the transfer to cold zones to achieve a new request balance and avoid request congestion caused by excessive requests from some nodes.

[0057] In one specific embodiment, the tilt balance management module 13 is also used to parse the request sending and response status in the cluster monitoring information. If a large number of abnormal failed requests occur, the subsequent forwarded requests need to be migrated in stages. For example, 50% of the requests are migrated initially. If there are still a large number of abnormal failed requests, the remaining 50% are migrated until the requests are normal. Then, other requests that exceed the threshold are rebalanced to the node.

[0058] Understandably, the tilt balance management module 13 is mainly used to coordinate the monitoring information and alarm information triggered by the dynamic balance monitoring module 12, execute tilt management actions according to the actual scenario problems encountered by the retrieval cluster, dynamically and quickly handle the current cluster resource tilt and request tilt problems, flexibly redistribute the data and request distribution of nodes, optimize resource utilization and protect the stable operation of the cluster.

[0059] Furthermore, as a preferred embodiment, the above-mentioned system optimization device based on the distributed retrieval system further includes:

[0060] The cluster state protection module is used to circuit break the retrieval service of the retrieval node when a large number of requests in the distributed retrieval cluster are abnormal, and to transfer subsequent requests to other retrieval nodes to protect the stable operation of the distributed retrieval cluster.

[0061] Understandably, shard migration and request transfer consume resources and have latency characteristics, which may lead to a large backlog of failed requests. In severe cases, this can cause the request queue to be blocked, and requests on overloaded nodes may fail completely. Therefore, cluster state protection needs to be added. That is, after skew balancing is triggered, shards and requests will not be immediately redistributed. Therefore, request anomaly handling needs to be performed on hot nodes that are under close monitoring. If a large number of request anomalies occur on hot nodes, the retrieval service of the node needs to be immediately circuit-broken, and subsequent requests should be forcibly transferred to cold or warm nodes for processing to avoid the hot nodes being overloaded and causing anomalies, thus ensuring the stable operation of the retrieval service and business. For example, the skew balancing strategy module 11 sets a default request circuit breaker threshold of 80%. This means that when a node experiences an 80% query request failure rate and is detected as having triggered data migration or request transfer, the node's request processing service is stopped. Subsequent requests entering that shard are forwarded at the same level and forcibly transferred to cold and warm zone replicas of the request partition for processing. Cold zone replicas have higher priority than warm zone replicas. Once the node's data migration or request transfer is complete, the request task distribution to the circuit breaker service node is restarted at 10-minute retry intervals. If the request service is processed normally, the node officially comes online and resumes request processing service. In other words, cluster state protection is mainly used to perform circuit breaker protection operations in extreme situations encountered by nodes, buying time and space for troubleshooting and protecting the cluster's operational security.

[0062] For example, such as Figure 2As shown, dynamic load balancing monitoring provides basic cluster detection and analysis information. For example, it dynamically monitors the sharding storage, request distribution, and segment caching of each retrieval node in the cluster, analyzes the resource status of the basic scenario, and summarizes and outputs historical monitoring information and alarm analysis information such as sharding storage, request distribution, and segment caching. The load balancing strategy provides users with a parameter window that can be configured for the current scenario environment. It forms the source data of the load balancing strategy from the scenario itself and defines the most reasonable load balancing strategy rules. The load balancing management module provides strategy operations for actual resource balancing and request balancing protection. That is, based on the monitoring data, it evaluates and balances the resource distribution and manages the distribution of requests in layers, so that the retrieval cluster can quickly and effectively optimize the load balancing, solve the load balancing problem, and protect the healthy and stable operation of the cluster. Then, it is combined with the cluster state protection module to maintain the stable and secure operation of the retrieval cluster. That is, the cluster state protection module provides a request circuit breaker mechanism in addition to load balancing management to ensure timely response to the transfer node request under extreme node operation conditions, avoid retrieval request blocking and abnormal cluster load caused by a large number of abnormal requests, and thus ensure the safe operation of the cluster. In other words, in routine business scenarios where the cluster load is high, data request imbalance occurs, and retrieval performance degrades abnormally, the distributed retrieval system monitors and schedules data skew and request skew scenarios. Through a series of skew balancing management operations, the retrieval service is rebalanced in terms of data and requests, and partitioned and layered management is implemented to optimize retrieval requests. Dynamic balancing monitoring can effectively monitor and report cluster node and resource skew and request skew and abnormal alarms. Skew balancing strategies can be defined with custom parameters based on actual cluster needs and scenarios, providing thresholds for balanced processing of cluster resources and redistribution of requests. Skew balancing management performs retrieval sharding based on cluster resource skew and partitions and layers requests for management, optimizing the cluster's resource balance and request balance to protect the cluster's stable operation. Cluster state protection provides circuit breaker protection for extreme abnormal scenarios, ensuring performance stability and service security when the cluster is not operating normally under abnormal conditions. This effectively optimizes the data distribution and retrieval performance distribution of the distributed retrieval cluster, making the retrieval service more stable, reliable, and secure.

[0063] Therefore, in this embodiment of the application, by adding a skew monitoring mechanism, a skew balance management mechanism, and a request partitioning and hierarchical control mechanism to the sharded storage and retrieval process of the distributed retrieval system, the current distributed retrieval cluster can be balanced and optimized, maintaining the balance of resources and requests, thereby avoiding the problems of request anomalies and resource waste caused by skew.

[0064] Accordingly, embodiments of the present invention disclose a system optimization method based on a distributed retrieval system, applied to the aforementioned system optimization device based on a distributed retrieval system, see [link to relevant documentation]. Figure 3 As shown, the method includes:

[0065] Step S11: Set the monitoring rules, monitoring scope, and monitoring indicator thresholds for the current distributed retrieval cluster.

[0066] Step S12: Monitor the tilt of each retrieval node in the distributed retrieval cluster according to the monitoring rules, the monitoring range, and the monitoring index threshold to obtain the corresponding cluster monitoring information.

[0067] Step S13: Perform dynamic balanced management of the resource distribution of the distributed retrieval cluster based on the cluster monitoring information, and perform partitioned and hierarchical management of the request distribution of the distributed retrieval cluster based on the cluster monitoring information to achieve balanced optimization of the current distributed retrieval cluster.

[0068] As can be seen, in this embodiment, by setting monitoring rules, monitoring scope, and monitoring indicator thresholds for the current distributed retrieval cluster; monitoring the skewness of each retrieval node in the distributed retrieval cluster according to the monitoring rules, monitoring scope, and monitoring indicator thresholds to obtain corresponding cluster monitoring information; dynamically balancing the resource distribution of the distributed retrieval cluster based on the cluster monitoring information; and partitioning and hierarchically managing the distribution of requests in the distributed retrieval cluster based on the cluster monitoring information. This application adds a skew monitoring mechanism, a skew balancing management mechanism, and a request partitioning and hierarchical control mechanism to the sharded storage and retrieval process of the distributed retrieval system, thereby achieving balanced optimization of the current distributed retrieval cluster, maintaining the balance between resources and requests, and thus avoiding the problems of request anomalies and resource waste caused by skewness.

[0069] For details regarding the system optimization method based on the distributed retrieval system, please refer to the aforementioned detailed introduction to the system optimization device based on the distributed retrieval system, which will not be repeated here.

[0070] Furthermore, embodiments of this application also provide an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0071] Figure 4This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the system optimization method based on a distributed retrieval system disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0072] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0073] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0074] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the system optimization method based on a distributed retrieval system executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0075] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the system optimization method steps based on a distributed retrieval system disclosed in any of the foregoing embodiments.

[0076] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0077] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] The above provides a detailed description of a system optimization device, method, equipment, and medium based on a distributed retrieval system provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A system optimization apparatus based on a distributed search system, characterized by comprising: The method comprises the following steps: An inclination balance strategy module is configured to set monitoring rules, monitoring ranges, and monitoring index thresholds of a current distributed retrieval cluster; A dynamic balance monitoring module is configured to monitor the inclination of each retrieval node in the distributed retrieval cluster according to the monitoring rules, the monitoring ranges, and the monitoring index thresholds set by the inclination balance strategy module to obtain corresponding cluster monitoring information; An inclination balance management module is configured to dynamically balance the resource distribution of the distributed retrieval cluster according to the cluster monitoring information reported by the dynamic balance monitoring module, and to manage the request distribution of the distributed retrieval cluster in a partitioned and hierarchical manner according to the cluster monitoring information to achieve the balance optimization of the current distributed retrieval cluster; The inclination balance management module comprises: A data inclination control unit is configured to determine the data inclination of each retrieval node in the current distributed retrieval cluster according to the cluster monitoring information, and to dynamically balance the resource distribution of the distributed retrieval cluster based on the data inclination until resource balance is achieved; A request hierarchical control unit is configured to determine the request inclination of each retrieval node in the current distributed retrieval cluster according to the cluster monitoring information, and to manage the request distribution of the distributed retrieval cluster in a partitioned and hierarchical manner based on the request inclination until request balance is achieved; The data inclination control unit is specifically configured to determine the retrieval node with an abnormal resource storage difference in the distributed retrieval cluster based on the data inclination, to perform data balance transfer and receiving operations on the resource distribution of the retrieval node, to evaluate whether the new balance condition achieves data balance between the transfer node and the receiving node, and if not, to continue to dynamically balance the resource distribution of the distributed retrieval cluster based on the new balance condition until resource balance is achieved; The request hierarchical control unit is specifically configured to statistically summarize the historical requests and current requests of each retrieval node in the cluster monitoring information to determine the request frequency distribution of each retrieval node, to perform cold, warm, and hot partition management on each retrieval node based on the request frequency distribution to obtain partitioned hot zone nodes, warm zone nodes, and cold zone nodes, wherein the hot zone nodes, the warm zone nodes, and the cold zone nodes represent nodes with different request busy levels, and for the hot zone nodes with busy requests, the location of the shard replica corresponding to the hot zone node is retrieved, and the request is transferred to the cold zone node or the warm zone node corresponding to the location; The inclination balance management module is further configured to determine the request sending and response condition according to the cluster monitoring information, to perform a transfer operation on the request according to a preset proportion when the proportion of failed requests is greater than a preset proportion threshold, and to return to the step of determining the request sending and response condition according to the cluster monitoring information until the proportion of failed requests is not greater than the preset proportion threshold. The device also comprises a cluster state protection module, configured to fuse the retrieval service of the retrieval node when a large-scale request anomaly occurs in the distributed retrieval cluster, and to transfer subsequent requests to other retrieval nodes to protect the operation stability of the distributed retrieval cluster. The cluster state protection module is specifically configured to fuse the retrieval service of the hot area node when a large-scale request anomaly occurs in the hot area node in the distributed retrieval cluster, and to transfer subsequent requests to the corresponding cold area node or warm area node to protect the operation stability of the distributed retrieval cluster.

2. The system optimization apparatus based on a distributed search system according to Claim 1, wherein The tilt balance strategy module runs on the master retrieval node of the distributed retrieval cluster, the dynamic balance monitoring module runs on the data storage retrieval node of the distributed retrieval cluster, and the tilt balance management module runs on the data storage retrieval node and the master retrieval node of the distributed retrieval cluster.

3. The system optimization apparatus based on a distributed search system according to Claim 1, wherein The dynamic balance monitoring module is specifically configured to: monitor each retrieval node in the distributed retrieval cluster from the perspectives of sharded storage and request distribution according to the monitoring rules, the monitoring range and the monitoring index threshold values set by the tilt balance strategy module to obtain corresponding node information; perform alarm analysis on related index data in the node information based on the monitoring index threshold values set by the tilt balance strategy module to obtain corresponding alarm analysis information; summarize the node information and the alarm analysis information to obtain corresponding cluster monitoring information.

4. A method for optimizing a system based on a distributed search system, characterized by, The system optimization device based on the distributed retrieval system according to any one of the preceding claims 1 to 3 comprises: setting monitoring rules, a monitoring range and monitoring index threshold values of a current distributed retrieval cluster; monitoring the tilt of each retrieval node in the distributed retrieval cluster according to the monitoring rules, the monitoring range and the monitoring index threshold values to obtain corresponding cluster monitoring information; performing dynamic balance management on the resource distribution of the distributed retrieval cluster according to the cluster monitoring information, and performing zoned and layered management on the distribution of requests of the distributed retrieval cluster according to the cluster monitoring information to achieve balance optimization of the current distributed retrieval cluster; wherein the dynamic balance management on the resource distribution of the distributed retrieval cluster according to the cluster monitoring information, and the zoned and layered management on the distribution of requests of the distributed retrieval cluster according to the cluster monitoring information to achieve balance optimization of the current distributed retrieval cluster comprises: determining the data tilt of each retrieval node in the current distributed retrieval cluster according to the cluster monitoring information, and performing dynamic balance management on the resource distribution of the distributed retrieval cluster based on the data tilt until resource balance is achieved; determining the request tilt of each retrieval node in the current distributed retrieval cluster according to the cluster monitoring information, and performing zoned and layered management on the request distribution of the distributed retrieval cluster based on the request tilt until request balance is achieved. According to the cluster monitoring information, the data skew of each retrieval node in the current distributed retrieval cluster is determined, and the resource distribution of the distributed retrieval cluster is dynamically balanced based on the data skew until resource balance is achieved, including: determining the retrieval node with abnormal resource storage difference in the distributed retrieval cluster based on the data skew; performing data balancing transfer and receiving operation on the resource distribution of the retrieval node; evaluating whether the new balanced condition achieves data balance between the transfer node and the receiving node; if not, the resource distribution of the distributed retrieval cluster is continuously dynamically balanced based on the new balanced condition until resource balance is achieved; According to the cluster monitoring information, the request skew of each retrieval node in the current distributed retrieval cluster is determined, and the request distribution of the distributed retrieval cluster is partitioned and managed based on the request skew until request balance is achieved, including: the historical requests and current requests of each retrieval node in the cluster monitoring information are summarized and counted to determine the request frequency distribution of each retrieval node; the cold, warm and hot partition management of each retrieval node is performed based on the request frequency distribution to obtain partitioned hot zone nodes, warm zone nodes and cold zone nodes; the hot zone nodes, warm zone nodes and cold zone nodes respectively represent nodes with different request busy degrees; for the hot zone nodes with request busy, the location of the shard replica corresponding to the hot zone node is retrieved, and the request is transferred to the cold zone node or the warm zone node corresponding to the location; The method further comprises: determining the request sending response condition according to the cluster monitoring information, when the proportion of failed requests is greater than a preset proportion threshold, performing a transfer operation on the request according to a preset proportion, and returning to the step of determining the request sending response condition according to the cluster monitoring information until the proportion of failed requests is not greater than the preset proportion threshold; When the distributed retrieval cluster has a large range of request abnormalities, the retrieval service of the retrieval node is fused, and subsequent requests are transferred to other retrieval nodes to protect the operation stability of the distributed retrieval cluster; When the distributed retrieval cluster has a large range of request abnormalities, the retrieval service of the retrieval node is fused, and subsequent requests are transferred to other retrieval nodes to protect the operation stability of the distributed retrieval cluster, including: when the hot zone node in the distributed retrieval cluster has a large range of request abnormalities, the retrieval service of the hot zone node is fused, and subsequent requests are transferred to the corresponding cold zone node or warm zone node to protect the operation stability of the distributed retrieval cluster.

5. An electronic device, comprising: Comprise: a memory for saving a computer program; a processor for executing the computer program to implement the steps of the system optimization method based on the distributed retrieval system according to claim 4.

6. A computer-readable storage medium, characterized in that, for storing a computer program; wherein the computer program is executed by a processor to implement the steps of the system optimization method based on the distributed retrieval system according to claim 4.

Images