Method and device for selecting nodes in cluster, electronic equipment and medium

Abstract

The invention provides a node selection method and device in a cluster, electronic equipment and a medium, and relates to the technical field of query engines, and the method comprises the following steps: obtaining load indexes which are used for processing a plurality of nodes of a database service and influence the load size; clustering the plurality of nodes according to the load indexes of the plurality of nodes to obtain a plurality of types of clusters; wherein the type of the clustering cluster is determined according to the type of the load; determining load values of the plurality of nodes according to preset load values corresponding to types to which the plurality of clusters belong; and determining the node meeting the load value of the first preset requirement from the plurality of nodes, and distributing the database service to the determined node, so that the determined node processes the database service. According to the embodiment of the invention, the node is determined for processing based on the load of the node in the plurality of nodes for processing the database service, so that the problem of service processing delay caused by the problem of node processing capability is relieved.

Description

Technical Field

[0001] The present invention relates to the field of query engine technology, and in particular to a method, device, electronic device and medium for selecting nodes in a cluster. Background Art

[0002] With the explosive growth of information in today's society, the speed and scale of data generation have far exceeded the processing capabilities of traditional databases. Data volumes across all industries are expanding exponentially, with information constantly emerging from various channels such as social media, the Internet of Things, and e-commerce. Traditional relational databases are increasingly unable to cope with this vast and complex data. To cope with this data deluge, OLAP (Online Analytical Processing) query engines have emerged. With their efficient and flexible multidimensional data analysis capabilities, they have gradually replaced traditional databases in the field of complex data analysis.

[0003] In an OLAP engine, multiple nodes are created to process database tasks, forming a cluster. During operation, large volumes of tasks may need to be processed, but the processing capacity of the nodes is limited. This can easily lead to a backlog of tasks at the nodes, causing processing delays. Summary of the Invention

[0004] The present invention provides a node selection method, device, electronic device and medium in a cluster, which determines the node to be processed based on the node load among multiple nodes processing database services, thereby alleviating the problem of service processing delay caused by problems with node processing capacity.

[0005] In a first aspect, an embodiment of the present invention provides a method for selecting a database service processing node in a cluster, comprising:

[0006] Obtain load indicators that affect load sizes of multiple nodes used to process database services;

[0007] Clustering the multiple nodes according to their load indicators to obtain multiple types of clusters; wherein the types of the clusters are determined according to the load types;

[0008] Determining load values ​​of multiple nodes according to preset load values ​​corresponding to the types of the multiple clusters;

[0009] A node that meets a first preset load requirement is determined from a plurality of nodes, and the database service is allocated to the determined node so that the determined node processes the database service.

[0010] The above method can determine the load value of each node through the load indicator that affects the load size in the node, and determine the node to be processed based on the load value of the node among multiple nodes processing database business. The present invention intends to determine the node that processes database business while taking the load into consideration, thereby alleviating the problem of business processing delay caused by the problem of node processing capacity.

[0011] In one possible implementation, multiple nodes are clustered according to their load indicators to obtain various types of clusters, including:

[0012] Determining a correlation between load indicators of the plurality of nodes and reference values ​​of the plurality of virtual nodes corresponding to each load indicator;

[0013] Determining an association degree between each node and each load type based on an association degree between at least a portion of the load indicator of each node and reference values ​​of a plurality of virtual nodes corresponding to at least a portion of the load indicators; wherein the type of the virtual node is the same as the load type;

[0014] Determine the absolute load matrix of each node based on the association between each node and each load type and the weight corresponding to each load type; where the weight corresponding to each element is determined based on the type of virtual node corresponding to each element;

[0015] The absolute load matrices of multiple nodes are processed by fuzzy clustering to obtain various types of clusters.

[0016] The above method can use the absolute load matrix of each node to perform fuzzy clustering processing to obtain the cluster to which each node belongs. Since fuzzy clustering processing allows a node to belong to multiple clusters, nodes with similar load conditions are distinguished based on the degree to which the nodes belong to different clusters, which improves the node differentiation and further facilitates the selection of nodes for processing business.

[0017] In one possible implementation, determining the correlation between load indicators of multiple nodes and reference values ​​of multiple virtual nodes corresponding to each load indicator includes:

[0018] determining a difference between load indicators of the plurality of nodes and a reference value of the plurality of virtual nodes corresponding to each load indicator;

[0019] For the difference between each load value and each reference value, the correlation between the load indicators of multiple nodes and the reference values ​​of multiple virtual nodes corresponding to each load indicator is determined based on the maximum and minimum values ​​of the differences between the load indicators of multiple nodes and the reference values ​​of multiple virtual nodes corresponding to each load indicator, as well as the differences.

[0020] The above method uses the difference between the load index of each node and the reference value of the virtual node corresponding to each load index to indicate the correlation between each load index and the virtual node, and then preliminarily forms the load value of each node, thereby improving the accuracy of subsequent clustering.

[0021] In one possible implementation, determining the association degree of each node with each load type based on the association degree between at least part of the load indicator of each node and reference values ​​of multiple virtual nodes corresponding to at least part of the load indicator includes:

[0022] determining a degree of association between each node and each virtual node based on a degree of association between at least a portion of the load indicator of each node and reference values ​​of a plurality of virtual nodes corresponding to at least a portion of the load indicator;

[0023] The association degree of each node with each load type is determined based on the association degree of each node with each type of virtual node.

[0024] In the above manner, since the type of the virtual node is the same as the load type, the present invention determines the association between multiple load types and multiple nodes, and then clusters the nodes based on the association, thereby improving the accuracy of subsequent clustering.

[0025] In one possible implementation, determining the load values ​​of the plurality of nodes according to the preset load values ​​corresponding to the types of the plurality of clusters includes:

[0026] Determining load values ​​of the plurality of nodes according to preset load values ​​corresponding to the types of the plurality of clusters and the product of the membership degrees of the plurality of nodes and the plurality of types of clusters;

[0027] For any node and any type of cluster, the membership of the node to the cluster is determined in the following manner:

[0028] Determine a plurality of first distances of the node; wherein any one of the first distances is the distance from each element in the absolute load matrix of the node to the cluster center of any type of cluster;

[0029] The membership of the node to the cluster is determined according to the ratio between the multiple first distances of the node and the second distance of the node; wherein the second distance is the distance from each element in the absolute load matrix of the node to the cluster center of the cluster.

[0030] The above method can determine the membership of each node to each cluster based on the distance from each element in the absolute load matrix of each node to the cluster center of any type of cluster. The membership, that is, the degree to which the node belongs to each type of cluster, is used as a weight to add the preset load value to determine the load value of each node. This can more clearly distinguish the load values ​​of the nodes, thereby enhancing the accuracy of subsequent node selection.

[0031] In one possible implementation, determining, from a plurality of nodes, a node having a load value that meets a first preset requirement includes:

[0032] If the node is the first node for receiving user requests and allocating database services, the first preset requirement is the first node with the smallest load value;

[0033] If the node is a second node for executing a database service, and the database service is a query service, the first preset requirement is the second node in the cluster that executes the query service submitted by the user;

[0034] If the node is a second node for executing database business, and the database business is an imported business, then the first preset requirement is the target second node with the largest load value among at least one target second node; wherein, the target second node is the second node in the cluster that executes the query business submitted by the user.

[0035] The above method can select the first node with the smallest load and select the second node according to different services, thereby ensuring efficient utilization of system resources and balanced distribution of load, bringing more efficient and stable system response and performance guarantee.

[0036] In one possible implementation, the method further includes:

[0037] If the database service is a query service, then when the resource size of at least one target second node that meets the third preset condition is the smallest among the resource upper limit required by the user submitting the database service, the resource upper limit configured by the cluster for the user, and the resource size of at least one target second node that meets the second preset condition, the user's resource upper limit is modified to a preset value determined based on the resource size of the target second node that meets the second preset condition among the at least one target second node;

[0038] If the database business is an imported business, then when the user's own resource upper limit exceeds the resource upper limit configured for the user by the cluster, and the load value in at least one target second node meets the third preset condition, the user's own resource upper limit is modified to a preset value determined based on the resource size of the second node with the largest load value in at least one target second node.

[0039] The above method accurately corrects the import or query resources after conducting a multi-dimensional comprehensive assessment based on the user-configured resource limit, the execution database business regulations, the resource size of the second node, and the load value of the second node. This ensures the stability and efficiency of the import business while greatly reducing the dependence on manual intervention, realizing automated and intelligent resource management, demonstrating the flexibility and efficiency of dynamic resource allocation, and improving the stability and overall performance of the system.

[0040] In a second aspect, an embodiment of the present invention provides a device for selecting a database service processing node in a cluster, comprising:

[0041] An acquisition module, configured to acquire load indicators affecting load sizes of multiple nodes used to process database services;

[0042] A clustering module, configured to cluster multiple nodes according to their load indicators to obtain multiple types of clusters; wherein the types of clusters are determined according to the load types;

[0043] A determination module, configured to determine load values ​​of a plurality of nodes according to preset load values ​​corresponding to types of a plurality of clusters;

[0044] The selection module is configured to determine a node that meets a first preset load value requirement from a plurality of nodes, and allocate the database service to the determined node so that the determined node processes the database service.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0046] processor;

[0047] The processor is configured to execute the computer program or instructions in the memory, so that the method for selecting a database service processing node in a cluster as described in any one of the first aspects is executed.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute the method for selecting a database business processing node in a cluster as described in any one of the first aspects.

[0049] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method for selecting a database business processing node in a cluster as described in any one of the first aspects.

[0050] In addition, the technical effects brought about by any implementation method in the second to fifth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here.

[0051] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of a device for cluster-based database processing provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of a process for selecting a database service processing node in a cluster provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a process for clustering multiple load indicators provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a method for determining a node load value based on multiple load indicators provided by an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of a judgment matrix provided by an embodiment of the present invention;

[0057] Figure 6 A flowchart of another method for selecting a database service processing node in a cluster provided by an embodiment of the present invention;

[0058] Figure 7 A structural diagram of a device for selecting a database service processing node in a cluster provided by an embodiment of the present invention;

[0059] Figure 8 A structural diagram of a device for selecting a database service processing node in a cluster provided by an embodiment of the present invention;

[0060] Figure 9 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0062] In the description of the embodiments of the present application, unless otherwise specified, in the description of the embodiments of the present application, "plurality" refers to two or more than two.

[0063] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0064] Glossary:

[0065] Online Analytical Processing (OLAP): A high-performance data analysis technology widely used in business intelligence and data warehousing. Using a multidimensional data model, OLAP organizes massive amounts of data into different dimensions, enabling users to quickly perform queries and conduct in-depth analysis. Leveraging pre-calculation and optimized storage strategies, OLAP achieves efficient query response, supports flexible operations such as slicing and drilling, and delivers accurate analytical results in a fraction of the time.

[0066] Load balancing: In the database world, load balancing distributes database query or operation requests across multiple database servers, ensuring a balanced workload on each server and improving overall system performance and reliability. This technique helps prevent a single database server from becoming a bottleneck due to excessive requests, thereby improving system responsiveness, availability, and scalability.

[0067] With the explosive growth of information in today's society, most data is stored in databases, which leads to the query and import of databases. The servers that process the query and import services are clustered together. Figure 1 As shown, an embodiment of the present invention provides a cluster including first nodes 1-n and second nodes 1-m. The first node is also called an FE node. EF nodes can be FE-Flowers (EF nodes - followers), FE-Leaders (FE nodes - leaders), or FE-Observers (FE nodes - observers), depending on their functions. The second node is also called a BE node. The first node is used to receive user requests and allocate database services, while the second node is used to execute database services. The second nodes 1-m are connected to a database 100, and the user's terminal is connected to the first nodes 1-n. Specifically, after receiving a user request, the first node parses the user request to extract the database service and sends it to the second node, which then executes the database service.

[0068] For example, an OLAP data system based on StarRocks (a cluster) often balances the load of FE nodes through a database connection interface (such as JDBC Connector) and the middleware of the relational database management system Mysql (ProxySql). The JDBC Connector uses an algorithm to establish connections randomly, while ProxySql routes queries based on the query content, user, source IP, and other conditions. However, neither of these two methods takes into account the hardware resources of the FE node, such as the CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, disk, network card, and software resources such as service internal indicators. In high-concurrency scenarios, the use of the above two methods is prone to uneven use of hardware resources between FE nodes, causing the StarRocks cluster to have a weak point effect and slow database business processing.

[0069] In the OLAP system, the dynamic parameters of cluster resources (CPU and memory for a single task, CPU and memory for a single import task) are specified by the user when submitting the request. After receiving the database business, the FE node will check the user's resource request. If it is within the set range, it will be submitted to the BE node for query. When the BE node is at its peak query period and the overall resources are insufficient, the query will fail. At this time, the user's resource request must be manually modified to ensure the query is correct, resulting in slow business processing. Moreover, this process lacks flexibility and dynamic adaptability, is complex and difficult to maintain.

[0070] Based on this, an embodiment of the present invention provides a method, device, electronic device and medium for selecting database business processing nodes in a cluster, which aims to determine the nodes that process database business while taking into account the load, thereby alleviating the problem of business processing delays caused by problems with node processing capacity.

[0071] The present invention is described in detail below with reference to the accompanying drawings:

[0072] Combine Figure 2 As shown, an embodiment of the present invention provides a method for selecting a database service processing node in a cluster, comprising:

[0073] S210: Obtain load indicators that affect load sizes of multiple nodes used to process database services.

[0074] The load indicators include indicators that characterize the node hardware usage and indicators that characterize the node service status.

[0075] Exemplarily, a method for obtaining load indicators includes collecting node indicators based on the open source tool Prometheus. The node proposed in the embodiment of the present invention includes a first node and a second node.

[0076] Combine Figure 1 As shown in the example, StarRocks deploys data collection software on each node in the cluster. This software periodically (at a controllable frequency) collects hardware metrics for the CPU, GPU, memory, disk, network card, motherboard, heat sink, power supply, and other components, and stores them in a TSDB (Time Series Database). For example, TSDB can store data for the past 30 days. Table 1 shows the hardware metrics for each node.

[0077] Table 1

[0078]

[0079] To collect service status indicators, configure the IP (Internet Protocol) and HTTP (Hypertext Transfer Protocol) port of each FE node and BE node in the open-source tool Prometheus, and set periodic (controllable frequency) collection. The open-source tool Prometheus will periodically pull the service status indicators of the FE nodes and BE nodes into the TSDB. The service status indicators of the FE nodes and BE nodes are shown in Table 2 below.

[0080] Table 2

[0081]

[0082] Acquiring the load index of the node in step 110 includes acquiring the load index of the first node and acquiring the load index of the second node.

[0083] Load indicators of the first node, such as CPU utilization, CPU instruction execution speed, CPU hyperthreading utilization, CPU temperature, GPU utilization, GPU memory usage, GPU performance status, GPU temperature, memory utilization, memory swap, disk I / O per second (IOPS), disk throughput, disk utilization, disk read and write speed, disk temperature, network card latency, network card bandwidth, network card temperature, FE JVM (Java Virtual Machine) information, FE RPS (Requests Per Second), FE QPS (Queries-per-second), FE number of connections, and FE Avg response latency.

[0084] Load indicators of the second node, such as CPU usage, CPU instruction execution speed, CPU hyperthreading utilization, CPU temperature, GPU usage, GPU memory usage, GPU performance status, GPU temperature, memory usage, memory swap, disk I / O per second (IOPS), disk throughput, disk utilization, disk read and write speed, disk temperature, network card latency, network card bandwidth, network card temperature, BE Compacition Core (computer core), and BE read and write speed.

[0085] S220: Clustering the multiple nodes according to the load indicators of the multiple nodes to obtain multiple types of clusters.

[0086] The type of cluster is determined according to the load type.

[0087] For example, if the load types are low load, medium-low load, medium load, medium-high load, and high load, then the cluster types are low load, medium-low load, medium load, medium-high load, and high load. In this embodiment of the present invention, multiple nodes can be divided into five clusters: low load, medium-low load, medium load, medium-high load, and high load.

[0088] S230: Determine load values ​​of multiple nodes according to preset load values ​​corresponding to the types of the multiple clusters.

[0089] For example, the preset load value corresponding to the low load type cluster is 0, the preset load value corresponding to the medium-low load type cluster is 1, the preset load value corresponding to the medium load type cluster is 2, the preset load value corresponding to the medium-high load type cluster is 3, and the preset load value corresponding to the high load type cluster is 4.

[0090] Specifically, the load value of each node is determined by considering the preset load value corresponding to the type of cluster to which each node belongs.

[0091] S240: Determine a node that meets a first preset load value from a plurality of nodes, and allocate the database service to the determined node so that the determined node processes the database service.

[0092] The embodiment of the present invention can determine the load value of each node through the load indicator that affects the load size in the node, and determine the node to be processed based on the load of the node among multiple nodes processing database services. The present invention intends to determine the node that processes database services while taking the load into consideration, thereby alleviating the problem of business processing delay caused by problems with node processing capacity.

[0093] Figure 2 The nodes in can be the first node and the second node.

[0094] Illustratively, the embodiment of the present invention may further combine the existing node selection method with the node selection method provided by the present invention to select a first node from multiple first nodes in the cluster.

[0095] For example, the load balancing of FE nodes is performed through a database connection interface (such as JDBC Connector) and the middleware (ProxySql) of the relational database management system Mysql. ProxySql determines the FE node based on conditions such as the query content, user, source IP, and determines the FE node based on multiple load indicators (the method provided in an embodiment of the present invention). The determined FE node continues to parse the business and sends the business to the BE node, which executes the business.

[0096] It should be noted that the properties of cluster clusters and iterative clusters are the same. Cluster clusters are clusters after the clustering process is completed, and iterative clusters are clusters obtained after each iteration during the clustering process.

[0097] The clustering process may be a fuzzy clustering process.

[0098] Fuzzy clustering uses the Fuzzy C-Means (FCM) algorithm for clustering. The FCM algorithm allows each data point to belong to multiple cluster centers and calculates a membership degree for each data point. The membership degree of each node indicates the degree to which it belongs to a particular cluster at that stage.

[0099] In some embodiments, clustering is performed on the multiple nodes according to their load indicators to obtain multiple types of clusters. That is, step 220 may be implemented in multiple ways.

[0100] In some embodiments, combined Figure 3As shown, one implementation may include:

[0101] S300: Determine the correlation between load indicators of multiple nodes and reference values ​​of multiple virtual nodes corresponding to each load indicator.

[0102] The virtual node is pre-set, and the reference value of the virtual node corresponding to each load indicator can be determined based on historical data.

[0103] Optionally, step 300 specifically includes:

[0104] determining a difference between load indicators of the plurality of nodes and a reference value of the plurality of virtual nodes corresponding to each load indicator;

[0105] For the difference between each load value and each reference value, the correlation between the load indicators of multiple nodes and the reference values ​​of multiple virtual nodes corresponding to each load indicator is determined based on the maximum and minimum values ​​of the differences, as well as the differences, between the load indicators of multiple nodes and the reference values ​​of multiple virtual nodes corresponding to each load indicator.

[0106] For example, the correlation between a load indicator and a reference value of a virtual node corresponding to the load indicator is determined by the following formula:

[0107]

[0108] Among them, γ ij (k) is the correlation between the i-th load index and the reference value of the j-th virtual node corresponding to the i-th load index; ρ is the resolution coefficient, which is usually set to 0.5; Δ ij (k) = |x′ i (k)-y′ j (k)|,x′ i (k) is the i-th load index, y′ j (k) is the reference value of the jth virtual node corresponding to the i-th load index, Δ ij (k) is the difference between the i-th load index and the reference value of the j-th virtual node corresponding to the i-th load index; Δ min is the minimum value among the differences between all load indices and the reference values ​​of all virtual nodes. For example, when the difference between the load index 1 of node 1 and the reference value of virtual node V4 is the smallest, then Δ min is the difference between the load index 1 of node 1 and the reference value of virtual node V4; Δ max is the maximum value of the difference between all load indices and the reference values ​​of all virtual nodes. For example, when the difference between the load index 1 of node 1 and the reference value of virtual node V10 is the largest, Δ maxis the difference between the load index 1 of node 1 and the reference value of virtual node V10.

[0109] S301: Determine an association degree between each node and each load type according to an association degree between at least part of the load indicator of each node and reference values ​​of multiple virtual nodes corresponding to at least part of the load indicator.

[0110] The type of the virtual node is the same as the type of the load.

[0111] When the load type is low load, medium-low load, medium load, medium-high load, or high load, the type of the virtual node can be low load, medium-low load, medium load, medium-high load, or high load.

[0112] When the load type is low load, medium load, or high load, the type of the virtual node can be low load, medium load, or high load.

[0113] The multiple virtual nodes proposed in the present invention can be determined according to the load type. For example, the multiple virtual nodes include 6 low-load type virtual nodes (represented by V1 to V6), 3 medium-load type virtual nodes (represented by V7 to V9), and 1 high-load type virtual node (represented by V10).

[0114] Optionally, step 301 specifically includes:

[0115] determining a degree of association between each node and each virtual node based on a degree of association between at least a portion of the load indicator of each node and reference values ​​of a plurality of virtual nodes corresponding to at least a portion of the load indicator;

[0116] The association degree of each node with each load type is determined based on the association degree of each node with each type of virtual node.

[0117] Exemplarily, the association degree between each node and each virtual node is determined by the following formula:

[0118]

[0119] Among them, each node includes k load indicators, and the correlation between each load indicator and each virtual node is γ ij (t), the correlation degree between a node and a virtual node is the sum of the correlation degrees between all load indices of a node and a virtual node, divided by the number of load indices of a node.

[0120] For example, if a node has two load indicators and multiple virtual nodes V1 to V10, the average of the correlation between the first load indicator and virtual node V1 and the correlation between the second load indicator and virtual node V1 is the correlation between the node and virtual node V1. Similarly, nine correlations can be calculated between the node and virtual nodes V2 to V10.

[0121] Among them, multiple virtual nodes are classified according to load type. Taking the load types of low load, medium load and high load as an example, multiple virtual nodes can be divided into one group of low load, one group of medium load and one group of high load. For example, virtual nodes V1~V10 are divided into virtual nodes V1~V6 as low load, virtual nodes V7~V9 as medium load, and virtual node V10 as high load.

[0122] The average value of the association between each node and each type of virtual node is used as the association between each node and each load type. The association between the low load type and each node is calculated using the following formula:

[0123]

[0124] Among them, γ i低 is the association degree between the i-th node and the low load type, |S 低 | is the number of virtual nodes of low load type, for example 6, then |S 低 | is 6; It is the sum of the associations between the i-th node and the virtual nodes belonging to the low-load type. For example, the virtual nodes belonging to the low-load type are V1 to V6. Then the sum of the six associations between the i-th node and the virtual nodes V1 to V6 divided by 6 is the association between the low-load type and the i-th node.

[0125] When the types of virtual nodes include low load, medium load and high load, determine an element of a node corresponding to low load, an element of a node corresponding to medium load and an element of a node corresponding to high load, and form an absolute load matrix of a node by combining an element corresponding to low load, an element corresponding to medium load and an element corresponding to high load.

[0126] S302: Determine an absolute load matrix of each node according to the association degree between each node and each load type and the weight corresponding to each load type.

[0127] The weight corresponding to each load type is determined according to the number of virtual nodes belonging to each load type.

[0128] Specifically, the weight corresponding to a load type is the number of virtual nodes belonging to the load type divided by the total number of virtual nodes.

[0129] For example, if the number of virtual nodes belonging to the low load type is 6 and the total number of virtual nodes is 10, then the weight corresponding to the low load type is 6 / 10=0.6; if the number of virtual nodes belonging to the medium load type is 3, then the weight corresponding to the medium load type is 3 / 10=0.3; if the number of virtual nodes belonging to the high load type is 1, then the weight corresponding to the high load type is 1 / 10=0.1.

[0130] In detail, step 302 is to weight the association between a node and a load type using the weight corresponding to a load type for each node and each load type, and use the weighted association between a node and a load type as an element in the absolute load matrix of the node.

[0131] The elements in the absolute load matrix of each node correspond to the load type one by one. For example, if the load type is low load, medium load, and high load, then the elements in the absolute load matrix of each node include the elements corresponding to the low load type, the medium load type, and the high load type.

[0132] S303: Perform fuzzy clustering processing on the absolute load matrices of multiple nodes to obtain multiple types of clusters.

[0133] Exemplarily, the multiple nodes include FE1 and FE2, and both FE1 and FE2 include load index 1 and load index 2. The virtual nodes are V1 to V10, and the specific values ​​are shown in Table 1:

[0134]

[0135]

[0136] Formula (1) is used to determine the correlation between the load index 1 of FE1 and the reference values ​​of the virtual nodes V1 to V10 corresponding to the load index 1, the correlation between the load index 2 of FE1 and the reference values ​​of the virtual nodes V1 to V10 corresponding to the load index 2, the correlation between the load index 1 of FE2 and the reference values ​​of the virtual nodes V1 to V10 corresponding to the load index 1, and the correlation between the load index 2 of FE2 and the reference values ​​of the virtual nodes V1 to V10 corresponding to the load index 2.

[0137] The calculation process of the correlation between the load index 1 of FE1 and the reference value of the virtual node V1 corresponding to the load index 1 is as follows:

[0138] First, determine the difference between the load index 1 of FE1 and the reference value of the virtual node V1 corresponding to the load index 1;

[0139] As shown in Table 1, 1.0 is the value of load index 1 of FE1, and 0.95 is the reference value of virtual node V1 corresponding to load index 1. Then their difference is:

[0140] Δ ij (k)=|1.0-0.95|=0.05

[0141] Here, 0.05 is the difference between the load index 1 of FE1 and the reference value of the virtual node V1 corresponding to the load index 1.

[0142] Then, find the maximum and minimum values ​​of the difference between all load indicators of the two nodes and the reference values ​​of each corresponding virtual node;

[0143] For example, if the minimum value is 0.05, then Δ min = 0.05, the maximum value is the difference between the load index 1 of FE2 and the reference value of the virtual node V10 corresponding to the load index 1, which is 0.55, then Δ max =0.55;

[0144] Finally, the correlation between the load index 1 of FE1 and the reference value of the virtual node V1 corresponding to the load index 1 is determined.

[0145] Combining formula (1) and the above-mentioned content, the correlation degree is determined as follows:

[0146]

[0147] The correlation degree between the load index 1 of FE1 and the reference value of the virtual node V1 corresponding to the load index 1 is 1.

[0148] Similarly, the difference between the load index 2 of FE1 and the reference value of the virtual node V1 corresponding to the load index 2 is calculated to be |0.8-0.85|=0.05;

[0149] The correlation between the load index 2 of FE1 and the reference value of the virtual node V1 corresponding to the load index 2 is calculated as:

[0150]

[0151] The correlation between FE1 and virtual node V1 is calculated by formula (2), namely:

[0152]

[0153] In the same way, the association degrees between FE1 and virtual nodes V2 to V10, and the association degrees between FE2 and virtual nodes V1 to V10 are calculated.

[0154] Table 2 shows the correlation between FE1 and virtual nodes V1 to V10 calculated using the above method.

[0155] Table 2

[0156] node Correlation FE1 and V1 1.0 FE1 and V2 0.85 FE1 and V3 0.90 FE1 and V4 0.82 FE1 and V5 0.88 FE1 and V6 0.80 FE1 and V7 0.78 FE1 and V8 0.72 FE1 and V9 0.85 FE1 and V10 0.65

[0157] The virtual nodes belonging to the low load type are V1 to V6. Then, combined with Table 2, the correlation between the low load type and FE1 is calculated by formula (3) as follows:

[0158]

[0159] The virtual nodes belonging to the medium load type are V7 to V9. Then, combined with Table 2, the correlation between the load type and FE1 in formula (3) is as follows:

[0160]

[0161] The virtual node belonging to the high load type is V10. Then, combined with Table 2, the correlation between the high load type and FE1 is similar to formula (3):

[0162]

[0163] The weight corresponding to the low load type is 0.6, the weight corresponding to the medium load type is 0.3, and the weight corresponding to the high load type is 0.1. The elements in the absolute load matrix of FE1 are:

[0164] The contribution of the low load type is: 0.875×0.6=0.525;

[0165] The contribution of the medium load type is: 0.783×0.3=0.235;

[0166] The contribution of high load type is: 0.65×0.1=0.065;

[0167] The absolute loading matrix of FE1 is [0.525 0.235 0.065].

[0168] Similarly, the absolute loading matrix of FE2 is calculated, which is [0.480 0.210 0.058].

[0169] In some embodiments, another implementation of clustering multiple nodes based on their load indicators to obtain multiple types of clusters may be:

[0170] According to the load indicators of multiple nodes, the initial load value of each node is determined; the initial load values ​​of multiple nodes are subjected to fuzzy clustering processing to obtain various types of clusters.

[0171] The initial load value of each node may be an average value of load indicators of multiple nodes, or a weighted average value of the load indicators of each node.

[0172] Step 303 performs fuzzy clustering processing on the absolute load values ​​of multiple nodes to obtain multiple types of clusters. The above fuzzy clustering processing schemes are the same, specifically:

[0173] During each iteration, the cluster center of each iteration is determined;

[0174] Determine the distance between the cluster centers and the absolute loading matrices of multiple nodes for each iteration;

[0175] Fuzzy clustering is performed based on the distance between the cluster center of each iteration and the absolute load matrix of multiple nodes to obtain the iteration cluster of each iteration;

[0176] Determining the value of the objective function of each iteration according to the membership between the multiple nodes and the multiple iteration clusters;

[0177] According to the value of the objective function of each iteration, the cluster center of the next iteration is determined;

[0178] After the iteration stopping condition is reached, the iteration is stopped; and the iteration cluster obtained in the last iteration is the clustering cluster.

[0179] The objective function formula of the fuzzy clustering algorithm is:

[0180]

[0181] in, is the membership of the i-th node to the j-th cluster, c j is the cluster center of the jth cluster, m is a constant (here 2), indicating the degree of fuzziness of the cluster, ||x i -c j || is the distance between the i-th node and the cluster center of the j-th cluster.

[0182] Optionally, the distance can be calculated using Euclidean distance.

[0183] In some embodiments, for each node and each iteration cluster, the node whose membership is currently being calculated is called a target node, and the iteration cluster whose membership is currently being calculated is called a target iteration cluster. The membership of the target node and the target iteration cluster is determined by:

[0184] Determine the distance from each element in the absolute load matrix of the target node to the cluster center of any type of iterative cluster;

[0185] The membership of the target node to the target cluster is determined according to the ratio of the distance from each element in the absolute load matrix of the target node to the cluster center of any type of iteration cluster and the distance from each element in the absolute load matrix of the target node to the cluster center of the target iteration cluster.

[0186] Exemplarily, the membership of the i-th node and the j-th iteration cluster is determined by the following formula (5):

[0187]

[0188] Among them, ‖x i -c k ‖ is the Euclidean distance between the i-th node and the cluster center of the k-th iteration cluster; ‖x i -c j ‖ is the Euclidean distance between the i-th node and the cluster center of the j-th iteration cluster; U ij is the membership degree between the i-th node and the j-th iteration cluster. j belongs to k.

[0189] For example, a cluster includes 5 nodes with load types of low load, medium load, and high load. The absolute load matrix of the 5 nodes is:

[0190] The absolute load matrix of node 1 is [0.8 0.1 0.1];

[0191] The absolute load matrix of node 2 is [0.2 0.7 0.1];

[0192] The absolute load matrix of node 3 is [0.1 0.1 0.8];

[0193] The absolute load matrix of node 4 is [0.6 0.3 0.1];

[0194] The absolute load matrix of node 5 is [0.3 0.5 0.2].

[0195] During the first iteration, three initial cluster centers can be randomly selected, such as the first three data points, for example:

[0196] The cluster center of the low-load cluster C1 is [0.8 0.1 0.1];

[0197] The cluster center of the medium-load cluster C2 is [0.2 0.7 0.1];

[0198] The cluster center of the high-load cluster C3 is [0.1 0.1 0.8].

[0199] Calculate the Euclidean distance between the five nodes and the cluster center of each cluster. Taking node 4 as an example, the specific distance is:

[0200] The Euclidean distance between node 4 and the low-load cluster C1 is:

[0201]

[0202] The Euclidean distance between node 4 and the medium-load cluster C2 is:

[0203]

[0204] The Euclidean distance between node 4 and the high-load cluster C3 is:

[0205]

[0206] The membership degree of node 4 and low-load cluster C1 is calculated by formula (5):

[0207]

[0208] By the above method, the membership degree of node 4 with the medium-load cluster C2, the membership degree of node 4 with the high-load cluster C3, and the membership degrees of other nodes with the three clusters are calculated;

[0209] According to the membership between all nodes and the three clusters and formula (4), the value of the objective function of the first iteration is determined;

[0210] According to the value of the objective function of the first iteration, the cluster center of the second iteration is determined, and so on, multiple iterations are performed to obtain the cluster clusters after clustering processing.

[0211] In some embodiments, the implementation of determining the load values ​​of multiple nodes according to the preset load values ​​corresponding to the types of the multiple clusters includes:

[0212] The load values ​​of the multiple nodes are determined according to the preset load values ​​corresponding to the types of the multiple clusters and the product of the membership degrees of the multiple nodes and the multiple types of clusters.

[0213] Specifically, the load value of a node is determined by the following formula:

[0214]

[0215] Where: L i is the load value of the i-th node, μ ik is the membership of the i-th node to the k-th cluster, L k is the preset load value corresponding to the type of the k-th cluster.

[0216] For example, when the clusters include low-load type clusters, medium-load type clusters, and high-load type clusters, then the load value of the i-th node is the sum of the preset load value corresponding to the low-load type cluster multiplied by the i-th node's membership to the low-load type cluster, the preset load value corresponding to the medium-load type cluster multiplied by the i-th node's membership to the medium-load type cluster, and the preset load value corresponding to the high-load type cluster multiplied by the i-th node's membership to the high-load type cluster.

[0217] For any node and any iterative cluster, the node whose membership is currently being sought is called the target node, and the cluster whose membership is currently being sought is called the target cluster. The membership of the target node and the target cluster is determined by the following method:

[0218] Determine multiple first distances of the target node; wherein any first distance is the distance from each element in the absolute load matrix of the target node to the cluster center of any type of cluster;

[0219] The membership of the target node to the target cluster is determined according to the ratio between the multiple first distances of the target node and the second distance of the target node; wherein the second distance is the distance from each element in the absolute load matrix of the target node to the cluster center of the target cluster.

[0220] Of course, the calculation method of the membership degree of each iterative cluster and each node is also determined by formula (5).

[0221] In some embodiments, the value of the load index needs to be standardized. For example, the standard value of the load index is determined by formula (6):

[0222]

[0223] Where: x ij is the jth load index of the i-th node, min(x j ) and max(x j ) are the minimum and maximum values ​​of the j-th load index, respectively, and can be the maximum and minimum values ​​of the historical values ​​of the j-th load index, x′ ij is the standardized data.

[0224] In some embodiments, after determining the standard value of the load indicator, the standard value of the load indicator is weighted using the weight corresponding to the load indicator, and the load value of the node is determined using the weighted load indicator.

[0225] For example, combined Figure 4As shown, the embodiment of the present invention first obtains multiple load indicators of FE nodes and multiple load indicators of BE nodes, and performs data preprocessing on the multiple load indicators of FE nodes and multiple load indicators of BE nodes, that is, standardizing the multiple load indicators of FE nodes and multiple load indicators of BE nodes as described in the above article, using the multiple load indicators of FE nodes and multiple load indicators of BE nodes to determine the weight of FE nodes and the weight of BE nodes, using the weight of FE nodes to weight the load indicators of FE nodes, using the weight of BE nodes to weight the load indicators of BE nodes, and using the weighted multiple load indicators of FE nodes and multiple load indicators of BE nodes to determine the load values ​​of FE nodes and BE nodes.

[0226] The weight corresponding to each load indicator is determined by:

[0227] According to each load index, the entropy weight method and the analytic hierarchy process are used to determine the weight corresponding to the entropy weight method and the weight corresponding to the analytic hierarchy process;

[0228] The weights corresponding to the entropy weight method and the weights corresponding to the hierarchical analysis method are normalized to obtain the weight corresponding to each load indicator.

[0229] In detail, the entropy weight method and the hierarchical analysis method are used to calculate the two types of weights respectively. The purpose is to use the entropy weight method to avoid the influence of subjective factors and determine the weights according to the characteristics of the data itself, so that the results are more objective and reliable; the hierarchical analysis method is used to increase expert judgment, which not only takes into account quantitative data, but also incorporates the subjective judgment of experts into the weight calculation.

[0230] The specific method of using the entropy weight method to determine the weight corresponding to the load index of each node is as follows:

[0231] Calculate the information entropy of each load indicator to reflect its uncertainty. The formula for information entropy is:

[0232]

[0233] Where: p ij is the proportion of the normalized value of the i-th node on the j-th load index to the total normalized value of the load index: k is a constant, usually 1 / ln(n), where n is the number of samples.

[0234] According to the information entropy of each load indicator, the weight of each load indicator is calculated. The weight calculation formula is:

[0235]

[0236] Where: w j is the weight of the jth indicator, H jIt is the entropy value of the information entropy of the j-th load indicator. The smaller the entropy value (that is, the greater the indicator change), the greater the weight.

[0237] The specific method of using hierarchical analysis to determine the weight corresponding to the load index of each node is as follows:

[0238] First, construct the judgment matrix;

[0239] Specifically, a judgment matrix is ​​constructed by comparing each load indicator pairwise. The matrix containing the host core hardware and service (25) indicators is as follows: Figure 5 As shown in the figure, matrix explanation: The diagonal elements of the matrix are all 1, because the relative importance of any indicator to itself is 1. For example, in the first row and second column of the matrix, A12 = 3, indicating that CPU utilization is 3 times more important than CPU utilization; while A21 = 1 / 3, indicating that CPU temperature is 3 times less important than CPU utilization.

[0240] Secondly, the eigenvector is determined based on the judgment matrix. For example, the geometric mean of each row of data in the judgment matrix is ​​calculated, and then the geometric mean of each row of data is normalized to obtain the eigenvector.

[0241] Normalize the eigenvector to get the weight of each load indicator of the hierarchical analysis method. The normalization method is to divide each element of the eigenvector by the sum of all elements of the eigenvector:

[0242]

[0243] Among them, v j is the jth element of the eigenvector, w j is the normalized weight.

[0244] Then, the weights are fused and normalized, and the two types of weights calculated by the entropy weight method and the hierarchical analysis method are weighted combined. The weighted average method is used here to combine the two types of weights. α is adjusted according to objective factors and expert experience. This time α is 0.4.

[0245]

[0246] in, is the weight determined by the entropy weight method, The weights determined by the AHP method are: is the weight after weight fusion.

[0247] In some embodiments, step 240 of determining a node that satisfies a first preset load value from a plurality of nodes includes:

[0248] If the node is the first node for receiving user requests and distributing database services, the first preset requirement is to be the first node with the smallest load value.

[0249] This method enables intelligent load balancing by precisely analyzing the hardware status and internal service indicators of each primary node to intelligently assess the primary node's load. This scientific and rational decision mechanism automatically identifies and selects the primary node that best suits the current query requirement, ensuring efficient utilization of system resources and balanced load distribution, resulting in more efficient and stable system response and performance assurance.

[0250] In some embodiments, step 240 of determining a node that satisfies a first preset load value from a plurality of nodes includes:

[0251] If the node is a second node for executing a database service, and the database service is a query service, the first preset requirement is that the second node in the cluster executes the query service submitted by the user;

[0252] If the node is a second node for executing database business, and the database business is an imported business, the first preset requirement is the target second node with the largest load value among at least one target second node; wherein, the target second node is the second node in the cluster that executes the query business submitted by the user.

[0253] Among them, the second node in the cluster that executes the query business submitted by the user, that is, the target second node is determined by Tablet distribution. Tablet is in Bigtable (distributed data storage system). Since a single table (Table) may store a lot of data, the reading and writing efficiency will be very low. Therefore, Bigtable divides the Table into fixed-size Tablets and uses them as the basic units of data storage and search.

[0254] For example, if the target second nodes are the second node A, the second node B, and the third node C, and the node is a second node for executing a database service, and the database service is a query service, then the second nodes that meet the first preset requirement are the second node A, the second node B, and the third node C.

[0255] For example, the load value of the third node C is the largest. If the node is the second node for executing database business, and the database business is an import business, then the second node that meets the first preset requirement is the third node C.

[0256] Furthermore, since the user's resource request will be checked after receiving the database task, when the resource limit proposed by the user exceeds the resource size of the second node, an error will be automatically reported, causing the database business processing to stop and manual error processing is required.

[0257] To avoid error reporting, an embodiment of the present invention further proposes: if the database service is a query service, then among the resource upper limit required by the user submitting the database service, the resource upper limit configured by the cluster for the user, and the resource size of at least one target second node that meets the second preset condition, if the resource size of at least one target second node that meets the third preset condition is the smallest, then the user's own resource upper limit is modified to a preset value determined based on the resource size of the target second node that meets the second preset condition.

[0258] If the database business is an imported business, then when the user's own resource upper limit exceeds the resource upper limit configured for the user by the cluster, and the load value in at least one target second node meets the third preset condition, the user's own resource upper limit will be modified to a preset value determined based on the resource size of the second node with the largest load value in at least one target second node.

[0259] The second preset condition is the minimum resource size of at least one target second node. The third preset condition is that the maximum load value of at least one target second node is greater than a preset load value corresponding to a medium-high load type.

[0260] In detail, obtain the resource limit and Tablet distribution of the user who submits the database business: obtain the user's connected FE-Leader node through the database business-related requests submitted by the user, and thus obtain the user's own resource limit. At the same time, according to the library table name in the database business-related requests, obtain the corresponding Tablet distribution.

[0261] The upper limit of the user's own resources is the upper limit of the idle CPU and the upper limit of the idle memory of the second node required by the user.

[0262] Use the Prometheus tool to obtain the current idle CPU and free memory of the second node from the second node.

[0263] For query tasks:

[0264] Rule 1: Determines whether the resource limit of the user submitting the database service exceeds the resource limit configured by the cluster for users submitting database services. If so, the user's resource limit is adjusted to the resource limit configured by the cluster for users submitting database services. If not, the user's resource limit remains unchanged.

[0265] Rule 2: Based on the modified result of Rule 1, determine whether the resource limit of the user submitting the database service exceeds the minimum resource size of at least one target second node. If so, modify the resource limit of the user submitting the database service to 50% of the minimum resource size of at least one target second node. If not, do not modify the resource limit of the user submitting the database service.

[0266] The resource limits required by the user include the second node CPU usage limit, the second node query memory usage limit, and the second node import memory usage limit.

[0267] For example, determining whether the resource limit of the user submitting the database service exceeds the resource limit configured by the cluster for the user submitting the database service includes:

[0268] Determine whether the idle CPU usage of the user submitting the database service exceeds the idle CPU usage configured by the cluster for the user submitting the database service, and determine whether the idle memory usage of the user submitting the database service exceeds the idle memory configured by the cluster for the user submitting the database service.

[0269] If any of these exceeds the limit, the excess resources will be modified to the resources configured by the cluster for the user submitting database services. For example, if the idle CPU limit exceeds the limit, the idle CPU limit of the user submitting database services will be modified to the idle CPU limit configured by the cluster for the user submitting database services. If the free memory limit exceeds the limit, the free memory limit of the user submitting database services will be modified to the free memory limit configured by the cluster for the user submitting database services.

[0270] Taking the second node specified to execute the database service as the second node A, the second node B, and the third node C as an example, determining whether the resource limit of the user submitting the database service exceeds the minimum resource size of the second node specified to execute the database service includes:

[0271] From the second node A, the second node B, and the third node C, the one with the smallest resource size is selected.

[0272] Determine whether the idle CPU limit of the user submitting the database business exceeds the minimum idle CPU limit; determine whether the idle memory limit of the user submitting the database business exceeds the minimum idle memory limit;

[0273] If any one of them exceeds, the resources of the user submitting the database service will be modified to 50% of the minimum resource size.

[0274] Selecting the smallest resource size from the second node A, the second node B, and the third node C includes obtaining the smallest idle CPU and the smallest idle memory from the idle CPUs and the idle memories of the second node A, the second node B, and the third node C. When the smallest resource size is the resource size of a second node, for example, the resource size of the second node A is the smallest resource size, the second node is selected.

[0275] For import tasks:

[0276] Rule 1: Determine whether the resource limit of the user submitting the database service exceeds the resource limit configured by the cluster for users submitting database services. If so, the resource limit of the user submitting the database service is modified to the resource limit configured by the cluster for users submitting database services. If not, the resource limit of the user submitting the database service remains unchanged.

[0277] Rule 2: Determine whether the maximum load value in at least one target second node is greater than the preset load value corresponding to the medium and high load types. If it is, the resource upper limit of the user submitting the database business will be modified to 50% of the resource size of the second node with the maximum load value in at least one target second node.

[0278] Exemplarily, rule one is the same as the query service, and reference may be made to the understanding of query service rule one.

[0279] In rule 2, taking at least one target second node as the second node A, the second node B, and the third node C as an example, the maximum load value among the at least one target second node, for example, the third node C, is selected.

[0280] Determine whether the third node C is greater than the preset load value corresponding to the medium and high load type, where the preset load value corresponding to the cluster of the medium and high load type is 3. Then, it is necessary to determine whether the third node C is greater than 3. If it is greater, the resource upper limit of the user submitting the database business will be modified to 50% of the resource size of the third node C.

[0281] As described above, the embodiments of the present invention enable precise dynamic resource allocation. Based on the user group resource limit, the distribution of tablets in the library table, the idle CPU and memory of the second node, and the load value of the second node, after a multi-dimensional comprehensive evaluation, business resources are accurately allocated. This strategy significantly reduces reliance on manual intervention while ensuring query stability and efficiency, achieving automated and intelligent resource management, demonstrating the flexibility and efficiency of dynamic resource allocation, and improving system stability and overall performance.

[0282] Combine Figure 6 As shown, in the practice process, the specific embodiments are as follows:

[0283] S610: Obtain load indicators of EF nodes and BE nodes.

[0284] For example, a StarRocks cluster has 9FE and 30BE specifications. By deploying Node Exporter and custom Exporter on FE and BE nodes, it collects 18 types of load indicators related to node hardware, such as CPU, GPU, memory bar, disk, and network card. Prometheus periodically (with controllable frequency) pulls the indicators collected by the Exporter and stores them in the internal time series database TSDB (which stores the last 30 days of data by default). At the same time, in the Prometheus configuration file, configure the IP and HTTP port of each FE and BE, set the collection period (with controllable frequency), and Prometheus will collect load indicators related to FE and BE services and store them in the time series database TSDB, totaling 7 types.

[0285] S620: Determine the load values ​​of the EF nodes and BE nodes in the SR cluster based on the load indicators of the EF nodes and BE nodes. The determination method can refer to the above content.

[0286] S630: Receive a request submitted by a user, and identify the task type and library table name of the submitting user from the request submitted by the user.

[0287] For example, the request submitted by the user may be a SQL (Structured Query Language) request.

[0288] S640: Based on the task type and library table name submitted by the user, obtain the resource limit and Tablet distribution required by the user through the FE-Leader node, and obtain the current idle CPU and idle memory of the BE node through Prometheus.

[0289] S650: Generate an optimal resource application plan based on the user's required resource limit, tablet distribution, free CPU and free memory of the BE node, and the load value of the BE node;

[0290] The resource application plan includes the modified resource upper limit of the user submitting the database service and the second node processing the database service.

[0291] S660: Select the first node with the lowest load according to the load value of the FE node, and submit the database task to the corresponding FE node.

[0292] like Figure 7 As shown, the present invention also provides a device for selecting a database service processing node in a cluster, comprising:

[0293] An acquisition module 710 is configured to acquire load indicators affecting load sizes of multiple nodes for processing database services;

[0294] A clustering module 720 is configured to cluster multiple nodes according to their load indicators to obtain multiple types of clusters; wherein the types of clusters are determined according to the load types;

[0295] A determination module 730 is configured to determine load values ​​of multiple nodes according to preset load values ​​corresponding to the types of the multiple clusters;

[0296] The selection module 740 is configured to determine a node that meets a first preset load value requirement from among multiple nodes, and allocate the database service to the determined node so that the determined node processes the database service.

[0297] Optionally, the clustering module 720 is specifically configured to:

[0298] Determining a correlation between load indicators of the plurality of nodes and reference values ​​of the plurality of virtual nodes corresponding to each load indicator;

[0299] Determining an association degree between each node and each load type based on an association degree between at least a portion of the load indicator of each node and reference values ​​of a plurality of virtual nodes corresponding to at least a portion of the load indicators; wherein the type of the virtual node is the same as the load type;

[0300] Determine the absolute load matrix of each node based on the association degree between each node and each load type and the weight corresponding to each load type;

[0301] The absolute load matrices of multiple nodes are processed by fuzzy clustering to obtain various types of clusters.

[0302] Optionally, the clustering module 720 is further configured to:

[0303] determining a difference between load indicators of the plurality of nodes and a reference value of the plurality of virtual nodes corresponding to each load indicator;

[0304] For the difference between each load value and each reference value, the correlation between the load indicators of multiple nodes and the reference values ​​of multiple virtual nodes corresponding to each load indicator is determined based on the maximum and minimum values ​​of the differences between the load indicators of multiple nodes and the reference values ​​of multiple virtual nodes corresponding to each load indicator, as well as the differences.

[0305] Optionally, the clustering module 720 is further configured to:

[0306] determining a degree of association between each node and each virtual node based on a degree of association between at least a portion of the load indicator of each node and reference values ​​of a plurality of virtual nodes corresponding to at least a portion of the load indicator;

[0307] The association degree of each node with each load type is determined based on the association degree of each node with each type of virtual node.

[0308] Optionally, the determination module 730 is specifically configured to:

[0309] Determining load values ​​of the plurality of nodes according to preset load values ​​corresponding to the types of the plurality of clusters and the product of the membership degrees of the plurality of nodes and the plurality of types of clusters;

[0310] For any node and any type of cluster, the membership of the node to the cluster is determined in the following manner:

[0311] Determine a plurality of first distances of the node; wherein any one of the first distances is the distance from each element in the absolute load matrix of the node to the cluster center of any type of cluster;

[0312] The membership of the node to the cluster is determined according to the ratio between the multiple first distances of the node and the second distance of the node; wherein the second distance is the distance from each element in the absolute load matrix of the node to the cluster center of the cluster.

[0313] Optionally, the selection module 740 is specifically configured to:

[0314] If the node is the first node for receiving user requests and allocating database services, the first preset requirement is the first node with the smallest load value;

[0315] If the node is a second node for executing a database service, and the database service is a query service, the first preset requirement is the second node in the cluster that executes the query service submitted by the user;

[0316] If the node is a second node for executing database business, and the database business is an imported business, then the first preset requirement is the target second node with the largest load value among at least one target second node; wherein, the target second node is the second node in the cluster that executes the query business submitted by the user.

[0317] Optionally, the selection module 740 is further configured to:

[0318] If the database service is a query service, then when the resource size of at least one target second node that meets the third preset condition is the smallest among the resource upper limit required by the user submitting the database service, the resource upper limit configured by the cluster for the user, and the resource size of at least one target second node that meets the second preset condition, the user's resource upper limit is modified to a preset value determined based on the resource size of the target second node that meets the second preset condition among the at least one target second node;

[0319] If the database business is an imported business, then when the user's own resource upper limit exceeds the resource upper limit configured for the user by the cluster, and the load value in at least one target second node meets the third preset condition, the user's own resource upper limit is modified to a preset value determined based on the resource size of the second node with the largest load value in at least one target second node.

[0320] The module division of the device provided by the present invention is only exemplary and cannot be used as a specific limitation of the present invention. In addition to the above module division, the embodiment of the present invention can also be divided into other functional modules according to function, for example, Figure 8 As shown, the apparatus provided by the embodiment of the present invention includes an analysis module 810, a resource calculation module 820, a load balancing module 830, and a load calculation module 840;

[0321] The load calculation module 840 is used to obtain load indicators that affect the load size of multiple first nodes and second nodes used to process database business; cluster the first nodes and second nodes according to the load indicators of the multiple first nodes and second nodes to obtain multiple types of cluster clusters; determine the load values ​​of the first nodes and second nodes according to the preset load values ​​corresponding to the types of the multiple cluster clusters.

[0322] The analysis module 810 is used to obtain the resource upper limit and tablet distribution of the user who submits the database service;

[0323] The resource calculation module 820 is used to obtain the resource size of at least one target second node through the Prometheus tool, and determine at least one target second node according to the Tablet distribution. If the database business is a query business, then among the resource upper limit required by the user submitting the database business, the resource upper limit configured by the cluster for the user submitting the database business, and the resource size of at least one target second node that meets the second preset condition, when the resource size of at least one target second node that meets the second preset condition is the smallest, the resource upper limit of the user submitting the database business itself will be modified to a preset value determined according to the resource size of at least one target second node that meets the second preset condition; and the at least one target second node will be used as the second node for executing the query business;

[0324] If the database business is an imported business, then when the resource upper limit of the user submitting the database business exceeds the resource upper limit configured by the cluster for the user submitting the database business, and the load value in at least one target second node meets the third preset condition, the resource upper limit of the user submitting the database business will be modified to a preset value determined based on the resource size of the second node with the largest load value in at least one target second node, and the second node with the largest load value in at least one target second node will be used as the second node for executing the query business.

[0325] The load balancing module 830 is configured to determine a first node having a load value that meets a first preset requirement from among a plurality of first nodes, and output the database service to the first node.

[0326] In addition, combined Figures 1-8 The method and apparatus for selecting a database service processing node in a cluster according to the embodiments of the present invention described above may be implemented by an electronic device.

[0327] Electronic devices, including: processors;

[0328] a memory for storing instructions executable by the processor;

[0329] The processor is configured to execute the instructions to implement the method for selecting a database service processing node in a cluster as described in any one of the above-mentioned methods.

[0330] Based on the above introduction, for example, Figure 9 electronic equipment structure.

[0331] The electronic device may include a processor 910 and a memory 920 storing computer program instructions.

[0332] Specifically, the processor 910 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured as one or more integrated circuits for implementing the embodiments of the present invention.

[0333] The memory 920 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 920 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 920 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 920 may be inside or outside the data processing device. In a specific embodiment, the memory 920 is a non-volatile solid-state memory. In a specific embodiment, the memory 920 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0334] The processor 910 implements any one of the methods for performing tasks in the above embodiments by reading and executing computer program instructions stored in the memory 920 .

[0335] In one example, the electronic device may further include a communication interface 930 and a bus 940. Figure 9 As shown, the processor 910 , the memory 920 , and the communication interface 930 are connected via a bus 940 and communicate with each other.

[0336] The communication interface 930 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present invention.

[0337] Bus 940 comprises hardware, software or both, couples the parts of electronic equipment to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 940 can comprise one or more buses.Although the embodiment of the present invention describes and shows specific bus, the present invention considers any suitable bus or interconnection.

[0338] The electronic device can execute the method for selecting database service processing nodes in a cluster according to the embodiment of the present invention based on the received task, thereby realizing the combination Figures 1-8 The invention describes a method and device for selecting a database service processing node in a cluster.

[0339] The electronic device provided by the present invention proposes to select a first node, a second node, and a resource upper limit required by a user, and then send its business, the second node, and the resource upper limit required by the user to the first node. The first node parses the database business and determines whether the resource upper limit required by the user is within a preset range. If so, the database business is sent to the second node, and the second node executes the database business.

[0340] The present invention provides another electronic device and a first node; the electronic device selects the first node according to the method provided by the present invention and sends the service to the first node; the first node selects the second node according to the method provided by the present invention and modifies the resource upper limit required by the user, and determines whether the resource upper limit required by the user is within a preset range. If so, the database service is sent to the second node, and the second node executes the database service.

[0341] In addition, in combination with the electronic device in the above embodiments, an embodiment of the present invention may provide a storage medium, which, when the instructions in the storage medium are executed by the processor of the electronic device, enables the electronic device to execute the database business processing node selection method in the cluster as described in any one of the above items.

[0342] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0343] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0344] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0345] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0346] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for selecting database service processing nodes in a cluster, characterized in that: include: Obtain load indicators that affect load sizes of multiple nodes used to process database services; Clustering the multiple nodes according to their load indicators to obtain multiple types of clusters; wherein the types of the clusters are determined according to the load types; Determining load values ​​of multiple nodes according to preset load values ​​corresponding to the types of the multiple clusters; A node that meets a first preset load requirement is determined from a plurality of nodes, and the database service is allocated to the determined node so that the determined node processes the database service.

2. The method for selecting a database service processing node in a cluster according to claim 1, wherein: Based on the load indicators of multiple nodes, multiple nodes are clustered to obtain various types of clusters, including: Determining a correlation between load indicators of the plurality of nodes and reference values ​​of the plurality of virtual nodes corresponding to each load indicator; Determining an association degree between each node and each load type based on an association degree between at least a portion of the load indicator of each node and reference values ​​of a plurality of virtual nodes corresponding to at least a portion of the load indicators; wherein the type of the virtual node is the same as the load type; Determine the absolute load matrix of each node based on the association degree between each node and each load type and the weight corresponding to each load type; The absolute load matrices of multiple nodes are processed by fuzzy clustering to obtain various types of clusters.

3. The method for selecting a database service processing node in a cluster according to claim 2, wherein: Determining a correlation between load indicators of the plurality of nodes and reference values ​​of the plurality of virtual nodes corresponding to each load indicator includes: determining a difference between load indicators of the plurality of nodes and a reference value of the plurality of virtual nodes corresponding to each load indicator; For the difference between each load value and each reference value, the correlation between the load indicators of multiple nodes and the reference values ​​of multiple virtual nodes corresponding to each load indicator is determined based on the maximum and minimum values ​​of the differences between the load indicators of multiple nodes and the reference values ​​of multiple virtual nodes corresponding to each load indicator, as well as the differences.

4. The method for selecting a database service processing node in a cluster according to claim 2, wherein: Determining, based on a correlation between at least a portion of the load indicators of each node and reference values ​​of a plurality of virtual nodes corresponding to at least a portion of the load indicators, a correlation degree between each node and each load type, includes: determining a degree of association between each node and each virtual node based on a degree of association between at least a portion of the load indicator of each node and reference values ​​of a plurality of virtual nodes corresponding to at least a portion of the load indicator; The association degree of each node with each load type is determined based on the association degree of each node with each type of virtual node.

5. The method for selecting a database service processing node in a cluster according to claim 2, wherein: Determining load values ​​of multiple nodes according to preset load values ​​corresponding to the types of the multiple clusters includes: Determining load values ​​of the plurality of nodes according to preset load values ​​corresponding to the types of the plurality of clusters and the product of the membership degrees of the plurality of nodes and the plurality of types of clusters; For any node and any type of cluster, the membership of the node to the cluster is determined in the following manner: Determine a plurality of first distances of the node; wherein any one of the first distances is the distance from each element in the absolute load matrix of the node to the cluster center of any type of cluster; The membership of the node to the cluster is determined according to the ratio between the multiple first distances of the node and the second distance of the node; wherein the second distance is the distance from each element in the absolute load matrix of the node to the cluster center of the cluster.

6. The method for selecting a database service processing node in a cluster according to any one of claims 1 to 5, characterized in that: Determining, from the plurality of nodes, a node having a load value that meets a first preset requirement, includes: If the node is the first node for receiving user requests and allocating database services, the first preset requirement is the first node with the smallest load value; If the node is a second node for executing a database service, and the database service is a query service, the first preset requirement is the second node in the cluster that executes the query service submitted by the user; If the node is a second node for executing database business, and the database business is an imported business, then the first preset requirement is the target second node with the largest load value among at least one target second node; wherein, the target second node is the second node in the cluster that executes the query business submitted by the user.

7. The method for selecting a database service processing node in a cluster according to claim 6, wherein: The method further comprises: If the database service is a query service, then when the resource size of at least one target second node that meets the third preset condition is the smallest among the resource upper limit required by the user submitting the database service, the resource upper limit configured by the cluster for the user, and the resource size of at least one target second node that meets the second preset condition, the user's resource upper limit is modified to a preset value determined based on the resource size of the target second node that meets the second preset condition among the at least one target second node; If the database business is an imported business, then when the user's own resource upper limit exceeds the resource upper limit configured for the user by the cluster, and the load value in at least one target second node meets the third preset condition, the user's own resource upper limit is modified to a preset value determined based on the resource size of the second node with the largest load value in at least one target second node.

8. A device for selecting a database service processing node in a cluster, characterized in that: include: An acquisition module, configured to acquire load indicators affecting load sizes of multiple nodes used to process database services; A clustering module, configured to cluster multiple nodes according to their load indicators to obtain multiple types of clusters; wherein the types of clusters are determined according to the load types; A determination module, configured to determine load values ​​of a plurality of nodes according to preset load values ​​corresponding to types to which a plurality of clusters belong; The selection module is configured to determine a node that meets a first preset load requirement from among a plurality of nodes, and allocate the database service to the determined node so that the determined node processes the database service.

9. An electronic device, characterized in that: include: Memory, used to store computer programs or instructions; The processor is configured to execute the computer program or instructions in the memory so that the method for selecting a database service processing node in a cluster as claimed in any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method for selecting a database service processing node in a cluster as described in any one of claims 1 to 7.

Images