A distributed large-scale parallel processing database load balancing management system
Through the distributed collaborative service platform, the optimal nodes are filtered out in the distributed large-scale parallel processing database, which solves the load skew and single-point failure problems in load balancing management, improves system performance and reliability, and saves network resources.
Patent Information
- Application Number
- CN202410953863.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-17
AI Technical Summary
The existing distributed large-scale parallel processing database load balancing management system has problems such as load slop, high network resource costs and single point failures, and cannot accurately monitor the cluster load, resulting in insufficient load evaluation and difficulty in achieving dynamic adjustment.
Adopt a distributed collaborative service platform, through node deployment modules, first filter modules and second filter modules, based on node load information and focus on feature dimension requirements, the optimal nodes are selected for connection management, to achieve load balancing and avoid additional Network IO overhead and single point of failure.
It realizes flexible and reliable load balancing management of distributed large-scale parallel processing databases, improves cluster performance, saves network resource costs, and reduces the probability of service loss in the load balancing management system.
Smart Images

Figure CN118502966B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a distributed large-scale parallel processing database load balancing management system. Background Art
[0002] Currently, there are two main solutions for connection management of distributed large-scale parallel processing databases:
[0003] Connection string: List the IP and port information of all nodes, randomly select one of the service nodes to connect to, and if the node connection fails (node abnormality or downtime), reselect the service node. This solution is simple to implement, but the node screening strategy is fixed and only supports random selection. The components cannot guarantee even distribution of connections, which is prone to load skew, and cannot guarantee that the connection node selected each time is the optimal choice to meet the requirements of the corresponding node connection request (such as node response speed, number of node connections, data processing performance, etc.).
[0004] SQL Proxy: The query requests from the client and the result returned by the server are uniformly forwarded through the SQL proxy server. The SQL proxy can monitor the forwarded requests, evaluate the load of each service node, and achieve load balancing. However, request forwarding means that additional network IO overhead is required, which increases the cost of network resources to a certain extent. Secondly, the proxy can only monitor the requests forwarded by it, and cannot monitor other requests, so there will be a deviation between its load statistics and the actual situation, resulting in inaccurate load assessment. Finally, the proxy is generally a single-node system. If the proxy server is abnormal or down (single point failure), it will cause service loss.
[0005] In summary, it is necessary to develop a flexible and reliable load balancing management system for distributed large-scale parallel processing databases, accurately monitor the real-time load of the cluster, make dynamic adjustments, achieve load balancing of distributed large-scale parallel processing databases, and improve cluster performance.
[0006] Therefore, the present invention proposes a distributed large-scale parallel processing database load balancing management system. Summary of the invention
[0007] The present invention provides a distributed large-scale parallel processing database load balancing management system, which is used to use a distributed collaborative service platform to accurately monitor node load information in a node cluster, and considers the requirements of node connection requests for each focused feature dimension and load balancing in the node cluster to reasonably determine the optimal node for the current required connection, that is, it provides flexible and reliable node connection management for distributed large-scale parallel processing databases to achieve load balancing; and by considering the requirements of node connection requests for each focused feature dimension to screen connection nodes, the cluster performance is improved, and no additional network IO is required, thus saving network resource costs, and there is no single node system forwarding and processing node connection requests, thus reducing the service loss probability of the load balancing management system.
[0008] The present invention provides a distributed large-scale parallel processing database load balancing management system, comprising:
[0009] A node deployment module is used to treat the distributed large-scale parallel processing database as a service node, and deploy the service node and all nodes in the corresponding node cluster to the distributed collaborative service platform;
[0010] The first screening module is used to screen out all connectable nodes of the service node in the node cluster based on the latest load information of each node in the node cluster in the distributed collaborative service platform when receiving a node connection request from the service node;
[0011] A second screening module, used to screen out the best node in the node cluster based on the demand degree of the node connection request for each focused feature dimension and / or the compliance degree of each connectable node for each focused feature dimension;
[0012] The connection request response module is used to obtain a node connection request response from a service node based on the optimal node.
[0013] Preferably, the node deployment module includes:
[0014] A node registration submodule is used to treat the distributed large-scale parallel processing database as a service node and register the service node and all nodes in the corresponding node cluster to the distributed collaborative service platform;
[0015] The information update submodule is used to collect statistics on the node information of the service node and the node information of all nodes in the node cluster in real time, and to update the node information of the service node and the node information of all nodes in the node cluster to the distributed collaborative service platform at regular intervals.
[0016] Preferably, the first screening module includes:
[0017] The latest node information acquisition submodule is used to obtain the latest load information of each node in the node cluster from the distributed collaborative service platform when receiving a node connection request from the service node;
[0018] A node connection number determination submodule is used to determine the node connection number of each node in the node cluster based on the latest load information of each node;
[0019] The connectable node screening submodule is used to treat all nodes in the node cluster whose node connection numbers do not exceed the average value of the node connection numbers of all nodes in the node cluster as all connectable nodes of the service node.
[0020] Preferably, the second screening module includes:
[0021] A first demand analysis submodule is used to determine the demand of the node connection request for each focused feature dimension;
[0022] The conformity analysis submodule is used to retrieve the conformity of each connectable node to each focused feature dimension;
[0023] A first priority evaluation submodule is used to evaluate the priority of each connectable node in terms of demand based on the compliance of each connectable node with each focused feature dimension and the demand of the node connection request with each focused feature dimension;
[0024] A second priority evaluation submodule is used to evaluate the load balancing priority of each connectable node based on the latest load information of all connectable nodes;
[0025] The first node screening submodule is used to take the sum of the demand priority and the load priority of each connectable node as the priority of each connectable node, and take the connectable node corresponding to the maximum priority in the node cluster as the optimal node.
[0026] Preferably, the first demand analysis submodule includes:
[0027] a demand contribution value determining unit, configured to determine the performance characteristic information of all performance characteristic items of each focused characteristic dimension in the node connection request, retrieve the performance characteristic information-demand contribution value table of the corresponding performance characteristic item of the corresponding focused characteristic dimension based on each performance characteristic information, and obtain the demand contribution value of each performance characteristic information;
[0028] The demand degree determining unit is used to take the sum of the demand degree contribution values of the performance feature information of all performance feature items of each focused feature dimension possessed by the node connection request as the demand degree of the node connection request for each focused feature dimension.
[0029] Preferably, the first priority evaluation submodule includes:
[0030] A sub-preferential degree determination unit, used to take the product of the conformity of each connectable node to each emphasized feature dimension and the demand of the node connection request to the corresponding emphasized feature dimension as the sub-preferential degree of each connectable node in each emphasized feature dimension;
[0031] The demand aspect preference determination unit is used to take the sum of the sub-preferences of each connectable node in all feature dimensions as the demand aspect preference of each connectable node.
[0032] Preferably, the second priority evaluation submodule includes:
[0033] A node connection number determination unit, configured to determine the node connection number of each connectable node based on the latest load information of each connectable node;
[0034] A sorting ordinal number determination unit is used to sort all connectable nodes from small to large according to the number of node connections to obtain a sorting ordinal number for each connectable node;
[0035] The load balancing preference determining unit is used to take the quotient of the ranking number of each connectable node and the total number of all connectable nodes as the load balancing preference of each connectable node.
[0036] Preferably, the second screening module includes:
[0037] The second demand analysis submodule is used to determine the demand of the node connection request for each focused feature dimension;
[0038] The emphasis balance determination submodule is used to determine the emphasis balance of the node connection request in all emphasis feature dimensions based on the demand of the node connection request for each emphasis feature dimension;
[0039] A node screening strategy determination submodule is used to determine the current connection node screening strategy based on the emphasis balance of the node connection request in all emphasis feature dimensions;
[0040] The second node screening submodule is used to screen out the best node in the node cluster based on the node screening strategy.
[0041] Preferably, the focus is on the balance determination submodule, including:
[0042] The average demand determination unit is used to determine the average of the demand of the node connection demand for all the focused feature dimensions as the average demand;
[0043] The emphasis imbalance determination unit is used to take the quotient of the absolute value of the difference between the demand degree of the node connection request for each emphasized feature dimension and the average demand degree and the average demand degree as the emphasis imbalance degree of the node connection request for each emphasized feature dimension;
[0044] The emphasis balance determination unit is used to take the difference between 1 and the sum of the emphasis imbalances of the node connection request on all emphasis feature dimensions as the emphasis balance of the node connection request on all emphasis feature dimensions.
[0045] Preferably, the node screening strategy determination submodule includes:
[0046] A first strategy determination unit is used to use a random strategy as a current connection node screening strategy when the emphasis balance of the node connection request in all emphasis feature dimensions is approximately equal;
[0047] The second strategy determination unit is used to use the node screening strategy corresponding to the maximum emphasis balance degree of the service node among all emphasis balance degrees of the emphasis feature dimensions as the current connection node screening strategy when the emphasis balance degrees of the node connection requests in the emphasis feature dimensions are not approximately equal.
[0048] The beneficial effects of the present invention compared with the prior art are as follows: accurate monitoring of node load information in a node cluster is achieved by utilizing a distributed collaborative service platform, and the optimal node for the current connection demand is reasonably determined taking into account the requirements of node connection requests for each focused feature dimension and load balancing in the node cluster, that is, flexible and reliable node connection management is provided for distributed large-scale parallel processing databases to achieve load balancing; and by screening connection nodes taking into account the requirements of node connection requests for each focused feature dimension, cluster performance is improved, and no additional network IO is required, thus saving network resource costs, and there is no single node system to forward and process node connection requests, thus reducing the probability of service loss in the load balancing management system.
[0049] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in this application document.
[0050] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0052] Figure 1A schematic diagram of a distributed large-scale parallel processing database load balancing management system in an embodiment of the present invention;
[0053] Figure 2 This is a diagram showing the execution architecture of a distributed large-scale parallel processing database load balancing management system in an embodiment of the present invention;
[0054] Figure 3 is a flowchart of query request processing in an embodiment of the present invention;
[0055] Figure 4 The figure is a schematic diagram of the node deployment process in an embodiment of the present invention. DETAILED DESCRIPTION
[0056] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0057] Embodiment 1:
[0058] The present invention provides a distributed large-scale parallel processing database load balancing management system, referring to Figure 1 , 2 , 3, including:
[0059] The node deployment module is used to treat the distributed large-scale parallel processing database as a service node and to associate the service node with the corresponding node cluster (i.e. Figure 2 All nodes in the Clickhouse cluster in the distributed collaborative service platform are deployed;
[0060] The first screening module is used to screen out all connectable nodes of the service node in the node cluster based on the latest load information of each node in the node cluster in the distributed collaborative service platform when receiving a node connection request from the service node;
[0061] The second screening module is used to screen out the best node (i.e. Figure 3 );
[0062] The connection request response module is used to obtain a node connection request response from a service node based on the optimal node.
[0063] In this embodiment, the distributed large-scale parallel processing database is a database that can use a distributed processing method to execute large-scale data processing tasks in parallel.
[0064] In this embodiment, the node cluster corresponding to the service node is a node with data processing function that can be connected to the service node and share the data processing task of the service node.
[0065] In this embodiment, the distributed collaborative service platform is Zookeeper.
[0066] In this embodiment, the node connection request is a request issued by the service node, including information such as data tasks that need to be processed next by the connecting node, and is used to request to connect to a node in the node cluster and let it share the data tasks to be processed next.
[0067] In this embodiment, the latest load information of a node is the most recently acquired number of other nodes to which the node is connected.
[0068] In this embodiment, the feature dimension focuses on the node feature aspects that are considered when selecting nodes, or the node advantage features required by the service node, such as: the response speed of the node (for example, Figure 2 The average response time in (expressed in ), number of node connections, data processing performance, etc.
[0069] In this embodiment, the demand degree of the node connection request for each emphasized feature dimension indicates the degree to which the node connection request requires each emphasized feature dimension.
[0070] In this embodiment, the compliance degree of the connectable node to each emphasized feature dimension indicates the compliance degree of the connectable node to each emphasized feature dimension of the node connection requirement.
[0071] In this embodiment, obtaining the node connection request response of the service node based on the optimal node is: establishing a connection relationship between the service node and the optimal node, that is, obtaining the node connection request response.
[0072] The beneficial effects of the above technology are: using the distributed collaborative service platform to accurately monitor the node load information in the node cluster, taking into account the node connection request for each focus feature dimension and the load balancing in the node cluster, and reasonably determining the optimal node for the current demand connection, that is, providing flexible and reliable node connection management for distributed large-scale parallel processing databases to achieve load balancing; and by considering the node connection request for each focus feature dimension to screen the connection nodes, the cluster performance is improved, and no additional network IO is required, saving network resource costs, and there is no single node system to forward and process node connection requests, reducing the probability of service loss in the load balancing management system.
[0073] Embodiment 2:
[0074] Based on Example 1, the node deployment module, refer to Figure 4 ,include:
[0075] A node registration submodule is used to treat the distributed large-scale parallel processing database as a service node and register the service node and all nodes in the corresponding node cluster to the distributed collaborative service platform;
[0076] The information update submodule is used to collect statistics on the node information of the service node and the node information of all nodes in the node cluster in real time, and to update the node information of the service node and the node information of all nodes in the node cluster to the distributed collaborative service platform at regular intervals.
[0077] In this embodiment, the node information of the service node includes, for example, the number of node connections and the IP address of the service node.
[0078] In this embodiment, the node information of the node includes, for example, the number of node connections, IP address, average response time, average data processing speed, current available running memory value, etc.
[0079] The beneficial effect of the above technology is: by registering the service node and all nodes in the corresponding node cluster to the distributed collaborative service platform, and regularly updating the node information of the service node and the node information of all nodes in the node cluster to the distributed collaborative service platform, accurate monitoring of the service node and the node information in the corresponding node cluster can be achieved.
[0080] Embodiment 3:
[0081] Based on Example 1, the first screening module includes:
[0082] The latest node information acquisition submodule is used to obtain the latest load information of each node in the node cluster from the distributed collaborative service platform when receiving a node connection request from the service node;
[0083] A node connection number determination submodule is used to determine the node connection number of each node in the node cluster based on the latest load information of each node;
[0084] The connectable node screening submodule is used to treat all nodes in the node cluster whose node connection numbers do not exceed the average value of the node connection numbers of all nodes in the node cluster as all connectable nodes of the service node.
[0085] In this embodiment, the node connection number of a node is the total number of other nodes connected to the node, and this parameter indicates the occupied load size of the node.
[0086] The beneficial effect of the above technology is: when a node connection request is received from a service node, the node connection number of each node in the node cluster obtained from the distributed collaborative service platform is compared with the average number of node connections of all nodes in the node cluster, and nodes with smaller node connection numbers are selected as all connectable nodes of the service node, thereby achieving preliminary filtering and screening of connection nodes to ensure load balancing of the node cluster.
[0087] Embodiment 4:
[0088] Based on Example 1, the second screening module includes:
[0089] A first demand analysis submodule is used to determine the demand of the node connection request for each focused feature dimension;
[0090] The conformity analysis submodule is used to retrieve the conformity of each connectable node to each focused feature dimension;
[0091] A first priority evaluation submodule is used to evaluate the priority of each connectable node in terms of demand based on the compliance of each connectable node with each focused feature dimension and the demand of the node connection request with each focused feature dimension;
[0092] A second priority evaluation submodule is used to evaluate the load balancing priority of each connectable node based on the latest load information of all connectable nodes;
[0093] The first node screening submodule is used to take the sum of the demand priority and the load priority of each connectable node as the priority of each connectable node, and take the connectable node corresponding to the maximum priority in the node cluster as the optimal node.
[0094] In this embodiment, the compliance of each connectable node to each emphasized feature dimension is retrieved from a preset list, wherein the preset list is a preset evaluated list containing the compliance of each connectable node to each emphasized feature dimension.
[0095] In this embodiment, the demand-side preference of the connectable node indicates the degree of priority of selecting the connectable node as the optimal node in terms of satisfying the demand for node connection request.
[0096] In this embodiment, the load balancing priority of the connectable node indicates the priority of selecting the connectable node as the optimal node from the perspective of load balancing of the node cluster.
[0097] In this embodiment, the priority of the connectable node indicates the priority of the connectable node to be selected as the optimal node when the two aspects of the demand for satisfying the node connection request and the load balancing of the node cluster are comprehensively considered.
[0098] The beneficial effects of the above technology are: by determining the demand degree of the node connection request for each emphasized feature dimension and the compliance degree of each connectable node for each emphasized feature dimension, the demand preference of each connectable node is further determined, and the load balancing preference of each connectable node is further evaluated based on the latest load information of all connectable nodes. Finally, the optimal node is selected by comprehensively considering the load balancing preference and the demand preference, thereby realizing the selection of the optimal node by comprehensively considering both the demand for meeting the node connection request and the load balancing of the node cluster.
[0099] Embodiment 5:
[0100] Based on Example 4, the first demand analysis submodule includes:
[0101] a demand contribution value determining unit, configured to determine the performance characteristic information of all performance characteristic items of each focused characteristic dimension in the node connection request, retrieve the performance characteristic information-demand contribution value table of the corresponding performance characteristic item of the corresponding focused characteristic dimension based on each performance characteristic information, and obtain the demand contribution value of each performance characteristic information;
[0102] The demand degree determining unit is used to take the sum of the demand degree contribution values of the performance feature information of all performance feature items of each focused feature dimension possessed by the node connection request as the demand degree of the node connection request for each focused feature dimension.
[0103] In this embodiment, for all performance feature items focusing on each feature dimension, for example, the performance feature items focusing on the feature dimension of "data processing speed of the node" include: the task processing volume of tasks to be processed, the task parallelism of tasks to be processed, the maximum real-time running memory required for tasks to be processed, etc.
[0104] In this embodiment, the performance characteristic information of the performance characteristic item is, for example: the performance characteristic information of the task processing amount of the pending tasks is the task processing amount of the pending tasks of the service node, and the performance characteristic information of the task parallelism of the pending tasks is the task parallelism of the pending tasks of the service node.
[0105] In this embodiment, the performance characteristic information-demand contribution value table is a pre-acquired list containing demand contribution values corresponding to different value ranges of each performance characteristic information.
[0106] In this embodiment, the demand contribution value of the characteristic information represents the contribution value of the node connection request to the demand of the corresponding emphasis characteristic dimension.
[0107] The beneficial effects of the above technology are: based on the performance feature information of all performance feature items for each focused feature dimension possessed by the node connection request, the demand degree of the node connection request for each focused feature dimension is quantified in detail, and the demand degree of the node connection request for each focused feature dimension is accurately determined.
[0108] Embodiment 6:
[0109] Based on Example 4, the first priority evaluation submodule includes:
[0110] A sub-preferential degree determination unit, used to take the product of the conformity of each connectable node to each emphasized feature dimension and the demand of the node connection request to the corresponding emphasized feature dimension as the sub-preferential degree of each connectable node in each emphasized feature dimension;
[0111] The demand aspect preference determination unit is used to take the sum of the sub-preferences of each connectable node in all feature dimensions as the demand aspect preference of each connectable node.
[0112] In this embodiment, the sub-preferential degree of the connectable node in each feature-focused dimension indicates the priority degree to which the connectable node should be selected as the optimal node from the perspective of a single feature-focused dimension.
[0113] The beneficial effects of the above technology are: based on the product of the compliance of each connectable node for each emphasized feature dimension and the demand of the node connection request for the corresponding emphasized feature dimension, the sub-preferential degree of each connectable node in each emphasized feature dimension is accurately calculated, and by summing the sub-preferential degrees of each connectable node in all feature dimensions, the preference degree in terms of demand of each connectable node is accurately calculated.
[0114] Embodiment 7:
[0115] Based on Example 4, the second priority evaluation submodule includes:
[0116] A node connection number determination unit, configured to determine the node connection number of each connectable node based on the latest load information of each connectable node;
[0117] A sorting ordinal number determination unit is used to sort all connectable nodes from small to large according to the number of node connections to obtain a sorting ordinal number for each connectable node;
[0118] The load balancing preference determining unit is used to take the quotient of the ranking number of each connectable node and the total number of all connectable nodes as the load balancing preference of each connectable node.
[0119] The beneficial effect of the above technology is: based on the sorting ordinal number obtained after sorting all connectable nodes according to the number of node connections of all connectable nodes, accurate quantification of the relative priority of the connectable nodes to be selected as the optimal node from the perspective of load balancing is achieved.
[0120] Embodiment 8:
[0121] Based on Example 1, the second screening module includes:
[0122] The second demand analysis submodule is used to determine the demand of the node connection request for each focused feature dimension;
[0123] The emphasis balance determination submodule is used to determine the emphasis balance of the node connection request in all emphasis feature dimensions based on the demand of the node connection request for each emphasis feature dimension;
[0124] A node screening strategy determination submodule is used to determine the current connection node screening strategy based on the emphasis balance of the node connection request in all emphasis feature dimensions;
[0125] The second node screening submodule is used to screen out the best node in the node cluster based on the node screening strategy.
[0126] In this embodiment, the specific steps of the second demand analysis submodule determining the demand of the node connection request for each focused feature dimension are exactly the same as the specific steps of the first demand analysis submodule, that is, they are implemented using the technical solution in Embodiment 5.
[0127] In this embodiment, the emphasis balance degree of the node connection request in all emphasis feature dimensions represents the numerical approximation degree of the demand degree of the node connection request in all emphasis feature dimensions.
[0128] In this embodiment, the current connection node screening strategy is a node screening strategy currently determined to be used as a reference when selecting the optimal node, for example: selecting a connectable node with the smallest average response time among all connectable nodes as the optimal node;
[0129] Alternatively, the connectable node with the smallest number of node connections among all connectable nodes is selected as the optimal node.
[0130] The beneficial effects of the above technology are: based on the analyzed imbalance of the node connection request in all the emphasis feature dimensions, the current connection node screening strategy is reasonably determined, and based on the current connection node screening strategy, the optimal node that can comprehensively meet the multi-faceted needs of the node connection request to a high degree is reasonably screened.
[0131] Embodiment 9:
[0132] On the basis of Example 8, the focus is on the balance determination submodule, including:
[0133] The average demand determination unit is used to determine the average of the demand of the node connection demand for all the focused feature dimensions as the average demand;
[0134] The emphasis imbalance determination unit is used to take the quotient of the absolute value of the difference between the demand degree of the node connection request for each emphasized feature dimension and the average demand degree and the average demand degree as the emphasis imbalance degree of the node connection request for each emphasized feature dimension;
[0135] The emphasis balance determination unit is used to take the difference between 1 and the sum of the emphasis imbalances of the node connection request on all emphasis feature dimensions as the emphasis balance of the node connection request on all emphasis feature dimensions.
[0136] In this embodiment, the absolute value of the difference between the demand degree of the node connection request for each focused feature dimension and the average demand degree is the value obtained by taking the absolute value of the difference between the demand degree of the node connection request for each focused feature dimension and the average demand degree.
[0137] In this embodiment, the imbalance degree of emphasis of the node connection request in all emphasis feature dimensions indicates the degree of dissimilarity of the values of the demand degree of the node connection request in all emphasis feature dimensions.
[0138] The beneficial effects of the above technology are: based on the absolute value of the difference between the demand degree of the node connection request for each emphasized feature dimension and the average demand degree and the average demand degree of the node connection demand for all emphasized feature dimensions, the emphasis imbalance of the node connection request for each emphasized feature dimension is accurately calculated; and based on the emphasis imbalance of the node connection request for each emphasized feature dimension, the emphasis balance of the node connection request for each emphasized feature dimension is accurately calculated.
[0139] Embodiment 10:
[0140] Based on Example 8, the node screening strategy determination submodule includes:
[0141] A first strategy determination unit is used to use a random strategy as a current connection node screening strategy when the emphasis balance of the node connection request in all emphasis feature dimensions is approximately equal;
[0142] The second strategy determination unit is used to use the node screening strategy corresponding to the maximum emphasis balance degree of the service node among all emphasis balance degrees of the emphasis feature dimensions as the current connection node screening strategy when the emphasis balance degrees of the node connection requests in the emphasis feature dimensions are not approximately equal.
[0143] In this embodiment, the random strategy is to randomly select any connectable node in the node cluster as the optimal node.
[0144] In this embodiment, the approximately equal emphasis balance of the node connection request in all feature-focused dimensions means that the difference between any two emphasis balances of the node connection request in all feature-focused dimensions does not exceed 20% of their average (the average of the any two emphasis balances).
[0145] In this embodiment, the node screening strategy corresponding to the maximum emphasis on balance is the node screening strategy corresponding to the emphasis on feature dimension corresponding to the maximum emphasis on balance;
[0146] Among them, the node screening strategy corresponding to the focused feature dimension is the node screening strategy that performs well in the focused feature dimension (the node screening strategy is a pre-prepared strategy that contains only one screening condition (or considers one performance) and is used to screen out the required nodes), for example:
[0147] When the feature dimension focused on is “node response speed”, the connectable node corresponding to the minimum average response time is taken as the optimal node as the node screening strategy.
[0148] The beneficial effect of the above technology is: by determining whether the emphasis flatness of the node connection request in all emphasis feature dimensions is approximately equal, two situations for determining the current connection node screening strategy are divided, thereby achieving a reasonable determination of the current connection node screening strategy.
[0149] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A distributed large-scale parallel processing database load balancing management system, It is characterized in that include: A node deployment module is used to treat the distributed large-scale parallel processing database as a service node, and deploy the service node and all nodes in the corresponding node cluster to the distributed collaborative service platform; The first screening module is used to screen out all connectable nodes of the service node in the node cluster based on the latest load information of each node in the node cluster in the distributed collaborative service platform when receiving a node connection request from the service node; A second screening module, used to screen out the best node in the node cluster based on the demand degree of the node connection request for each focused feature dimension and / or the compliance degree of each connectable node for each focused feature dimension; A connection request response module, used to obtain a node connection request response from a service node based on an optimal node; Among them, the node deployment module includes: A node registration submodule is used to treat the distributed large-scale parallel processing database as a service node and register the service node and all nodes in the corresponding node cluster to the distributed collaborative service platform; The information update submodule is used to collect statistics on the node information of the service node and the node information of all nodes in the node cluster in real time, and to update the node information of the service node and the node information of all nodes in the node cluster to the distributed collaborative service platform at regular intervals; Among them, the first screening module includes: The latest node information acquisition submodule is used to obtain the latest load information of each node in the node cluster from the distributed collaborative service platform when receiving a node connection request from the service node; A node connection number determination submodule is used to determine the node connection number of each node in the node cluster based on the latest load information of each node; The connectable node screening submodule is used to treat all nodes in the node cluster whose node connection number does not exceed the average value of the node connection number of all nodes in the node cluster as all connectable nodes of the service node; Among them, the second screening module includes: A first demand analysis submodule is used to determine the demand of the node connection request for each focused feature dimension; The conformity analysis submodule is used to retrieve the conformity of each connectable node to each focused feature dimension; A first priority evaluation submodule is used to evaluate the priority of each connectable node in terms of demand based on the compliance of each connectable node with each focused feature dimension and the demand of the node connection request with each focused feature dimension; A second priority evaluation submodule is used to evaluate the load balancing priority of each connectable node based on the latest load information of all connectable nodes; A first node screening submodule is used to take the sum of the demand priority and the load priority of each connectable node as the priority of each connectable node, and take the connectable node corresponding to the maximum priority in the node cluster as the optimal node; The first demand analysis submodule includes: a demand contribution value determining unit, configured to determine the performance characteristic information of all performance characteristic items of each focused characteristic dimension in the node connection request, retrieve the performance characteristic information-demand contribution value table of the corresponding performance characteristic item of the corresponding focused characteristic dimension based on each performance characteristic information, and obtain the demand contribution value of each performance characteristic information; a demand degree determining unit, configured to take the sum of demand degree contribution values of the performance feature information of all performance feature items of each focused feature dimension possessed by the node connection request as the demand degree of the node connection request for each focused feature dimension; Among them, the first priority assessment submodule includes: A sub-preferential degree determination unit, used to take the product of the conformity of each connectable node to each emphasized feature dimension and the demand of the node connection request to the corresponding emphasized feature dimension as the sub-preferential degree of each connectable node in each emphasized feature dimension; A demand aspect preference determination unit, used to take the sum of the sub-preferences of each connectable node in all feature dimensions as the demand aspect preference of each connectable node; Among them, the second priority assessment submodule includes: A node connection number determination unit, configured to determine the node connection number of each connectable node based on the latest load information of each connectable node; A sorting ordinal number determination unit is used to sort all connectable nodes from small to large according to the number of node connections to obtain a sorting ordinal number for each connectable node; A load balancing preference determining unit, used to take the quotient of the ranking number of each connectable node and the total number of all connectable nodes as the load balancing preference of each connectable node; Among them, the feature-focused dimension refers to the node features that are considered when selecting nodes, or the node advantage features required by the service node, including: node response speed, number of node connections, and data processing performance; Among them, the demand degree of the node connection request for each focused feature dimension represents the degree of need of the node connection request for each focused feature dimension; The conformity of the connectable node to each emphasized feature dimension indicates the conformity of the connectable node to each emphasized feature dimension of the node connection requirement.
