TrinoCoordinator high-availability method

Abstract

The invention relates to the technical field of databases, in particular to a Tr inoCoord inator high availability method, which comprises the following steps that: Nginx receives a client query request transmitted by a Tr ino, and establishes a service agent node to be connected with each coordination node of the Tr ino based on the received client query request; the service agent node sends a state sharing task to each coordination node to obtain the cluster resource utilization rate of each Tr ino working node; based on the task response sequence of each coordination node and the current task execution rate of each coordination node, a plurality of alternative coordination nodes for executing the current query task are selected, and the number of the alternative coordination nodes is at least two; the service agent node sends the client query request to each alternative coordination node; and each alternative coordination node obtains a trigger condition from the isolated storage node based on the client query request, and the isolated storage node judges whether the alternative coordination node fails or not according to the update speed of the current query result.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a TrinoCoordinator high availability method. Background Art

[0002] Trino is a distributed QSQL query engine, a massively parallel processing (MPP) database and query engine. Trino achieves horizontal expansion by distributing processing tasks across the entire cluster of servers, rather than vertical expansion by improving the performance of a single server. Based on this architecture, the Tino query engine can process SQL queries on massive data in parallel on the computing nodes within the cluster. On each computing node, Trino runs as a single server process. A Trino cluster consists of multiple Trino nodes configured to collaborate with each other.

[0003] Chinese patent application publication number: CN112559022A discloses a Jenkins high-availability system and method, the system includes a task management module, which is used to receive task data created by users and persist the task data in a local database; a task scheduling module, which is used to call the Jenkins interface to use the task data as input to synchronously create a Jenkins task on an available Jenkins master node; write the task result data into the local database; a Jenkins management module, which is used to perform an abnormality check on the Jenkins master node; when an abnormal Jenkins master node is detected, check the Jenkins task being executed in the abnormal Jenkins master node. This application improves the availability and stability of the existing Jenkins high-availability solution.

[0004] It can be seen that the existing technology has the problem of low accuracy in real-time status analysis of task coordination nodes, which leads to failure to reasonably allocate query tasks, resulting in low cluster resource utilization and long query task response time. Summary of the invention

[0005] To this end, the present invention provides a TrinoCoordinator high availability method to overcome the problem in the prior art that the real-time status analysis of the task coordination node is of low accuracy, resulting in the inability to reasonably allocate query tasks, causing low cluster resource utilization and long query task response time.

[0006] To achieve the above object, on the one hand, the present invention provides a TrinoCoordinator high availability method, comprising:

[0007] Set up at least one service proxy node and at least one isolated storage node in Nginx, and use Nginx to proxy Trino's coordinator service, including:

[0008] Nginx receives the client query request transmitted by Trino, and establishes a connection between the service proxy node and each coordination node of Trino based on the received client query request;

[0009] The service proxy node sends a state sharing task to each of the coordination nodes to obtain the cluster resource utilization rate of each Trino working node;

[0010] Selecting a number of candidate coordination nodes for executing the current query task based on the task response order of each coordination node and the current task execution rate of each coordination node, wherein the number of the candidate coordination nodes is at least two;

[0011] The service proxy node sends the client query request to each of the candidate coordination nodes;

[0012] Each candidate coordination node obtains a trigger condition from the isolated storage node based on the client query request, wherein if the candidate coordination node finds that there is no query mapping corresponding to the client query request in the isolated storage node, the trigger condition is obtained to query the client query request;

[0013] In response to the trigger condition, the isolated storage node monitors and obtains the current query result of the corresponding candidate coordination node;

[0014] The isolated storage node determines whether the candidate coordination node is faulty according to the update speed of the current query result. If there is a fault, the isolated storage node generates an updated query mapping based on the stored client query request and the corresponding current query result and shares it with other candidate coordination nodes except the current candidate coordination node, so as to transmit the current query result to the client through other candidate coordination nodes;

[0015] Update the current task execution rate of each candidate coordination node based on the candidate number of times, the actual query execution number and the failure number of the candidate coordination node;

[0016] In response to the task completion status of the candidate coordination node, the isolation storage node saves the currently completed query mapping into the request comparison table and shares it with all coordination nodes.

[0017] Furthermore, a single candidate node constructs a query plan based on the trigger condition according to the client query request and distributes it to each working node for execution to obtain a request execution result, which is recorded as a current query result.

[0018] Furthermore, the selection of the candidate coordination node includes:

[0019] The service proxy node obtains the cluster resource usage rate data with timestamps transmitted by each of the coordination nodes;

[0020] Determine the task response order of the corresponding coordination node according to the sequence of each timestamp;

[0021] Obtaining the current task execution rate and preset task execution rate of each of the coordination nodes;

[0022] Eliminating the corresponding coordination nodes in the task response sequence based on a preset task execution rate;

[0023] After removing some coordination nodes, reorder the coordination nodes based on the task response order;

[0024] A preset number of coordination nodes in the reordering are selected as candidate coordination nodes.

[0025] Furthermore, the acquisition of the trigger condition also includes:

[0026] If the candidate coordination node finds that the query mapping corresponding to the client query request exists in the isolated storage node, the trigger condition is not obtained and the query of the client query request is not performed.

[0027] Furthermore, the step of determining whether the candidate coordination node is faulty includes:

[0028] The isolated storage node repeatedly monitors and obtains the current query result of the candidate coordination node of the trigger condition based on a preset period;

[0029] If the update speed of the current query result is less than or equal to the preset update speed, it is determined that the corresponding candidate coordination node is faulty.

[0030] Furthermore, the preset number is determined according to the average value of the task execution rate of each coordination node, and the preset number is negatively correlated with the average value of the task execution rate of each coordination node.

[0031] Furthermore, the current task execution rate of a single candidate coordination node is determined by the following formula:

[0032]

[0033] Among them, H represents the current task execution rate of a single candidate coordination node, N1 represents the actual query execution times, N2 represents the alternative times, α represents the influence coefficient of the task execution success rate on the task execution rate, Y represents the number of failures, Y0 represents the preset number of failures, and β represents the influence coefficient of the number of failures on the task execution rate.

[0034] Furthermore, each of the coordination nodes determines a delay time for transmitting the cluster resource usage rate data back to the service proxy node based on the execution percentage of the task being executed, wherein:

[0035] The execution percentage is inversely correlated to the lag time.

[0036] Furthermore, the query is mapped to a data packet with each client query request as the file name, and the data packet stores the query result corresponding to the client query request.

[0037] Furthermore, after obtaining the cluster resource utilization rate of each Trino working node, the service proxy node synchronously shares the received cluster resource utilization rate of each Trino working node with each coordination node so that the computing data of each coordination node remains consistent.

[0038] Compared with the prior art, the beneficial effect of the present invention lies in that the present invention selects Nginx as the front-end load balancing and proxy server to provide a unified access entrance for the client. The client does not need to directly interact with the various coordination nodes in the Trino cluster, which simplifies the use experience. Nginx can route the request to the appropriate Trino coordination node for processing according to the characteristics of the client request. At the same time, Nginx can realize load balancing distribution of the request according to the load of the coordination node, thereby improving the overall processing capacity of the cluster. Nginx can perform security review and access control on the client request to prevent illegal access and attacks, which helps to protect the security and stable operation of the Trino cluster. The service proxy node sends a status sharing task to the Trino coordination node, and can obtain the resource usage of each working node. These cluster status information can be used for Nginx's load balancing decision to improve the efficiency of request routing. Based on the collected cluster status information, Nginx can intelligently route the request to the coordination node with lower load for processing, thereby maximizing the overall response capability and resource utilization efficiency of the Trino cluster.

[0039] Furthermore, by collecting the real-time load status of the coordination nodes, the service proxy nodes of the present invention can dynamically perceive the usage status of the cluster resources. Based on this, the service proxy nodes can adjust the distribution of tasks among the coordination nodes to achieve a more balanced load distribution. According to the task response order and the preset execution rate, the service proxy nodes can give priority to the coordination nodes with lower load and faster response to process the query requests, which can shorten the delay of query processing and improve the overall query response efficiency. By selecting some prepared coordination nodes, the service proxy nodes can quickly switch to the backup nodes to continue to provide services when the main nodes fail, which can improve the availability and fault tolerance of the services and enhance the stability of the entire system. By dynamically adjusting the task allocation, the service proxy nodes can try to avoid idle or overused resources, which can improve the overall resource utilization efficiency of the cluster and reduce operating costs. By reserving some alternative nodes, the task allocation ratio can be quickly adjusted when the cluster is expanded or reduced to meet dynamic computing needs. This flexibility helps to better cope with changes in business needs.

[0040] Furthermore, the present invention determines whether to trigger the acquisition condition by judging whether there is a query mapping. If there is a query mapping, it means that the query has been executed before, and the result can be directly obtained from the cache without re-execution. This can greatly reduce unnecessary query execution and improve the overall query processing efficiency. By checking the query mapping, repeated query requests can be identified to avoid repeated calculation and processing of the same query, which not only improves efficiency, but also reduces the burden on cluster resources and improves the overall system performance. If there is a query mapping, the result data can be directly obtained from the cache and quickly responded to the client without going through the complete query execution process. This is very helpful for some query scenarios that require fast response and improves the user experience. By caching and reusing query results, unnecessary computing resource consumption can be reduced and the overall resource utilization efficiency of the cluster can be improved. This is particularly important for resource management in large-scale data processing scenarios. According to the existence of query mapping, incremental updates or differentiated processing can be performed in a targeted manner to avoid repeated calculation of the entire amount, which can further improve the efficiency and response speed of query processing.

[0041] Furthermore, the present invention can grasp the current operating status and computing power of the candidate nodes in real time by regularly checking the query result update speed of the candidate nodes, so that it can promptly discover whether the candidate nodes have faults or performance degradation. If the candidate node is found to be faulty or has poor performance, the node can be quickly removed from the candidate node list to avoid it affecting the overall service quality, which helps to maintain high availability of the service and ensure business continuity and stability. By comparing the query result update speed, the candidate nodes with better performance can be identified and they can be used to process query tasks preferentially, which can improve the overall query processing efficiency, shorten the response time, and improve the user experience. Once a candidate node is found to have a fault or performance degradation, it can be quickly removed from the load distribution. At the same time, the load weights of other healthy nodes can be appropriately increased to achieve dynamic load balancing adjustment. By monitoring the query result update speed of the candidate nodes, the system's operating indicator data can be collected, and these data help to further optimize the system configuration and improve overall performance and reliability.

[0042] Furthermore, the present invention can ensure that even if the current node fails, there are still other nodes that can take over the service and provide query results to the client by sharing the query mapping with other healthy alternative nodes. This helps to maintain the high availability of the entire system and avoid overall service interruption due to a single node failure. After other alternative nodes obtain the updated query mapping, they can directly pull the query results from the cache without re-executing the query. This can greatly shorten the query response time and improve the user experience. When the current alternative node fails, other nodes can quickly take over the service without the client being aware of the switching process. This can ensure service continuity and reduce the business impact caused by node failure. After sharing the query mapping with other nodes, the burden on the current failed node can be reduced and its recovery difficulty can be reduced. This helps the failed node to return to normal as soon as possible and rejoin the list of alternative nodes. By synchronizing the query mapping to other nodes in a timely manner, it can ensure that the entire system has self-healing capabilities and avoid overall service interruption due to single point failures. This helps to improve the reliability and stability of the system.

[0043] Furthermore, the present invention can predict resource requirements in the future by analyzing the current task execution progress of each node. The lag time of returning data can be adjusted according to this information to ensure that the service proxy node obtains more timely and accurate resource usage information, thereby better scheduling and allocating cluster resources and improving overall utilization efficiency. If the current task execution progress of a node is low, it means that its resource consumption is low and there is no need to frequently return usage rate data. The data return lag time of such node can be appropriately increased to reduce network communication overhead and overall network load. For nodes that are executing critical tasks, their task progress is high and resource consumption is also large, so they need to return to the service proxy node in a timely manner. Transmitting utilization data can shorten the data return lag time of these nodes and ensure that the service proxy nodes can obtain accurate resource usage information in the first time, so as to prioritize the scheduling of these key tasks. Timely acquisition of accurate cluster resource utilization data can enable service proxy nodes to perceive cluster status changes more quickly and make more timely and reasonable resource allocation decisions, which will help improve the responsiveness of the entire system and enhance the elasticity and stability of the service. Adjusting the data return lag time according to the progress of task execution at each node can allow service proxy nodes to grasp resource usage information closer to real time, which provides a basis for dynamically adjusting load distribution and helps to achieve dynamic and balanced utilization of cluster resources.

[0044] Furthermore, the present invention synchronizes the resource usage of each Trino working node to the coordination node in a timely manner through the service proxy node, which can ensure that all coordination nodes have the same cluster resource usage data to avoid data inconsistency. This helps the coordination nodes make more unified and correct resource scheduling decisions. When the coordination nodes have the resource usage data of the entire cluster, they can make more comprehensive and optimized resource allocation and scheduling decisions in combination with their own computing tasks. This decision based on global information is usually more accurate and effective than the local decision of a single node. If a coordination node fails or goes down, other coordination nodes can still continue to perform resource scheduling and load balancing through the shared cluster resource usage data to maintain the stable operation of the system. This improves the fault tolerance and reliability of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The following is a workflow diagram of the TrinoCoordinator high availability method of the present invention;

[0046] Figure 2 It is a structural schematic diagram of the TrinoCoordinator high availability method of the present invention;

[0047] Figure 3 It is a structural schematic diagram of the TrinoCoordinator high availability method of the present invention;

[0048] Figure 4 A flowchart of the TrinoCoordinator high availability method of the present invention for determining whether an alternative coordination node is faulty;

[0049] Figure 5 The present invention is a flowchart for selecting an alternative coordination node of the TrinoCoordinator high availability method. DETAILED DESCRIPTION

[0050] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0052] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0053] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0054] The tino architecture consists of (at least) one coordinator and multiple workers. Users connect to the coordinator through SQL tools. The coordinator manages the workers. The catalogscoordinator and the workers access the connected data sources by configuring them.

[0055] The execution of each query is a stateful operation, and the coordinator orchestrates the workload, scheduling tasks to workers for parallel execution. Each tino runs in a JM instance and further parallelizes tasks by using threads.

[0056] See also Figure 1-Figure 5 As shown, Figure 1 The following is a workflow diagram of the TrinoCoordinator high availability method of the present invention; Figure 2 It is a structural schematic diagram of the TrinoCoordinator high availability method of the present invention; Figure 3 It is a structural schematic diagram of the TrinoCoordinator high availability method of the present invention; Figure 4 A flowchart of the TrinoCoordinator high availability method of the present invention for determining whether an alternative coordination node is faulty; Figure 5 The present invention is a flowchart for selecting an alternative coordination node of the TrinoCoordinator high availability method.

[0057] A TrinoCoordinator high availability method according to an embodiment of the present invention includes:

[0058] Set up at least one service proxy node and at least one isolated storage node in Nginx, and use Nginx to proxy Trino's coordinator service, including:

[0059] Step S1, Nginx receives a client query request transmitted by Trino, and establishes a connection between the service proxy node and each coordination node of Trino based on the received client query request;

[0060] Step S2, the service proxy node sends a status sharing task to each of the coordination nodes to obtain the cluster resource utilization rate of each Trino working node;

[0061] Step S3, selecting a number of candidate coordination nodes for executing the current query task based on the task response order of each coordination node and the current task execution rate of each coordination node, wherein the number of the candidate coordination nodes is at least two;

[0062] Step S4, the service proxy node sends the client query request to each of the candidate coordination nodes;

[0063] Step S5, each candidate coordination node obtains a trigger condition from the isolated storage node based on the client query request, wherein if the candidate coordination node finds that there is no query mapping corresponding to the client query request in the isolated storage node, then the trigger condition is obtained and the query of the client query request is performed;

[0064] Step S6, in response to the trigger condition, the isolated storage node monitors and obtains the current query result of the corresponding candidate coordination node;

[0065] Step S7, the isolated storage node determines whether the candidate coordination node is faulty according to the update speed of the current query result. If there is a fault, the isolated storage node generates an updated query mapping based on the stored client query request and the corresponding current query result and shares it with other candidate coordination nodes except the current candidate coordination node, so as to transmit the current query result to the client through other candidate coordination nodes;

[0066] Step S8, updating the current task execution rate of each candidate coordination node based on the candidate number of times, the actual query execution number and the failure number of the candidate coordination node;

[0067] Step S9: In response to the task completion status of the candidate coordination node, the isolated storage node saves the currently completed query mapping into the request comparison table and shares it with all coordination nodes.

[0068] The present invention selects Nginx as a front-end load balancing and proxy server to provide a unified access entrance for the client. The client does not need to directly interact with each coordination node in the Trino cluster, which simplifies the use experience. Nginx can route the request to a suitable Trino coordination node for processing according to the characteristics of the client request. At the same time, Nginx can realize load balancing distribution of the request according to the load condition of the coordination node, thereby improving the overall processing capacity of the cluster. Nginx can perform security review and access control on the client request to prevent illegal access and attacks, which is helpful to protect the security and stable operation of the Trino cluster. The service proxy node sends a status sharing task to the Trino coordination node, and can obtain the resource usage of each working node. The cluster status information can be used for the load balancing decision of Nginx to improve the efficiency of request routing. Based on the collected cluster status information, Nginx can intelligently route the request to the coordination node with lower load for processing, thereby maximizing the overall response capability and resource utilization efficiency of the Trino cluster.

[0069] Specifically, in step S3, the step of selecting the candidate coordination node includes:

[0070] Step S3301, the service proxy node obtains the cluster resource usage data with timestamps transmitted by each of the coordination nodes;

[0071] Step S3302, determining the task response order of the corresponding coordination node according to the sequence of each timestamp;

[0072] Step S3303, obtaining the current task execution rate and preset task execution rate of each coordination node;

[0073] Step S3304, eliminating the corresponding coordination nodes in the task response sequence based on the preset task execution rate;

[0074] Step S3305, reordering the coordination nodes after removing some coordination nodes based on the task response order;

[0075] Step S3306: Select a preset number of coordination nodes in the reordering as candidate coordination nodes.

[0076] Specifically, in step S3304, the coordination nodes that can be eliminated are determined based on the comparison result of the current task execution rate B of each coordination node and the preset task execution rate B0;

[0077] When B≥B0, determining that the coordination node is not a removable coordination node;

[0078] When B<B0, it is determined that the coordination node is a removable coordination node.

[0079] Specifically, the current task execution rate H of a single candidate coordination node is determined by the following formula:

[0080]

[0081] Among them, N1 represents the actual query execution times, N2 represents the alternative times, α represents the influence coefficient of task execution success rate on task execution rate, and its value is set to 0.56, Y represents the number of failures, Y0 represents the preset number of failures, and β represents the influence coefficient of the number of failures on task execution rate, and its value is set to 0.44.

[0082] The value of the preset task execution rate in the embodiment of the present invention is the value of the task execution rate obtained when N1=0.8N2 and Y=0.2Y0, and the preset number of failures is the average number of failures of several alternative coordination nodes, but the above values ​​are not limited to this, and technical personnel in this field can also adjust the value according to actual needs.

[0083] By collecting the real-time load status of the coordination node, the service proxy node can dynamically perceive the usage status of the cluster resources. Based on this, the service proxy node can adjust the distribution of tasks among the coordination nodes to achieve a more balanced load distribution. According to the task response order and the preset execution rate, the service proxy node can give priority to the coordination nodes with lower load and faster response to process the query request, which can shorten the query processing delay and improve the overall query response efficiency. By selecting some prepared coordination nodes, the service proxy node can quickly switch to the backup node to continue to provide services when the main node fails, which can improve the availability and fault tolerance of the service and enhance the stability of the entire system. By dynamically adjusting the task allocation, the service proxy node can try to avoid idle or overused resources, which can improve the overall resource utilization efficiency of the cluster and reduce operating costs. By reserving some alternative nodes, the task allocation ratio can be quickly adjusted when the cluster is expanded or reduced to meet dynamic computing needs. This flexibility helps to better cope with changes in business needs.

[0084] Specifically, in step S3306, the preset number is determined according to the average of the task execution rates of the coordination nodes, and the preset number D is negatively correlated with the average of the task execution rates of the coordination nodes. The preset number can be calculated by the following formula:

[0085] D = (1-H1) × D0;

[0086] Among them, H1 represents the mean of the task execution rate of each coordination node, and D0 represents the number of coordination nodes.

[0087] Specifically, in step S5, when determining whether to obtain the trigger condition, it is determined whether to obtain the trigger condition according to whether there is a query mapping corresponding to the client query request in the isolated storage node;

[0088] When the candidate coordination node finds that the isolated storage node has a query mapping corresponding to the client query request, it is determined not to obtain the trigger condition;

[0089] When the candidate coordination node finds that the query mapping corresponding to the client query request does not exist in the isolated storage node, it determines to obtain the trigger condition.

[0090] The query mapping in the embodiment of the present invention is a data packet with each client query request as the file name, and the query result corresponding to the client query request is stored in the data packet.

[0091] The present invention determines whether to trigger the acquisition condition by judging whether there is a query mapping. If there is a query mapping, it means that the query has been executed before, and the result can be directly obtained from the cache without re-execution. This can greatly reduce unnecessary query execution and improve the overall query processing efficiency. By checking the query mapping, repeated query requests can be identified to avoid repeated calculation and processing of the same query, which not only improves efficiency, but also reduces the burden on cluster resources and improves the overall system performance. If there is a query mapping, the result data can be directly obtained from the cache and quickly responded to the client without going through the complete query execution process. This is very helpful for some query scenarios that require fast response and improves the user experience. By caching and reusing query results, unnecessary computing resource consumption can be reduced and the overall resource utilization efficiency of the cluster can be improved. This is particularly important for resource management in large-scale data processing scenarios. According to the existence of query mapping, incremental updates or differentiated processing can be performed in a targeted manner to avoid repeated calculation of the entire amount, which can further improve the efficiency and response speed of query processing.

[0092] Specifically, in step S7, when determining whether the candidate coordination node is faulty, the isolated storage node repeatedly monitors the current query result of the candidate coordination node of the trigger condition based on a preset period, and determines whether the candidate coordination node is faulty according to the comparison result of the update speed V of the current query result and the preset update speed V0;

[0093] When V>V0, it is determined that the candidate coordination node is not faulty;

[0094] When V≤V0, it is determined that the candidate coordination node is faulty;

[0095] The value of the preset update speed V0 is the historical average update speed of the current query result of the candidate coordination node, but the above value is not limited thereto, and those skilled in the art may also adjust the value according to actual needs.

[0096] In the embodiment of the present invention, the value range of the preset period is set to 15 seconds-60 seconds, and the preset period is preferably 30 seconds, but the above value is not limited thereto, and those skilled in the art may also adjust the value according to actual needs.

[0097] By regularly checking the query result update speed of the alternative nodes, the present invention can grasp the current operating status and computing power of the alternative nodes in real time, so that it can timely discover whether the alternative nodes have faults or performance degradation. If the alternative node is found to be faulty or has poor performance, the node can be quickly removed from the alternative node list to avoid it affecting the overall service quality, which helps to maintain high availability of the service and ensure business continuity and stability. By comparing the query result update speed, the alternative nodes with better performance can be identified and they can be preferentially selected to process query tasks, which can improve the overall query processing efficiency, shorten the response time, and improve the user experience. Once a candidate node is found to have a fault or performance degradation, it can be quickly removed from the load distribution. At the same time, the load weights of other healthy nodes can be appropriately increased to achieve dynamic load balancing adjustment. By monitoring the query result update speed of the alternative nodes, the system's operating indicator data can be collected, and these data help to further optimize the system configuration and improve the overall performance and reliability.

[0098] Specifically, when it is determined that the alternative coordination node has a fault, the isolated storage node will store the client query request and the corresponding current query result to generate an updated query mapping and share it with other alternative coordination nodes except the current alternative coordination node, so as to transmit the current query result to the client through other alternative coordination nodes.

[0099] By sharing the query mapping with other healthy alternative nodes, the present invention can ensure that even if the current node fails, there are still other nodes that can take over the service and provide query results to the client, which helps to maintain the high availability of the entire system and avoid overall service interruption due to a single node failure. After other alternative nodes obtain the updated query mapping, they can directly pull the query results from the cache without re-executing the query, which can greatly shorten the query response time and improve the user experience. When the current alternative node fails, other nodes can quickly take over the service without the client being aware of the switching process, which can ensure the continuity of the service and reduce the business impact caused by the node failure. After sharing the query mapping with other nodes, the burden of the current failed node can be reduced and its recovery difficulty can be reduced, which helps the failed node to return to normal as soon as possible and rejoin the list of alternative nodes. By synchronizing the query mapping to other nodes in a timely manner, it can be ensured that the entire system has self-healing capabilities and avoids overall service interruption due to single point failures, which helps to improve the reliability and stability of the system.

[0100] Specifically, when determining the delay time for transmitting the cluster resource usage data back to the service proxy node, the delay time T for transmitting the cluster resource usage data back to the service proxy node is determined based on the execution percentage R of the tasks being executed by each coordination node.

[0101] Specifically, the lag time T of the service proxy node returning the cluster resource usage rate data can be calculated by the following formula, set:

[0102] T = (1-R) ​​× T0;

[0103] Wherein, R represents the execution percentage of each coordination node based on the task being executed, and R0 represents the initial lag time.

[0104] In the embodiment of the present invention, the execution percentage of each coordination node based on the task being executed can be calculated by the completed workload / total expected workload×100%, and the initial lag time can be set as an initial default lag time based on the number of nodes in the cluster, CPU, memory and other hardware resources.

[0105] By analyzing the current task execution progress of each node, the present invention can predict the resource demand in the future period of time, and adjust the lag time of returning data according to this information, so as to ensure that the service proxy node obtains more timely and accurate resource usage, so as to better schedule and allocate cluster resources and improve the overall utilization efficiency. If the current task execution progress of a node is low, it means that its resource consumption is low and there is no need to frequently return the utilization rate data. The data return lag time of such node can be appropriately increased to reduce the network communication overhead and the overall network load. For nodes that are executing key tasks, their task progress is high and resource consumption is also large, so they need to return the usage data to the service proxy node in time. Usage data can shorten the data return lag time of these nodes and ensure that the service proxy nodes can obtain accurate resource usage information in the first time, so as to prioritize the scheduling of these key tasks. Timely acquisition of accurate cluster resource utilization data can enable service proxy nodes to perceive cluster status changes more quickly and make more timely and reasonable resource allocation decisions, which will help improve the responsiveness of the entire system and enhance the elasticity and stability of the service. Adjusting the data return lag time according to the progress of task execution at each node can allow service proxy nodes to grasp resource usage information closer to real time, which provides a basis for dynamically adjusting load distribution and helps to achieve dynamic and balanced utilization of cluster resources.

[0106] Specifically, the service proxy node will collect the resource usage of each Trino working node in the cluster regularly or as needed through monitoring or API, and integrate the collected resource usage data of each node into a unified data structure. The integrated cluster resource usage data will be pushed synchronously to all coordination nodes in the system through network transmission. Each coordination node receives the cluster resource usage data from the service proxy node and updates it to its locally maintained computing data to ensure data consistency.

[0107] The present invention synchronizes the resource usage of each Trino working node to the coordination node in time through the service proxy node, which can ensure that all coordination nodes have the same cluster resource usage data to avoid data inconsistency. This helps the coordination nodes to make more unified and correct resource scheduling decisions. When the coordination nodes have the resource usage data of the entire cluster, they can make more comprehensive and optimized resource allocation and scheduling decisions in combination with their own computing tasks. This decision based on global information is usually more accurate and effective than the local decision of a single node. If a coordination node fails or goes down, other coordination nodes can still continue to perform resource scheduling and load balancing through the shared cluster resource usage data to maintain the stable operation of the system. This improves the fault tolerance and reliability of the entire system.

[0108] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A TrinoCoordinator high availability method, characterized in that: Set up at least one service proxy node and at least one isolated storage node in Nginx, and use Nginx to proxy Trino's coordinator service, including: Nginx receives the client query request transmitted by Trino, and establishes a connection between the service proxy node and each coordination node of Trino based on the received client query request; The service proxy node sends a state sharing task to each of the coordination nodes to obtain the cluster resource utilization rate of each Trino working node; Selecting a number of candidate coordination nodes for executing the current query task based on the task response order of each coordination node and the current task execution rate of each coordination node, wherein the number of the candidate coordination nodes is at least two; The service proxy node sends the client query request to each of the candidate coordination nodes; Each candidate coordination node obtains a trigger condition from the isolated storage node based on the client query request, wherein if the candidate coordination node finds that there is no query mapping corresponding to the client query request in the isolated storage node, the trigger condition is obtained to query the client query request; In response to the trigger condition, the isolated storage node monitors and obtains the current query result of the corresponding candidate coordination node; The isolated storage node determines whether the candidate coordination node is faulty according to the update speed of the current query result. If there is a fault, the isolated storage node generates an updated query mapping based on the stored client query request and the corresponding current query result and shares it with other candidate coordination nodes except the current candidate coordination node, so as to transmit the current query result to the client through other candidate coordination nodes; Update the current task execution rate of each candidate coordination node based on the candidate number of times, the actual query execution number and the failure number of the candidate coordination node; In response to the task completion status of the candidate coordination node, the isolation storage node saves the currently completed query mapping into the request comparison table and shares it with all coordination nodes.

2. The TrinoCoordinator high availability method according to claim 1, characterized in that: The single candidate node constructs a query plan based on the trigger condition and the query request of the client and distributes it to each working node for execution to obtain the request execution result, which is recorded as the current query result.

3. The TrinoCoordinator high availability method according to claim 2, characterized in that: The selection of the candidate coordination node includes: The service proxy node obtains the cluster resource usage rate data with timestamps transmitted by each of the coordination nodes; Determine the task response order of the corresponding coordination node according to the sequence of each timestamp; Obtaining the current task execution rate and preset task execution rate of each of the coordination nodes; Eliminating the corresponding coordination nodes in the task response sequence based on a preset task execution rate; After removing some coordination nodes, reorder the coordination nodes based on the task response order; A preset number of coordination nodes in the reordering are selected as candidate coordination nodes.

4. The TrinoCoordinator high availability method according to claim 1, characterized in that: The acquisition of the trigger condition also includes: If the candidate coordination node finds that the query mapping corresponding to the client query request exists in the isolated storage node, the trigger condition is not obtained and the query of the client query request is not performed.

5. The TrinoCoordinator high availability method according to claim 2, characterized in that: The step of determining whether the candidate coordination node is faulty includes: The isolated storage node repeatedly monitors and obtains the current query result of the candidate coordination node of the trigger condition based on a preset period; If the update speed of the current query result is less than or equal to the preset update speed, it is determined that the corresponding candidate coordination node is faulty.

6. The TrinoCoordinator high availability method according to claim 3, characterized in that: The preset number is determined according to the average value of the task execution rate of each coordination node, and the preset number is negatively correlated with the average value of the task execution rate of each coordination node.

7. The TrinoCoordinator high availability method according to claim 1, characterized in that: The current task execution rate of a single candidate coordination node is determined by the following formula: Among them, H represents the current task execution rate of a single candidate coordination node, N1 represents the actual query execution times, N2 represents the alternative times, α represents the influence coefficient of the task execution success rate on the task execution rate, Y represents the number of failures, Y0 represents the preset number of failures, and β represents the influence coefficient of the number of failures on the task execution rate.

8. The TrinoCoordinator high availability method according to claim 3, characterized in that: Each of the coordination nodes determines a delay time for transmitting the cluster resource usage rate data back to the service proxy node based on the execution percentage of the task being executed, wherein: The execution percentage is inversely correlated to the lag time.

9. The TrinoCoordinator high availability method according to claim 1, characterized in that: The query mapping is a data packet with each client query request as the file name, and the query result corresponding to the client query request is stored in the data packet.

10. The TrinoCoordinator high availability method according to claim 1, characterized in that: After obtaining the cluster resource utilization rate of each Trino working node, the service proxy node synchronously shares the received cluster resource utilization rate of each Trino working node with each coordination node so that the computing data of each coordination node remains consistent.

Images