A TrinoCoordinator high availability method

Abstract

The present invention relates to the field of database technology, and in particular to a TrinoCoordinator high-availability method, which includes: Nginx receiving a client query request transmitted by Trino, establishing a connection between the service proxy node and each coordination node of Trino based on the received client query request; the service proxy node sending a status sharing task to each coordination node to obtain the cluster resource utilization rate of each Trino working node; selecting several alternative coordination nodes for executing the current query task based on the task response order of each coordination node and the current task execution rate of each coordination node, wherein the number of alternative coordination nodes is at least two; the service proxy node sending the client query request to each alternative coordination node; each alternative coordination node obtaining a trigger condition from the isolation storage node based on the client query request, and the isolation storage node determining whether the alternative coordination node is faulty according to the update speed of the current query result.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a TrinoCoordinator high availability method. Background Art

[0002] Trino is a distributed QSQL query engine, a massively parallel processing (MPP) database and query engine. Trino achieves horizontal scaling by distributing processing tasks across a cluster of servers, rather than vertical scaling by improving the performance of a single server. Based on this architecture, the Trino query engine can process SQL queries on massive amounts of data in parallel across the compute nodes within the cluster. On each compute node, Trino runs as a single server process. A Trino cluster consists of multiple Trino nodes configured to collaborate with each other.

[0003] Chinese patent application publication number CN112559022A discloses a Jenkins high-availability system and method. The system includes a task management module for receiving user-created task data and persisting it in a local database; a task scheduling module for invoking a Jenkins interface, using the task data as input to synchronously create Jenkins tasks on available Jenkins master nodes; and writing task result data to the local database. A Jenkins management module is used to check for anomalies in Jenkins master nodes. Upon detecting an abnormal Jenkins master node, the module checks the Jenkins tasks currently executing on the abnormal Jenkins master node. This application improves the availability and stability of existing Jenkins high-availability solutions.

[0004] It can be seen that the existing technology has the problem of low accuracy in real-time status analysis of task coordination nodes, which leads to unreasonable allocation of query tasks, low cluster resource utilization and long query task response time. Summary of the Invention

[0005] To this end, the present invention provides a TrinoCoordinator high availability method to overcome the problems in the prior art of low accuracy of real-time status analysis of task coordination nodes, which leads to inability to reasonably allocate query tasks, resulting in low cluster resource utilization and long query task response time.

[0006] To achieve the above objectives, the present invention provides a TrinoCoordinator high availability method, comprising:

[0007] Set up at least one service proxy node and at least one isolated storage node in Nginx, and use Nginx to proxy Trino's coordinator service, including:

[0008] Nginx receives the client query request transmitted by Trino, and establishes a connection between the service proxy node and each coordination node of Trino based on the received client query request;

[0009] The service agent node sends a state sharing task to each of the coordination nodes to obtain the cluster resource utilization rate of each Trino working node;

[0010] Selecting a number of candidate coordination nodes for executing the current query task based on the task response order of each coordination node and the current task execution rate of each coordination node, wherein the number of the candidate coordination nodes is at least two;

[0011] The service proxy node sends the client query request to each of the candidate coordination nodes;

[0012] Each candidate coordination node obtains a trigger condition from the isolated storage node based on the client query request, wherein if the candidate coordination node finds that there is no query mapping corresponding to the client query request in the isolated storage node, the trigger condition is obtained and a query is performed for the client query request;

[0013] In response to a trigger condition, the isolated storage node monitors and obtains a current query result of the corresponding candidate coordination node;

[0014] The isolated storage node determines whether the candidate coordination node is faulty based on the update speed of the current query result. If a fault exists, the isolated storage node generates an updated query mapping based on the stored client query request and the corresponding current query result and shares it with the other candidate coordination nodes except the current candidate coordination node, so as to transmit the current query result to the client through the other candidate coordination nodes.

[0015] Update the current task execution rate of each candidate coordination node based on the candidate number of times, the actual query execution number and the failure number of the candidate coordination node;

[0016] In response to the task completion status of the candidate coordination node, the isolated storage node saves the currently completed query mapping into the request comparison table and shares it with all coordination nodes.

[0017] Furthermore, a single candidate node constructs a query plan according to the client query request based on the trigger condition and distributes the query plan to each working node for execution to obtain a request execution result, which is recorded as a current query result.

[0018] Furthermore, the selection of the candidate coordination node includes:

[0019] The service agent node obtains the cluster resource usage data with timestamps transmitted by each coordination node;

[0020] Determine the task response order of the corresponding coordination node according to the sequence of each time stamp;

[0021] Obtaining the current task execution rate and preset task execution rate of each coordination node;

[0022] Eliminating the corresponding coordination nodes in the task response sequence based on a preset task execution rate;

[0023] After removing some coordination nodes, reorder the coordination nodes based on the task response order;

[0024] A preset number of coordination nodes in the reordering are selected as candidate coordination nodes.

[0025] Furthermore, the acquisition of the trigger condition also includes:

[0026] If the candidate coordination node finds that the query mapping corresponding to the client query request exists in the isolated storage node, the trigger condition is not obtained and the query request of the client is not performed.

[0027] Furthermore, the step of determining whether the candidate coordination node is faulty includes:

[0028] The isolated storage node repeatedly monitors and obtains the current query result of the candidate coordination node of the trigger condition based on a preset period;

[0029] If the update speed of the current query result is less than or equal to the preset update speed, it is determined that the corresponding candidate coordination node is faulty.

[0030] Furthermore, the preset number is determined according to an average value of the task execution rate of each coordination node, and the preset number is negatively correlated with the average value of the task execution rate of each coordination node.

[0031] Furthermore, the current task execution rate of a single candidate coordination node is determined by the following formula:

[0032]

[0033] Among them, H represents the current task execution rate of a single alternative coordination node, N1 represents the actual query execution times, N2 represents the alternative times, α represents the influence coefficient of the task execution success rate on the task execution rate, Y represents the number of failures, Y0 represents the preset number of failures, and β represents the influence coefficient of the number of failures on the task execution rate.

[0034] Furthermore, each of the coordination nodes determines a delay time for transmitting the cluster resource usage data back to the service agent node based on the execution percentage of the task being executed, wherein:

[0035] The execution percentage is inversely correlated with the lag time.

[0036] Furthermore, the query mapping is a data packet with each client query request as the file name, and the data packet stores the query result corresponding to the client query request.

[0037] Furthermore, after obtaining the cluster resource utilization rate of each Trino working node, the service agent node synchronously shares the received cluster resource utilization rate of each Trino working node with each coordination node so that the computing data of each coordination node remains consistent.

[0038] Compared with the prior art, the beneficial effect of the present invention is that the present invention selects Nginx as the front-end load balancing and proxy server to provide a unified access portal for the client. The client does not need to interact directly with the various coordination nodes in the Trino cluster, which simplifies the user experience. Nginx can route the request to the appropriate Trino coordination node for processing according to the characteristics of the client request. At the same time, Nginx can realize load balancing distribution of requests according to the load of the coordination node, thereby improving the overall processing capability of the cluster. Nginx can perform security review and access control on client requests to prevent illegal access and attacks, which helps to protect the security and stable operation of the Trino cluster. The service proxy node sends a status sharing task to the Trino coordination node, and can obtain the resource usage of each working node. These cluster status information can be used for Nginx's load balancing decision to improve the efficiency of request routing. Based on the collected cluster status information, Nginx can intelligently route the request to the coordination node with lower load for processing, thereby maximizing the overall response capability and resource utilization efficiency of the Trino cluster.

[0039] Furthermore, the present invention collects the real-time load status of the coordination node, and the service proxy node can dynamically perceive the usage status of the cluster resources. Based on this, the service proxy node can adjust the distribution of tasks among the coordination nodes to achieve a more balanced load distribution. According to the task response order and the preset execution rate, the service proxy node can give priority to the coordination node with lower load and faster response to process the query request, which can shorten the query processing delay and improve the overall query response efficiency. By selecting some prepared coordination nodes, the service proxy node can quickly switch to the backup node to continue providing services when the main node fails, which can improve the availability and fault tolerance of the service and enhance the stability of the entire system. By dynamically adjusting the task allocation, the service proxy node can try to avoid idle or overused resources, which can improve the overall resource utilization efficiency of the cluster and reduce operating costs. By reserving some alternative nodes, the task allocation ratio can be quickly adjusted when the cluster is expanded or reduced to meet dynamic computing needs. This flexibility helps to better respond to changes in business needs.

[0040] Furthermore, the present invention determines whether to trigger the acquisition condition by judging whether a query mapping exists. If a query mapping exists, it means that the query has been executed before, and the result can be directly obtained from the cache without re-execution. This can greatly reduce unnecessary query execution and improve the overall query processing efficiency. By checking the query mapping, repeated query requests can be identified to avoid repeated calculation and processing of the same query. This not only improves efficiency, but also reduces the burden on cluster resources and improves the overall system performance. If a query mapping exists, the result data can be directly obtained from the cache and quickly responded to the client without going through the complete query execution process. This is very helpful for some query scenarios that require fast responses and improves the user experience. By caching and reusing query results, unnecessary computing resource consumption can be reduced and the overall resource utilization efficiency of the cluster can be improved. This is particularly important for resource management in large-scale data processing scenarios. According to the existence of query mapping, incremental updates or differential processing can be performed in a targeted manner to avoid repeated calculation of the entire amount, which can further improve the efficiency and response speed of query processing.

[0041] Furthermore, the present invention can grasp the current operating status and computing power of alternative nodes in real time by regularly checking the query result update speed of alternative nodes, so as to timely discover whether there is a fault or performance degradation in the alternative node. If the alternative node is found to be faulty or has poor performance, the node can be quickly removed from the alternative node list to avoid it affecting the overall service quality, which helps to maintain high availability of the service and ensure business continuity and stability. By comparing the query result update speed, alternative nodes with better performance can be identified and they can be used to process query tasks preferentially, which can improve the overall query processing efficiency, shorten the response time, and enhance the user experience. Once a candidate node is found to be faulty or has performance degradation, it can be quickly removed from the load distribution. At the same time, the load weight of other healthy nodes can be appropriately increased to achieve dynamic load balancing adjustment. By monitoring the query result update speed of the alternative node, the system's operating indicator data can be collected. These data help to further optimize the system configuration and improve the overall performance and reliability.

[0042] Furthermore, the present invention can ensure that even if the current node fails, there are still other nodes that can take over the service and provide query results to the client by sharing the query mapping with other healthy alternative nodes. This helps to maintain the high availability of the entire system and avoid overall service interruption due to a single node failure. After other alternative nodes obtain the updated query mapping, they can directly pull the query results from the cache without re-executing the query. This can greatly shorten the query response time and improve user experience. When the current alternative node fails, other nodes can quickly take over the service without the client being aware of the switching process. This can ensure service continuity and reduce the business impact caused by node failure. After sharing the query mapping with other nodes, the burden on the current faulty node can be reduced and its recovery difficulty can be reduced. This helps the faulty node to return to normal as soon as possible and rejoin the list of alternative nodes. By synchronizing the query mapping to other nodes in a timely manner, it can ensure that the entire system has self-healing capabilities and avoid overall service interruption due to single point failures. This helps to improve the reliability and stability of the system.

[0043] Furthermore, the present invention can predict resource requirements in the future by analyzing the current task execution progress of each node. The lag time of returning data can be adjusted according to this information, which can ensure that the service proxy node obtains more timely and accurate resource usage information, thereby better scheduling and allocating cluster resources and improving overall utilization efficiency. If the current task execution progress of a node is low, it means that its resource consumption is low and there is no need to frequently return utilization data. The data return lag time of such node can be appropriately increased to reduce network communication overhead and reduce the overall network load. For nodes that are performing critical tasks, their task progress is high and resource consumption is also large, so they need to return to the service proxy node in a timely manner. Transmitting utilization data can shorten the data return lag time of these nodes and ensure that the service proxy node can obtain accurate resource usage information in the first time, so as to prioritize the scheduling of these key tasks. Obtaining accurate cluster resource utilization data in a timely manner can enable the service proxy node to perceive cluster status changes more quickly and make more timely and reasonable resource allocation decisions, which helps to improve the responsiveness of the entire system and enhance the elasticity and stability of the service. Adjusting the data return lag time according to the task execution progress of each node can allow the service proxy node to grasp the resource usage information closer to real time, which provides a basis for dynamically adjusting load distribution and helps to achieve dynamic and balanced utilization of cluster resources.

[0044] Furthermore, the present invention uses a service proxy node to promptly synchronize the resource usage of each Trino working node to the coordination node, ensuring that all coordination nodes have the same cluster resource usage data, avoiding data inconsistencies. This helps coordination nodes make more unified and accurate resource scheduling decisions. When coordination nodes have resource usage data for the entire cluster, they can make more comprehensive and optimized resource allocation and scheduling decisions based on their own computing tasks. This type of decision-making based on global information is generally more accurate and effective than the local decision-making of a single node. If a coordination node fails or goes down, other coordination nodes can still continue to perform resource scheduling and load balancing through shared cluster resource usage data, maintaining stable system operation. This improves the fault tolerance and reliability of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a workflow diagram of the TrinoCoordinator high availability method of the present invention;

[0046] Figure 2 Schematic diagram of the structure of the TrinoCoordinator high availability method of the present invention;

[0047] Figure 3 Schematic diagram of the structure of the TrinoCoordinator high availability method of the present invention;

[0048] Figure 4 This is a workflow diagram for determining whether an alternative coordination node is faulty in the TrinoCoordinator high availability method of the present invention;

[0049] Figure 5 This is a workflow diagram for selecting an alternative coordination node for the TrinoCoordinator high-availability method of the present invention. DETAILED DESCRIPTION

[0050] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention.

[0051] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0052] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside", and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0053] Furthermore, it should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0054] The tino architecture consists of (at least) one coordinator and multiple workers. Users connect to the coordinator through SQL tools. The coordinator manages the workers. The coordinator and the workers access connected data sources by configuring the catalog.

[0055] The execution of each query is a stateful operation. The coordinator orchestrates the workload by dispatching tasks to workers for parallel execution. Each tino runs in a JM instance and further parallelizes tasks by using threads.

[0056] See also Figure 1-Figure 5 As shown, Figure 1 This is a workflow diagram of the TrinoCoordinator high availability method of the present invention; Figure 2 Schematic diagram of the structure of the TrinoCoordinator high availability method of the present invention; Figure 3 Schematic diagram of the structure of the TrinoCoordinator high availability method of the present invention; Figure 4 This is a workflow diagram for determining whether an alternative coordination node is faulty in the TrinoCoordinator high availability method of the present invention; Figure 5 This is a workflow diagram for selecting an alternative coordination node for the TrinoCoordinator high-availability method of the present invention.

[0057] An embodiment of the present invention provides a TrinoCoordinator high availability method, including:

[0058] Set up at least one service proxy node and at least one isolated storage node in Nginx, and use Nginx to proxy Trino's coordinator service, including:

[0059] Step S1, Nginx receives a client query request transmitted by Trino, and establishes a connection between the service proxy node and each coordination node of Trino based on the received client query request;

[0060] Step S2: the service agent node sends a status sharing task to each of the coordination nodes to obtain the cluster resource usage of each Trino working node;

[0061] Step S3, selecting a number of candidate coordination nodes for executing the current query task based on the task response order of each coordination node and the current task execution rate of each coordination node, wherein the number of candidate coordination nodes is at least two;

[0062] Step S4: the service proxy node sends the client query request to each of the candidate coordination nodes;

[0063] Step S5: Each candidate coordinating node obtains a trigger condition from the isolated storage node based on the client query request. If the candidate coordinating node finds that no query mapping corresponding to the client query request exists in the isolated storage node, the candidate coordinating node obtains the trigger condition and performs a query for the client query request.

[0064] Step S6: In response to the trigger condition, the isolated storage node monitors and obtains the current query result of the corresponding candidate coordination node;

[0065] Step S7: The isolated storage node determines whether the candidate coordination node is faulty based on the update speed of the current query result. If a fault exists, the isolated storage node generates an updated query mapping based on the stored client query request and the corresponding current query result and shares it with the other candidate coordination nodes except the current candidate coordination node, so as to transmit the current query result to the client through the other candidate coordination nodes.

[0066] Step S8: updating the current task execution rate of each candidate coordination node based on the candidate number of times, actual query execution times, and failure times of the candidate coordination node;

[0067] Step S9: In response to the task completion status of the candidate coordination node, the isolated storage node saves the currently completed query mapping into the request comparison table and shares it with all coordination nodes.

[0068] The present invention selects Nginx as the front-end load balancing and proxy server to provide a unified access portal for the client. The client does not need to interact directly with the various coordination nodes in the Trino cluster, which simplifies the user experience. Nginx can route the request to the appropriate Trino coordination node for processing according to the characteristics of the client request. At the same time, Nginx can realize load balancing distribution of the request according to the load of the coordination node, thereby improving the overall processing capability of the cluster. Nginx can perform security review and access control on the client request to prevent illegal access and attacks, which helps to protect the security and stable operation of the Trino cluster. The service proxy node sends a status sharing task to the Trino coordination node, and can obtain the resource usage of each working node. This cluster status information can be used for Nginx's load balancing decision to improve the efficiency of request routing. Based on the collected cluster status information, Nginx can intelligently route the request to the coordination node with lower load for processing, thereby maximizing the overall responsiveness and resource utilization efficiency of the Trino cluster.

[0069] Specifically, in step S3, the step of selecting the candidate coordination node includes:

[0070] Step S3301: The service agent node obtains the cluster resource usage data with timestamps transmitted by each coordination node;

[0071] Step S3302: determining the task response order of the corresponding coordination node according to the order of the timestamps;

[0072] Step S3303, obtaining the current task execution rate and preset task execution rate of each coordination node;

[0073] Step S3304: Eliminate the corresponding coordination nodes in the task response sequence based on the preset task execution rate;

[0074] Step S3305: reorder the coordination nodes after removing some coordination nodes based on the task response order;

[0075] Step S3306: Select a preset number of coordination nodes in the reordering as candidate coordination nodes.

[0076] Specifically, in step S3304, the coordination nodes that can be eliminated are determined based on the comparison result of the current task execution rate B of each coordination node and the preset task execution rate B0;

[0077] When B≥B0, determining that the coordination node is not a removable coordination node;

[0078] When B<B0, it is determined that the coordination node is a removable coordination node.

[0079] Specifically, the current task execution rate H of a single candidate coordination node is determined by the following formula:

[0080]

[0081] Among them, N1 represents the actual number of query executions, N2 represents the number of alternatives, α represents the influence coefficient of task execution success rate on task execution rate, and its value is set to 0.56, Y represents the number of failures, Y0 represents the preset number of failures, and β represents the influence coefficient of the number of failures on task execution rate, and its value is set to 0.44.

[0082] In the embodiment of the present invention, the value of the preset task execution rate is the value of the task execution rate obtained when N1=0.8N2 and Y=0.2Y0, and the preset number of failures is the average number of failures of several alternative coordination nodes, but the above value is not limited to this, and technical personnel in this field can also adjust the value according to actual needs.

[0083] By collecting the real-time load status of the coordination node, the service proxy node can dynamically perceive the usage status of the cluster resources. Based on this, the service proxy node can adjust the distribution of tasks among the coordination nodes to achieve a more balanced load distribution. According to the task response order and the preset execution rate, the service proxy node can give priority to the coordination node with lower load and faster response to process the query request, which can shorten the query processing delay and improve the overall query response efficiency. By selecting some prepared coordination nodes, the service proxy node can quickly switch to the backup node to continue providing services when the main node fails, which can improve the availability and fault tolerance of the service and enhance the stability of the entire system. By dynamically adjusting the task allocation, the service proxy node can try to avoid idle or overused resources, which can improve the overall resource utilization efficiency of the cluster and reduce operating costs. By reserving some alternative nodes, the task allocation ratio can be quickly adjusted when the cluster is expanded or reduced to meet dynamic computing needs. This flexibility helps to better cope with changes in business needs.

[0084] Specifically, in step S3306, the preset number is determined according to the average of the task execution rates of the coordination nodes, and the preset number D is negatively correlated with the average of the task execution rates of the coordination nodes. The preset number can be calculated and set by the following formula:

[0085] D = (1-H1) × D0;

[0086] Among them, H1 represents the mean of the task execution rate of each coordination node, and D0 represents the number of coordination nodes.

[0087] Specifically, in step S5, when determining whether a trigger condition is obtained, it is determined whether the trigger condition is obtained according to whether a query mapping corresponding to the client query request exists in the isolated storage node;

[0088] When the candidate coordination node finds that the isolated storage node has a query mapping corresponding to the client query request, it determines not to obtain the trigger condition;

[0089] When the candidate coordination node finds that the query mapping corresponding to the client query request does not exist in the isolated storage node, it determines to obtain the trigger condition.

[0090] In the embodiment of the present invention, the query mapping is a data packet with each client query request as the file name, and the data packet stores the query result corresponding to the client query request.

[0091] The present invention determines whether to trigger the acquisition condition by judging whether there is a query mapping. If there is a query mapping, it means that the query has been executed before, and the result can be directly obtained from the cache without re-execution. This can greatly reduce unnecessary query execution and improve the overall query processing efficiency. By checking the query mapping, repeated query requests can be identified to avoid repeated calculation and processing of the same query. This not only improves efficiency, but also reduces the burden on cluster resources and improves the overall system performance. If there is a query mapping, the result data can be directly obtained from the cache and quickly responded to the client without going through the complete query execution process. This is very helpful for some query scenarios that require fast response and improves the user experience. By caching and reusing query results, unnecessary computing resource consumption can be reduced and the overall resource utilization efficiency of the cluster can be improved. This is particularly important for resource management in large-scale data processing scenarios. According to the existence of query mapping, incremental updates or differential processing can be performed in a targeted manner to avoid repeated calculation of the entire amount, which can further improve the efficiency and response speed of query processing.

[0092] Specifically, in step S7, when determining whether the candidate coordinating node is faulty, the isolated storage node repeatedly monitors the current query result of the candidate coordinating node of the trigger condition based on a preset period, and determines whether the candidate coordinating node is faulty based on a comparison result of an update speed V of the current query result and a preset update speed V0;

[0093] When V>V0, it is determined that the candidate coordination node is not faulty;

[0094] When V≤V0, it is determined that the candidate coordination node is faulty;

[0095] The preset update speed V0 is set to a historical average of the update speeds of the current query results of the candidate coordination node, but the value is not limited thereto, and those skilled in the art may adjust the value according to actual needs.

[0096] In the embodiment of the present invention, the preset period has a value range of 15 seconds to 60 seconds, and the preset period is preferably 30 seconds, but the above value is not limited thereto, and those skilled in the art may also adjust the value according to actual needs.

[0097] By regularly checking the query result update speed of the alternative nodes, the present invention can grasp the current operating status and computing power of the alternative nodes in real time, so that it can promptly discover whether the alternative nodes have faults or performance degradation. If the alternative node is found to be faulty or has poor performance, the node can be quickly removed from the alternative node list to avoid it affecting the overall service quality, which helps to maintain high availability of the service and ensure business continuity and stability. By comparing the query result update speed, the alternative nodes with better performance can be identified and they can be used to process query tasks preferentially, which can improve the overall query processing efficiency, shorten the response time, and enhance the user experience. Once a candidate node is found to have a fault or performance degradation, it can be quickly removed from the load distribution. At the same time, the load weight of other healthy nodes can be appropriately increased to achieve dynamic load balancing adjustment. By monitoring the query result update speed of the alternative nodes, the system's operating indicator data can be collected. These data help to further optimize the system configuration and improve the overall performance and reliability.

[0098] Specifically, when it is determined that the alternative coordination node has a fault, the isolated storage node will generate an updated query mapping based on the stored client query request and the corresponding current query result and share it with other alternative coordination nodes except the current alternative coordination node, so as to transmit the current query result to the client through other alternative coordination nodes.

[0099] By sharing the query mapping with other healthy alternative nodes, the present invention can ensure that even if the current node fails, there are still other nodes that can take over the service and provide query results to the client. This helps to maintain the high availability of the entire system and avoid overall service interruption due to a single node failure. After other alternative nodes obtain the updated query mapping, they can directly pull the query results from the cache without having to re-execute the query, which can greatly shorten the query response time and improve the user experience. When the current alternative node fails, other nodes can quickly take over the service without the client being aware of the switching process, which can ensure service continuity and reduce the business impact caused by node failure. After sharing the query mapping with other nodes, the burden on the current faulty node can be reduced and its recovery difficulty can be reduced, which helps the faulty node to return to normal as soon as possible and rejoin the list of alternative nodes. By synchronizing the query mapping to other nodes in a timely manner, it can be ensured that the entire system has self-healing capabilities and avoids overall service interruption due to single point failures, which helps to improve the reliability and stability of the system.

[0100] Specifically, when determining the lag time for transmitting the cluster resource usage data back to the service proxy node, the lag time T for transmitting the cluster resource usage data back to the service proxy node is determined based on the execution percentage R of the tasks being executed by each coordination node.

[0101] Specifically, the lag time T of the service proxy node returning the cluster resource usage data can be calculated by the following formula:

[0102] T=(1-R)×T0;

[0103] Here, R represents the execution percentage of each coordination node based on the task being executed, and R0 represents the initial lag time.

[0104] In an embodiment of the present invention, the execution percentage of each coordination node based on the task being executed can be calculated by the completed workload / total expected workload × 100%, and the initial lag time can be set as an initial default lag time based on the number of nodes in the cluster, CPU, memory and other hardware resources.

[0105] By analyzing the current task execution progress of each node, the present invention can predict the resource demand in the future period of time, and adjust the lag time of returning data based on this information, so as to ensure that the service proxy node obtains more timely and accurate resource usage, thereby better scheduling and allocating cluster resources and improving overall utilization efficiency. If the current task execution progress of a node is low, it means that its resource consumption is low and there is no need to frequently return utilization data. The data return lag time of such node can be appropriately increased to reduce network communication overhead and reduce the overall network load. For nodes that are performing critical tasks, their task progress is high and resource consumption is also large, so they need to return the usage data to the service proxy node in a timely manner. Usage data can shorten the data return lag time of these nodes and ensure that the service proxy node can obtain accurate resource usage information in the first time, so as to prioritize the scheduling of these key tasks. Timely and accurate acquisition of cluster resource utilization data can enable the service proxy node to perceive cluster status changes more quickly and make more timely and reasonable resource allocation decisions, which helps to improve the responsiveness of the entire system and enhance the elasticity and stability of the service. Adjusting the data return lag time according to the task execution progress of each node can allow the service proxy node to grasp the resource usage situation closer to real time, which provides a basis for dynamically adjusting load distribution and helps to achieve dynamic and balanced utilization of cluster resources.

[0106] Specifically, the service proxy node will collect the resource usage of each Trino working node in the cluster regularly or as needed through monitoring or API, and integrate the collected resource usage data of each node into a unified data structure. The integrated cluster resource usage data will be pushed to all coordination nodes in the system synchronously through network transmission. Each coordination node receives the cluster resource usage data from the service proxy node and updates it to its own locally maintained operation data to ensure data consistency.

[0107] The present invention uses a service proxy node to promptly synchronize the resource usage of each Trino working node to the coordination node, ensuring that all coordination nodes have the same cluster resource usage data, avoiding data inconsistencies. This helps coordination nodes make more unified and accurate resource scheduling decisions. When coordination nodes have resource usage data for the entire cluster, they can make more comprehensive and optimized resource allocation and scheduling decisions based on their own computing tasks. This type of decision-making based on global information is generally more accurate and effective than the local decision-making of a single node. If a coordination node fails or goes down, other coordination nodes can still continue to perform resource scheduling and load balancing through shared cluster resource usage data, maintaining stable system operation. This improves the fault tolerance and reliability of the entire system.

[0108] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A TrinoCoordinator high availability method, characterized in that: Set up at least one service proxy node and at least one isolated storage node in Nginx, and use Nginx to proxy Trino's coordinator service, including: Nginx receives the client query request transmitted by Trino, and establishes a connection between the service proxy node and each coordination node of Trino based on the received client query request; The service agent node sends a state sharing task to each of the coordination nodes to obtain the cluster resource utilization rate of each Trino working node; Selecting a number of candidate coordination nodes for executing the current query task based on the task response order of each coordination node and the current task execution rate of each coordination node, wherein the number of the candidate coordination nodes is at least two; The service proxy node sends the client query request to each of the candidate coordination nodes; Each candidate coordination node obtains a trigger condition from the isolated storage node based on the client query request, wherein if the candidate coordination node finds that there is no query mapping corresponding to the client query request in the isolated storage node, the trigger condition is obtained and a query is performed for the client query request; In response to a trigger condition, the isolated storage node monitors and obtains a current query result of the corresponding candidate coordination node; The isolated storage node determines whether the candidate coordination node is faulty based on the update speed of the current query result. If a fault exists, the isolated storage node generates an updated query mapping based on the stored client query request and the corresponding current query result and shares it with the other candidate coordination nodes except the current candidate coordination node, so as to transmit the current query result to the client through the other candidate coordination nodes. Update the current task execution rate of each candidate coordination node based on the candidate number of times, the actual query execution number and the failure number of the candidate coordination node; In response to the task completion status of the candidate coordination node, the isolated storage node saves the currently completed query mapping into the request comparison table and shares it with all coordination nodes; A single candidate node constructs a query plan based on the trigger condition and the client query request and distributes it to each working node for execution, so as to obtain a request execution result, which is recorded as the current query result; The selection of the candidate coordination node includes: The service agent node obtains the cluster resource usage data with timestamps transmitted by each coordination node; Determine the task response order of the corresponding coordination node according to the sequence of each time stamp; Obtaining the current task execution rate and preset task execution rate of each coordination node; Eliminating the corresponding coordination nodes in the task response sequence based on a preset task execution rate; After removing some coordination nodes, reorder the coordination nodes based on the task response order; Selecting a preset number of coordination nodes in the reordering as candidate coordination nodes; The acquisition of the trigger condition also includes: If the candidate coordination node finds that the isolated storage node has a query mapping corresponding to the client's query request, it does not obtain the trigger condition and does not query the client's query request; The step of determining whether the candidate coordination node is faulty includes: The isolated storage node repeatedly monitors and obtains the current query result of the candidate coordination node of the trigger condition based on a preset period; If the update speed of the current query result is less than or equal to the preset update speed, it is determined that the corresponding candidate coordination node is faulty; The preset number is determined according to an average value of the task execution rates of the coordination nodes, and the preset number is negatively correlated with the average value of the task execution rates of the coordination nodes.

2. The TrinoCoordinator high availability method according to claim 1, characterized in that: The current task execution rate of a single candidate coordination node is determined by the following formula: Among them, H represents the current task execution rate of a single alternative coordination node, N1 represents the actual query execution times, N2 represents the alternative times, α represents the influence coefficient of the task execution success rate on the task execution rate, Y represents the number of failures, Y0 represents the preset number of failures, and β represents the influence coefficient of the number of failures on the task execution rate.

3. The TrinoCoordinator high availability method according to claim 2, characterized in that: Each of the coordination nodes determines a delay time for transmitting the cluster resource usage data back to the service agent node based on the execution percentage of the task being executed, wherein: The execution percentage is inversely correlated with the lag time.

4. The TrinoCoordinator high availability method according to claim 1, characterized in that: The query mapping is a data packet with each client query request as the file name, and the data packet stores the query result corresponding to the client query request.

5. The TrinoCoordinator high availability method according to claim 1, characterized in that: After obtaining the cluster resource usage rate of each Tri no working node, the service agent node synchronously shares the received cluster resource usage rate of each Tri no working node with each coordination node so that the calculation data of each coordination node remains consistent.

Images