Dual-mode driven distributed cluster resource scheduling method and device

Abstract

The invention discloses a dual-mode driven distributed cluster resource scheduling method, which comprises the following steps: starting a cluster task and creating a GRPC channel corresponding to the cluster task, and establishing signal communication between the cluster task and a main program module to obtain communication data; judging the working state of the corresponding cluster task based on the communication data or the heartbeat data; driving a monitor module to capture abnormal information data of the cluster task based on the working state; and obtaining monitoring data based on the cluster task in a triggering manner.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of cloud computing, and particularly relates to a dual-mode driven distributed cluster resource scheduling method and device. BACKGROUND

[0002] In a cluster computing system, task scheduling and resource allocation directly affect system running efficiency, resource utilization and task stability. With the expansion of cluster size and the diversification of task types, the accuracy and real-time performance of resource allocation are required to be higher. However, the current mainstream technology has obvious defects. The traditional scheme relies on periodic calibration of resource and task matching relationship, and it is difficult to respond to task quantity fluctuations in time. For example, when the actual number of 5000 tasks is 4000, resource waste or task blocking is easy to occur. In addition, the resource adjustment strategy is aggressive and lacks flexibility. When the task is insufficient, the resource is still allocated according to full load, which further aggravates the mismatch problem. At the same time, the existing technology lacks real-time feedback mechanism of task data and resource state, and information is obtained through periodic polling. It is impossible to perceive the number of deliverable tasks and the state of resources in time when the task is submitted or the connection is established, which leads to scheduling decision deviation. The above problems superimposed will also cause resource pollution risk. When the task is insufficient, full load allocation of resources is easy to cause resource queue congestion and increase the task failure rate, which affects the system stability. These defects have become the key bottleneck restricting the performance improvement of the cluster system. SUMMARY

[0003] To solve the above problems, the present application provides a dual-mode driven distributed cluster resource scheduling method and device. The request trend is predicted through a quadratic exponential smoothing model, the number of concurrent connections is dynamically adjusted by combining an adaptive congestion window algorithm, and efficient utilization and low delay response of resources are realized. Based on weighted round robin and long connection pool maintenance zero delay GRPC channel, the repeated handshake process is skipped, and the throughput performance of high-frequency business is significantly improved. The task state is determined by heartbeat mechanism and communication data. When the task state is abnormal, the information is actively captured and the hierarchical processing strategy is triggered, so that the fault response is rapid and accurate. The task management list dynamically counts the task quantity difference, automatically restarts or rebuilds the task to match the load demand, and prevents the avalanche effect by combining the intelligent avoidance strategy. The modules such as main program, task scheduler and monitor have clear division of labor, realize the full-link automation of resource scheduling, state monitoring and exception handling, and reduce the operation and maintenance complexity.

[0004] The first aspect of the present application provides a dual-mode driven distributed cluster resource scheduling method, which further comprises: starting a cluster task and creating a GRPC channel corresponding to the cluster task, the cluster task establishing signal communication with a main program module to obtain communication data; determining the working state of the corresponding cluster task based on the communication data or heartbeat data; The abnormal information data of the cluster task is captured based on the working state driving monitor module; The monitoring data is acquired based on the cluster task by triggering.

[0005] Preferably, the dual-mode driven distributed cluster resource scheduling method according to claim 1, the step of starting a cluster task and creating a GRPC channel corresponding to the cluster task further comprises: The predicted arrival rate and the interval of the predicted step length are acquired by smoothing extrapolation of the secondary exponential smoothing model based on the RPC arrival interval sequence, and the connection establishment strength is acquired based on the interval of the predicted step length and the predicted arrival rate; The optimal concurrent connection establishment number is acquired based on the arrival rate by an adaptive congestion window algorithm; The zero-latency GRPC channel is acquired based on the weighted round-robin weight vector by maintaining the minimum idle long connection pool.

[0006] Preferably, the step of acquiring the predicted arrival rate and the interval of the predicted step length by smoothing extrapolation of the secondary exponential smoothing model based on the RPC arrival interval sequence further comprises: The smoothing baseline and the smoothing trend are acquired based on the RPC arrival interval sequence, and the calculation expressions are respectively: In the formula, , The smoothing baseline and the smoothing trend are respectively, , The hyperparameters are respectively, The RPC arrival interval sequence is; The interval of the predicted step length and the predicted arrival rate are acquired based on the smoothing baseline and the smoothing trend, and the calculation expression is: In the formula, , The interval of the predicted step length and the predicted arrival rate are respectively, The predicted step length is; if the predicted arrival rate is greater than 0 and the active number of the cluster task is less than the preset active number, the next step is executed, otherwise this step is executed in a loop.

[0007] Preferably, the step of acquiring the optimal concurrent connection establishment number based on the arrival rate by an adaptive congestion window algorithm further comprises: The growth factor and the decay factor are acquired based on the exponential smoothing idle degree, and the calculation expressions are respectively: wherein, the exponential smoothing idle degree, is a hyperparameter, , are respectively a growth factor and a decay factor; Based on the growth factor, the decay factor, and the exponential load penalty, the optimal concurrent connection number is obtained, and the calculation expression is respectively: wherein, , are respectively the optimal concurrent connection number at time t and t-1, is an exponential smoothing success rate; The task set of the optimal concurrent connection number is obtained by the order of the queue head to be connected.

[0008] Preferably, the step of obtaining a zero-latency GRPC channel based on a weighted round-robin weight vector through maintaining a minimum idle long connection pool further comprises: Send HTTP / 2 PING to each of the cluster tasks through a periodic task to obtain the round-trip time and the idle degree, and obtain the weight vector in real time based on the round-trip time and the idle degree; Based on the weight vector, a dynamic pool structure is obtained by reserving a preset number of long connections and releasing weight connections with an idle degree greater than a preset value; Based on the active connection in the dynamic pool structure, an HTTP / 2 idle state and zero-latency ready channel are obtained through TCP / TLS handshake, SessionTicket cache local saving; Based on the ready channel, the calling path that skips the TCP / TLS handshake process is obtained by selecting the connection through weighted round-robin and sending the page number request.

[0009] Preferably, the step of judging the working state of the corresponding cluster task based on the communication data or the heartbeat data further comprises: Start the cluster task and drive the thread to send heartbeat information to the main program module through the corresponding heartbeat thread, if the main program module receives the heartbeat information, the working state of the cluster task is normal, if the main program module cannot receive continuous time sequence heartbeat information, the working state of the cluster task is abnormal; if the working state is abnormal, the monitoring data is obtained through the monitor module.

[0010] Preferably, if the working state is abnormal, the step of obtaining monitoring data by the monitor module further comprises: if the working state is abnormal, judging an abnormal condition of the cluster task, and driving to probe the state of the cluster task based on the abnormal condition, and the specific rule is that if the abnormal condition is network abnormality, insufficient cluster resources, or machine resource abnormality, the state of the cluster task is periodically probed by a command script and is updated.

[0011] Preferably, the step of obtaining monitoring data based on the cluster task by triggering further comprises: If the cluster task is determined to be successfully started and the working state is abnormal, the cluster task is driven to be restarted, and the task information of the cluster task is updated by the task management list module; if the cluster task is determined to be successfully started and the cluster resources are insufficient, the current cluster task is exited, a new cluster task is constructed and started, and the task information of the new cluster task is updated to the task management list module. The task management list module is driven to restart the corresponding cluster task by periodic task detection, and the specific rule is that the number of cluster task running is counted by the task management list module, if the number of cluster task running is less than a preset cluster task number, the serial number of the cluster task to be restarted is obtained and is iterated to be restarted.

[0012] Preferably, the step of driving the cluster task to be restarted and updating the task information of the cluster task by the task management list module further comprises: If the cluster task is in an inactive state and the cluster resources are insufficient, the task scheduler module closes the cluster task and restarts the cluster task consistent with the cluster task, and the task management list module is updated. If the cluster resources are insufficient or the main program module calls a preset number of restart commands and the execution result is failure, the intelligent back-off strategy of the task scheduler module is used for retry, and the specific rule is that the task management list module sets a retry parameter, the retry parameter at least includes a retry number, a retry time, a retry interval, and a maximum waiting time; and the cluster task is started based on the retry parameter.

[0013] The second aspect of the application provides a dual-mode driven distributed cluster resource scheduling device, comprising: A main program module is configured to set a plurality of cluster tasks, construct a command for starting the cluster tasks, and drive the tasks to execute the command to establish communication with the main program module. A task management list module is configured to record working data of the cluster tasks, calculate difference data between an expected task amount and the working data based on the working data, and the working data at least includes a task number, a job identification, a cluster state, a restart number, and a heartbeat time. The task scheduler module is used to periodically check the task management list module and count abnormal cluster tasks, and trigger commands to restart cluster tasks by comparing the currently running cluster tasks with preset cluster tasks. The monitoring module is used to obtain abnormal information data based on heartbeat information.

[0014] This invention, by employing the above technical solutions, possesses the following advantages and positive effects compared to existing technologies: It predicts request trends using a quadratic exponential smoothing model and dynamically adjusts the number of concurrent connections using an adaptive congestion window algorithm, achieving efficient resource utilization and low-latency response. It maintains zero-latency GRPC channels based on weighted round-robin and long connection pooling, skipping repeated handshake processes and significantly improving the throughput performance of high-frequency services. It uses a dual-determination of task status through a heartbeat mechanism and communication data, proactively capturing information and triggering tiered processing strategies in case of anomalies, resulting in rapid and accurate fault response. The task management list dynamically calculates task volume differences, automatically restarting or rebuilding tasks to match load demands, combined with intelligent fallback strategies to prevent avalanche effects. The main program, task scheduler, and monitor modules have clearly defined roles, achieving end-to-end automation of resource scheduling, status monitoring, and anomaly handling, reducing operational complexity. Attached Figure Description

[0015] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is the main flowchart of a dual-mode driven distributed cluster resource scheduling method in this invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description and claims. It should be noted that the drawings are all in a very simplified form and use non-precise ratios, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0017] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0018] First Embodiment See Figure 1 The first aspect of the present invention provides a dual-mode driven distributed cluster resource scheduling method, further comprising: Start the cluster task and create the corresponding GRPC channel for the cluster task. The cluster task establishes signal communication with the main program module to obtain communication data. Judge the working state of the corresponding cluster task based on communication data or heartbeat data; Capture abnormal information data of the cluster task based on the working state driving the monitor module; Based on the cluster task, the monitoring data is acquired by triggering.

[0019] Through the exclusive GRPC channel and zero delay technology, the repeated handshake overhead is eliminated, the microsecond level service response is realized, the low delay demand in high concurrency scene is supported. Combined with business data analysis and heartbeat signal double check, the network jitter, process abnormality and other risks are accurately identified, and the fault discovery time efficiency is improved to millisecond level. Based on the prediction algorithm, the number of concurrent connections and task instances are dynamically adjusted, the resource utilization rate is improved, and the business peak and valley fluctuation is calmly coped with. A hierarchical diagnosis and treatment mechanism is constructed, the mild abnormality is automatically expanded, the severe fault is fused and degraded, the system availability is close to the normal, and the operation and maintenance intervention frequency is greatly reduced. The whole link tracking and visual matrix management are realized, the fault root cause is located one key, the operation and maintenance troubleshooting efficiency is doubled, and the business continuity is guaranteed.

[0020] Preferably, the dual-mode driven distributed cluster resource scheduling method according to claim 1, the step of starting the cluster task and creating the GRPC channel corresponding to the cluster task further comprises: Based on the RPC arrival interval sequence, the predicted arrival rate, the interval of the predicted step length and the connection building strength are obtained by the smoothing extrapolation of the quadratic exponential smoothing model; Based on the arrival rate, the optimal number of concurrent connection building is obtained by the adaptive congestion window algorithm; Based on the weighted round-robin weight vector, the zero-delay GRPC channel is obtained by maintaining the minimum idle long connection pool.

[0021] Through the quadratic exponential smoothing model, the RPC request interval sequence is extrapolated and predicted, the future traffic trend and the connection building strength are dynamically calculated, and the resource preposition is highly matched with the actual load. Based on the adaptive congestion window algorithm, the optimal number of concurrent connection building is adjusted in real time, the system throughput and resource competition are balanced, and the performance degradation caused by overload is avoided. The weighted round-robin strategy is used to maintain the minimum idle long connection pool, and the channel is reused by using the 0-RTT technology, so that the repeated TCP / TLS handshake overhead is completely eliminated. Through the weight vector, the life cycle of the long connection is dynamically managed, the high-value active connection is preferentially retained, the inefficient idle connection is eliminated, and the efficient turnover of the connection pool is realized. By comprehensively considering the prediction accuracy, the concurrent rationality and the connection reuse technology, the service response time delay is significantly reduced, and the overall processing capacity and resource utilization rate of the distributed cluster are improved.

[0022] Preferably, the step of obtaining the predicted arrival rate, the interval of the predicted step length and the connection building strength based on the RPC arrival interval sequence by the smoothing extrapolation of the quadratic exponential smoothing model further comprises: The smooth baseline and the smooth trend are obtained based on the RPC arrival interval sequence, and the expression is calculated as follows: In the formula, , The smooth baseline and the smooth trend are respectively, , The hyperparameters are respectively, The RPC arrival interval sequence is; The interval of the prediction step and the predicted arrival rate are obtained based on the smooth baseline and the smooth trend, and the expression is calculated as follows: In the formula, , The interval of the prediction step and the predicted arrival rate are respectively, The prediction step is; if the predicted arrival rate is greater than 0 and the active number of cluster tasks is less than the preset active number, the next step is executed, otherwise this step is executed in a loop.

[0023] The baseline and trend components are separated by the quadratic exponential smoothing model to realize fine modeling of the RPC request arrival rate and interval, and the prediction accuracy of future load is significantly improved. Based on the prediction result, the connection building strength is dynamically calculated to make the connection pool capacity highly match the actual traffic demand, avoiding resource idling or overload. Only when the predicted arrival rate is greater than 0 and the active task is not saturated, the expansion is triggered to ensure that the resource allocation has both forward-looking and constraint, reducing invalid expansion and contraction operations. Through the hyperparameter weighting mechanism, short-term noise interference is filtered, and stable prediction output can still be maintained under complex traffic mode, improving system reliability. A negative feedback calibration mechanism is adopted, and if the prediction result does not meet the execution condition, the calculation process is automatically restarted to form a closed loop of prediction, verification and correction, and the optimal solution is continuously obtained.

[0024] Preferably, the step of obtaining the optimal concurrent connection building number based on the arrival rate through the adaptive congestion window algorithm further comprises: The growth factor and the decay factor are obtained based on the exponential smoothing idle degree, and the expressions are calculated as follows: In the formula, The exponential smoothing idle degree is, The hyperparameters are respectively, , The growth factor and the decay factor are respectively; The optimal concurrent connection building number is obtained based on the growth factor, the decay factor and the exponential load penalty, and the expressions are calculated as follows: wherein, , are the optimal concurrent connection numbers at time t and t-1 respectively, is the exponential smoothing success rate; The optimal concurrent connection number is obtained by sequentially obtaining the task set from the head of the to-be-pre-connected queue.

[0025] The growth / decay factor is calculated in real time by exponential smoothing the idle degree, so that the concurrent connection number automatically adapts to the load change, realizes the precise matching of resource supply and demand, filters short-term traffic noise by using the hyperparameter weighting mechanism, avoids the instability of the connection pool caused by frequent jitter, and guarantees the smooth operation of the system. The exponential load penalty term is introduced to actively suppress excessive expansion of connections at high load, form a safety boundary, and effectively prevent the avalanche effect. The time series recursive model is used to fuse historical experience, so that the concurrent number adjustment has both immediate response and long-term optimization, and the system tends to be stable. The connection resources are allocated according to the head of the to-be-pre-connected queue, taking into account fairness and priority of urgent tasks, and improving the efficiency of key business processing.

[0026] Preferably, the step of obtaining a zero-latency GRPC channel based on a weighted round-robin weight vector through maintaining a minimum idle long connection pool further comprises: Send HTTP / 2 PING to each cluster task through periodic tasks to obtain round-trip time and idle degree, and obtain a weight vector based on the round-trip time and the idle degree in real time; Based on the weight vector, a dynamic pool structure is obtained by reserving a preset number of long connections and releasing weight connections with an idle degree greater than a preset value; Based on the active connections in the dynamic pool structure, HTTP / 2 idle state and zero-latency ready channels are obtained through TCP / TLS handshake, SessionTicket cache local saving; Based on the ready channel, the calling path that skips the TCP / TLS handshake process is obtained by selecting a connection through weighted round-robin and sending a page request.

[0027] HTTP / 2 PING is an application-layer mechanism used to detect and manage the health of HTTP / 2 connections, including verifying connection validity, measuring round-trip latency, and triggering a response from the other party to confirm normal bidirectional communication. HTTP / 2 is an application-layer protocol that optimizes data transmission efficiency through multiplexing, header compression, and server push technologies, aiming to significantly improve webpage loading speed and resource utilization. TCP in TCP / TLS is responsible for reliable data transmission over the network, while TLS provides encryption, authentication, and data integrity protection. Together, they ensure reliable and secure network communication. SessionTicket is a protocol extension mechanism that enables rapid authentication of subsequent connections by pre-stored encrypted session credentials, eliminating the need to repeat the complete handshake process, thus significantly reducing latency and improving performance. By periodically probing the round-trip latency and idle time of cluster tasks through HTTP / 2 PING, a dynamic weight evaluation system is constructed to quantify the health and availability of each connection. Based on weight vectors, inefficient and idle connections are dynamically eliminated, retaining only a preset number of high-value long-lived connections, forming a lightweight and highly active dynamic pool structure, reducing unnecessary resource consumption. By leveraging SessionTicket local caching technology, active connections are solidified into ready channels in the HTTP / 2 idle state, completely eliminating TCP / TLS handshake time and achieving true zero-latency communication. Through a weight-based connection selection strategy, low-latency, high-idleness high-quality connections are prioritized for reuse, allowing business requests to directly access established channels and significantly improving concurrent processing efficiency.

[0028] Preferably, the step of determining the working status of the corresponding cluster task based on communication data or heartbeat data further includes: The cluster task is started and a corresponding heartbeat thread drives the thread to send heartbeat information to the main program module. If the main program module receives the heartbeat information, the cluster task is in a normal working state. If the main program module does not receive continuous heartbeat information, the cluster task is in an abnormal working state. If the working state is abnormal, monitoring data is obtained through the monitoring module.

[0029] By periodically reporting status through an independent heartbeat thread, the main program module can perceive the cluster task liveness status in real time, achieving fault detection within seconds. A continuous time-series heartbeat verification mechanism effectively filters out occasional network jitter, accurately distinguishes between normal and abnormal states, and reduces the false alarm rate. Once an anomaly is detected, the monitor is automatically triggered to collect detailed diagnostic data, providing a basis for quickly locating the root cause. Clear health status identifiers provide reliable triggering conditions for subsequent automated recovery strategies. This transforms passive fault diagnosis into proactive status monitoring, significantly improving the observability and operational efficiency of the distributed cluster.

[0030] Preferably, if the working status is abnormal, the step of obtaining monitoring data through the monitoring module further includes: if the working status is abnormal, determining the abnormal situation of the cluster task, and driving the detection of the status of the cluster task based on the abnormal situation. The specific rule is: if the abnormal situation is a network abnormality, insufficient cluster resources, or abnormal machine resources, the status of the cluster task is periodically detected and updated through a command script.

[0031] By refining scenarios such as network anomalies, insufficient cluster resources, and abnormal machine resources, the system achieves precise identification of fault types, avoiding ineffective responses caused by general alarms. Periodic command script probes are triggered for specific anomaly types to dynamically verify the true status of tasks and effectively distinguish between short-term fluctuations and persistent faults. The cluster task status is automatically updated after each probe, ensuring the main program module has the latest health information and providing a reliable data foundation for subsequent decisions. Differentiated probe strategies are matched according to the anomaly level, optimizing the balance between resource consumption and fault recovery efficiency. This forms a complete closed loop of anomaly detection, root cause localization, and status updates, providing precise triggering criteria for automated operation and maintenance.

[0032] Preferably, the step of obtaining monitoring data based on cluster tasks via triggering further includes: If the cluster task is determined to have started successfully but its working status is abnormal, the driver restarts the cluster task and updates the task information of the cluster task through the task management list module; if the cluster task is determined to have started successfully but the cluster resources are insufficient, the current cluster task is exited, a new cluster task is created and started, and the task information of the new cluster task is updated to the task management list module. The task management list module drives the restart of corresponding cluster tasks through periodic task detection. The specific rules are as follows: the task management list module counts the number of cluster tasks running. If the number of cluster tasks running is less than the preset number of cluster tasks, the sequence number of the cluster tasks that need to be restarted is obtained and the process is repeated to restart them.

[0033] When a cluster task fails to start but starts successfully, the system automatically triggers a restart and updates the task management list synchronously, shortening business interruption time and ensuring service continuity. For scenarios with insufficient cluster resources, inefficient tasks are proactively terminated and new tasks are rebuilt, achieving efficient resource replacement and dynamic scaling, avoiding global crashes caused by resource exhaustion. By periodically counting the number of running tasks and comparing it with preset values, missing tasks are proactively detected and restarted, ensuring the cluster always maintains the service capacity of the target scale. Combining task status and resource conditions, restart or creation strategies are dynamically executed, maximizing resource utilization while ensuring service quality. The task lifecycle is centrally managed through the task management list, forming a complete closed loop of monitoring, decision-making, execution, and feedback, significantly improving the self-healing capability and operational efficiency of the distributed cluster.

[0034] Preferably, the step of driving the restart of the cluster task and updating the task information of the cluster task through the task management list module further includes: If a cluster task is inactive and cluster resources are insufficient, the task scheduler module shuts down the cluster task and restarts the corresponding cluster task, and updates the task management list module. If cluster resources are insufficient or the main program module calls the restart command a preset number of times and the execution result fails, the intelligent fallback strategy of the task scheduler module will be used for retrying. The specific rules are as follows: the task management list module sets retry parameters, which include at least the number of retries, retry time, retry interval, and maximum waiting time; the cluster task is started based on the retry parameters.

[0035] When cluster tasks are inactive and resources are insufficient, the task scheduler automatically shuts down old tasks and starts new instances, achieving efficient resource reclamation and reallocation to ensure the continuous operation of critical businesses. For scenarios with frequent restart failures, an intelligent fallback strategy avoids resource exhaustion caused by infinite retries, improving system resilience. Flexible retry parameter configuration is supported, allowing for customized retry strategies based on business needs, balancing fault recovery success rate and resource consumption. Task status is centrally maintained through a task management list, ensuring the atomicity and consistency of task start, stop, and update operations, avoiding cascading problems caused by state chaos. The restart and replacement of abnormal tasks are handled automatically, reducing the need for manual intervention, while providing observable retry logs to accelerate troubleshooting and system optimization.

[0036] Second Embodiment A second aspect of the present invention provides a dual-mode driven distributed cluster resource scheduling device, comprising: The main program module is used to set up several cluster tasks, construct commands to start cluster tasks, execute task commands, and drive tasks to establish communication with the main program module. The task management list module is used to record the working data of cluster tasks and calculate the difference between the working data and the expected task volume. The working data includes at least the task number, job identifier, cluster status, number of restarts, and heartbeat time. The task scheduler module is used to periodically check the task management list module and count abnormal cluster tasks, and to trigger commands to restart cluster tasks by comparing the currently running cluster tasks with preset cluster tasks. The monitoring module is used to obtain abnormal information data based on heartbeat information.

[0037] The main program module centrally manages the creation and communication of cluster tasks, standardizing task deployment and maximizing command execution efficiency, thus reducing management complexity. The task management list module quantifies task status through multi-dimensional work data and analyzes expected task volume differences, providing precise data for resource scheduling. The task scheduler module periodically detects abnormal tasks and proactively triggers restart commands, forming a closed loop of detection, response, and repair, significantly improving the cluster's self-healing capabilities and availability. The monitoring module continuously collects abnormal data based on heartbeat information, providing a fine-grained view of task health, facilitating rapid location of network, resource, or process-level failures. Dynamic management of metadata such as task numbers and job identifiers, combined with a preset task volume comparison mechanism, provides a flexible and reliable execution foundation for the elastic scaling of cluster resources.

[0038] In the description of this application, it should be noted that the terms "inner" and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. They are used only for the convenience of describing this application and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0039] It should also be noted that, unless otherwise explicitly specified and limited, the terms "setup" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0040] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific identification content executed by the system and device described above can be referred to the corresponding process in the foregoing method embodiments.

[0041] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they shall still fall within the protection scope of the present invention.

Claims

1. A dual-mode driven distributed cluster resource scheduling method, characterized in that, Also includes: Start the cluster task and create the corresponding GRPC channel for the cluster task. The cluster task establishes signal communication with the main program module to obtain communication data. The working status of the corresponding cluster task is determined based on communication data or heartbeat data. The monitoring module captures abnormal information data of the cluster tasks based on their working status. Based on the cluster task, monitoring data is obtained through triggering.

2. The dual-mode driven distributed cluster resource scheduling method according to claim 1, characterized in that, The steps of starting a cluster task and creating a corresponding gRPC channel for the cluster task further include: Based on the RPC arrival interval sequence, the predicted arrival rate is obtained through smooth extrapolation using a quadratic exponential smoothing model, and the connection strength is obtained through the interval of the predicted step size. The optimal number of concurrent connections is obtained based on the arrival rate using an adaptive congestion window algorithm. Zero-latency GRPC channels are obtained by maintaining a minimum idle long connection pool based on weighted polling weight vectors.

3. The dual-mode driven distributed cluster resource scheduling method according to claim 2, characterized in that, The steps of obtaining the predicted arrival rate and the connection strength based on the RPC arrival interval sequence through smooth extrapolation using a quadratic exponential smoothing model further include: The smoothed baseline and smoothed trend are obtained based on the RPC arrival interval sequence, and the calculation expressions are as follows: In the formula, , The smoothing baseline and the smoothing trend are respectively, , These are hyperparameters, The RPC arrival interval sequence; Based on the smoothed baseline and the smoothed trend, the interval of the prediction step and the prediction arrival rate are obtained, and the calculation expression is: In the formula, , These are the interval of the prediction step size and the prediction arrival rate, respectively. The prediction step size is determined by the following steps: if the predicted arrival rate is greater than 0 and the number of active tasks in the cluster is less than the preset number of active tasks, then proceed to the next step; otherwise, repeat this step in a loop.

4. The dual-mode driven distributed cluster resource scheduling method according to claim 2, characterized in that, The step of obtaining the optimal number of concurrent connections based on the arrival rate using an adaptive congestion window algorithm further includes: The growth factor and decay factor are obtained based on exponentially smoothed idle time, and their calculation expressions are as follows: In the formula, Exponentially smoothed idle time, For hyperparameters, , These are the growth factor and the decay factor, respectively. The optimal number of concurrent connections is obtained based on the growth factor, the decay factor, and the exponential load penalty, and the calculation expressions are as follows: In the formula, , These represent the optimal number of concurrent connections established for time series t and t-1, respectively. To smooth the success rate exponentially; The optimal set of tasks for concurrent connection establishment is obtained sequentially from the head of the queue to be pre-connected.

5. The dual-mode driven distributed cluster resource scheduling method according to claim 2, characterized in that, The step of obtaining zero-latency gRPC channels by maintaining a minimum idle long connection pool based on a weighted round-robin weight vector further includes: The round-trip time and idle time are obtained by sending HTTP / 2 PING to each cluster task through periodic tasks, and the weight vector is obtained in real time based on the round-trip time and idle time. Based on the weight vector, a dynamic pool structure is obtained by retaining a preset number of long connections and releasing weighted connections with idle time greater than a preset value. Based on the active connections in the dynamic pool structure, a ready channel with zero latency is obtained through TCP / TLS handshake and local storage of SessionTicket cache; Based on the ready channel, a connection is selected through weighted polling and a page number request is sent to obtain a calling path that skips the TCP / TLS handshake process.

6. The dual-mode driven distributed cluster resource scheduling method according to claim 1, characterized in that, The step of determining the working status of the corresponding cluster task based on communication data or heartbeat data further includes: The cluster task is started and the corresponding heartbeat thread drives the thread to send heartbeat information to the main program module. If the main program module receives the heartbeat information, the working state of the cluster task is normal. If the main program module does not receive continuous heartbeat information, the working state of the cluster task is abnormal. If the working state is abnormal, monitoring data is obtained through the monitoring module.

7. The dual-mode driven distributed cluster resource scheduling method according to claim 6, characterized in that, If the working status is abnormal, the step of obtaining monitoring data through the monitoring module further includes: if the working status is abnormal, determining the abnormal situation of the cluster task, and driving the detection of the status of the cluster task based on the abnormal situation. The specific rule is: if the abnormal situation is a network abnormality, insufficient cluster resources, or abnormal machine resources, the status of the cluster task is periodically detected and updated through a command script.

8. The dual-mode driven distributed cluster resource scheduling method according to claim 1, characterized in that, The step of obtaining monitoring data through triggering based on the cluster task further includes: If the cluster task is determined to have started successfully but its working status is abnormal, the driver restarts the cluster task and updates the task information of the cluster task through the task management list module; if the cluster task is determined to have started successfully but the cluster resources are insufficient, the current cluster task is exited, a new cluster task is created and started, and the task information of the new cluster task is updated to the task management list module. The task management list module is used to drive the corresponding cluster tasks to restart through periodic task detection. The specific rule is as follows: the task management list module counts the number of cluster tasks running. If the number of cluster tasks running is less than the preset number of cluster tasks, the sequence number of the cluster tasks that need to be restarted is obtained and the task is restarted by iterating through them.

9. The dual-mode driven distributed cluster resource scheduling method according to claim 8, characterized in that, The steps of restarting the cluster task and updating the task information of the cluster task through the task management list module further include: If a cluster task is inactive and cluster resources are insufficient, the task scheduler module shuts down the cluster task and restarts the corresponding cluster task, and updates the task management list module. If cluster resources are insufficient or the main program module calls the restart command a preset number of times and the execution result is a failure, a retry will be performed through the intelligent fallback strategy of the task scheduler module. The specific rules are as follows: the task management list module sets retry parameters, which include at least the number of retries, retry time, retry interval, and maximum waiting time; the cluster task is started based on the retry parameters.

10. A dual-mode driven distributed cluster resource scheduling device, characterized in that, include: The main program module is used to set up several cluster tasks, construct commands to start the cluster tasks, execute commands to drive the tasks to establish communication with the main program module; The task management list module is used to record the working data of the cluster tasks and calculate the difference data between the working data and the expected task volume. The working data includes at least the task number, job identifier, cluster status, number of restarts, and heartbeat time. The task scheduler module is used to periodically check the task management list module and count abnormal cluster tasks, and trigger commands to restart cluster tasks by comparing the currently running cluster tasks with preset cluster tasks. The monitoring module is used to obtain abnormal information data based on heartbeat information.

Images