MPP cluster exception handling method and device and storage medium

Abstract

Embodiments of the invention provide an MPP cluster exception handling method and apparatus, and a storage medium. The method comprises the steps of obtaining a to-be-executed job; when the to-be-executed jobs are dominant faults, whether the queuing time of the to-be-executed jobs exceeds a first threshold value or not is judged, and if the queuing time of the to-be-executed jobs exceeds the first threshold value, the jobs are submitted and removed out of a to-be-distributed job queue; if the first threshold value is not exceeded, the to-be-executed job is queued continuously; when the to-be-executed job is a hidden fault, whether the queuing time of the to-be-executed job exceeds a second threshold value or not is judged, and if the queuing time of the to-be-executed job exceeds the second threshold value, the to-be-executed job is submitted and removed out of the queuing queue; if the number does not exceed a second threshold value, judging the number of job instances which are associated with the metadata cluster in the to-be-executed job and are in operation, and if the number of the job instances does not exceed a third threshold value, submitting the to-be-executed job and removing the to-be-distributed job queue; and if so, continuing to queue the to-be-executed jobs. According to the method, invalid scheduling submission is reduced, and the automation level is improved.

Description

Technical Field

[0001] This application relates to the field of computer and job batch processing scheduling technology, specifically to a method, apparatus and storage medium for handling anomalies in an MPP cluster. Background Technology

[0002] In the field of batch processing scheduling, Extract-Transform-Load (ETL) jobs in traditional data warehouses are the core of data processing and analysis. These jobs need to be executed in an orderly manner according to preset logic through a scheduling system to achieve data extraction, transformation, and loading. Among them, the actual operation of some ETL jobs heavily relies on the underlying Massively Parallel Processing (MPP) clusters—MPP clusters, with their distributed computing architecture, provide computing power support for large-scale data processing and are the key infrastructure to ensure the efficient operation of ETL jobs.

[0003] As business data volume grows, the number of ETL jobs relying on MPP clusters gradually increases. When the number of jobs reaches a certain level, the operational status of the MPP cluster has a more significant impact on job scheduling. In practical applications, MPP clusters may experience various abnormal situations such as metadata cluster failures and compute cluster failures due to hardware failures, software anomalies, network interruptions, etc. These anomalies will directly lead to the interruption of ETL jobs that depend on the cluster, thereby affecting the data processing flow and business analysis timeliness of the entire data warehouse.

[0004] To address the job scheduling issue when MPP clusters malfunction, the industry currently mainly adopts the following two technical solutions: manual intervention rerun solution and automatic rerun solution by the scheduling system.

[0005] The manual intervention rerun scheme relies on manual operation to handle job failures caused by MPP cluster anomalies, and the scheduling system lacks proactive control logic for MPP cluster anomalies. The specific process is as follows: In the scheduling system, corresponding MPP data source information is configured for each job that depends on the MPP cluster, ensuring that the job can associate with the target MPP cluster during runtime; the scheduling system triggers job execution according to a preset cycle or dependency relationship; when the MPP data source associated with the job (i.e., the underlying MPP cluster) experiences an anomaly, the job cannot be executed normally, and the scheduling system marks the job status as "failed"; after project personnel detect job failure, they need to confirm the cause of the failure and wait for the MPP cluster maintenance personnel to complete the fault repair until the MPP cluster returns to normal operation; after the MPP cluster recovers, project personnel manually filter out the jobs that failed due to MPP cluster anomalies in the scheduling system and trigger the job rerun operation to complete the unexecuted job process.

[0006] The existing solution has obvious drawbacks: regardless of the number of jobs affected by the MPP cluster anomaly, project personnel must be involved throughout the entire process, from monitoring job failures and confirming the cause of the failure to manually rerunning the jobs after the cluster recovers. The entire process relies on manual operation, which is costly in terms of manpower and is prone to missed or incorrect job runs due to human error. At the same time, the recovery and execution of jobs depend entirely on the speed and efficiency of manual response. If the MPP cluster failure lasts for a long time or project personnel fail to handle it in a timely manner, the jobs will remain in a failed state for a long time, which will delay the data warehouse data update and business analysis progress, and the job execution efficiency cannot be reliably guaranteed.

[0007] The automatic rerun scheme of the scheduling system utilizes the built-in automatic rerun function of the scheduling system to reduce manual intervention. The specific process is as follows: two configurations are completed in the scheduling system. First, the corresponding MPP data source information is configured for jobs that depend on the MPP cluster. Second, the "automatic rerun" function is enabled for the job, and rerun parameters are set (including but not limited to rerun interval and maximum number of reruns). After the scheduling system triggers the job to run, if the MPP cluster associated with the job fails, the scheduling system marks the job status as waiting to be rerun. The scheduling system triggers the job to rerun periodically according to the preset rerun parameters until the MPP cluster recovers and the job is successfully executed.

[0008] This solution also has significant problems: To reduce manual intervention, the automatic rerun function needs to be enabled indiscriminately for all jobs that rely on the MPP cluster; however, since the scheduling system cannot predict the recovery time of the MPP cluster in advance, during the cluster failure, the job will repeatedly rerun at preset intervals. These reruns all fail due to the cluster anomaly, which is an ineffective scheduling. This will consume a large amount of the scheduling system's normal scheduling resources (such as scheduling queues and task threads), causing delays in the scheduling of other normal jobs that do not rely on the MPP cluster, and significantly reducing the overall scheduling efficiency. In addition, the scheduling system needs to schedule the job to the execution node and start the job before it can detect the MPP cluster anomaly through job execution failure. Therefore, during the MPP cluster failure, each automatic rerun will consume the execution node's computing resources (such as CPU and memory), but the job will ultimately fail to complete due to the cluster anomaly, resulting in the ineffective use of computing resources and serious resource waste. Summary of the Invention

[0009] The purpose of this application is to provide a method, apparatus, and storage medium for handling MPP cluster anomalies. This method reduces invalid scheduling submissions and improves the level of automation.

[0010] To achieve the above objectives, the first aspect of this application provides a method for handling MPP cluster anomalies, the method comprising: Obtain the jobs to be executed from the job queue to be dispatched, wherein the data source of the jobs to be executed involves an MPP data source; Determine whether the task to be executed is an explicit or implicit fault; If the pending job is an explicit fault, it is determined whether the queuing time of the pending job exceeds a first threshold. If it exceeds the first threshold, the job is submitted and removed from the pending job queue; if it does not exceed the first threshold, the pending job continues to be queued. If the pending job is a hidden fault, it is determined whether the queuing time of the pending job exceeds the second threshold. If it exceeds the second threshold, the pending job is submitted and removed from the queuing queue. If it does not exceed the second threshold, it is determined whether the number of job instances associated with the metadata cluster of the pending job is running. If the number of job instances does not exceed the third threshold, the pending job is submitted and removed from the job queue to be dispatched. If it exceeds the third threshold, the pending job continues to be queued.

[0011] Optionally, the method further includes: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs is in an explicit fault and the queuing time of the job to be executed has not exceeded the first threshold, the latest metadata cluster status is obtained by calling the MPP resource scheduling service. If the metadata cluster has recovered to health, the job to be executed is started normally and removed from the job queue to be dispatched.

[0012] Optionally, the method further includes: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs is in a hidden fault and the queuing time of the job to be executed does not exceed the second threshold, the MPP resource scheduling service is called to obtain the latest metadata cluster status. If the metadata cluster has recovered to health, the job is started normally and removed from the queuing queue.

[0013] Optionally, the method further includes: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs is in an explicit fault, and the queuing time of the job to be executed does not exceed the first threshold, then register the current system time and queuing type for storage in the job queue to be dispatched.

[0014] Optionally, the explicit failure is an anomaly in the MPP computing cluster and / or metadata cluster involved in the job; The latent fault is a CPU and / or memory anomaly occurring on a node of the MPP computing cluster involved in the job.

[0015] Optionally, the method further includes: periodically collecting the status of the MPP cluster and metadata cluster of the job to be executed, and caching them in a Redis database for periodic fault diagnosis.

[0016] A second aspect of this application provides an apparatus for handling MPP cluster anomalies, the apparatus comprising: The acquisition module is used to acquire the jobs to be executed in the job queue to be dispatched, wherein the data source of the jobs to be executed involves an MPP data source; The first processing module is used to determine whether the task to be executed is an explicit fault or a hidden fault; The second processing module is used to determine whether the queuing time of the job to be executed exceeds a first threshold when the job to be executed is an explicit fault. If it exceeds the first threshold, the job is submitted and removed from the job queue. If it does not exceed the first threshold, the job to be executed continues to be queued. The third processing module is used to determine whether the queuing time of the job to be executed exceeds a second threshold when the job to be executed is a hidden fault. If it exceeds the second threshold, the job to be executed is submitted and removed from the queuing queue. If it does not exceed the second threshold, the module determines the number of job instances associated with the metadata cluster of the job to be executed and in operation. If the number of job instances does not exceed a third threshold, the job to be executed is submitted and removed from the job queue to be dispatched. If it exceeds the third threshold, the job to be executed continues to be queued.

[0017] Optionally, the device further includes: a third processing module, configured to, when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs is in an explicit fault and the queuing time of the job to be executed has not exceeded a first threshold, obtain the latest metadata cluster status by calling the MPP resource scheduling service; if the metadata cluster has recovered to health, then normally initiate the job to be executed and remove it from the job queue to be dispatched.

[0018] Optionally, the device further includes: a fourth processing module, used to call the MPP resource scheduling service to obtain the latest metadata cluster status when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs to has a hidden fault and the queuing time of the job to be executed has not exceeded the second threshold; if the metadata cluster has recovered to health, the job is started normally and removed from the queuing queue.

[0019] A third aspect of this application provides a processor configured to execute the method for handling MPP cluster anomalies as described above.

[0020] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method for handling MPP cluster anomalies as described above.

[0021] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method for handling MPP cluster anomalies as described above.

[0022] A method for handling MPP cluster anomalies according to this application includes: obtaining jobs to be executed in a queue of jobs to be dispatched, wherein the data source of the jobs to be executed involves an MPP data source; determining whether the jobs to be executed are explicit or implicit; if the jobs to be executed are explicit, determining whether the queuing time of the jobs to be executed exceeds a first threshold; if it exceeds the first threshold, submitting the jobs and removing them from the queue of jobs to be dispatched; if it does not exceed the first threshold, the jobs to be executed continue to be queued; if the jobs to be executed are implicit, determining whether the queuing time of the jobs to be executed exceeds a second threshold; if it exceeds the second threshold, submitting the jobs to be executed and removing them from the queue of jobs to be dispatched; if it does not exceed the second threshold, determining the number of job instances associated with the metadata cluster of the jobs to be executed and currently running; if the number of job instances does not exceed a third threshold, submitting the jobs to be executed and removing them from the queue of jobs to be dispatched; if it exceeds the third threshold, the jobs to be executed continue to be queued. This method distinguishes between explicit and implicit faults. In the case of explicit MPP faults, it reduces invalid scheduling submissions; in the case of implicit MPP faults, it prevents excessive job submissions from overwhelming the cluster, thus improving cluster stability. Overall, it improves job runtime efficiency, prevents other computing resources from being wasted, reduces manual intervention costs for project teams, and increases automation.

[0023] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The illustration shows a flowchart of a method for handling MPP cluster anomalies according to an embodiment of this application. Figure 2 A schematic diagram illustrating an embodiment of this application is shown. Figure 3 This illustration shows a schematic diagram of the MPP cluster status acquisition sub-process according to this application; Figure 4 This schematic diagram illustrates the scheduling queuing control flow under abnormal conditions of an MPP cluster according to this application. Figure 5 This schematic diagram illustrates a structural block diagram of an apparatus for handling MPP cluster anomalies according to an embodiment of this application. Figure 6 The diagram illustrates the internal structure of a computer device according to an embodiment of this application.

[0025] Explanation of reference numerals in the attached figures Device for handling 100-MPP cluster anomalies; 200 - Acquisition Module; 300 - First Processing Module; 400 - Second Processing Module; 500 - Third processing module. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0027] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0028] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0029] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant laws and regulations. Furthermore, it should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0030] Figure 1 This illustration schematically shows a flowchart of a method for handling MPP cluster anomalies according to an embodiment of this application, such as... Figure 1 and Figure 2 As shown, the method includes: Step 101: Obtain the jobs to be executed from the job queue to be dispatched. The data source of the jobs to be executed involves an MPP data source. Specifically, once all the dependencies of the job are ready, they are placed in the job queue to be dispatched, waiting to be pulled and executed by the job execution service. MPP is a distributed database system for massively parallel processing, often used to build data warehouses. The MPP architecture of this application is a storage-compute separation architecture, including a computing cluster and a metadata cluster, wherein one metadata cluster contains multiple computing clusters.

[0031] In this system, a job is the basic configuration unit of the scheduling system; it is a user-defined logical unit that completes a specific task, including the program to be executed and its parameters. In task scheduling, a job (or task) is the smallest unit of execution. The dispatch queue is implemented using RocketMQ (a high-performance, reliable message middleware that supports message publishing and subscription, and automatic retry mechanisms for failed consumption). Job instances are placed in the job queue for distributed processing by the job execution service. The job execution service is a component of the task scheduling system, responsible for initiating jobs and monitoring their status.

[0032] Step 102: Determine whether the task to be executed is an explicit fault or a implicit fault.

[0033] According to a preferred embodiment, the explicit fault is an anomaly in the MPP computing cluster and / or metadata cluster involved in the job; the implicit fault is a CPU and / or memory anomaly in a node of the MPP computing cluster involved in the job.

[0034] The explicit failure (MPP explicit failure) refers to the state when the computing cluster or metadata cluster is explicitly unable to provide services. The implicit failure (MPP implicit failure) only occurs in the computing cluster. When the physical resources (CPU / memory) of a node in the computing cluster are abnormal, it slows down the performance of the entire computing cluster, causing the computing cluster to be in a sub-healthy state.

[0035] The core characteristic of explicit failures is the complete offline status of the computing cluster or metadata cluster, resulting in direct failure of business requests. The failure symptoms are obvious and cannot be ignored. Whether the entire cluster is unavailable or a core component (such as a metadata management node) fails, users will be unable to perform any operations; this is a zero-sum game failure. The core characteristic of implicit failures is that the computing cluster is not completely interrupted, but the physical resources (CPU / memory) of some nodes are abnormal, leading to an overall performance decline. The failure symptoms are not obvious (easily mistaken for "normal delays during peak business hours"). This occurs only in the computing cluster, and the faulty nodes are not down, but their resources are in an abnormal state (such as full CPU load or memory leaks), thus slowing down the entire parallel computing task.

[0036] Specifically, the explicit fault refers to a failure in the MPP computing cluster / metadata cluster involved in the job. In this type of fault scenario, abnormalities are predictable during job execution. The implicit fault, for example, is a CPU / memory failure on a node of the MPP computing cluster, which slows down the computing performance of the entire cluster. In this case, errors or backlogs may occur after the job is submitted for MPP execution.

[0037] For example, the metadata cluster is the brain of the MPP architecture, responsible for storing critical information such as table structure, data shard locations, and permission configurations. All computational tasks (such as queries and writes) must first obtain the data location from the metadata cluster before scheduling computation nodes for execution. Explicit failure triggers include the sudden downtime of the metadata cluster's master node due to hardware damage (e.g., motherboard burnout), system crash (e.g., kernel panic), or software bug (e.g., metadata service deadlock), and the failure of the backup node to automatically switch over in a timely manner (or the absence of a backup node), causing the entire MPP cluster to completely lose its service capability. This is a typical explicit failure. MPP computation relies on the parallel processing of sharded data by each node. If the CPU of a node is occupied by non-business processes (e.g., virus processes, accidentally started batch scripts), the processing speed of that node will decrease significantly, thus slowing down the entire task (because parallel tasks must wait for the slowest node to complete). Explicit failure triggers include node A of the MPP computation cluster starting an infinite loop test script due to operator error, causing node A's CPU utilization to remain at 99%-100% for an extended period (normal business CPU utilization is approximately 30%-60%).

[0038] like Figure 3As shown, the MPP cluster status collection sub-process is responsible for collecting various metrics of the underlying computing platform and caching them in Redis (a high-performance cache database) for the job scheduling and execution service to use when initiating MPP jobs. The MPP resource scheduling service periodically (e.g., every 5 minutes) queries the MPP console service for the status of each MPP computing cluster and its associated metadata cluster, and updates the Redis cache with the corresponding explicit and implicit cluster faults. Explicit faults include two situations: metadata cluster failure and the computing cluster explicitly unable to provide service. Implicit faults include the following situation: a node in the computing cluster has a CPU / memory ratio that deviates significantly from normal values ​​compared to other nodes. To avoid error control, all other situations are considered normal for the MPP cluster.

[0039] Step 103: When the job to be executed is an explicit fault, determine whether the queuing time of the job to be executed exceeds the first threshold. If it exceeds the first threshold, submit the job and remove it from the job queue to be dispatched. If it does not exceed the first threshold, the job to be executed continues to queue.

[0040] According to a preferred embodiment, the method further includes: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs is experiencing an explicit failure and the queuing time of the job to be executed has not exceeded a first threshold, then the latest metadata cluster status is obtained by calling the MPP resource scheduling service; if the metadata cluster has recovered to health, then the job to be executed is started normally and removed from the job queue. The MPP resource scheduling service is mainly responsible for collecting the status of each MPP cluster and caching it in Redis.

[0041] The method further includes: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs is in an explicit fault, and the queuing time of the job to be executed does not exceed the first threshold, the current system time and queuing type are registered for storage in the job queue to be dispatched.

[0042] Specifically, such as Figure 4As shown, the scheduling queuing control process under abnormal MPP cluster conditions includes the following steps: Step 1: The scan and retry thread starts running to obtain a set of MPP job instances waiting in the queue for retry; Step 2: Determine whether the queuing type of the queued job instances is an MPP explicit fault or a implicit fault. If so, proceed to Step 3; otherwise, take the next instance and continue processing; Step 3: Since the self-healing process of the MPP cluster takes a certain amount of time, to avoid excessively frequent scanning, determine whether the scan processing interval of the next job has exceeded 15 seconds. If so, take the next instance for processing; otherwise, proceed to Step 4; If the queuing type is an MPP explicit fault, determine whether the queuing time of the job instance has exceeded the set threshold (this step is to avoid the job being in a queue for a long time). If it has exceeded the threshold, the job is started normally and removed from the queue. If the queuing time has not exceeded the threshold, the latest cluster status is obtained by calling the MPP resource scheduling service. If the cluster has recovered to health, the job is started normally and removed from the queue. If the cluster is still in an explicit fault state, the queuing continues.

[0043] This method prioritizes avoiding jobs that fail directly due to temporary faults (such as situations where a restart of the metadata cluster would restore the job), allowing sufficient recovery time through queuing; it leverages Redis caching to quickly obtain the latest cluster status, avoiding repeated access to the faulty cluster; and it uses a first threshold to control queuing costs, preventing jobs from waiting indefinitely and impacting business operations, thus balancing fault tolerance and timeliness.

[0044] Step 104: If the job to be executed is a hidden fault, determine whether the queuing time of the job to be executed exceeds the second threshold. If it exceeds the second threshold, submit the job to be executed and remove it from the queuing queue. If it does not exceed the second threshold, determine the number of job instances associated with the metadata cluster of the job to be executed and in operation. If the number of job instances does not exceed the third threshold, submit the job to be executed and remove it from the job queue to be dispatched. If it exceeds the third threshold, the job to be executed continues to be queued.

[0045] According to a preferred embodiment, the method further includes: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs is in a hidden fault and the queuing time of the job to be executed does not exceed the second threshold, the MPP resource scheduling service is called to obtain the latest metadata cluster status; if the metadata cluster has recovered to health, the job is started normally and removed from the queuing queue.

[0046] Specifically, if the queuing type is MPP latent fault, it checks whether the queuing time of the job instance has exceeded a set threshold (this threshold is less than the time threshold for explicit faults; in the case of a latent fault, MPP is likely still able to provide normal service). If it exceeds the threshold, the job is started normally and removed from the queue. If the queuing time does not exceed the threshold, the latest cluster status is obtained by calling the MPP resource scheduling service. If the cluster has recovered, the job is started normally and removed from the queue. If the cluster is still in a latent fault state, it checks the number of job instances associated with and running in the cluster. If it does not exceed the threshold, the job is submitted again and removed from the queue; if it exceeds the threshold, the queue continues to be queued to control the rate at which jobs are submitted to the cluster.

[0047] This method, in the sub-healthy state of an MPP cluster, uses fine-grained queuing control and concurrency limits to maximize the use of the cluster's remaining capacity (avoiding resource waste) while preventing fault deterioration (avoiding overload and crash), ultimately achieving a balance between business continuity and system stability.

[0048] Specifically, the entire process of an MPP job instance being pulled by the job execution service and submitted to the underlying MPP cluster in this application includes: all the job's dependencies are ready and placed in the dispatched job queue, waiting to be pulled and executed by the job execution service; the job execution service checks the job instance parameters and obtains the MPP data source information involved in the parameters. If the job involves an MPP data source, it sequentially determines whether the MPP computing cluster corresponding to the MPP data source involved in the job and the metadata cluster to which the computing cluster belongs are in an explicit fault state (a job may involve multiple MPP data sources). If so, it registers the current system time and queuing type and saves them in the local queuing queue of the job execution service for processing by the job re-extraction sub-thread. If not, then sequentially check whether the MPP computing cluster corresponding to the MPP data source involved in the job is in a latent fault state (latent faults only exist in the computing cluster; a latent fault in the computing cluster is defined as: the CPU / memory of a certain node in the MPP computing cluster is in an abnormal state). If so, check the number of jobs running in the global scheduling system involving that data source. If it exceeds a specified threshold, register the current system time and queuing type, and save them in the local queuing queue of the job execution service for the job to re-extract its sub-thread for processing. If it does not exceed the specified threshold, then the job instance is launched normally.

[0049] This application significantly reduces invalid scheduling submissions when MPP is experiencing explicit failures, improving job runtime efficiency and preventing the unauthorized use of other computing resources (since the MPP anomaly can only be detected when the job is running). It also greatly reduces the manual intervention costs for the project team and improves the level of automation. It avoids configuring indiscriminate automatic reruns for all MPP job types to cover MPP anomalies, and prevents interference with other anomaly types that don't require automatic reruns (such as normal SQL logic errors). When MPP is experiencing implicit failures, it prevents excessive job submissions from overwhelming the MPP cluster, thus improving cluster stability.

[0050] In one embodiment, such as Figure 5 As shown, an apparatus for handling MPP cluster anomalies is provided. The apparatus 100 for handling MPP cluster anomalies includes an acquisition module 200, a first processing module 300, a second processing module 400, and a third processing module 500, wherein: The acquisition module 200 is used to acquire the jobs to be executed in the job queue to be dispatched, wherein the data source of the jobs to be executed involves an MPP data source; The first processing module 300 is used to determine whether the task to be executed is an explicit fault or a hidden fault; The second processing module 400 is used to determine whether the queuing time of the job to be executed exceeds a first threshold when the job to be executed is an explicit fault. If it exceeds the first threshold, the job is submitted and removed from the job queue. If it does not exceed the first threshold, the job to be executed continues to be queued. The third processing module 500 is used to determine whether the queuing time of the job to be executed exceeds a second threshold when the job to be executed is a hidden fault. If it exceeds the second threshold, the job to be executed is submitted and removed from the queuing queue. If it does not exceed the second threshold, the module determines the number of job instances associated with the metadata cluster of the job to be executed and in operation. If the number of job instances does not exceed a third threshold, the job to be executed is submitted and removed from the job queue to be dispatched. If it exceeds the third threshold, the job to be executed continues to be queued.

[0051] Specifically, the device also includes a third processing module, which is used to obtain the latest metadata cluster status by calling the MPP resource scheduling service when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs to has an explicit failure and the queuing time of the job to be executed has not exceeded the first threshold. If the metadata cluster has recovered to health, the job to be executed is normally started and removed from the job queue to be dispatched.

[0052] The device also includes a fourth processing module, which is used to call the MPP resource scheduling service to obtain the latest metadata cluster status when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the job to be executed belongs to has a hidden fault and the queuing time of the job to be executed has not exceeded the second threshold. If the metadata cluster has recovered to health, the job is started normally and removed from the queuing queue.

[0053] A method for handling MPP cluster anomalies according to this application includes: obtaining jobs to be executed in a queue of jobs to be dispatched, wherein the data source of the jobs to be executed involves an MPP data source; determining whether the jobs to be executed are explicit or implicit; if the jobs to be executed are explicit, determining whether the queuing time of the jobs to be executed exceeds a first threshold; if it exceeds the first threshold, submitting the jobs and removing them from the queue of jobs to be dispatched; if it does not exceed the first threshold, the jobs to be executed continue to be queued; if the jobs to be executed are implicit, determining whether the queuing time of the jobs to be executed exceeds a second threshold; if it exceeds the second threshold, submitting the jobs to be executed and removing them from the queue of jobs to be dispatched; if it does not exceed the second threshold, determining the number of job instances associated with the metadata cluster of the jobs to be executed and currently running; if the number of job instances does not exceed a third threshold, submitting the jobs to be executed and removing them from the queue of jobs to be dispatched; if it exceeds the third threshold, the jobs to be executed continue to be queued. This method distinguishes between explicit and implicit faults. In the case of explicit MPP faults, it reduces invalid scheduling submissions; in the case of implicit MPP faults, it prevents excessive job submissions from overwhelming the cluster, thus improving cluster stability. Overall, it improves job runtime efficiency, prevents other computing resources from being wasted, reduces manual intervention costs for project teams, and increases automation.

[0054] The MPP cluster anomaly handling device 100 includes a processor and a memory. The aforementioned acquisition module 200, first processing module 300, second processing module 400, and third processing module 500 are all stored in the memory as program units. The processor executes the aforementioned program modules stored in the memory to implement the corresponding functions.

[0055] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and methods for handling MPP cluster anomalies can be implemented by adjusting kernel parameters.

[0056] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0057] This application provides a storage medium storing a program that, when executed by a processor, implements the above-described method for handling MPP cluster anomalies.

[0058] This application provides a processor for running a program, wherein the program executes the above-described method for handling MPP cluster exceptions.

[0059] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown in the figure, the computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements a method for handling MPP cluster exceptions. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0060] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0061] In one embodiment, the MPP cluster anomaly handling apparatus provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 6 The computer program, comprised of various program modules, runs on the computer device shown. It causes the processor to execute the steps of the methods for handling MPP cluster exceptions described in the various embodiments of this application.

[0062] This application provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: obtaining jobs to be executed from a queue of jobs to be dispatched, wherein the data source of the jobs to be executed relates to an MPP data source; determining whether the jobs to be executed have a visible or hidden fault; if the jobs to be executed have a visible fault, determining whether the queuing time of the jobs exceeds a first threshold; if it exceeds the first threshold, submitting the jobs and removing them from the queue of jobs to be dispatched; if it does not exceed the first threshold, the jobs to be executed continue to be queued; if the jobs to be executed have a hidden fault, determining whether the queuing time of the jobs to be executed exceeds a second threshold; if it exceeds the second threshold, submitting the jobs to be executed and removing them from the queue of jobs to be dispatched; if it does not exceed the second threshold, determining the number of job instances associated with the metadata cluster of the jobs to be executed and currently running; if the number of job instances does not exceed a third threshold, submitting the jobs to be executed and removing them from the queue of jobs to be dispatched; if it exceeds the third threshold, the jobs to be executed continue to be queued.

[0063] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: obtaining jobs to be executed in a job queue to be dispatched, wherein the data source of the jobs to be executed involves an MPP data source; determining whether the jobs to be executed are explicit or implicit faults; if the jobs to be executed are explicit faults, determining whether the queuing time of the jobs to be executed exceeds a first threshold; if it exceeds the first threshold, submitting the jobs and removing them from the job queue to be dispatched; if it does not exceed the first threshold, the jobs to be executed continue to be queued; if the jobs to be executed are implicit faults, determining whether the queuing time of the jobs to be executed exceeds a second threshold; if it exceeds the second threshold, submitting the jobs to be executed and removing them from the queue to be dispatched; if it does not exceed the second threshold, determining the number of job instances associated with the metadata cluster of the jobs to be executed and currently running; if the number of job instances does not exceed a third threshold, submitting the jobs to be executed and removing them from the job queue to be dispatched; if it exceeds the third threshold, the jobs to be executed continue to be queued.

[0064] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0069] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0070] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0071] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0072] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for handling exceptions in a MPP cluster, the method comprising: The method comprises: obtaining a to-be-executed job in a to-be-dispatched job queue, wherein a data source of the to-be-executed job relates to an MPP data source; judging whether the to-be-executed job is an explicit failure or an implicit failure; when the to-be-executed job is an explicit failure, judging whether a queuing time of the to-be-executed job exceeds a first threshold value, if yes, submitting the job and removing the job from the to-be-dispatched job queue, if no, the to-be-executed job continues to be queued; when the to-be-executed job is an implicit failure, judging whether a queuing time of the to-be-executed job exceeds a second threshold value, if yes, submitting the to-be-executed job and removing the job from the queuing queue, if no, judging a number of job instances associated with a metadata cluster of an MPP computing cluster corresponding to the to-be-executed job and running, if the number of job instances does not exceed a third threshold value, submitting the to-be-executed job and removing the job from the to-be-dispatched job queue, if the number of job instances exceeds the third threshold value, the to-be-executed job continues to be queued.

2. The method of claim 1, wherein, The method further comprises: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the to-be-executed job belongs is an explicit failure and the queuing time of the to-be-executed job does not exceed the first threshold value, obtaining a latest metadata cluster state by calling an MPP resource scheduling service, if the metadata cluster has recovered to be healthy, normally starting the to-be-executed job and removing the job from the to-be-dispatched job queue.

3. The method of claim 1, wherein, The method further comprises: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the to-be-executed job belongs is an implicit failure and the queuing time of the to-be-executed job does not exceed the second threshold value, calling an MPP resource scheduling service to obtain a latest metadata cluster state, if the metadata cluster has recovered to be healthy, normally starting the job and removing the job from the queuing queue.

4. The method of claim 1, wherein, The method further comprises: when the metadata cluster to which the MPP computing cluster corresponding to the MPP data source of the to-be-executed job belongs is an explicit failure and the queuing time of the to-be-executed job does not exceed the first threshold value, registering a current system time and a queuing type for saving in the to-be-dispatched job queue.

5. The method according to any one of claims 1-4, wherein the explicit failure is an abnormality of an MPP computing cluster and / or a metadata cluster related to the job; and the implicit failure is a CPU and / or memory abnormality of a node of the MPP computing cluster related to the job. The method further comprises: collecting states of the MPP cluster and the metadata cluster of the to-be-executed job at a regular time and caching the states to a redis database, for regularly judging failures.

6. The method of claim 1, wherein, The apparatus comprises: an obtaining module, configured to obtain a to-be-executed job in a to-be-dispatched job queue, wherein a data source of the to-be-executed job relates to an MPP data source; 7. An apparatus for handling exceptions in a MPP cluster, the apparatus comprising: a first processing module, configured to judge whether the to-be-executed job is an explicit failure or an implicit failure; a second processing module, configured to, when the to-be-executed job is an explicit failure, judge whether a queuing time of the to-be-executed job exceeds a first threshold value, if yes, submit the job and remove the job from the to-be-dispatched job queue, if no, the to-be-executed job continues to be queued; ​ ​ The third processing module is configured to, when the to-be-executed job is a hidden fault, determine whether the queuing time of the to-be-executed job exceeds a second threshold, and if the queuing time exceeds the second threshold, submit the to-be-executed job and remove the to-be-executed job from the queuing queue; if the queuing time does not exceed the second threshold, determine the number of job instances associated with the metadata cluster and running, and if the number of job instances does not exceed a third threshold, submit the to-be-executed job and remove the to-be-executed job from the to-be-dispatched job queue; if the number of job instances exceeds the third threshold, the to-be-executed job continues to be queued.

8. The apparatus of claim 7, wherein, The apparatus further includes: The third processing module is configured to, when the to-be-executed job is a hidden fault, determine whether the queuing time of the to-be-executed job exceeds a second threshold, and if the queuing time exceeds the second threshold, submit the to-be-executed job and remove the to-be-executed job from the queuing queue; if the queuing time does not exceed the second threshold, determine the number of job instances associated with the metadata cluster and running, and if the number of job instances does not exceed a third threshold, submit the to-be-executed job and remove the to-be-executed job from the to-be-dispatched job queue; if the number of job instances exceeds the third threshold, the to-be-executed job continues to be queued.

9. The apparatus of claim 7, wherein, The apparatus further includes: The fourth processing module is configured to, when the to-be-executed job is a hidden fault, determine whether the queuing time of the to-be-executed job exceeds a second threshold, and if the queuing time exceeds the second threshold, submit the to-be-executed job and remove the to-be-executed job from the queuing queue; if the queuing time does not exceed the second threshold, determine the number of job instances associated with the metadata cluster and running, and if the number of job instances does not exceed a third threshold, submit the to-be-executed job and remove the to-be-executed job from the to-be-dispatched job queue; if the number of job instances exceeds the third threshold, the to-be-executed job continues to be queued.

10. A processor, comprising: The computer program is configured to, when executed by a processor, implement the method of MPP cluster exception processing according to any one of claims 1 to 6.

11. A machine-readable storage medium having instructions stored thereon, the instructions comprising: The computer program is configured to, when executed by a processor, implement the method of MPP cluster exception processing according to any one of claims 1 to 6.

12. A computer program product comprising a computer program, characterized in that, The computer program is configured to, when executed by a processor, implement the method of MPP cluster exception processing according to any one of claims 1 to 6.

Images