Distributed cluster-based job intervention method, apparatus and device, medium and product

Abstract

The invention discloses an operation intervention method and device based on a distributed cluster, electronic equipment, a storage medium and a computer program product, relates to the technical field of big data processing, is applied to a monitoring unit of the distributed cluster, and comprises the following steps: for any real-time operation in an operation list of the distributed cluster, obtaining an operation index of the real-time operation, determining monitoring analysis data of the real-time operation based on the operation indexes; and under the condition that the monitoring analysis data hits the corresponding monitoring rule in the local database, judging that the real-time job is an abnormal job, and executing an intervention operation on the abnormal job so as to enable the abnormal job to run normally. The technical problem of poor stability of cluster operation is solved.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a distributed cluster-based job intervention method, device, electronic device, storage medium and computer program product. Background Art

[0002] In the era of big data, real-time data processing has become a key link in enterprise data analysis and business decision-making. With the explosive growth of data volume and the continuous improvement of business requirements for real-time performance, the existing real-time data processing mechanism faces severe challenges.

[0003] At present, most real-time jobs in real-time data processing frameworks usually run on distributed clusters (such as Hadoop clusters) and use cluster resource manager systems for resource scheduling and management. However, the current real-time data processing mechanism still has limitations. When the system is abnormal, it cannot automatically intervene. If the cluster scale grows, manual operations can no longer meet the operation and maintenance needs. Since the system cannot provide fast enough intervention capabilities, abnormal problems cannot be solved in time, affecting the stability of cluster operations.

[0004] Therefore, how to ensure the stable operation of cluster operations is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a distributed cluster-based job intervention method, device, electronic device, storage medium and computer program product, aiming to solve the technical problem of poor stability of cluster jobs.

[0006] To achieve the above objectives, the present application proposes a distributed cluster-based job intervention method, which is applied to a monitoring unit of a distributed cluster. The distributed cluster-based job intervention method includes:

[0007] For any real-time job in the job list of the distributed cluster, obtain a job index of the real-time job, and determine monitoring and analysis data of the real-time job based on the job index;

[0008] When the monitoring analysis data hits the corresponding monitoring rule in the local database, the real-time operation is determined to be an abnormal operation, and an intervention operation is performed on the abnormal operation to enable the abnormal operation to run normally.

[0009] In one embodiment, the step of determining the monitoring and analysis data of the real-time operation based on the operation indicator includes:

[0010] Aggregating the operation indicators according to preset dimensions to filter out invalid operation indicators;

[0011] Based on the job indicators after data aggregation, the monitoring and analysis data is calculated according to the preset calculation logic, and the monitoring and analysis data includes at least one of job processing delay, transactions processed per second, average memory usage, and maximum memory usage.

[0012] In one embodiment, when the monitoring analysis data hits the corresponding monitoring rule in the local database, the step of determining that the real-time job is an abnormal job includes:

[0013] When the job processing delay of the real-time job exceeds a first preset abnormal threshold, and / or the number of transactions processed per second is lower than a second preset abnormal threshold, and / or the average memory usage exceeds a third preset abnormal threshold, and / or the maximum memory usage exceeds a fourth preset abnormal threshold, it is determined that the monitoring and analysis data hits a corresponding monitoring rule in a local database, and it is determined that the real-time job belongs to an abnormal job of an abnormal job type;

[0014] When there is a target job identical to the real-time job in the job list of the distributed cluster, it is determined that the monitoring analysis data hits the corresponding monitoring rule in the local database, and it is determined that the real-time job is an abnormal job of a repeated job type.

[0015] In one embodiment, the step of performing an intervention operation on the abnormal operation includes:

[0016] In the case where the type of the abnormal operation is the abnormal operation, restarting the abnormal operation and recording the number of restarts, and when the number of restarts reaches a preset restart threshold, notifying the user to process the abnormal operation;

[0017] In the case that the type of the abnormal job includes the repeated job, the abnormal job is terminated and the number of terminations is recorded. When the number of terminations reaches a preset termination threshold, the user is notified to process the abnormal job.

[0018] In one embodiment, the distributed cluster-based job intervention method further includes:

[0019] When the monitoring and analysis data hits the corresponding alarm rule in the local database, an alarm event is determined, and the context information of the alarm event is encapsulated into an alarm message, wherein the alarm message includes the job name, the alarm event type, the alarm event details, the alarm event time, and handling suggestions;

[0020] The warning message is output.

[0021] In one embodiment, the distributed cluster includes an alarm unit, and the step of outputting the alarm message includes:

[0022] The alarm message is sent to the target topic of the preset message subscription system, so that the alarm unit obtains the alarm message by subscribing to the target topic, parses the alarm message and sends the parsed result to the user terminal.

[0023] In addition, to achieve the above objectives, the present application also proposes a distributed cluster-based job intervention device, the distributed cluster-based job intervention device comprising:

[0024] A data generation module, configured to obtain a job index of any real-time job in a job list of the distributed cluster, and determine monitoring and analysis data of the real-time job based on the job index;

[0025] The intervention module is used to determine that the real-time operation is an abnormal operation when the monitoring and analysis data hits the corresponding monitoring rules in the local database, and perform intervention operations on the abnormal operation to make the abnormal operation run normally.

[0026] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the distributed cluster-based job intervention method as described above.

[0027] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the distributed cluster-based job intervention method as described above are implemented.

[0028] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the distributed cluster-based job intervention method as described above.

[0029] The present application is applied to a distributed cluster and provides a job intervention method based on a distributed cluster. The job intervention method based on a distributed cluster includes: for any real-time job in the job list of the distributed cluster, obtaining the job index of the real-time job, and determining the monitoring and analysis data of the real-time job based on the job index; when the monitoring and analysis data hits the corresponding monitoring rule in the local database, the real-time job is determined to be an abnormal job, and an intervention operation is performed on the abnormal job to enable the abnormal job to operate normally.

[0030] This application monitors the job indicators of real-time jobs and generates monitoring and analysis data based on these job indicators. The monitoring and analysis data is an important basis for evaluating whether the job is running healthily and whether it meets the performance requirements. By analyzing the monitoring and analysis data, the system can detect job anomalies in a timely manner. When the monitoring and analysis data hits the monitoring rules stored in the local database, the system can detect the anomalies in a timely manner and trigger corresponding intervention operations, thereby solving job anomalies, reducing the need for manual intervention, improving the efficiency and accuracy of fault handling, and enhancing the stability of real-time jobs in distributed clusters, thereby improving the reliability and availability of the entire system. Compared with related solutions that rely on manual monitoring and manual intervention and lack automatic intervention capabilities, this application obtains monitoring and analysis data for evaluating job anomalies. Based on preset monitoring rules, when a system anomaly is detected, timely intervention operations are performed to reduce job anomalies caused by human delays or negligence, and significantly improve the stability of real-time jobs in distributed clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A flowchart of a distributed cluster-based job intervention method embodiment 1 of the present application is provided;

[0034] Figure 2 A flow chart of the second embodiment of the distributed cluster-based job intervention method of the present application;

[0035] Figure 3 This is a system architecture diagram of the distributed cluster-based job intervention method for this application;

[0036] Figure 4 This is a schematic diagram of the module structure of a distributed cluster-based job intervention device according to an embodiment of the present application;

[0037] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the distributed cluster-based job intervention method in an embodiment of the present application.

[0038] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0040] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0041] The embodiment of the present application is applied to a distributed cluster, and mainly includes: for any real-time job in the job list of the distributed cluster, obtaining the job index of the real-time job, and determining the monitoring and analysis data of the real-time job based on the job index; when the monitoring and analysis data hits the corresponding monitoring rules in the local database, the real-time job is determined to be an abnormal job, and an intervention operation is performed on the abnormal job to enable the abnormal job to operate normally.

[0042] In this embodiment, for the convenience of description, the following description is made with the operation intervention system as the execution subject.

[0043] In the existing technology, when the system encounters an abnormality, operation and maintenance usually relies on manual intervention, which not only increases the workload of operation and maintenance, but may also lead to untimely intervention, thus affecting the stability of real-time operations. As the cluster expands and the workload surges, the method of relying on manual operations for operation and maintenance can no longer meet the operation and maintenance needs, resulting in the failure to ensure the stable operation of real-time operations.

[0044] The present application provides a solution, which monitors the operation indicators in real time, analyzes and generates monitoring and analysis data as the key basis for the health and performance of the operation, and automatically monitors anomalies and performance bottlenecks by parsing the monitoring and analysis data. Once the monitoring and analysis data triggers the preset monitoring rules, the intervention operation is automatically triggered to quickly respond to abnormal problems in the operation, reduce manual intervention, improve the efficiency and accuracy of fault handling, and strengthen the stability of real-time operations in distributed clusters, thereby enhancing the reliability and availability of the entire system and optimizing the operation and maintenance process.

[0045] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a work intervention system, etc. The following takes the work intervention system as an example to illustrate this embodiment and the following embodiments.

[0046] Based on this, the embodiment of the present application provides a distributed cluster-based job intervention method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the distributed cluster-based job intervention method of the present application.

[0047] In this embodiment, the distributed cluster-based job intervention method is applied to the monitoring unit of the distributed cluster. The distributed cluster-based job intervention method includes steps S01 to S02:

[0048] Step S01, for any real-time job in the job list of the distributed cluster, obtain the job index of the real-time job, and determine the monitoring and analysis data of the real-time job based on the job index;

[0049] It should be noted that a distributed cluster refers to a computing system composed of multiple computers connected through a network, which can collaboratively complete large-scale computing tasks, such as Hadoop clusters, Spark clusters, Flink clusters, etc. The monitoring unit is a submodule in the distributed cluster, responsible for monitoring the job status in the distributed cluster. The job list refers to the list of tasks currently being executed or to be executed in the distributed cluster. Real-time jobs refer to the tasks currently being executed in the job list, and their status and performance can be monitored and analyzed in real time. Job indicators are quantitative indicators for measuring job performance, resource usage, etc., such as job status, startup time, response time, etc. Monitoring and analysis data refers to higher-level data based on job indicator analysis that reflects the health and performance of the job, such as job processing delay, transactions processed per second, average memory usage, maximum memory usage, etc.

[0050] In addition, it should be noted that for any real-time job in the job list of a distributed cluster, the job indicators of the real-time job are collected through cluster management tools, APIs provided by the job execution environment (such as YARN, Kubernetes), etc. After obtaining the job indicators of the real-time job, the job indicators are analyzed in real time, and the monitoring and analysis data are calculated and output to identify potential abnormal problems in the cluster.

[0051] It is understandable that since the current real-time data processing mechanism can only view the status, logs, and task execution records of the currently running job, and cannot obtain monitoring and analysis data such as latency and average memory usage, step S01 can be performed to obtain job indicators in more dimensions, including but not limited to execution time, throughput, error rate, etc., to provide a more comprehensive job status view and output the monitoring and analysis data. Users do not need to visit the Yarn management page to view job running indicators, thereby improving access convenience.

[0052] Step S02, when the monitoring analysis data hits the corresponding monitoring rule in the local database, the real-time operation is determined to be an abnormal operation, and an intervention operation is performed on the abnormal operation to make the abnormal operation run normally.

[0053] It should be noted that the local database is the background system that stores information related to cluster jobs. The monitoring rules define how to evaluate the health status of jobs based on job indicators, including a series of thresholds, ratios, time windows and other conditions, which are used to determine whether the job is within the normal operating range. Abnormal jobs refer to jobs with abnormal performance, resource exhaustion or inability to complete tasks as expected. Intervention operations refer to measures taken to restore the normal operation of abnormal jobs, including restart operations and termination operations.

[0054] In addition, it should be noted that the monitoring and analysis data obtained in step S01 are compared with the monitoring rules predefined in the local database. When certain indicators or combinations in the monitoring and analysis data meet the abnormal conditions defined in the monitoring rules, the system determines that the real-time job is an abnormal job. The system performs intervention operations on the abnormal job according to the preset intervention strategy, such as restarting the job, reallocating resources, adjusting parameters, etc., to restore its normal operating state.

[0055] It is understandable that since the current real-time data processing mechanism lacks automatic intervention capabilities, it cannot diagnose abnormal indicators and provide sufficiently fast intervention capabilities. When the cluster scale grows and the number of jobs exceeds 10,000, manual operations can no longer meet the operation and maintenance needs, resulting in the lack of guarantee for the stable operation of real-time jobs. Therefore, step S02 is performed to achieve automatic detection and abnormality judgment of the job status through real-time comparison with monitoring rules, thereby improving accuracy and efficiency. After detecting an abnormality, the system can automatically perform intervention operations, reducing the delay and uncertainty of manual intervention and ensuring the stability and availability of the cluster.

[0056] In a feasible implementation, in step S01, the step of determining and outputting real-time operation monitoring and analysis data based on the operation index includes steps A01 to A02:

[0057] Step A01, aggregate the operation indicators according to the preset dimensions;

[0058] It should be noted that preset dimensions refer to data classification standards set according to business needs and data characteristics before analysis, such as time dimensions (by hours, days, etc.), job type dimensions (such as batch jobs, real-time processing jobs), node dimensions (different servers or computing nodes), etc. Data aggregation refers to the process of organizing scattered data points into more regular and easier to analyze data sets based on preset dimensions.

[0059] In addition, it should be noted that relevant job indicators are collected from various data sources of the distributed cluster (such as log files, performance monitoring tools, databases, etc.), such as the status of the currently running job, logs, task execution records, etc., and the data in the same time period are merged according to the time dimension, and the average value, total and other statistical information of the time period are calculated, or the job indicators of the same type of services are aggregated according to the service type dimension for horizontal comparison.

[0060] Step A02, based on the job indicators after data aggregation, the monitoring and analysis data is calculated according to the preset calculation logic, and the monitoring and analysis data includes at least one of the job processing delay, transactions processed per second, average memory usage, and maximum memory usage.

[0061] It should be noted that the preset calculation logic refers to the calculation method and rules set according to business needs and data characteristics before analysis, which is used to extract valuable monitoring and analysis data from the aggregated data. The job processing delay refers to the total time from the time the job is submitted to the cluster for processing to the time the job is completed and the results are returned. Transactions per second (TPS) refers to the number of transactions that the cluster can successfully process per second. The average memory usage refers to the proportion of the average amount of memory used by the cluster over a period of time to the total amount of memory. The maximum memory usage refers to the highest memory usage reached by the cluster during the monitoring period.

[0062] In addition, it should be noted that the system further calculates higher-level monitoring and analysis data based on the job indicators after data aggregation. These monitoring and analysis data include at least one or more of job processing delay, transactions per second (TPS), average memory usage, and maximum memory usage. In addition, more high-level performance indicators can be calculated by configuring preset calculation logic according to user needs. Each indicator reflects different aspects of job performance, such as response speed, processing power, resource utilization efficiency, etc. After calculating the monitoring and analysis data, the system integrates it into a unified format through data formatting, unit conversion (such as converting timestamps into readable date and time formats), data sorting, etc., so that it can be displayed on the reporting platform. The preset reporting platform is a system or tool used to display, analyze and report monitoring and analysis data. It is used to receive and display job monitoring and analysis data to help administrators or analysts quickly understand the performance status and potential problems of the cluster, such as FineReport, Smartbi, Tableau, etc. The system establishes a connection with the preset reporting platform and transmits the monitoring and analysis data to the preset reporting platform. On the reporting platform, the system displays the monitoring and analysis data in the form of charts, tables, etc., which intuitively reflects the performance status of the job, which is convenient for users to quickly understand and analyze. At the same time, the preset reporting platform not only displays the current monitoring and analysis data, but also records and saves the historical versions of these data. Users can view data changes in any time period through the preset reporting platform, thereby analyzing performance trends, identifying performance problems and formulating corresponding optimization strategies.

[0063] Exemplarily, taking transactions processed per second (TPS) as an example, for Spark jobs and Flink jobs running on a distributed cluster (Hadoop cluster), in the Spark job, collect the job data of the 10 most recently completed batches, including the records (amount of data processed) and processTime (the time required to process these data) of each batch, take the sum of the records as the total data processing amount during this period, take the sum of the processTime as the total processing time, and divide the two to get the TPS of the Spark job; in the Flink job, for each task, use KafkaConsumer.records-consumed-rate to get the data processing rate of the current task, for each time point, add the data processing rates of all tasks to get the total data processing rate at that time point, and aggregate the total data processing rates of all time points over time (such as taking the average or summing them, depending on the requirements) to get the TPS of the Flink job.

[0064] In this embodiment, data aggregation significantly reduces the amount of data required for subsequent analysis and processing, speeds up data processing, and improves the responsiveness of the monitoring system. By calculating monitoring and analysis data, the system can provide richer dimensional data, thereby more accurately evaluating the health status of the job.

[0065] In a feasible implementation, in step S02, when the monitoring analysis data hits the corresponding monitoring rule in the local database, the step of determining that the real-time operation is an abnormal operation includes steps A11 to A12:

[0066] Step A11, when the job processing delay of the real-time job exceeds the first preset abnormal threshold, and / or the number of transactions processed per second is lower than the second preset abnormal threshold, and / or the average memory usage exceeds the third preset abnormal threshold, and / or the maximum memory usage exceeds the fourth preset abnormal threshold, it is determined that the monitoring analysis data hits the corresponding monitoring rule in the local database, and the real-time job is determined to be an abnormal job of the abnormal job type;

[0067] It should be noted that the abnormal job type refers to the job type that exhibits unexpected or potential problems during the execution process. The first preset abnormal threshold is a specific value or time limit, which is used to determine whether the processing delay of the job exceeds an acceptable range, such as the upper limit of the processing delay. The second preset abnormal threshold is a value related to the job processing capacity, which is used to determine whether the number of transactions processed per second by the job is lower than expected, such as the lower limit of the number of transactions processed per second. The third preset abnormal threshold is a value related to memory usage, which is used to determine whether the average memory usage of the job is too high, such as the upper limit of the average memory usage. The fourth preset abnormal threshold is a value related to the peak memory usage, which is used to determine whether the maximum memory usage of the job exceeds the system or container limit, such as the upper limit of the maximum memory usage. The performance indicators targeted by the monitoring rules can be adjusted and set according to user needs. Corresponding to the monitoring and analysis data, the actual rules may contain more content than the first to fourth preset abnormal thresholds mentioned above.

[0068] In addition, it should be noted that the system checks multiple performance indicators of real-time jobs, including job processing delay, number of transactions processed per second, average memory usage and maximum memory usage, etc., and compares these performance indicators with preset abnormality thresholds (first preset abnormality threshold to fourth preset abnormality threshold). If any performance indicator of the real-time job exceeds the corresponding abnormality threshold in the monitoring rule (or is lower than the number of transactions processed per second), it is determined that the monitoring and analysis data hits the monitoring rule corresponding to the real-time job in the local database, and the real-time job is classified as an abnormal job type.

[0069] Among them, the monitoring rules can be personalized according to the abnormal demand keywords input by the user. When setting or modifying the monitoring rules, based on the abnormal demand keywords input by the user, the local abnormal threshold table is queried to intelligently recommend abnormal thresholds associated with the abnormal demand keywords. If the user refuses to select the intelligently recommended abnormal threshold, the abnormal threshold table is output to determine the target abnormal threshold that the user searches for based on the abnormal category or abnormal keyword, and the target abnormal threshold is updated to the monitoring rules. In addition, the system intelligently recommends a derived abnormal threshold associated with the target abnormal threshold, and updates the derived abnormal threshold selected by the user to the monitoring rules.

[0070] Exemplarily, in a distributed cluster, the scheduleAtFixedRate method of ScheduledExecutorService is used to set the inspection interval (for example, 5 minutes). When each inspection thread is started, the monitoring configuration of the current job is read from the local database (MySQL), including various abnormal thresholds (first preset abnormal threshold, second preset abnormal threshold, third preset abnormal threshold, fourth preset abnormal threshold), and the latest monitoring indicators of the current job are queried, such as processing delay, TPS, average memory usage, and maximum memory usage. According to the queried monitoring indicators, they are compared with the preset abnormal thresholds. If any monitoring and analysis data of the real-time job does not meet the corresponding abnormal threshold in the monitoring rule, the real-time job is recorded as an abnormal job, and it is determined that the monitoring and analysis data hits the corresponding monitoring rule in the local database.

[0071] Step A12, when there is a target job identical to the real-time job in the job list of the distributed cluster, it is determined that the monitoring analysis data hits the corresponding monitoring rule in the local database, and the real-time job is determined to be an abnormal job of the duplicate job type.

[0072] It should be noted that the system traverses the job list of the distributed cluster to find the same job as the real-time job. Based on the pre-set matching logic (such as job name, parameters, execution target, etc.), it determines whether the real-time job has the same target job in the job list. The target job is a job with the same execution target, parameters or tasks as the real-time job. If the target job is found, the real-time job is classified as a duplicate job type. The duplicate job type means that in the distributed cluster, there are two or more job types with the same or highly similar execution targets, parameters or tasks. The identification of duplicate jobs also means that the monitoring and analysis data hits the corresponding monitoring rules of the real-time job in the local database.

[0073] Exemplarily, in a distributed cluster, there is a real-time job named "data cleaning". The scheduleAtFixedRate method of ScheduledExecutorService is used to set the inspection interval (for example, 5 minutes), and the properties of all jobs in the job list of the distributed cluster are queried from the job configuration table (job_config) to determine whether there is a job that is repeated with "data cleaning" (that is, the target job). If the target job is found, the "data cleaning" real-time job is recorded as a duplicate job, and it is determined whether the monitoring and analysis data hits the corresponding monitoring rules in the local database.

[0074] In this embodiment, real-time status monitoring of cluster operations is achieved through automated monitoring tools, which improves the automation level of system monitoring, reduces the burden on operation and maintenance personnel, and reduces the possibility of human errors. Based on preset thresholds and rules, abnormal operations and repeated operations are identified in real time and accurately, which helps to take remedial measures in time to prevent the problem from expanding. Once an abnormal or repeated operation is found, the corresponding processing mechanism is immediately triggered to reduce the scope of the fault impact, effectively avoid waste of resources and potential conflicts, and improve the execution efficiency of cluster operations.

[0075] In a feasible implementation manner, in step S02, the step of performing intervention operation on abnormal operation includes steps A21 to A22:

[0076] Step A21, when the type of the abnormal operation is an abnormal operation, restart the abnormal operation and record the number of restarts. When the number of restarts reaches a preset restart threshold, notify the user to handle the abnormal operation;

[0077] It should be noted that the number of restarts refers to the number of times an abnormal job is automatically restarted by the system. Each restart will increase the value of the counter. The preset restart threshold is a value set by the system to determine whether the abnormal job cannot return to normal after multiple restarts. When the number of restarts reaches or exceeds this threshold, the system will trigger a notification mechanism.

[0078] In addition, it should be noted that in order to try to restore the normal execution of the abnormal job, the system will restart the abnormal job of the abnormal job type. The system maintains a restart counter for the abnormal job of the abnormal job type. Each time the abnormal job of the abnormal job type is restarted, the counter will increase to record the number of times the abnormal job of the abnormal job type is restarted. At the same time, the system also sets a preset restart threshold to determine whether the abnormal job of the abnormal job type cannot be restored to normal after multiple restarts. When the number of restarts reaches or exceeds this threshold, the system determines that the abnormal job has a more serious problem and requires manual intervention. Once the number of restarts reaches the preset restart threshold, the system notifies the user or the operation and maintenance team through a preset notification mechanism (such as email, text message, system log, etc.) so that they can conduct further investigation and processing.

[0079] Step A22, when the type of the abnormal job includes a repeated job, terminate the abnormal job and record the number of terminations. When the number of terminations reaches a preset termination threshold, notify the user to process the abnormal job.

[0080] It should be noted that the number of terminations refers to the number of times a repetitive job is automatically terminated by the system. Each termination will increase the value of the counter. The preset termination threshold is a value set by the system to determine whether repetitive jobs occur frequently. When the number of terminations reaches or exceeds the preset termination threshold, the system will trigger a notification mechanism.

[0081] In addition, it should be noted that in order to avoid waste of resources and potential conflicts, the system will terminate (kill operation) the job in the cluster that is the same as the current real-time job (i.e., the target job). The system maintains a counter of the number of terminations for each repeated job. Each time a repeated job is terminated, the counter will be incremented to record the number of times the repeated job has been terminated. At the same time, the system also sets a preset termination threshold to determine whether the repeated job is frequently terminated. When the number of terminations reaches or exceeds the preset termination threshold, the system determines that there are deeper problems such as configuration errors, scheduling logic problems, or human errors. Once the number of terminations reaches the preset termination threshold, the system notifies the user or operation and maintenance team through the preset notification mechanism so that they can conduct further investigation and processing.

[0082] In this implementation, by presetting the restart threshold and notification mechanism as well as the presetting the termination threshold and notification mechanism, it is ensured that abnormal jobs can be promptly handled manually when they cannot be automatically recovered, thereby improving the automatic recovery capability of abnormal jobs and reducing service interruption time. At the same time, it also improves the utilization rate of cluster resources and the uniqueness of job execution, and enhances the stability and reliability of cluster jobs.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 The distributed cluster-based job intervention method further includes steps S11 to S12:

[0084] Step S11, when the monitoring and analysis data hits the corresponding alarm rule in the local database, an alarm event is determined, and the context information of the alarm event is encapsulated into an alarm message, which includes the job name, alarm event type, alarm event details, alarm event time, and handling suggestions;

[0085] It should be noted that the alarm rule defines what kind of monitoring and analysis data will trigger an alarm event, such as high CPU usage, response time timeout, insufficient disk space, etc. Alarm events refer to any abnormal situation in the cluster that requires attention and may affect the stable operation of the job. Context information refers to a collection of data, logs, configuration and other information closely related to the alarm event, which together constitute the overall picture of the alarm event. Alarm message refers to the notification information automatically generated and sent to subscribers when the system monitors an abnormality or potential problem, containing detailed information about the alarm event. Job name refers to the name of a single task running in the cluster, which is used to identify and distinguish different jobs. Alarm event type refers to the specific category of the abnormality or potential problem identified in the alarm message. Alarm event details refer to a detailed description of the alarm event, including the specific conditions and indicator values ​​that trigger the alarm. Alarm event time refers to the time point when the alarm event occurs, usually expressed in the form of a timestamp. Processing suggestions refer to preliminary processing suggestions provided based on the alarm event type to help the recipient quickly take solutions.

[0086] In addition, it should be noted that once it is determined that the monitoring and analysis data hits the alarm rule, the system will determine that an alarm event has occurred. This alarm event represents a specific problem in the distributed cluster that requires the user's attention and the need to take action. Contextual information related to the alarm event is collected, including but not limited to the specific data points that trigger the alarm, related system logs, job configuration information, etc., which together constitute the overall picture of the alarm event. The context information of the alarm event is encapsulated into a structured alarm message. This alarm message contains multiple key fields, such as the job name (identifies the job where the alarm occurs), the alarm event type (indicates the specific nature of the alarm), the alarm event details (provides a detailed description of the alarm event), the alarm event time (records the specific time point when the alarm occurs), and processing suggestions (based on historical experience and preset rules, providing users with preliminary solutions or inspection directions).

[0087] Among them, the alarm rules can be personalized according to the alarm demand keywords input by the user. When setting or modifying the alarm rules, based on the alarm demand keywords input by the user, the local alarm threshold table is queried to intelligently recommend alarm thresholds related to the alarm demand keywords. If the user refuses to select the intelligently recommended alarm threshold, the alarm threshold table will be output to determine the target abnormality threshold retrieved by the user based on the alarm category or alarm keyword, and the target alarm threshold will be updated to the alarm rule. In addition, the system intelligently recommends a derivative alarm threshold related to the target alarm threshold, and updates the derived alarm threshold selected by the user to the alarm rule.

[0088] Step S12: outputting an alarm message.

[0089] It should be noted that after the alarm message is encapsulated, the output method is selected according to the operation and maintenance strategy or user needs, the corresponding output operation is performed, all issued alarm messages are recorded and archived, and the progress of the alarm event processing is continuously monitored. If feedback is received from the operation and maintenance personnel or the problem is resolved, the alarm status is updated in a timely manner and the relevant personnel are notified.

[0090] In this implementation, through real-time monitoring and analysis, the system can quickly discover abnormal conditions in the cluster and identify alarm events, thereby ensuring the timeliness and accuracy of the alarm. The context information of the alarm event is encapsulated into a structured alarm message, which facilitates the subsequent parsing and processing of this information, improves the efficiency of alarm information processing, and ensures that the alarm message is effectively conveyed to relevant personnel by outputting the alarm message, providing strong support for operation and maintenance work.

[0091] In a feasible implementation manner, the distributed cluster includes an alarm unit. In step S12, the step of outputting an alarm message includes step E01:

[0092] Step E01, sending an alarm message to a target topic of a preset message subscription system, so that the alarm unit obtains the alarm message by subscribing to the target topic, parses the alarm message and sends the parsed result to the user terminal.

[0093] It should be noted that the preset message subscription system (Kafka) is a middleware for message publishing and subscription, which allows different components within the system to receive and send messages by subscribing to specific topics (target topics). The alarm unit is an important component in the distributed cluster, responsible for receiving, parsing and processing alarm messages, and notifying users of alarm information in a timely manner. The alarm unit needs to subscribe to the target topic in the preset message subscription system in advance so that it can receive the alarm message when the alarm message is published. When the alarm message is sent to the target topic, the alarm module subsystem subscribed to the topic will immediately receive and parse the alarm message, extract the content, including the job name, alarm event type, alarm event details, alarm event time and processing suggestions, etc., and notify the user of the parsed alarm message content in an appropriate manner (such as email, SMS, instant message, etc.), so that the user can be aware of the alarm events occurring in the cluster in a timely manner and take corresponding processing measures according to the processing suggestions provided.

[0094] Exemplarily, the alarm module subsystem subscribes to the specified topic through the Kafka client so that it can receive the alarm message when it arrives, and parses the alarm message to extract the content, including the job name, alarm event type, alarm event details, alarm event time, and processing suggestions. The alarm module subsystem notifies the user of the alarm information through a preset notification method (such as SMS, phone, email, etc.) based on the parsed content, so that the user can promptly understand the alarm events occurring in the cluster and take corresponding processing measures based on the processing suggestions provided.

[0095] In this implementation, an effective notification mechanism is used to ensure that the alarm message can be conveyed to the user in a timely and accurate manner, so that the user can quickly understand the abnormal situation in the cluster and take corresponding processing measures, thereby ensuring the stable operation of the cluster operation.

[0096] For example, to help understand the implementation process of the distributed cluster-based job intervention method, please refer to Figure 3 , Figure 3 A system architecture diagram of a distributed cluster-based job intervention method is provided, specifically:

[0097] The Spark framework and Flink framework are deployed on the distributed cluster (Hadoop cluster), including Spark jobs and Flink jobs. The monitoring service obtains the monitoring indicators (i.e., monitoring and analysis data) of the Spark jobs and Flink jobs from the cluster, stores them in the local MySQL database, and performs intervention operations and alarm operations on abnormal problems in the cluster through the intervention rules and alarm rules stored in the local MySQL database. At the same time, the alarm messages generated in the alarm operation are received and parsed by the downstream alarm system, thereby notifying the user to solve the abnormal problems.

[0098] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the distributed cluster-based job intervention method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0099] This application also provides a distributed cluster-based operation intervention device, please refer to Figure 4 , a monitoring unit applied to a distributed cluster, and a distributed cluster-based operation intervention device includes:

[0100] The data generation module 10 is used to obtain the job index of any real-time job in the job list of the distributed cluster, and determine the monitoring and analysis data of the real-time job based on the job index;

[0101] The intervention module 20 is used to determine that the real-time operation is an abnormal operation when the monitoring and analysis data hits the corresponding monitoring rule in the local database, and perform intervention operations on the abnormal operation to make the abnormal operation run normally.

[0102] Optionally, the data generating module 10 is further used for:

[0103] Aggregate the data of the operation indicators according to the preset dimensions;

[0104] Based on the job indicators after data aggregation, the monitoring and analysis data is calculated according to the preset calculation logic, and the monitoring and analysis data includes at least one of the job processing delay, transactions processed per second, average memory usage, and maximum memory usage.

[0105] Optionally, the intervention module 20 is further configured to:

[0106] When the job processing delay of the real-time job exceeds the first preset abnormal threshold, and / or the number of transactions processed per second is lower than the second preset abnormal threshold, and / or the average memory usage exceeds the third preset abnormal threshold, and / or the maximum memory usage exceeds the fourth preset abnormal threshold, it is determined that the monitoring analysis data hits the corresponding monitoring rule in the local database, and the real-time job is determined to be an abnormal job of the abnormal job type;

[0107] When there is a target job identical to the real-time job in the job list of the distributed cluster, it is determined that the monitoring analysis data hits the corresponding monitoring rule in the local database, and the real-time job is determined to be an abnormal job of a duplicate job type.

[0108] Optionally, the intervention module 20 is further configured to:

[0109] If the type of the abnormal operation is an abnormal operation, the abnormal operation is restarted and the number of restarts is recorded. When the number of restarts reaches a preset restart threshold, the user is notified to handle the abnormal operation;

[0110] In the case that the type of the abnormal job includes a repeated job, the abnormal job is terminated, and the number of terminations is recorded. When the number of terminations reaches a preset termination threshold, the user is notified to process the abnormal job.

[0111] Optionally, the distributed cluster-based operation intervention device further includes an alarm module 30, and the alarm module 30 is further used to:

[0112] When the monitoring and analysis data hits the corresponding alarm rule in the local database, the alarm event is determined and the context information of the alarm event is encapsulated into an alarm message. The alarm message includes the job name, alarm event type, alarm event details, alarm event time, and handling suggestions;

[0113] Outputs a warning message.

[0114] Optionally, the alarm module 30 is further used for:

[0115] The alarm message is sent to a target topic of a preset message subscription system, so that the alarm unit obtains the alarm message by subscribing to the target topic, parses the alarm message and sends the parsed result to the user terminal.

[0116] The distributed cluster-based job intervention device provided by the present application adopts the distributed cluster-based job intervention method in the above-mentioned embodiment, which can solve the technical problem of ensuring the stable operation of cluster jobs. Compared with the prior art, the beneficial effects of the distributed cluster-based job intervention device provided by the present application are the same as the beneficial effects of the distributed cluster-based job intervention method provided by the above-mentioned embodiment, and other technical features in the distributed cluster-based job intervention device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0117] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed cluster-based job intervention method in the above-mentioned embodiment one.

[0118] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, notebook computers, PADs (Portable Application Description: tablet computers), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0119] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a hard disk; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0120] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0121] The electronic device provided by the present application adopts the distributed cluster-based job intervention method in the above embodiment, which can solve the technical problem of ensuring the stable operation of cluster operations. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the distributed cluster-based job intervention method provided in the above embodiment, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0122] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0124] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the distributed cluster-based job intervention method in the above-mentioned embodiment.

[0125] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0126] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0127] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: for any real-time job in the job list of the distributed cluster, obtains the job indicators of the real-time job, and determines the monitoring and analysis data of the real-time job based on the job indicators; when the monitoring and analysis data hits the corresponding monitoring rules in the local database, the real-time job is determined to be an abnormal job, and an intervention operation is performed on the abnormal job to enable the abnormal job to run normally.

[0128] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0131] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned distributed cluster-based job intervention method, and can solve the technical problem of ensuring the stable operation of cluster jobs. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the distributed cluster-based job intervention method provided in the above-mentioned embodiment, and will not be repeated here.

[0132] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned distributed cluster-based job intervention method when executed by a processor.

[0133] The computer program product provided in this application can solve the technical problem of ensuring the stable operation of cluster operations. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the distributed cluster-based operation intervention method provided in the above embodiment, which will not be repeated here.

[0134] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A distributed cluster-based job intervention method, characterized in that: The monitoring unit applied to the distributed cluster, the distributed cluster-based job intervention method includes: For any real-time job in the job list of the distributed cluster, obtain a job index of the real-time job, and determine monitoring and analysis data of the real-time job based on the job index; When the monitoring analysis data hits the corresponding monitoring rule in the local database, the real-time operation is determined to be an abnormal operation, and an intervention operation is performed on the abnormal operation to enable the abnormal operation to run normally.

2. The distributed cluster-based job intervention method according to claim 1, characterized in that: The step of determining the monitoring and analysis data of the real-time operation based on the operation index comprises: Aggregating the data of the operation indicators according to preset dimensions; Based on the job indicators after data aggregation, the monitoring and analysis data is calculated according to the preset calculation logic, and the monitoring and analysis data includes at least one of job processing delay, transactions processed per second, average memory usage, and maximum memory usage.

3. The distributed cluster-based job intervention method according to claim 2, characterized in that: When the monitoring analysis data hits the corresponding monitoring rule in the local database, the step of determining that the real-time operation is an abnormal operation includes: When the job processing delay of the real-time job exceeds a first preset abnormal threshold, and / or the number of transactions processed per second is lower than a second preset abnormal threshold, and / or the average memory usage exceeds a third preset abnormal threshold, and / or the maximum memory usage exceeds a fourth preset abnormal threshold, it is determined that the monitoring and analysis data hits a corresponding monitoring rule in a local database, and it is determined that the real-time job belongs to an abnormal job of an abnormal job type; When there is a target job identical to the real-time job in the job list of the distributed cluster, it is determined that the monitoring analysis data hits the corresponding monitoring rule in the local database, and it is determined that the real-time job is an abnormal job of a repeated job type.

4. The distributed cluster-based job intervention method according to claim 3, characterized in that: The step of performing intervention operation on the abnormal operation comprises: In the case where the type of the abnormal operation is the abnormal operation, restarting the abnormal operation and recording the number of restarts, and when the number of restarts reaches a preset restart threshold, notifying the user to process the abnormal operation; In the case that the type of the abnormal job includes the repeated job, the abnormal job is terminated and the number of terminations is recorded. When the number of terminations reaches a preset termination threshold, the user is notified to process the abnormal job.

5. The distributed cluster-based job intervention method according to claim 1, characterized in that: The distributed cluster-based job intervention method further includes: When the monitoring and analysis data hits the corresponding alarm rule in the local database, an alarm event is determined, and the context information of the alarm event is encapsulated into an alarm message, wherein the alarm message includes the job name, the alarm event type, the alarm event details, the alarm event time, and handling suggestions; The warning message is output.

6. The distributed cluster-based job intervention method according to claim 5, characterized in that: The distributed cluster includes an alarm unit, and the step of outputting the alarm message includes: The alarm message is sent to a target topic of a preset message subscription system, so that the alarm unit obtains the alarm message by subscribing to the target topic, parses the alarm message and sends the parsed result to the user terminal.

7. A distributed cluster-based job intervention device, characterized in that: A monitoring unit applied to a distributed cluster, wherein the distributed cluster-based operation intervention device comprises: A data generation module, configured to obtain a job index of any real-time job in a job list of the distributed cluster, and determine monitoring and analysis data of the real-time job based on the job index; The intervention module is used to determine that the real-time operation is an abnormal operation when the monitoring and analysis data hits the corresponding monitoring rules in the local database, and perform intervention operations on the abnormal operation to make the abnormal operation run normally.

8. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the distributed cluster-based job intervention method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the distributed cluster-based job intervention method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the distributed cluster-based job intervention method according to any one of claims 1 to 6 are implemented.

Images