A big data cluster task monitoring method, device, equipment and storage medium

Abstract

The present application relates to a big data cluster task monitoring method, apparatus, equipment and storage medium. Based on engine data of multiple engines collected in real time, the resource contention degree of different engine pairs is calculated to generate a resource contention matrix; based on a pre-trained time series prediction model, the engine indicator baseline prediction value corresponding to each of the multiple engines is predicted, and the real-time engine indicator value corresponding to each of the multiple engines is compared with the engine indicator baseline prediction value. If the real-time engine indicator value does not meet the engine indicator baseline prediction value, the abnormal classification model corresponding to the target engine is obtained to perform root cause diagnosis and determine the engine indicator abnormality type; based on the engine indicator abnormality type, a recovery strategy set is determined, and based on the resource contention matrix, each recovery strategy in the recovery strategy set is scored and processed respectively, and based on the scoring results, the optimal recovery strategy is determined and executed; compared with the existing technology, the technical solution of the present application can improve cluster stability and operation and maintenance efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a big data cluster task monitoring method, apparatus, device, and storage medium. Background Art

[0002] In big data cluster operation and maintenance scenarios, the coexistence of multi-engine heterogeneous environments including engines such as Flink, Spark, and Tez faces problems such as significant differences in abnormal characteristics, easily aggravated resource competition, and rapid cross-engine abnormality propagation, posing severe challenges to the stable operation of the cluster and business continuity.

[0003] To ensure reliable cluster operation, existing technologies typically use a solution that combines multi-source data collection and unified observation, anomaly detection and root cause location, and self-healing strategies with automated execution. By integrating monitoring data, identifying anomalies, and triggering recovery operations, they attempt to achieve efficient cluster operation and maintenance.

[0004] However, existing technologies have many defects in practical applications: First, the accuracy of anomaly detection is insufficient. In complex cluster environments, the rule engine that uses a combination of static and dynamic thresholds is easily affected by environmental fluctuations, frequently misjudging anomalies, and then triggering unnecessary self-healing operations such as restarts, which in turn undermines cluster stability; Second, the fault handling capability is limited: it can only deal with standardized, high-frequency simple faults, lacks effective means to deal with deep and complex problems, and still relies on manual intervention, resulting in low fault recovery efficiency; Third, the self-healing mechanism has poor adaptability: self-healing rules need to be continuously maintained as cluster components are upgraded, and related scripts need to be re-adapted to the new environment, which not only leads to a low self-healing rate, but also significantly increases operation and maintenance costs; Fourth, the root cause analysis efficiency is inefficient: key data such as logs, indicators, and tracking are stored in different systems such as ELK, Prometheus, and Zabbix. Root cause location requires cross-platform query and integration, which seriously delays the timing of fault handling. Summary of the Invention

[0005] This application provides a big data cluster task monitoring method, device, equipment and storage medium, which can improve cluster stability and operation and maintenance efficiency.

[0006] In the first aspect, the present application provides a big data cluster task monitoring method, including: based on the engine data corresponding to each of the multiple engines collected in real time, calculating the resource contention degree corresponding to each of the different engines, and generating a resource contention matrix based on the resource contention degree; predicting the engine indicator baseline prediction value corresponding to each of the multiple engines based on a pre-trained time series prediction model, and comparing the real-time engine indicator value corresponding to each of the multiple engines with the engine indicator baseline prediction value. If the real-time engine indicator value does not meet the engine indicator baseline prediction value, obtaining the abnormality classification model corresponding to the target engine for root cause diagnosis and determining the engine indicator abnormality type; based on the engine indicator abnormality type, determining a recovery strategy set, and based on the resource contention matrix, scoring each recovery strategy in the recovery strategy set respectively, and determining and executing the optimal recovery strategy based on the scoring results.

[0007] In a possible implementation, the engine data corresponding to each of the multiple engines collected in real time is used to calculate the resource contention degree corresponding to each of the different engine pairs, and based on the resource contention degree, a resource contention matrix is ​​generated, specifically including: when executing the current cluster task, real-time collection of the engine data corresponding to each of the multiple engines within a preset time window, wherein the engine data includes engine CPU data, engine memory data and engine throughput data; combining the multiple engines in pairs to obtain multiple engine pairs, and based on the engine CPU data, respectively calculating the CPU contention coefficient corresponding to each of the multiple engine pairs, based on the engine memory data, respectively calculating the memory contention coefficient corresponding to each of the multiple engine pairs, and based on the engine throughput data, respectively calculating the throughput of each of the multiple engine pairs. The method comprises the following steps: selecting any one of the multiple engines as the task engine in turn, obtaining all CPU contention coefficients, all memory contention coefficients and all network contention coefficients of the task engine in different engine pairs, and respectively obtaining the maximum values ​​of all CPU contention coefficients, all memory contention coefficients and all network contention coefficients to obtain the target CPU contention coefficient, target memory contention coefficient and target network contention coefficient; performing weighted fusion processing on the target CPU contention coefficient, the target memory contention coefficient and the target network contention coefficient to obtain the resource contention degree between different engine pairs, and taking the resource contention degree as the comprehensive resource contention degree corresponding to the task engine; integrating the comprehensive resource contention degrees corresponding to each of the multiple engines to determine a resource contention degree matrix.

[0008] In a possible implementation, the CPU contention coefficients corresponding to each of the multiple engine pairs are calculated based on the engine CPU data, the memory contention coefficients corresponding to each of the multiple engine pairs are calculated based on the engine memory data, and the network contention coefficients corresponding to each of the multiple engine pairs are calculated based on the engine throughput data. Specifically, the method includes: obtaining first engine CPU data corresponding to the first engine in each engine pair, and second engine CPU data corresponding to the second engine, calculating engine CPU total usage data based on the first engine CPU data and the second engine CPU data, obtaining cluster total CPU data corresponding to the current cluster, and determining the CPU contention coefficients corresponding to each of the multiple engine pairs based on the cluster total CPU data and the engine CPU total usage data; obtaining The first engine memory data corresponding to the first engine in each engine pair and the second engine memory data corresponding to the second engine are obtained. Based on the first engine memory data and the second engine memory data, the engine memory usage data is calculated to obtain the total cluster memory data corresponding to the current cluster. Based on the total cluster memory data and the engine memory usage data, the memory contention coefficients corresponding to each of the multiple engine pairs are determined. The first engine throughput data corresponding to the first engine in each engine pair and the second engine throughput data corresponding to the second engine are obtained. Based on the first engine throughput data and the second engine throughput data, the engine total throughput data is calculated to obtain the physical bandwidth data corresponding to the current cluster. Based on the total engine throughput data and the physical bandwidth data, the network contention coefficients corresponding to each of the multiple engine pairs are determined.

[0009] In one possible implementation, the pre-trained time series prediction model predicts the engine indicator baseline prediction value corresponding to each of the multiple engines, specifically including: collecting historical engine indicator data corresponding to the multiple engines, pre-processing the historical engine indicator data to obtain pre-processed historical engine indicator data, and arranging the pre-processed historical engine indicator data based on the collection timestamp to generate a historical engine indicator sequence; setting model parameters for the pre-built model to obtain an initial time series prediction model; using the historical engine indicator sequence as model input, and using the engine indicator baseline prediction value at a future preset moment and the confidence interval of the engine indicator baseline prediction value as model output, model training the initial time series prediction model to obtain a time series prediction model; obtaining the target historical engine indicator sequence corresponding to each of the multiple engines, and inputting the target historical engine indicator sequence into the time series prediction model respectively to obtain the engine indicator baseline prediction value at the future target preset moment corresponding to each of the multiple engines.

[0010] In one possible implementation, obtaining an abnormality classification model corresponding to the target engine for root cause diagnosis and determining the abnormality type of the engine indicator specifically includes: matching the corresponding abnormality classification model based on the target engine; collecting the target real-time engine indicator value of the target engine, and inputting the target real-time engine indicator value into the abnormality classification model corresponding to the target engine, so that the abnormality classification model outputs an abnormality type probability vector; and determining the abnormality type of the engine indicator of the target engine based on the abnormality type probability vector.

[0011] In one possible implementation, based on the resource contention matrix, each recovery strategy in the recovery strategy set is scored separately, specifically including: extracting the target comprehensive resource contention corresponding to the target engine from the resource contention matrix, obtaining the task type and task priority corresponding to the current cluster task, and obtaining the policy cost value corresponding to each recovery strategy in the recovery strategy set, wherein the recovery strategy set contains at least one recovery strategy for the target engine abnormality type; based on the target comprehensive resource contention, the task type, the task priority and the policy cost value, each recovery strategy in the recovery strategy set is scored separately to obtain the score value corresponding to each recovery strategy in the recovery strategy set.

[0012] In one possible implementation, executing the optimal recovery strategy specifically includes: determining whether the optimal recovery strategy is a multi-strategy collaborative strategy; if the optimal recovery strategy is a multi-strategy collaborative strategy, performing conflict detection on all sub-recovery strategies in the optimal recovery strategy to determine conflicting sub-recovery strategy pairs; performing dependency analysis on all sub-recovery strategies in the optimal recovery strategy to determine the dependencies corresponding to all sub-recovery strategies; monitoring cluster resources in real time based on a resource coordinator, and allocating corresponding resource budgets to all sub-recovery strategies in the optimal recovery strategy; obtaining the sub-recovery strategy priority corresponding to each sub-recovery strategy in the optimal recovery strategy, and allocating all sub-recovery strategies in the optimal recovery strategy to different time windows based on the sub-recovery strategy priority, the dependencies, and the conflicting sub-recovery strategy pairs, and executing the sub-recovery strategy corresponding to each time window in sequence based on the time window sequence; monitoring the execution status of each sub-recovery strategy in real time, and outputting an execution completion result if all sub-recovery strategies in the optimal recovery strategy are successfully executed; and triggering a preset fallback plan if any sub-recovery strategy in the optimal recovery strategy fails to execute.

[0013] In the second aspect, the present application provides a big data cluster task monitoring device, including: a resource contention matrix generation module, an engine indicator abnormality type determination module and an optimal recovery strategy determination module; wherein the resource contention matrix generation module is used to calculate the resource contention corresponding to each engine of different engines based on the engine data corresponding to each of the multiple engines collected in real time, and generate a resource contention matrix based on the resource contention; the engine indicator abnormality type determination module is used to predict the engine indicator baseline prediction value corresponding to each of the multiple engines based on a pre-trained time series prediction model, and compare the real-time engine indicator value corresponding to each of the multiple engines with the engine indicator baseline prediction value. If the real-time engine indicator value does not meet the engine indicator baseline prediction value, the abnormality classification model corresponding to the target engine is obtained for root cause diagnosis to determine the engine indicator abnormality type; the optimal recovery strategy determination module is used to determine a recovery strategy set based on the engine indicator abnormality type, and score each recovery strategy in the recovery strategy set based on the resource contention matrix, and determine and execute the optimal recovery strategy based on the scoring result.

[0014] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0016] The present invention provides a method that has the following advantages over the prior art:

[0017] Based on the engine data corresponding to each of the multiple engines collected in real time, the resource contention degree of different engines for each of the corresponding engines is calculated, and based on the resource contention degree, a resource contention matrix is ​​generated; based on the pre-trained time series prediction model, the engine indicator baseline prediction value corresponding to each of the multiple engines is predicted, and the real-time engine indicator value corresponding to each of the multiple engines is compared with the engine indicator baseline prediction value. If the real-time engine indicator value does not meet the engine indicator baseline prediction value, the abnormal classification model corresponding to the target engine is obtained to perform root cause diagnosis and determine the engine indicator abnormality type; based on the engine indicator abnormality type, a recovery strategy set is determined, and based on the resource contention matrix, each recovery strategy in the recovery strategy set is scored and processed respectively. Based on the scoring results, the optimal strategy is determined and executed. Recovery strategy; Compared with the existing technology, the technical solution of the present application quantifies the intensity of resource contention between engines by calculating the resource competition matrix in real time, and combines the pre-trained time series prediction model to generate engine indicator baseline prediction values ​​corresponding to multiple engines to achieve accurate anomaly detection, and then locates the root cause through the anomaly classification model corresponding to the target engine, and scores and selects the recovery strategy based on the resource competition situation. It can adapt to cluster environment fluctuations to reduce anomaly misjudgment and unnecessary self-healing operations, accurately identify deep and complex faults and locate the causes of resource conflicts, and dynamically adapt to environmental changes after cluster component upgrades to improve strategy adaptability. It effectively solves the problems of high anomaly detection misjudgment rate, weak complex fault handling capabilities, and poor self-healing strategy adaptability in the existing technology, thereby improving cluster stability, reducing manual dependence and improving operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1 This is a flowchart of an embodiment of a big data cluster task monitoring method provided by the present application;

[0022] Figure 2This is a structural diagram of an embodiment of a big data cluster task monitoring device provided by the present application;

[0023] Figure 3 This is a structural diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0026] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0027] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0030] Example 1, see Figure 1 , Figure 1 This is a flow chart of an embodiment of a big data cluster task monitoring method provided by this application. Figure 1 As shown, the method includes steps 101 to 103, which are specifically as follows:

[0031] Step 101: Based on the real-time collected engine data corresponding to the plurality of engines, the resource contention degrees corresponding to the different engines are calculated, and based on the resource contention degrees, a resource competition matrix is ​​generated.

[0032] In one embodiment, when executing the current cluster task, engine data corresponding to each of the multiple engines within a preset time window is collected in real time, wherein the engine data includes engine CPU data, engine memory data, and engine throughput data.

[0033] Specifically, the multiple engines include a Flink engine, a Spark engine, and a Tez engine.

[0034] Specifically, the preset time window is 5 seconds.

[0035] Specifically, the engine data corresponding to each of the multiple engines includes Flink engine data, Spark engine data, and Tez engine data; wherein, the Flink engine data includes but is not limited to Flink engine CPU data, Flink engine memory data, and Flink engine throughput data; the Spark engine data includes but is not limited to Spark engine CPU data, Spark engine memory data, and Spark engine throughput data; the Tez engine data includes but is not limited to Tez engine CPU data, Tez engine memory data, and Tez engine throughput data.

[0036] Preferably, when the engine data corresponding to each of the multiple engines is obtained, the engine data is preprocessed by using NTP time synchronization, aggregating indicators by 5-second windows, and using linear interpolation to fill missing values.

[0037] In one embodiment, the multiple engines are combined in pairs to obtain multiple engine pairs.

[0038] Specifically, the multiple engine pairs are a Flink-Spark engine pair, a Spark-Tez engine pair, and a Flink-Tez engine pair.

[0039] In one embodiment, based on the engine CPU data, the CPU contention coefficients corresponding to the plurality of engines are calculated respectively.

[0040] Specifically, obtain the first engine CPU data corresponding to the first engine in each engine pair, and the second engine CPU data corresponding to the second engine, calculate the engine CPU total usage data based on the first engine CPU data and the second engine CPU data, obtain the cluster total CPU data corresponding to the current cluster, and determine the CPU contention coefficient corresponding to each of the multiple engine pairs based on the cluster total CPU data and the engine CPU total usage data.

[0041] Specifically, the engine CPU data sum of the first engine CPU data and the second engine CPU data is calculated, and the engine CPU data sum is used as the total engine CPU usage data.

[0042] Specifically, a first ratio between the cluster total CPU data and the engine CPU total usage data is calculated, and a square value of the first ratio is calculated, and the square value of the first ratio is used as the CPU contention coefficient.

[0043] Specifically, the CPU contention coefficient includes a Flink-Spark engine CPU contention coefficient, a Spark-Tez engine CPU contention coefficient, and a Flink-Tez engine CPU contention coefficient.

[0044] Specifically, the CPU contention coefficient quantifies the intensity of competition for CPU computing resources between two computing engines. When the sum of the CPU usage data of multiple engines is close to the total CPU usage data of the cluster, it means that the CPU resources are close to saturation. At this time, the competition pressure between engines does not increase linearly with the resource occupancy rate, but instead increases exponentially. The proportion of the CPU usage data of the engine pair to the total CPU usage data of the cluster is also important. This relative proportion can intuitively reflect that the higher the weight of different engine pairs in the CPU resource competition, the more CPU resources the two engines consume together, the greater their influence on the overall CPU resource competition, and the corresponding CPU contention coefficient will also be higher. The CPU contention coefficient captures the risk of intensified competition when resources are saturated through the nonlinear amplification effect, and clarifies the influence weight of the engine pair in the competition through relative proportion measurement. The combination of the two realizes the accurate quantification of the intensity of CPU resource competition between the two engines.

[0045] In one embodiment, based on the engine memory data, the memory contention coefficients corresponding to the plurality of engine pairs are calculated respectively.

[0046] Specifically, first engine memory data corresponding to the first engine in each engine pair and second engine memory data corresponding to the second engine are obtained, engine memory usage data is calculated based on the first engine memory data and the second engine memory data, cluster total memory data corresponding to the current cluster is obtained, and memory contention coefficients corresponding to each of the multiple engine pairs are determined based on the cluster total memory data and the engine memory usage data.

[0047] Specifically, the engine memory data sum of the first engine memory data and the second engine memory data is calculated, and the engine memory data sum is used as the engine memory usage data.

[0048] Preferably, if a shared cache exists, the engine memory data sum of the first engine memory data and the second engine memory data is calculated, and the shared cache value of the shared cache is subtracted from the engine memory data sum to obtain the engine memory usage data.

[0049] Specifically, a second ratio between the engine memory usage data and the cluster total memory data is calculated, and the second ratio is used as the memory contention coefficient, that is, memory contention coefficient=engine memory usage data / cluster total memory data.

[0050] Specifically, the CPU contention coefficient includes a Flink-Spark engine memory contention coefficient, a Spark-Tez engine memory contention coefficient, and a Flink-Tez engine memory contention coefficient.

[0051] Specifically, the memory contention coefficient measures the intensity of competition for memory resources between engines, especially considering the impact of shared cache. If a shared cache exists, the shared cache portion is removed from the total memory usage, retaining only the exclusive memory consumption of each engine. This avoids miscounting the shared cache into the actual memory requirements of the engine, ensuring that the contention coefficient only reflects the competition between engines for non-shareable exclusive memory resources, making the results more in line with real resource competition scenarios.

[0052] In one embodiment, based on the engine throughput data, the network contention coefficients corresponding to the plurality of engine pairs are calculated respectively.

[0053] Specifically, first engine throughput data corresponding to the first engine in each engine pair and second engine throughput data corresponding to the second engine are obtained, total engine throughput data is calculated based on the first engine throughput data and the second engine throughput data, physical bandwidth data corresponding to the current cluster is obtained, and network contention coefficients corresponding to each of the multiple engine pairs are determined based on the total engine throughput data and the physical bandwidth data.

[0054] Specifically, the sum of the engine throughput data of the first engine throughput data and the second engine throughput data is calculated, and the sum of the engine throughput data is used as the total engine throughput data.

[0055] Specifically, the engine total throughput data and the physical bandwidth data are substituted into a preset network contention coefficient calculation formula to obtain a network contention coefficient; wherein the network contention coefficient calculation formula is: network contention coefficient=min(1.0, engine total throughput data / physical bandwidth data).

[0056] Specifically, the CPU contention coefficient includes a Flink-Spark engine network contention coefficient, a Spark-Tez engine network contention coefficient, and a Flink-Tez engine network contention coefficient.

[0057] Specifically, the network contention coefficient evaluates the degree of network bandwidth occupation by the engine combination. The calculation of the network contention coefficient mainly evaluates the intensity of the engine combination's competition for network bandwidth from two dimensions: whether the bandwidth limit is reached and the actual occupation ratio. The combination of the two enables an accurate assessment of the engine combination's network bandwidth competition intensity.

[0058] In one embodiment, any one of the multiple engines is selected as the task engine in turn, and all CPU contention coefficients, all memory contention coefficients, and all network contention coefficients of the task engine in different engine pairs are obtained, and the maximum values ​​of all CPU contention coefficients, all memory contention coefficients, and all network contention coefficients are obtained respectively to obtain the target CPU contention coefficient, target memory contention coefficient, and target network contention coefficient.

[0059] Specifically, when selecting any of multiple engines as a task engine, you must first identify the engine pairs formed between that task engine and all other engines. For example, when selecting Flink as the task engine, the corresponding engine pairs are Flink-Spark and Flink-Tez; when selecting Spark as the task engine, the corresponding engine pairs are Flink-Spark and Spark-Tez; and when selecting Tez as the task engine, the corresponding engine pairs are Spark-Tez and Flink-Tez. These engine pairs cover the resource competition relationships between the task engine and all other engines.

[0060] Specifically, for each task engine, its CPU contention coefficient, memory contention coefficient, and network contention coefficient across all engine pairs must be extracted. For example, if the task engine is the Flink engine, the CPU contention coefficient, memory contention coefficient, and network contention coefficient for the Flink-Spark engine pair can be obtained from the Flink-Spark engine pair; the CPU contention coefficient, memory contention coefficient, and network contention coefficient for the Flink-Tez engine pair can be obtained from the Flink-Tez engine pair. In this case, the total CPU contention coefficient for the Flink task engine is [Flink-Spark engine pair CPU contention coefficient, Flink-Tez engine pair CPU contention coefficient], the total memory contention coefficient is [Flink-Spark engine pair memory contention coefficient, Flink-Tez engine pair memory contention coefficient], and the total network contention coefficient is [Flink-Spark engine pair network contention coefficient, Flink-Tez engine pair network contention coefficient].

[0061] Specifically, after obtaining all the aforementioned contention coefficients, the maximum value of the coefficients for each of the three resource dimensions is taken to obtain the target contention coefficient for the task engine. For example, among all the CPU contention coefficients corresponding to the Flink task engine, if the CPU contention coefficient for the Flink-Spark engine is 0.7 and the CPU contention coefficient for the Flink-Tez engine is 0.9, then the target CPU contention coefficient is 0.9. Similarly, the maximum contention coefficients for the memory and network dimensions are taken to obtain the target memory contention coefficients and the target network contention coefficients. The core of this process is to focus on the most severe competition scenarios faced by the task engine in each resource dimension, and to reflect the extreme pressure of its resource competition through the maximum value, providing a key basis for the subsequent calculation of the comprehensive resource contention degree.

[0062] In one embodiment, the target CPU contention coefficient, the target memory contention coefficient, and the reduced target network contention coefficient are weighted and fused to obtain the resource contention degree between different engine pairs, and the resource contention degree is used as the comprehensive resource contention degree corresponding to the task engine.

[0063] Specifically, the target CPU contention coefficient, the target memory contention coefficient and the target network contention coefficient are substituted into the preset comprehensive resource contention degree calculation formula to obtain the comprehensive resource contention degree corresponding to the task engine; wherein, the comprehensive resource contention degree calculation formula is: comprehensive contention degree = (0.6×CPU contention coefficient) + (0.3×memory contention coefficient) + (0.1×network contention coefficient).

[0064] Specifically, the default weight value of the CPU contention coefficient is 0.6, the weight value of the memory contention coefficient is 0.3, and the weight value of the network contention coefficient is 0.1; preferably, the priority of each contention coefficient can also be dynamically adjusted based on the importance of the business; for example, for a compute-intensive cluster, the CPU weight is increased; for a memory-intensive cluster, the memory weight is increased; for a network-intensive cluster, the network weight is increased.

[0065] In one embodiment, the comprehensive resource contention corresponding to each of the multiple engines is integrated to determine a resource contention matrix.

[0066] Step 102: Based on the pre-trained time series prediction model, the engine indicator baseline prediction value corresponding to each of the multiple engines is predicted, and the real-time engine indicator value corresponding to each of the multiple engines is compared with the engine indicator baseline prediction value. If the real-time engine indicator value does not meet the engine indicator baseline prediction value, the abnormality classification model corresponding to the target engine is obtained to perform root cause diagnosis and determine the engine indicator abnormality type.

[0067] In one embodiment, when predicting the engine indicator baseline prediction values ​​corresponding to each of the multiple engines based on a pre-trained time series prediction model, historical engine indicator data corresponding to the multiple engines are collected, the historical engine indicator data are pre-processed to obtain pre-processed historical engine indicator data, and the pre-processed historical engine indicator data are arranged based on the collection timestamp to generate a historical engine indicator sequence; model parameters are set for the pre-built model to obtain an initial time series prediction model; the historical engine indicator sequence is used as model input, and the engine indicator baseline prediction value at a future preset moment and the confidence interval of the engine indicator baseline prediction value are used as model output, and the initial time series prediction model is trained to obtain a time series prediction model; the target historical engine indicator sequence corresponding to each of the multiple engines is obtained, and the target historical engine indicator sequence is respectively input into the time series prediction model to obtain the engine indicator baseline prediction value at the future target preset moment corresponding to each of the multiple engines.

[0068] Specifically, historical engine indicator data corresponding to the multiple engines are collected at fixed time intervals; preferably, the fixed time interval is 5 minutes.

[0069] Specifically, the collected historical engine indicator data corresponding to the multiple engines include Flink historical engine indicator data, Spark historical engine indicator data and Tez historical engine indicator data; wherein, the Flink historical engine indicator data includes but is not limited to Checkpoint duration and backpressure rate; the Spark historical engine indicator data includes but is not limited to Stage execution duration and Shuffle overflow; the Tez historical engine indicator data includes but is not limited to DAG completion time.

[0070] Preferably, the Flink historical engine indicator data also includes Checkpoint success rate, Watermark offset, start timestamp, duration, size, and interval; the Spark historical engine indicator data also includes Executor heartbeat, data volume, read data volume, number of disk overflows, network transmission volume, and time consumption; the Tez historical engine indicator data also includes tracking DAG progress, container allocation delay, AM health, DAG execution tasks, dependencies, execution time, and resource usage.

[0071] Specifically, the preprocessing includes but is not limited to supplementing missing values ​​through linear interpolation and filtering outliers using sliding window quantiles to ensure the integrity and accuracy of the data.

[0072] Specifically, the preprocessed historical engine indicator data is arranged in order according to the collection timestamp to form a historical engine indicator sequence; this chronological sequence can fully reflect the changing trend of the indicator over time and provide a time series prediction model with an input format that meets the requirements.

[0073] Specifically, model parameter setting and training are the key to building an effective prediction model; when setting model parameters for a pre-built model, the set model parameters include but are not limited to setting growth trends (such as linear growth trends), seasonal patterns (such as multiplicative seasonality), and custom seasonal items (daily and weekly seasonality) based on the Prophet algorithm to obtain an initial time series prediction model; during the training process, each historical engine indicator sequence is input into the corresponding initial model, and the indicator baseline forecast value and confidence interval at a preset time in the future are used as the output target. By continuously iteratively optimizing the model parameters, the model can learn the time series laws in historical data, such as periodic fluctuations, long-term trends, etc., and finally obtain a time series prediction model corresponding to each mature historical engine indicator sequence; for example, the model is trained using the historical sequences of the past 7 days to enable it to grasp the changing characteristics of the indicators within a day and a week, thereby accurately predicting the baseline value for the next 4 hours.

[0074] Specifically, the target historical engine indicator sequence is input into the corresponding time series prediction model respectively, so that the time series prediction model outputs the engine indicator baseline prediction value of the engine at the future target preset moment and the confidence interval of the engine indicator baseline prediction value based on the learned time series rules; this process realizes the transformation from historical data to future predictions, and provides a quantitative standard for subsequent judgment of whether the engine indicator is abnormal.

[0075] In one embodiment, the real-time engine index value corresponding to each of the multiple engines is compared with the confidence interval corresponding to the engine index baseline prediction value. If the real-time engine index value is not within the confidence interval, it is determined that the real-time engine index value does not meet the engine index baseline prediction value; if the real-time engine index value is within the confidence interval, it is determined that the real-time engine index value meets the engine index baseline prediction value.

[0076] In one embodiment, the target engine refers to an engine whose real-time engine index value does not meet the engine index baseline prediction value, that is, an engine with an abnormal index.

[0077] In one embodiment, an abnormality classification model corresponding to a target engine is obtained for root cause diagnosis. When determining the abnormality type of an engine indicator, the corresponding abnormality classification model is matched based on the target engine; a target real-time engine indicator value of the target engine is collected, and the target real-time engine indicator value is input into the abnormality classification model corresponding to the target engine, so that the abnormality classification model outputs an abnormality type probability vector; and based on the abnormality type probability vector, the abnormality type of the engine indicator of the target engine is determined.

[0078] Specifically, the target engine matches corresponding exception classification models including the Flink exception classification model, the Spark exception classification model, and the Tez exception classification model.

[0079] Specifically, the Flink anomaly classification model is built based on a long short-term memory network and is specifically designed to detect checkpoint anomaly patterns. Its core function is to accurately identify anomaly types by learning the changing patterns of checkpoint temporal features. Model training relies on historically annotated checkpoint anomaly event data, which is divided into training and test sets in a 5:1 ratio. By learning the temporal feature patterns of historical anomalies, the model is able to identify new anomalies and provide accurate root cause classification for checkpoint anomalies in Flink tasks.

[0080] Specifically, the model structure of the Flink exception classification model includes a three-layer long short-term memory network, each layer of which contains 128 units. The three-layer long short-term memory network is connected to a fully connected layer, and the fully connected layer outputs a five-dimensional probability vector. The five-dimensional probability vector corresponds to five preset Flink engine indicator exception types: timeout, failure, partial completion, slow speed, and other.

[0081] Specifically, when using the Flink exception classification model for classification, the target real-time engine indicator value of the Flink engine is input into the Flink exception classification model, so that the Flink exception classification model outputs a five-dimensional exception type probability vector, and the type corresponding to the maximum value of the probability vector in the five-dimensional exception type probability vector is taken as the engine indicator exception type of the target engine; wherein, the target real-time engine indicator value of the Flink engine is a multi-dimensional time series feature related to Checkpoint, including Checkpoint success rate, backpressure rate, Watermark offset, duration, start timestamp, duration, size, interval, etc.; after these features are standardized, a sequence of preset time steps is formed to capture the dynamic changes of Checkpoint in a continuous time period.

[0082] Specifically, the Spark anomaly classification model adopts the isolation forest algorithm, which is mainly used to identify anomalies in the Shuffle process. Its core logic is to judge the anomaly type by detecting the degree of deviation between the Shuffle feature and the normal mode. The Spark anomaly classification model only uses normal samples in the model training phase, such as Shuffle process data without anomalies. By constructing 100 trees, sampling 256 samples per tree, and calculating the path length threshold of normal samples, this is used as the benchmark for judging anomalies. During classification, the target real-time engine indicator value of the Spark engine is input into the Spark anomaly classification model to calculate its anomaly score. If the anomaly score is If it exceeds the preset anomaly score threshold, it is determined to be an anomaly; at the same time, the contribution vector of each feature to the anomaly score is combined to obtain the maximum value of the contribution vector to determine the engine indicator anomaly type corresponding to the specific Spark engine, such as data skew, network congestion, disk IO bottleneck, etc.; Among them, the target real-time engine indicator value of the Spark engine includes statistical features related to Shuffle, including Stage duration, Executor heartbeat, Shuffle overflow, data volume, read data volume, number of disk overflows, network transmission volume, time consumption, etc. These features are aggregated and calculated with a 60-second window to form a feature vector reflecting the short-term state of Shuffle.

[0083] Specifically, the Tez anomaly classification model is built based on a graph neural network, focusing on the deviation analysis of the execution path of a directed acyclic graph. It identifies global anomaly types by learning the association between the directed acyclic graph topology and node features.

[0084] Specifically, the model structure of the Tez anomaly classification model includes a three-layer graph convolutional network, each layer of the graph convolutional network is 64-dimensional, and the three-layer graph convolutional network is connected to a fully connected classifier. The fully connected classifier outputs a three-dimensional probability vector, and the three-dimensional probability vector corresponds to three preset engine indicator anomaly types: critical path delay, node failure, and insufficient resources. When performing classification based on the Tez anomaly classification model, the target real-time engine indicator value of the Tez engine is converted into a graph data structure input model. The model calculates the anomaly probability of each node by aggregating node features and topological relationships, and then obtains the global anomaly type through global aggregation. Among them, the target real-time engine indicator value of the Tez engine includes but is not limited to the topological structure of the DAG execution graph (nodes are tasks, edges are dependencies) and node features (execution time, resource usage, DAG progress, container allocation delay, AM health, etc.).

[0085] Preferably, the abnormal type of the engine indicator of the target engine is determined, and an alarm notification is triggered, such as an email or text message.

[0086] Specifically, when the real-time indicator value does not meet the baseline prediction value, the root cause diagnosis process needs to be initiated for the corresponding target engine; the system will call the anomaly classification model corresponding to the target engine, which can identify the specific anomaly type based on the characteristics of the anomaly indicator; this process realizes the transformation from discovering anomalies to clarifying the anomaly type, providing a key basis for the subsequent formulation of targeted recovery strategies.

[0087] Step 103: Based on the engine indicator abnormality type, a recovery strategy set is determined, and based on the resource contention matrix, each recovery strategy in the recovery strategy set is scored, and based on the scoring results, an optimal recovery strategy is determined and executed.

[0088] In one embodiment, when scoring each recovery strategy in the recovery strategy set based on the resource contention matrix, the target comprehensive resource contention corresponding to the target engine is extracted from the resource contention matrix to obtain the task type and task priority corresponding to the current cluster task, and the policy cost value corresponding to each recovery strategy in the recovery strategy set is obtained, wherein the recovery strategy set includes at least one recovery strategy for the target engine abnormality type; based on the target comprehensive resource contention, the task type, the task priority and the policy cost value, each recovery strategy in the recovery strategy set is scored to obtain the scoring value corresponding to each recovery strategy in the recovery strategy set.

[0089] Specifically, based on the task type corresponding to the current cluster task, the task real-time coefficient is determined; wherein the task type includes real-time tasks and batch tasks; the task real-time coefficient corresponding to the real-time task is greater than the task real-time coefficient corresponding to the batch task.

[0090] Specifically, the policy cost value refers to the resource consumption required to execute the recovery policy.

[0091] Specifically, based on the target comprehensive resource competition degree, the task type, the task priority and the strategy cost value, when scoring each recovery strategy in the recovery strategy set, the task real-time coefficient corresponding to the task type is obtained, and the target comprehensive resource competition degree, the task real-time coefficient, the task priority and the strategy cost value are substituted into the preset scoring calculation formula to obtain the scoring value corresponding to each recovery strategy in the recovery strategy set; wherein, the preset scoring calculation formula is as follows: scoring value = W1*task real-time coefficient + W2*task priority + W3*target comprehensive resource competition degree-W4*strategy cost value.

[0092] In one embodiment, when determining and executing the optimal recovery strategy based on the scoring results, the scoring value corresponding to each recovery strategy in the recovery strategy set is selected, and the recovery strategy corresponding to the maximum scoring value is selected as the optimal recovery strategy.

[0093] In one embodiment, when executing the optimal recovery strategy, it is determined whether the optimal recovery strategy is a multi-strategy collaborative strategy; if the optimal recovery strategy is a multi-strategy collaborative strategy, conflict detection is performed on all sub-recovery strategies in the optimal recovery strategy to determine conflicting sub-recovery strategy pairs; dependency analysis is performed on all sub-recovery strategies in the optimal recovery strategy to determine the dependencies corresponding to all sub-recovery strategies; cluster resources are monitored in real time based on a resource coordinator, and corresponding resource budgets are allocated to all sub-recovery strategies in the optimal recovery strategy; the sub-recovery strategy priority corresponding to each sub-recovery strategy in the optimal recovery strategy is obtained, and based on the sub-recovery strategy priority, the dependencies and the conflicting sub-recovery strategy pairs, all sub-recovery strategies in the optimal recovery strategy are allocated to different time windows, and the sub-recovery strategy corresponding to each time window is executed in sequence based on the time window sequence; the execution status of each sub-recovery strategy is monitored in real time, and if all sub-recovery strategies in the optimal recovery strategy are successfully executed, the execution completion result is output; if any sub-recovery strategy in the optimal recovery strategy fails to execute, a preset fallback plan is triggered.

[0094] Specifically, since multiple recovery strategies in a multi-strategy collaborative strategy may need to be executed simultaneously, in order to avoid conflicts, conflict handling, dependency analysis, resource allocation, and timing scheduling are used to ensure the overall collaborative effect and achieve the orderly execution of each sub-recovery strategy.

[0095] Specifically, all sub-recovery strategies within the optimal recovery strategy are individually checked for execution conflicts. For example, a sub-recovery strategy pair consisting of a Flink engine's Savepoint restart (which requires stopping tasks) and a parallelism adjustment (which can be executed dynamically) is identified as a conflicting sub-recovery strategy pair. Furthermore, the dependencies between all sub-strategies are analyzed to identify which strategies require prior execution, such as stopping tasks before resource adjustments. Through conflict detection and dependency analysis, we can identify conflicting scenarios that cannot be run in parallel and clarify the execution order of sub-strategies, providing a basis for subsequent sorting.

[0096] Specifically, resource allocation is performed based on the resource coordinator to avoid sub-recovery strategy execution failure due to resource competition; cluster resources are monitored in real time based on the resource coordinator. After monitoring the currently available resources, a corresponding resource budget is allocated to each sub-recovery strategy based on the resource requirements of each sub-recovery strategy; ensuring that the total resource request of each sub-recovery strategy does not exceed the upper limit of the cluster's available resources to prevent the execution of any sub-strategy from being interrupted due to resource exhaustion; this step provides a stable resource environment for the parallel or serial execution of sub-strategies through resource isolation and quota control.

[0097] Preferably, the purpose of resource isolation is to avoid affecting other tasks during the recovery process and ensure the stability of the recovery environment; for example, the execution process of resource isolation includes creating a temporary queue on YARN and specifying resource quotas; using labels to isolate K8sPods, restricting network policies, and only allowing communication with necessary services; allocating exclusive storage paths for recovery tasks to avoid disk IO competition; among which, the temporary queue is set to be automatically deleted when it times out to prevent resource leakage.

[0098] Specifically, time windows are allocated and execution is sorted based on the sub-strategy's priority, dependencies, and conflicting sub-recovery strategy pairs. Each sub-recovery strategy has a preset priority. The system first filters the execution order by priority, then assigns all sub-strategies to different time windows based on dependencies and conflicting sub-recovery strategy pair processing rules. For example, a high-priority sub-recovery strategy is assigned to the first time window. After its execution completes, the remaining sub-recovery strategies are executed in the second time window, and so on, forming an orderly sequence of time windows.

[0099] Specifically, each sub-strategy is executed in sequence according to the time window sequence, and the execution effect is guaranteed through real-time monitoring and exception handling; the system calls the API of the corresponding engine to execute the sub-strategy in the order of the time windows, and tracks the execution status of each sub-recovery strategy in real time; if all sub-recovery strategies are executed successfully, the overall completion result is output; once a sub-strategy fails to execute, such as not completing due to timeout or returning an error status code, the preset fallback plan is immediately triggered, such as rolling back to the fault site and enabling the backup strategy to ensure system stability under abnormal conditions.

[0100] Preferably, in order to smoothly roll back to the fault site, it is also necessary to perform fault site preservation in the early stage. Its purpose is to capture the fault site information before recovery to facilitate subsequent analysis and rollback. The execution process of fault site preservation includes, for Java applications, using jmap to generate HeapDump and save it to HDFS; for Flink tasks, triggering a Checkpoint to save the status; recording key logs and thread snapshots, and marking all on-site data with timestamps and task IDs, and storing them in the distributed file system for a preset time.

[0101] Specifically, through a multi-dimensional coordination mechanism, we not only resolve the conflicts and dependencies between sub-recovery strategies, but also achieve the overall optimal execution effect through the rational allocation of resources and time, ensuring that the multi-strategy coordination strategy can achieve the recovery goals safely and efficiently.

[0102] In one embodiment, after executing the optimal recovery strategy, it also includes: after executing the optimal recovery strategy, performing a running status check on the current cluster task to determine the task running status check result; if the task running status check result is a pass, then detecting whether self-recovery is completed, and if so, generating a recovery report, and sending a recovery success event to trigger a knowledge precipitation process; wherein, the recovery report includes the recovery time, the execution order of the optimal recovery strategy, and the task running status check result; if it is detected that self-recovery is not complete, then rolling back to the pre-retained fault site data, and triggering an alarm; if the task running status check result is a fail, then automatically trying a backup recovery strategy.

[0103] Specifically, the running status verification of the current cluster task includes but is not limited to: comparing key indicators before and after recovery, checking data consistency, and verifying whether resource usage is within the limit of the isolation queue; wherein, the comparison of key indicators before and after recovery includes comparing whether the processing delay of the Flink task returns to the baseline ±10%, whether the throughput recovers to more than 90% of the normal level, and whether the error rate drops below 0.5%; the data consistency verification includes confirming that there is no data loss through Flink's checkpoint or Spark's checkpoint.

[0104] Specifically, if the status verification passes, the system enters the self-recovery completion detection phase. To verify the completion of self-recovery, the system checks four criteria: whether all recovery operations return a success status code, whether key metrics remain stable within normal ranges for five consecutive minutes, whether temporary isolated resources (such as YARN temporary queues) have been released, and whether key services have passed end-to-end smoke testing. Only when all four conditions are met is self-recovery considered complete. At this point, the system generates a recovery report that includes the recovery time, the execution sequence of the optimal recovery strategy, and the status verification results. A recovery success event is also sent, triggering the knowledge accumulation process and incorporating the recovery experience into the historical case library.

[0105] Specifically, if the task running status check result is a failure, the system will automatically try the backup recovery strategy and retry according to the preset maximum number of retries to further fix the problem; if the self-recovery is not completed, such as temporary resources not being released or business verification failing, it will roll back to the pre-retained fault site data and trigger an alarm to notify the operation and maintenance personnel to intervene; through the immediate rollback and retry mechanism, the impact of recovery failure on the cluster is minimized to ensure business continuity.

[0106] In one embodiment, when the knowledge precipitation process is triggered, feature extraction is performed on the abnormal event corresponding to the engine indicator abnormal type to obtain a core abnormal indicator, and the core abnormal indicator is vectorized to obtain a core abnormal indicator vector. The similarity between the core abnormal indicator vector and the historical abnormal pattern is calculated based on the cosine similarity algorithm; if the similarity is greater than 0.85, the abnormal event is classified as a historical abnormal pattern, and the historical occurrence frequency of the historical abnormal pattern is updated; if the similarity is less than 0.85, the abnormal event is determined to be a new abnormal pattern, and a unique ID is assigned to the new abnormal pattern and the feature information is stored.

[0107] Specifically, when the knowledge precipitation process is triggered, the prediction model parameters are also optimized based on the feedback of abnormal events; by collecting prediction records of recent abnormal events, including positive samples of real anomalies and negative samples of false alarms, the model prediction loss and gradient are calculated, and the model weights are updated using the AdaGrad algorithm, and then the new weights are released after normalization; the update strategy adopts a combination of timing and event triggering: an update is automatically performed every hour, and if a major false alarm or omission occurs, the update is triggered immediately, and the historical weight version is retained for reference; this step improves the model's recognition accuracy of abnormal patterns and enhances the reliability of subsequent anomaly detection through continuous feedback iteration.

[0108] Specifically, when the knowledge accumulation process is triggered, multi-dimensional knowledge output is generated and experience accumulation is completed. For example, by combining the characteristics of abnormal tasks, the comparison of key indicators before and after recovery, resource consumption fluctuations and the recovery operation timeline data, matching the historical optimization case library, and generating practical information such as parameter adjustment recommended values ​​and architecture transformation suggestions; these contents will be presented in the form of Markdown / PDF reports, Grafana visual dashboards, etc., and push notifications will be sent; at the same time, the abnormal pattern, recovery strategy and optimization suggestions of this time will be incorporated into the knowledge system to provide experience support for the rapid diagnosis and recovery of similar faults in the future, and realize the continuous accumulation and reuse of knowledge.

[0109] Example 2, see Figure 2 , Figure 2 This is a schematic diagram of the structure of an embodiment of a big data cluster task monitoring device provided by this application. Corresponding to the above-described big data cluster task monitoring method, this application also provides a big data cluster task monitoring device. This big data cluster task monitoring device includes a module for executing the above-described big data cluster task monitoring method. This big data cluster task monitoring device can be configured in a desktop computer, tablet computer, laptop computer, or other terminal. Specifically, this big data cluster task monitoring device includes a resource contention matrix generation module 201, an engine indicator anomaly type determination module 202, and an optimal recovery strategy determination module 203.

[0110] The resource contention matrix generation module 201 is configured to calculate the resource contention corresponding to different engines based on engine data corresponding to multiple engines collected in real time, and generate a resource contention matrix based on the resource contention.

[0111] The engine indicator abnormality type determination module 202 is used to predict the engine indicator baseline prediction value corresponding to each of the multiple engines based on a pre-trained time series prediction model, and compare the real-time engine indicator value corresponding to each of the multiple engines with the engine indicator baseline prediction value. If the real-time engine indicator value does not meet the engine indicator baseline prediction value, the abnormality classification model corresponding to the target engine is obtained to perform root cause diagnosis and determine the engine indicator abnormality type.

[0112] The optimal recovery strategy determination module 203 is used to determine a recovery strategy set based on the engine indicator abnormality type, and score each recovery strategy in the recovery strategy set based on the resource competition matrix, and determine and execute the optimal recovery strategy based on the scoring results.

[0113] In one embodiment, the resource contention matrix generation module 201 is used to calculate the resource contention corresponding to each of different engine pairs based on the engine data corresponding to each of the multiple engines collected in real time, and generate a resource contention matrix based on the resource contention, specifically including: when executing the current cluster task, collecting the engine data corresponding to each of the multiple engines in a preset time window in real time, wherein the engine data includes engine CPU data, engine memory data and engine throughput data; combining the multiple engines in pairs to obtain multiple engine pairs, and calculating the CPU contention coefficient corresponding to each of the multiple engine pairs based on the engine CPU data, calculating the memory contention coefficient corresponding to each of the multiple engine pairs based on the engine memory data, and calculating the multiple engine throughput data based on the engine throughput data. The network contention coefficient corresponding to each engine pair; any engine among the multiple engines is selected in turn as the task engine, and all CPU contention coefficients, all memory contention coefficients and all network contention coefficients of the task engine in different engine pairs are obtained, and the maximum values ​​of all CPU contention coefficients, all memory contention coefficients and all network contention coefficients are obtained respectively to obtain the target CPU contention coefficient, target memory contention coefficient and target network contention coefficient; the target CPU contention coefficient, the target memory contention coefficient and the target network contention coefficient are weightedly fused to obtain the resource contention degree between different engine pairs, and the resource contention degree is used as the comprehensive resource contention degree corresponding to the task engine; the comprehensive resource contention degree corresponding to each of the multiple engines is integrated to determine the resource contention degree matrix.

[0114] In one embodiment, the resource contention matrix generation module 201 is used to calculate the CPU contention coefficients corresponding to each of the multiple engine pairs based on the engine CPU data, calculate the memory contention coefficients corresponding to each of the multiple engine pairs based on the engine memory data, and calculate the network contention coefficients corresponding to each of the multiple engine pairs based on the engine throughput data. Specifically, the module 201 includes: obtaining the first engine CPU data corresponding to the first engine in each engine pair, and the second engine CPU data corresponding to the second engine, calculating the total engine CPU usage data based on the first engine CPU data and the second engine CPU data, obtaining the total cluster CPU data corresponding to the current cluster, and determining the CPU contention coefficients corresponding to each of the multiple engine pairs based on the total cluster CPU data and the total engine CPU usage data. coefficient; obtain first engine memory data corresponding to the first engine in each engine pair, and second engine memory data corresponding to the second engine, calculate engine memory usage data based on the first engine memory data and the second engine memory data, obtain cluster total memory data corresponding to the current cluster, and determine the memory contention coefficients corresponding to each of the multiple engine pairs based on the cluster total memory data and the engine memory usage data; obtain first engine throughput data corresponding to the first engine in each engine pair, and second engine throughput data corresponding to the second engine, calculate engine total throughput data based on the first engine throughput data and the second engine throughput data, obtain physical bandwidth data corresponding to the current cluster, and determine the network contention coefficients corresponding to each of the multiple engine pairs based on the engine total throughput data and the physical bandwidth data.

[0115] In one embodiment, the engine indicator abnormality type determination module 202 is used to predict the engine indicator baseline prediction value corresponding to each of the multiple engines based on a pre-trained time series prediction model, specifically including: collecting historical engine indicator data corresponding to the multiple engines, pre-processing the historical engine indicator data to obtain pre-processed historical engine indicator data, and arranging the pre-processed historical engine indicator data based on the collection timestamp to generate a historical engine indicator sequence; setting model parameters for the pre-built model to obtain an initial time series prediction model; using the historical engine indicator sequence as model input, and using the engine indicator baseline prediction value at a future preset moment and the confidence interval of the engine indicator baseline prediction value as model output, model training the initial time series prediction model to obtain a time series prediction model; obtaining the target historical engine indicator sequence corresponding to each of the multiple engines, and inputting the target historical engine indicator sequence into the time series prediction model respectively to obtain the engine indicator baseline prediction value at the future target preset moment corresponding to each of the multiple engines.

[0116] In one embodiment, the engine indicator abnormality type determination module 202 is used to obtain the abnormality classification model corresponding to the target engine to perform root cause diagnosis and determine the engine indicator abnormality type, specifically including: matching the corresponding abnormality classification model based on the target engine; collecting the target real-time engine indicator value of the target engine, inputting the target real-time engine indicator value into the abnormality classification model corresponding to the target engine, so that the abnormality classification model outputs an abnormality type probability vector; and determining the engine indicator abnormality type of the target engine based on the abnormality type probability vector.

[0117] In one embodiment, the optimal recovery strategy determination module 203 is used to score each recovery strategy in the recovery strategy set based on the resource contention matrix, specifically including: extracting the target comprehensive resource contention corresponding to the target engine from the resource contention matrix, obtaining the task type and task priority corresponding to the current cluster task, and obtaining the policy cost value corresponding to each recovery strategy in the recovery strategy set, wherein the recovery strategy set includes at least one recovery strategy for the target engine abnormality type; based on the target comprehensive resource contention, the task type, the task priority and the policy cost value, scoring each recovery strategy in the recovery strategy set to obtain the scoring value corresponding to each recovery strategy in the recovery strategy set.

[0118] In one embodiment, the optimal recovery strategy determination module 203 is used to execute the optimal recovery strategy, specifically including: determining whether the optimal recovery strategy is a multi-strategy collaborative strategy; if the optimal recovery strategy is a multi-strategy collaborative strategy, performing conflict detection on all sub-recovery strategies in the optimal recovery strategy to determine conflicting sub-recovery strategy pairs; performing dependency analysis on all sub-recovery strategies in the optimal recovery strategy to determine the dependencies corresponding to all sub-recovery strategies; monitoring cluster resources in real time based on a resource coordinator, and allocating corresponding resource budgets to all sub-recovery strategies in the optimal recovery strategy; obtaining the sub-recovery strategy priority corresponding to each sub-recovery strategy in the optimal recovery strategy, and allocating all sub-recovery strategies in the optimal recovery strategy to different time windows based on the sub-recovery strategy priority, the dependencies, and the conflicting sub-recovery strategy pairs, and sequentially executing the sub-recovery strategy corresponding to each time window based on the time window sequence; monitoring the execution status of each sub-recovery strategy in real time, outputting an execution completion result if all sub-recovery strategies in the optimal recovery strategy are successfully executed, and triggering a preset fallback plan if any sub-recovery strategy in the optimal recovery strategy fails to execute.

[0119] The above-mentioned big data cluster task monitoring device can implement the big data cluster task monitoring method of the above-mentioned method embodiment. The optional options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here.

[0120] like Figure 3 As shown, Figure 3 This is a structural diagram of a computer device provided by the present application; it includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114, and the memory 113 is used to store computer programs.

[0121] In one embodiment of the present application, the processor 111 is configured to implement the big data cluster task monitoring method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113 .

[0122] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0123] Therefore, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the big data cluster task monitoring method provided in any of the aforementioned method embodiments.

[0124] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, which can store program code. The computer-readable storage medium can be either non-volatile or volatile.

[0125] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0127] The steps in the method of the embodiment of the present application can be adjusted in order, combined, and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0128] If this integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application.

[0129] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, as long as these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0131] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A big data cluster task monitoring method, characterized in that: include: Calculating resource contention degrees corresponding to different engines based on engine data corresponding to each of the multiple engines collected in real time, and generating a resource contention matrix based on the resource contention degrees; Predicting the engine indicator baseline prediction values ​​corresponding to each of the multiple engines based on the pre-trained time series prediction model, and comparing the real-time engine indicator values ​​corresponding to each of the multiple engines with the engine indicator baseline prediction values. If the real-time engine indicator values ​​do not meet the engine indicator baseline prediction values, obtaining the anomaly classification model corresponding to the target engine to perform root cause diagnosis and determine the type of engine indicator anomaly; Determine a recovery strategy set based on the engine indicator anomaly type, and score each recovery strategy in the recovery strategy set based on the resource contention matrix, and determine and execute the optimal recovery strategy based on the scoring results; The method of calculating the resource contention of different engines based on the engine data corresponding to each of the multiple engines collected in real time, and generating a resource competition matrix based on the resource contention, specifically includes: When executing the current cluster task, real-time collection of engine data corresponding to each of the multiple engines within a preset time window, wherein the engine data includes engine CPU data, engine memory data, and engine throughput data; Combining the multiple engines in pairs to obtain multiple engine pairs, and calculating CPU contention coefficients corresponding to each of the multiple engine pairs based on the engine CPU data, calculating memory contention coefficients corresponding to each of the multiple engine pairs based on the engine memory data, and calculating network contention coefficients corresponding to each of the multiple engine pairs based on the engine throughput data; Sequentially selecting any one of the multiple engines as a task engine, obtaining all CPU contention coefficients, all memory contention coefficients, and all network contention coefficients of the task engine in different engine pairs, and respectively obtaining the maximum values ​​of all the CPU contention coefficients, all the memory contention coefficients, and all the network contention coefficients to obtain a target CPU contention coefficient, a target memory contention coefficient, and a target network contention coefficient; Performing weighted fusion processing on the target CPU contention coefficient, the target memory contention coefficient, and the target network contention coefficient to obtain resource contention degrees between different engine pairs, and using the resource contention degrees as the comprehensive resource contention degrees corresponding to the task engines; The comprehensive resource contention corresponding to each of the multiple engines is integrated to determine a resource contention matrix.

2. A big data cluster task monitoring method according to claim 1, characterized in that: The calculating of the CPU contention coefficients corresponding to the respective pairs of engines based on the engine CPU data, the calculating of the memory contention coefficients corresponding to the respective pairs of engines based on the engine memory data, and the calculating of the network contention coefficients corresponding to the respective pairs of engines based on the engine throughput data specifically includes: Obtain first engine CPU data corresponding to the first engine in each engine pair, and second engine CPU data corresponding to the second engine, calculate total engine CPU usage data based on the first engine CPU data and the second engine CPU data, obtain cluster total CPU data corresponding to the current cluster, and determine CPU contention coefficients corresponding to each of the multiple engine pairs based on the cluster total CPU data and the engine total CPU usage data; Obtaining first engine memory data corresponding to the first engine and second engine memory data corresponding to the second engine in each engine pair, calculating engine memory usage data based on the first engine memory data and the second engine memory data, obtaining cluster total memory data corresponding to the current cluster, and determining a memory contention coefficient corresponding to each of the multiple engine pairs based on the cluster total memory data and the engine memory usage data; Obtain first engine throughput data corresponding to the first engine in each engine pair, and second engine throughput data corresponding to the second engine, calculate total engine throughput data based on the first engine throughput data and the second engine throughput data, obtain physical bandwidth data corresponding to the current cluster, and determine network contention coefficients corresponding to each of the multiple engine pairs based on the total engine throughput data and the physical bandwidth data.

3. A big data cluster task monitoring method according to claim 1, characterized in that: The pre-trained time series prediction model is used to predict the baseline prediction values ​​of the engine indicators corresponding to the multiple engines, specifically including: Collecting historical engine indicator data corresponding to the multiple engines, preprocessing the historical engine indicator data to obtain preprocessed historical engine indicator data, and arranging the preprocessed historical engine indicator data based on a collection timestamp to generate a historical engine indicator sequence; Set the model parameters for the pre-built model to obtain the initial time series prediction model; The historical engine indicator sequence is used as a model input, and the engine indicator baseline prediction value at a preset future time and the confidence interval of the engine indicator baseline prediction value are used as model outputs, and the initial time series prediction model is trained to obtain a time series prediction model; Obtain target historical engine indicator sequences corresponding to each of the multiple engines, input the target historical engine indicator sequences into the time series prediction model respectively, and obtain engine indicator baseline prediction values ​​corresponding to each of the multiple engines at future target preset moments.

4. A big data cluster task monitoring method according to claim 1, characterized in that: Obtaining the abnormal classification model corresponding to the target engine to perform root cause diagnosis and determine the abnormal type of the engine indicator specifically includes: Matching a corresponding anomaly classification model based on the target engine; Collecting a target real-time engine index value of the target engine, and inputting the target real-time engine index value into the abnormality classification model corresponding to the target engine, so that the abnormality classification model outputs an abnormality type probability vector; An engine indicator abnormality type of the target engine is determined based on the abnormality type probability vector.

5. A big data cluster task monitoring method according to claim 1, characterized in that: The scoring process for each recovery strategy in the recovery strategy set based on the resource contention matrix specifically includes: Extracting the target comprehensive resource contention corresponding to the target engine from the resource contention matrix, obtaining the task type and task priority corresponding to the current cluster task, and obtaining the policy cost value corresponding to each recovery policy in the recovery policy set, wherein the recovery policy set includes at least one recovery policy for the target engine exception type; Based on the target comprehensive resource competition degree, the task type, the task priority and the strategy cost value, each recovery strategy in the recovery strategy set is scored to obtain a score value corresponding to each recovery strategy in the recovery strategy set.

6. A big data cluster task monitoring method according to claim 1, characterized in that: The executing of the optimal recovery strategy specifically includes: Determining whether the optimal recovery strategy is a multi-strategy collaborative strategy; If the optimal recovery strategy is a multi-strategy collaborative strategy, performing conflict detection on all sub-recovery strategies in the optimal recovery strategy to determine conflicting sub-recovery strategy pairs; Perform dependency analysis on all sub-recovery strategies in the optimal recovery strategy to determine the dependency relationships corresponding to all sub-recovery strategies; Based on the resource coordinator, cluster resources are monitored in real time, and corresponding resource budgets are allocated to all sub-recovery strategies in the optimal recovery strategy; Obtaining the sub-recovery strategy priority corresponding to each sub-recovery strategy in the optimal recovery strategy, and assigning all sub-recovery strategies in the optimal recovery strategy to different time windows based on the sub-recovery strategy priorities, the dependencies, and the conflicting sub-recovery strategy pairs, and sequentially executing the sub-recovery strategy corresponding to each time window based on a time window sequence; Monitor the execution status of each sub-recovery strategy in real time. If all sub-recovery strategies in the optimal recovery strategy are successfully executed, output the execution completion result. If any sub-recovery strategy in the optimal recovery strategy fails to execute, trigger the preset fallback plan.

7. A big data cluster task monitoring device, characterized in that: include: Resource competition matrix generation module, engine indicator abnormality type determination module and optimal recovery strategy determination module; The resource contention matrix generation module is configured to calculate the resource contention corresponding to each of the multiple engines based on the engine data corresponding to each of the multiple engines collected in real time, and generate a resource contention matrix based on the resource contention; The engine indicator anomaly type determination module is configured to predict the engine indicator baseline prediction values ​​corresponding to each of the multiple engines based on a pre-trained time series prediction model, and compare the real-time engine indicator values ​​corresponding to each of the multiple engines with the engine indicator baseline prediction values. If the real-time engine indicator values ​​do not meet the engine indicator baseline prediction values, an anomaly classification model corresponding to the target engine is obtained to perform root cause diagnosis and determine the engine indicator anomaly type. The optimal recovery strategy determination module is configured to determine a recovery strategy set based on the engine indicator anomaly type, and to score each recovery strategy in the recovery strategy set based on the resource contention matrix, and to determine and execute the optimal recovery strategy based on the scoring results; Among them, the resource contention matrix generation module is used to calculate the resource contention degree corresponding to each of different engine pairs based on the engine data corresponding to each of the multiple engines collected in real time, and generate a resource contention matrix based on the resource contention degree, specifically including: when executing the current cluster task, collecting the engine data corresponding to each of the multiple engines in a preset time window in real time, wherein the engine data includes engine CPU data, engine memory data and engine throughput data; combining the multiple engines in pairs to obtain multiple engine pairs, and calculating the CPU contention coefficient corresponding to each of the multiple engine pairs based on the engine CPU data, calculating the memory contention coefficient corresponding to each of the multiple engine pairs based on the engine memory data, and calculating the throughput data of the multiple engine pairs based on the engine throughput data. The target CPU contention coefficient, the target memory contention coefficient and the target network contention coefficient are obtained; any one of the multiple engines is selected as the task engine in turn, and all CPU contention coefficients, all memory contention coefficients and all network contention coefficients of the task engine in different engine pairs are obtained, and the maximum values ​​of all CPU contention coefficients, all memory contention coefficients and all network contention coefficients are obtained respectively to obtain the target CPU contention coefficient, the target memory contention coefficient and the target network contention coefficient; the target CPU contention coefficient, the target memory contention coefficient and the target network contention coefficient are weightedly fused to obtain the resource contention degree between different engine pairs, and the resource contention degree is used as the comprehensive resource contention degree corresponding to the task engine; the comprehensive resource contention degree corresponding to each of the multiple engines is integrated to determine the resource contention degree matrix.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 can be implemented.

Images