Data output timeliness measurement method and device of big data platform and storage medium

Abstract

The invention discloses a data output timeliness measurement method and device for a big data platform and a storage medium, and the method comprises the steps: obtaining the actual completion timeliness of each data processing operation in the big data platform and the corresponding reference timeliness, and calculating the individual timeliness index of each data processing operation; acquiring preset attribute information and a data dependency relationship of each data processing job; determining a dynamic factor according to preset attribute information and the data dependency relationship, and determining an aggregation weight of the data processing job in each evaluation dimension based on the dynamic factor; calculating a dimension timeliness index through the individual timeliness index and the aggregation weight, and generating a comprehensive timeliness index of the big data platform based on the dimension timeliness index; and inputting the comprehensive timeliness index into a job scheduler of the big data platform to monitor the scheduling state of the big data platform. According to the method and the device, the technical effects of prospective scheduling and resource allocation and global aging landslide caused by local delay are realized by quantifying the aging risk conduction dynamic factors of the data dependency relationship between the operations.

Description

Technical Field

[0001] This application relates to the field of game server development technology, and in particular to a method, device and storage medium for measuring the timeliness of data output on a big data platform. Background Technology

[0002] During game server development, developers typically need to write and call test code to perform functional verification, parameter adjustments, and scenario simulations. A common practice is to distribute the test code across various functional modules or adjacent locations, or to use external text documents to record test commands. However, as project versions iterate and features expand, this distributed approach to debugging code management has gradually revealed several problems.

[0003] First, test code is often scattered across functional modules, lacking a unified directory structure, making it difficult to maintain and reuse debugging logic as a whole. Inconsistent test code styles and locations introduced by different developers or at different times further increase management complexity. Second, when multiple versions of a project are developed in parallel, changes to test code in one version often cannot be automatically synchronized to other versions, resulting in inconsistent debugging logic between versions, a large amount of repetitive modification work, and an increased risk of human error.

[0004] Furthermore, because test code and production code are often physically located close to each other or coexisting, they are easily mistakenly included in the repository during code submission, leading to online functionalities and increasing unnecessary release risks. On the other hand, certain configuration parameters initialized during server startup, such as database connections and underlying service addresses, are difficult to dynamically adjust during runtime once initialized. Modifications typically require a service restart, disrupting service continuity and reducing debugging flexibility and efficiency.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a method, device and storage medium for measuring the timeliness of data output of a big data platform, which aims to solve the technical problem that existing big data platforms based on static weight scheduling cannot prevent the cascading transmission of job delays.

[0007] To achieve the above objectives, this application proposes a method for measuring the timeliness of data output on a big data platform. The method includes: Obtain the actual completion time and corresponding baseline time for each data processing job in the big data platform; Based on the actual completion time and the benchmark time, calculate the individual timeliness index for each of the data processing jobs; Obtain the preset attribute information and data dependencies of each of the data processing jobs; Dynamic factors are determined based on the preset attribute information and the data dependency relationship, and the aggregation weight of the data processing job in each evaluation dimension is determined based on the dynamic factors. The individual timeliness indicators and the aggregate weights are used to calculate the dimensional timeliness indicators for each evaluation dimension, and the comprehensive timeliness indicators of the big data platform are generated based on the dimensional timeliness indicators. The comprehensive timeliness index is input into the job scheduler of the big data platform. The job scheduler monitors the scheduling status of the big data platform. When the comprehensive timeliness index is detected to be lower than the preset scheduling intervention threshold, the job scheduler is triggered to dynamically adjust the scheduling priority of the data processing job and allocate computing resources to the data processing job based on the scheduling priority.

[0008] In one embodiment, the step of determining dynamic factors based on the preset attribute information and the data dependencies, and determining the aggregation weight of the data processing job in each evaluation dimension based on the dynamic factors, includes: Obtain job information corresponding to each data processing job and its respective service level agreement; Based on the preset impact weight coefficients for each service level, calculate the static impact value of each data processing job based on the job information; Based on the data dependencies, a directed acyclic graph is constructed for each of the data processing jobs, and the downstream dependent jobs of each of the data processing jobs are identified through the directed acyclic graph. Calculate the timeliness risk transmission coefficient of the data processing operation based on the number of downstream dependent operations, static impact value, and dependency edge attributes; The combined impact value of the data processing operation is obtained by fusing the static impact value and the time-sensitive risk transmission coefficient. Based on the comprehensive impact value, the aggregate weight of the data processing operation in each evaluation dimension is determined.

[0009] In one embodiment, the step of calculating the timeliness risk transmission coefficient of the data processing job based on the number of downstream dependent jobs, static impact value, and dependency edge attributes includes: Assign a weight to each dependent edge in the directed acyclic graph, the weight being determined based on the sensitivity of the downstream operation to the output data of the upstream operation or the data processing time of the dependent edge. Starting with the data processing job, all downstream dependent jobs are traversed along the dependency edge, and the product of the static impact value of each downstream dependent job and its corresponding dependency edge weight is accumulated to obtain the total value of the transmitted impact. The total value of the transmission impact is normalized to obtain the time-sensitive risk transmission coefficient.

[0010] In one embodiment, the step of obtaining preset attribute information for each data processing job and determining the aggregation weight of the data processing job in each evaluation dimension based on the preset attribute information includes: Obtain important system information regarding the different Service Level Agreements (SLAs) affected by each data processing operation; Calculate the comprehensive impact value of each data processing job on the important system based on the preset impact weight coefficients of different SLA levels; Based on the comprehensive impact value of each data processing task, its relative weight within its respective SLA level is determined, which serves as its aggregate weight in the corresponding evaluation dimension.

[0011] In one embodiment, the step of calculating the comprehensive impact value of each data processing job on the important system based on preset impact weighting coefficients for different SLA levels includes: Obtain the number of important systems that the data processing operation affects at different service level protocol levels, and the weighting coefficients corresponding to each service level protocol level; The number of important systems for each service level agreement level is multiplied by the corresponding weight coefficient, and all the multiplication results are summed to obtain the comprehensive impact value of the data processing operation.

[0012] In one embodiment, the step of calculating the dimensional timeliness index of each evaluation dimension using the individual timeliness index and the aggregate weight includes: The weighted index is obtained by multiplying the individual timeliness indicators of all data processing jobs within an evaluation dimension by their corresponding aggregate weights. Sum all the weighted indicators to obtain the timeliness index of this evaluation dimension.

[0013] In one embodiment, the data output timeliness measurement method of the big data platform further includes: Based on the aforementioned comprehensive timeliness indicators, analyze the changing trend of data output timeliness of the big data platform; If the comprehensive timeliness index is below the first threshold or continues to decline for multiple consecutive periods, it is determined that the timeliness of the data output of the big data platform is deteriorating. If the comprehensive timeliness index mentioned above is higher than the second threshold or continues to rise for multiple consecutive periods, it is determined that the timeliness of the platform's data output is showing an improving trend. Wherein, the first threshold is a first percentage value of the preset scheduling intervention threshold, the second threshold is a second percentage value of the preset scheduling intervention threshold, and the first threshold is less than the second threshold.

[0014] In one embodiment, after determining that the data output timeliness of the big data platform data shows a deteriorating trend when the comprehensive timeliness index is below a first threshold for multiple consecutive periods or continues to decline, the method further includes: Extract dimension timeliness indicators of multiple preset evaluation dimensions from the comprehensive timeliness indicator data, and locate key evaluation dimensions among the dimension timeliness indicators of multiple preset evaluation dimensions. Based on the attributes of the key evaluation dimensions, perform operational bottleneck analysis or attribute optimization effects.

[0015] In addition, to achieve the above objectives, this application also proposes a data output timeliness measurement device for a big data platform. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the data output timeliness measurement method for the big data platform as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data output timeliness measurement method of the big data platform as described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: The technical solution of this application involves obtaining the actual completion time and corresponding benchmark time of each data processing job in a big data platform; calculating the individual timeliness index of each data processing job based on the actual completion time and the benchmark timeliness; obtaining the preset attribute information and data dependency relationship of each data processing job; determining dynamic factors based on the preset attribute information and the data dependency relationship, and determining the aggregation weight of the data processing job in each evaluation dimension based on the dynamic factors; calculating the dimension timeliness index of each evaluation dimension through the individual timeliness index and the aggregation weight, and generating the comprehensive timeliness index of the big data platform based on the dimension timeliness index; inputting the comprehensive timeliness index into the job scheduler of the big data platform, and monitoring the scheduling status of the big data platform through the job scheduler, wherein, when the comprehensive timeliness index is detected to be lower than the preset scheduling intervention threshold, the job scheduler is triggered to dynamically adjust the scheduling priority of the data processing job, and the computing resource quota of the data processing job is based on the scheduling priority.

[0018] This application achieves the technical effect of proactive scheduling and resource allocation by quantifying the timeliness risk transmission factors of data dependencies between operations, thereby avoiding the global timeliness decline caused by local delays. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the data output timeliness measurement method of the big data platform of this application. Figure 2 This is a detailed step diagram based on step S40 in the first embodiment; Figure 3 This is a detailed step diagram based on step S30 in the first embodiment; Figure 4 This is a detailed step diagram based on step S50 in the first embodiment; Figure 5 This is a schematic diagram of the hardware operating environment involved in the data output timeliness measurement method of the big data platform in this application embodiment.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of this application embodiment is as follows: Obtain the actual completion time and corresponding baseline time of each data processing job in the big data platform; calculate the individual timeliness index of each data processing job based on the actual completion time and the baseline timeliness; obtain the preset attribute information and data dependency relationship of each data processing job; determine dynamic factors according to the preset attribute information and the data dependency relationship, and determine the aggregation weight of the data processing job in each evaluation dimension based on the dynamic factors; calculate the dimension timeliness index of each evaluation dimension through the individual timeliness index and the aggregation weight, and generate the comprehensive timeliness index of the big data platform based on the dimension timeliness index; input the comprehensive timeliness index to the job scheduler of the big data platform, and monitor the scheduling status of the big data platform through the job scheduler, wherein, when the comprehensive timeliness index is detected to be lower than a preset scheduling intervention threshold, the job scheduler is triggered to dynamically adjust the scheduling priority of the data processing job, and the computing resource quota of the data processing job is based on the scheduling priority.

[0026] In existing technologies, big data platform job scheduling relies on static attributes (such as fixed SLA levels or business priorities) to assign weights, which cannot quantitatively assess and proactively intervene in the risk of delay chain transmission caused by complex data dependencies between jobs. This leads to the technical problem that local job timeouts may cause a decline in the timeliness of global data output.

[0027] This application provides a solution that achieves proactive scheduling and resource allocation by quantifying the dynamic factors of timeliness risk transmission in data dependencies between operations, thereby avoiding the technical effect of local delays causing a global timeliness decline.

[0028] Based on this, embodiments of this application provide a method for measuring the timeliness of data output on a big data platform, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data output timeliness measurement method for the big data platform according to this application. In this embodiment, the data output timeliness measurement method for the big data platform includes steps S10 to S60: Step S10: Obtain the actual completion time and corresponding baseline time of each data processing job in the big data platform; In this embodiment, within the big data platform environment, two types of key time-series data are collected: the actual completion time of each data processing job within the monitoring period, and the corresponding, scientifically set baseline completion time. The big data platform typically consists of a job scheduling system, a metadata management system, a distributed file system, and a computing engine.

[0029] Obtaining the actual completion time relies on in-depth analysis of the job scheduling system's execution logs. Specifically, by deploying a log collection agent or directly calling the scheduling system's open API, the "end" status event of each data processing job instance is captured in real-time or periodically, and a precise timestamp is extracted from it; this timestamp is the actual completion time. To ensure data integrity and accuracy, the collection process needs to include merging processing logic for multiple instances of the data processing job (such as retries and make-up runs), typically using the completion time of the finally successful data processing job instance as the standard.

[0030] Furthermore, the setting of the baseline timeliness is a dynamic process involving multiple parallel strategies, based on the job's historical behavior, business contracts, and platform context. In specific implementation, the static attributes of the data processing job, such as job identifier, project affiliation, and service level agreement (SLA) level, are first obtained from the metadata management system. For existing jobs with a long operational history, the baseline timeliness is determined using statistical methods. Specifically, within a specified historical baseline period (such as the previous stable operating quarter or year), the average or median completion time of all successful instances of the data processing job is calculated. For jobs with explicit business timeliness requirements, the baseline timeliness is directly taken from the target completion time specified in their SLA.

[0031] For new jobs or data processing jobs with insufficient historical data, the baseline lead time is generated through an estimation model. The core logic of the estimation model is to identify the position of the data processing job in the data dependency graph. If the new job is determined to be a leaf node in a dependency link, the system traces all its upstream jobs, finds the time window in which the upstream jobs were completed within the historical baseline period, and calculates the average completion time of leaf jobs with similar resource characteristics to the new job that were completed within the event window as an estimate.

[0032] In addition, a mechanism for reviewing and adjusting baseline timeliness needs to be maintained. For example, when a major event such as hardware expansion, core data model reconstruction, or change in collection frequency is detected on the platform, a batch recalculation of the baseline timeliness for the affected jobs is triggered. All actual completion times and baseline timeliness obtained from collection and calculation are stored in a dedicated table in a time-series database or relational database after consistency verification, and an index is established to associate them with job metadata, providing high-quality, associative input data for subsequent indicator calculations.

[0033] Step S20: Calculate the individual timeliness index for each data processing job based on the actual completion timeliness and the baseline timeliness. In this embodiment, the individual timeliness index is defined as the relative rate of change of the actual completion time of a single task within a specific reporting period compared to its own baseline timeliness. The calculation formula is: Individual Timeliness Index = (Actual Completion Timeliness in Reporting Period / Baseline Timeliness) × 100%. This calculation aims to eliminate the influence of the inherent cyclical characteristics of the task, transforming absolute time into a proportional value representing changes in efficiency. An index equal to 100% means the task was completed on time; greater than 100% indicates a delay compared to the baseline; and less than 100% indicates an improvement in timeliness.

[0034] In practice, the computing engine retrieves records of all active data processing jobs from the constructed time-sensitive data storage, categorized by reporting period time range (e.g., daily, weekly). For each data processing job, the actual completion time of all successful instances of the job within the reporting period is aggregated (typically using an average value to smooth out individual fluctuations), and then divided by the currently valid baseline time of the data processing job. When processing large-scale jobs, this computation process is efficiently completed in batch processing using a distributed computing framework (such as Spark).

[0035] The key technical complexity of the above calculation process lies in the robust handling of anomalies and edge cases. For example, for a data processing job that fails on its first run due to an external dependency failure during the reporting period but subsequently succeeds on a rerun, its multiple completion times may contain outliers. Data cleaning rules need to be introduced, such as using an outlier detection method based on box plots, or using a truncated mean when calculating the average actual completion time for the reporting period. Another scenario is that the baseline completion time of the data processing job is dynamically updated due to changes in business requirements.

[0036] At this point, time alignment must be ensured when calculating individual metrics. This means comparing the actual completion time of the reporting period with the baseline completion time version effective during the same period, requiring the data model to maintain the version history of the baseline completion time. The calculated individual completion time metrics, along with metadata such as job identifier, reporting period date, and the number of actual completion time samples used for calculation, are stored together to form a fine-grained basic metric layer. This data metric layer not only serves subsequent aggregation but can also be used for root cause drill-down analysis of specific jobs.

[0037] Step S30: Obtain the preset attribute information and data dependency relationships of each data processing job; In this embodiment, a comprehensive digital profile is constructed for each data processing job in the big data platform. This digital profile includes the static business attributes and dynamic topological relationships of the data processing job. These static attributes and dynamic topological relationships form the context for meaningful aggregation and in-depth analysis of individual metrics. The preset attribute information is a structured set, with typical fields including, but not limited to, a unique job identifier, job name, business line or data domain, responsible team, job type (e.g., Hive SQL, Spark JAR, Flink task), configured service level agreement (e.g., L1 to L5), and a list of business systems it serves or affects. The impact information on business systems needs to be further correlated with the key classifications of these systems.

[0038] The data dependency relationship describes the logical sequence of data processing jobs formed by data output and consumption, and is usually represented as a directed acyclic graph. The two sources of the directed acyclic graph specifically include declarative dependency and inferred lineage. The declarative dependency is the list of upstream jobs explicitly configured by the job developer in the scheduling system. The inferred lineage is obtained by parsing the SQL statements or program logic in the job script, automatically identifying its input and output tables, and then deriving the inter-job dependencies through the table-level lineage relationship chain.

[0039] During implementation, information is gathered through multiple integration methods. First, the API of the metadata management system is called to retrieve the preset attribute information of data processing jobs in batches and cache it in a local dimension table. Second, the task definitions of data processing jobs are pulled from the configuration library of the job scheduling system (such as Airflow or DolphinScheduler), and explicit task dependencies are parsed. Simultaneously, a lineage analysis engine is started to periodically perform static parsing on SQL scripts and JAR packages submitted to the platform, or collect lineage logs generated by computing engines (such as Hive or Spark) during runtime, constructing and continuously updating table-level and job-level data lineage graphs. Finally, declarative dependencies and inferred lineages are compared and merged to generate an authoritative job dependency graph, which is stored in a graph database or a relational database with recursive query capabilities. This process ensures the accuracy and timeliness of dependencies, reflecting topological changes brought about by changes in job logic. The complete attribute and dependency data obtained provides all the necessary inputs for the next step of calculating dynamic factors reflecting the importance and scope of influence of jobs.

[0040] Step S40: Determine dynamic factors based on the preset attribute information and the data dependency relationship, and determine the aggregation weight of the data processing job in each evaluation dimension based on the dynamic factors; In this embodiment, a quantitative model is established to transform information such as the business importance, scope of influence, and topological location of data processing jobs into a scalar value called a "dynamic factor". Based on the scalar value, appropriate aggregation weights are assigned to each data processing job under different analysis dimensions, thereby ensuring that subsequent aggregation indicators can reflect the actual impact of different jobs in a weighted manner.

[0041] The dynamic factor calculation model comprehensively considers the breadth and depth of the direct impact of data processing operations on business systems. The formula is: Dynamic Factor = ∑ (Number of business systems affected by the operation at Service Level Agreement (SLA) level Li × Weight coefficient of level Li). The weight coefficient is typically predefined based on business consensus; for example, L1 (highest level) has a weight of 10, L2 has 8, and so on, with L5 having a weight of 1. In practice, based on the obtained preset attribute information, particularly the "list of affected business systems" and their corresponding SLA levels, statistics and weighted summation are performed. This calculation model implies that if a data processing operation affects multiple high-level critical business systems, its dynamic factor value will significantly increase.

[0042] After determining the dynamic factors, the aggregate weights of data processing jobs under multiple evaluation dimensions are calculated. Common evaluation dimensions include: grouping by SLA level, grouping by data region (or business domain), and grouping by job type. For a specific evaluation dimension (e.g., "L1 level job"), first, all job sets belonging to this dimension are found. Then, the proportion of the dynamic factor of each data processing job in the set to the sum of the dynamic factors of all jobs in the set is calculated. This proportion is the aggregate weight of the data processing job under this dimension, expressed by the formula: the weight of job j under dimension D w_{j,D} = F_j / (∑_{m ∈ D} F_m), where F represents the dynamic factor.

[0043] The above calculation process requires traversing all preset evaluation dimensions. Specifically, it is completed through distributed aggregation calculation based on multidimensional grouping (GROUP BY). First, at the data processing job level, its dynamic factors and all dimension labels are associated. Then, it is grouped by each dimension, and weights are calculated within each group. The calculation results are persisted to form a weight mapping table, recording the corresponding weight value of each data processing job-dimensional pair. The aggregated weight is a normalized influence measure, ensuring that in subsequent weighted averaging, data processing jobs with higher importance contribute more to the overall index, thus making the index more reflective of the platform's overall health and business assurance.

[0044] Step S50: Calculate the dimensional timeliness index of each evaluation dimension using the individual timeliness index and the aggregate weight, and generate the comprehensive timeliness index of the big data platform based on the dimensional timeliness index. In this embodiment, the final decision-making index is generated by the gradual aggregation of measurement indicators from micro to macro. The calculation of this generation process is divided into two main steps: first, the dimensional timeliness indicators of each evaluation dimension are calculated, and then the platform-level comprehensive timeliness indicator is synthesized based on the dimensional timeliness indicators.

[0045] Specifically, the first-level calculation is performed independently for each preset evaluation dimension, retrieving the individual timeliness indicators for the reporting period and the aggregate weights of each data processing job under this evaluation dimension from storage. For a specific dimension D (e.g., "Data Zone A"), its dimension timeliness indicator I_D is calculated as a weighted arithmetic mean: I_D = ∑{j∈D} (Individual Timeliness Indicator_j × Aggregate Weight{j, D}). This calculation operation is performed in parallel for each evaluation dimension, producing intermediate results such as "L1 Level Timeliness Index", "Data Zone A Timeliness Index", and "Spark Job Type Timeliness Index". These dimension indicators reveal the distribution of timeliness issues across different business partitions, task types, or priority levels, aiding in the initial identification of problems.

[0046] After obtaining the timeliness indicators for all key dimensions, the second-level composite calculation is performed to generate the comprehensive timeliness indicator representing the overall performance of the platform. This composite calculation is not a simple averaging, but rather assigns different strategic importance weights (dimensional weights) to different dimensions based on management strategies. For example, timeliness assurance for high SLA-level operations is usually more important than for low-level operations; therefore, during composite calculation, the index of an L1-level dimension may be assigned a higher dimension weight than that of an L5-level dimension. The formula for calculating the comprehensive platform timeliness indicator I_platform is: I_platform = ∑_{D ∈ All Dimensions} (Dimension Timeliness Indicator_D × Dimension Weight_W_D). The sum of the dimension weights is 1, and they are typically set and maintained in the configuration library by the platform governance committee based on business objectives.

[0047] In practice, the entire two-layer aggregation process can be achieved through a pre-arranged computation task. This task first loads individual indicators and job-dimension weight tables, performing a first round of grouped weighted aggregation; then it loads the dimension weight configurations and performs a second round of linear combination. The calculation results, including the timeliness indicators for each dimension and the final comprehensive timeliness indicator, are stored and indexed along with historical values. The system can also simultaneously calculate the month-on-month and year-on-year growth rates of key indicators, for example: Comprehensive Timeliness Index Growth Rate = [(Current Period I_platform / Previous Period I_platform) - 1] × 100%. This hierarchical weighted aggregation system provides both macro-trend judgment and retains the ability to drill down for analysis, forming the core data support for the timeliness measurement dashboard.

[0048] Step S60: Input the comprehensive timeliness index into the job scheduler of the big data platform, and monitor the scheduling status of the big data platform through the job scheduler. When the comprehensive timeliness index is detected to be lower than the preset scheduling intervention threshold, the job scheduler is triggered to dynamically adjust the scheduling priority of the data processing job, and the computing resource quota of the data processing job is allocated based on the scheduling priority.

[0049] In this embodiment, an automated closed loop from "measurement" to "regulation" is constructed, making the timeliness index not only a monitoring indicator but also a direct optimization driving factor. The core of this is to feed back the comprehensive timeliness index, along with related dimensional timeliness indicators, to the intelligent decision-making module of the job scheduler in real-time or near real-time, thereby enabling dynamic adjustments to the platform's scheduling strategy and resource allocation.

[0050] Specifically, after the comprehensive timeliness indicator is calculated, it is pushed to the scheduler's monitoring agent via a message queue (such as Kafka) or direct API call. The scheduler internally maintains a status dashboard to continuously track the indicator and its historical trends. Based on business tolerance and SLA targets, multi-level scheduling intervention thresholds are pre-set, such as yellow warning thresholds and red intervention thresholds.

[0051] When the monitoring module detects that the comprehensive timeliness index is consistently below the red intervention threshold (e.g., below 95% for three consecutive calculation cycles), it automatically triggers the intervention process. The intervention logic first assists in root cause localization: the scheduler analyzes the timeliness indexes of each dimension to identify which SLA level and which data area has experienced the most severe index decline. Next, for jobs within the problem dimension, the scheduler combines its real-time individual timeliness index (whether there is a severe delay) with its calculated dynamic factor (business importance) to recalculate the emergency scheduling priority. This calculation rule typically dictates that jobs with more severe delays and higher dynamic factors receive a greater temporary priority increase.

[0052] Then, the scheduler performs dynamic adjustments. Specifically, these dynamic adjustments include adjusting the scheduling order of job queues, inserting jobs with higher urgency priority into earlier positions; and coordinating with resource managers (such as YARN or Kubernetes) to dynamically adjust the computing resource quotas of different job queues or pools based on the new priority ratios. For example, increasing the number of containers or CPU cores in queues containing high-priority jobs while appropriately reducing the resource quotas for low-priority jobs to ensure the timeliness of critical jobs. The entire adjustment process can be configured with incremental and gradual strategies to avoid drastic resource fluctuations.

[0053] After the adjustments take effect, the scheduler continues to monitor the timeliness metrics of subsequent cycles. If the metrics recover and stabilize above the threshold, the temporary priority and resource adjustments can be gradually rolled back to restore the normal scheduling strategy. This closed-loop mechanism implements effect-based feedback control, enabling the big data platform to adapt to load changes and anomalies, continuously optimizing global resource efficiency while meeting high timeliness requirements. This process relies on the scheduler's open programmable interface, fine-grained resource management capabilities, and stable, low-latency data integration with the metrics system.

[0054] Furthermore, the data output timeliness measurement method of the big data platform also includes: Based on the aforementioned comprehensive timeliness indicators, analyze the changing trend of data output timeliness of the big data platform; If the comprehensive timeliness index is below the first threshold or continues to decline for multiple consecutive periods, it is determined that the timeliness of the data output of the big data platform is deteriorating. If the comprehensive timeliness index mentioned above is higher than the second threshold or continues to rise for multiple consecutive periods, it is determined that the timeliness of the platform's data output is showing an improving trend. Wherein, the first threshold is a first percentage value of the preset scheduling intervention threshold, the second threshold is a second percentage value of the preset scheduling intervention threshold, and the first threshold is less than the second threshold.

[0055] In this embodiment, based on the calculated comprehensive timeliness index, automated trend analysis and status determination of the comprehensive timeliness index are realized, thereby transforming the quantitative measurement results into qualitative judgments and early warnings of the platform's operational status. The latest calculated comprehensive timeliness index is periodically (e.g., daily or weekly) and stored in a historical time series database in chronological order to form a continuous trend line.

[0056] Meanwhile, load the preset scheduling intervention threshold Tintervene, which is usually a key level value set according to the business tolerance bottom line (for example, the comprehensive timeliness index is 98%). Based on the preset scheduling intervention threshold Tintervene, two derivative thresholds are automatically calculated: the first threshold Tlow and the second threshold Thigh. Specifically, Tlow = Tintervene × P1%, Thigh = Tintervene × P2%, where P1 and P2 are configurable percentage parameters, and P1 < 100 < P2. For example, if Tintervene = 98%, P1 = 95, and P2 = 105 are set, then Tlow = 93.1% and Thigh = 102.9%. Tlow is defined as a warning line stricter than the intervention line, while Thigh is the baseline for a good state.

[0057] Perform a sliding window analysis on the historical trend line (the window size is configurable, such as 3 consecutive periods or 5 consecutive periods). The determination logic is as follows: Determine the deteriorating trend: If it is detected that the values of the comprehensive timeliness index for multiple consecutive periods (meeting the preset number of periods N) are all lower than Tlow, or although some index values are higher than Tlow, but the overall shows a statistically significant continuous downward trend (for example, the linear regression slope is less than -0.5% per period), then it is automatically determined that the data output timeliness of the big data platform shows a deteriorating trend, triggering a high-level warning and recording it in the platform operation status report.

[0058] Determine the optimizing trend: On the contrary, if it is detected that the values of the comprehensive timeliness index for multiple consecutive periods are all higher than Thigh, or the overall shows a statistically significant continuous upward trend (for example, the linear regression slope is greater than +0.5% per period), then it is determined that the data output timeliness of the platform shows an optimizing trend, as a positive feedback on the effectiveness of platform governance.

[0059] By dynamically associating the first threshold and the second threshold with the preset scheduling intervention threshold, this method ensures the internal consistency of the warning system and the management action line. The determination of the deteriorating trend is earlier than the threshold breakthrough that may trigger scheduling intervention, providing an earlier buffer period for the administrator to conduct root cause analysis; while the determination of the optimizing trend provides an objective basis for verifying the effectiveness of governance measures. The entire analysis process can be automatically executed by a background scheduled task, and the determination results are pushed to the monitoring dashboard and the management portal.

[0060] Specifically, after the step of determining that the data output timeliness of the big data platform shows a deteriorating trend if the comprehensive timeliness index is lower than the first threshold or continuously declines for multiple consecutive periods, the following steps are further included: Extract dimension timeliness indicators of multiple preset evaluation dimensions from the comprehensive timeliness indicator data, and locate key evaluation dimensions among the dimension timeliness indicators of multiple preset evaluation dimensions. Based on the attributes of the key evaluation dimensions, perform operational bottleneck analysis or attribute optimization effects.

[0061] In this embodiment, the process is initiated after determining that the platform's timeliness is deteriorating. It quickly pinpoints the macro-level deterioration signal to a specific responsible dimension or work group causing the problem, thereby initiating precise root cause analysis and optimization actions, achieving a closed loop from "problem discovery" to "problem localization." Upon determining that the timeliness of the big data platform's data output is deteriorating, the analysis steps are immediately executed: First, extract and locate key evaluation dimensions. Extract the timeliness indicators of multiple preset evaluation dimensions corresponding to the current assessment period from storage. These dimensions typically include, but are not limited to: different SLA levels (L1-L5), different data regions (or business domains), different job types (e.g., Hive / Spark / Flink), and different responsible teams. Compare the current indicators of these dimensions with their historical baselines (e.g., the average of the previous 3 months) to calculate their relative change (degree of deterioration).

[0062] Key evaluation dimensions are located using sorting and filtering algorithms. The location logic can be based on a combination of one or more strategies, including the maximum decline strategy, the influence weight strategy, and the threshold breakthrough strategy. The maximum decline strategy directly selects the top K dimensions whose timeliness indicators have fallen the most compared to the historical baseline. The influence weight strategy calculates the contribution of each dimension's deterioration to the overall decline in the comprehensive timeliness indicator. This calculation needs to consider the dimension's weight in the synthesis of the comprehensive indicator (dimensional weight). Contribution = (Dimension Weight × Dimension Indicator Change). The dimension with the highest contribution is selected. The threshold breakthrough strategy identifies dimensions whose indicator values ​​have fallen below their own preset health threshold. The identified key evaluation dimensions are the main contradictions causing the overall deterioration of the platform's timeliness. For example, the "L1 level operation timeliness index" or the "transaction data area timeliness index" might be identified as key dimensions.

[0063] Secondly, based on the attributes of key evaluation dimensions, bottleneck analysis or optimization effect attribution is performed. The corresponding analysis mode is initiated according to the specific attributes of the identified key evaluation dimensions. These analysis modes include bottleneck analysis and optimization effect attribution. Bottleneck analysis is performed when the key dimension points to a technical or resource category (such as a specific job type or a specific data area), aiming to identify operational bottlenecks. For example, if the key dimension is "Spark job," the system will drill down further to analyze operational metrics such as resource utilization (CPU / memory), queuing time, failure rate, and data skew, comparing them with historical normal periods to identify specific bottlenecks such as insufficient resources, decreased code efficiency, or slower external dependencies. Optimization effect attribution, on the other hand, is performed when the key dimension points to a management or business category (such as a specific responsible team or a specific SLA level), aiming to assess the impact of recent changes or optimization measures. The system will connect to the change management system to check for recent events such as job logic changes, data model reconstruction, or scheduling strategy adjustments under this dimension. By comparing the timeliness indicators before and after the changes, the effectiveness of the optimization measures is analyzed to determine whether the deterioration is an unexpected negative effect.

[0064] The analysis results will generate a diagnostic report, clearly identifying the core dimensions, possible causes, or related events leading to the deterioration in timeliness. This report can be directly pushed to the relevant data manager or operations team, providing precise data support and action directions for subsequent targeted optimization measures (such as scheduling intervention in step S60, code optimization, and resource expansion). Through this process, platform administrators can quickly focus from macro-level warnings to micro-level actionable entry points, greatly improving the efficiency of problem-solving.

[0065] Furthermore, you can also view Figure 2 , Figure 2 This is a detailed step diagram based on step S40 in the first embodiment. Figure 2 The steps of determining dynamic factors based on the preset attribute information and the data dependency relationship, and determining the aggregation weight of the data processing job in each evaluation dimension based on the dynamic factors, include S41~46: Step S41: Obtain job information corresponding to each data processing job under each service level agreement; Step S42: Calculate the static impact value of each data processing job based on the job information according to the preset impact weight coefficients for each service level; Step S43: Construct a directed acyclic graph of each data processing job based on the data dependency relationship, and identify the downstream dependent jobs of each data processing job through the directed acyclic graph; Step S44: Calculate the timeliness risk transmission coefficient of the data processing job based on the number of downstream dependent jobs, static impact value, and dependency edge attributes. Step S45: The static impact value and the timeliness risk transmission coefficient are fused to obtain the comprehensive impact value of the data processing operation; Step S46: Based on the comprehensive impact value, determine the aggregate weight of the data processing operation in each evaluation dimension.

[0066] In this embodiment, a hierarchical operation influence measurement model is constructed. This model not only considers the direct static impact of operations on the business system, but also incorporates the potential time-sensitive risk transmission effect brought about by the topological position of the operation in the data-dependent network. Finally, a comprehensive influence value is synthesized as the basis for calculating the aggregate weight.

[0067] Obtain job information corresponding to each data processing job and its respective Service Level Agreement (SLA). Extract detailed attribute records for each data processing job from the metadata management system. The core of this process is to obtain the job information, which clarifies the support relationship between the job and the business system and its corresponding SLA level. Specifically, for each data processing job, a structured list needs to be obtained, where each record contains: the "business system identifier" directly supported by the data produced by the data processing job, and the "SLA level" (e.g., L1 to L5) agreed upon by this business system and the data team. This job information typically originates from the platform's requirements management module or configuration management database. In practice, a mapping table of "job identifier - business system identifier - SLA level" is established by calling the relevant system's API or querying the configuration table, serving as direct input for subsequent calculations.

[0068] Based on the preset impact weight coefficients for each service level, the static impact value of each data processing job based on the job information is calculated. A preset SLA level weight coefficient mapping table is used to assign a quantified weight coefficient to each SLA level (e.g., L1:10, L2:8, L3:6, L4:4, L5:1). For any job, the static impact value (S_Impact) is calculated as follows: S_Impact = Σ (Number of business systems belonging to SLA level i affected by the job × Weight coefficient of level i). During the calculation, each piece of "job information" of the acquired data processing job is traversed, the number of affected business systems is counted by grouping according to SLA level, and then multiplied by the corresponding preset weight coefficient before summing.

[0069] Based on the data dependencies, a directed acyclic graph (DAG) is constructed for each data processing job, and the downstream dependent jobs of each data processing job are identified through the DAG. Using the dependencies obtained from scheduling configuration and lineage analysis, a platform-level DAG is constructed with jobs as nodes and data flow dependencies as edges. The DAG is stored in a graph database or in storage with good adjacency query performance. For each job node in the graph, by executing a graph traversal algorithm (such as breadth-first search), starting from this node and traversing along the outgoing edges, the set of all downstream job nodes that directly or indirectly depend on its output data can be identified; these are the downstream dependent jobs.

[0070] Based on the number of downstream dependent jobs, their static impact values, and dependency edge attributes, the timeliness risk propagation coefficient of the data processing job is calculated. The timeliness risk propagation coefficient (R_Propagate) measures the potential impact on downstream jobs when the current job experiences a delay. A basic calculation model is: R_Propagate = Σ(Static impact value of downstream job j × Propagation attenuation factor_{current job->j}). The propagation attenuation factor can be set according to the attributes of the dependency edge (such as the urgency of data exchange or whether it is a critical path). In a simple case, it can be set to 1 (full propagation) or attenuated based on dependency depth. In another simplified method, R_Propagate = Total number of downstream dependent jobs × Average static impact value adjustment coefficient. In specific implementation, for each data processing job, its downstream job set is queried, and the static impact value of the downstream jobs is obtained. Then, the timeliness risk propagation coefficient is calculated according to the configured propagation model. The higher the value of the timeliness risk propagation coefficient, the more critical the data processing job is in the dependency network.

[0071] The static impact value and the time-sensitive risk transmission coefficient are fused to obtain the comprehensive impact value of the data processing operation. Direct business importance and network topology importance are combined into a unified metric, namely the comprehensive impact value (F_Composite). Specifically, a common linear fusion formula is: F_Composite = α×S_Impact + β×R_Propagate. Here, α and β are configurable fusion coefficients, satisfying α+β=1, used to adjust the relative weight of static impact and risk transmission in the final evaluation. Another non-linear fusion method can use a multiplicative model to amplify the impact of high-risk, high-value operations, for example: F_Composite = S_Impact × log(1+R_Propagate). Based on the preset fusion strategy and coefficients, the static impact value and time-sensitive risk transmission coefficient of each data processing operation are calculated to obtain the final comprehensive impact value.

[0072] Based on the comprehensive impact value, the aggregation weights of the data processing jobs in each evaluation dimension are determined. According to the calculated comprehensive impact value, appropriate weights are assigned to the data processing jobs under different analysis dimensions for subsequent weighted aggregation calculations. For any predefined evaluation dimension D (e.g., "all L1 jobs", "jobs within data region X"), first, the set of all data processing jobs belonging to dimension D is determined. Then, for each job j in the set, its aggregation weight (w_{j, D}) under dimension D is calculated as: the proportion of the comprehensive impact value (F_Composite_j) of the data processing job to the sum of the comprehensive impact values ​​of all jobs within the dimension. That is: w_{j, D} = F_Composite_j / (Σ_{k∈D} F_Composite_k). This calculation method ensures that within the same dimension, the sum of the weights of all data processing jobs is 1, and the weight of a data processing job is proportional to its comprehensive impact value. Through a single calculation of dimension-grouped aggregation, a weight mapping table for all job-dimension combinations can be generated.

[0073] Based on the above Figure 2 The content described in step S44 is further refined, specifically the step of calculating the timeliness risk transmission coefficient of the data processing operation based on the number of downstream dependent operations, static impact value, and dependency edge attributes, including steps S44-1 to S44-3: Step S44-1: Assign a weight to each dependent edge in the directed acyclic graph. The weight is determined based on the sensitivity of the downstream operation to the output data of the upstream operation or the data processing time of the dependent edge. Step S44-2: Starting from the data processing job, traverse all downstream dependent jobs along the dependent edge, and accumulate the product of the static influence value of each downstream dependent job and its corresponding dependent edge weight to obtain the total value of the transmission influence. Step S44-3: Normalize the total value of the transmission impact to obtain the time-sensitive risk transmission coefficient.

[0074] In this embodiment, the process of calculating the timeliness risk transmission coefficient is further refined and specific operationally defined, establishing a refined quantitative method based on a weighted network transmission model. This method not only considers the number and static impact value of downstream dependent operations but also characterizes the differences in sensitivity and urgency of different links in the data flow chain by assigning different weights to dependent edges, thereby more accurately assessing the intensity of the cascading risks that may be triggered by delays in upstream operations.

[0075] First, the sensitivity of downstream jobs to upstream data is configured. Sensitivity levels are defined based on the business type and scheduling logic of the downstream jobs. For example, if a downstream job is a real-time or near-real-time application directly related to business needs, it is extremely sensitive to the latency of upstream data readiness and can be assigned a higher weight (e.g., 0.9 or 1.0); if a downstream job is an offline analytical job with a relaxed time window, its sensitivity is lower and it is assigned a lower weight (e.g., 0.3 or 0.5). This sensitivity level can be predefined as a metadata attribute of the job and automatically associated when constructing dependency edges.

[0076] Secondly, the weights are determined based on the data processing time of the dependent edges. This data processing time is not the runtime of the job itself, but specifically refers to the additional waiting and transmission time typically required for the downstream job to successfully receive, load, or prepare its output data after the upstream job completes. This transmission time can be averaged through statistical analysis of historical operation logs. The configuration rule can be: the longer the processing time, the looser the dependency relationship and the lower the urgency of timeliness transmission, thus assigning a lower weight; conversely, dependent edges with extremely short processing times (near-zero latency) indicate tight coupling between upstream and downstream jobs, and timeliness delays will be immediately transmitted, thus assigning a higher weight. The specific mapping function can be linear or piecewise, for example, mapping the average processing time to a weight range of [0.1, 1.0].

[0077] In specific implementation, a dependency edge weight configuration table or calculation function is maintained. When constructing or updating the directed acyclic graph, a weight value We is calculated and stored for each edge, where 0 ≤ We ≤ 0. <We ≤ 1。

[0078] For the source job (referred to as job A) whose risk transmission coefficient is to be calculated, a graph traversal algorithm (such as breadth-first search or depth-first search) is performed on the weighted directed acyclic graph, starting with job A, to identify the set of all its downstream dependent job nodes {D1, D2, ..., Dn}. During the traversal, for each reachable path from job A to any direct or indirect downstream job Dk, the combined effect of the dependent edge weights on that path needs to be determined. The transmission path weight from job A to downstream job Dk is calculated, and the path weight is defined as the product of the weights of all dependent edges on the path. For example, if the path is A->B->Dk, and the edge weights are W_{A->B} and W_{B->Dk}, then the path transmission weight from A to Dk is W_{A->B} × W_{B->Dk}. Then, the static impact values ​​of all downstream jobs after weighted attenuation are accumulated. The formula for calculating the total propagated impact (Total_Propagated_Impact) is: Total_Propagated_Impact = Σ_{k=1}^{n}[S_Impact(Dk) × Path_Weight(A->Dk) ], where S_Impact(Dk) is the static impact value of downstream job Dk, and Path_Weight(A->Dk) is the product of the dependency edge weights on the minimum or critical path from source job A to downstream job Dk. In actual calculations, to reduce complexity, only critical propagation paths are considered, or a minimum threshold is set for the path weights, ignoring distant downstream jobs with weak propagation effects.

[0079] A commonly used and effective normalization method is to scale based on a platform-wide baseline. In practice, after calculating the total propagation impact of all jobs, the maximum value, Max_Total_Impact, is found. For any job A, its time-related risk propagation coefficient (R_Propagate_A) is calculated as: R_Propagate_A = Total_Propagated_Impact_A / Max_Total_Impact. Through processing, the R_Propagate coefficient of all jobs will be normalized to the range [0, 1]. Specifically, a job with a value of 1 means that it has the greatest potential time-related risk propagation capability in the current platform's dependent network, meaning its delay may have the widest and strongest impact on downstream services; jobs with a value of 0 or close to 0 are usually end-leaf jobs on dependent links, and their time-related delays will generally not affect other jobs.

[0080] Another normalization method is to use the sum of the total impact values ​​of all operations as the denominator, converting the total impact value of each operation into a proportional share. However, dividing by the maximum value better preserves the differences in relative impact between operations and effectively controls the range of the coefficients, facilitating subsequent weighted fusion with static impact values.

[0081] Ultimately, the calculated time-sensitive risk transmission coefficient is a quantitative indicator of the topological importance of an operation in the network. Together with the static impact value, which characterizes the direct business importance, it constitutes the two core dimensions for assessing the overall impact of an operation.

[0082] Furthermore, you can also view Figure 3 , Figure 3 This is a detailed step diagram based on step S30 in the first embodiment. Figure 3 The steps of obtaining preset attribute information for each data processing job and determining the aggregation weight of the data processing job in each evaluation dimension based on the preset attribute information include S31~33: Step S31: Obtain important system information on different Service Level Agreements (SLAs) affected by each data processing job; Step S32: Calculate the comprehensive impact value of each data processing job on the important system based on the preset impact weight coefficients for different SLA levels; Step S33: Based on the comprehensive impact value of each data processing job, determine its relative weight within its respective SLA level, and use it as its aggregate weight in the corresponding evaluation dimension.

[0083] In this embodiment, the support relationship between each data processing job and the business system and its agreed service quality level are extracted from the platform metadata. Important system information is usually stored in the platform's "job-business system mapping table" or configuration management database. The core fields recorded include at least: the unique identifier of the data processing job, the unique identifier of the business system directly supported by the data produced by the data processing job, and the service level agreement (SLA) level reached between the business system and the data platform on the timeliness of data delivery (for example, defined as L1 to L5, where L1 is the highest guarantee level).

[0084] In practice, the above mapping relationships are obtained in batches by calling the application programming interface of the metadata management service or executing predefined database queries. For each data processing job, the output is a list or set containing several records, each record specifying a (job, business system, SLA level) triple relationship.

[0085] The acquired qualitative relationships are transformed into a quantitative scalar value that characterizes the direct business importance of the operations, namely the comprehensive impact value. To this end, a pre-defined "SLA Level Weighting Coefficient Table" is used, assigning a fixed weighting coefficient to each SLA level (e.g., L1:10, L2:8, L3:6, L4:4, L5:1). This coefficient reflects the differences in the importance of business systems at different assurance levels.

[0086] For any given data processing job J, its composite impact value (denoted as Composite_Impact_J) is calculated as follows: Composite_Impact_J = Σ (Count_SLA_i × Weight_SLA_i), where: i traverses all SLA levels (L1 to L5), Count_SLA_i represents the number of important systems with SLA level i affected by job J, and Weight_SLA_i represents the preset impact weight coefficient corresponding to SLA level i. The specific calculation process is implemented through an aggregation calculation task, loading the important system information of all jobs, grouping and counting them according to job identifier and SLA level, and then connecting them to the SLA level weight coefficient table for weighted summation. The higher the composite impact value, the more critical (high SLA level) business systems the data processing job directly supports, and the more prominent its importance at the business level.

[0087] The overall impact value of each task is converted into a normalized relative weight within its respective SLA level group. This weight will be directly used as the aggregation weight of the data processing task under the specific evaluation dimension of the "SLA level," and will be used for subsequent weighted averaging of the timeliness indicators of all tasks within that dimension. The implementation includes two phases: Dimensional Grouping: First, determine the SLA level to which each data processing job belongs. This is typically determined based on the job's core assurance requirements and recorded in the job's basic attributes. Based on this attribute, the system divides all jobs into different SLA level groups (such as L1 job group, L2 job group, etc.).

[0088] Intra-group weight calculation: For each SLA level group G, calculate the relative weight w_J of each job J within the group. The calculation formula is: w_J = Composite_Impact_J / Σ_(KG) Composite_Impact_K, where the denominator is the sum of the combined impact values ​​of all data processing jobs within group G. This calculation ensures that within the same SLA level group, the sum of the weights of all jobs is 1. The weight w_J represents the importance share of job J under the stated SLA level dimension.

[0089] After the calculation is complete, a "job-dimension-weight" mapping table is generated and persisted, which records the aggregate weight of each job under its own SLA level dimension. The weights are then used in the aggregation calculation to synthesize the timeliness index of the SLA level.

[0090] Based on the above Figure 3 The content described in step S32 is further refined, specifically the step of calculating the comprehensive impact value of each data processing operation on the important system based on the preset impact weight coefficients for different SLA levels, including steps S32-1 to S32-2: Step S32-1: Obtain the number of important systems affected by the data processing operation at different service level protocol levels, and the weight coefficient corresponding to each service level protocol level; Step S32-2: Multiply the number of important systems for each service level agreement level by the corresponding weight coefficient, and sum all the multiplication results to obtain the comprehensive impact value of the data processing operation.

[0091] In this embodiment, the number of critical systems at different SLA levels affected by each target data processing job is obtained. This data source is the critical system information table for each data processing job. Each record in the critical system information table contains two key fields: "Business System Identifier" and "SLA Level". For each data processing job to be calculated, a grouping and statistical operation is performed. In this grouping and shutdown operation, using the unique identifier as the key, all records belonging to the data processing job in the critical system information table are grouped (GROUP BY) according to the "SLA Level" field, and the number of records in each group is counted. This statistical result represents the number of critical systems affected by the data processing job at each SLA level (e.g., L1, L2, L3, L4, L5). For example, for job "Job_A", the statistical results might be: 2 L1 level systems, 1 L2 level system, 0 L3 level systems, 3 L4 level systems, and 1 L5 level system.

[0092] Secondly, preset weight coefficients corresponding to each Service Level Agreement (SLA) level are retrieved in parallel or from the configuration storage. These weight coefficients are global configuration parameters independent of specific jobs. They typically exist in the form of configuration tables or key-value pairs, for example: {L1:10, L2:8, L3:6, L4:4, L5:1}. During implementation, the mapping relationship of these weight coefficients is loaded from predefined configuration files, configuration databases, or environment variables to ensure the authority and consistency of the weight coefficients. Thus, for each data processing job to be computed, two sets of aligned data are prepared: one is a count list grouped by SLA level, and the other is a mapping table from SLA level to weight coefficients.

[0093] The number of important systems for each service level agreement level is multiplied by its corresponding weighting coefficient, and all multiplication results are summed to obtain the overall impact value of the data processing job. For the specific data processing job currently being calculated, the following calculation process is executed: Iterate through all predefined SLA levels (e.g., process L1 to L5 sequentially). For each SLA level i encountered during iteration; Extraction Count: From the SLA level count list of the data processing job, obtain the number of important systems belonging to level i that it affects, denoted as Count_i. If the data processing job does not affect any system of level i, then Count_i = 0; Extracting coefficients: Obtain the weight coefficient corresponding to level i from the global SLA level weight coefficient mapping table, denoted as Weight_i; Calculate the product: Calculate the contribution value for the specified level: Contribution_i = Count_i × Weight_i. This represents the weighted impact component generated by the operation through its support of the system at level i.

[0094] After iterating through and multiplying all predefined SLA levels, the contribution values ​​Contribution_i calculated for all levels are summed. The final formula for calculating the composite impact value (Composite_Impact) of the data processing job is: Composite_Impact = Σ_(i∈{L1,L2, L3, L4, L5}) (Count_i ×Weight_i), where Σ represents the summation operation.

[0095] In a specific example, assuming the system count affected by job "Job_A" is {L1:2, L2:1, L4:3, L5:1}, and the weight coefficient mapping is {L1:10, L2:8, L3:6, L4:4, L5:1}, then the calculation process of its comprehensive impact value is: (2×10)+ (1×8) + (0×6) + (3×4) + (1×1) = 20 + 8 + 0 + 12 + 1 = 41.

[0096] In terms of technical implementation, the above calculation process can be efficiently completed through a single map-reduce operation. This involves joining the count list of data processing jobs with the weight coefficient table, performing multiplication on each row, and then summing the results grouped by job identifier. For large-scale job sets, the calculation process can be batch-processed using a distributed computing framework. The calculation result, i.e., the comprehensive impact value of each job, will be stored and associated with the job identifier, serving as the core input for weight allocation. By clearly combining business rules (weight coefficients) with objective facts (the number of systems affected) through multiplication and addition operations, a comparable and interpretable quantitative indicator of importance is obtained.

[0097] You can also view Figure 4 , Figure 4 This is a detailed step diagram based on step S50 in the first embodiment. Figure 4 The steps for calculating the dimensional timeliness index of each evaluation dimension using the individual timeliness index and the aggregate weight include S51~52: Step S51: Multiply the individual timeliness indicators of all data processing jobs within an evaluation dimension by their corresponding aggregate weights to obtain a weighted index. Step S52: Sum all the weighted indicators to obtain the timeliness index of the evaluation dimension.

[0098] In this embodiment, the core of the first stage of the weighted aggregation calculation is the "weighting" operation. First, a specific evaluation dimension D is defined, such as "all jobs with an SLA level of L1" or "all jobs within data area A". Then, the following operations are performed: Data Acquisition and Alignment: Individual timeliness metrics for all data processing jobs belonging to dimension D are read from the metric repository. Simultaneously, the aggregated weights corresponding to the data processing jobs under dimension D are read from the weight mapping table (generated by processes such as S33 or S46 mentioned above). These aggregated weights must be normalized, meaning the sum of the weights of all jobs within dimension D is 1.

[0099] Mapping and Calculation: A one-to-one mapping is established between the individual timeliness index and its aggregate weight for each data processing job. For the j-th job in dimension D, let its individual timeliness index be \(I_j\), and its aggregate weight in the dimension be \(w_{j, D}\). The formula for calculating the weighted index \(WI_j\) of the data processing job is: \(WI_j = I_j \times w_{j, D}\). This algorithm defines the contribution of the timeliness performance of job j to the overall index of dimension D. The larger the weight \(w_{j, D}\), the more important the data processing job is in dimension D, and the greater the influence of the individual index \(I_j\) on the final dimension index.

[0100] In the second stage of the weighted aggregation calculation, namely the "aggregation" operation, after calculating the weighted index \(WI_j\) for each data processing job within dimension D, all contribution values ​​are combined into a sum. The calculation formula is: \[ \text{Dimension Timeliness Index}_D = \sum_{j \in D} WI_j \], that is, the timeliness index \(I_D\) of dimension D is equal to the sum of the weighted indices \(WI_j\) of all jobs within that dimension.

[0101] Since the weights are normalized (\( \sum_{j \in D} w_{j, D} = 1 \)), the above summation operation is essentially equivalent to calculating the weighted arithmetic mean of the timeliness indicators of all individual tasks within the dimension. \(I_D \) is a single value that comprehensively reflects the overall timeliness performance of evaluation dimension D. If \(I_D \) is greater than 100%, it indicates that the overall completion time of tasks under this dimension is delayed compared to the benchmark; if it is less than 100%, it indicates that the overall timeliness has improved.

[0102] In specific implementations, the summation operation is typically performed as part of a computational task within a single grouping and aggregation operation. Specifically, after performing a join operation to obtain an intermediate dataset containing \(WI_j\), the dataset is grouped by "dimension identifier D" (Group By), and the summation (Sum) aggregation function is applied to the \(WI_j\) field of all records within each group. The final output is the timeliness indicator \(I_D\) for that dimension. This timeliness indicator is persistently stored and can be used to generate monitoring dashboards, perform historical trend comparisons, and serve as input data for further calculations of the platform's comprehensive timeliness indicator.

[0103] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data output timeliness measurement method of the big data platform of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0104] This application provides a data output timeliness measurement device for a big data platform. The data output timeliness measurement device for a big data platform includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data output timeliness measurement method of the big data platform in the above embodiment 1.

[0105] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a data output timeliness measurement device suitable for implementing the big data platform of this application embodiments. The data output timeliness measurement device of the big data platform in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The data output timeliness measurement device of the big data platform shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0106] like Figure 5As shown, the data output timeliness measurement device of the big data platform may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data output timeliness measurement device of the big data platform. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the data output timeliness measurement device of the big data platform to exchange data with other devices wirelessly or via wired communication. Although the figure shows a data output timeliness measurement device of a big data platform with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented or possessed alternatively.

[0107] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0108] The data output timeliness measurement device for big data platforms provided in this application adopts the data output timeliness measurement method for big data platforms in the above embodiments, which can solve the technical problem that existing big data platforms based on static weight scheduling cannot prevent the cascading propagation of job delays. Compared with the prior art, the beneficial effects of the data output timeliness measurement device for big data platforms provided in this application are the same as the beneficial effects of the data output timeliness measurement method for big data platforms provided in the above embodiments, and other technical features in the data output timeliness measurement device for big data platforms are the same as the features disclosed in the method of the previous embodiment, and will not be repeated here.

[0109] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0110] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0111] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon. The computer-readable program instructions are used to execute the data output timeliness measurement method of the big data platform in the above embodiments.

[0112] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0113] The aforementioned computer-readable storage medium may be included in the data output timeliness measurement device of the big data platform; or it may exist independently and not be assembled into the data output timeliness measurement device of the big data platform.

[0114] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the data output timeliness measurement device of the big data platform, the data output timeliness measurement device of the big data platform implements the technical content of the data output timeliness measurement method embodiment of the big data platform as shown above.

[0115] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0118] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the data output timeliness measurement method of the above-described big data platform. This addresses the technical problem of existing big data platforms, which rely on static weight scheduling and cannot prevent the cascading propagation of job delays. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data output timeliness measurement method of the big data platform provided in the above embodiments, and will not be elaborated upon here.

Claims

1. A method for measuring the timeliness of data output from a big data platform, characterized in that, The data output timeliness measurement method of the big data platform includes the following steps: Obtain the actual completion time and corresponding baseline time for each data processing job in the big data platform; Based on the actual completion time and the benchmark time, calculate the individual timeliness index for each of the data processing jobs; Obtain the preset attribute information and data dependencies of each of the data processing jobs; Dynamic factors are determined based on the preset attribute information and the data dependency relationship, and the aggregation weight of the data processing job in each evaluation dimension is determined based on the dynamic factors. The individual timeliness indicators and the aggregate weights are used to calculate the dimensional timeliness indicators for each evaluation dimension, and the comprehensive timeliness indicators of the big data platform are generated based on the dimensional timeliness indicators. The comprehensive timeliness index is input into the job scheduler of the big data platform. The job scheduler monitors the scheduling status of the big data platform. When the comprehensive timeliness index is detected to be lower than the preset scheduling intervention threshold, the job scheduler is triggered to dynamically adjust the scheduling priority of the data processing job and allocate computing resources to the data processing job based on the scheduling priority.

2. The data output timeliness measurement method of the big data platform as described in claim 1, characterized in that, The steps of determining dynamic factors based on the preset attribute information and the data dependencies, and determining the aggregation weight of the data processing job in each evaluation dimension based on the dynamic factors, include: Obtain job information corresponding to each data processing job and its respective service level agreement; Based on the preset impact weight coefficients for each service level, calculate the static impact value of each data processing job based on the job information; Based on the data dependencies, a directed acyclic graph is constructed for each of the data processing jobs, and the downstream dependent jobs of each of the data processing jobs are identified through the directed acyclic graph. Calculate the timeliness risk transmission coefficient of the data processing operation based on the number of downstream dependent operations, static impact value, and dependency edge attributes; The combined impact value of the data processing operation is obtained by fusing the static impact value and the time-sensitive risk transmission coefficient. Based on the comprehensive impact value, the aggregate weight of the data processing operation in each evaluation dimension is determined.

3. The data output timeliness measurement method of the big data platform as described in claim 2, characterized in that, The step of calculating the timeliness risk transmission coefficient of the data processing job based on the number of downstream dependent jobs, static impact value, and dependency edge attributes includes: Assign a weight to each dependent edge in the directed acyclic graph, the weight being determined based on the sensitivity of the downstream operation to the output data of the upstream operation or the data processing time of the dependent edge. Starting with the data processing job, all downstream dependent jobs are traversed along the dependency edge, and the product of the static impact value of each downstream dependent job and its corresponding dependency edge weight is accumulated to obtain the total value of the transmitted impact. The total value of the transmission impact is normalized to obtain the time-sensitive risk transmission coefficient.

4. The data output timeliness measurement method of the big data platform as described in claim 1, characterized in that, The step of obtaining preset attribute information for each of the data processing jobs and determining the aggregation weight of the data processing jobs in each evaluation dimension based on the preset attribute information includes: Obtain important system information regarding the different Service Level Agreements (SLAs) affected by each data processing operation; Calculate the comprehensive impact value of each data processing job on the important system based on the preset impact weight coefficients of different SLA levels; Based on the comprehensive impact value of each data processing task, its relative weight within its respective SLA level is determined, which serves as its aggregate weight in the corresponding evaluation dimension.

5. The data output timeliness measurement method of the big data platform as described in claim 4, characterized in that, The steps for calculating the comprehensive impact of each data processing job on the critical system based on preset impact weighting coefficients for different SLA levels include: Obtain the number of important systems that the data processing operation affects at different service level protocol levels, and the weighting coefficients corresponding to each service level protocol level; The number of important systems for each service level agreement level is multiplied by the corresponding weight coefficient, and all the multiplication results are summed to obtain the comprehensive impact value of the data processing operation.

6. The data output timeliness measurement method of the big data platform as described in claim 1, characterized in that, The steps for calculating the dimensional timeliness index of each evaluation dimension using the individual timeliness index and the aggregate weight include: The weighted index is obtained by multiplying the individual timeliness indicators of all data processing jobs within an evaluation dimension by their corresponding aggregate weights. Sum all the weighted indicators to obtain the timeliness index of this evaluation dimension.

7. The method for measuring the timeliness of data output from a big data platform as described in any one of claims 1 to 6, characterized in that, The data output timeliness measurement method of the big data platform also includes: Based on the aforementioned comprehensive timeliness indicators, analyze the changing trend of data output timeliness of the big data platform; If the comprehensive timeliness index is below the first threshold or continues to decline for multiple consecutive periods, it is determined that the timeliness of the data output of the big data platform is deteriorating. If the comprehensive timeliness index mentioned above is higher than the second threshold or continues to rise for multiple consecutive periods, it is determined that the timeliness of the platform's data output is showing an improving trend. Wherein, the first threshold is a first percentage value of the preset scheduling intervention threshold, the second threshold is a second percentage value of the preset scheduling intervention threshold, and the first threshold is less than the second threshold.

8. The data output timeliness measurement method of the big data platform as described in claim 7, characterized in that, If the comprehensive timeliness index is below the first threshold for multiple consecutive periods or continues to decline, after determining that the timeliness of the data output of the big data platform is deteriorating, the method further includes: Extract dimension timeliness indicators of multiple preset evaluation dimensions from the comprehensive timeliness indicator data, and locate key evaluation dimensions among the dimension timeliness indicators of multiple preset evaluation dimensions. Based on the attributes of the key evaluation dimensions, perform operational bottleneck analysis or attribute optimization effects.

9. A data output timeliness measurement device for a big data platform, characterized in that, The data output timeliness measurement device of the big data platform stores a computer program, which, when executed by a processor, implements the data output timeliness measurement method of the big data platform according to any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the data output timeliness measurement method of the big data platform according to any one of claims 1-8.

Images