Data processing job execution management method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing job execution management method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring computing resource parameters of a big data platform at a preset time point and job information of a plurality of data processing jobs; determining the estimated total processing duration of each data processing job according to the job parameters of the data processing jobs; when the computing resource parameter is not lower than the first threshold value, the multiple data processing jobs are executed in sequence from low to high according to the estimated total processing duration; and when the computing resource parameter is lower than a first threshold value, determining execution priorities of the corresponding data processing jobs according to the estimated total processing duration, the job deadline, the job level and the job type of each data processing job, and executing the plurality of data processing jobs in sequence from high to low according to the execution priorities. The technical problem that in the prior art, the execution sequence of the data processing operation is set according to a fixed rule or priority, and consequently calculation resource arrangement is unreasonable is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, in particular to a data processing job execution management method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the deepening of digital transformation, enterprises need to extract valuable information from massive data to support decision-making, optimize business processes and improve customer experience. Batch jobs, as the main form of data processing, undertake important tasks from data integration, cleaning, conversion to analysis. They usually involve reading, processing and writing a large amount of data, as well as complex SQL (Structured Query Language) queries and data operation logic. Big data platforms have become the first choice for batch job execution with their distributed processing capabilities and high scalability. However, as the number of jobs on the platform increases, the dependency relationship between jobs and resource competition also become more complex. How to reasonably arrange the execution order of jobs to ensure that critical business is completed in time while avoiding resource waste has become an important technical challenge for big data platform operation.

[0003] Currently, job scheduling systems often rely on static rules (such as deadlines or fixed priorities) when arranging the execution order of such jobs. This static strategy cannot dynamically adjust the job scheduling strategy in combination with the real-time system computing resource state and business urgency, resulting in poor flexibility in arranging task execution order and causing system computing resource waste or high-priority job delay.

[0004] In view of the above problems, no effective solutions have been proposed so far. SUMMARY

[0005] The main purpose of the present application is to provide a data processing job execution management method and device, a storage medium and an electronic device, to at least solve the technical problem that the existing technology sets the execution order of data processing jobs according to fixed rules or priority, resulting in unreasonable arrangement of computing resources.

[0006] According to an aspect of some embodiments of the present application, a data processing job execution management method is provided, including: obtaining computing resource parameters of a big data platform at a preset time point and job information of a plurality of data processing jobs, wherein the job information includes job parameters, job levels, job types, and job deadlines; for each data processing job, determining an estimated total processing duration of the data processing job according to the job parameters of the data processing job; in a case where the computing resource parameters are not lower than a first threshold, controlling the big data platform to execute the plurality of data processing jobs in order from low to high according to the estimated total processing durations; in a case where the computing resource parameters are lower than the first threshold, determining an execution priority of each data processing job according to the estimated total processing duration, the job deadline, the job level, and the job type of the data processing job, and controlling the big data platform to execute the plurality of data processing jobs in order from high to low according to the execution priority.

[0007] In some embodiments of the present application, the job parameters at least include data volume of at least one data table, processing order, at least one processing operation of each data table, and a corresponding execution script, the data volume is determined by row number and length of all fields in each column of each row, and the processing operation includes a read operation, a write operation, a calculation operation, and an export operation. The estimated total processing duration of the data processing job is determined according to the job parameters of the data processing job, including: for each data table in the data processing job, analyzing the data volume of the data table, each processing operation, and the corresponding execution script by using a prediction model to obtain an estimated processing duration of the data table for performing each processing operation; and adding the estimated processing durations of all processing operations of each data table in the data processing job in order according to the processing order to obtain the estimated total processing duration of the data processing job.

[0008] In some embodiments of the present application, a plurality of meta-models for predicting processing durations of different processing operations are included in the prediction model, and a training process of each meta-model includes: obtaining a plurality of groups of training sample data, wherein each group of training sample data includes data volume of a data table to be executed by the big data platform at a historical time point for a target processing operation, an execution script of the target processing operation as a training sample, and an actual processing duration of the data table for performing the target processing operation as a sample label corresponding to the training sample; constructing a neural network model; and iteratively training the neural network model by using the plurality of groups of training sample data to obtain a meta-model for predicting the processing duration of the target processing operation.

[0009] In some embodiments of this application, after obtaining the estimated processing time for each processing operation of the data table, the method further includes: determining the target data table for performing computational operations within the data processing job, parsing the execution script of the target data table when performing computational operations to obtain multiple keywords; determining the complexity of the computational operations performed on the target data table based on the keywords, and multiplying the complexity by the estimated processing time of the computational operations performed on the target data table to obtain the target estimated processing time of the computational operations performed on the target data table.

[0010] In some embodiments of this application, the keyword type includes at least one of the following: join operation type, aggregation operation type, and window operation type. Determining the complexity of the computational operation performed on the target data table based on the keyword includes: when the keyword type is a join operation type, obtaining the row counts of the associated data tables of the target data table from a preset database, and using a first multiple of the sum of the row counts of the target data table and the associated data tables as the complexity of the computational operation performed on the target data table, wherein the preset database stores at least the row counts of multiple data tables; when the keyword type is an aggregation operation type, using a second multiple of the row counts of the target data table as the complexity of the computational operation performed on the target data table; when the keyword type is a window operation type, using a third multiple of the row counts of the target data table as the complexity of the computational operation performed on the target data table; wherein the first multiple is greater than the second multiple, and the second multiple is greater than the third multiple.

[0011] In some embodiments of this application, the execution priority of each data processing job is determined based on the estimated total processing time, job deadline, job level, and job type. This includes: for each data processing job, determining the timeliness coefficient of the data processing job based on the difference between the sum of the estimated total processing time and preset time points and the job deadline; determining the business weight of the data processing job based on the job level; determining the resource peak-shaving weight of the data processing job based on the job type and computing resource parameters; and multiplying the sum of the business weight and the resource peak-shaving weight by the timeliness coefficient to obtain the execution priority of the data processing job.

[0012] In some embodiments of this application, the job type includes: processing and computing type or data export type, and the computing resource parameters include: CPU utilization and IO wait rate. Determining the resource peak-shaving weight for the data processing job based on the job type and computing resource parameters includes: setting the resource peak-shaving weight for the processing and computing type data processing job to a first value when the CPU utilization is below a second threshold; setting the resource peak-shaving weight for the processing and computing type data processing job to a second value when the CPU utilization is not below the second threshold, wherein the second value is less than the first value; setting the resource peak-shaving weight for the data export type data processing job to a third value when the IO wait rate is below a third threshold; and setting the resource peak-shaving weight for the data export type data processing job to a fourth value when the IO wait rate is not below the third threshold, wherein the third value is less than the fourth value.

[0013] According to another aspect of the embodiments of this application, a data processing job execution management device is also provided, comprising: an acquisition module, configured to acquire computing resource parameters of a big data platform at a preset time point and job information of multiple data processing jobs, wherein the job information includes: job information, job level, job type, and job deadline; an analysis module, configured to determine the estimated total processing time of each data processing job based on the job parameters of the data processing job; a first management module, configured to control the big data platform to execute multiple data processing jobs sequentially in order of estimated total processing time from low to high when the computing resource parameters are not lower than a first threshold; and a second management module, configured to determine the execution priority of each data processing job based on the estimated total processing time, job deadline, job level, and job type when the computing resource parameters are lower than the first threshold, and control the big data platform to execute multiple data processing jobs sequentially in order of execution priority from high to low.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein the device on which the computer-readable storage medium is located executes the above-described data processing operation management method by running the computer program.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-described data processing job execution management method.

[0016] In this embodiment, the computing resource parameters of the big data platform at a preset time point and the job information of multiple data processing jobs are obtained. The job information includes job parameters, job level, job type, and job deadline. For each data processing job, the estimated total processing time is determined based on its job parameters. If the computing resource parameters are not lower than a first threshold, the big data platform executes multiple data processing jobs sequentially in ascending order of estimated total processing time. If the computing resource parameters are lower than the first threshold, the execution priority of each data processing job is determined based on its estimated total processing time, job deadline, job level, and job type, and the big data platform executes multiple data processing jobs sequentially in descending order of execution priority. This achieves dynamic adjustment of job execution order based on estimated job duration and real-time system resource status, ensuring that short-duration jobs are prioritized when resources are abundant, while intelligent scheduling is performed based on job criticality and deadline when resources are scarce. This optimizes the job scheduling efficiency and resource utilization of the big data platform while ensuring timely completion of critical business operations. This solves the technical problem that existing technologies set the execution order of data processing jobs according to fixed rules or priorities, resulting in unreasonable allocation of computing resources. Attached Figure Description

[0017] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a hardware structure block diagram of an optional mobile terminal for implementing a data processing job execution management method according to an embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating an optional data processing job execution management method according to an embodiment of this application.

[0020] Figure 3 This is a structural block diagram of an optional data processing job execution management device according to an embodiment of this application. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] As described in the background section, existing job scheduling systems often rely on fixed rules or preset priorities to set the execution order of batch jobs that run regularly on a daily or monthly basis. This static setting method has the following obvious drawbacks: First, fixed rules cannot set a reasonable execution order based on the actual complexity and resource requirements of the data processing jobs, resulting in simple data processing jobs being executed too early while complex data processing jobs are delayed, reducing the overall utilization of resources. Second, when system computing resources are strained or sudden changes occur, static priority settings cannot respond quickly, which may cause high-priority data processing jobs to be delayed due to the long running time of low-priority data processing jobs, affecting business timeliness. Furthermore, due to the lack of a dynamic adjustment mechanism, the job scheduling system cannot respond to the actual system resource usage, especially when CPU and I / O resources are scarce, making it impossible to prioritize the scheduling of critical jobs, which may lead to business delays.

[0025] To address the aforementioned issues, embodiments of this application provide a data processing job execution management method. This method aims to dynamically optimize the execution order of tasks, thereby improving resource utilization efficiency, ensuring the priority of critical tasks, and increasing overall job processing speed.

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a data processing job execution management method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing job execution management method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] This embodiment provides a data processing job execution management method that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0030] Figure 2 This is a flowchart illustrating a data processing job execution management method according to an embodiment of this application, such as... Figure 2 As shown, the specific steps of this process are detailed below.

[0031] Step S202: Obtain the computing resource parameters of the big data platform at a preset time point and the job information of each of the multiple data processing jobs. The job information includes: job parameters, job level, job type and job deadline.

[0032] Specifically, the aforementioned big data platform is a computing architecture used to process and analyze massive amounts of data, including but not limited to distributed data processing frameworks such as Hadoop, Spark, and Flink.

[0033] The aforementioned data processing tasks are computational tasks that the big data platform runs at preset time points. They typically include reading, computing, writing, or exporting operations from one or more data tables.

[0034] The above job information describes the characteristic dataset of each data processing job, including but not limited to: job parameters (i.e., the amount of data in each data table contained in the job, the processing order between each data table, the execution script of each data table and at least one processing operation, etc.), job level (i.e., according to the business importance of the job, such as high, medium and low), job type (such as calculation and processing job, data export job, etc.), and job deadline (i.e. the final time point when the job must be completed).

[0035] The aforementioned computing resource parameters refer to quantitative indicators of the current available computing power of the big data platform, including but not limited to real-time resource data such as CPU utilization, memory usage, disk I / O wait rate, and network bandwidth.

[0036] Step S204: For each data processing job, determine the estimated total processing time of the data processing job based on the job parameters.

[0037] Specifically, the estimated total processing time mentioned above is the estimated time required from the start of the preset time point to the completion of the data processing operation, calculated based on the operation parameters.

[0038] Step S206: If the computational resource parameters are not lower than the first threshold, control the big data platform to execute multiple data processing jobs in order of estimated total processing time from low to high.

[0039] Specifically, when the computing resource parameters of the big data platform reach or exceed a pre-set first threshold, the scheduling system can use the estimated total processing time of the data processing jobs as the main sorting criterion, starting with the data processing job with the shortest processing time, and sequentially schedule and execute each data processing job.

[0040] Step S208: When the computational resource parameters are lower than the first threshold, determine the execution priority of each data processing job based on the estimated total processing time, job deadline, job level, and job type, and control the big data platform to execute multiple data processing jobs in order of execution priority from high to low.

[0041] Specifically, when the computing resource parameters of the big data platform do not reach the pre-set first threshold, the scheduling system can dynamically determine the execution priority of each data processing job based on multiple dimensions of information. This allows the big data platform to execute multiple data processing jobs in descending order of execution priority. This sorting mechanism fully considers the urgency of the jobs and the efficiency of resource utilization, ensuring that, given limited resources, jobs with higher value and more urgent needs for the business are completed first.

[0042] In this embodiment, when resources are sufficient, multiple data processing jobs are executed sequentially in order of their estimated total processing time from low to high; when resources are scarce, the big data platform is controlled to execute multiple data processing jobs sequentially in order of their execution priority from high to low, thereby achieving the goal of improving the processing efficiency and resource utilization balance of the big data platform.

[0043] The following section explains each step of the data processing job execution management method in conjunction with the specific implementation process.

[0044] As an optional implementation, in the technical solution provided in step S204 above, for each data processing job, the scheduling system can determine the estimated total processing time of the data processing job according to the following steps: for each data table in the data processing job, the data volume of the data table, each processing operation, and the corresponding execution script are analyzed using a prediction model to obtain the estimated processing time for each processing operation of the data table. The job parameters include at least: the data volume of all (at least one) data tables, the processing order between the data tables, at least one processing operation for each data table, and the execution script corresponding to each processing operation. The data volume is determined by the number of rows and the length of all fields in each column of each row. The processing operations include: read operations, write operations, calculation operations, and export operations. The estimated processing time of each data table in the data processing job is added together sequentially according to the processing order to obtain the estimated total processing time of the data processing job.

[0045] In the above embodiments, for each data table within the data processing job, the scheduling system calls a pre-trained prediction model to analyze the data volume of the data table, the various processing operations to be executed, and the corresponding execution scripts to obtain the estimated processing time for each processing operation of the data table. Compared to predicting job processing time based on static rules or historical averages, this method utilizes machine learning technology to capture the non-linear relationship between data volume, execution script complexity, and processing time, thus providing more accurate prediction results. Simultaneously, by analyzing the estimated time of each processing operation individually, compared to overall estimation, errors caused by simplified calculations are reduced, improving the accuracy of time estimation. Furthermore, according to the processing order of each data table within the data processing job, the estimated processing times of all processing operations are accumulated to obtain the estimated total processing time for the data processing job to complete all predetermined operations from beginning to end. This accumulation principle considers the sequential relationship between processing flows within the job, ensuring the rationality and accuracy of the estimated time.

[0046] Optionally, the above prediction model includes multiple meta-models for predicting the processing time of different processing operations.

[0047] The training process for each meta-model includes: Step 1: Acquiring multiple sets of training sample data. Each set of training sample data includes: the data volume of the target processing operation to be executed at historical time points by the big data platform, the execution script of the target processing operation as the training sample, and the actual processing time of the target processing operation in the data table as the sample label of the corresponding training sample. Step 2: Constructing a neural network model. Step 3: Iteratively training the neural network model using multiple sets of training sample data to obtain a meta-model used to predict the processing time of the target processing operation.

[0048] The model architecture of this neural network can be a multilayer perceptron, a recurrent neural network, etc. The structure of the neural network model can include: an input layer, at least one hidden layer, at least one recurrent layer, and an output layer. The hidden layer can contain fully connected layers (to capture complex relationships between features) and dropout layers (to reduce the risk of overfitting). The recurrent layer can be a long short-term memory network or a gated recurrent unit, used to process the sequential features of the execution script, helping the model understand the order and hierarchy of operations within the script.

[0049] Specifically, in the technical solution provided in the third step above, the method may include the following steps: data preprocessing, forward propagation, loss calculation, back propagation, and parameter update, and the specific implementation of each step is detailed in the following description.

[0050] Data preprocessing: The data volume of the data table that performed the target processing operation at historical time points in the big data platform, and the field length of each column in each row of the data table are standardized to eliminate the influence of feature scaling; at the same time, text features such as the execution script corresponding to the target processing operation are embedded and encoded to convert them into numerical vectors; the numerical features and text vectorized features are combined to form the final input feature vector.

[0051] Forward propagation: The input layer first receives the input feature vector of the training samples; the hidden layer (composed of one or more fully connected layers) performs a nonlinear transformation on the input feature vector to extract high-level abstract features; simultaneously, the recurrent layer (such as a long short-term memory network) processes the execution script sequence features output by the last hidden layer, with each time step corresponding to a statement in the execution script, and the dependencies and features of the entire sequence are associated through the internal recurrent state propagation mechanism; finally, the output layer (containing one neuron) uses a linear activation function to fuse and analyze the feature vectors output by the hidden layer and the recurrent layer to obtain the predicted processing time corresponding to the training samples.

[0052] Loss calculation: Construct a target loss function based on the predicted processing time and the actual processing time for each training sample. The target loss function can be of the following types: mean squared error loss function or mean absolute error loss function, etc.

[0053] Backpropagation: The gradient of the target loss function with respect to the model parameters of the neural network model is calculated using the backpropagation method.

[0054] Parameter update: Adjust the model parameters based on the gradient and the preset learning rate to minimize the value of the target loss function.

[0055] Repeat the above process of forward propagation, loss calculation, backpropagation, and parameter update until the model parameters converge or the preset number of iterations is reached.

[0056] Through the above model training process, an accurate predictive model for job processing time can be obtained, which can be used for real-time scheduling and optimization of data processing job execution.

[0057] Furthermore, after obtaining the estimated processing time for each processing operation of the data table, since the complexity of read, write, and export operations is relatively fixed, the estimated processing time of the data table output by the prediction model for these operations can be directly used as the final prediction result.

[0058] As for computational operations, the estimated processing time based solely on the data table output by the prediction model cannot accurately reflect the actual processing time. This is because the time consumption of computational operations largely depends on the complexity of the execution script and the data processing logic.

[0059] Therefore, the scheduling system can accurately obtain the estimated processing time for performing calculation operations on a data table using the following method: First, determine the target data table for the calculation operations within the data processing job, and parse the execution script for the target data table to obtain multiple keywords; then, determine the complexity of the calculation operations on the target data table based on the keywords, and multiply the complexity by the estimated processing time of the calculation operations on the target data table to obtain the target estimated processing time of the calculation operations on the target data table. The types of the keywords include at least one of the following: join operation type, aggregation operation type, window operation type, etc. Keywords for join operation types include, but are not limited to: Inner Join, Left Join, Right Join, Full Join, etc.; keywords for aggregation operation types include "Group By," etc.; keywords for window operation types include, but are not limited to: "Sum," "Avg," "Count," "Min," "Max," "Order By," etc.

[0060] Therefore, the scheduling system can determine the complexity of the computational operation performed on the target data table according to the following rules: When the key type is a join operation, obtain the number of rows in the associated data tables of the target data table from the preset database, and use the first multiple of the sum of the number of rows in the target data table and the number of rows in the associated data tables as the complexity of the computational operation performed on the target data table. The preset database stores the row counts of at least multiple data tables. When the key type is an aggregation operation, use the second multiple of the number of rows in the target data table as the complexity of the computational operation performed on the target data table. When the key type is a window operation, use the third multiple of the number of rows in the target data table as the complexity of the computational operation performed on the target data table. The first multiple is greater than the second multiple, and the second multiple is greater than the third multiple.

[0061] It should be noted that if the execution script for performing calculations on the target data table includes multiple types of keywords, the complexity corresponding to each type of keyword can be added together and divided by the number of rows in the target data table to obtain the final complexity. For example, if the execution script for performing calculations on target data table A includes the keywords Join, Group by, and SUM, where the first multiplier can be 2, the second multiplier can be 1, and the third multiplier can be 0.5, and target data table X has 100 rows and the associated data table Y has 200 rows, then the formula for calculating the complexity of performing calculations on target data table X can be written as: .

[0062] Therefore, this application embodiment deeply analyzes the execution script of the data processing job to perform computational operations. It uses a keyword recognition algorithm to locate computationally intensive instructions within the script and accurately quantifies the complexity of the computational operations based on the frequency and type of these instructions. Combined with model prediction results, it obtains the target processing time for the computational operations on the target data table. This evaluation method no longer relies on the overall data volume and simple operations of the data processing job to assess its processing time. Instead, it deeply understands the inherent complexity of the job, including data connections, summarization, and analysis, thus more accurately assessing computation time. Furthermore, combining model prediction results with complexity reflects the additional time cost brought to the job by computationally intensive operations. This approach overcomes the shortcomings of traditional estimation methods, which neglect the impact of the statement structure and complexity of the execution script on execution time.

[0063] For example, if a data processing job includes four data tables, and the processing order of these four data tables is: Data Table A → Data Table B → Data Table C → Data Table D, where for the source data table A, read operations and calculation operations are performed sequentially; for intermediate tables B and C, read operations, write operations, and calculation operations are performed sequentially; and for the result data table D, write operations and export operations are performed sequentially, then the estimated processing time for each of the above data tables can be determined by the following formula: , , , In the formula, This represents the complexity of performing calculations on the source data table A. This represents the complexity of performing calculations on intermediate data table B. This represents the complexity of performing calculations on the intermediate data table C.

[0064] Therefore, the expression for the estimated total processing time of this data processing job can be written as: .

[0065] Furthermore, the scheduling system can compare the computing resource usage (such as CPU utilization) of the big data platform with a preset first threshold, and the comparison results can be divided into the following two cases.

[0066] Scenario 1: The calculated resource parameters are not lower than the first threshold.

[0067] In the above situation, the scheduling system can sort multiple data processing jobs in order of estimated total processing time from low to high, and control the big data platform to prioritize the execution of jobs with shorter processing times based on the sorting results, effectively balancing the system load and preventing resource bottlenecks.

[0068] Scenario 2: The calculated resource parameters are below the first threshold.

[0069] In the above situation, the scheduling system can determine the execution priority of each data processing job based on the estimated total processing time, job deadline, job level, and job type.

[0070] As an optional implementation, the scheduling system can determine the execution priority of each data processing job according to the following steps: Step S1, determine the timeliness coefficient of the data processing job based on the difference between the sum of the estimated total processing time and the preset time point and the job deadline; Step S2, determine the business weight of the data processing job based on the job level; Step S3, determine the resource peak-shifting weight of the data processing job based on the job type and computing resource parameters; Step S4, multiply the sum of the business weight and the resource peak-shifting weight by the timeliness coefficient to obtain the execution priority of the data processing job.

[0071] The technical solution provided in step S1 above can be understood as converting the difference between the completion time of the data processing job and the deadline required by the business into a quantitative coefficient that can be directly applied to scheduling decisions. This step can usually be achieved through a preset time window mapping table, which includes timeliness coefficients corresponding to multiple time windows (i.e., different time intervals). The longer the time window, the smaller the corresponding timeliness coefficient; conversely, the shorter the time window, the larger the corresponding timeliness coefficient.

[0072] Specifically, if the difference falls within a shorter time window (e.g., 0-3 hours), it indicates that the completion time of the data processing job is closer to the business deadline, and the corresponding timeliness coefficient will be higher (e.g., a timeliness coefficient of 3). If the difference falls within a shorter time window (e.g., 3-6 hours), it indicates that the completion time of the data processing job is relatively far from the business deadline, and the corresponding timeliness coefficient will be relatively low (e.g., a timeliness coefficient of 2). If the difference falls within a shorter time window (e.g., greater than 6 hours), it indicates that the completion time of the data processing job is even further from the business deadline, and the corresponding timeliness coefficient will be even lower (e.g., a timeliness coefficient of 1). Therefore, the timeliness coefficient ensures that data processing jobs that are about to expire or jobs that are extremely time-sensitive can receive higher priority, avoiding delays in critical operations and reducing business risks.

[0073] Next, the technical solution provided in step S2 above can be understood as follows: the big data platform defines different data processing job levels. Therefore, the scheduling system can determine the business weight of a data processing job according to its job level and a pre-set level mapping rule. In other words, the business weight emphasizes the impact of data processing jobs on core business, ensuring that the tasks most critical to business operations are processed first, thereby improving the overall business response speed and service quality.

[0074] For example, if job levels are divided into "high," "medium," and "low," then the business weight of "high" level data processing jobs can be higher (e.g., set to 1), the business weight of "medium" level data processing jobs can be relatively lower (e.g., set to 0.8), and the business weight of "low" level data processing jobs can be set to the lowest (e.g., set to 0.5). These weight values ​​reflect the priority of data processing jobs at the business level; therefore, the higher the job level, the greater the business weight.

[0075] In the technical solution provided by step S3 above, the above-mentioned job types include: processing calculation type or data export type. In addition to the CPU occupancy rate, the calculation resource parameters may also include the input / output I / O wait rate.

[0076] Therefore, the aforementioned resource staggering weights are set based on the different computing resource requirements of different types of jobs. The setting rules are as follows: when the CPU utilization rate is below the second threshold, the resource staggering weight for data processing jobs involving computational processing is set to the first value; when the CPU utilization rate is not lower than the second threshold, the resource staggering weight for such jobs is set to the second value, where the second value is less than the first value. When the IO wait rate is below the third threshold, the resource staggering weight for data exporting jobs is set to the third value; when the IO wait rate is not lower than the third threshold, the resource staggering weight for data exporting jobs is set to the fourth value, where the third value is less than the fourth value. Therefore, the dynamic adjustment of the resource staggering weights allows the system to guide different types of data processing jobs to be executed in an interleaved manner, reducing resource bottlenecks and improving resource utilization efficiency.

[0077] It should be noted that both the second and third thresholds mentioned above are less than the first threshold, and the second and third thresholds can be the same or different, depending on the specific application scenario. Furthermore, the first, second, third, and fourth values ​​mentioned above can all be set according to the actual application scenario; for example, the first and third values ​​can both be set to 0.3, while the second and fourth values ​​can both be set to -0.3.

[0078] In the technical solution provided in step S4 above, the business weight and the resource peak-shifting weight are added together to form a weighted value that reflects the comprehensive priority of the operation at the business level and the resource peak-shifting level. The final priority order is obtained by multiplying the weighted value by the timeliness coefficient. The larger the product, the higher the execution priority of the operation, and the earlier it should be scheduled for execution.

[0079] Therefore, by integrating business weight, resource peak-shifting weight, and timeliness coefficient, a comprehensive execution priority calculation model is formed. This model aims to comprehensively consider the business importance of the job, the system resource status, and the time urgency to determine the execution order of the job in the scheduling system, ensuring that critical business is processed first, while maintaining the efficient operation of the system.

[0080] For example, if the big data platform needs to process three data processing jobs, A1, A2, and A3, at 14:00, the CPU utilization rate of the big data platform at 14:00 is less than 50%, and the IO wait rate is higher than 50%.

[0081] If the estimated processing time for data processing job A1 is 1 hour, and the deadline is 16:30 on the same day, and the business system associated with data processing job A1 is an important system, then its job type is processing and calculation; if the estimated processing time for data processing job A2 is 2 hours, and the deadline is 21:00 on the same day, and the business system associated with data processing job A2 is a non-important system, then its job type is data export; if the estimated processing time for data processing job A3 is 3 hours, and the deadline is 23:11 on the same day, and the business system associated with data processing job A3 is an important system, then its job type is processing and calculation. Therefore, it can be determined that there are 1.5 hours remaining between the estimated completion time and the deadline for data processing job A1, 5 hours remaining between the estimated completion time and the deadline for data processing job A2, and 6 hours remaining between the estimated completion time and the deadline for data processing job A3.

[0082] Therefore, the timeliness coefficient of data processing job A1 can be set to 3, the timeliness coefficient of data processing job A2 to 2, and the timeliness coefficient of data processing job A3 to 1. In addition, data processing jobs A1 and A3 are both associated with important systems, so their business weight is set to 1, while data processing job A2 is associated with non-important systems, so its business weight is set to 0.5. Based on the CPU utilization rate, IO wait rate and business type of the big data platform at 14:00, the resource peak-shifting weight of data processing job A1 can be set to -0.3, the resource peak-shifting weight of data processing job A2 can be set to -0.3, and the resource peak-shifting weight of data processing job A3 can be set to -0.3.

[0083] Based on the above information, the execution priorities of data processing jobs A1, A2, and A3 can be calculated using the following formulas: , , .

[0084] Therefore, the scheduling system can control the big data platform to run the jobs in the following order at 14:00: first execute data processing job A1, then execute data processing job A3, and finally execute data processing job A2.

[0085] It should be noted that after the big data platform completes multiple data processing jobs at preset time points, it can perform backtracking analysis and parameter correction on the prediction model based on the actual processing time of each processing operation on different data tables under each data processing job, so as to reduce the model prediction error, especially when processing periodic jobs with similar characteristics, to ensure the accuracy of the model output results.

[0086] In summary, the data processing job execution management method provided in this application has the following technical advantages compared with the prior art: (1) This application provides a more accurate prediction of the total processing time of a job by analyzing the job parameters in detail and combining the real-time computing resource parameters of the big data platform. Compared with the prior art, which relies on rough experience estimates or historical data averages, this method can more closely match the actual job requirements and reduce prediction errors; (2) This application can determine the execution priority based on the estimated total processing time of the job, the job deadline, the job level, and the job type when computing resources are scarce, thereby realizing dynamic optimization of resource scheduling. This contrasts sharply with the static or single-factor-based scheduling strategies in the prior art, improving job execution efficiency and system response speed under resource-constrained conditions; (3) The embodiments of this application introduce a resource staggered weighting mechanism, which can dynamically adjust according to the complementarity between job type and current queue resource load, guide different types of data processing jobs to be executed in an interleaved manner, effectively avoid performance bottlenecks caused by resource contention, and improve the balance of resource utilization and overall processing efficiency; (4) The embodiments of this application dynamically adjust the model performance of the prediction model through backtracking and prediction correction mechanisms, enabling the model to learn and continuously optimize itself, overcoming the shortcomings of the lack of self-correction and iterative improvement mechanisms in the prior art.

[0087] In summary, the embodiments of this application significantly improve the intelligence, efficiency, and reliability of big data platform job scheduling by introducing innovative technologies such as dynamic resource scheduling, accurate duration prediction, multi-dimensional priority calculation, intelligent peak-shifting scheduling, and adaptive model correction, providing a brand-new solution to common problems in big data job scheduling.

[0088] This application also provides a data processing job execution management device. It should be noted that the data processing job execution management device of this application can be used to execute the data processing job execution management method provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0089] The following describes the data processing job execution management device provided in the embodiments of this application.

[0090] Figure 3 This is a structural block diagram of a data processing job execution management device according to an embodiment of this application. Figure 3 As shown, the device includes: an acquisition module 32, an analysis module 34, a first management module 36, and a second management module 38.

[0091] The acquisition module 32 is used to acquire the computing resource parameters of the big data platform at a preset time point and the job information of each of the multiple data processing jobs. The job information includes: job information, job level, job type and job deadline.

[0092] Analysis module 34 is used to analyze the job information of each data processing job to obtain the estimated total processing time of the data processing job.

[0093] The first management module 36 is used to control the big data platform to execute multiple data processing jobs in order of estimated total processing time from low to high, provided that the computing resource parameters are not lower than the first threshold.

[0094] The second management module 38 is used to determine the execution priority of each data processing job based on the estimated total processing time, job deadline, job level and job type when the computing resource parameters are lower than the first threshold, and to control the big data platform to execute multiple data processing jobs in order of execution priority from high to low.

[0095] Optionally, the job parameters may include at least: the amount of data in at least one data table, the processing order, at least one processing operation for each data table and the corresponding execution script. The amount of data is determined by the number of rows and the length of all fields in each column of each row. The processing operations include: read operations, write operations, calculation operations and export operations.

[0096] Then, the analysis module 34 can determine the estimated total processing time of the data processing job according to the following steps: for each data table in the data processing job, the prediction model is used to analyze the data volume of the data table, each processing operation and the corresponding execution script to obtain the estimated processing time of each processing operation of the data table; the estimated processing time of each data table in the data processing job is added together in the order of processing to obtain the estimated total processing time of the data processing job.

[0097] The method involves utilizing multiple meta-models within the prediction model to predict the processing time of different processing operations. The training process for each meta-model includes: acquiring multiple sets of training sample data, wherein each set of training sample data includes: the data volume of the data table of the target processing operation to be executed at a historical time point from the big data platform, the execution script of the target processing operation as training samples, and the actual processing time of the target processing operation executed in the data table as the sample label of the corresponding training sample; constructing a neural network model; and iteratively training the neural network model using multiple sets of training sample data to obtain a meta-model for predicting the processing time of the target processing operation.

[0098] In addition, after obtaining the estimated processing time for each processing operation of the data table, the analysis module 34 can also determine the target data table for performing calculation operations within the data processing job, and parse the execution script of the target data table when performing calculation operations to obtain multiple keywords. The types of keywords include at least one of the following: join operation type, aggregation operation type, and window operation type. Based on the keywords, the complexity of the calculation operation performed on the target data table is determined, and the complexity is multiplied by the estimated processing time of the calculation operation performed on the target data table to obtain the target estimated processing time of the calculation operation performed on the target data table.

[0099] Specifically, the analysis module 34 can determine the complexity of the calculation operation performed on the target data table using the following methods: When the key type is a join operation, the number of rows in the associated data tables of the target data table is obtained from a preset database. The first multiple of the sum of the number of rows in the target data table and the number of rows in the associated data tables is used as the complexity of the calculation operation performed on the target data table. The preset database stores the row counts of at least multiple data tables. When the key type is an aggregation operation, the second multiple of the number of rows in the target data table is used as the complexity of the calculation operation performed on the target data table. When the key type is a window operation, the third multiple of the number of rows in the target data table is used as the complexity of the calculation operation performed on the target data table. The first multiple is greater than the second multiple, and the second multiple is greater than the third multiple.

[0100] Optionally, the second management module 38 can determine the execution priority of each data processing job in the following way: determining the timeliness coefficient of the data processing job based on the difference between the sum of the estimated total processing time and the preset time point and the job deadline; determining the business weight of the data processing job based on the job level; determining the resource peak-shaving weight of the data processing job based on the job type and computing resource parameters; and multiplying the sum of the business weight and the resource peak-shaving weight by the timeliness coefficient to obtain the execution priority of the data processing job.

[0101] The job types include: processing and calculation type or data export type, and the calculation resource parameters include: CPU utilization rate and IO wait rate.

[0102] Therefore, the second management module 38 can set the resource peak-shaving weight of data processing jobs as follows: when the CPU utilization rate is lower than the second threshold, the resource peak-shaving weight of data processing jobs of the processing and computing type is set to the first value; when the CPU utilization rate is not lower than the second threshold, the resource peak-shaving weight of data processing jobs of the processing and computing type is set to the second value, wherein the second value is less than the first value; when the IO wait rate is lower than the third threshold, the resource peak-shaving weight of data processing jobs of the data export type is set to the third value; when the IO wait rate is not lower than the third threshold, the resource peak-shaving weight of data processing jobs of the data export type is set to the fourth value, wherein the third value is less than the fourth value.

[0103] It should be noted that, Figure 3 The data processing job execution management device shown is used to execute Figure 2 The data processing job execution management device shown, therefore, Figure 2 The relevant explanations and instructions in the data processing job execution management method also apply to Figure 3 The data processing job execution management device shown will not be described in detail here.

[0104] Furthermore, the data processing job execution management device includes a processor and a memory. The aforementioned acquisition module, analysis module, first management module, and second management module are all stored as program units in the memory, and the processor executes these program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the various modules may be located in different processors in any combination.

[0105] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to execute a data processing job execution management method.

[0106] Specifically, the data processing job execution management method includes: acquiring the computing resource parameters of the big data platform at a preset time point and the job information of each of the multiple data processing jobs, wherein the job information includes: job parameters, job level, job type, and job deadline; for each data processing job, determining the estimated total processing time of the data processing job based on the job parameters; when the computing resource parameters are not lower than a first threshold, controlling the big data platform to execute the multiple data processing jobs sequentially in order of estimated total processing time from low to high; when the computing resource parameters are lower than the first threshold, determining the execution priority of each data processing job based on the estimated total processing time, job deadline, job level, and job type, and controlling the big data platform to execute the multiple data processing jobs sequentially in order of execution priority from high to low.

[0107] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DROM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CDROM), digital versatile disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this article, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0108] This invention provides a processor for running a program, wherein the program executes a data processing job execution management method during runtime.

[0109] Specifically, the data processing job execution management method includes: acquiring the computing resource parameters of the big data platform at a preset time point and the job information of each of the multiple data processing jobs, wherein the job information includes: job parameters, job level, job type, and job deadline; for each data processing job, determining the estimated total processing time of the data processing job based on the job parameters; when the computing resource parameters are not lower than a first threshold, controlling the big data platform to execute the multiple data processing jobs sequentially in order of estimated total processing time from low to high; when the computing resource parameters are lower than the first threshold, determining the execution priority of each data processing job based on the estimated total processing time, job deadline, job level, and job type, and controlling the big data platform to execute the multiple data processing jobs sequentially in order of execution priority from high to low.

[0110] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: acquiring computing resource parameters of a big data platform at a preset time point and job information of multiple data processing jobs, wherein the job information includes: job parameters, job level, job type, and job deadline; for each data processing job, determining the estimated total processing time of the data processing job based on the job parameters; when the computing resource parameters are not lower than a first threshold, controlling the big data platform to execute multiple data processing jobs sequentially in ascending order of estimated total processing time; when the computing resource parameters are lower than the first threshold, determining the execution priority of each data processing job based on the estimated total processing time, job deadline, job level, and job type, and controlling the big data platform to execute multiple data processing jobs sequentially in descending order of execution priority.

[0111] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps: obtaining the computing resource parameters of a big data platform at a preset time point and the job information of multiple data processing jobs, wherein the job information includes: job parameters, job level, job type, and job deadline; for each data processing job, determining the estimated total processing time of the data processing job based on the job parameters of the data processing job; when the computing resource parameters are not lower than a first threshold, controlling the big data platform to execute multiple data processing jobs sequentially in order of estimated total processing time from low to high; when the computing resource parameters are lower than the first threshold, determining the execution priority of each data processing job based on the estimated total processing time, job deadline, job level, and job type of the data processing job, and controlling the big data platform to execute multiple data processing jobs sequentially in order of execution priority from high to low.

[0112] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0113] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0118] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing job execution management method characterized by comprising: The method comprises the following steps: Obtain the computing resource parameters of the big data platform at a preset time point and the job information of each data processing job, wherein the job information comprises job parameters, job level, job type and job deadline; For each data processing job, determine the estimated total processing time of the data processing job according to the job parameters of the data processing job; If the computing resource parameters are not lower than a first threshold, control the big data platform to execute the data processing jobs in order from low to high according to the estimated total processing time; If the computing resource parameters are lower than the first threshold, determine the execution priority of each data processing job according to the estimated total processing time, job deadline, job level and job type of the data processing job, and control the big data platform to execute the data processing jobs in order from high to low according to the execution priority.

2. The method of claim 1, wherein, The job parameters at least include the data volume of at least one data table, the processing order, at least one processing operation of each data table and the corresponding execution script, the data volume is determined by the number of rows and the length of all fields in each column, and the processing operation includes reading operation, writing operation, calculation operation and exporting operation, wherein determining the estimated total processing time of the data processing job according to the job parameters of the data processing job comprises: For each data table in the data processing job, analyze the data volume, each processing operation and the corresponding execution script of the data table by using a prediction model to obtain the estimated processing time of the data table for executing each processing operation; Add the estimated processing time of each data table in the data processing job for executing all processing operations in order according to the processing order to obtain the estimated total processing time of the data processing job.

3. The method of claim 2, wherein, The prediction model comprises multiple meta-models for predicting the processing time of different processing operations, and the training process of each meta-model comprises: Obtain multiple groups of training sample data, wherein each group of training sample data comprises the data volume of a data table to be executed by the big data platform at a historical time point, the execution script of a target processing operation as a training sample, and the actual processing time of the target processing operation executed by the data table as a sample label corresponding to the training sample; Construct a neural network model; Iteratively train the neural network model by using multiple groups of training sample data to obtain a meta-model for predicting the processing time of the target processing operation.

4. The method of claim 2, wherein, After obtaining the estimated processing time of each data table for executing each processing operation, the method further comprises: Determine a target data table for executing a calculation operation in the data processing job, and parse the execution script when the target data table executes the calculation operation to obtain multiple keywords; Determine the complexity of the target data table for executing the calculation operation according to the keywords, and multiply the complexity by the estimated processing time of the target data table for executing the calculation operation to obtain a target estimated processing time of the target data table for executing the calculation operation.

5. The method of claim 4, wherein, The type of the keyword comprises at least one of a join operation type, an aggregation operation type, and a window operation type, wherein the complexity of the target data table in performing the calculation operation is determined according to the keyword, and the complexity of the target data table in performing the calculation operation comprises: When the type of the keyword is the join operation type, the number of rows of an associated data table of the target data table is obtained from a preset database, and a first multiple of a sum of the number of rows of the target data table and the number of rows of the associated data table is taken as the complexity of the target data table in performing the calculation operation, wherein the preset database stores at least the number of rows of each data table; When the type of the keyword is the aggregation operation type, a second multiple of the number of rows of the target data table is taken as the complexity of the target data table in performing the calculation operation; When the type of the keyword is the window operation type, a third multiple of the number of rows of the target data table is taken as the complexity of the target data table in performing the calculation operation; Wherein the first multiple is greater than the second multiple, and the second multiple is greater than the third multiple.

6. The method of claim 1, wherein, The execution priority of each data processing job is determined according to the estimated total processing time of the data processing job, the job deadline, the job level, and the job type, and the execution priority of each data processing job comprises: For each data processing job, a timeliness coefficient of the data processing job is determined according to a difference between a sum of the estimated total processing time of the data processing job and a preset time point and the job deadline, a business weight of the data processing job is determined according to the job level, a resource peak-shaving weight of the data processing job is determined according to the job type and the calculation resource parameter, and the execution priority of the data processing job is obtained by multiplying a sum of the business weight and the resource peak-shaving weight by the timeliness coefficient.

7. The method of claim 1, wherein, The job type comprises a processing calculation type or a data export type, and the calculation resource parameter comprises a CPU occupancy rate and an IO waiting rate, wherein the resource peak-shaving weight of the data processing job is determined according to the job type and the calculation resource parameter, and the resource peak-shaving weight of the data processing job comprises: In a case where the CPU occupancy rate is lower than a second threshold value, the resource peak-shaving weight of the data processing job of the processing calculation type is set to a first value; In a case where the CPU occupancy rate is not lower than the second threshold value, the resource peak-shaving weight of the data processing job of the processing calculation type is set to a second value, wherein the second value is smaller than the first value; In a case where the IO waiting rate is lower than a third threshold value, the resource peak-shaving weight of the data processing job of the data export type is set to a third value; In a case where the IO waiting rate is not lower than the third threshold value, the resource peak-shaving weight of the data processing job of the data export type is set to a fourth value, wherein the third value is smaller than the fourth value.

8. A data processing job execution management apparatus characterized by comprising: The method comprises: An obtaining module is configured to obtain calculation resource parameters of a big data platform at a preset time point and job information of a plurality of data processing jobs, wherein the job information comprises job parameters, a job level, a job type, and a job deadline. an analysis module configured to determine, for each data processing job, an estimated total processing time of the data processing job according to job parameters of the data processing job; a first management module configured to, when the computing resource parameter is not lower than a first threshold, control the big data platform to execute a plurality of the data processing jobs in a sequence from low to high according to the estimated total processing time; a second management module configured to, when the computing resource parameter is lower than the first threshold, determine an execution priority of each data processing job according to the estimated total processing time, a job deadline, a job level and a job type of the data processing job, and control the big data platform to execute a plurality of the data processing jobs in a sequence from high to low according to the execution priority.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, controls a device in which the computer-readable storage medium is located to perform the data processing job execution management method of any one of claims 1 to 7.

10. An electronic device, comprising: comprise: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a program for executing the data processing job execution management method of any one of claims 1 to 7.

Images