Big data task scheduling model training method and device, equipment and storage medium
By generating a multi-layer neural network model and training it using preprocessing and classification rules, the problem of low hardware resource utilization was solved, achieving efficient scheduling of hardware resources and cost reduction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2022-08-15
- Publication Date
- 2026-06-02
AI Technical Summary
The unreasonable allocation of hardware resources in existing technologies leads to low overall utilization and high computing costs.
By generating a multi-layer neural network model, processing big data task sample data using preprocessing rules and preset classification rules, training the model with a loss function, and generating scheduling information to optimize hardware resource scheduling.
It improved the overall utilization of hardware resources and reduced computing costs.
Smart Images
Figure CN115328631B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data engines, and in particular to a method, apparatus, device, and storage medium for training a big data task scheduling model. Background Technology
[0002] As the demand for big data processing tasks continues to grow, hardware resources are also increasing. However, the sheer volume of hardware resources results in high computing costs. Moreover, these hardware resources are often configured based on peak demand, leading to low overall utilization.
[0003] Therefore, it is necessary to find a new training method for big data task scheduling models to obtain a big data task scheduling model that can generate scheduling information. Through scheduling information, big data tasks can be scheduled reasonably, improving the overall utilization of hardware resources and reducing computing costs. Summary of the Invention
[0004] Therefore, it is necessary to provide a big data task scheduling model training method, apparatus, device, and storage medium to address the above-mentioned technical problems, so as to obtain a big data task scheduling model that can generate scheduling information, schedule big data tasks through scheduling information, improve the overall utilization of hardware resources, and reduce computing costs.
[0005] A method for training a big data task scheduling model, comprising:
[0006] Obtain sample data for big data tasks;
[0007] The big data task sample data is processed by preprocessing rules to generate model input parameters for a multi-layer neural network model; the big data task sample data is processed by preset classification rules to generate feature classification data.
[0008] Configure the output layer of the multi-layer neural network model based on the feature classification data;
[0009] Obtain a pre-configured loss function, and train the multi-layer neural network model based on the loss function and the model input parameters;
[0010] Calculate the accuracy of the trained multilayer neural network model;
[0011] When the accuracy of the multilayer neural network model meets the preset requirements, the trained multilayer neural network model is determined as a big data task scheduling model. The big data task scheduling model is used to process the task information of big data tasks and generate the scheduling information of the big data tasks.
[0012] A big data task scheduling model training device, comprising:
[0013] The sample acquisition module is used to acquire sample data for big data tasks.
[0014] The preprocessing module is used to process the big data task sample data according to preprocessing rules to generate model input parameters for a multi-layer neural network model; and to process the big data task sample data according to preset classification rules to generate feature classification data.
[0015] The output layer setting module is used to configure the model output layer of the multi-layer neural network model according to the feature classification data;
[0016] The model training module is used to obtain a pre-configured loss function and train the multi-layer neural network model based on the loss function and the model input parameters.
[0017] An accuracy calculation module is used to calculate the accuracy of the trained multilayer neural network model.
[0018] The model generation module is used to determine the trained multi-layer neural network model as a big data task scheduling model when the accuracy of the multi-layer neural network model meets the preset requirements. The big data task scheduling model is used to process the task information of big data tasks and generate the scheduling information of the big data tasks.
[0019] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-described big data task scheduling model training method when executing the computer-readable instructions.
[0020] One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the big data task scheduling model training method described above.
[0021] The aforementioned big data task scheduling model training method, apparatus, computer equipment, and storage medium process big data task sample data through preprocessing rules to generate model input parameters for a multi-layer neural network model. After processing with preset classification rules, feature classification data for configuring the model's output layer is generated, increasing the adaptability of the neural model to big data tasks. Training the multi-layer neural network model using a loss function may improve the model's feature extraction capability. Calculating the accuracy of the trained multi-layer neural network model allows for its validation, ensuring its predictive accuracy. Finally, a big data task scheduling model is generated. The scheduling information generated by this model can improve the overall utilization of hardware resources and reduce computational costs. Attached Figure Description
[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a schematic diagram of an application environment for a big data task scheduling model training method according to an embodiment of the present invention;
[0024] Figure 2 This is a flowchart illustrating a big data task scheduling model training method in one embodiment of the present invention;
[0025] Figure 3 This is a schematic diagram of a big data task scheduling model training device in one embodiment of the present invention;
[0026] Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation
[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0028] The big data task scheduling model training method provided in this embodiment can be applied to, for example, Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.
[0029] In one embodiment, such as Figure 2 As shown, a method for training a big data task scheduling model is provided, and this method is applied to... Figure 1 Taking the server side as an example, the explanation includes the following steps S10-S60.
[0030] S10. Obtain sample data for big data tasks;
[0031] S20. Process the big data task sample data according to preprocessing rules to generate model input parameters for a multi-layer neural network model; process the big data task sample data according to preset classification rules to generate feature classification data;
[0032] S30. Configure the output layer of the multi-layer neural network model according to the feature classification data.
[0033] Understandably, the big data task sample data includes task data from multiple historical big data tasks. Here, task data includes, but is not limited to, task name, planned execution time, frequency, execution duration, task priority, task dependencies, and resource consumption.
[0034] Preprocessing rules can be set according to actual needs. Preprocessing rules include, but are not limited to, data cleaning rules and data standardization rules. Among them, data cleaning rules can deduplicate sample data for big data tasks and remove outliers; data standardization rules can standardize data from different units, such as unifying data from 12-hour format to 24-hour format if some time data uses 12-hour format and others use 24-hour format.
[0035] After preprocessing according to rules, the sample data from big data tasks can be directly input into the input layer of a multilayer neural network model as input parameters. A multilayer neural network model can be an extension of the perceptron model. Specifically, multiple hidden layers can be added to the perceptron to increase the model's expressive power.
[0036] Preset classification rules can be set according to actual needs. These rules can classify large-scale task sample data, generating feature classification data. This feature classification data can involve execution time, task dependencies, resource consumption, etc. The feature classification data can be used to configure the output layer of a multi-layer neural network model.
[0037] S40. Obtain the pre-configured loss function, and train the multi-layer neural network model according to the loss function and the model input parameters.
[0038] Understandably, when training a neural network model multiple times, gradient descent and backpropagation algorithms can be used, combined with a pre-configured loss function, to iteratively calculate the model input parameters within the multi-layer network model. Specifically, after the model input parameters are input into the multi-layer network model, the predicted output value is output through the model's output layer. This predicted output value is then processed by the loss function to obtain the loss value. Based on this loss value, parameter compensation is performed on the multi-layer network model, and then the next round of training begins.
[0039] S50. Calculate the accuracy of the trained multilayer neural network model.
[0040] Understandably, after a sufficient number of iterations (which can be defined as needed), the trained multilayer neural network model can be tested using validation samples to calculate its accuracy. Here, the validation samples can be samples from a large data task that differ from the large data task sample data.
[0041] S60. When the accuracy of the multi-layer neural network model meets the preset requirements, the trained multi-layer neural network model is determined as a big data task scheduling model. The big data task scheduling model is used to process the task information of big data tasks and generate the scheduling information of the big data tasks.
[0042] Understandably, preset requirements can be set according to actual needs, such as an accuracy rate greater than a certain threshold (e.g., 95%). The trained multi-layer neural network model can be designated as a big data task scheduling model. Here, the big data task scheduling model is used to process task information for big data tasks to be executed and generate scheduling information for these tasks. The scheduling information generated by the big data task scheduling model can improve the overall utilization of hardware resources and reduce computational costs.
[0043] In this embodiment, the big data task sample data is processed by preprocessing rules to generate model input parameters for a multi-layer neural network model. After processing by preset classification rules, feature classification data for configuring the model's output layer is generated, increasing the adaptability of the neural model to the big data task. Training the multi-layer neural network model using a loss function may improve its feature extraction capability. Calculating the accuracy of the trained multi-layer neural network model allows for its validation, ensuring its predictive accuracy. Finally, a big data task scheduling model is generated. The scheduling information generated by this model can improve the overall utilization of hardware resources and reduce computational costs.
[0044] Optionally, step S40, namely obtaining the pre-configured loss function and training the multi-layer neural network model based on the loss function and the model input parameters, includes:
[0045] S401. Input the model input parameters into the multilayer neural network model, and obtain the first output data output by the multilayer neural network model;
[0046] S402. Input the first output data into the loss function to obtain the first loss value;
[0047] S403. Adjust the model parameters of the multilayer neural network model according to the first loss value.
[0048] Understandably, during the training of a multilayer neural network model, the model input parameters are fed into the model to obtain the first output data of the output layer (first round of training). Then, a loss function is used to process the first output data, generating a first loss value (first round of training). The model parameters are then adjusted based on this first loss value. The model input parameters are then fed back into the adjusted multilayer neural network model to generate new second output data and a second loss value (second round of training). Through iterative training, when the number of iterations reaches a preset threshold, the trained multilayer neural network model can be obtained.
[0049] This embodiment uses model input parameters and loss function to train a multi-layer neural network model, which can improve the accuracy of the multi-layer neural network model.
[0050] Optionally, the loss function includes:
[0051]
[0052] Where Loss is the loss value;
[0053] y is the label value of the input parameter of the model;
[0054] The weighting values are those of the multilayer neural network model.
[0055] Understandably, a loss function measures the difference between the predicted output (label value) and the target output (weighted value). Here, the loss function can be the cross-entropy loss function.
[0056] In the loss function, y represents the label value of the model input parameters, which can specifically be the execution time and plan output by the model output layer. This refers to the weighting value in a multi-layer neural network model. The magnitude of the weighting value directly affects the number of nodes the scheduling server has within the consistency ring (hash ring). A larger number of nodes increases the probability of being assigned a node, thus achieving better load balancing. The consistency ring is a computing resource allocation algorithm that distributes data as evenly as possible across all computing nodes, avoiding uneven distribution of computing resources.
[0057] This embodiment adjusts the relationship between label values and weighted values using the cross-entropy loss function to achieve better allocation of computing resources.
[0058] Optionally, step S50, namely calculating the accuracy of the trained multilayer neural network model, includes:
[0059] S501. Obtain verification sample data;
[0060] S502. Use the verification sample data to input the trained multilayer neural network model to obtain the verification prediction result;
[0061] S503. Determine the accuracy rate based on the verification prediction results and the actual results of the verification sample data.
[0062] Understandably, validation sample data can be different from the big data task sample data. When multiple sample data are obtained, a portion of the big data task sample data can be designated as big data task sample data, while another portion can be designated as validation sample data.
[0063] By inputting validation sample data into the trained multilayer neural network model, validation prediction results can be generated. These prediction results can then be compared with the actual results from the validation sample data to obtain the accuracy of the multilayer neural network model.
[0064] In this embodiment, validation sample data is used to validate the multilayer neural network model, which can prevent overfitting of the multilayer neural network model and obtain the accuracy of the model.
[0065] Optionally, the preprocessing rules include feature normalization rules and feature selection rules;
[0066] Step S20, namely, processing the big data task sample data through preprocessing rules to generate model input parameters for the multi-layer neural network model, includes:
[0067] S201. Extract features from the big data task sample data according to the feature normalization rules to generate a standardized feature set;
[0068] S202. Extract features from the standardized feature set according to the feature selection rules to generate the model input parameters.
[0069] In essence, feature normalization rules refer to the rules that standardize the individual data points in a big data task sample data set. Feature normalization rules encompass normalization rules for various types of data, and can standardize all data points within the big data task sample data. After processing by feature normalization rules, the big data task sample data can generate a normalized feature set. Here, the normalized feature set contains a large amount of feature data.
[0070] Feature selection rules can be a type of feature query rule, such as random search rules, heuristic search rules, and complete search rules. By using feature selection rules to extract features from a standardized feature set, a subset of the standardized feature set is generated, which becomes the model input parameters. Feature selection rules can reduce the feature dimensionality of the model input parameters, improving the efficiency and accuracy of model training.
[0071] In this embodiment, feature normalization rules can normalize big data task sample data; feature selection rules can reduce the feature dimension of model input parameters and improve the efficiency and accuracy of model training.
[0072] Optionally, the multilayer neural network model includes several hidden layers;
[0073] Step S401, namely, inputting the model input parameters into the multilayer neural network model and obtaining the first output data of the multilayer neural network model, includes:
[0074] S4011. Extract the input parameters of the hidden layer from the input parameters of the model. The input parameters include task plan execution time, historical trend of task resource consumption, and queue utilization rate.
[0075] S4012. Extract the constraints of the hidden layer from the model input parameters. The constraints include indicator time, cluster resources, and task dependencies.
[0076] S4013. Determine the overall resource consumption of the cluster based on the input parameters and the constraints.
[0077] S4014. Input the overall resource consumption of the cluster into the activation function to generate the input value of the model output layer.
[0078] Understandably, a multi-layer neural network model includes several hidden layers. Parameters such as task execution time, historical trends in task resource consumption, and queue utilization can be extracted from the model's input parameters and used as input parameters for the hidden layers. Indicator time, cluster resources, and task dependencies are set as constraints for the hidden layers. The input parameters and constraints are calculated by the hidden layers to generate the overall cluster resource consumption. Processing the overall cluster resource consumption using activation functions yields the input values for the model's output layer.
[0079] This embodiment uses hidden layers and activation functions to transform model input parameters into input values for the model output layer.
[0080] Optionally, the excitation function includes:
[0081]
[0082] Where f(x) is the input value of the output layer of the model;
[0083] x represents the total resource consumption of the cluster.
[0084] Understandably, a sigmoid curve can be used as the activation function here. The sigmoid curve can effectively adjust the parameters between layers in a multi-layer neural network model, improving the accuracy of the multi-layer neural network.
[0085] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0086] In one embodiment, a big data task scheduling model training device is provided, which corresponds one-to-one with the big data task scheduling model training method in the above embodiments. For example... Figure 3 As shown, the big data task scheduling model training device includes a sample acquisition module 10, a preprocessing module 20, an output layer setting module 30, a model training module 40, an accuracy calculation module 50, and a model generation module 60. Detailed descriptions of each functional module are as follows:
[0087] The sample acquisition module 10 is used to acquire sample data for big data tasks.
[0088] The preprocessing module 20 is used to process the big data task sample data according to preprocessing rules to generate model input parameters for a multi-layer neural network model; and to process the big data task sample data according to preset classification rules to generate feature classification data.
[0089] The output layer setting module 30 is used to configure the model output layer of the multi-layer neural network model according to the feature classification data;
[0090] The model training module 40 is used to obtain a pre-configured loss function and train the multi-layer neural network model according to the loss function and the model input parameters;
[0091] Accuracy calculation module 50 is used to calculate the accuracy of the trained multilayer neural network model;
[0092] The model generation module 60 is used to determine the trained multi-layer neural network model as a big data task scheduling model when the accuracy of the multi-layer neural network model meets the preset requirements. The big data task scheduling model is used to process the task information of big data tasks and generate the scheduling information of the big data tasks.
[0093] Optionally, the model training module 40 includes:
[0094] A first output data unit is used to input the model input parameters into the multilayer neural network model and obtain the first output data output by the multilayer neural network model.
[0095] A first loss value acquisition unit is used to input the first output data into the loss function to obtain a first loss value;
[0096] The parameter adjustment unit is used to adjust the model parameters of the multilayer neural network model according to the first loss value.
[0097] Optionally, the loss function includes:
[0098]
[0099] Where Loss is the loss value;
[0100] y is the label value of the input parameter of the model;
[0101] The weighting values are those of the multilayer neural network model.
[0102] Optionally, the accuracy calculation module 50 includes:
[0103] The verification data acquisition unit is used to acquire verification sample data.
[0104] The verification prediction result acquisition unit is used to input the verification sample data into the trained multilayer neural network model to obtain the verification prediction result;
[0105] An accuracy generation unit is used to determine the accuracy based on the validation prediction results and the actual results of the validation sample data.
[0106] Optionally, the preprocessing rules include feature normalization rules and feature selection rules;
[0107] Preprocessing module 20 includes:
[0108] A normalization unit is used to extract features from the big data task sample data according to the feature normalization rules and generate a normalized feature set.
[0109] The feature extraction unit is used to extract features from the standardized feature set according to the feature selection rules and generate the model input parameters.
[0110] Optionally, the multilayer neural network model includes several hidden layers;
[0111] The unit for acquiring the first output data includes:
[0112] The input parameter extraction unit is used to extract the input parameters of the hidden layer from the input parameters of the model. The input parameters include task plan execution time, historical trend of task resource consumption, and queue utilization rate.
[0113] The constraint extraction unit is used to extract the constraints of the hidden layer from the model input parameters. The constraints include index time, cluster resources, and task dependencies.
[0114] A resource consumption calculation unit is used to determine the overall resource consumption of the cluster based on the input parameters and the constraints.
[0115] The output layer input value generation unit is used to input the overall resource consumption of the cluster into the activation function to generate the input value of the model output layer.
[0116] Optionally, the excitation function includes:
[0117]
[0118] Where f(x) is the input value of the output layer of the model;
[0119] x represents the total resource consumption of the cluster.
[0120] Specific limitations regarding the training device for the big data task scheduling model can be found in the limitations on the training method for the big data task scheduling model mentioned above, and will not be repeated here. Each module in the aforementioned big data task scheduling model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0121] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database stores data related to a big data task scheduling model training method. The network interface communicates with external terminals via a network connection. When the computer-readable instructions are executed by the processor, they implement a big data task scheduling model training method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.
[0122] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:
[0123] Obtain sample data for big data tasks;
[0124] The big data task sample data is processed by preprocessing rules to generate model input parameters for a multi-layer neural network model; the big data task sample data is processed by preset classification rules to generate feature classification data.
[0125] Configure the output layer of the multi-layer neural network model based on the feature classification data;
[0126] Obtain a pre-configured loss function, and train the multi-layer neural network model based on the loss function and the model input parameters;
[0127] Calculate the accuracy of the trained multilayer neural network model;
[0128] When the accuracy of the multilayer neural network model meets the preset requirements, the trained multilayer neural network model is determined as a big data task scheduling model. The big data task scheduling model is used to process the task information of big data tasks and generate the scheduling information of the big data tasks.
[0129] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:
[0130] Obtain sample data for big data tasks;
[0131] The big data task sample data is processed by preprocessing rules to generate model input parameters for a multi-layer neural network model; the big data task sample data is processed by preset classification rules to generate feature classification data.
[0132] Configure the output layer of the multi-layer neural network model based on the feature classification data;
[0133] Obtain a pre-configured loss function, and train the multi-layer neural network model based on the loss function and the model input parameters;
[0134] Calculate the accuracy of the trained multilayer neural network model;
[0135] When the accuracy of the multilayer neural network model meets the preset requirements, the trained multilayer neural network model is determined as a big data task scheduling model. The big data task scheduling model is used to process the task information of big data tasks and generate the scheduling information of the big data tasks.
[0136] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0137] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0138] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A training method for a big data task scheduling model, characterized in that, include: Obtain big data task sample data, which includes task name, planned execution time, frequency, execution duration, task priority, task dependencies, and resource consumption. The big data task sample data is processed by preprocessing rules to generate model input parameters for a multi-layer neural network model; the big data task sample data is processed by preset classification rules to generate feature classification data. The preprocessing rules include feature normalization rules and feature selection rules. The feature normalization rules are used to standardize all data in the big data task sample data, and the feature selection rules are used to extract features. The feature selection rules include random search rules, heuristic search rules, and complete search rules. Configure the output layer of the multi-layer neural network model based on the feature classification data; Obtain a pre-configured loss function, which measures the difference between the label value of the model input parameter and the weighted value of the multilayer neural network model, and the weighted value is used to adjust the number of nodes of the scheduling server in the consistency loop; The multilayer neural network model is trained based on the loss function and the model input parameters; Calculate the accuracy of the trained multilayer neural network model; When the accuracy of the multilayer neural network model meets the preset requirements, the trained multilayer neural network model is determined as a big data task scheduling model. The big data task scheduling model is used to process the task information of big data tasks and generate the scheduling information of the big data tasks.
2. The big data task scheduling model training method as described in claim 1, characterized in that, The step of obtaining a pre-configured loss function and training the multi-layer neural network model based on the loss function and the model input parameters includes: Input the model input parameters into the multi-layer neural network model, and obtain the first output data of the multi-layer neural network model; The first output data is input into the loss function to obtain the first loss value; The model parameters of the multilayer neural network model are adjusted based on the first loss value.
3. The big data task scheduling model training method as described in claim 2, characterized in that, The loss function includes: Where Loss is the loss value; y is the label value of the input parameter of the model; The weighting values are those of the multilayer neural network model.
4. The big data task scheduling model training method as described in claim 1, characterized in that, The calculation of the accuracy of the multilayer neural network model after training includes: Obtain validation sample data; The validation sample data is used as input to the trained multi-layer neural network model to obtain validation prediction results. The accuracy rate is determined based on the verification prediction results and the actual results of the verification sample data.
5. The big data task scheduling model training method as described in claim 1, characterized in that, The process of processing the big data task sample data through preprocessing rules to generate model input parameters for a multi-layer neural network model includes: Based on the feature normalization rules, feature extraction is performed on the big data task sample data to generate a standardized feature set; Based on the feature selection rules, feature extraction is performed on the standardized feature set to generate the model input parameters.
6. The big data task scheduling model training method as described in claim 2, characterized in that, The multi-layer neural network model includes several hidden layers; The step of inputting the model input parameters into the multilayer neural network model and obtaining the first output data of the multilayer neural network model includes: The input parameters of the hidden layer are extracted from the input parameters of the model. The input parameters include task plan execution time, historical trend of task resource consumption, and queue utilization rate. The constraints of the hidden layer are extracted from the input parameters of the model. The constraints include indicator time, cluster resources, and task dependencies. The overall resource consumption of the cluster is determined based on the input parameters and the constraints. The total resource consumption of the cluster is input into the activation function to generate the input value of the model output layer.
7. The big data task scheduling model training method as described in claim 6, characterized in that, The activation function includes: in, The input values for the output layer of the model; x represents the total resource consumption of the cluster.
8. A training device for a big data task scheduling model, characterized in that, include: The sample acquisition module is used to acquire big data task sample data, which includes task name, planned execution time, frequency, execution duration, task priority, task dependency, and resource consumption. The preprocessing module is used to process the big data task sample data according to preprocessing rules to generate model input parameters for a multi-layer neural network model; and to process the big data task sample data according to preset classification rules to generate feature classification data. The preprocessing rules include feature normalization rules and feature selection rules. The feature normalization rules are used to standardize all data in the big data task sample data, and the feature selection rules are used to extract features. The feature selection rules include random search rules, heuristic search rules, and complete search rules. The output layer setting module is used to configure the model output layer of the multi-layer neural network model according to the feature classification data; The model training module is used to obtain a pre-configured loss function, which measures the difference between the label values of the model input parameters and the weighted values of the multi-layer neural network model. The weighted values are used to adjust the number of nodes in the scheduling server within the consistency loop. The multilayer neural network model is trained based on the loss function and the model input parameters; An accuracy calculation module is used to calculate the accuracy of the trained multilayer neural network model. The model generation module is used to determine the trained multi-layer neural network model as a big data task scheduling model when the accuracy of the multi-layer neural network model meets the preset requirements. The big data task scheduling model is used to process the task information of big data tasks and generate the scheduling information of the big data tasks.
9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the big data task scheduling model training method as described in any one of claims 1 to 7.
10. One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the big data task scheduling model training method as described in any one of claims 1 to 7.