Real-time big data task diagnosis and optimization method and device
By using a serial structure diagnostic and optimization model in big data tasks, combined with random forest and gradient enhancement tree model, the problem of inefficient diagnosis and optimization of big data tasks is solved, and real-time and efficient diagnostic and optimization strategy generation is achieved.
Patent Information
- Application Number
- CN202411356571.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-27
AI Technical Summary
The existing technology cannot effectively solve the problems of inefficient diagnosis and optimization of real-time big data tasks and insufficient accuracy in the field of big data, and cannot meet the requirements of real-time and efficiency.
The big data task diagnosis and optimization model adopts a serial structure. By obtaining task operation data in real time and training with historical data sets, a random forest model and gradient enhancement tree model is built, and the dynamic weight cross-validation method is used for model training and optimization to achieve Real-time diagnostics and optimization strategies generation.
It significantly improves the efficiency of real-time big data task diagnosis and optimization, and can quickly generate accurate diagnostic results and optimization strategies to meet the real-time and efficient needs of big data tasks.
Smart Images
Figure CN119204269A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data technology, and in particular to a real-time big data task diagnosis and optimization method and device. Background Art
[0002] In the field of big data, with the continuous growth of data volume and the complexity of data processing tasks, data tasks may encounter various problems during execution, including but not limited to slow task execution due to insufficient task performance, task errors, data quality monitoring rule errors, and uneven resource allocation. At present, the diagnosis and optimization of big data tasks mainly rely on manual analysis and experience judgment. This method is inefficient and difficult to guarantee accuracy, and cannot meet the real-time and high-efficiency requirements of big data tasks.
[0003] In summary, how to improve the efficiency of real-time big data task diagnosis and optimization is a problem that technical personnel in this field need to solve urgently. Summary of the invention
[0004] In view of this, the present application provides a real-time big data task diagnosis and optimization method and device, aiming to improve the efficiency of real-time big data task diagnosis and optimization.
[0005] In a first aspect, the present application provides a real-time big data task diagnosis and optimization method, comprising:
[0006] During the execution of the big data task, the task running data of the big data task is obtained in real time;
[0007] The task running data is input into a big data task diagnosis and optimization model to obtain a diagnosis result and an optimization strategy for the big data task; the big data task diagnosis and optimization model is a serial structure model; the big data task diagnosis and optimization model includes a big data task diagnosis model and a big data task optimization model.
[0008] Optionally, before inputting the task running data into the big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task, the method further includes:
[0009] Acquire a historical data set; the historical data set includes historical operation data of the big data task, diagnostic data of the historical operation task, historical optimization strategy of the big data task, and operation comparison data before and after the implementation of the historical optimization strategy;
[0010] The historical data set is used to train the big data task diagnosis and optimization model.
[0011] Optionally, before using the historical data set to train the big data task diagnosis and optimization model, the method further includes:
[0012] Construct big data task diagnosis model and big data task optimization model;
[0013] The big data task diagnosis model and the big data task optimization model are serially constructed into a big data task diagnosis and optimization model.
[0014] Optionally, the building of the big data task diagnosis model and the big data task optimization model includes:
[0015] Constructing a random forest model for diagnosing the performance of a big data task; the input of the random forest model for diagnosing the performance of a big data task is the task running data of the big data task; the output of the random forest model for diagnosing the performance of a big data task is the diagnosis result;
[0016] A gradient boosting tree model for generating an optimization strategy is constructed, wherein the input of the gradient boosting tree model for generating an optimization strategy is a diagnosis result; and the output of the gradient boosting tree model for generating an optimization strategy is an optimization strategy category.
[0017] Optionally, the step of serially constructing a big data task diagnosis and optimization model using the big data task diagnosis model and the big data task optimization model includes:
[0018] The output of the random forest model for diagnosing the performance of big data tasks is used as the input of the gradient boosting tree model for generating optimization strategies to achieve serial construction of the big data task diagnosis and optimization model.
[0019] Optionally, after obtaining the historical data set, the method further includes:
[0020] Cleaning the data in the historical data set to obtain a cleaned historical data set;
[0021] Feature extraction and normalization are performed on the data in the cleaned historical data set to obtain a processed historical data set.
[0022] Optionally, the using the historical data set to train the big data task diagnosis and optimization model includes:
[0023] Using a dynamic weight cross-validation method and the historical data set, respectively training the big data task diagnosis model and the big data task optimization model;
[0024] The dynamic weight cross-validation method and the historical data set are used to train a big data task diagnosis and optimization model using the historical data set.
[0025] Optionally, after inputting the task running data into a big data task diagnosis and optimization model to obtain a diagnosis result and an optimization strategy for the big data task, the method further includes:
[0026] Saving the diagnosis result, the optimization strategy, and the operation data before and after the execution of the optimization strategy;
[0027] The big data task diagnosis and optimization model is iterated using the diagnosis result, the optimization strategy, and the operating data before and after the execution of the optimization strategy.
[0028] In a second aspect, the present application provides a real-time big data task diagnosis and optimization device, comprising:
[0029] A first acquisition module is used to acquire task running data of the big data task in real time during the execution of the big data task;
[0030] An input module is used to input the task operation data into a big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task; the big data task diagnosis and optimization model is a serial structure model; the big data task diagnosis and optimization model includes a big data task diagnosis model and a big data task optimization model.
[0031] Optionally, the device further comprises:
[0032] The second acquisition module is used to acquire a historical data set; the historical data set includes historical operation data of the big data task, diagnostic data of the historical operation task, historical optimization strategy of the big data task, and operation comparison data before and after the implementation of the historical optimization strategy;
[0033] The training module is used to train the big data task diagnosis and optimization model using the historical data set.
[0034] Optionally, the device further comprises:
[0035] The first building module is used to build a big data task diagnosis model and a big data task optimization model;
[0036] The second construction module is used to construct a big data task diagnosis and optimization model by serially combining the big data task diagnosis model and the big data task optimization model.
[0037] Optionally, the first building block includes:
[0038] A first construction unit is used to construct a random forest model for diagnosing the performance of a big data task; the input of the random forest model for diagnosing the performance of a big data task is the task running data of the big data task; the output of the random forest model for diagnosing the performance of a big data task is a diagnosis result;
[0039] The second construction unit is used to construct a gradient boosting tree model for generating an optimization strategy, wherein the input of the gradient boosting tree model for generating an optimization strategy is a diagnosis result; and the output of the gradient boosting tree model for generating an optimization strategy is an optimization strategy category.
[0040] Optionally, the second building block includes:
[0041] The third construction unit is used to use the output of the random forest model for diagnosing the performance of big data tasks as the input of the gradient boosting tree model for generating optimization strategies, so as to realize the serial construction of the big data task diagnosis and optimization model.
[0042] Optionally, the device further comprises:
[0043] A data cleaning module, used to clean the data in the historical data set to obtain a cleaned historical data set;
[0044] The processing module is used to perform feature extraction and normalization processing on the data in the cleaned historical data set to obtain a processed historical data set.
[0045] Optionally, the training module includes:
[0046] A first training unit is used to train the big data task diagnosis model and the big data task optimization model respectively by using a dynamic weight cross-validation method and the historical data set;
[0047] The second training module is used to train the big data task diagnosis and optimization model on the historical data set by using the dynamic weight cross-validation method and the historical data set.
[0048] Optionally, the device further comprises:
[0049] A storage module, used for storing the diagnosis result, the optimization strategy and the operation data before and after the execution of the optimization strategy;
[0050] An iteration module is used to iterate the big data task diagnosis and optimization model using the diagnosis result, the optimization strategy, and the operating data before and after the execution of the optimization strategy.
[0051] The present application provides a real-time big data task diagnosis and optimization method. When executing the method, first, in the process of executing the big data task, the task operation data of the big data task is obtained in real time, and then the task operation data is input into the big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task. Among them, the big data task diagnosis and optimization model is a serial structure model, including a big data task diagnosis model and a big data task optimization model. In this way, by inputting the task operation data into the big data task diagnosis and optimization model, the diagnosis result and optimization strategy of the big data task can be directly obtained, which can greatly improve the efficiency of real-time big data task diagnosis and optimization. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0053] Figure 1 A flow chart of a real-time big data task diagnosis and optimization method provided in an embodiment of the present application;
[0054] Figure 2 A schematic diagram of the structure of a real-time big data task diagnosis and optimization device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. The present application provides a real-time big data task diagnosis and optimization method and device for use in the field of big data technology. The above is only an example and does not limit the application field of the method and device name provided in the present application.
[0056] In the field of big data, with the continuous growth of data volume and the complexity of data processing tasks, data tasks may encounter various problems during execution, including but not limited to slow task execution due to insufficient task performance, task errors, data quality monitoring rule errors, and uneven resource allocation. At present, the diagnosis and optimization of big data tasks mainly rely on manual analysis and experience judgment. This method is inefficient and difficult to guarantee accuracy, and cannot meet the real-time and high-efficiency requirements of big data tasks.
[0057] After research, the inventor proposed the technical solution of this application. First, during the execution of the big data task, the task operation data of the big data task is obtained in real time. Then, the task operation data is input into the big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task. Among them, the big data task diagnosis and optimization model is a serial structure model, including a big data task diagnosis model and a big data task optimization model. In this way, by inputting the task operation data into the big data task diagnosis and optimization model, the diagnosis result and optimization strategy of the big data task can be directly obtained, which can greatly improve the efficiency of real-time big data task diagnosis and optimization.
[0058] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. It should be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.
[0059] See also Figure 1 , Figure 1 A flow chart of a real-time big data task diagnosis and optimization method provided in an embodiment of the present application includes:
[0060] S101: During the execution of a big data task, task running data of the big data task is obtained in real time.
[0061] During the execution of the big data task, the task running data of the big data task is collected in real time.
[0062] S102: Input the task running data into the big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task.
[0063] The task operation data collected in real time is input into the big data task diagnosis and optimization model to obtain the diagnosis results and optimization strategies of the big data task. Among them, the big data task diagnosis and optimization model is constructed in series by the big data task diagnosis model and the big data task optimization model.
[0064] Before executing step S102, it is necessary to first build and train a big data task diagnosis and optimization model. The details are as follows:
[0065] First, collect the historical data of the big data task to obtain the historical data set. The historical data includes the log data of the big data task collected by means of technical metadata, etc. The log data of the big data task includes but is not limited to the execution time of the task, the resource usage of the task, the success of the task, etc. The data example is shown in Table 1:
[0066] Table 1
[0067] Task Number Success Task execution time CPU usage Memory usage ... Task 1 yes 1 hour 20 minutes 670 11776 Task 2 yes 50 minutes 400 7500 Task 3 no 25 minutes 280 3500
[0068] The historical data also includes different optimization strategies and the optimization effects of tasks before and after the optimization strategies are implemented. The data examples are shown in Table 2:
[0069] Table 2
[0070]
[0071]
[0072] After obtaining the historical data of the big data task, you can first perform data cleaning on the historical data. Specifically, the obtained historical data of the big data task is cleaned and preprocessed, including processing missing values, outliers, noise, etc. of the data.
[0073] Next, feature extraction may be performed on the cleaned data. Specifically, feature extraction and normalization may be performed on the cleaned data to ensure the quality and consistency of the data. The data includes but is not limited to execution time, CPU usage, memory usage, etc.
[0074] The specific method of constructing the big data task diagnosis and optimization model is as follows: First, construct a big data task diagnosis model, including: construct a random forest model, which is used to diagnose the performance problems of the task, with the extracted features as input and the labels of the diagnosed performance problems as output. Next, construct a big data task optimization model, including: construct a gradient boosting tree model, which is used to generate optimization suggestions, with the diagnosis results and related features as input and the categories of optimization suggestions as output. Finally, the big data task diagnosis model and the big data task optimization model are constructed serially to obtain the big data task diagnosis and optimization model. Specifically, the random forest model and the gradient boosting tree model are constructed into a serial structure, that is, the output of the random forest model is used as the input of the gradient boosting model. The gradient boosting model can adaptively select the most useful features for diagnosis and optimization, rather than relying on a preset feature set, and can have multiple optimization goals, such as reducing running time and reducing resource consumption at the same time.
[0075] The big data task diagnosis model and the big data task optimization model are trained and evaluated respectively. The samples selected for training the big data task diagnosis model and the big data task optimization model are all historical data of the big data task, that is, the historical data set is used as the training data set to train the above two models. It should be noted that the training and optimization methods of the big data task diagnosis model and the big data task optimization model are the same, and the training and optimization methods of the big data task diagnosis model and the big data task optimization model will not be explained separately below.
[0076] The specific methods for training and evaluating the big data task diagnosis model or the big data task optimization model are as follows:
[0077] (1) Define error costs: Determine the relative costs of different types of prediction and suggestion errors. In big data tasks, some incorrect diagnoses or incorrect optimization suggestions may have a greater impact on system performance or business after implementation. For example, if the task error is caused by data quality problems that trigger data quality verification rules, if the optimization suggestion is to set the task to success, it may result in erroneous data that affects the business.
[0078] (2) Dataset weighting: Assign a weight to each sample in the training dataset based on the error cost. The weight is based on the error cost of the sample or the importance of the sample. For example, when performing a task, the weight of a task reporting an error is greater than the weight of a task running slowly.
[0079] (3) Dynamic weight adjustment: In each fold of cross-validation, the sample weights are dynamically adjusted based on the model’s performance in that fold. For example, if the model performs poorly on a particular category, the weight of samples in that category can be increased.
[0080] (4) Model training: Use weighted data sets for model training. During the training process, the model pays more attention to samples with higher weights, thereby reducing the prediction error of these samples.
[0081] (5) Cross-validation: In the validation process of each fold, different weight configurations are used to evaluate the model performance. This can be achieved by changing the sample weights or through stratified sampling.
[0082] (6) Performance evaluation: Evaluate the performance of the model under different weight configurations, rather than just the average performance on the original dataset. This helps understand how the model performs in different situations.
[0083] Next, we need to complete the joint training of the big data task diagnosis and optimization model, that is, use the historical data set to input the separately trained random forest model. After the random model diagnoses the performance problem, the diagnosis result is used as the input of the gradient boosting tree model to train the gradient boosting tree model so that it can learn how to generate effective optimization suggestions based on the diagnosis result. Similarly, we also need to use the dynamic weight cross-validation method mentioned above to evaluate the effectiveness of the joint model.
[0084] In an embodiment of the present application, during the execution of a big data task, the task operation data of the big data task is first obtained in real time, and then the task operation data is input into the big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task. Among them, the big data task diagnosis and optimization model is a serial structure model, including a big data task diagnosis model and a big data task optimization model. In this way, by inputting the task operation data into the big data task diagnosis and optimization model, the diagnosis result and optimization strategy of the big data task can be directly obtained, which can greatly improve the efficiency of real-time big data task diagnosis and optimization.
[0085] In some implementations, the task diagnosis data, optimization suggestion data, and actual conditions before and after optimization predicted by the big data task diagnosis and optimization model can be retained to facilitate subsequent data backtracking and query. The algorithm model can also be iteratively optimized through these data to further improve the accuracy of data fluctuation verification, iteratively optimize the machine learning model, and improve the accuracy and efficiency of diagnosis.
[0086] The above are some specific implementations of the method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided in the embodiment of the present application will be introduced from the perspective of functional modularization.
[0087] See also Figure 2 , Figure 2 A schematic diagram of the structure of a real-time big data task diagnosis and optimization device provided in an embodiment of the present application, the real-time big data task diagnosis and optimization device 200 includes:
[0088] The first acquisition module 210 is used to acquire the task running data of the big data task in real time during the execution of the big data task;
[0089] The input module 220 is used to input the task operation data into the big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task; the big data task diagnosis and optimization model is a serial structure model; the big data task diagnosis and optimization model includes a big data task diagnosis model and a big data task optimization model.
[0090] Optionally, the device 200 further includes:
[0091] The second acquisition module is used to acquire a historical data set; the historical data set includes historical operation data of the big data task, diagnostic data of the historical operation task, historical optimization strategy of the big data task, and operation comparison data before and after the implementation of the historical optimization strategy;
[0092] The training module is used to train the big data task diagnosis and optimization model using the historical data set.
[0093] Optionally, the device 200 further includes:
[0094] The first building module is used to build a big data task diagnosis model and a big data task optimization model;
[0095] The second construction module is used to construct a big data task diagnosis and optimization model by serially combining the big data task diagnosis model and the big data task optimization model.
[0096] Optionally, the first building block includes:
[0097] A first construction unit is used to construct a random forest model for diagnosing the performance of a big data task; the input of the random forest model for diagnosing the performance of a big data task is the task running data of the big data task; the output of the random forest model for diagnosing the performance of a big data task is a diagnosis result;
[0098] The second construction unit is used to construct a gradient boosting tree model for generating an optimization strategy, wherein the input of the gradient boosting tree model for generating an optimization strategy is a diagnosis result; and the output of the gradient boosting tree model for generating an optimization strategy is an optimization strategy category.
[0099] Optionally, the second building block includes:
[0100] The third construction unit is used to use the output of the random forest model for diagnosing the performance of big data tasks as the input of the gradient boosting tree model for generating optimization strategies, so as to realize the serial construction of the big data task diagnosis and optimization model.
[0101] Optionally, the device 200 further includes:
[0102] A data cleaning module, used to clean the data in the historical data set to obtain a cleaned historical data set;
[0103] The processing module is used to perform feature extraction and normalization processing on the data in the cleaned historical data set to obtain a processed historical data set.
[0104] Optionally, the training module includes:
[0105] A first training unit is used to train the big data task diagnosis model and the big data task optimization model respectively by using a dynamic weight cross-validation method and the historical data set;
[0106] The second training module is used to train the big data task diagnosis and optimization model on the historical data set by using the dynamic weight cross-validation method and the historical data set.
[0107] Optionally, the device 200 further includes:
[0108] A storage module, used for storing the diagnosis result, the optimization strategy and the operation data before and after the execution of the optimization strategy;
[0109] An iteration module is used to iterate the big data task diagnosis and optimization model using the diagnosis result, the optimization strategy, and the operating data before and after the execution of the optimization strategy.
[0110] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.
[0111] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the method described in any embodiment of the present application.
[0112] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the method described in any embodiment of the present application.
[0113] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.
[0114] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0115] It should also be noted that the various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the equipment and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The device embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components prompted as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0116] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A real-time big data task diagnosis and optimization method, characterized in that: include: During the execution of the big data task, the task running data of the big data task is obtained in real time; The task running data is input into a big data task diagnosis and optimization model to obtain a diagnosis result and an optimization strategy for the big data task; the big data task diagnosis and optimization model is a serial structure model; the big data task diagnosis and optimization model includes a big data task diagnosis model and a big data task optimization model.
2. The method according to claim 1, characterized in that Before inputting the task operation data into the big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task, the method further includes: Acquire a historical data set; the historical data set includes historical operation data of the big data task, diagnostic data of the historical operation task, historical optimization strategy of the big data task, and operation comparison data before and after the implementation of the historical optimization strategy; The historical data set is used to train the big data task diagnosis and optimization model.
3. The method according to claim 2, characterized in that Before using the historical data set to train the big data task diagnosis and optimization model, the method further includes: Construct big data task diagnosis model and big data task optimization model; The big data task diagnosis model and the big data task optimization model are serially constructed into a big data task diagnosis and optimization model.
4. The method according to claim 3, characterized in that: The construction of the big data task diagnosis model and the big data task optimization model includes: Constructing a random forest model for diagnosing the performance of a big data task; the input of the random forest model for diagnosing the performance of a big data task is the task running data of the big data task; the output of the random forest model for diagnosing the performance of a big data task is the diagnosis result; A gradient boosting tree model for generating an optimization strategy is constructed, wherein the input of the gradient boosting tree model for generating an optimization strategy is a diagnosis result; and the output of the gradient boosting tree model for generating an optimization strategy is an optimization strategy category.
5. The method according to claim 3, characterized in that: The step of serially constructing a big data task diagnosis and optimization model using the big data task diagnosis model and the big data task optimization model includes: The output of the random forest model for diagnosing the performance of big data tasks is used as the input of the gradient boosting tree model for generating optimization strategies to achieve serial construction of the big data task diagnosis and optimization model.
6. The method according to claim 2, characterized in that After obtaining the historical data set, the method further includes: Cleaning the data in the historical data set to obtain a cleaned historical data set; Feature extraction and normalization are performed on the data in the cleaned historical data set to obtain a processed historical data set.
7. The method according to claim 2, characterized in that: The using of the historical data set to train the big data task diagnosis and optimization model includes: Using a dynamic weight cross-validation method and the historical data set, respectively training the big data task diagnosis model and the big data task optimization model; The dynamic weight cross-validation method and the historical data set are used to train a big data task diagnosis and optimization model using the historical data set.
8. The method according to claim 1, characterized in that: After inputting the task operation data into the big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task, the method further includes: Saving the diagnosis result, the optimization strategy, and the operation data before and after the execution of the optimization strategy; The big data task diagnosis and optimization model is iterated using the diagnosis result, the optimization strategy, and the operating data before and after the execution of the optimization strategy.
9. A real-time big data task diagnosis and optimization device, characterized in that: include: A first acquisition module is used to acquire task running data of the big data task in real time during the execution of the big data task; An input module is used to input the task operation data into a big data task diagnosis and optimization model to obtain the diagnosis result and optimization strategy of the big data task; the big data task diagnosis and optimization model is a serial structure model; the big data task diagnosis and optimization model includes a big data task diagnosis model and a big data task optimization model.
10. The device according to claim 9, characterized in that The device also includes: The second acquisition module is used to acquire a historical data set; the historical data set includes historical operation data of the big data task, diagnostic data of the historical operation task, historical optimization strategy of the big data task, and operation comparison data before and after the implementation of the historical optimization strategy; The training module is used to train the big data task diagnosis and optimization model using the historical data set.