Artificial intelligence-based task optimization method and related devices
By collecting, reducing, preprocessing, and clustering historical task data, a task classification model is trained, which solves the problem of low task optimization accuracy in existing technologies and achieves efficient and accurate task optimization results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA PING AN PROPERTY INSURANCE CO LTD
- Filing Date
- 2022-07-22
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies are insufficient to comprehensively and accurately analyze the strengths and weaknesses of enterprise data tasks, resulting in low accuracy of task optimization results.
By collecting multiple data related to historical tasks, performing dimensionality reduction and preprocessing, clustering and labeling sample data, training a task classification model, and using this model to classify and optimize the tasks to be evaluated.
It improved the accuracy and efficiency of task optimization, reduced the time complexity of data analysis, reduced the waste of human resources, and improved the accuracy of data analysis and the performance of models.
Smart Images
Figure CN115146865B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a task optimization method, apparatus, electronic device and storage medium based on artificial intelligence. Background Technology
[0002] As businesses diversify their operations, the number of big data tasks they undertake is also increasing. This has led to a surge in repetitive computational and development tasks, driving up data analysis costs. A key challenge is how to quickly identify and optimize unreasonable or redundant tasks within this massive workload.
[0003] Currently, companies typically develop metrics based on experience, such as complexity, link redundancy, and code efficiency, to reflect the quality of data analysis tasks. This approach is difficult to guarantee the comprehensiveness of the analysis, and the number of metrics used to measure the quality of tasks is limited, which leads to low accuracy of the analysis results. Summary of the Invention
[0004] In view of the above, it is necessary to provide an artificial intelligence-based task optimization method and related equipment to solve the technical problem of how to improve the accuracy of task optimization. The related equipment includes an artificial intelligence-based task optimization device, electronic equipment, and storage medium.
[0005] This application provides a task optimization method based on artificial intelligence, the method comprising:
[0006] Collect multiple data points related to historical tasks as raw data, and the raw data contains multiple dimensions;
[0007] The original data is subjected to dimensionality reduction processing to obtain dimensionality-reduced data and multiple target dimensions;
[0008] The dimensionality-reduced data is preprocessed to obtain sample data;
[0009] The sample data is clustered to obtain multiple sample groups, and the sample data in each sample group is labeled to obtain label data;
[0010] A task classification model is trained based on the sample data and the label data;
[0011] Based on the target dimension, collect the data to be evaluated related to the task to be evaluated, input the data to be evaluated into the task classification model to obtain the classification result, and optimize the task to be evaluated based on the classification result.
[0012] The aforementioned AI-based task optimization method constructs multi-dimensional raw data by collecting various data related to historical tasks. It then performs dimensionality reduction and preprocessing on the raw data to obtain sample data, thereby improving data quality. Furthermore, it labels the sample data to obtain corresponding tag data. The sample data and tag data are used to train a task classification model, which is then used to classify the task to be evaluated. Based on the classification results, the task classification model is continuously optimized to ensure that the performance of the task classification model can be continuously improved, thereby continuously enhancing the accuracy of task optimization.
[0013] In some embodiments, the dimensionality reduction processing of the original data to obtain dimensionality-reduced data and multiple target dimensions includes:
[0014] The importance of each dimension in the original data is calculated based on a preset dimensionality reduction algorithm;
[0015] A predetermined number of the dimensions are selected as target dimensions based on their importance.
[0016] The target dimension in the original data is used as the dimensionality reduction data corresponding to the original data.
[0017] In this way, by calculating the importance of each dimension in the original data and selecting multiple dimensions with higher importance as dimensionality reduction data, the data dimensionality is reduced, thereby reducing the time complexity of subsequent data analysis and improving the efficiency of data analysis.
[0018] In some embodiments, the preprocessing of the dimensionality-reduced data to obtain sample data includes:
[0019] Remove the dimensionality-reduced data with missing values from the dimensionality-reduced data, and use the remaining dimensionality-reduced data as the first candidate data;
[0020] Delete the first candidate data with outliers from the first candidate data, and use the remaining first candidate data as the second candidate data;
[0021] Sample data is obtained by normalizing the values of each dimension in the second candidate data.
[0022] In this way, by preprocessing the original data, missing values and outliers are removed to obtain the second alternative original data. The values of each dimension in the second alternative original data are normalized to eliminate the problem of inconsistent dimensions, thereby improving the quality of the sample data and thus improving the accuracy of subsequent data analysis.
[0023] In some embodiments, clustering the sample data to obtain multiple sample groups, and labeling the sample data in each sample group to obtain label data, includes:
[0024] The sample data is clustered to obtain multiple sample groups, and the mean of the sample data in each sample group is calculated.
[0025] For each sample group, the Euclidean distance between each sample data and the mean of the sample data is calculated, and the sample data corresponding to the minimum Euclidean distance is used as the representative data to obtain the representative data of each sample group, wherein each representative data corresponds to a historical task;
[0026] Based on the preset historical optimization records, the optimization results of the historical tasks corresponding to the representative data are queried, and all sample data in the sample group to which the representative data belongs are labeled according to the optimization results to obtain label data. The optimization results include no optimization required and optimization required.
[0027] In this way, by querying a small amount of representative data corresponding to the optimization results of historical tasks, label data is obtained by annotating all sample data, avoiding the waste of human resources caused by repeated queries and annotations, and improving the efficiency of data annotation.
[0028] In some embodiments, clustering the sample data to obtain multiple sample groups includes:
[0029] a. Select multiple target data from all sample data according to preset grouping parameters, wherein the number of target data is the same as the grouping parameters;
[0030] b. Select any unselected sample data as the current data, calculate the cosine similarity between the current data and each target data, and group the current data and the target data corresponding to the maximum cosine similarity into the same candidate group;
[0031] c. Repeat step b until all sample data are assigned to candidate groups, resulting in multiple candidate groups;
[0032] d. Calculate the mean of all sample data in each sample group, and calculate the difference between the mean of the same sample group and the target data. If the difference is less than a preset termination threshold, the multiple candidate groups are used as multiple sample groups. If the difference is greater than the preset termination threshold, the mean is used as the target data, and steps b to d are repeated to obtain multiple sample groups.
[0033] In this way, by clustering all sample data, multiple sample groups are obtained, and each sample group has highly similar data dimensions, which can provide guidance for data labeling.
[0034] In some embodiments, training a task classification model based on the sample data and the label data includes:
[0035] a. The sample data and the label data are matched one-to-one as training data. The training data is divided into multiple subsets according to a preset division ratio. All subsets are marked as unvisited. One subset is selected as the validation set and marked as visited. All sample data in the remaining subsets are used as the training set.
[0036] b. Use all the sample data in the training set to train a preset initial classification model to obtain alternative classification models;
[0037] c. Calculate the classification accuracy of the candidate classification model based on the sample data and label data in the validation set;
[0038] d. Select one subset marked as unvisited as the validation set and use the remaining subsets as the training set. Repeat steps b to d to obtain multiple candidate classification models and the classification accuracy corresponding to each candidate classification model until all subsets are marked as visited, and obtain multiple candidate classification models and the classification accuracy corresponding to each candidate classification model.
[0039] e. Select the candidate classification model corresponding to the maximum value among the classification accuracies as the task classification model. The function of the task classification model is to receive the sample data corresponding to the historical task and output the category corresponding to the historical task.
[0040] In this way, obtaining a better-performing task classification model through multiple cross-validations can reduce the risk of underfitting in the task classification model and improve its performance.
[0041] In some embodiments, the step of collecting assessment data related to the task to be assessed based on the target dimension, inputting the assessment data into the task classification model to obtain classification results, and optimizing the assessment task based on the classification results includes:
[0042] Based on the target dimensions, multiple data points related to the task to be evaluated are collected as the original data to be evaluated.
[0043] The original data to be evaluated is normalized to obtain the data to be evaluated.
[0044] The data to be evaluated is input into the task classification model to obtain the category of the task to be evaluated;
[0045] The cosine similarity between the data to be evaluated and each sample data is calculated. The category of the sample data corresponding to the maximum cosine similarity is taken as the baseline category. If the baseline category is the same as the category corresponding to the task to be evaluated, the category corresponding to the task to be evaluated is pushed to the user as the classification result. Otherwise, the data to be evaluated is used as training data to retrain the task classification model.
[0046] In this way, by using the target dimension to select the data to be evaluated corresponding to the task to be evaluated, and inputting the data to be evaluated into the task classification model to obtain the category of the task to be evaluated, the task to be evaluated can be evaluated based on the sample data and label data to determine whether the task to be evaluated is correctly classified. The task classification model can be continuously optimized through the data that is not correctly classified, thereby improving the performance of the task classification model.
[0047] This application embodiment also provides a task optimization device based on artificial intelligence, the device comprising:
[0048] The data acquisition unit is used to collect multiple data related to historical tasks as raw data, and the raw data contains multiple dimensions.
[0049] The dimensionality reduction unit is used to perform dimensionality reduction processing on the original data to obtain dimensionality-reduced data and multiple target dimensions;
[0050] A preprocessing unit is used to preprocess the dimensionality-reduced data to obtain sample data;
[0051] The annotation unit is used to cluster the sample data to obtain multiple sample groups, and to annotate the sample data in each sample group to obtain label data.
[0052] The training unit is used to train a task classification model based on the sample data and the label data;
[0053] The classification unit is used to collect data related to the task to be evaluated based on the target dimension, input the data to be evaluated into the task classification model to obtain the classification result, and optimize the task to be evaluated based on the classification result.
[0054] This application embodiment also provides an electronic device, the electronic device comprising:
[0055] Memory, which stores computer-readable instructions; and
[0056] The processor executes computer-readable instructions stored in the memory to implement the AI-based task optimization method.
[0057] This application also provides a computer-readable storage medium storing computer-readable instructions, which are executed by a processor in an electronic device to implement the artificial intelligence-based task optimization method.
[0058] The aforementioned AI-based task optimization method constructs multi-dimensional raw data by collecting various data related to historical tasks. It then performs dimensionality reduction and preprocessing on the raw data to obtain sample data, thereby improving data quality. Furthermore, it labels the sample data to obtain corresponding tag data. The sample data and tag data are used to train a task classification model, which is then used to classify the task to be evaluated. Based on the classification results, the task classification model is continuously optimized to ensure that the performance of the task classification model can be continuously improved, thereby continuously enhancing the accuracy of task optimization. Attached Figure Description
[0059] Figure 1 This is a flowchart of a preferred embodiment of an artificial intelligence-based task optimization method involved in this application.
[0060] Figure 2 This is a functional block diagram of a preferred embodiment of the artificial intelligence-based task optimization device involved in this application.
[0061] Figure 3 This is a schematic diagram of the structure of an electronic device according to a preferred embodiment of the task optimization method based on artificial intelligence involved in this application. Detailed Implementation
[0062] To better understand the purpose, dimensions, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and dimensions within the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a portion of, and not all, of the embodiments of this application.
[0063] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical dimensions indicated. Therefore, a dimension defined as "first" or "second" may explicitly or implicitly include one or more of the stated dimensions. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0065] This application provides an artificial intelligence-based task optimization method that can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0066] The electronic device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.
[0067] The electronic device may also include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.
[0068] The networks in which the electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).
[0069] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the task optimization method based on artificial intelligence according to this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.
[0070] S10, collect multiple data related to historical tasks as raw data, the raw data containing multiple dimensions.
[0071] In this optional embodiment, the historical tasks refer to various data processing tasks that an enterprise runs on a server to improve business efficiency. The functions of these various data processing tasks include data querying, data cleaning, and data aggregation.
[0072] In this optional embodiment, the various data related to the historical task include the category of the historical task, runtime, execution result, resource utilization rate, the amount of data analyzed by the historical task, and the program interface called by the historical task. The resource utilization rate refers to the proportion of server resources occupied by the historical task, and the resource utilization rate includes CPU utilization rate, memory utilization rate, and I / O utilization rate.
[0073] In this optional embodiment, multiple data items related to each historical task can be queried from a preset task operation log as the original data corresponding to each historical task. The original data is a vector with 1 row and n columns, where n represents the number of the multiple data items, and each column in the original data represents a dimension of the original data.
[0074] In this way, a large amount of raw data was obtained by extensively collecting data related to historical tasks, which comprehensively covered various information of historical tasks and provided data support for subsequent dimension extraction.
[0075] S11, perform dimensionality reduction processing on the original data to obtain dimensionality-reduced data and multiple target dimensions.
[0076] In an optional embodiment, the dimensionality reduction processing of the original data to obtain dimensionality-reduced data and multiple target dimensions includes:
[0077] The importance of each dimension in the original data is calculated based on a preset dimensionality reduction algorithm;
[0078] A predetermined number of the dimensions are selected as target dimensions based on their importance.
[0079] The target dimension in the original data is used as the dimensionality reduction data corresponding to the original data.
[0080] In this optional embodiment, the preset dimensionality reduction algorithm can be an existing data dimensionality reduction algorithm such as PCA (Principal Component Analysis) or SVD (Singular Value Decomposition). This application does not limit this algorithm. The output of the preset dimensionality reduction algorithm is the importance of each dimension in the original data. The higher the importance, the more important the dimension corresponding to that importance is for the representation of the original data.
[0081] In this optional embodiment, the preset number can be 0.8n, 0.6n, 0.4n, etc., and this application does not limit it, where n represents the number of dimensions of the original data. For example, when the preset number is 0.8n, the dimensions can be sorted according to their importance from high to low, and the first 0.8n dimensions are selected as the target dimensions.
[0082] In this optional embodiment, the target dimension in the original data can be used as dimensionality-reduced data.
[0083] In this way, by calculating the importance of each dimension in the original data and selecting multiple dimensions with higher importance as dimensionality reduction data, the data dimensionality is reduced, thereby reducing the time complexity of subsequent data analysis and improving the efficiency of data analysis.
[0084] S12, preprocess the dimensionality-reduced data to obtain sample data.
[0085] In an optional embodiment, the preprocessing of the dimensionality-reduced data to obtain sample data includes:
[0086] Remove the dimensionality-reduced data with missing values from the dimensionality-reduced data, and use the remaining dimensionality-reduced data as the first candidate data;
[0087] Delete the first candidate data with outliers from the first candidate data, and use the remaining first candidate data as the second candidate data;
[0088] Sample data is obtained by normalizing the values of each dimension in the second candidate data.
[0089] In this optional embodiment, in order to avoid the negative impact of missing data on the accuracy of subsequent data analysis, it is possible to query whether each dimensionality reduction data has missing values. If the value of any dimension in a certain dimensionality reduction data is missing, the dimensionality reduction data is deleted until all dimensionality reduction data with missing values are deleted to obtain the first candidate data.
[0090] In this optional embodiment, if the value of a certain dimension in the first candidate data deviates from the mean of the values of that dimension in all the first candidate data by more than twice the standard deviation, then the value of that dimension is an outlier, and all the first candidate data with outliers can be deleted to obtain the second candidate data.
[0091] In this optional embodiment, in order to eliminate the dimensional differences of each dimension in the second candidate data, the values of each dimension in the second candidate data can be normalized according to a preset normalization algorithm to obtain sample data. The preset normalization algorithm can be an existing normalization algorithm such as a maximization algorithm, a minimization algorithm, an arctangent function algorithm, or an S-shaped growth curve algorithm, and this application does not limit it.
[0092] The dimension of the sample data is the same as the dimension of the dimensionality-reduced data, and the value range of each dimension in the sample data is [0,1].
[0093] In this way, by preprocessing the original data, missing values and outliers are removed to obtain the second alternative original data. The values of each dimension in the second alternative original data are normalized to eliminate the problem of inconsistent dimensions, thereby improving the quality of the sample data and thus improving the accuracy of subsequent data analysis.
[0094] S13, cluster the sample data to obtain multiple sample groups, and label the sample data in each sample group to obtain label data.
[0095] In an optional embodiment, the step of clustering the sample data to obtain multiple sample groups, and labeling the sample data in each sample group to obtain label data, includes:
[0096] The sample data is clustered to obtain multiple sample groups, and the mean of the sample data in each sample group is calculated.
[0097] For each sample group, the Euclidean distance between each sample data and the mean of the sample data is calculated, and the sample data corresponding to the minimum Euclidean distance is used as the representative data to obtain the representative data of each sample group, wherein each representative data corresponds to a historical task;
[0098] Based on the preset historical optimization records, the optimization results of the historical tasks corresponding to the representative data are queried, and all sample data in the sample group to which the representative data belongs are labeled according to the optimization results to obtain label data. The optimization results include no optimization required and optimization required.
[0099] In an optional embodiment, the step of clustering the sample data to obtain multiple sample groups includes:
[0100] a. Select multiple target data from all sample data according to preset grouping parameters. The number of target data is the same as the grouping parameters. For example, the preset grouping parameters can be 2.
[0101] b. Select any unselected sample data as the current data, calculate the cosine similarity between the current data and each target data, and group the current data and the target data corresponding to the maximum cosine similarity into the same candidate group;
[0102] c. Repeat step b until all sample data are classified into candidate groups, obtaining multiple candidate groups, the number of sample groups being the same as the preset grouping parameters;
[0103] d. Calculate the mean of all sample data in each sample group, and calculate the difference between the mean of the same sample group and the target data. If the difference is less than a preset termination threshold, the multiple candidate groups are used as multiple sample groups. If the difference is greater than the preset termination threshold, the mean is used as the target data, and steps b to d are repeated to obtain multiple sample groups.
[0104] In this optional embodiment, since all the sample data in each sample group are quite similar, the mean of all the sample data in the sample group can be calculated, and the sample data with the smallest Euclidean distance from the mean can be selected as the representative data of the sample group. The representative data can characterize the dimension of all the sample data in the sample group.
[0105] In this optional embodiment, the preset historical optimization record is used to record the optimization results of all historical tasks. The optimization results include those that do not require optimization and those that require optimization. The optimization results of the historical tasks corresponding to the representative data can be used to mark all sample data in the sample group to which the representative data belongs.
[0106] In this way, by querying a small amount of representative data corresponding to the optimization results of historical tasks, label data is obtained by annotating all sample data, avoiding the waste of human resources caused by repeated queries and annotations, and improving the efficiency of data annotation.
[0107] S14, Train a task classification model based on the sample data and the label data.
[0108] In an optional embodiment, training the task classification model based on the sample data and the label data includes:
[0109] a) The sample data and the label data are matched one-to-one as training data. The training data is then divided into multiple subsets according to a preset partitioning ratio. All subsets are marked as unvisited. One subset is randomly selected as the validation set and marked as visited. All sample data from the remaining subsets are used as the training set. For example, when the preset partitioning ratio is 10%, the training data can be divided into 10 subsets, each containing 10% of the sample data.
[0110] b. Using all the sample data in the training set, a preset initial classification model is trained to obtain candidate classification models. The preset initial classification model can be an existing classification model such as GBDT (Gradient Boosting Decision Tree), XGBoost (Extreme Gradient Boosting), or LightGBM (Light Gradient Boosting Machine). This application does not limit the specific classification model.
[0111] The initial classification model can be trained using Bayesian parameter tuning.
[0112] c. Calculate the classification accuracy of the candidate classification model based on the sample data and label data in the validation set.
[0113] The sample data in the validation set is input into the candidate classification model to obtain the classification result. The ratio of the number of correctly classified samples to the total number of sample data in the validation set is used as the classification accuracy of the candidate classification model.
[0114] d. Select one subset marked as unvisited as the validation set and use the remaining subsets as the training set. Repeat steps b to d to obtain multiple candidate classification models and the classification accuracy corresponding to each candidate classification model until all subsets are marked as visited, and obtain multiple candidate classification models and the classification accuracy corresponding to each candidate classification model.
[0115] e. Select the candidate classification model corresponding to the maximum value among the classification accuracies as the task classification model. The function of the task classification model is to receive the sample data corresponding to the historical task and output the category corresponding to the historical task.
[0116] In this way, obtaining a better-performing task classification model through multiple cross-validations can reduce the risk of underfitting in the task classification model and improve its performance.
[0117] S15, collect the data to be evaluated related to the task to be evaluated according to the target dimension, input the data to be evaluated into the task classification model to obtain the classification result, and optimize the task to be evaluated according to the classification result.
[0118] In an optional embodiment, the step of collecting evaluation data related to the task to be evaluated based on the target dimension, inputting the evaluation data into the task classification model to obtain classification results, and optimizing the evaluation task based on the classification results includes:
[0119] Based on the target dimensions, multiple data points related to the task to be evaluated are collected as the original data to be evaluated.
[0120] The original data to be evaluated is normalized to obtain the data to be evaluated.
[0121] The data to be evaluated is input into the task classification model to obtain the category of the task to be evaluated;
[0122] The cosine similarity between the data to be evaluated and each sample data is calculated. The category of the sample data corresponding to the maximum cosine similarity is taken as the baseline category. If the baseline category is the same as the category corresponding to the task to be evaluated, the category corresponding to the task to be evaluated is pushed to the user as the classification result. Otherwise, the data to be evaluated is used as training data to retrain the task classification model.
[0123] The task to be evaluated refers to a data processing task that requires classification.
[0124] In this optional embodiment, various data related to the task to be evaluated can be collected as raw data to be evaluated based on the target dimension. For example, when the target dimension includes runtime and CPU utilization, the runtime and CPU utilization of the task to be evaluated can be collected as raw data to be evaluated.
[0125] In this optional embodiment, the original data to be evaluated can be normalized according to the preset normalization algorithm to obtain the data to be evaluated. The data to be evaluated contains multiple dimensions, and the number of dimensions in the data to be evaluated is the same as the number of dimensions in the dimensionality-reduced data. The value range of each dimension in the data to be evaluated is [0,1].
[0126] In this optional embodiment, the data to be evaluated can be input into the task classification model to obtain the category corresponding to the task to be evaluated, and the category corresponding to the task to be evaluated includes no optimization or optimization required.
[0127] In this optional embodiment, the cosine similarity between the data to be evaluated and each of the sample data can be calculated respectively, and the sample data corresponding to the maximum cosine similarity can be selected as the benchmark data. The label data corresponding to the benchmark data can be used as the benchmark category. If the benchmark category is the same as the category corresponding to the task to be evaluated, the classification is correct, and the category corresponding to the task to be evaluated can be pushed to the user as the classification result.
[0128] If the baseline category is different from the category corresponding to the task to be evaluated, the classification fails. The baseline category can be used as the label data corresponding to the data to be evaluated, and the data to be evaluated can be used as training data to retrain the task classification model to improve the performance of the task classification model.
[0129] In this way, by using the target dimension to select the data to be evaluated corresponding to the task to be evaluated, and inputting the data to be evaluated into the task classification model to obtain the category of the task to be evaluated, the task to be evaluated can be evaluated based on the sample data and label data to determine whether the task to be evaluated is correctly classified. The task classification model can be continuously optimized through the data that is not correctly classified, thereby improving the performance of the task classification model.
[0130] The aforementioned AI-based task optimization method constructs multi-dimensional raw data by collecting various data related to historical tasks. It then performs dimensionality reduction and preprocessing on the raw data to obtain sample data, thereby improving data quality. Furthermore, it labels the sample data to obtain corresponding tag data. The sample data and tag data are used to train a task classification model, which is then used to classify the task to be evaluated. Based on the classification results, the task classification model is continuously optimized to ensure that the performance of the task classification model can be continuously improved, thereby continuously enhancing the accuracy of task optimization.
[0131] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the task optimization device based on artificial intelligence provided in this application. The task optimization device 11 based on artificial intelligence includes a data acquisition unit 110, a dimensionality reduction unit 111, a preprocessing unit 112, a labeling unit 113, a training unit 114, and a classification unit 115. The module / unit referred to in this application refers to a series of computer program segments that can be executed by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.
[0132] In an optional embodiment, the acquisition unit 110 is used to acquire multiple data related to historical tasks as raw data, the raw data containing multiple dimensions.
[0133] In this optional embodiment, the historical tasks refer to various data processing tasks that an enterprise runs on a server to improve business efficiency. The functions of these various data processing tasks include data querying, data cleaning, and data aggregation.
[0134] In this optional embodiment, the various data related to the historical task include the category of the historical task, runtime, execution result, resource utilization rate, the amount of data analyzed by the historical task, and the program interface called by the historical task. The resource utilization rate refers to the proportion of server resources occupied by the historical task, and the resource utilization rate includes CPU utilization rate, memory utilization rate, and I / O utilization rate.
[0135] In this optional embodiment, multiple data items related to each historical task can be queried from a preset task operation log as the original data corresponding to each historical task. The original data is a vector with 1 row and n columns, where n represents the number of the multiple data items, and each column in the original data represents a dimension of the original data.
[0136] In an optional embodiment, the dimensionality reduction unit 111 is used to perform dimensionality reduction processing on the original data to obtain dimensionality-reduced data and multiple target dimensions.
[0137] In an optional embodiment, the dimensionality reduction processing of the original data to obtain dimensionality-reduced data and multiple target dimensions includes:
[0138] The importance of each dimension in the original data is calculated based on a preset dimensionality reduction algorithm;
[0139] A predetermined number of the dimensions are selected as target dimensions based on their importance.
[0140] The target dimension in the original data is used as the dimensionality reduction data corresponding to the original data.
[0141] In this optional embodiment, the preset dimensionality reduction algorithm can be an existing data dimensionality reduction algorithm such as PCA (Principal Component Analysis) or SVD (Singular Value Decomposition). This application does not limit this algorithm. The output of the preset dimensionality reduction algorithm is the importance of each dimension in the original data. The higher the importance, the more important the dimension corresponding to that importance is for the representation of the original data.
[0142] In this optional embodiment, the preset number can be 0.8n, 0.6n, 0.4n, etc., and this application does not limit it, where n represents the number of dimensions of the original data. For example, when the preset number is 0.8n, the dimensions can be sorted according to their importance from high to low, and the first 0.8n dimensions are selected as the target dimensions.
[0143] In this optional embodiment, the target dimension in the original data can be used as dimensionality-reduced data.
[0144] In an optional embodiment, the preprocessing unit 112 is used to preprocess the dimensionality-reduced data to obtain sample data.
[0145] In an optional embodiment, the preprocessing of the dimensionality-reduced data to obtain sample data includes:
[0146] Remove the dimensionality-reduced data with missing values from the dimensionality-reduced data, and use the remaining dimensionality-reduced data as the first candidate data;
[0147] Delete the first candidate data with outliers from the first candidate data, and use the remaining first candidate data as the second candidate data;
[0148] Sample data is obtained by normalizing the values of each dimension in the second candidate data.
[0149] In this optional embodiment, in order to avoid the negative impact of missing data on the accuracy of subsequent data analysis, it is possible to query whether each dimensionality reduction data has missing values. If the value of any dimension in a certain dimensionality reduction data is missing, the dimensionality reduction data is deleted until all dimensionality reduction data with missing values are deleted to obtain the first candidate data.
[0150] In this optional embodiment, if the value of a certain dimension in the first candidate data deviates from the mean of the values of that dimension in all the first candidate data by more than twice the standard deviation, then the value of that dimension is an outlier, and all the first candidate data with outliers can be deleted to obtain the second candidate data.
[0151] In this optional embodiment, in order to eliminate the dimensional differences of each dimension in the second candidate data, the values of each dimension in the second candidate data can be normalized according to a preset normalization algorithm to obtain sample data. The preset normalization algorithm can be an existing normalization algorithm such as a maximization algorithm, a minimization algorithm, an arctangent function algorithm, or an S-shaped growth curve algorithm, and this application does not limit it.
[0152] The dimension of the sample data is the same as the dimension of the dimensionality-reduced data, and the value range of each dimension in the sample data is [0,1].
[0153] In an optional embodiment, the labeling unit 113 is used to cluster the sample data to obtain multiple sample groups, and to label the sample data in each sample group to obtain label data.
[0154] In an optional embodiment, the step of clustering the sample data to obtain multiple sample groups, and labeling the sample data in each sample group to obtain label data, includes:
[0155] The sample data is clustered to obtain multiple sample groups, and the mean of the sample data in each sample group is calculated.
[0156] For each sample group, the Euclidean distance between each sample data and the mean of the sample data is calculated, and the sample data corresponding to the minimum Euclidean distance is used as the representative data to obtain the representative data of each sample group, wherein each representative data corresponds to a historical task;
[0157] Based on the preset historical optimization records, the optimization results of the historical tasks corresponding to the representative data are queried, and all sample data in the sample group to which the representative data belongs are labeled according to the optimization results to obtain label data. The optimization results include no optimization required and optimization required.
[0158] In an optional embodiment, the step of clustering the sample data to obtain multiple sample groups includes:
[0159] a. Select multiple target data from all sample data according to preset grouping parameters. The number of target data is the same as the grouping parameters. For example, the preset grouping parameters can be 2.
[0160] b. Select any unselected sample data as the current data, calculate the cosine similarity between the current data and each target data, and group the current data and the target data corresponding to the maximum cosine similarity into the same candidate group;
[0161] c. Repeat step b until all sample data are classified into candidate groups, obtaining multiple candidate groups, the number of sample groups being the same as the preset grouping parameters;
[0162] d. Calculate the mean of all sample data in each sample group, and calculate the difference between the mean of the same sample group and the target data. If the difference is less than a preset termination threshold, the multiple candidate groups are used as multiple sample groups. If the difference is greater than the preset termination threshold, the mean is used as the target data, and steps b to d are repeated to obtain multiple sample groups.
[0163] In this optional embodiment, since all the sample data in each sample group are quite similar, the mean of all the sample data in the sample group can be calculated, and the sample data with the smallest Euclidean distance from the mean can be selected as the representative data of the sample group. The representative data can characterize the dimension of all the sample data in the sample group.
[0164] In this optional embodiment, the preset historical optimization record is used to record the optimization results of all historical tasks. The optimization results include those that do not require optimization and those that require optimization. The optimization results of the historical tasks corresponding to the representative data can be used to mark all sample data in the sample group to which the representative data belongs.
[0165] In an optional embodiment, training unit 114 is used to train a task classification model based on the sample data and the label data.
[0166] In an optional embodiment, training the task classification model based on the sample data and the label data includes:
[0167] In an optional embodiment, training the task classification model based on the sample data and the label data includes:
[0168] a) The sample data and the label data are matched one-to-one as training data. The training data is then divided into multiple subsets according to a preset partitioning ratio. All subsets are marked as unvisited. One subset is randomly selected as the validation set and marked as visited. All sample data from the remaining subsets are used as the training set. For example, when the preset partitioning ratio is 10%, the training data can be divided into 10 subsets, each containing 10% of the sample data.
[0169] b. Using all the sample data in the training set, a preset initial classification model is trained to obtain candidate classification models. The preset initial classification model can be an existing classification model such as GBDT (Gradient Boosting Decision Tree), XGBoost (Extreme Gradient Boosting), or LightGBM (Light Gradient Boosting Machine). This application does not limit the specific classification model.
[0170] The initial classification model can be trained using Bayesian parameter tuning.
[0171] c. Calculate the classification accuracy of the candidate classification model based on the sample data and label data in the validation set.
[0172] The sample data in the validation set is input into the candidate classification model to obtain the classification result. The ratio of the number of correctly classified samples to the total number of sample data in the validation set is used as the classification accuracy of the candidate classification model.
[0173] d. Select one subset marked as unvisited as the validation set and use the remaining subsets as the training set. Repeat steps b to d to obtain multiple candidate classification models and the classification accuracy corresponding to each candidate classification model until all subsets are marked as visited, and obtain multiple candidate classification models and the classification accuracy corresponding to each candidate classification model.
[0174] e. Select the candidate classification model corresponding to the maximum value among the classification accuracies as the task classification model. The function of the task classification model is to receive the sample data corresponding to the historical task and output the category corresponding to the historical task.
[0175] In an optional embodiment, the classification unit 115 is used to collect data related to the task to be evaluated based on the target dimension, input the data to be evaluated into the task classification model to obtain classification results, and optimize the task to be evaluated based on the classification results.
[0176] In an optional embodiment, the step of collecting evaluation data related to the task to be evaluated based on the target dimension, inputting the evaluation data into the task classification model to obtain classification results, and optimizing the evaluation task based on the classification results includes:
[0177] Based on the target dimensions, multiple data points related to the task to be evaluated are collected as the original data to be evaluated.
[0178] The original data to be evaluated is normalized to obtain the data to be evaluated.
[0179] The data to be evaluated is input into the task classification model to obtain the category of the task to be evaluated;
[0180] The cosine similarity between the data to be evaluated and each sample data is calculated. The category of the sample data corresponding to the maximum cosine similarity is taken as the baseline category. If the baseline category is the same as the category corresponding to the task to be evaluated, the category corresponding to the task to be evaluated is pushed to the user as the classification result. Otherwise, the data to be evaluated is used as training data to retrain the task classification model.
[0181] The task to be evaluated refers to a data processing task that requires classification.
[0182] In this optional embodiment, various data related to the task to be evaluated can be collected as raw data to be evaluated based on the target dimension. For example, when the target dimension includes runtime and CPU utilization, the runtime and CPU utilization of the task to be evaluated can be collected as raw data to be evaluated.
[0183] In this optional embodiment, the original data to be evaluated can be normalized according to the preset normalization algorithm to obtain the data to be evaluated. The data to be evaluated contains multiple dimensions, and the number of dimensions in the data to be evaluated is the same as the number of dimensions in the dimensionality-reduced data. The value range of each dimension in the data to be evaluated is [0,1].
[0184] In this optional embodiment, the data to be evaluated can be input into the task classification model to obtain the category corresponding to the task to be evaluated, and the category corresponding to the task to be evaluated includes no optimization or optimization required.
[0185] In this optional embodiment, the cosine similarity between the data to be evaluated and each of the sample data can be calculated respectively, and the sample data corresponding to the maximum cosine similarity can be selected as the benchmark data. The label data corresponding to the benchmark data can be used as the benchmark category. If the benchmark category is the same as the category corresponding to the task to be evaluated, the classification is correct, and the category corresponding to the task to be evaluated can be pushed to the user as the classification result.
[0186] If the baseline category is different from the category corresponding to the task to be evaluated, the classification fails. The baseline category can be used as the label data corresponding to the data to be evaluated, and the data to be evaluated can be used as training data to retrain the task classification model to improve the performance of the task classification model.
[0187] The aforementioned AI-based task optimization method constructs multi-dimensional raw data by collecting various data related to historical tasks. It then performs dimensionality reduction and preprocessing on the raw data to obtain sample data, thereby improving data quality. Furthermore, it labels the sample data to obtain corresponding tag data. The sample data and tag data are used to train a task classification model, which is then used to classify the task to be evaluated. Based on the classification results, the task classification model is continuously optimized to ensure that the performance of the task classification model can be continuously improved, thereby continuously enhancing the accuracy of task optimization.
[0188] like Figure 3 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 stores computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement the artificial intelligence-based task optimization method of any of the above embodiments.
[0189] In an alternative embodiment, the electronic device 1 further includes a bus and a computer program stored in memory 12 and executable on processor 13, such as an AI-based task optimization program.
[0190] Figure 3 Only the electronic device 1 with components 12-13 is shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0191] Combination Figure 1 The memory 12 in electronic device 1 stores multiple computer-readable instructions to implement an artificial intelligence-based task optimization method, and the processor 13 can execute multiple instructions to achieve:
[0192] Collect multiple data points related to historical tasks as raw data, and the raw data contains multiple dimensions;
[0193] The original data is subjected to dimensionality reduction processing to obtain dimensionality-reduced data and multiple target dimensions;
[0194] The dimensionality-reduced data is preprocessed to obtain sample data;
[0195] The sample data is clustered to obtain multiple sample groups, and the sample data in each sample group is labeled to obtain label data;
[0196] A task classification model is trained based on the sample data and the label data;
[0197] Based on the target dimension, collect the data to be evaluated related to the task to be evaluated, input the data to be evaluated into the task classification model to obtain the classification result, and optimize the task to be evaluated based on the classification result.
[0198] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.
[0199] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. Electronic device 1 can be either a bus topology or a star topology. Electronic device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, electronic device 1 may also include input / output devices, network access devices, etc.
[0200] It should be noted that electronic device 1 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.
[0201] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive of the electronic device 1. In other embodiments, the memory 12 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 1. Furthermore, the memory 12 can include both internal storage units and external storage devices of the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as code for task optimization programs based on artificial intelligence, but also to temporarily store data that has been output or will be output.
[0202] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 through various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing AI-based task optimization programs) and calls data stored in the memory 12 to perform various functions and process data in the electronic device 1.
[0203] Processor 13 executes the operating system of electronic device 1 and various installed applications. Processor 13 executes these applications to implement the steps in the various AI-based task optimization method embodiments described above, for example... Figure 1 The steps are shown.
[0204] For example, the computer program may be divided into one or more modules / units, which are stored in memory 12 and executed by processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in electronic device 1. For example, the computer program may be divided into a data acquisition unit 110, a dimensionality reduction unit 111, a preprocessing unit 112, a labeling unit 113, a training unit 114, and a classification unit 115.
[0205] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the artificial intelligence-based task optimization methods described in the various embodiments of this application.
[0206] If the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.
[0207] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.
[0208] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.
[0209] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0210] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.
[0211] Although not shown, electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power sources, a recharging device, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components. Electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.
[0212] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.
[0213] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.
[0214] This application also provides a computer-readable storage medium (not shown), which stores computer-readable instructions that are executed by a processor in an electronic device to implement the artificial intelligence-based task optimization method described in any of the above embodiments.
[0215] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.
[0216] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0217] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0218] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0219] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.
[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.
Claims
1. A task optimization method based on artificial intelligence, characterized in that, The method includes: Multiple data points related to historical tasks are collected as raw data. These raw data include multiple dimensions, such as the category of the historical task, runtime, execution result, resource utilization, the amount of data analyzed by the historical task, and the program interfaces called by the historical task. Resource utilization refers to the proportion of server resources used by the historical task, including CPU utilization, memory utilization, and I / O utilization. The historical tasks include various data processing tasks running on the server, such as data querying, data cleaning, and data aggregation. The original data is subjected to dimensionality reduction processing to obtain dimensionality-reduced data and multiple target dimensions, including: calculating the importance of each dimension in the original data according to a preset dimensionality reduction algorithm; selecting a preset number of the dimensions as target dimensions according to the order of importance from high to low; and using the data of the target dimensions in the original data as the dimensionality-reduced data corresponding to the original data. The dimensionality-reduced data is preprocessed to obtain sample data; The process involves clustering the sample data to obtain multiple sample groups, and labeling the sample data in each sample group to obtain label data. This includes: clustering the sample data to obtain multiple sample groups, and calculating the mean of the sample data in each sample group; for each sample group, calculating the Euclidean distance between each sample data point and the mean of the sample data, and using the sample data with the smallest Euclidean distance as representative data to obtain representative data for each sample group, wherein each representative data point corresponds to a historical task; querying the optimization results of the historical task corresponding to the representative data based on preset historical optimization records, and labeling all sample data in the sample group to which the representative data belongs based on the optimization results to obtain label data, wherein the optimization results include no optimization required and optimization required; and training a task classification model based on the sample data and the label data. Based on the target dimension, collect the data to be evaluated related to the task to be evaluated, input the data to be evaluated into the task classification model to obtain the classification result, and optimize the task to be evaluated based on the classification result.
2. The task optimization method based on artificial intelligence as described in claim 1, characterized in that, The preprocessing of the dimensionality-reduced data to obtain sample data includes: Remove the dimensionality-reduced data with missing values from the dimensionality-reduced data, and use the remaining dimensionality-reduced data as the first candidate data; Delete the first candidate data with outliers from the first candidate data, and use the remaining first candidate data as the second candidate data; Sample data is obtained by normalizing the values of each dimension in the second candidate data.
3. The task optimization method based on artificial intelligence as described in claim 1, characterized in that, The process of clustering the sample data to obtain multiple sample groups includes: a. Select multiple target data from all sample data according to preset grouping parameters, wherein the number of target data is the same as the grouping parameters; b. Select any unselected sample data as the current data, calculate the cosine similarity between the current data and each target data, and group the current data and the target data corresponding to the maximum cosine similarity into the same candidate group; c. Repeat step b until all sample data are assigned to candidate groups, resulting in multiple candidate groups; d. Calculate the mean of all sample data in each sample group, and calculate the difference between the mean of the same sample group and the target data. If the difference is less than a preset termination threshold, the multiple candidate groups are used as multiple sample groups. If the difference is greater than the preset termination threshold, the mean is used as the target data, and steps b to d are repeated to obtain multiple sample groups.
4. The task optimization method based on artificial intelligence as described in claim 1, characterized in that, The step of training a task classification model based on the sample data and the label data includes: a. The sample data and the label data are matched one-to-one as training data. The training data is divided into multiple subsets according to a preset division ratio. All subsets are marked as unvisited. One subset is selected as the validation set and marked as visited. All sample data in the remaining subsets are used as the training set. b. Use all the sample data in the training set to train a preset initial classification model to obtain alternative classification models; c. Calculate the classification accuracy of the candidate classification model based on the sample data and label data in the validation set; d. Select one subset marked as unvisited as the validation set and use the remaining subsets as the training set. Repeat steps b to d to obtain multiple candidate classification models and the classification accuracy corresponding to each candidate classification model until all subsets are marked as visited, and obtain multiple candidate classification models and the classification accuracy corresponding to each candidate classification model. e. Select the candidate classification model corresponding to the maximum value among the classification accuracies as the task classification model. The function of the task classification model is to receive the sample data corresponding to the historical task and output the category corresponding to the historical task.
5. The task optimization method based on artificial intelligence as described in claim 1, characterized in that, The process involves collecting assessment data related to the task to be evaluated based on the target dimension, inputting the assessment data into the task classification model to obtain classification results, and optimizing the assessment task based on the classification results, including: Based on the target dimensions, multiple data points related to the task to be evaluated are collected as the original data to be evaluated. The original data to be evaluated is normalized to obtain the data to be evaluated. The data to be evaluated is input into the task classification model to obtain the category of the task to be evaluated; The cosine similarity between the data to be evaluated and each sample data is calculated. The category of the sample data corresponding to the maximum cosine similarity is taken as the baseline category. If the baseline category is the same as the category corresponding to the task to be evaluated, the category corresponding to the task to be evaluated is pushed to the user as the classification result. Otherwise, the data to be evaluated is used as training data to retrain the task classification model.
6. A task optimization device based on artificial intelligence, characterized in that, The device includes: The data acquisition unit is used to collect multiple data related to historical tasks as raw data. This raw data includes multiple dimensions. The data related to the historical tasks includes the category of the historical task, runtime, execution results, resource utilization, the amount of data analyzed by the historical task, and the program interfaces called by the historical task. Resource utilization refers to the proportion of server resources used by the historical task, including CPU utilization, memory utilization, and I / O utilization. The historical tasks include various data processing tasks running on the server, such as data querying, data cleaning, and data aggregation. The dimensionality reduction unit is used to perform dimensionality reduction processing on the original data to obtain dimensionality-reduced data and multiple target dimensions, including: calculating the importance of each dimension in the original data according to a preset dimensionality reduction algorithm; selecting a preset number of the dimensions as target dimensions according to the order of importance from high to low; and using the data of the target dimensions in the original data as the dimensionality-reduced data corresponding to the original data. A preprocessing unit is used to preprocess the dimensionality-reduced data to obtain sample data; The labeling unit is used to cluster the sample data to obtain multiple sample groups, and to label the sample data in each sample group to obtain label data. This includes: clustering the sample data to obtain multiple sample groups, and calculating the mean of the sample data in each sample group; for each sample group, calculating the Euclidean distance between each sample data and the mean of the sample data, and using the sample data corresponding to the minimum Euclidean distance as representative data to obtain representative data for each sample group, wherein each representative data corresponds to a historical task; querying the optimization results of the historical task corresponding to the representative data based on preset historical optimization records, and labeling all sample data in the sample group to which the representative data belongs based on the optimization results to obtain label data, wherein the optimization results include no optimization required and optimization required. The training unit is used to train a task classification model based on the sample data and the label data; The classification unit is used to collect data related to the task to be evaluated based on the target dimension, input the data to be evaluated into the task classification model to obtain the classification result, and optimize the task to be evaluated based on the classification result.
7. An electronic device, characterized in that, The electronic device includes: Memory, which stores computer-readable instructions; and The processor executes computer-readable instructions stored in the memory to implement the AI-based task optimization method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which are executed by a processor in an electronic device to implement the AI-based task optimization method as described in any one of claims 1 to 5.