An automatic machine learning (AutoML) system, method and device

Abstract

The application relates to the field of artificial intelligence (AI), and provides an automatic machine learning (AutoML) method, which comprises the following steps: an AutoML system receives a task target and a first data set of a user; an initial AI model is determined according to the task target, and is used to realize the task target of the user; the AutoML system trains the initial AI model according to the received first data set, and obtains a trained AI model; further, the training of the initial AI model according to the first data set is analyzed, and an analysis result is obtained, wherein the analysis result comprises the influence of at least one type of data in the first data set on the training of the initial AI model; and the AutoML system provides an optimization mode of the trained AI model to the user according to the analysis result, wherein the optimization mode can be uploading a second data set to optimize the trained AI model. According to the analysis of the training of the initial AI model, the optimization mode provided by the AutoML system to the user can efficiently optimize the prediction accuracy of the AI model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an automatic machine learning (AutoML) system, method, and device. Background Technology

[0002] Artificial intelligence (AI) has received widespread attention from academia and industry, and its applications are becoming increasingly widespread, surpassing the capabilities of ordinary humans in many fields. For example, AI technology has enabled machine vision (human recognition, image classification, object detection, etc.) to achieve higher accuracy than humans, and it also shows promising applications in natural language processing and recommendation systems.

[0003] Machine learning is a core method for realizing AI. Computers, based on existing data, build an AI model to solve a technical problem, and then use this model to predict the outcome. This method is similar to how computers learn a certain ability (e.g., cognitive, discriminative, or classification abilities) like humans; hence the name machine learning. Various applications of AI using machine learning require various AI models (e.g., neural network models). An AI model is essentially an algorithm, containing numerous parameters and calculation formulas (or rules). When using machine learning to solve a technical problem, the key challenges are how to build or select a suitable AI model, how to optimize the model during training (i.e., how to achieve the optimal combination of parameters in the selected AI model), and how to maximize the accuracy of the AI ​​model in solving the technical problem. These issues make applying machine learning to real-world problems a technology that can only be implemented by a small group of highly specialized technical personnel.

[0004] In reality, many enterprises or organizations with real-world application data who want to use AI to solve practical problems lack AI capabilities. Meanwhile, AI providers with a large pool of AI technology and talent often struggle to obtain real-world application data as training datasets for their AI models. Against this backdrop, AutoML systems have emerged. AutoML systems provide users with services such as selecting, building, and training AI models based on user-defined task objectives and user-collected datasets. This allows users with limited AI expertise to obtain AI models capable of performing specific tasks and solving business problems. However, existing AutoML systems offer relatively simple feedback on user data quality when training AI models. When an AI model trained on a user-uploaded dataset hasn't reached its ideal state, existing AutoML platforms simply provide the user with the current results (e.g., overall model accuracy) or general optimization methods. This leaves users confused after obtaining an AI model with suboptimal performance. If users want to further optimize the AI ​​model trained in AutoML, they can only blindly optimize the AI ​​model by increasing the dataset or adjusting the proportion of each type of dataset. Since these methods do not consider the correlation between data, they usually result in less effective optimization of the AI ​​model. Summary of the Invention

[0005] This application provides a method, system, and device for automated machine learning (AutoML). This AutoML method can analyze the training of AI models and further provide users with efficient optimization methods for optimizing trained AI models.

[0006] In a first aspect, this application provides a method for Automated Machine Learning (AutoML), the method comprising: an AutoML system receiving a user's task objective and a first dataset; determining an initial artificial intelligence (AI) model based on the task objective, the initial AI model being used to achieve the task objective for the user; training the initial AI model based on the first dataset to obtain a trained AI model; analyzing the training of the initial AI model based on the first dataset to obtain analysis results, wherein the analysis results include the influence of at least one type of data in the first dataset on the training of the initial AI model; and providing the user with optimization methods for the trained AI model based on the analysis results, the optimization methods including: uploading a second dataset for optimizing the trained AI model.

[0007] It should be understood that the task objectives received by the AutoML system are the functions that the user hopes the final AI model trained by the AutoML system will provide. Users can select or input a task objective into the AutoML system via the GUI or through the command line. It should also be understood that the order in which the AutoML system receives the user's task objectives and the first dataset is not limited; it can receive the user's task objectives first, followed by the first dataset uploaded by the user.

[0008] This method allows users to obtain more specific optimization methods for trained AI models. This enables users to more effectively collect, label, and upload data based on the optimization suggestions from the AutoML system, avoiding unnecessary additional workload and making the optimization of trained AI models more efficient. Optimizing and analyzing the initial AI model training and providing reliable optimization methods truly allows users without professional AI knowledge to easily obtain a satisfactory final AI model, which can then be used to achieve the task objectives.

[0009] In one possible implementation of the first aspect, the method further includes: providing the user with the expected performance of the optimized trained AI model, the expected performance indicating the prediction accuracy that the trained AI model will achieve after optimized training based on the second dataset.

[0010] By providing users with the expected results of optimizing a trained AI model, users can understand the optimization potential of the model and decide whether to follow the optimization methods suggested by the AutoML system based on this information and their actual situation. Alternatively, users can choose to abandon further optimization of the trained AI model after balancing the prediction accuracy, expected results after optimization, and time and manpower costs.

[0011] In one possible implementation of the first aspect, the first dataset includes a training dataset and a test dataset; before analyzing the training of the initial AI model based on the first dataset to obtain analysis results, the method further includes: evaluating the prediction accuracy of the trained AI model for each type of data in the test dataset; the step of analyzing the training of the initial AI model based on the first dataset to obtain analysis results specifically includes: determining at least one type of data in the training dataset for analyzing the training of the initial AI model based on the prediction accuracy of each type of data in the test dataset; and using an incremental experiment method to analyze the impact of at least one type of data in the training dataset on the training of the initial AI model to obtain the analysis results.

[0012] The above evaluation of the trained AI model and analysis of the initial AI model training fully consider the different impacts of different types of data in the training dataset on AI model training, ensuring that the optimization methods provided by the AutoML system can more efficiently optimize the trained AI model.

[0013] In one possible implementation of the first aspect, the step of using incremental experimentation to analyze the impact of at least one type of data in the training dataset on the training of the initial AI model and obtaining the analysis results specifically includes: dividing the training dataset into a baseline set and an incremental set; training the initial AI model using the baseline set to obtain a baseline AI model; for each type of data in the at least one type of data in the incremental set, dividing each type of data into multiple parts, and successively adding one part of the multiple parts of data to train the baseline AI model to obtain an intermediate AI model; calculating the change in prediction accuracy of the intermediate AI model relative to the baseline AI model after each training; and obtaining the benefit coefficient of each type of data on the prediction accuracy of the intermediate AI model based on the change in prediction accuracy and each type of data.

[0014] This method utilizes mathematical experiments to fully analyze the impact of at least one type of data in the training dataset on the training of the initial AI model. It uses the benefit coefficient of each type of data in the at least one type of data on the prediction accuracy of the intermediate AI model as the result of the analysis. This mathematically quantified analysis result not only enables the AutoML system to accurately propose optimization methods for the trained AI model based on the analysis results, but also provides them to users intuitively, making the optimization methods more convincing to users.

[0015] In one possible implementation of the first aspect, the second dataset includes one or more types of data, wherein the data type in the second dataset is the type of data whose gain coefficient for the prediction accuracy of the intermediate AI model is greater than a preset threshold. The data types in the second dataset are obtained through further analysis based on the analysis results of the initial AI model. When providing optimization methods to users, guiding them to continue uploading data types whose gain coefficient for the prediction accuracy of the intermediate AI model is greater than the preset threshold can improve the optimization efficiency of the trained AI model and save users unnecessary time and manpower costs.

[0016] In one possible implementation of the first aspect, the method further includes: receiving the second dataset uploaded by the user; and optimizing the trained AI model based on the second dataset. After the user uploads the second dataset, the trained AI model is further optimized so that the optimized AI model can better achieve the user's task objectives.

[0017] In one possible implementation of the first aspect, before analyzing the training of the initial AI model based on the first dataset to obtain a trained AI model, the method further includes: classifying the data in the first dataset according to the attributes of the data in the first dataset. This method allows the AutoML system to analyze the categories under each attribute of the data in the dataset separately when analyzing the training of the initial AI model, which can fully analyze the impact of different attribute classifications of the data on the training of the AI ​​model and provide users with more optimization methods.

[0018] In one possible implementation of the first aspect, the data in the first and second datasets are labeled, and the data types in the first and second datasets are the same as the labels on the data in the first and second datasets. The AutoML system can analyze the impact of each labeled data type on the training of the AI ​​model based on the labels in the user-uploaded dataset, and finally provide optimization methods for adding one or more labeled data types, allowing users to continue collecting the second dataset in the same way as the first dataset, and this optimization method is simple and efficient.

[0019] In one possible implementation of the first aspect, the method further includes: preprocessing the data in the received first dataset and the second dataset respectively, wherein the preprocessing includes one or more of the following operations: (1) modifying the size specifications of the data; (2) verifying the data; (3) encoding and converting the data; (4) classifying the data by attributes; and (5) extracting features from the data.

[0020] Preprocessing the data in the first or second dataset before training can make the data more suitable for training the AI ​​model, thereby improving the efficiency of training the AI ​​model and the prediction accuracy of the AI ​​model trained with this data.

[0021] Secondly, this application provides an AutoML (Autonomous Machine Learning) system, comprising: a user input / output (I / O) module for receiving a user's task objective and a first dataset; a model determination module for determining an initial artificial intelligence (AI) model based on the task objective, the initial AI model being used to achieve the task objective for the user; a model training module for training the initial AI model based on the first dataset to obtain a trained AI model; a model optimization analysis module for analyzing the training of the initial AI model based on the first dataset to obtain analysis results, wherein the analysis results include the influence of at least one type of data in the first dataset on the training of the initial AI model; and the user I / O module further comprising providing the user with optimization methods for the trained AI model based on the analysis results, the optimization methods including: uploading a second dataset for optimizing the trained AI model.

[0022] In one possible implementation of the second aspect, the user I / O module is further configured to provide the user with the expected performance of the optimized trained AI model, the expected performance indicating the prediction accuracy that the trained AI model will achieve after optimized training based on the second dataset.

[0023] In one possible implementation of the second aspect, the first dataset includes a training dataset and a test dataset; the model optimization analysis module is further configured to: evaluate the prediction accuracy of the trained AI model for each type of data in the test dataset; the model optimization analysis module is configured to analyze the training of the initial AI model based on the first dataset, and when obtaining the analysis results, specifically configured to: determine at least one type of data in the training dataset for analyzing the training of the initial AI model based on the prediction accuracy of each type of data in the test dataset; and analyze the impact of at least one type of data in the training dataset on the training of the initial AI model using an incremental experiment method to obtain the analysis results.

[0024] In one possible implementation of the second aspect, the model optimization analysis module is used to analyze the impact of at least one type of data in the training dataset on the training of the initial AI model using an incremental experiment method. Specifically, when obtaining the analysis results, it is used to: divide the training dataset into a baseline set and an incremental set; train the initial AI model using the baseline set to obtain a baseline AI model; for each type of data in the incremental set, divide each type of data into multiple parts, and successively add one part of the multiple parts of data to train the baseline AI model to obtain an intermediate AI model; calculate the change in prediction accuracy of the intermediate AI model relative to the baseline AI model after each training iteration; and obtain the benefit coefficient of each type of data on the prediction accuracy of the intermediate AI model based on the change in prediction accuracy and the data of each type.

[0025] In one possible implementation of the second aspect, the second dataset includes one or more types of data, wherein the data in the second dataset is of the type whose profit coefficient for the prediction accuracy of the intermediate AI model is greater than a preset threshold.

[0026] In one possible implementation of the second aspect, the user I / O module is further configured to receive the second dataset uploaded by the user; the model training module is further configured to optimize the trained AI model based on the second dataset.

[0027] In one possible implementation of the second aspect, the model optimization analysis module is further configured to: classify the data in the first dataset according to the attributes of the data in the first dataset.

[0028] In one possible implementation of the second aspect, the data in the first dataset and the second dataset are labeled, and the data types of the first dataset and the second dataset are the same as the labels on the data in the first dataset and the second dataset.

[0029] In one possible implementation of the second aspect, the system further includes: a data preprocessing module, configured to preprocess the received first dataset and the second dataset respectively, wherein the preprocessing includes one or more of the following operations: (1) modifying the size specifications of the data; (2) verifying the data; (3) encoding and converting the data; (4) classifying the data by attributes; and (5) extracting features from the data.

[0030] Thirdly, this application provides a computing device, the computing device including a memory and a processor, the memory being used to store a set of computer instructions; the processor executing the set of computer instructions stored in the memory to cause the computing device to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0031] Fourthly, this application provides a non-transient readable storage medium storing computer program code. When the computer program code is executed by a computing device, the computing device performs the method provided in the first aspect or any possible implementation thereof. This storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0032] Fifthly, this application provides a computer program product comprising computer program code. When the computer program code is executed by a computing device, the computing device performs the methods provided in the first aspect or any possible implementation thereof. The computer program product may be a software installation package. When it is necessary to use the methods provided in the first aspect or any possible implementation thereof, the computer program product may be downloaded and executed on a computing device. Attached Figure Description

[0033] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0034] Figure 1 This is a schematic diagram of the structure of an AutoML system 100 provided in an embodiment of this application;

[0035] Figure 2 This application provides a schematic diagram of an application scenario for an AutoML system 100.

[0036] Figure 3 A deployment diagram of an AutoML system 100 provided for an embodiment of this application;

[0037] Figure 4 A schematic diagram of the structure of a computing device 200 equipped with an AutoML system 100, provided for an embodiment of this application;

[0038] Figure 5 A flowchart illustrating an AutoML (Automotive Machine Learning) method provided in this application embodiment;

[0039] Figure 6 A flowchart illustrating a method for analyzing the training of an initial AI model, provided as an embodiment of this application;

[0040] Figure 7 A GUI diagram illustrating the prediction accuracy of a trained AI model on a test dataset for various types of data, as provided in an embodiment of this application.

[0041] Figure 8 A schematic diagram illustrating the calculation of the total revenue coefficient of an intermediate AI model using newly added Class A data, as provided in an embodiment of this application;

[0042] Figure 9 A GUI diagram illustrating the optimization method and analysis results provided in this application embodiment;

[0043] Figure 10 A GUI diagram illustrating the prediction accuracy curve of an AI model, provided as an embodiment of this application;

[0044] Figure 11 A flowchart illustrating another method for automated machine learning (AutoML) provided in an embodiment of this application;

[0045] Figure 12 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0046] The solutions in the embodiments provided in this application will now be described with reference to the accompanying drawings.

[0047] Currently, the field of artificial intelligence (AI) is booming, and machine learning is a core means of realizing AI. Machine learning has permeated various industries such as medicine, transportation, education, and finance. Not only professionals, but also non-AI professionals in various industries are looking forward to using AI and machine learning to complete specific tasks.

[0048] To facilitate understanding of the technical solutions and embodiments provided in this application, the concepts of AI model, AI model training, and automatic machine learning (AutoML) system are explained in detail below:

[0049] AI models are mathematical algorithm models that use machine learning concepts to solve practical problems. AI models include numerous parameters and calculation formulas (or rules). The parameters in an AI model are numerical values ​​obtained by training the model on a dataset; for example, the weights of calculation formulas or factors within the model. AI models also contain hyperparameters, which are parameters that cannot be obtained through training on a dataset. Hyperparameters are used to guide the construction or training of the AI ​​model. There are various types of hyperparameters, such as the number of iterations, learning rate, batch size, number of layers, and number of neurons per layer. In other words, the difference between hyperparameters and parameters in an AI model is that the values ​​of hyperparameters cannot be obtained by analyzing the data in the dataset, while the values ​​of model parameters can be modified and determined based on the analysis of the dataset.

[0050] AI models are diverse, with neural network models being one of the most widely used. Neural network models are mathematical algorithm models that mimic the structure and function of biological neural networks (the central nervous system of animals). A neural network model can include multiple neural network layers with different functions, each containing parameters and calculation formulas. Depending on the calculation formula or function, different layers in a neural network model have different names; for example, the layer performing convolution calculations is called a convolutional layer, which is often used for feature extraction from input signals (e.g., images). A neural network model can also be composed of a combination of multiple existing neural network models. Different neural network models can be used in different scenarios (e.g., classification, recognition) or provide different results when used in the same scenario. Differences in neural network model structure specifically include one or more of the following: different numbers of network layers, different order of network layers, and different weights, parameters, or calculation formulas in each network layer. The industry already has many different neural network models with high accuracy for applications such as recognition or classification. Some neural network models can be trained on specific datasets and used alone to complete a task or combined with other neural network models (or other functional modules) to complete a task.

[0051] Besides neural network models, most other AI models need to be trained before they can be used to perform a task. Training an AI model refers to using existing data and employing certain methods to make the AI ​​model fit the patterns in the existing data, and determining the parameters in the AI ​​model. Training an AI model requires a dataset. Depending on whether the data in the dataset is labeled (i.e., whether the data has a specific type or name), AI model training can be divided into supervised training and unsupervised training. In supervised training, the data in the training dataset is labeled. When training an AI model, the data in the dataset is used as the input to the AI ​​model, and the corresponding labels are used as a reference for the AI ​​model's output value. A loss function is used to calculate the loss between the AI ​​model's output value and the corresponding labels, and the parameters in the AI ​​model are adjusted based on the loss value. The AI ​​model is trained iteratively with each data point in the dataset, and the parameters of the AI ​​model are continuously adjusted until the AI ​​model can output the same value as the corresponding label with high accuracy based on the input data. Unsupervised training of AI models involves using unlabeled data in the training dataset. The data is sequentially input into the AI ​​model, which gradually identifies relationships and underlying rules between the data points until it can determine or identify the type or characteristics of the input data. For example, in clustering, an AI model receiving a large amount of data can learn the characteristics of each data point and the relationships and differences between them, automatically classifying the data into multiple types. Different task types can use different AI models. Some AI models can only be trained using supervised learning, some can only be trained using unsupervised learning, and some can be trained using both supervised and unsupervised learning. A trained AI model can then be used to complete a specific task. Generally, AI models in machine learning require supervised learning training. Supervised learning allows the AI ​​model to more effectively learn the relationships between data points and their corresponding labels in a labeled dataset, resulting in higher accuracy when predicting other input data.

[0052] Here's an example of training a neural network model for image classification using supervised learning: To train such a model, data is first collected based on the task, creating a dataset containing images of three categories: apples, pears, and bananas. The collected images are stored in three folders, each named after its corresponding image label. After the dataset is built, a suitable neural network model for image classification (e.g., a convolutional neural network (CNN)) is chosen. The images from the dataset are input into the CNN. The convolutional kernels in each layer of the CNN extract and classify features, outputting the confidence score of each image belonging to its category. Based on the confidence score and the corresponding image label, a loss function is used to calculate the loss value. The parameters of each layer in the CNN are then updated based on the loss value and the CNN structure. This training process continues until the loss function converges or all images in the dataset have been used for training, at which point training ends.

[0053] The loss function is used to measure the degree to which an AI model has been trained (that is, to calculate the difference between the AI ​​model's prediction and the true target). During AI model training, because we want the AI ​​model's output to be as close as possible to the desired predicted value, we compare the current AI model's predicted value based on the input image with the actual target value (i.e., the input image's label), and update the AI ​​model's parameters based on the difference (of course, there is usually an initialization process before the first update, i.e., pre-configuring initial values ​​for the AI ​​model's parameters). Each training iteration uses the loss function to determine the difference between the current AI model's predicted value and the true target value, updating the AI ​​model's parameters accordingly, until the AI ​​model can predict the actual target value or a value very close to the actual target value, at which point the AI ​​model is considered to have completed training.

[0054] Automated Machine Learning (AutoML) systems are designed to automate the machine learning process. AutoML systems incorporate various AI models or sub-models to solve different problems. They can search for and build suitable AI models based on user needs. Users simply define their requirements on the AutoML platform and upload their prepared datasets. AutoML then trains an AI model to meet those needs. Users can then use this trained model to complete their specific tasks. Because machine learning is a complex development process requiring technical expertise, AutoML effectively reduces development costs and lowers the barrier to entry for AI applications.

[0055] Existing AutoML systems generally suffer from weak analytical capabilities and an inability to provide users with effective model optimization methods during AI model training. To address this issue, this application provides an AutoML system that can deeply analyze the impact of different data types on AI model training, predict the effect of adding one or more data types on AI model optimization, and further provide users with suggestions for AI model optimization. This system performs operations such as data preprocessing, searching for or selecting suitable AI models based on user tasks, AI model training and hyperparameter optimization, and deep optimization analysis of the AI ​​model.

[0056] Figure 1 This is a schematic diagram of the structure of the AutoML system 100 in the embodiments of this application. It should be understood that... Figure 1 This is merely an illustrative structural diagram of the AutoML system 100, and this application does not limit the division of modules in the AutoML system 100. Figure 1 As shown, the AutoML system 100 includes a user input / output (I / O) module 101, a data preprocessing module 102, a model determination module 103, a model training module 104, a model optimization and analysis module 105, a dataset storage module 106, and an AI model storage module 107.

[0057] The following is a brief description of the functions of each module in the AutoML system 100:

[0058] User I / O module 101 is used to receive user-inputted or selected task objectives, receive user-uploaded datasets, and provide users with analysis results of AI model training, model optimization methods, and / or the expected effects of AI model optimization. As an example, user I / O module 101 can be implemented using a graphical user interface (GUI). For example, the GUI displays that the AutoML system can provide users with four AI services: image classification, face recognition, video similarity detection, and license plate recognition. Users can select a task objective on the GUI, such as face recognition, and then upload a dataset for training the AI ​​model for face recognition on the AutoML GUI. After receiving the task objective and dataset, the GUI communicates with dataset storage module 102 and model determination module 103. Dataset storage module 102 stores the user-uploaded dataset, and model determination module 103 selects or searches for and constructs an AI model that can be used to complete the user's task objective based on the user-defined task objective. User I / O module 101 is also used to receive analysis results and optimization methods of AI model training from model optimization analysis module 105.

[0059] Optionally, the user I / O module 101 can also be used to receive user input regarding the expected performance of the AI ​​model in achieving the task objective, such as inputting or selecting that the final AI model used for face recognition should have an accuracy rate higher than 99%.

[0060] Optionally, the user I / O module 101 can also be used to provide a variety of pre-built initial AI models for the user to choose from. For example, the user can select an initial AI model on the GUI according to their task objectives.

[0061] Optionally, the user I / O module 101 can also be used to receive various configuration information from the user regarding the initial AI model and dataset.

[0062] Data preprocessing module 102: Used to preprocess user-uploaded datasets. Data preprocessing module 102 can read the user-uploaded dataset from dataset storage module 106, or it can directly receive the user-uploaded dataset and preprocess the data within it. Preprocessing the user-uploaded dataset ensures consistency in data size and removes inappropriate data. The preprocessed dataset is suitable for training the initial AI model and can improve training performance. After preprocessing, data preprocessing module 102 either stores the preprocessed dataset in dataset storage module 106 or sends the preprocessed dataset to model training module 104.

[0063] Model Determination Module 103: This module determines an initial AI model for the user to complete their task objectives based on the user's task goals. Model Determination Module 103 can communicate with User I / O Module 101, Model Training Module 104, and AI Model Storage Module 107. Model Determination Module 103 selects a ready-made initial AI model from the AI ​​model library stored in AI Model Storage Module 107 based on the user's task goals, the user's expected performance, or user-input configuration parameters. Alternatively, it searches the AI ​​model library for initial AI sub-model structures, specifying hyperparameters such as the number of layers and neurons per layer, and constructs the initial AI model to obtain a complete initial AI model. After determining the initial AI model for the task objectives, Model Determination Module 103 sends the initial AI model to Model Training Module 104 or sends its name and address information in AI Model Storage Module 104, enabling Model Training Module 104 to train the initial AI model. It is worth noting that some hyperparameters of the initial AI model can be determined by the AutoML system based on experience in building and training the initial AI model.

[0064] Optionally, the model determination module 103 can also be used to determine the AI ​​model selected by the user on the GUI as the initial AI model.

[0065] Model training module 104: This module automatically trains the predetermined initial AI model based on the preprocessed dataset. Model training module 104 reads the preprocessed dataset from data preprocessing module 102 or dataset storage module 106, and obtains the predetermined initial AI model from model determination module 103 or AI model storage module 107. Model training module 104 determines some hyperparameters for training the initial AI model based on the characteristics of the dataset and the structure of the initial AI model, such as iteration count, learning rate, and batch size. After setting the hyperparameters, model training module 104 uses the acquired dataset to automatically train the initial AI model, continuously updating the internal parameters of the AI ​​model during training. It is worth noting that some hyperparameters for training the initial AI model can be hyperparameters determined by the AutoML system based on model training experience.

[0066] Model optimization analysis module 105: This module analyzes the training of the initial AI model, assessing its effectiveness and identifying potential ways to further optimize the trained AI model obtained from the model training module 104, along with expected results. During the initial AI model training process, the module analyzes the impact of each data type in the dataset on the initial AI model's training, identifying data types that significantly improve its performance. It further analyzes the expected optimization potential of the initial AI model if these data types are added for further training. Based on the analysis results, the module provides optimization methods to the user and sends the analysis results and optimization methods to the user I / O module 101.

[0067] Dataset storage module 106: Used to store user-uploaded datasets, and also to store datasets processed by data preprocessing module 102. It should be understood that in another embodiment, dataset storage module 106 may also be part of data preprocessing module 102, even if data preprocessing module 102 has the function of storing datasets.

[0068] AI model storage module 107: Used to store preset AI models and AI sub-model structures, and can also be used to store newly constructed initial AI models based on AI sub-model structures. It should be understood that, in another embodiment, AI model storage module 107 can also be part of model determination module 103.

[0069] Due to the functions of the above modules, the AutoML system provided in this application embodiment can provide users with the business of determining and training AI models. Moreover, the system can deeply analyze the impact of different types of data on AI model training, predict the effect of adding one or more types of data on AI model optimization, and other analysis results, further providing users with ways to optimize AI models.

[0070] Figure 2 This is a schematic diagram illustrating an application scenario of an AutoML system 100 provided in an embodiment of this application, such as... Figure 2As shown, in one embodiment, the AutoML system 100 can be entirely deployed in a cloud environment. A cloud environment is an entity that provides cloud services to users using basic resources under a cloud computing model. The cloud environment includes a cloud data center and a cloud service platform. The cloud data center includes a large amount of basic resources (including computing resources, storage resources, and network resources) owned by the cloud service provider. The computing resources included in the cloud data center can be a large number of computing devices (e.g., servers). The AutoML system 100 can be deployed independently on servers or virtual machines in the cloud data center, or it can be distributed across multiple servers in the cloud data center, or distributed across multiple virtual machines in the cloud data center, or distributed across servers and virtual machines in the cloud data center. Figure 2 As shown, the AutoML system 100 is abstracted into an AutoML cloud service by a cloud service provider on a cloud service platform and provided to users. After purchasing this cloud service on the cloud service platform (pre-payment is possible, with settlement based on final resource usage), the cloud environment utilizes the AutoML system 100 deployed in the cloud data center to provide the AutoML cloud service to users. When using the AutoML cloud service, users can specify the task to be completed by the AI ​​model and upload the dataset to the cloud environment through an application program interface (API) or GUI. The AutoML system 100 in the cloud environment receives the user's task information and dataset, and performs operations such as data preprocessing, AI model determination, AI model training, and AI model optimization analysis. The AutoML system returns the effect of the trained AI model, the optimization method of the trained AI model, and the expected optimization effect to the user through the API or GUI. Users can further upload datasets or abandon optimization according to the optimization method. The trained AI model can be downloaded or used online by users to complete specific tasks.

[0071] In another embodiment of this application, when the AutoML system 100 in the cloud environment is abstracted into an AutoML cloud service and provided to users, it can be divided into two parts: a basic AutoML cloud service and a value-added AI model optimization and analysis cloud service. Users can initially purchase only the basic AutoML cloud service on the cloud service platform, and then purchase the value-added AI model optimization and analysis cloud service when needed. After purchase, the cloud service provider provides the value-added AI model optimization and analysis API, and the value-added AI model optimization and analysis cloud service is charged additionally based on the number of API calls.

[0072] The AutoML system 100 provided in this application has relatively flexible deployment, such as Figure 3As shown, in another embodiment, the AutoML system 100 provided in this application can also be deployed in a distributed manner in different environments. The AutoML system 100 provided in this application can be logically divided into multiple parts, each with different functions. For example, in one embodiment, the AutoML system 100 includes a user I / O module 101, a data preprocessing module 102, a model determination module 103, a model training module 104, a model optimization analysis module 105, a dataset storage module 106, and an AI model storage module 107. Each part of the AutoML system 100 can be deployed in any two or three environments: a terminal computing device, an edge environment, and a cloud environment. Terminal computing devices include: terminal servers, smartphones, laptops, tablets, personal desktop computers, smart cameras, etc. An edge environment is an environment that includes a set of edge computing devices located close to the terminal computing device. Edge computing devices include: edge servers, edge stations with computing power, etc. The various parts of the AutoML system 100 deployed in different environments or devices work together to provide users with initial AI model determination and training functions. For example, in one scenario, the user I / O module 101, data preprocessing module 102, and dataset storage module 106 of the AutoML system 100 are deployed in the terminal computing device, and the model determination module 103, model training module 104, model optimization analysis module 105, and AI model storage module 107 of the AutoML system 100 are deployed in the edge computing device in the edge environment. The user sends the collected dataset to the user I / O module 101 in the terminal computing device, and the terminal computing device stores the dataset in the dataset storage module 106. The data preprocessing module 102 preprocesses the dataset and stores the preprocessed dataset in the dataset storage module 106 as well. The module determination module 103 in the edge computing device determines the initial AI model according to the user's task objectives. Furthermore, the model training module 104 and the model optimization analysis module 105 use the preprocessed dataset stored in the data storage device to train and optimize the determined initial AI model in the AI ​​model storage module 107. It should be understood that this application does not restrict the specific deployment environment of which parts of the AutoML system 100 are deployed. In actual application, the deployment can be adapted according to the computing power of the terminal computing device, the resource availability of the edge environment and cloud environment, or the specific application requirements.

[0073] The AutoML system 100 can also be deployed independently on a computing device in any environment (e.g., on an edge server in an edge environment). Figure 4 This is a schematic diagram of the hardware structure of a computing device 200 equipped with an AutoML system 100. Figure 4The computing device 200 shown includes a memory 201, a processor 202, a communication interface 203, and a bus 204. The memory 201, processor 202, and communication interface 203 are interconnected via the bus 204.

[0074] The memory 201 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 201 can store programs. When the programs stored in the memory 201 are executed by the processor 202, the processor 202 and the communication interface 203 are used to execute the methods used by the AutoML system 100 to train and optimize AI models for the user. The memory can also store datasets. For example, a portion of the storage resources in the memory 201 can be allocated to a dataset storage module 106 to store the datasets required by the AutoML system 100, and a portion of the storage resources in the memory 201 can be allocated to an AI model storage module 107 to store an AI model library.

[0075] The processor 202 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits.

[0076] The processor 202 can also be an integrated circuit chip with signal processing capabilities. In implementation, the functions of the AutoML system 100 of this application can be accomplished through integrated logic circuits in the hardware of the processor 202 or through software instructions. The aforementioned processor 202 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments below of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments below of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. Processor 202 reads the information in memory 201 and, in conjunction with its hardware, completes the functions of the AutoML system 100 of this application embodiment.

[0077] Communication interface 203 uses transceiver modules, such as, but not limited to, transceivers, to enable communication between computing device 200 and other devices or communication networks. For example, a dataset can be obtained through communication interface 203.

[0078] Bus 204 may include a pathway for transmitting information between various components of computing device 200 (e.g., memory 201, processor 202, communication interface 203).

[0079] The following is combined Figure 5 The following describes the specific flow of a method for automating machine learning (AutoML) in one embodiment, which is executed by an AutoML system 100:

[0080] S301: Receive the user's task objectives and dataset.

[0081] Specifically, the AutoML system 100 can receive the user's task objective through a user I / O module (e.g., a GUI). The task objective could be, for example, that the user wants an AI model to detect and recognize text on express delivery tracking numbers, or that the user wants an AI model to accurately identify images containing various fruits. After receiving the user's task, the system provides prompts, allowing the user to upload the collected dataset. The AutoML system then receives the uploaded dataset.

[0082] It is worth noting that the AutoML system 100 can also receive two datasets uploaded by the user: a training dataset and a test dataset. The training dataset is used to train the initial AI model to determine the task objective, while the test dataset is used to test the trained AI model and evaluate its prediction accuracy. It is also noteworthy that when the AutoML system 100 receives only one dataset uploaded by the user, it can automatically divide the uploaded dataset into a training dataset and a test dataset.

[0083] Optionally, the AutoML system 100 can also receive the user's expectations for the final AI model's performance as input on the GUI (e.g., the expectation that the final AI model's detection and recognition accuracy will reach over 99%).

[0084] Optionally, the AutoML system 100 can also receive a user-selected preset AI model and use the user-selected preset AI model as the initial AI model.

[0085] Optionally, the AutoML system 100 can also receive various configuration information from the user regarding the initial AI model and dataset.

[0086] S302: Preprocess the dataset uploaded by the user.

[0087] In this step, the preprocessing method includes one or more of the following operations:

[0088] 1. Automatically scale or standardize the size and specifications of data in the user-uploaded dataset;

[0089] 2. Verify the data in the user-uploaded dataset and remove individual data that seriously affect the model training effect;

[0090] 3. Verify the annotations of the data in the user-uploaded datasets, and remove or correct data in the labeled data whose content does not match the annotations;

[0091] 4. Transform and encode the data in the dataset;

[0092] 5. Extract features from the data in the dataset;

[0093] 6. Divide the data in the dataset into training dataset and test dataset. The ratio of division can vary depending on the user's task objectives. When a dataset contains multiple different types of data, both the training dataset and the test dataset should contain data of each type.

[0094] 7. Perform attribute classification on the dataset. For example, when the dataset contains license plates from multiple countries, it can be classified according to attributes such as license plate color or the length of characters in the license plate. It should be understood that the preprocessing operations on the dataset are not limited to those mentioned above; other adaptive preprocessing can be performed based on the task objective and the user-uploaded dataset. It should also be understood that when performing multiple preprocessing operations on a dataset, the dataset can be preprocessed sequentially according to the type of preprocessing operation.

[0095] It is worth noting that when the user uploads two datasets, a training dataset and a test dataset, the same preprocessing operations are performed on both datasets. It is also worth noting that when the user uploads a single dataset, S302 first divides the uploaded dataset into a training dataset and a test dataset, and then performs the same other preprocessing operations on both datasets.

[0096] S303: Determine the initial AI model based on the user's task objectives.

[0097] In this step, the AutoML system 100 determines a complete AI model from the AI ​​model database based on the user's task objective as the initial AI model for completing the user's task. Alternatively, the AutoML system 100 determines some hyperparameters of the initial AI model based on the user's task objective, such as the number of layers and the number of neurons per layer. The AutoML system then searches the AI ​​model library for AI sub-model structures based on the user's task objective. Further, the AutoML system 100 constructs the AI ​​model based on the hyperparameters and AI sub-model structures, ultimately obtaining a complete initial AI model. It should be understood that this application does not limit the method for determining the initial AI model; other methods for determining and constructing initial AI models in the prior art are also applicable to this step in the embodiments of this application. It should be understood that the initial AI model in this application refers to the AI ​​model determined by the AutoML system 100 based on the user's task objective, but which has not been trained on a dataset uploaded by the user.

[0098] S304: Train the initial AI model using the preprocessed dataset.

[0099] This step uses the preprocessed training dataset obtained in S302 to train the initial AI model determined in S303. Before training, some hyperparameters for model training, such as the number of iterations, learning rate, and batch size, can be determined based on training experience, the characteristics of the preprocessed training dataset, and the characteristics of the initial AI model. In one training method, the initial AI model is trained according to the set hyperparameters. During training, a loss function is used to calculate the loss value between the AI ​​model's predicted value and the target value of the input image. The parameters of the AI ​​model are updated according to the loss value until all data in the training dataset are used for training according to the set hyperparameters. It should be understood that this application does not limit the specific method of training the initial AI model. The training method will change accordingly depending on the structure of the initial AI model and the set training hyperparameters. However, all training requires the use of the training dataset, and the purpose of training is to enable the initial AI model to learn the characteristics and patterns of the data in the training dataset, so that the initial AI model can predict any other data that is similar to or of the same type as the data in the training dataset.

[0100] S305: Evaluate the trained AI model and analyze the training of the initial AI model.

[0101] In step S304, the initial AI model is trained using the training dataset. In step S305, the AutoML system 100 evaluates the trained AI model using the test dataset. Specifically, it uses data from the test dataset as input to the trained AI model and calculates the prediction accuracy of the trained AI model for the test data. When the dataset contains multiple data types, the evaluation of the trained AI model can calculate the prediction accuracy for each type of data in the test dataset separately. After evaluation, the results are compared with the user's pre-input expectation of the final AI model's performance on the GUI. When the trained AI model fails to meet the expected performance, further analysis is conducted on the impact of several data types with poor prediction accuracy on the AI ​​model's accuracy in predicting the same type of data and other types of data during the initial AI model training process. Incremental experiments can be used to analyze the change in the AI ​​model's prediction accuracy after each addition of a fixed amount of training data for training. Furthermore, based on the curve relationship between the amount of data used to train the AI ​​model and the prediction accuracy of the AI ​​model, the expected effect on the prediction accuracy of the AI ​​model after further adding one or more types of data for optimized training can be predicted. A specific process for evaluating the trained AI model and training the initial AI model in one embodiment will be described in subsequent S3051-S3054.

[0102] Optionally, after evaluating the trained AI model, S305 compares the evaluation results with the user's pre-input expectations for the AI ​​model's performance on the GUI. When the trained AI model meets the performance expectations, S305 does not perform further analysis, but instead prompts the user through the GUI that an AI model that meets the user's performance expectations has been trained, provides the user with the download of the trained AI model, or prompts the user that the trained AI model can be used online.

[0103] S306: Provide users with feedback on the evaluation results, analysis results, optimization methods, and expected results after optimization.

[0104] Based on the aforementioned evaluation in S305, the evaluation results of the trained AI model can be obtained. These results include: the prediction accuracy of the currently trained AI model on the test dataset (for datasets with multiple data types, the evaluation results also include the prediction accuracy of the trained AI model for each data type). Based on the aforementioned analysis in S305, the analysis results of the initial AI model training can be obtained. These results include the change in prediction accuracy of the intermediate AI model relative to the baseline AI model after each training iteration. Based on the change in prediction accuracy and each data type, the benefit coefficient of each data type on the prediction accuracy of the intermediate AI model is obtained. The optimization method is a method suggested by the AutoML system 100 to the user based on the analysis results to optimize the trained AI model. For example, if the training dataset includes four data types: A, B, C, and D, and the analysis results show that adding 10% of the total training dataset data of type A data can improve the AI ​​model's prediction accuracy for type A data, as well as the prediction accuracy for types B and C data, then the optimization method is to "add 10% of the total training dataset data of type A data". The AutoML system 100 also provides feedback to users on the expected optimization effects of the optimization methods. For example, after increasing the total amount of training dataset data by 10% to include Class A data, the expected effect of the AI ​​model is: the AI ​​model's prediction accuracy for Class A data is expected to increase by 4.2%, the AI ​​model's prediction accuracy for Class B data is expected to increase by 1.5%, and the AI ​​model's prediction accuracy for Class C data is expected to increase by 6.3%.

[0105] It should be understood that when a user uploads a new training dataset according to the optimization method provided by the AutoML system 100, the AutoML system 100 will use the already trained AI model as the initial AI model and execute a process similar to the above S302, S304, S305 and S306 using the new training dataset. That is, the data in the new training dataset will be preprocessed; the preprocessed new training dataset will be used to continue to optimize and train the already trained AI model that has been determined by the above S303 and trained by the above S304; the optimized AI model will be evaluated and analyzed, and further analysis results, optimization methods and expected optimization effects will be provided to the user. Until the user chooses not to follow the optimization method, or after S304 has completed training the AI ​​model and compared the prediction accuracy of the currently trained AI model with the user's preset performance expectations, and it is determined that the currently trained AI model has met the user's performance expectations, the AutoML system will no longer execute processes similar to those described in S302, S304, S305, and S306. Instead, it will prompt the user on the GUI that the AI ​​model has been trained according to the user's needs and is available for download or online use.

[0106] Through the automated machine learning process described in S301-S306 of the AutoML system 100, users can obtain more informative analysis results of AI model training, optimization methods for the trained AI model, and expected results after optimization. This allows users to decide whether to follow the optimization methods suggested by the AutoML system based on this information and their actual situation. Alternatively, users can choose to abandon further optimization of the trained AI model after balancing the prediction accuracy, expected results after optimization, and time and manpower costs. Optimizing and analyzing AI model training and providing reliable optimization methods can truly enable users without professional AI knowledge to easily obtain satisfactory AI models to achieve their task objectives.

[0107] Figure 6 The following is a flowchart illustrating a specific method for evaluating a trained AI model and analyzing the training of an initial AI model in one embodiment. Figure 6 Taking the user's task objective as obtaining an AI model for image classification, and the user-uploaded dataset consisting of a training dataset and a test dataset containing four classes (A, B, C, and D), the method for evaluating and analyzing the AI ​​model in S305 above is described in detail:

[0108] S3051: Evaluate the trained AI model using the test dataset and calculate the prediction accuracy of the trained AI model on various types.

[0109] Specifically, the data in the test dataset are sequentially input into the trained AI model, which then outputs a predicted type for each input data point. The predicted types are further compared with the true types of the input data, and the prediction accuracy of the trained AI model for each of the four types (A, B, C, and D) in the test dataset is calculated. The prediction accuracy for each type is the ratio of the number of correctly predicted images of that type in the test dataset to the total number of images of that type in the test dataset. For example, if there are 20 images of type A in the test dataset, and the trained AI model accurately predicts 18 of them as type A, then the prediction accuracy for type A is 90%.

[0110] It is worth noting that in S306, the prediction accuracy of the trained AI model on various types of test datasets can be displayed to the user via the GUI, allowing the user to intuitively understand the performance of the currently trained AI model on various types of data. For example: Figure 7 This is a diagram illustrating the prediction accuracy of a trained AI model on a test dataset, as presented on a GUI.

[0111] S3052: Analyze the impact of one or more types of data on the predictive accuracy of an AI model when used to train it.

[0112] Specifically, based on the prediction accuracy of the trained AI model across various types obtained in S3051, N types with poor prediction accuracy in the training dataset are identified, and incremental experiments are conducted on these N types respectively. Here, N is a positive integer greater than or equal to 1, and the value of N can be determined by a combination of factors, such as training time cost, the current ranking of AI model prediction accuracy, etc. For example, for types such as... Figure 7 Based on the prediction accuracy shown, N is set to 2, and incremental experiments are conducted using types A and B.

[0113] The main idea behind incremental experiments is as follows: First, retrain the initial AI model using a baseline set to obtain a baseline AI model. Then, evaluate the prediction accuracy of the baseline AI model for each type of data in the test dataset. Next, gradually add one type of data to the baseline AI model for training, obtaining the correlation coefficient between the incremental data sequence for that type and the AI ​​model's prediction accuracy for each type of data in the test dataset. The incremental data sequence for one type can be represented as [NA1, NA2, ..., NA...]. i , ..., NA k ], where i and k are both positive integers greater than 0, and i is less than or equal to k. NA iNA represents the number of data points of this type used to train the AI ​​model after the i-th addition of data. k This represents the number of data points of this type used to train the AI ​​model after the last data addition. As the amount of data of one type is gradually increased during AI model training, the sequence of prediction accuracy of the AI ​​model for the j-th type of data in the test dataset can be represented as [ΔPA]. j 1, ΔPA j 2, ..., ΔPA j 1, ..., ΔPA j k [j], where j is a positive integer greater than 0. It should be understood that gradually adding a type of data to the AI ​​model training process yields an incremental sequence of prediction accuracy for all data types in the test dataset. For example, gradually using class A data to train the AI ​​model yields an incremental sequence of prediction accuracy for the AI ​​model on the four classes A, B, C, and D in the test dataset as class A data is added during training. The following describes the specific method for incrementally adding class A data to the AI ​​model training process and its impact on the prediction accuracy of the AI ​​model on data types A, B, C, and D. The specific steps are as follows:

[0114] 1. Divide the preprocessed training dataset into a baseline set and an increment set. The ratio between the baseline set and the increment set can be determined by the AutoML system 100 based on empirical values. Different ratios can be set for different task objectives.

[0115] 2. Retrain the initial AI model determined in step S303 using the benchmark set to obtain a benchmark AI model. Evaluate the prediction accuracy of the benchmark AI model for each type using the test dataset to obtain the benchmark prediction accuracy for each type. The benchmark prediction accuracy for the j-th type of data is denoted as PA. j 0.

[0116] It should be understood that the specific methods for retraining and evaluating the AI ​​model in steps 1 and 2 are similar to those in steps S304 and S305, and will not be repeated here.

[0117] 3. Divide the data of class A in the incremental set into k parts, where the number of data in each part can be the same or different. Each time a part of class A data is added, train the AI ​​model. After each training iteration, calculate the prediction accuracy of the currently trained intermediate AI model for classes A, B, C, and D in the test dataset. Calculate the change in each prediction accuracy relative to the baseline prediction accuracy. The change in the prediction accuracy of the intermediate AI model for class j after the i-th addition of class A data relative to the baseline AI model's prediction accuracy for class j is denoted as ΔPA.j i After adding class A data for the kth time, four prediction accuracy variable sequences are obtained (these are the prediction accuracy sequences of the intermediate AI model for classes A, B, C, and D relative to the baseline prediction accuracy during the training process of adding class A data). The prediction accuracy sequence for class j data represents the set of changes in the intermediate AI model's prediction accuracy for class j data in the test dataset relative to the baseline prediction accuracy after adding class A data for the (1-k)th times. For example, after each addition of class A data, the prediction accuracy of the intermediate AI model for class B data in the test dataset may change; the prediction accuracy sequence for class B data represents each change.

[0118] 4. Calculate the correlation coefficient between the incremental sequence of data of type A and the prediction accuracy variable sequence corresponding to each data type. The correlation coefficient can be calculated using the Pearson correlation coefficient or other commonly used statistical correlation coefficients, such as the Spearman coefficient and Kendall coefficient. For example, after adding type A data for AI model training, obtain the prediction accuracy variable sequences corresponding to types A, B, C, and D. Calculate the correlation coefficient between the incremental sequence of type A and the prediction accuracy variable sequences corresponding to types A, B, C, and D, respectively. The correlation coefficients for types A, B, C, and D are denoted as rA. A rA B rA C rA D Therefore, this step reveals the impact of adding type A data for AI model training on the AI ​​model's prediction of type A, B, C, and D data. This impact can be determined by the correlation coefficient. When the correlation coefficient between the incremental sequence of type A data and the corresponding prediction accuracy variable sequence is large and positive (correlation coefficient is positive), it can be concluded that adding type A data for AI model training has a positive impact on the prediction accuracy of type A data, improving the AI ​​model's accuracy in predicting type A data. When the correlation coefficient between the incremental sequence of type A data and the corresponding prediction accuracy variable sequence of type B data is large and negative (correlation coefficient is negative), it can be concluded that adding type A data for AI model training has a negative impact on the prediction accuracy of type B data, reducing the AI ​​model's accuracy in predicting type B data. When the correlation coefficient between the incremental sequence of type A data and the corresponding prediction accuracy variable sequence of type C data is small, it can be concluded that adding type A data for AI model training has little impact on the prediction accuracy of type C data.

[0119] It is worth noting that the methods in steps 3 and 4 above are performed once for all N types of data, thereby obtaining the correlation coefficient that increases the change in the prediction accuracy of each type of data for AI models in predicting the same type of data and other types of data.

[0120] S3053: Calculate the benefit factor for increasing the predictive accuracy of the intermediate AI model by adding a type of data.

[0121] Specifically, a pre-set correlation coefficient threshold is compared with each obtained correlation coefficient. The incremental sequences and prediction accuracy variable sequences corresponding to correlation coefficients greater than or equal to the threshold are then subjected to further regression analysis. The regression analysis method can be linear regression analysis. Taking the incremental sequence as the incremental sequence of type A data and the corresponding prediction accuracy sequence as the prediction accuracy variable sequence of the AI ​​model for type B data after adding type A data as an example, the incremental sequences [NA1, NA2, ..., NA1] are used. i , ..., NA k ] and the corresponding prediction accuracy sequence [ΔPA] B 1, ΔPA B 2, ..., ΔPA B i , …, ΔPA B k According to the formula:

[0122] [ΔPA B t ΔPA B 2, ..., ΔPA B i , …, ΔPA B k ]=bA B *[NA1, NA2, ..., NA] i , ..., NA k ]+hA B

[0123] The calculation represents the gain coefficient bA, which indicates the improvement in the prediction accuracy of an AI model trained on type A data for type B data. B Similarly, according to the above formula, calculate all the benefit coefficients of the AI ​​model trained with additional type A data relative to the prediction accuracy of similar data and other data. The total benefit coefficient of the AI ​​model trained with additional type A data is the sum of all the benefit coefficients of the AI ​​model trained with additional type A data relative to the prediction accuracy of similar data and other data, denoted as BA.

[0124] The calculations in the aforementioned steps S3052 and S3053 can be used Figure 8 The calculation diagram shown is an example, such as... Figure 8As shown, in step S3052, the correlation coefficient rA between the incremental sequence of the newly added A-class data used to train the baseline AI model and the prediction accuracy of the intermediate AI model for the three classes A, B, and C data was calculated. A rA B rA C Since the pre-set correlation coefficient threshold is 0.6, it can be determined that the addition of Class A data has a significant impact on the prediction accuracy of Class A and Class B data, but a smaller impact on the prediction accuracy of Class C data. Therefore, the benefit coefficient bA of using the added Class A data for AI model training on the AI ​​model's prediction of Class A and Class B data is further calculated. A bA B According to bA A bA B The total revenue coefficient for the prediction accuracy of the intermediate AI model based on the newly added Class A data was calculated to be 5.6.

[0125] It should be understood that, in S3051, the N types with poor prediction accuracy are used to calculate, in S3052 and S3053, the impact (correlation coefficient, benefit coefficient) of each of these N types on the data of each type in the intermediate AI model prediction test dataset, and the total benefit coefficient for the AI ​​model. The N total benefit coefficients are then sorted, and one or more new types corresponding to larger benefit coefficients can be selected as the one or more data types that should be prioritized for users.

[0126] It is worth noting that in S306, the impact of adding a new type of data obtained in the aforementioned steps S3052 and S3053 on the intermediate AI model's prediction of the same type of data and its impact on the prediction of different types of data, as well as the overall benefit coefficient of adding a new type of data on the prediction accuracy of the intermediate AI model, can all be displayed to the user on the GUI. Furthermore, the AutoML system 100 also recommends one or more data types that should be added to the user based on these analysis results. For example: Figure 9 As shown, after the analysis in steps S3052 and S3053, the AutoML system 100 displays the optimization method to the user on the GUI. The user can clearly see from the GUI the data types that the AutoML system 100 recommends to be added. Furthermore, the user can choose to view the analysis results to understand why the AutoML system 100 recommends adding this or more types of data types.

[0127] S3054: Calculate the expected effect on the prediction accuracy of the AI ​​model after training with an additional class of data.

[0128] Based on the aforementioned S3053, after each addition of a new data category to the AI ​​model, the prediction accuracy of the intermediate AI model for each data category is calculated. The overall prediction accuracy of the intermediate AI model after each training iteration can be the average or a weighted average of the prediction accuracy for each category in each iteration (the weighting coefficient can be determined based on the amount of data of each category in the test dataset). For example, the incremental data of type A used to train the AI ​​model is [NA1, NA2, ..., NA...]. i , ..., NA k As type A data increases, the prediction accuracy sequence of the trained intermediate AI model for type A data is [PA]. A 1, PA A 2, ..., PA A i , ..., PA A k The prediction accuracy sequence of the trained intermediate AI model for type B data is [PA]. B 1, PA B 2, ..., PA B i , ..., PA B k The prediction accuracy sequence of the trained intermediate AI model for type C data is [PA]. C 1, PA C 2, ..., PA C i , ..., PA C k The prediction accuracy sequence of the trained intermediate AI model for type D data is [PA]. D 1, PA D 2, ..., PA D i , ..., PA D k By calculating the average prediction accuracy of each of these four sequences, the prediction accuracy sequence of the trained intermediate AI model during the increase of type A data can be obtained as [PA1, PA2, ..., PA...]. i , ..., PA k The data increment of type A [NA1, NA2, ..., NA] will be calculated. i , ..., NA k The prediction accuracy sequence [PA1, PA2, ..., PA] of the intermediate AI model trained. i , ..., PA kCurve fitting is performed to obtain a formula G that can characterize the relationship of the curve. Based on formula G, the expected prediction accuracy of the AI ​​model after training with a certain amount of additional A-type data can be calculated. Based on the expected accuracy, the expected effect of the prediction accuracy of the AI ​​model after training with a certain amount of additional A-type data can be further calculated.

[0129] Optionally, for calculating the prediction accuracy sequence [PA1, PA2, ..., PA] of the AI ​​model trained during the data increment of type A. i , ..., PA k Alternatively, following the incremental experiment method described in S3052 above, type A data can be gradually added to progressively train the baseline AI model. The intermediate AI model after each training iteration is evaluated using test data to obtain the prediction accuracy of the intermediate AI model on the entire test data, thus obtaining the prediction accuracy sequence [PA1, PA2, ..., PA...]. i , ..., PA k ].

[0130] It should be understood that, in one embodiment, after S3053 is completed, for the data types (which can be one or more data types) suggested in the optimization method, the expected effect of calculating the overall prediction accuracy of the AI ​​model after training with the data of the suggested data types mentioned in S3054 can be performed. For example, in S3053, the AutoML system 100 suggests to the user to continue adding data of type A based on analysis. In S3054, the AutoML system 100 continues to calculate the expected effect of the prediction accuracy of the AI ​​model after training with the data of type A, so as to display it to the user. In another embodiment, S3054 can calculate the expected effect of the prediction accuracy of the AI ​​model after training with the data of each type for each data type analyzed in S3053.

[0131] It is worth noting that in S306, the prediction accuracy curve obtained by the aforementioned fitting, as well as the expected effect of the AI ​​model's prediction accuracy after further calculation of adding a certain amount of data, can be displayed on the GUI, so that users can decide whether to add data in the optimization manner based on the expected effect of the AI ​​model's prediction accuracy. Figure 10 This is a GUI that displays a graph showing the prediction accuracy of an AI model trained using Class A data. In the graph, the horizontal axis represents the amount of Class A data, and the vertical axis represents the prediction accuracy of the AI ​​model after that amount of Class A data was used to train the AI ​​model. Figure 10As shown, users can see that adding 200 Class A data points for training will improve the AI ​​model's overall prediction accuracy to 95.6%; adding 1000 Class A data points will improve the overall prediction accuracy to 97.9%. Optionally, users can... Figure 10 You can also click on any point on the curve with the mouse arrow, and the GUI will display the number of type A data points added to that point on the curve, as well as the expected effect on the AI ​​model's prediction accuracy after using this number of type A data points to continue training the AI ​​model.

[0132] It should be understood that although the methods described in S3051-S3054 are illustrated using image classification as an example of the user's task objective, the methods for analyzing AI models, providing users with optimization methods and expected optimization results, as described in S3051-S3054, can actually be applied to various task objectives. This application does not limit the type of task objective; any AI model that requires training on different datasets can be optimized and analyzed using the aforementioned methods, providing users with more accurate and reliable optimization methods and expected results. For example, the user's task objective could be license plate recognition, face recognition, object detection, video review, etc.

[0133] The AutoML system 100 provided in this application, when performing optimization analysis, can classify the dataset based on one or more attributes of the data in the user-uploaded dataset, rather than on the labeled classification of the data. These attributes include factors such as the background color of an image, the creation date of a video, and the country of origin of the text. Furthermore, the impact of each type of data under one or more attribute classifications on the training of the AI ​​model is analyzed.

[0134] The following is combined Figure 11 Another embodiment provided in this application is described below:

[0135] S401: The AutoML system 100 receives the task objective and dataset selected by the user on the GUI. The task objective is license plate recognition, and the dataset includes different license plates from various countries. Each license plate in the dataset is labeled with the string corresponding to its license plate number.

[0136] S402: The AutoML system 100 preprocesses the dataset based on the user's dataset. The preprocessing operations include one or more of those mentioned in S302 above, which will not be described in detail here.

[0137] S403: The AutoML system 100 determines an initial AI model for the user to achieve the task objectives based on the task objectives.

[0138] S404: AutoML system 100 uses a dataset to train an AI model and obtain a trained AI model.

[0139] S405: AutoML system 100 classifies license plates in both the training and test datasets according to their background color. Background color is an attribute of the data in the dataset; for example, it can be categorized into four types: black, green, blue, and red. The color-classified test dataset is used to evaluate the performance of the trained AI model, and the color-classified training dataset is used to analyze the training of the initial AI model.

[0140] The license plates in the test dataset were input into the trained AI model. Similar to S3051 above, the prediction accuracy of the currently trained AI model for green, blue, black, and red license plates was evaluated. It was found that the prediction accuracy of the trained AI model for strings in license plates with black and red backgrounds was poor.

[0141] Based on the methods described in S3052-S3054 above, this paper analyzes the impact of using license plates with black and red backgrounds in the training dataset to train the initial AI model on the prediction accuracy of the AI ​​model for license plates of the same color and other colors. It calculates the total benefit coefficient of adding a color type of data to the AI ​​model's prediction accuracy, and further calculates the expected effect on the overall prediction accuracy of the AI ​​model after training with an additional color type of data. The specific implementation of the above evaluation and analysis methods is the same as described in S3051-S3054 above, and will not be repeated here.

[0142] S406: Based on the evaluation and analysis in S405 above, display the analysis results and optimization methods to the user. Optimization methods could include: adding license plates with a black background to further optimize the AI ​​model. The user can also be provided with the expected effects after adding a certain number of optimized AI models, such as the percentage increase in the AI ​​model's prediction accuracy.

[0143] In the above embodiments, although the data uploaded by users is not classified by color attribute, the AutoML system 100 performs attribute (color) classification on the dataset when optimizing the AI ​​model in order to analyze the impact of license plate background on character recognition. This allows the prediction accuracy of the trained AI model for license plates of different colors to be analyzed, providing users with another way to optimize the AI ​​model.

[0144] Optionally, in another embodiment, when analyzing the trained AI model and the training dataset, the AutoML system 100 can classify the dataset according to multiple attributes and analyze the impact of each dataset category under each attribute on the AI ​​model training. For example, when the user's task is face recognition, the training and test datasets can be classified according to the gender of the faces in the dataset, into male and female, and the accuracy of the trained AI model in recognizing males and females, as well as the impact of male and female training data on the accuracy of the AI ​​model, can be analyzed. The training and test datasets can also be classified according to the age of the faces in the dataset, into 20-30, 30-40, 40-50, 50-60, and 60 and above, and the accuracy of the trained AI model in recognizing faces of different age groups, as well as the impact of training data for each age group on the accuracy of the AI ​​model, can be analyzed. This is because the AutoML system analyzes the training of the AI ​​model from two aspects of attributes. The AutoML system 100 offers users optimization methods through its GUI, including adding female facial data and facial data of people aged 60 and above.

[0145] This application also provides a method such as Figure 1 The AutoML system 100 shown includes modules and functions as described above, which will not be repeated here. In one embodiment, the user I / O module 101 in the AutoML system 100 is specifically used to execute the methods described in steps S301 and S306 above or to execute the methods described in steps S401 and S406 above; the data preprocessing module 102 is specifically used to execute the methods described in step S302 above or to execute the methods described in step S402 above; the model determination module 103 is specifically used to execute the methods described in step S303 above or to execute the methods described in step S403 above; the model training module 104 is specifically used to execute the methods described in step S304 above or to execute the methods described in step S404 above; and the model optimization analysis module 105 is specifically used to execute the methods described in step S305 above or to execute the methods described in step S405 above.

[0146] It is worth noting that, in one embodiment, the model optimization analysis module is also specifically used to execute the aforementioned S3051-S3054.

[0147] This application also provides a method such as Figure 4 The computing device 200 shown has a processor 202 that reads the program and data set stored in the memory 201 to execute the aforementioned AutoML system execution method.

[0148] Since the various modules in the AutoML system 100 provided in this application can be distributed and deployed on multiple computers in the same or different environments, this application also provides a... Figure 12 The computing device shown includes multiple computers 500, each computer 500 including a memory 501, a processor 502, a communication interface 503, and a bus 504. The memory 501, processor 502, and communication interface 503 are interconnected via the bus 504.

[0149] The memory 501 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 501 can store programs. When the program stored in the memory 501 is executed by the processor 502, the processor 502 and the communication interface 503 are used to execute parts of the AutoML system's methods for training and optimizing AI models for the user. The memory can also store datasets. For example, a portion of the storage resources in the memory 501 can be allocated as a dataset storage module to store the datasets required by the AutoML system, and a portion of the storage resources in the memory 501 can be allocated as an AI model storage module to store an AI model library.

[0150] The processor 502 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0151] Processor 502 can also be an integrated circuit chip with signal processing capabilities. In implementation, some or all functions of the AutoML system of this application can be completed through integrated logic circuits in the hardware of processor 502 or instructions in software form. The aforementioned processor 502 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application below. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. Processor 502 reads the information in memory 501 and, in conjunction with its hardware, completes some of the functions of the AutoML system in this application embodiment.

[0152] Communication interface 503 uses transceiver modules, such as, but not limited to, transceivers, to enable communication between computer 500 and other devices or communication networks. For example, a dataset can be obtained through communication interface 503.

[0153] Bus 504 may include a pathway for transmitting information between various components of computer 500 (e.g., memory 501, processor 502, communication interface 503).

[0154] Each of the aforementioned computers 500 establishes a communication path through a communication network. Each computer 500 runs any one or more of the following modules: user I / O module 101, data preprocessing module 102, model determination module 103, model training module 104, model optimization and analysis module 105, dataset storage module 106, and AI model storage module 107. Any computer 500 can be a computer in a cloud data center (e.g., a server), a computer in an edge data center, or a terminal computing device.

[0155] The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis. For parts of a process that are not described in detail, please refer to the relevant descriptions of other processes.

[0156] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product providing AutoML includes one or more computer instructions for performing AutoML, which, when loaded and executed on a computer, produce, in whole or in part, the functionality according to the embodiments of the present invention. Figure 5 , Figure 6 or Figure 11 The process or function described.

[0157] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide AutoML. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD).

Claims

1. A method for automated machine learning (AutoML), characterized in that, include: Receive the user's task objective and the first dataset; An initial artificial intelligence (AI) model is determined based on the task objective, and the initial AI model is used to achieve the task objective for the user. The initial AI model is trained based on the first dataset to obtain a trained AI model. The training of the initial AI model is analyzed based on the first dataset to obtain analysis results. The analysis results include the impact of at least one type of data in the first dataset on the training of the initial AI model. The impact includes the effect of increasing the training of the initial AI model with the at least one type of data on the prediction accuracy of the initial AI model for each type of data in the first dataset. Based on the analysis results, the system provides the user with optimization methods for the trained AI model, including uploading a second dataset for optimizing the trained AI model.

2. The method as described in claim 1, characterized in that, The method further includes: The expected performance of the trained AI model optimization is provided to the user, the expected performance indicating the prediction accuracy that the trained AI model will achieve after optimized training based on the second dataset.

3. The method as described in claim 1, characterized in that, The first dataset includes a training dataset and a test dataset; Before analyzing the training of the initial AI model based on the first dataset to obtain the analysis results, the method further includes: Evaluate the prediction accuracy of the trained AI model for each type of data in the test dataset; The step of analyzing the training of the initial AI model based on the first dataset to obtain analysis results specifically includes: Based on the prediction accuracy of each type of data in the test dataset, at least one type of data in the training dataset is determined to be used for analyzing the training of the initial AI model; Using an incremental experiment method, the impact of at least one type of data in the training dataset on the training of the initial AI model is analyzed, and the analysis results are obtained.

4. The method as described in claim 3, characterized in that, The incremental experiment method is used to analyze the impact of at least one type of data in the training dataset on the training of the initial AI model, and the analysis results are obtained. Specifically, this includes: The training dataset is divided into a baseline set and an incremental set; The initial AI model is trained using the benchmark set to obtain a benchmark AI model; For each type of data in at least one type of data in the incremental set, the data of each type is divided into multiple parts, and one part of the multiple parts of data is added successively to train the baseline AI model to obtain the intermediate AI model. Calculate the change in prediction accuracy of the intermediate AI model relative to the baseline AI model after each training iteration; Based on the change in the prediction accuracy and each type of data, obtain the benefit coefficient of each type of data in the at least one type of data on the prediction accuracy of the intermediate AI model.

5. The method as described in claim 4, characterized in that, The second dataset includes one or more types of data, wherein the data in the second dataset is of the type whose profit coefficient for the prediction accuracy of the intermediate AI model is greater than a preset threshold.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Receive the second dataset uploaded by the user; The trained AI model is optimized and trained based on the second dataset.

7. The method according to any one of claims 1-5, characterized in that, Before analyzing the training of the initial AI model based on the first dataset to obtain the trained AI model, the method further includes: The data in the first dataset is classified according to the attributes of the data in the first dataset.

8. The method according to any one of claims 1-5, characterized in that, The data in the first dataset and the second dataset are labeled, and the data types in the first dataset and the second dataset are the same as the data labeled in the first dataset and the second dataset.

9. The method according to any one of claims 1-5, characterized in that, The method further includes: The data in the received first dataset and second dataset are preprocessed respectively, wherein the preprocessing includes one or more of the following operations: (1) Modify the dimensions and specifications of the data; (2) Verify the data; (3) Encode and convert the data; (4) Classify the data by attributes; (5) Extract features from the data.

10. An AutoML (Auto Machine Learning) system, characterized in that, The system includes: The user input / output I / O module is used to receive the user's task objectives and the first dataset. A model determination module is used to determine an initial artificial intelligence (AI) model based on the task objective, the initial AI model being used to achieve the task objective for the user. The model training module is used to train the initial AI model based on the first dataset to obtain a trained AI model. The model optimization analysis module is used to analyze the training of the initial AI model based on the first dataset and obtain analysis results. The analysis results include the impact of at least one type of data in the first dataset on the training of the initial AI model. The impact includes the effect of increasing the training of the initial AI model with the at least one type of data on the prediction accuracy of the initial AI model for each type of data in the first dataset. The user I / O module is also used to provide the user with optimization methods for the trained AI model based on the analysis results. The optimization methods include uploading a second dataset for optimizing the trained AI model.

11. The system as claimed in claim 10, characterized in that, The user I / O module is also used to provide the user with the expected effect of the optimized trained AI model, the expected effect indicating the prediction accuracy that the trained AI model will achieve after optimized training based on the second dataset.

12. The system as described in claim 10, characterized in that, The first dataset includes a training dataset and a test dataset; the model optimization analysis module is further used to: evaluate the prediction accuracy of the trained AI model for each type of data in the test dataset; The model optimization analysis module is used to analyze the training of the initial AI model based on the first dataset, and when obtaining the analysis results, it is specifically used for: Based on the prediction accuracy of each type of data in the test dataset, at least one type of data in the training dataset is determined to be used for analyzing the training of the initial AI model; Using an incremental experiment method, the impact of at least one type of data in the training dataset on the training of the initial AI model is analyzed, and the analysis results are obtained.

13. The system as described in claim 12, characterized in that, The model optimization analysis module is used to analyze the impact of at least one type of data in the training dataset on the training of the initial AI model using an incremental experiment method. When obtaining the analysis results, it is specifically used for: The training dataset is divided into a baseline set and an incremental set; The initial AI model is trained using the benchmark set to obtain a benchmark AI model; For each type of data in at least one type of data in the incremental set, the data of each type is divided into multiple parts, and one part of the multiple parts of data is added successively to train the baseline AI model to obtain the intermediate AI model. Calculate the change in prediction accuracy of the intermediate AI model relative to the baseline AI model after each training iteration; Based on the change in the prediction accuracy and each type of data, obtain the benefit coefficient of each type of data in the at least one type of data for the prediction accuracy of the intermediate AI model.

14. The system as described in claim 13, characterized in that, The second dataset includes one or more types of data, wherein the data type in the second dataset is the type of data whose profit coefficient for the prediction accuracy of the intermediate AI model is greater than a preset threshold.

15. The system according to any one of claims 10-14, characterized in that, The user I / O module is also used to receive the second dataset uploaded by the user; The model training module is also used to optimize the trained AI model based on the second dataset.

16. The system according to any one of claims 10-14, characterized in that, The model optimization analysis module is also used for: The data in the first dataset is classified according to the attributes of the data in the first dataset.

17. The system according to any one of claims 10-14, characterized in that, The data in the first dataset and the second dataset are labeled, and the data types in the first dataset and the second dataset are the same as the data labeled in the first dataset and the second dataset.

18. The system according to any one of claims 10-14, characterized in that, The system also includes: A data preprocessing module is used to preprocess the received first dataset and the second dataset respectively, wherein the preprocessing includes one or more of the following operations: (1) Modify the dimensions and specifications of the data; (2) Verify the data; (3) Encode and convert the data; (4) Classify the data by attributes; (5) Extract features from the data.

19. A computing device, characterized in that, The computing device includes a memory and a processor, the memory being used to store a set of computer instructions; The processor executes a set of computer instructions stored in the memory to perform the method of any one of claims 1 to 9.

20. A non-transient readable storage medium, characterized in that, The non-transient readable storage medium stores computer program code, which, when executed by a computing device, performs the method described in any one of claims 1 to 9.

21. A computer program product, characterized in that, The computer program product includes computer program code, which, when executed by a computing device, performs the method described in any one of claims 1 to 9.

Images