Image classification method, device, equipment and storage medium based on integrated meta-learning

Through the integrated meta-learning method, multiple meta-learners and meta-integration modules are used to integrate the image classification results, which solves the adaptive problem of image classification tasks in small data sets and different scenarios, and achieves fast and accurate image classification , and effectively train with limited computing resources.

CN114037859BActive Publication Date: 2024-11-26TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111258636.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-27
Publication Date
2024-11-26
Estimated Expiration
2041-10-27

AI Technical Summary

Technical Problem

The prior art is difficult to implement fast training on small data sets that have never been seen in image classification tasks, and it is unable to adapt to different scenarios, and the meta-learning task cannot be completed under the limitation of computing resources.

Method used

Using the integrated meta-learning method, the images to be classified are transmitted to multiple meta-learners through the central controller, and the results are classified and integrated. The meta-integration module is used to integrate multiple classification results, and the training process is dynamically adjusted to adapt to computing resources and time. limit.

Benefits of technology

It realizes adaptive completion of image classification tasks with less training data, improves classification speed and accuracy, and can be used in different scenarios without retraining, and is suitable for environments with limited resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114037859B_ABST
    Figure CN114037859B_ABST
Patent Text Reader

Abstract

The embodiments of the present application relate to the field of data processing technology, and specifically to an image classification method, device, equipment and storage medium based on integrated meta-learning. It is intended to dynamically control the training process of image classification tasks and adaptively complete image classification tasks in various scenarios when there is less training data. The method comprises: inputting the image to be classified into a central controller, transmitting the image to be classified to multiple meta-learners through the central controller; classifying the image to be classified by each meta-learner in the multiple meta-learners to obtain multiple classification results, and sending the multiple classification results to the central controller; sending the multiple classification results to the meta-integration module through the central controller, and integrating the multiple classification results through the meta-integration module to obtain the final classification result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically, to an image classification method, apparatus, device and storage medium based on integrated meta-learning. Background Art

[0002] Image classification has a wide range of applications in various scenarios in life, such as image review, object recognition, etc. Using deep learning to solve image classification problems is a hot topic in current research. However, the use of deep learning requires a large amount of high-quality training data, which is often difficult to obtain. How to achieve good and fast training on a small data set that has never been seen before is an urgent problem to be solved. In order to achieve this goal, the concept of meta-learning is proposed, that is, to allow the model to quickly learn new tasks based on the existing "knowledge". In the prior art, the first method is to extract the meta-knowledge of the data set to the best model by learning a mapping from a data set to model parameters; the second method is to learn a good model initialization parameter and predict each step of the gradient update, so that the relevant image classification model can complete the meta-learning task within a limited data set and a limited number of steps; the third method is to use an encoder to map the image to a representation space, and then use manually defined metrics to guide the learning of the mapping.

[0003] In the existing technology, the required hyperparameters and model architectures for metadata datasets of different scenarios are different, which makes it impossible to truly achieve scene adaptation. If the scene changes too drastically, their meta-learning models will need to reset the hyperparameters and retrain. In addition, the existing technology has no way to take into account the limitations of computing resources and cannot complete meta-learning tasks with limited resources. Summary of the invention

[0004] The embodiments of the present application provide an image classification method, apparatus, device and storage medium based on integrated meta-learning, which aims to dynamically control the training process of image classification tasks and adaptively complete image classification tasks in various scenarios with less training data.

[0005] A first aspect of an embodiment of the present application provides an image classification method integrating meta-learning, the method comprising:

[0006] Inputting the to-be-classified images into a central controller, and transmitting the to-be-classified images to a plurality of meta-learners through the central controller;

[0007] Classify the to-be-classified images by each of the multiple meta-learners to obtain multiple classification results, and send the multiple classification results to a central controller;

[0008] The central controller sends the multiple classification results to the meta-integration module, and the meta-integration module performs integration operations on the multiple classification results to obtain a final classification result.

[0009] Optionally, the method further comprises:

[0010] Inputting a first preset number of labeled images into the central controller, and transmitting the first preset number of labeled images to the plurality of meta-learners through the central controller;

[0011] The features of the first preset number of labeled images are recorded by the multiple meta-learners.

[0012] Optionally, the multiple meta-learners are obtained through training, and the steps of training and verifying each meta-learner in the multiple meta-learners include:

[0013] Inputting a meta-training data set including a second preset number of labeled images into the meta-learner;

[0014] Using the meta-training data set, fine-tuning parameters of the pre-trained convolutional network encoder in the meta-learner through a direction prediction enhancement task and an image label classification task to obtain a fine-tuned convolutional network encoder;

[0015] Inputting the meta-validation dataset into the fine-tuned convolutional network encoder to obtain an encoding vector;

[0016] The encoding vector is input into a parameter-free decoder in the meta-learner to obtain a prediction result.

[0017] Optionally, the meta-integration module is obtained through training, and the training steps include:

[0018] Inputting the prediction results obtained by the multiple meta-learners into the meta-integration module;

[0019] The meta-integration module uses multiple classifiers to perform integration operations on the prediction results respectively to obtain multiple integration operation results;

[0020] According to the multiple integrated operation results, a classifier whose integrated operation is most accurate among the multiple classifiers is selected as the best classifier.

[0021] Optionally, each of the multiple meta-learners classifies the to-be-classified image to obtain multiple classification results, including:

[0022] Extracting features of the image to be classified by using the fine-tuned convolutional network in the meta-learner to obtain image features;

[0023] Predicting the category probability of each image according to the image features by means of a parameter-free decoder in the meta-learner to obtain a category probability distribution of the image to be classified;

[0024] The category probability distribution of the image to be classified is updated by using an iterative update method. After the iterative update is completed, the final category probability distribution is obtained.

[0025] Optionally, the method further comprises:

[0026] Setting the start time and stop time of training in the central controller;

[0027] When the start time is reached, the central controller controls the multiple meta-learners and the meta-integration module to start training;

[0028] When the stop time is reached, the central controller controls the plurality of meta-learners and the meta-integration module to stop training.

[0029] Optionally, the method further comprises:

[0030] Obtaining the remaining time of the training through the central controller;

[0031] The algorithm used in the training process is dynamically adjusted according to the computing resources and the remaining time.

[0032] A second aspect of an embodiment of the present application provides an image classification device based on integrated meta-learning, the device comprising:

[0033] A first data input module, used to input the to-be-classified images into a central controller, and transmit the to-be-classified images to a plurality of meta-learners through the central controller;

[0034] An image classification module is used to classify the to-be-classified image through each of the multiple meta-learners to obtain multiple classification results, and send the multiple classification results to a central controller;

[0035] The integrated operation module is used to send the multiple classification results to the meta-integration module through the central controller, and perform integrated operation on the multiple classification results through the meta-integration module to obtain a final classification result.

[0036] Optionally, the device further comprises:

[0037] A second data input module, configured to input a first preset number of labeled images into the central controller, and transmit the first preset number of labeled images to the plurality of meta-learners through the central controller;

[0038] A feature recording module is used to record the features of the first preset number of labeled images through the multiple meta-learners.

[0039] Optionally, the device further comprises:

[0040] A third data input module, configured to input a meta-training data set including a second preset number of labeled images into the meta-learner;

[0041] A parameter fine-tuning module, used to use the meta-training data set to fine-tune the parameters of the pre-trained convolutional network encoder in the meta-learner through a direction prediction enhancement task and an image label classification task to obtain a fine-tuned convolutional network encoder;

[0042] A meta-verification module, used for inputting a meta-verification dataset into the fine-tuned convolutional network encoder to obtain an encoding vector;

[0043] The result acquisition module is used to input the encoding vector into the parameter-free decoder in the meta-learner to obtain a prediction result.

[0044] Optionally, the device further comprises:

[0045] a fourth data input module, configured to input the prediction results obtained by the plurality of meta-learners into the meta-integration module;

[0046] An integrated operation module, used for the meta-integration module to use multiple classifiers to perform integrated operations on the prediction results respectively to obtain multiple integrated operation results;

[0047] The classifier determination module is used to select a classifier with the most accurate ensemble operation among the multiple classifiers as the best classifier based on the multiple ensemble operation results.

[0048] Optionally, the picture classification module includes:

[0049] A feature extraction submodule, used to extract features of the image to be classified through the fine-tuned convolutional network in the meta-learner to obtain image features;

[0050] A probability prediction submodule, used to predict the category probability of each picture according to the picture features through the parameter-free decoder in the meta-learner, so as to obtain the category probability distribution of the picture to be classified;

[0051] The iterative update submodule is used to update the category probability distribution of the image to be classified by using an iterative update method, and obtain the final category probability distribution after the iterative update is completed.

[0052] Optionally, the device further comprises:

[0053] A time setting module, used to set the start time and stop time of training in the central controller;

[0054] A first control module, configured to control the plurality of meta-learners and the meta-integration module to start training through the central controller when the start time is reached;

[0055] The second control module is used to control the multiple meta-learners and the meta-integration module to stop training through the central controller when the stop time is reached.

[0056] Optionally, the device further comprises:

[0057] A time acquisition module, used to acquire the remaining time of the training through the central controller;

[0058] The dynamic adjustment module is used to dynamically adjust the algorithm used in the training process according to the computing resources and the remaining time.

[0059] A third aspect of an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0060] A fourth aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect of the present application are implemented.

[0061] Using the image classification method of integrated meta-learning provided by the present application, the image to be classified is input into the central controller, and the image to be classified is transmitted to multiple meta-learners through the central controller; the image to be classified is classified by each meta-learner in the multiple meta-learners to obtain multiple classification results, and the multiple classification results are sent to the central controller; the multiple classification results are sent to the meta-integration module through the central controller, and the multiple classification results are integrated by the meta-integration module to obtain the final classification result. In the present application, a central controller is used to control multiple meta-learners, resources are reasonably allocated, and each meta-learner is fully utilized, which improves the speed of image classification. The training and prediction of the meta-learner and the meta-integration module can be dynamically adjusted according to the computing resources and the required time during training, so that the entire system can complete the training in a short time, and multiple meta-learners are used to identify the images to be classified, which is suitable for image recognition in various scenes, and can be well adapted in various scenes with huge differences, without retraining, and realizes the adaptation of different scenes, which is more convenient and quick, and the prediction results of multiple meta-learners are integrated by using the meta-integration module, and the prediction results obtained are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0063] Figure 1 is a flowchart of an image classification method based on integrated meta-learning proposed in one embodiment of the present application;

[0064] Figure 2 is a schematic diagram of an integrated meta-learning system proposed in an embodiment of the present application;

[0065] Figure 3 Schematic diagram of the training and verification process of the meta-learner proposed in one embodiment of the present application;

[0066] Figure 4 This is a control schematic diagram of a central controller in a meta-training phase proposed in an embodiment of the present application;

[0067] Figure 5 It is a schematic diagram of an image classification device based on integrated meta-learning proposed in one embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] refer to Figure 1 , Figure 1 : is a flowchart of an image classification method based on integrated meta-learning proposed in one embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0070] S11: Inputting the image to be classified into a central controller, and transmitting the image to be classified to multiple meta-learners through the central controller.

[0071] In this embodiment, the central controller is a processor that uses a combination of multi-threading and multi-processing to perform data processing, data distribution, and round number estimation. The meta-learner includes a pre-trained convolutional neural network and a parameter-free decoder, which determines the category of feature data based on the distance between the input features. The convolutional neural network in the meta-learner is pre-trained. Based on the "knowledge" that has been learned, the meta-learner only needs a small amount of training data to quickly adapt to new scenarios and achieve adaptation to new scenarios.

[0072] In this embodiment, when performing the task of image classification, the images to be classified are first input into the central controller, and the central controller transmits the images to be classified to multiple meta-learners. Each meta-learner will receive all the images to be classified input into the central controller. Each meta-learner focuses on different scenes. Using multiple meta-learners can include various types and scenes as much as possible.

[0073] For example, there are four central controllers, and the pre-trained convolutional neural networks in each central controller are different, namely ResNet50, ResNet152, WRN50 and MobileNet. The pictures to be classified can be pictures of different animals, such as pictures containing cats, pictures containing dogs, etc.

[0074] S12: Classify the to-be-classified images by each of the multiple meta-learners to obtain multiple classification results, and send the multiple classification results to the central controller.

[0075] In this embodiment, multiple meta-learners classify the images to be classified respectively, that is, each meta-learner classifies all the images to be classified and gives a classification result to each image to be classified. The classification result is expressed as the probability distribution of each image. After the classification results are obtained, the multiple classification results are sent to the central controller.

[0076] For example, there are four meta-learners, each of which classifies a picture to be processed. The result output by meta-learner 1 is that the probability that the animal in the picture is a dog is 90%, and the probability that it is a cat is 10%; the result output by meta-learner 2 is that the probability that the animal in the picture is a dog is 95%, and the probability that it is a cat is 5%; the result output by meta-learner 3 is that the probability that the animal in the picture is a dog is 87%, and the probability that it is a cat is 13%; the result output by meta-learner 4 is that the probability that the animal in the picture is a dog is 82%, and the probability that it is a cat is 18%.

[0077] S13: The central controller sends the multiple classification results to a meta-integration module, and the meta-integration module performs integration operations on the multiple classification results to obtain a final classification result.

[0078] In this embodiment, the meta-integration module includes a classifier, which is used to perform an integration operation on the classification results output by multiple meta-learners to obtain a final classification result, that is, which category the object in each picture belongs to.

[0079] For example, for a picture to be classified, the input is input into four meta-learners respectively to obtain four classification results, and these four classification results are input into the classifier in the meta-integration module. The classifier integrates these four classification results to obtain the final probability distribution of the picture. The classifier can be LightGBM, Random Forest, Voting and Naive Bayes classifier.

[0080] In another embodiment of the present application, before inputting the to-be-classified image into the central controller and transmitting the to-be-classified image to the plurality of meta-learners through the central controller, the method further includes:

[0081] S21: Input a first preset number of labeled images into the central controller, and transmit the small number of labeled images to the multiple meta-learners through the central controller.

[0082] S22: Recording features of the first preset number of labeled images through the multiple meta-learners.

[0083] In this embodiment, the first preset number can be set by itself, and a small number of pictures can be searched for annotation to obtain labeled pictures. When the pictures to be classified are identified by the meta-learner, a small number of labeled pictures need to be input into the meta-learner. The types of labeled pictures are the same as the types of pictures to be classified. The meta-learner records the features of these labeled pictures for classification of the pictures to be classified according to the recorded features. This process is different from the training process, and the parameters of the meta-learner will not be adjusted, but only the features of a small number of labeled pictures are recorded. When the scene to be identified changes greatly, the labeled pictures corresponding to the new scene can be input into the meta-learner.

[0084] For example, if all the images to be classified are animal images, then first input 3 cat images, and these 3 images are labeled as cats, and then input 3 dog images, and these 3 images are labeled as dogs. The meta-learner records the features of these 6 images and classifies the subsequent animal images. When the recognition scene changes, for example, the images to be classified become images of vehicles, then a small number of labeled images of vehicles are input into the meta-learner.

[0085] In this embodiment, a small number of labeled images can be input into the meta-learner, so that the meta-learner records the features of these images and can adapt to new scenes without changing its own parameters. It can achieve good scene adaptation and there is no need to retrain the network for each new scene.

[0086] In another embodiment of the present application, the multiple meta-learners are obtained through training, and the steps of training and verifying each meta-learner in the multiple meta-learners include:

[0087] S31: Inputting a meta-training dataset containing a second preset number of labeled images into the meta-learner.

[0088] In this embodiment, the meta-training data set includes a second preset number of labeled images. The second preset number is much lower than the amount of data required for traditional training and is a controllable number that can be determined based on existing resources. The meta-learner is composed of a pre-trained network, and its training process is the same as that of an ordinary image feature extraction network. Therefore, only a small amount of training data is needed to train the entire learner. The labels on the images are pre-marked to indicate the categories of the objects in the images.

[0089] For example, multiple types of data are divided into multiple meta-training data sets, each of which contains a small amount of training data. These training data are all labeled. For example, the transportation category includes car pictures, airplane pictures, and train pictures. Only a few representative pictures are needed for each transportation, and there is no need to find a large number of pictures for the same transportation. The animal category includes cat pictures, dog pictures, lion pictures, etc., and only a few representative pictures are needed.

[0090] In this embodiment, when training the meta-learner, because the feature extraction network in the meta-learner is pre-trained, when it is necessary to perform image classification tasks, only a small amount of data needs to be used to fine-tune the parameters of the feature extraction network, which greatly reduces the time spent on training, and when the recognized image scenes are very different, there is no need to retrain the entire network, which greatly improves the generalization of the entire model.

[0091] S32: Using the meta-training data set, fine-tune the parameters of the pre-trained convolutional network encoder in the meta-learner through a direction prediction enhancement task and an image label classification task to obtain a fine-tuned convolutional network encoder.

[0092] In this embodiment, the direction prediction enhancement task is to generate several pictures with changed angles for each picture, and use the meta-learner to predict the angle of the picture change. The label classification task is to classify the labeled pictures, use the labels of the pictures to verify the classification results, and adjust the parameters of the network according to the verification results. The pre-trained convolutional network encoder is a network that extracts features from pictures and recognizes them. Parameter fine-tuning is to adaptively modify the parameters of the network. The modification range is smaller than that of traditional training, and less training sample data is required, so the parameter adjustment time is shorter.

[0093] In this embodiment, multiple types of annotated pictures are used as training data and put into a meta-training data set, so that the meta-training data set contains multiple types of subsets, each subset contains a small number of training samples, the meta-training data set is input into the meta-learner, and for each picture, a picture with a transformed angle is generated, and the meta-learner is used to predict the transformed angle. The meta-learner fine-tunes the parameters of the pre-trained convolutional network encoder during the training process, and then obtains the fine-tuned convolutional network encoder. The direction prediction task can enable the model to obtain a better image representation space.

[0094] For example, the meta-training dataset contains 3 types of samples, each type of sample constitutes a subset, each subset contains 5 pictures, and the categories can be transportation, food, and animals. For each picture, 3 pictures are generated that are rotated 90 degrees, 180 degrees, and 270 degrees respectively. The meta-training dataset and the pictures with changed angles are input into the meta-learner, and the meta-learner predicts the types of these pictures and the angles of the changes. The parameters of the pre-trained convolutional network encoder in the meta-learner are fine-tuned through the loss function to obtain the fine-tuned convolutional network encoder.

[0095] S33: Input the meta-validation dataset into the fine-tuned convolutional network encoder to obtain an encoding vector.

[0096] In this embodiment, the meta-validation dataset includes a support set query set, the support set is annotated images, the query set is unannotated images, and the encoding vector is a vector extracted from the input image by a fine-tuned convolutional network encoder.

[0097] In this embodiment, after the meta-learner is trained, the meta-verification data set is input into the fine-tuned convolutional network encoder, and the fine-tuned convolutional network performs feature extraction on the images in the meta-verification data set to obtain encoding vectors.

[0098] For example, the meta-validation dataset includes 5 labeled images as a support set and 20 unlabeled images as a query set. These 25 images in the meta-validation dataset are input into the fine-tuned convolutional network encoder, and the fine-tuned convolutional network encoder extracts the corresponding encoding vector.

[0099] S34: Input the encoding vector into the parameter-free decoder in the meta-learner to obtain a prediction result.

[0100] In this embodiment, the parameter-free decoder is used to calculate the category of each image based on the encoding vector, mainly to determine which category the image in the query set belongs to based on the distance between the encoding vector of the labeled image in the support set and the encoding vector of the image in the query set.

[0101] In this embodiment, in the meta-learner, the fine-tuned convolutional network encoder transmits the encoding vector to the parameter-free decoder. The parameter-free decoder uses the vector corresponding to the labeled image in the support set as a reference point, and calculates the distance between the encoding vector corresponding to the image in the query set and the reference point. The closer the encoding vector is to the reference point, the closer the corresponding image category is.

[0102] For example, the pictures in the support set include car pictures and airplane pictures and they have been labeled. The coding vectors corresponding to the pictures in the support set are used as reference points to obtain two reference points. Reference point one is the coding vector corresponding to the car picture, and reference point two is the coding vector corresponding to the airplane picture. If the coding vector corresponding to the picture in the query set is close to reference point one, the probability that the object in the picture corresponding to the coding vector is a car is relatively high. If the coding vector corresponding to the picture in the query set is close to reference point two, the probability that the object in the picture corresponding to the coding vector is an airplane is relatively high.

[0103] In another embodiment of the present application, the meta-integration module is obtained through training, and the training steps include:

[0104] S41: Inputting the prediction results obtained by the multiple meta-learners into the meta-integration module.

[0105] In this embodiment, during the verification phase of the meta-learner, the meta-integration module is also trained. A plurality of classifiers are pre-set in the meta-integration module for performing integration operations on the prediction results to obtain the final results.

[0106] By way of example, the classifier may be LightGBM, Random Forest, Voting, and Naive Bayes classifier, etc.

[0107] S42: The meta-integration module uses multiple classifiers to perform integration operations on the prediction results respectively to obtain multiple integration operation results.

[0108] In this embodiment, the prediction result is the probability distribution of each image in the query set, and the integrated operation is to perform an integrated operation on these probability distributions based on the probability distribution of each image output by multiple meta-learners, and to obtain the integrated operation result of classifying each image by combining the probability distribution results obtained by multiple meta-learners.

[0109] For example, there are 4 classifiers in the meta-integration module. Each of these 4 classifiers will perform an integration operation on the prediction results output by multiple meta-learners to obtain an integration operation result. For each image in the query set of the meta-validation set, an integration operation result will be obtained, that is, a classification result will be given to each image.

[0110] S43: According to the multiple integrated operation results, select a classifier with the most accurate integrated operation among the multiple classifiers as the best classifier.

[0111] In this embodiment, for each image in the meta-verification data set, each classifier will obtain an integrated operation result, and each classifier will output multiple integrated operation results. These integrated operation results are the categories of the images in the meta-verification data set.

[0112] In this embodiment, in the meta-verification stage, each classifier in the meta-integration module performs an integration operation on the prediction results output by multiple meta-learners to obtain multiple integration operation results, and selects a classifier with the most accurate integration operation from the multiple classifications as the best classifier. That is, the classifier that most accurately classifies the images in the meta-verification set is selected as the best classifier.

[0113] For example, there are 30 images in the meta-validation set, and there are 4 classifiers in the meta-integration module. The classification of 25 images in the ensemble operation results obtained by classifier 1 is correct, 28 images in the ensemble operation results obtained by classifier 2 are correct, 28 images in the ensemble operation results obtained by classifier 3 are correct, and 30 images in the ensemble operation results obtained by classifier 4 are correct. Classifier 4 is selected as the best classifier. The trained meta-integration module only uses classifier 4 for ensemble operation in subsequent classification tasks.

[0114] In this embodiment, in order to adapt to various scenarios, multiple classifiers are used to achieve meta-integration. When the categories of pictures to be classified are quite different, only a small number of labeled pictures need to be input, and the meta-integration module will select a suitable classifier for subsequent integration operations, further improving the overall model's ability to generalize to different data sets and scenarios.

[0115] In another embodiment of the present application, each of the multiple meta-learners classifies the image to be classified to obtain multiple classification results, including:

[0116] S51: extracting features of the image to be classified through the fine-tuned convolutional network in the meta-learner to obtain image features.

[0117] In this embodiment, after the entire model training is completed, the image to be classified is input into the meta-learner, and the fine-tuned convolutional network extracts features of the image to be classified to obtain the features of the image.

[0118] In this embodiment, the fine-tuned convolutional network extracts features from all samples of each class in the classified image dataset, and at the same time, performs representation mapping on all samples of each class and takes the average to obtain the average representation of the samples of each class, and uses these average representations as the initialized prototype representations.

[0119] For example, for a small data set containing N classes, each class has K samples, N average representations can be obtained through mapping, and K image features can be obtained.

[0120] S52: predicting the category probability of each image according to the image features through the parameter-free decoder in the meta-learner to obtain the category probability distribution of the image to be classified.

[0121] In this embodiment, the parameter-free decoder uses the extracted features of the image to be classified as sample points, and obtains the probability distribution of each sample point according to the distance from each sample point to the average representation, that is, the probability of each image belonging to that category.

[0122] For example, for a small data set containing N classes and K samples in each class, N average representations and K sample points can be obtained through mapping. The probability distribution of each sample point can be obtained by the distance from the average representation. For example, there are 3 classes and 5 sample points in each class. The probability that sample point 1 belongs to category 1 is 0.98, the probability that the sample point belongs to category 2 is 0.65, and the probability that the sample point belongs to category 3 is 0.21.

[0123] S53: using an iterative updating method to update the category probability distribution of the image to be classified, and after the iterative updating is completed, a final category probability distribution is obtained.

[0124] In this embodiment, an iterative update method is used, that is, the category probability of each sample point is used as a weight to update the prototype representation, and a new probability distribution of the sample point is obtained, and the final probability distribution of the sample point is obtained by iterating several rounds.

[0125] For example, there are 3 classes, each class has 5 sample points, the probability that sample point 1 belongs to class 1 is 0.98, the probability that sample point 1 belongs to class 2 is 0.65, and the probability that sample point 1 belongs to class 3 is 0.21. After several rounds of iterations, the probability that sample point 1 belongs to class 1 is 0.98, the probability that sample point 1 belongs to class 2 is 0.19, and the probability that sample point 1 belongs to class 3 is 0.18. It can be determined that sample point 1 belongs to class 1, that is, the class of the image 1 to be classified is class 1.

[0126] In another embodiment of the present application, the method further includes:

[0127] S61: Setting the start time and stop time of training in the central controller.

[0128] In this embodiment, when training the entire integrated meta-learning model, the start time and stop time of the entire training can be set by the central controller. Optionally, the number of training rounds can also be set. For example, the start time is set to 9.00am and the stop time is set to 9.30am. The number of training rounds can also be set to 200 rounds.

[0129] S62: When the start time is reached, the central controller controls the multiple meta-learners and the meta-integration module to start training.

[0130] S63: When the stop time is reached, the central controller controls the multiple meta-learners and the meta-integration module to stop training.

[0131] In this embodiment, when the preset training start time is reached, the central controller sends a training start signal to control the meta-learner and meta-integration module to start training. When the preset training end time is reached, the central controller sends a training end signal to control the meta-learner and meta-integration module to stop training. It is also possible not to set the training duration and number of rounds, and the training will automatically stop when the model training reaches convergence.

[0132] The training process also includes the following steps:

[0133] S71: Obtain the remaining time of the training through the central controller.

[0134] S72: Dynamically adjust the algorithm used in the training process according to the computing resources and the remaining time.

[0135] In this embodiment, the central controller will detect the entire training process in real time, obtain the remaining time of training, and dynamically adjust the algorithm used in the entire training process according to the existing computing resources and the remaining time. When there are more computing resources and remaining time, a more complex algorithm is used to train the entire model to ensure the training effect. When computing resources and remaining time are scarce, a simpler algorithm is used to train the entire model.

[0136] For example, the algorithm can be divided into three levels: the first level algorithm has low complexity and occupies less computing resources and time; the second level algorithm has medium complexity and occupies medium computing resources and time; the third level algorithm has high complexity and occupies more computing resources and time. During the training process, the central controller uses the third level algorithm to train the entire model when the training time and computing time are sufficient, and uses the first level algorithm to train the entire model when the training time and computing resources are scarce.

[0137] In this embodiment, a centralized controller is used to balance the learning process and prediction process of multiple meta-learners, a combination of multi-threading and multi-process is used for data processing, data distribution and round number estimation, and a grading system is used to dynamically adjust the model's decoding strategy according to the remaining time, so as to make full use of computing time and computing resources, which can play a greater role in scenarios where there are requirements for model training time and computing resources are limited.

[0138] In another embodiment of the present application, Figure 2 As shown, Figure 2 is a schematic diagram of an integrated meta-learning system proposed in an embodiment of the present application, combined with Figure 2 The embodiments of the present application are further described as follows. Figure 2 As stated, Figure 2 First, the meta-training / validation / testing data generator is used to generate the meta-training data set, the meta-validation data set, and the meta-testing data set. The above data are input into the central controller, which sends the data to the four meta-learners and sends control signals to control the four meta-learners for training. Each meta-learner inputs the predicted probability into the meta-integration module, and the meta-integration module is trained using the meta-validation data set. After the training is completed, the image to be classified is input into the central controller, which sends the image to the four meta-learners, and the four meta-learners send the prediction results to the meta-integration module, which outputs the final prediction results.

[0139] In this embodiment, a meta-learning system is designed to cope with image classification tasks in different scenarios and distributions. By means of enhancing generalization such as integrated learning and pre-training, this meta-learning system does not require any form of parameter adjustment for new data sets, and achieves true adaptation to different scenarios. At the same time, the present invention also includes a central controller designed independently, which can dynamically adjust the training process according to computing resources and remaining time while greatly improving the training efficiency.

[0140] In another embodiment of the present application, Figure 3 As shown, Figure 3 Schematic diagram of the training and verification process of the meta-learner proposed in one embodiment of the present application. Figure 3 As described above, in the meta-training stage, there are L batches of Z samples, i.e., L classes of samples, and the number of samples in each class is Z. Here, only a small number of samples are needed. The sample data is input into the pre-trained convolutional network encoder to obtain the image features of each sample, and the image is classified by the classifier to obtain the loss value, and the parameters of the pre-trained network convolutional network encoder are fine-tuned to obtain the fine-tuned convolutional network encoder.

[0141] In the meta-validation / testing phase, the images of the support set query set are input into the fine-tuned convolutional network encoder to obtain the encoding vector, which is then input into the parameter-free decoder. The parameter-free decoder obtains the prediction result, i.e., the classification result of each image in the query set, based on the label of the support set image and the image features corresponding to the image in the query set, i.e., the encoding vector.

[0142] In another embodiment of the present application, Figure 4 As shown, Figure 4 : is a schematic diagram of the control of the central controller in the meta-training phase proposed in an embodiment of the present application, such as Figure 4 As described, the data is input into the central controller, and the central controller uses 4 sub-processes to pre-process the data according to the time limit, and inputs the pre-processed data into 4 meta-learners respectively, and controls the training of the 4 meta-learners through 4 independent sub-processes.

[0143] Based on the same inventive concept, an embodiment of the present application provides an image classification device based on integrated meta-learning. Figure 5 , Figure 5 FIG. 5 is a schematic diagram of an image classification device 500 based on integrated meta-learning proposed in one embodiment of the present application. Figure 5 As shown, the device comprises:

[0144] A first data input module 501 is used to input the to-be-classified pictures into a central controller, and transmit the to-be-classified pictures to a plurality of meta-learners through the central controller;

[0145] The image classification module 502 is used to classify the to-be-classified image through each of the multiple meta-learners to obtain multiple classification results, and send the multiple classification results to the central controller;

[0146] The integrated operation module 503 is used to send the multiple classification results to the meta-integration module through the central controller, and perform integrated operation on the multiple classification results through the meta-integration module to obtain a final classification result.

[0147] Optionally, the device further comprises:

[0148] A second data input module, configured to input a first preset number of labeled images into the central controller, and transmit the first preset number of labeled images to the plurality of meta-learners through the central controller;

[0149] A feature recording module is used to record the features of the first preset number of labeled images through the multiple meta-learners.

[0150] Optionally, the device further comprises:

[0151] A third data input module, configured to input a meta-training data set including a second preset number of labeled images into the meta-learner;

[0152] A parameter fine-tuning module, used to use the meta-training data set to fine-tune the parameters of the pre-trained convolutional network encoder in the meta-learner through a direction prediction enhancement task and an image label classification task to obtain a fine-tuned convolutional network encoder;

[0153] A meta-verification module, used for inputting a meta-verification dataset into the fine-tuned convolutional network encoder to obtain an encoding vector;

[0154] The result acquisition module is used to input the encoding vector into the parameter-free decoder in the meta-learner to obtain a prediction result.

[0155] Optionally, the device further comprises:

[0156] a fourth data input module, configured to input the prediction results obtained by the plurality of meta-learners into the meta-integration module;

[0157] An integrated operation module, used for the meta-integration module to use multiple classifiers to perform integrated operations on the prediction results respectively to obtain multiple integrated operation results;

[0158] The classifier determination module is used to select a classifier with the most accurate ensemble operation among the multiple classifiers as the best classifier based on the multiple ensemble operation results.

[0159] Optionally, the picture classification module includes:

[0160] A feature extraction submodule, used to extract features of the image to be classified through the fine-tuned convolutional network in the meta-learner to obtain image features;

[0161] A probability prediction submodule, used to predict the category probability of each picture according to the picture features through the parameter-free decoder in the meta-learner, so as to obtain the category probability distribution of the picture to be classified;

[0162] The iterative update submodule is used to update the category probability distribution of the image to be classified by using an iterative update method, and obtain the final category probability distribution after the iterative update is completed.

[0163] Optionally, the device further comprises:

[0164] A time setting module, used to set the start time and stop time of training in the central controller;

[0165] A first control module, configured to control the plurality of meta-learners and the meta-integration module to start training through the central controller when the start time is reached;

[0166] The second control module is used to control the multiple meta-learners and the meta-integration module to stop training through the central controller when the stop time is reached.

[0167] Optionally, the device further comprises:

[0168] A time acquisition module, used to acquire the remaining time of the training through the central controller;

[0169] The dynamic adjustment module is used to dynamically adjust the algorithm used in the training process according to the computing resources and the remaining time.

[0170] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the image classification method based on integrated meta-learning as described in any of the above embodiments of the present application.

[0171] Based on the same inventive concept, another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps of the image classification method based on integrated meta-learning described in any of the above embodiments of the present application are implemented.

[0172] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0174] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0175] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0179] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0180] The above is a detailed introduction to the image classification method, device, equipment and storage medium based on integrated meta-learning provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image classification method based on ensemble meta-learning, characterized in that: The method comprises: Inputting the to-be-classified images into a central controller, and transmitting the to-be-classified images to a plurality of meta-learners through the central controller; Classify the to-be-classified images by each of the multiple meta-learners to obtain multiple classification results, and send the multiple classification results to a central controller; The central controller sends the multiple classification results to the meta-integration module, and the classifier in the meta-integration module performs integration operation on the multiple classification results to obtain a final classification result; The classifying the to-be-classified image by each of the multiple meta-learners to obtain multiple classification results includes: Extracting features of the image to be classified by using the fine-tuned convolutional network in the meta-learner to obtain image features; Predicting the category probability of each image according to the image features by means of a parameter-free decoder in the meta-learner to obtain a category probability distribution of the image to be classified; The category probability distribution of the image to be classified is updated by using an iterative update method. After the iterative update is completed, the final category probability distribution is obtained.

2. The method according to claim 1, before inputting the to-be-classified image into the central controller and transmitting the to-be-classified image to the plurality of meta-learners through the central controller, the method further comprises: Inputting a first preset number of labeled images into the central controller, and transmitting the first preset number of labeled images to the plurality of meta-learners through the central controller; The features of the first preset number of labeled images are recorded by the multiple meta-learners.

3. The method according to claim 1, characterized in that The multiple meta-learners are obtained through training, and the steps of training and verifying each meta-learner in the multiple meta-learners include: Inputting a meta-training data set including a second preset number of labeled images into the meta-learner; Using the meta-training data set, fine-tuning parameters of the pre-trained convolutional network encoder in the meta-learner through a direction prediction enhancement task and an image label classification task to obtain a fine-tuned convolutional network encoder; Inputting the meta-validation dataset into the fine-tuned convolutional network encoder to obtain an encoding vector; The encoding vector is input into a parameter-free decoder in the meta-learner to obtain a prediction result.

4. The method according to claim 3, characterized in that The meta-integration module is obtained through training, and the training steps include: Inputting the prediction results obtained by the multiple meta-learners into the meta-integration module; The meta-integration module uses multiple classifiers to perform integration operations on the prediction results respectively to obtain multiple integration operation results; According to the multiple integrated operation results, a classifier whose integrated operation is most accurate among the multiple classifiers is selected as the best classifier.

5. The method according to claim 1, characterized in that The method further comprises: Setting the start time and stop time of training in the central controller; When the start time is reached, the central controller controls the multiple meta-learners and the meta-integration module to start training; When the stop time is reached, the central controller controls the plurality of meta-learners and the meta-integration module to stop training.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining the remaining time of the training through the central controller; The algorithm used in the training process is dynamically adjusted according to the computing resources and the remaining time.

7. An image classification device based on integrated meta-learning, characterized in that: The device comprises: A first data input module, used to input the to-be-classified images into a central controller, and transmit the to-be-classified images to a plurality of meta-learners through the central controller; An image classification module is used to classify the to-be-classified image through each of the multiple meta-learners to obtain multiple classification results, and send the multiple classification results to a central controller; An integrated operation module, used for sending the multiple classification results to the meta-integration module through the central controller, and performing integrated operation on the multiple classification results through the classifier in the meta-integration module to obtain a final classification result; The classifying the to-be-classified image by each of the multiple meta-learners to obtain multiple classification results includes: Extracting features of the image to be classified by using the fine-tuned convolutional network in the meta-learner to obtain image features; Predicting the category probability of each image according to the image features by means of a parameter-free decoder in the meta-learner to obtain a category probability distribution of the image to be classified; The category probability distribution of the image to be classified is updated by using an iterative update method. After the iterative update is completed, the final category probability distribution is obtained.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.