Two-stage image classification method based on model search

Abstract

The present invention discloses a two-stage image classification method based on model search, the purpose of which is to solve the problems of low image classification accuracy and long classification time in existing image classification methods. The technical scheme is: construct an image classification model library M and an image classification system containing two search modules, train the first search module, use the trained first search module to search in M ​​according to the user input demand text q, obtain the 20 models that best match q, and the second search module scores and sorts the 20 models to obtain the optimal image classification model; train the optimal image classification model to obtain the trained optimal image classification model; use the trained optimal image classification model to classify the unlabeled pictures uploaded by the user. The present invention can quickly and accurately search for a suitable image classification model from the model library according to the user's selection requirements, improve the image classification accuracy and classification speed, and reduce the user's professional requirements.

Description

Technical Field

[0001] The present invention relates to the field of image classification, and in particular to a two-stage image classification method based on model search. Background Art

[0002] Image classification is an important task in computer vision, which aims to assign input images to predefined categories, such as cats, dogs, airplanes, etc. Image classification has a wide range of applications, such as face recognition, medical image analysis, autonomous driving, etc.

[0003] At present, there are mainly the following image classification methods:

[0004] Methods based on manual features: This type of method mainly uses manually designed feature extractors, such as SIFT, HOG, SURF, etc., to describe the image globally or locally, and then uses classifiers, such as KNN, SVM, decision trees, etc., to classify the image. The advantage of this type of method is that it is simple and easy to implement, and is suitable for some simple classification tasks. The disadvantage is that the design of the feature extractor depends on human experience and prior knowledge, and it is difficult to adapt to complex and changeable image data.

[0005] Methods based on deep learning: This type of method mainly uses deep neural networks as classification models, such as convolutional neural networks (CNNs), to perform end-to-end learning and classification of images. The advantages of this type of method are that it can automatically learn high-level semantic features of images, has strong expression and generalization capabilities, and is suitable for some complex and large-scale classification tasks. The disadvantage is that it requires a large amount of labeled data and computing resources, and it is difficult to explain the working mechanism inside the network.

[0006] Methods based on transfer learning: This type of method mainly uses deep neural network models that have been trained on large-scale data sets, such as VGG, ResNet, Inception, etc., as feature extractors or classification models, fine-tunes or adapts the classification model according to the target task, and then classifies the images of the target task. The advantage of this type of method is that it can use existing knowledge and models, reduce training time and data volume, and improve classification accuracy. The disadvantage is that it is necessary to select appropriate source and target domains to avoid negative transfer or overfitting. Even if the appropriate source and target domains are selected, a lot of experiments are still required to select the most suitable model for a specific task, and it is difficult to quickly achieve good classification results.

[0007] However, the classification models used in these methods are not the most suitable deep neural network models for image classification, and the model parameters of the models are not optimal. A lot of manual attempts and adjustments are required to make the models suitable for image classification, resulting in low image classification accuracy and long classification time. Moreover, these methods lack adaptability and flexibility, so they are difficult to adapt to different image data sets and image classification task requirements. The models need to be redesigned or modified for each problem, resulting in high professional requirements for users, huge workload, and low classification accuracy when using these methods.

[0008] There have been some image classification methods based on model search, such as REA (Regularized evolution for image classifier architecture search, 2019) and GDAS (Searching for a robust neural architecture in four GPU hours, 2019). These methods train all models in the model library, obtain the classification accuracy of the models on the user dataset, and then sort them according to the classification accuracy, and select the model with the highest accuracy to perform the image classification task. This type of method only considers the adaptation to the user dataset, without considering the needs of the user, so that the searched model is not suitable for classifying the images input by the user. In addition, the model quality in the model library of this type of method is poor, the evaluation takes a very long time, and it is easily affected by hyperparameters, resulting in the inability to correctly evaluate the model effect, so that the model ranked in front is not necessarily the most suitable model for classifying the image input by the user, resulting in low classification accuracy.

[0009] Later, researchers proposed the NASWOT (Neural Architecture Search without Training, 2021) method to evaluate model performance without training. This method can evaluate model performance without training the models in the model library and can quickly perform the model search process. However, this method also does not take user needs into consideration. Although it increases the search speed, it reduces the search effect. The classification accuracy of the searched model may not be as good as that of the REA and GDAS methods.

[0010] Therefore, if it is possible to search for the optimal or nearly optimal deep neural network model structure and parameters from a model library with high model quality according to a given data set and selection requirements, so as to quickly and accurately find an image classification model suitable for the image classification task, and use the searched image classification model for image classification, the image classification effect can be improved. This is a hot issue that technicians in this field are discussing. Summary of the invention

[0011] The technical problem to be solved by the present invention is that in the existing methods for image classification, whether by searching models or building models, the image classification accuracy is not high and the classification time is long, it is difficult to adapt to different image data sets and image classification task requirements, the user's professional requirements are high, and the workload is huge. A two-stage image classification method based on model search is provided. According to a given data set and selection requirements, a suitable image classification model is quickly and accurately searched from a model library, and the searched image classification model is used to classify images, thereby improving the image classification accuracy, increasing the classification speed, and reducing the user's professional requirements.

[0012] The technical solution of the present invention is:

[0013] Step 1: Build an image classification model library M as follows:

[0014] 1.1 Select N image classification models from the Paddleclas library (taken from https: / / pypi.org / project / paddleclas / , 2.5.1) to form the image classification model library M.

[0015] 1.2 Obtain various information about the model in M, including the name of the model, the number of model parameters, the floating-point calculation amount of the model, and the source code of the model. The method is:

[0016] 1.2.1 Get the model name and executable code file of each model in the model library M from the Paddleclas library. The executable code file of the model includes model definition, training and testing parts.

[0017] 1.2.2 Based on the ImageNet-1k dataset (taken from https: / / www.image-net.org / challenges / LSVRC / 2012 / , 2012), train and test all models in the model library, and obtain the classification accuracy of the model on the ImageNet-1k dataset. At the same time, record the inference speed of each model, that is, the time required for the model to output the prediction result from the input image.

[0018] 1.2.3 Through the thop library (taken from https: / / pypi.org / project / thop / 0.0.31-1909021322 / , 0.0.31-1909021322) to obtain the parameter quantity and floating-point calculation quantity of each model in the model library. The parameter quantity indicates the number of trainable weights in the model, reflecting the complexity and storage requirements of the model. The floating-point calculation quantity indicates the number of floating-point operations required for the model to perform a forward propagation, reflecting the computational overhead and energy consumption of the model.

[0019] 1.2.4 Read the model code from the model's executable code file to obtain the model code Code. Reorder the executable code in Code to make it clearer. Specifically, divide the model's executable code into three layers: the first layer is the model definition code, including network structure-related code and global parameter initialization code; the second layer is the definition code of other global class names and function names; the third layer is the remaining code in the code sequence Code. Put the first layer of code at the front of Code, the second layer of code after the first layer of code, and the third layer of code at the end.

[0020] 1.2.5 The model name, classification accuracy, inference speed, parameter amount, floating-point calculation amount and model code of each model in M ​​are formed into a model information sequence. Let the model information sequence of any model m = {Name, Accuracy, Latency, Param, Flops, Code}, where Name is the name of the model, Accuracy is the classification accuracy of the model, Latency is the inference speed of the model, Param is the parameter amount of the model, Flops is the floating-point calculation amount of the model, and Code is the model code.

[0021] Step 2: Build an image classification system. The image classification system consists of a model search module, an image classification module, and an image classification model library.

[0022] The model search module consists of a preprocessing module, a first search module and a second search module, wherein the first search module consists of a first feedforward neural network layer and a first linear classification layer, and the second search module consists of a scoring layer and a ranking recommendation layer.

[0023] The image classification module consists of a training module and a classification module.

[0024] The image classification model library M stores models and model information sequences corresponding to the models, including model name, classification accuracy, inference speed, parameter quantity, floating-point calculation quantity and model code.

[0025] The preprocessing module is connected to the first search module and the image classification model library, receives user input (including the user-input demand text q or the first training set Qa), uses the model information sequence in the image classification model library M to splice respectively, and processes the user input into a word level sequence.

[0026] If the preprocessing module receives the requirement text q input by the user, it concatenates q with all the model information sequences in the image classification model library M to form a set Set(qM) of q's requirement text-model information sequence, Set(qM) = {qm1,…qm n ,…,qm N}, qm n = {q,m n}, where 1≤n≤N, N is the number of model information sequences in the image classification model library M, and m n is the nth model information sequence in M. Use sentencepiece word segmentation (taken from https: / / github.com / google / sentencepiece / , 2023-11-07) to convert the N demand text-model information sequences in Set(qM) into word-level sequences, and obtain the word-level sequence set G of Set(qM), and send G to the first feedforward neural network layer. G={G(qm1),…,G(qm n ),…,G(qm N )},qm n The word level sequence G(qm n )={[CLS],q1,q2,...,q i ,...q I ,[SEP],mc1,mc2,...,mc j ,...,mc J ,[EOS]}(where q i is the i-th word in q, 1≤i≤I, I is the number of words in q, mc j Indicates m n The jth word in, 1≤j≤J, J is m n [CLS] is a special word indicating the beginning of a sequence, [SEP] is a special word that separates the requirement text sequence and the model information sequence, and [EOS] is a special word that indicates the end of a sequence).

[0027] The first feedforward neural network layer is connected to the preprocessing module and the first linear classification layer. The first feedforward neural network layer receives G from the preprocessing module and processes G(qm1),…,G(qm n ),…,G(qm N ) is linearly mapped, and the linear mapping is expressed as f(qm1),…,f(qm n ),…,f(qm N ), f(qm1),…,f(qm n ),…,f(qm N ) is sent to the first linear classification layer.

[0028] The first linear classification layer is connected to the first feedforward neural network layer and the second search module, and the first linear classification layer is connected to the first feedforward neural network layer and the second search module. n ),…,f(qm N ) to perform linear mapping, and obtain f(qm1),…,f(qm n ),…,f(qm N )’s original prediction vector (i.e., the first logits vector), performs Softmax operation on the first logits vector, obtains the probability of matching the model information sequence of the demand text q with each model in M, takes the top 20 models with the highest probability as the output of the first search module, and sends the 20 models to the second search module.

[0029] The scoring layer of the second search module is connected to the first linear classification layer and the sorting recommendation layer. The scoring layer receives 20 models from the first linear classification layer, and receives the second training set Pa, the second test set Pt, and the category table uploaded by the user. The scoring layer scores the 20 models output by the first search module according to the picture set P in the second training set Pa, and obtains an unordered score set Set (score) of the 20 models. Set (score) contains the scores of the 20 models, and sends Set (score), the second training set Pa, the second test set Pt, and the category table to the sorting recommendation layer.

[0030] The sorting recommendation layer is connected to the scoring layer and the image classification module. The scores in Set (score) received from the scoring layer are sorted in descending order, and the image classification model with the highest score is selected as the optimal image classification model (the optimal image classification model consists of the second feedforward neural network layer and the second linear classification layer). The optimal image classification model, the second training set Pa, the second test set Pt, and the category table are sent to the image classification module.

[0031] The training module in the image classification module is connected to the sorting recommendation layer and the classification module, and receives the optimal image classification model and the second training set Pa, the second test set Pt, and the category table from the sorting recommendation layer. Pa is input into the second feedforward neural network layer of the optimal image classification model, and the optimal image classification model is trained using Pa. The second feedforward neural network layer linearly maps the pictures in Pa one by one to obtain the representation ff(Pa) of Pa after linear mapping, and sends ff(Pa) to the second linear classification layer; the second linear classification layer linearly maps ff(Pa) to obtain the original prediction vector (second logits vector) of ff(Pa), and performs a Softmax operation on the second logits vector to obtain the probability of each category in the category table corresponding to the picture in Pa, and takes the category with the highest probability as the category of the picture. The training module uses the cross-loss entropy function to calculate the loss, and back-propagates the loss in the optimal image classification model to update the optimal image classification model parameters. The training stops when the loss no longer decreases. The trained optimal image classification model is sent to the classification module as the output of the training module.

[0032] The classification module is connected to the training module, and receives the trained optimal image classification model from the training module, and receives the unlabeled image p uploaded by the user. The size of p is adjusted to 64×64, and input into the trained optimal image classification model. The second feedforward neural network layer of the trained optimal image classification model performs linear mapping on p, and the linear mapping of p is represented as ff(p). ff(p) is sent to the second linear classification layer of the trained optimal image classification model. The second linear classification layer performs linear mapping on ff(p) to obtain a prediction vector (third logits vector) for p. The third logits vector is subjected to Softmax operation to obtain the probability of each category in the category table corresponding to p, and the category with the highest probability is taken as the category of p.

[0033] The third step is to build a dataset for training the model search module and the image classification module as follows:

[0034] 3.1 Prepare the data set for the first search module in the training model search module by:

[0035] 3.1.1 Describe the user requirement in natural language. The user requirement is "among the models with the top v% in inference speed, the model with the highest classification accuracy", where v represents the accuracy ratio of the user requirement, and its value range is 0<v≤100. Depending on the number of decimal places retained by v, there are a total of vt possible values, where vt is a positive integer, that is, v is a set consisting of vt possible values, v={v1,v2,…,v vvt ,…,v vt}; where vvt is a possible value of vt, 1≤vvt≤vt, and vvt is a positive integer; each possible value of vt corresponds to a user demand text. Let the user demand text set Q = {Q1, Q2, …, Q vvt ,…,Q vt}, Q vvt This is the user requirement text for Article vvt;

[0036] 3.1.2 Let vvt = 1;

[0037] 3.1.3 Take the vvtth user requirement text Q from Q vvt , find the one that matches Q from M vvt The model is as follows:

[0038] 3.1.3.1 Sort each model in M ​​from low to high according to the latency value in the model information sequence, and select the top v vvt % of the model, let it be the screening model set MM;

[0039] 3.1.3.2 Sort the models in MM from high to low according to the Accuracy values ​​in the model information sequence, and take the model with the highest Accuracy value as the one that meets the vvtth user requirement text Q vvt Model;

[0040] 3.1.4 Article vvt User Requirements Text Q vvt In line with Q vvt The model information sequence of the model of the model constitutes a demand-model pair, and the label of this demand-model pair is marked as 1; randomly select from MM that does not meet Q vvt The model of the model, and the model information sequence of the model form a demand-model pair, and the label of this demand-model pair is marked as 0; the demand-model pair and its corresponding actual label form a sample, and the form of each sample = {Q vvt ,m,lavel}, where Q vvt is the vvtth user requirement text, m is the model information sequence, label is the label of the requirement-model pair; put the sample into the data set;

[0041] 3.1.5 Let vvt = vvt + 1. If vvt ≤ vt, go to 3.1.3. Otherwise, go to 3.1.6.

[0042] 3.1.6 Construct a data set containing 2vt samples;

[0043] 3.1.7 Expand the data set to form a data set containing 6vt samples as follows:

[0044] 3.1.7.1 Let variable count = 1.

[0045] 3.1.7.2 Expand the data set by replacing synonyms. Replace the user demand text Q in the count-th sample in the data set count From "the model with the highest classification accuracy among the top v% of models in inference speed" to Q count ′, Q′ is “the model with the highest prediction accuracy among the models with the top v% in inference speed”, forming a new sample = {Q count ′,m,label}, and added to the dataset.

[0046] 3.1.7.3 Expand the data set by changing the word position. Change the user demand text Q in the demand-model pair of the count-th sample in the data set to count From "the model with the highest classification accuracy among the top v% of models in inference speed" to Q count ″, Q count ″ is “the inference speed is in the top v% among all models, and the classification accuracy is the highest”, forming a new sample = {Q count ″,m,label}, is added to the dataset.

[0047] 3.1.7.4 Let count = count + 1. If count ≤ 2000, go to 3.1.7.2; otherwise, go to 3.1.8.

[0048] 3.1.8 Divide the dataset into training set and test set in a ratio of 8 to 2. The demand-model pairs of all samples in the training set constitute the first training set Qa, and the labels of all samples in the training set constitute the actual label set Qa # ; Let Qa={Qa 1 ,Qa 2 ,···Qa o ,···,Qa O}, where Qa o represents the oth user requirement text-model pair in Qa (where 1≤o≤O, O is the total number of samples in Qa, O=0.8×6vt); Qa o = {Q o ,m},Q o is the oth user requirement text, and m is the corresponding model information sequence. The requirement-model pairs of all samples in the test set constitute the first test set Qt, and the labels of all samples in the test set constitute the actual label set Qt of the first test set # , Qa is used to train the first search module, Qa # It is used to calculate the training loss and calibrate the model when training the first search module. Qt is used to test the first search module. #Used to compare with the labels of the test output to measure the performance of the first search module under test.

[0049] 3.2 Construct the dataset of the training module in the training image classification module. The dataset comes from the collection of labeled pictures uploaded by users. The method is as follows:

[0050] 3.2.1 The user searches for pictures of the corresponding category on the Internet according to the target task and annotates them. The annotation method is as follows:

[0051] 3.2.1.1 Determine which categories of images need to be classified, set a label for each category, and make a category table.

[0052] 3.2.1.2 Collect a certain number of images for each category from the Internet, with no less than 500 images for each category, and label these images with the corresponding labels.

[0053] 3.2.2 Process the KK labeled images, delete duplicate and blurred images, adjust the size of the KK labeled images to 64×64, perform data enhancement on the KK labeled images by rotating, cropping, scaling, flipping, adjusting brightness and contrast, and obtain KK labeled images after data enhancement. KK is the total number of labeled images, KK ≥ 500.

[0054] 3.2.3 Divide the enhanced KK labeled images into training and test sets in a ratio of 8 to 2, and obtain the second training set Pa = {Pa 1 ,Pa 2 ,…,Pa kk ,…,Pa 0.8KK}, where 1≤kk≤0.8KK, 0.8KK is the total number of labeled images in Pa. At the same time, the second test set Pt is obtained, Pt={Pt 1 ,Pt 2 ,…,Pt kkk ,…,Pt 0.2KK}, where 1≤kkk≤0.2KK, 0.2KK is the total number of labeled images in Pt. The actual labels in Pa constitute the actual label set Pa of the second training set # , the pictures in Pa constitute the picture set P in Pa, and the actual labels in the Pt samples constitute the actual label set Pt of the second test set # Pa is used to train the optimal image classification model in the training module in the image classification module, Pa # It is used to calculate the training loss when training the optimal image classification model and to correct the optimal image classification model; Pt is used to test the optimal image classification model. #Used to compare with the test output label to measure the performance of the optimal image classification model for testing.

[0055] The fourth step is to use Qa to perform supervised training on the first search module in the model search module to obtain different network weight parameters. The method is as follows:

[0056] 4.1 Initialize the weight parameters and set the parameter set W of the first feedforward neural network layer F Initialize the elements in to random numbers in [0,1]; set the parameter set W of the first linear classification layer Y The elements in are initialized to random numbers in [0,1].

[0057] 4.2 Set the network training hyperparameters, set the batch size (batch_size) to 32, the network model learning rate (learning rate) to 0.001, the maximum sequence length to 200, and the control gradient accumulation parameter (gradient_accumulation_steps) to 1.

[0058] 4.3 The first feedforward neural network layer and the first linear classification layer are trained in a supervised manner. In each round of training, the predicted label sequence set Qa′ of each batch of samples is obtained, and the difference between Qa′ and the actual label sequence set Qa of each batch of samples is calculated. ## The distribution gap is obtained, the loss value is minimized and the weight parameters of the first feedforward neural network layer and the first linear classification layer are updated until the number of iterations is met. The weight parameters of the first feedforward neural network layer and the first linear classification layer are obtained as follows:

[0059] 4.3.1 Initialize the training iteration parameter itretation = 1; initialize the iteration threshold E, where E is an integer in [1, 10]. Experiments have shown that when E is 8, the Acc value of the model search (i.e., the precision rate, which indicates the ratio of the number of correct searches to the number of all samples) is the highest.

[0060] 4.3.2 Let variable bz=1.

[0061] 4.3.3 Let o=1 and let the predicted label set Qa′ be empty.

[0062] 4.3.4 Preprocessing module reads the oth sample Qa from Qa o , will Qa o Processed as a word-level sequence G(Qa o ), G(Qa o )={[CLS],Q1,…,Q ii ,…,Q II ,[SEP],mc1,…,mcjj ,…mc JJ ,[EOS]}, where 1≤ii≤II,1≤jj≤JJ, [CLS] is a special word indicating the beginning of a sequence, Q ii For QA o User Requirements Text Q o The iith word in, II is Q o The total number of words in [SEP] is a special word that separates the user requirement text and the model information sequence, mc jj Qa o The model information sequence m o The jjth word of o The number of words in G(Qa o ) is sent to the first feed-forward neural network layer.

[0063] 4.3.5 The first feedforward neural network layer receives G(Qa o ), using the linear mapping method (see the literature "Attention is all you need.", 2017, attention is all you need) to G(Qa o ) is linearly mapped to obtain G(Qa o ) after linear mapping o ), that is, f(Qa o )=W F ×G(Qa o ), change f(Qa o ) is sent to the first linear classification layer.

[0064] 4.3.6 The first linear classification layer receives f(Qa o ), change f(Qa o ) is linearly mapped to the first two-dimensional logits vector, and the first logits vector is subjected to softmax operation to obtain Qa o The probability of each category in the corresponding MM, taking the category with the highest probability as Qa o The predicted label Qa″ is added to the predicted label set Qa′.

[0065] 4.3.7 If o≤O and o=bz×batch_size, go to 4.3.8; if o>O, it means that all O samples of Qa have participated in the training and one iteration of training has been completed, go to 4.3.9; if o≤O and o≤bz×batch_size, let o=o+1, go to 4.3.4.

[0066] 4.3.8 The first linear classification layer uses the cross loss entropy function to calculate Qa′ and Qa## The difference between them is used to obtain the loss value Loss, and the Adam optimization algorithm (see the document "Adam: A Method for Stochastic Optimization", 2015, Adam, a stochastic optimization method) is used to minimize the Loss. The weight parameter W of the first search module (including the first feedforward neural network layer and the first linear classification layer) is updated once through back propagation. F and W Y Let o=o+1, bz=bz+1. If o>O, it means that all O samples of Qa have participated in the training and one iteration of training has been completed. Go to 4.3.9. Otherwise, go to 4.3.4.

[0067] 4.3.9 Let itretation = itretation + 1. If itretation ≤ iteration threshold E, go to 4.3.2. Otherwise, it means that the training meets the iteration number requirement, and the training ends. The weight parameter W of the first feedforward neural network layer in the first search module is obtained. F , the weight parameter W of the first linear classification layer Y , go to step 5.

[0068] The fifth step is to assign the trained weight parameters to the first search model to obtain the trained first search module, that is, the trained W F As the network weight parameter of the first feedforward neural network layer, the trained W Y As the weight parameter of the first linear classification layer, the trained first search module is obtained.

[0069] Step 6: Use the trained first search module to search in M ​​according to the user input requirement text q, and obtain the 20 models that best match q. The method is as follows:

[0070] 6.1 The preprocessing module receives the user input requirement text q and compares q with the model information sequence m1,…m in M. n ,…,m N Splice them separately to form sequences qm1,…qm n ,…,qm N , qm1,…qm n ,…,qm N Put it into the requirement text-model information sequence set Set (qm n ), that is, Set(qm n )={qm1,…qm n ,…,qm N}, qm n ={q,m n}, where 1≤n≤N, and N is the number of model information sequences in M.

[0071] 6.2 Let n=1 and create an empty list list.

[0072] 6.3 Take Set(qm n ) in the nth requirement text-model information sequence qm n , qm n Transformed into word-level sequence G(qm n ), G(qm n )={[CLS],q1,q2,...,q i ,...q I ,[SEP],mc1,mc2,...,mc j ,...,mc J ,[EOS]}(where q i represents the i-th word in q, 1≤i≤I, I is the number of words in q, mc j Indicates m n The jth word in, 1≤j≤J, J is m n The number of words in G(qm), [CLS] is a special word indicating the beginning of the sequence, [SEP] is a special word that separates the requirement text sequence and the model information sequence, and [EOS] is a special word that indicates the end of the sequence). n ) is sent to the first feed-forward neural network layer.

[0073] 6.4 The first feedforward neural network layer receives G(qm n ), for G(qm n ) is linearly mapped, and the linear mapping is expressed as f(qm n ), that is, f(qm)=W F ×G(qm n ), f(qm n ) is sent to the first linear classification layer.

[0074] 6.5 The first linear classification layer receives f(qm n ), f(qm n ) is mapped to a two-dimensional original prediction vector (i.e., the first logits vector), and a Softmax operation is performed on the first logits vector to obtain qm n The probability corresponding to each category. Among them, the predicted category is 1, which represents q and m n Correspondingly, the predicted category is 0 for q and m n No correspondence.

[0075] 6.6 Get qm n The probability value pqm of the predicted category being 1 n , and m nForm a probability information binary tuple (m n , pqm n ), and store it in the list list.

[0076] 6.7 If n < N, let n = n + 1, and go to 6.3; if n ≥ N, go to 6.8.

[0077] 6.8 Sort list in descending order according to the value of pqm n , take the first 20 binary tuples to get (m1, pqm1), …, (m n , pqm n ), …, (m 20 , pqm 20 ), obtain the image classification models corresponding to m1m n , …, m 20 , and output these 20 models to the second search module.

[0078] Step 7: The second search module receives 20 image classification models from the first search module, and also receives the second training set Pa, the second test set Pt, and the category table uploaded by the user. Score and sort the 20 image classification models as follows:

[0079] 7.1 The scoring layer of the second search module receives 20 models from the first search module and scores the 20 image classification models in turn as follows:

[0080] 7.1.1 Divide the image set P of the second training set Pa into multiple batches (batch) according to the batch size, and BZ is the total number of batches, which means rounding down 0.8KK ÷ batch_size. Among them, the images in one batch can be represented as the batch image set Pb = {Pb1, Pb2, …, Pb bb , …, Pb BB}, 1 ≤ bb ≤ BB, where BB is the number of images in Pb and BB is equal to batch_size, and Pb bb is the bb-th image in Pb.

[0081] 7.1.2 Let mcnt = 1, and let the unordered set Set(score) of model scores be empty.

[0082] 7.1.3 Let bz = 1.

[0083] 7.1.4 Take the images of the bz-th batch from P.

[0084] 7.1.5 Let the variable cnt = 1.

[0085] 7.1.6 Take the cntth image Pb from the bzth batch cnt , Pb cnt After passing through the second feedforward neural network layer of the mcnt-th image classification model, the linear mapping is obtained to represent g(Pb cnt ), namely g(Pb cnt )=ReLU(Pb cnt ), ReLU is the activation function in the second feedforward neural layer of the mcnt-th image classification model, ReLU(p)=max(0,p). The mapping of all activation functions in the second feedforward neural layer of the mcnt-th image classification model is Pb cnt Form a unique linear code u cnt , Among them, 1≤z≤N A , It is the picture Pb cnt The mapping at the zth activation unit, N A is the number of activation units in the second feedforward neural network layer in the mcnt-th image classification model.

[0086] 7.1.7 Let cnt = cnt + 1. If cnt ≤ BB, go to 7.1.6. Otherwise, it means that {u1, u2, …, u cnt ,…,u BB}, where 1≤cnt≤BB, go to 7.1.8.

[0087] 7.1.8 Using the Hamming distance d H (u a ,u b ) is used to measure the similarity of the linear codes between all the images in a batch, u a is the linear code of the ath sample, u b is the linear code of the b-th sample, 1≤a≤BB, 1≤b≤BB. Calculate the kernel matrix K H :

[0088] Calculate the score of the mcnt-th image classification model on P cnt , score cnt =log|K H |. Among them, |K H | indicates K H The value of the determinant. The model information sequence m of the mcnt-th image classification model cnt and m cnt The corresponding score cnt Composed of two tuples (m cnt ,score cnt), add it to the unordered set of model scores Set(score). Let bz=bz+1,mcnt=mcnt+1, if bz≤BZ and mcnt≤20, go to 7.1.4; if bz>BZ and mcnt≤20, go to 7.1.3; if mcnt>20, get the unordered set of model scores Set(score) containing 20 bigrams, go to 7.1.9.

[0089] 7.1.9 Send Set(score) to the sorting recommendation layer.

[0090] 7.2 The sorting recommendation layer receives Set(score) from the scoring layer, sorts Set(score) according to the scores corresponding to each model information sequence from high to low, and takes the model corresponding to the model information sequence with the highest score as the optimal image classification model, and sends it to the image classification module together with the second training set Pa, the second test set Pt, and the category table uploaded by the user.

[0091] In the eighth step, the training module in the image classification module receives the optimal image classification model and Pa, Pt, and the category table from the sorting recommendation layer, and uses Pa to train the optimal image classification model. The method is as follows:

[0092] 8.1 Initialize the weight parameters and set the parameter set W of the second feedforward neural network layer of the optimal image classification model FF Initialize the elements in to random numbers in [0,1]; Set the parameter set W of the second linear classification layer of the optimal image classification model YY The elements in are initialized to random numbers in [0,1].

[0093] 8.2 Set the network training hyperparameters, set the batch size (batch_size) to 64, and the network model learning rate (learning rate) to 0.001.

[0094] 8.3 Iteratively train the second feedforward neural network layer and the second linear classification layer of the optimal image classification model in a supervised manner. In each round of training, the predicted label set Pa′ of each batch of samples is obtained, and the difference between Pa′ and the actual label set Pa of each batch of samples is calculated. ## The distribution gap is obtained, the loss value is minimized and W is updated FF and W YY , until the number of iterations required is met, and the parameter set W of the second feedforward neural network layer is obtained. FF and the parameter set W of the second linear classification layer YY , the method is as follows:

[0095] 8.3.1 Initialize the training iteration parameter itretation = 1; initialize the second iteration threshold EE, EE is an integer in [1, 200]. Experiments show that when the training iteration is 100 (i.e. the second iteration threshold EE is 100), the Acc value (i.e. the precision rate, which indicates the proportion of the number of correctly classified images to the total number of images in the category) of image classification is the highest. Therefore, EE is preferably 100.

[0096] 8.3.2 Let bz=1.

[0097] 8.3.3 Let kk=1 and let the predicted label set Pa′ be empty.

[0098] 8.3.4 The second feedforward neural network layer of the optimal image classification model reads the kkth sample Pa from Pa kk , using the linear mapping method to kk Perform linear mapping and get Pa kk The representation after linear mapping f(Pa kk ), that is, f(Pa kk )=W FF ×Pa kk , f(Pa kk ) is sent to the second linear classification layer of the optimal image classification model.

[0099] 8.3.5 The second linear classification layer of the optimal image classification model receives f(Pa kk ), f(Pa kk ) is linearly mapped to the second logits vector of the number of categories in the category table, and the second logits vector contains the kkth sample Pa kk The unnormalized probability of each category. Perform a softmax operation on the second logits vector to obtain Pa kk The probability of each category in the corresponding category table, take the category with the highest probability as Pa kk The predicted label Pa″ is added to the predicted label set Pa′.

[0100] 8.3.6 If kk≤KK and kk=bz×batch_size, go to 8.3.7; if kk>KK, it means that all KK samples of Pa have participated in the training and one iteration of training has been completed, go to 8.3.8; if kk≤KK and kk≤bz×batch_size, let kk=kk+1, go to 8.3.4.

[0101] 8.3.7 The second linear classification layer uses the cross loss entropy function to calculate Pa′ and Pa ##The difference between them is used to obtain the loss value Loss. The SGD optimization algorithm (see the literature "A Stochastic Approximation Method", 1951, a stochastic approximation method) is used to minimize the Loss. The weight parameters of the optimal image classification model are updated through back propagation. Let kk=kk+1, bz=bz+1. If kk>KK, it means that the KK samples of Pa have participated in the training and one iteration of training has been completed. Go to 8.3.8. Otherwise, go to 8.3.4.

[0102] 8.3.8 Let itretation = itretation + 1. If itretation ≤ the second iteration threshold EE, go to 8.3.2. Otherwise, the training meets the iteration number requirement, and the training ends. The weight parameters of the second feedforward neural network layer and the second linear classification layer in the optimal image classification model are obtained, that is, the optimal W is obtained. FF and W YY , go to step 9.

[0103] In the ninth step, the training module will train the W FF As the network weight parameter of the second feedforward neural network layer of the optimal image classification model, the trained W YY As the weight parameter of the second linear classification layer of the optimal image classification model, the trained image classification model is obtained, and the trained optimal image classification model and Pa, Pt, and category table are sent to the classification module.

[0104] In the tenth step, the classification module receives the trained optimal image classification model and Pa, Pt, and category table from the training module, and receives the unlabeled image p uploaded by the user, classifies the image p, and obtains the image category of p. The method is as follows:

[0105] 10.1 The classification module receives the trained optimal image classification model from the training module;

[0106] 10.2 The second feedforward neural network layer of the trained optimal image classification model receives the unlabeled image p uploaded by the user. Linear mapping is performed on p to obtain the linear mapping representation f(p), that is, f(p) = W FF ×p, and sends f(p) to the second linear classification layer of the trained optimal image classification model.

[0107] 10.3 The second linear classification layer of the trained optimal image classification model receives f(p) from the second feedforward neural network layer, compares it with the category table uploaded by the user, obtains the number of categories, and linearly maps f(p) to the third logits vector of the category number dimension in the category table. The third logits vector contains the probability of each category of the unlabeled image p uploaded by the user without normalization. Perform a softmax operation on the third logits vector to obtain the probability of each category in the category table corresponding to p, take the category with the highest probability as the image category of p, and complete the classification of p.

[0108] The following technical effects can be achieved by using the present invention:

[0109] 1. In the model search module of the image classification system constructed in the second step of the present invention, the first search module preliminarily screens the models in the model library according to user needs, and the second search module finely sorts the preliminarily screened models according to the picture set uploaded by the user to obtain the optimal image classification model (first stage), and then trains the optimal image classification model to obtain a trained optimal image classification model with optimal weight parameters (second stage). These two stages ensure that the image classification model for classifying the unlabeled picture p input by the user is the most suitable for classifying p, which is more suitable for user needs than the existing method, is more matched with the user data set, and has higher classification accuracy.

[0110] 2. Since the present invention adopts a search model approach rather than a model construction approach to obtain an image classification model that is most suitable for classifying an unlabeled image p input by a user, the image classification speed of the present invention is faster than that of existing methods.

[0111] 3. The present invention uses a training module to train the optimal image classification model searched out, so that it better meets the needs of users and better matches the user data set. The complexity of the data set will affect the accuracy of image classification. Generally speaking, with the increase in the number of categories and the increase in the complexity of the image, the features that the model needs to learn will become more complex, and the accuracy of image classification will decrease. Experiments show that: on the MNIST data set with 60,000 pictures and 10 categories, the classification accuracy of the present invention reaches 99%; on the Cifar100 data set with 60,000 pictures and 100 categories, the classification accuracy of the present invention reaches 90%; on the ImageNet-1k data set with 1 million pictures and 1,000 categories, the classification accuracy of the present invention reaches 86%. When the quality of the pictures uploaded by the user is poor, such as a small number, uneven distribution, etc., the classification accuracy of the present invention can still reach more than 80%.

[0112] 4. Since the present invention searches for image classification models according to a given data set and selection requirements, and uses the searched image classification model to classify images, there is no need for users to use professional knowledge to build or select models by themselves, thereby reducing the professional requirements for users.

[0113] 5. The models in the image classification model library constructed by the present invention are all high-quality models with high classification accuracy on general data sets, and are more likely to perform better on user data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 It is a logical structure diagram of the image classification system constructed in the first step of the present invention;

[0115] Figure 2 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0116] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. The present invention is described below in conjunction with the accompanying drawings.

[0117] like Figure 2 As shown, the present invention comprises the following steps:

[0118] Step 1: Build an image classification model library M as follows:

[0119] 1.1 Select N image classification models from the Paddleclas library (taken from https: / / pypi.org / project / paddleclas / , 2.5.1) to form the image classification model library M.

[0120] 1.2 Obtain various information about the model in M, including the name of the model, the number of model parameters, the floating-point calculation amount of the model, and the source code of the model. The method is:

[0121] 1.2.1 Get the model name and executable code file of each model in the model library M from the Paddleclas library. The executable code file of the model includes model definition, training and testing parts.

[0122] 1.2.2 Based on the ImageNet-1k dataset (taken from https: / / www.image-net.org / challenges / LSVRC / 2012 / , 2012), train and test all models in the model library, and obtain the classification accuracy of the model on the ImageNet-1k dataset. At the same time, record the inference speed of each model, that is, the time required for the model to output the prediction result from the input image.

[0123] 1.2.3 Through the thop library (taken from https: / / pypi.org / project / thop / 0.0.31-1909021322 / , 0.0.31-1909021322) to obtain the parameter quantity and floating-point calculation quantity of each model in the model library. The parameter quantity indicates the number of trainable weights in the model, reflecting the complexity and storage requirements of the model. The floating-point calculation quantity indicates the number of floating-point operations required for the model to perform a forward propagation, reflecting the computational overhead and energy consumption of the model.

[0124] 1.2.4 Read the model code from the model's executable code file to obtain the model code Code. Reorder the executable code in Code to make it clearer. Specifically, divide the model's executable code into three layers: the first layer is the model definition code, including network structure-related code and global parameter initialization code; the second layer is the definition code of other global class names and function names; the third layer is the remaining code in the code sequence Code. Put the first layer of code at the front of Code, the second layer of code after the first layer of code, and the third layer of code at the end.

[0125] 1.2.5 The model name, classification accuracy, inference speed, parameter amount, floating-point calculation amount and model code of each model in M ​​are formed into a model information sequence. Let the model information sequence of any model m = {Name, Accuracy, Latency, Param, Flops, Code}, where Name is the name of the model, Accuracy is the classification accuracy of the model, Latency is the inference speed of the model, Param is the parameter amount of the model, Flops is the floating-point calculation amount of the model, and Code is the model code.

[0126] Step 2: Build an image classification system. The image classification system consists of a model search module, an image classification module, and an image classification model library.

[0127] The model search module consists of a preprocessing module, a first search module and a second search module, wherein the first search module consists of a first feedforward neural network layer and a first linear classification layer, and the second search module consists of a scoring layer and a ranking recommendation layer.

[0128] The image classification module consists of a training module and a classification module.

[0129] The image classification model library M stores models and model information sequences corresponding to the models, including model name, classification accuracy, inference speed, parameter quantity, floating-point calculation quantity and model code.

[0130] The preprocessing module is connected to the first search module and the image classification model library, receives user input (including the user-input demand text q or the first training set Qa), uses the model information sequence in the image classification model library M to splice respectively, and processes the user input into a word level sequence.

[0131] If the preprocessing module receives the requirement text q input by the user, it concatenates q with all the model information sequences in the image classification model library M to form a set Set(qM) of q's requirement text-model information sequence, Set(qM) = {qm1,…qm n ,…,qm N}, qm n = {q,m n}, where 1≤n≤N, N is the number of model information sequences in the image classification model library M, and m n is the nth model information sequence in M. Use sentencepiece word segmentation (taken from https: / / github.com / google / sentencepiece / , 2023-11-07) to convert the N demand text-model information sequences in Set(qM) into word-level sequences, and obtain the word-level sequence set G of Set(qM), and send G to the first feedforward neural network layer. G={G(qm1),…,G(qm n ),…,G(qm N )},qm n The word level sequence G(qm n )={[CLS],q1,q2,...,q i ,...q I ,[SEP],mc1,mc2,...,mc j ,...,mc J ,[EOS]}(where q i is the i-th word in q, 1≤i≤I, I is the number of words in q, mc j Indicates m n The jth word in, 1≤j≤J, J is m n [CLS] is a special word indicating the beginning of a sequence, [SEP] is a special word that separates the requirement text sequence and the model information sequence, and [EOS] is a special word that indicates the end of a sequence).

[0132] The first feedforward neural network layer is connected to the preprocessing module and the first linear classification layer. The first feedforward neural network layer receives G from the preprocessing module and processes G(qm1),…,G(qm n ),…,G(qm N ) to perform linear mapping, and the linear mapping is expressed as f(qm1),…,f(qmn ),…,f(qm N ), f(qm1),…,f(qm n ),…,f(qm N ) is sent to the first linear classification layer.

[0133] The first linear classification layer is connected to the first feedforward neural network layer and the second search module, and the first linear classification layer is connected to the first feedforward neural network layer and the second search module. n ),…,f(qm N ) to perform linear mapping, and obtain f(qm1),…,f(qm n ),…,f(qm N )’s original prediction vector (i.e., the first logits vector), performs Softmax operation on the first logits vector, obtains the probability of matching the model information sequence of the demand text q with each model in M, takes the top 20 models with the highest probability as the output of the first search module, and sends the 20 models to the second search module.

[0134] The scoring layer of the second search module is connected to the first linear classification layer and the sorting recommendation layer. The scoring layer receives 20 models from the first linear classification layer, and receives the second training set Pa, the second test set Pt, and the category table uploaded by the user. The scoring layer scores the 20 models output by the first search module according to the picture set P in the second training set Pa, and obtains an unordered score set Set (score) of the 20 models. Set (score) contains the scores of the 20 models, and sends Set (score), the second training set Pa, the second test set Pt, and the category table to the sorting recommendation layer.

[0135] The sorting recommendation layer is connected to the scoring layer and the image classification module. The scores in Set (score) received from the scoring layer are sorted in descending order, and the image classification model with the highest score is selected as the optimal image classification model (the optimal image classification model consists of the second feedforward neural network layer and the second linear classification layer). The optimal image classification model, the second training set Pa, the second test set Pt, and the category table are sent to the image classification module.

[0136] The training module in the image classification module is connected to the sorting recommendation layer and the classification module, and receives the optimal image classification model and the second training set Pa, the second test set Pt, and the category table from the sorting recommendation layer. Pa is input into the second feedforward neural network layer of the optimal image classification model, and the optimal image classification model is trained using Pa. The second feedforward neural network layer linearly maps the pictures in Pa one by one to obtain the representation ff(Pa) of Pa after linear mapping, and sends ff(Pa) to the second linear classification layer; the second linear classification layer linearly maps ff(Pa) to obtain the original prediction vector (second logits vector) of ff(Pa), and performs a Softmax operation on the second logits vector to obtain the probability of each category in the category table corresponding to the picture in Pa, and takes the category with the highest probability as the category of the picture. The training module uses the cross-loss entropy function to calculate the loss, and back-propagates the loss in the optimal image classification model to update the optimal image classification model parameters. The training stops when the loss no longer decreases. The trained optimal image classification model is sent to the classification module as the output of the training module.

[0137] The classification module is connected to the training module, and receives the trained optimal image classification model from the training module, and receives the unlabeled image p uploaded by the user. The size of p is adjusted to 64×64, and input into the trained optimal image classification model. The second feedforward neural network layer of the trained optimal image classification model performs linear mapping on p, and the linear mapping of p is represented as ff(p). ff(p) is sent to the second linear classification layer of the trained optimal image classification model. The second linear classification layer performs linear mapping on ff(p) to obtain a prediction vector (third logits vector) for p. The third logits vector is subjected to Softmax operation to obtain the probability of each category in the category table corresponding to p, and the category with the highest probability is taken as the category of p.

[0138] The third step is to build a dataset for training the model search module and the image classification module as follows:

[0139] 3.1 Prepare the data set for the first search module in the training model search module by:

[0140] 3.1.1 Describe the user requirement in natural language. The user requirement is "among the models with the top v% in inference speed, the model with the highest classification accuracy", where v represents the accuracy ratio of the user requirement, and its value range is 0<v≤100. Depending on the number of decimal places retained by v, there are a total of vt possible values, where vt is a positive integer, that is, v is a set consisting of vt possible values, v={v1,v2,…,v vvt ,…,v vt}; where vvt is a possible value of vt, 1≤vvt≤vt, and vvt is a positive integer; each possible value of vt corresponds to a user demand text. Let the user demand text set Q = {Q1, Q2, …, Q vvt ,…,Q vt}, Q vvt This is the user requirement text for Article vvt;

[0141] 3.1.2 Let vvt = 1;

[0142] 3.1.3 Take the vvtth user requirement text Q from Q vvt , find the one that matches Q from M vvt The model is as follows:

[0143] 3.1.3.1 Sort each model in M ​​from low to high according to the latency value in the model information sequence, and select the top v vvt % of the model, let it be the screening model set MM;

[0144] 3.1.3.2 Sort the models in MM from high to low according to the Accuracy values ​​in the model information sequence, and take the model with the highest Accuracy value as the one that meets the vvtth user requirement text Q vvt Model;

[0145] 3.1.4 Article vvt User Requirements Text Q vvt In line with Q vvt The model information sequence of the model of the model constitutes a demand-model pair, and the label of this demand-model pair is marked as 1; randomly select from MM that does not meet Q vvt The model of the model, and the model information sequence of the model form a demand-model pair, and the label of this demand-model pair is marked as 0; the demand-model pair and its corresponding actual label form a sample, and the form of each sample = {Q vvt ,m,label}, where Q vvt is the vvtth user requirement text, m is the model information sequence, label is the label of the requirement-model pair; put the sample into the data set;

[0146] 3.1.5 Let vvt = vvt + 1. If vvt ≤ vt, go to 3.1.3. Otherwise, go to 3.1.6.

[0147] 3.1.6 Construct a data set containing 2vt samples;

[0148] 3.1.7 Expand the data set to form a data set containing 6vt samples as follows:

[0149] 3.1.7.1 Let variable count = 1.

[0150] 3.1.7.2 Expand the data set by replacing synonyms. Replace the user demand text Q in the count-th sample in the data set count From "the model with the highest classification accuracy among the top v% of models in inference speed" to Q count ′, Q′ is “the model with the highest prediction accuracy among the models with the top v% in inference speed”, forming a new sample = {Q count ′,m,label} and added to the dataset.

[0151] 3.1.7.3 Expand the data set by changing the word position. Change the user demand text Q in the demand-model pair of the count-th sample in the data set to count From "the model with the highest classification accuracy among the top v% of models in inference speed" to Q count ″, Q count ″ is “the inference speed is in the top v% among all models, and the classification accuracy is the highest”, forming a new sample = {Q count ″,m,label}, is added to the dataset.

[0152] 3.1.7.4 Let count = count + 1. If count ≤ 2000, go to 3.1.7.2; otherwise, go to 3.1.8.

[0153] 3.1.8 Divide the dataset into training set and test set in a ratio of 8 to 2. The demand-model pairs of all samples in the training set constitute the first training set Qa, and the labels of all samples in the training set constitute the actual label set Qa # ;Let Qa={Q 1 ,Q 2 ,···Qa o ,···,Qa o}, where Qa o represents the oth user requirement text-model pair in Qa (where 1≤o≤O, O is the total number of samples in Qa, O=0.8×6vt); Qa o = {Q o ,m},Q o is the oth user requirement text, and m is the corresponding model information sequence. The requirement-model pairs of all samples in the test set constitute the first test set Qt, and the labels of all samples in the test set constitute the actual label set Qt of the first test set # , Qa is used to train the first search module, Qa # It is used to calculate the training loss and calibrate the model when training the first search module. Qt is used to test the first search module. #Used to compare with the labels of the test output to measure the performance of the first search module under test.

[0154] 3.2 Construct the dataset of the training module in the training image classification module. The dataset comes from the collection of labeled pictures uploaded by users. The method is as follows:

[0155] 3.2.1 The user searches for pictures of the corresponding category on the Internet according to the target task and annotates them. The annotation method is as follows:

[0156] 3.2.1.1 Determine which categories of images need to be classified, set a label for each category, and make a category table.

[0157] 3.2.1.2 Collect a certain number of images for each category from the Internet, with no less than 500 images for each category, and label these images with the corresponding labels.

[0158] 3.2.2 Process the KK labeled images, delete duplicate and blurred images, adjust the size of the KK labeled images to 64×64, perform data enhancement on the KK labeled images by rotating, cropping, scaling, flipping, adjusting brightness and contrast, and obtain KK labeled images after data enhancement. KK is the total number of labeled images, KK ≥ 500.

[0159] 3.2.3 Divide the enhanced KK labeled images into training and test sets in a ratio of 8 to 2, and obtain the second training set Pa = {Pa 1 ,Pa 2 ,…,Pa kk ,…,Pa 0.8KK}, where 1≤kk≤0.8KK, 0.8KK is the total number of labeled images in Pa. At the same time, the second test set Pt is obtained, Pt={Pt 1 ,Pt 2 ,…,Pt kkk ,…,Pt 0.2KK}, where 1≤kkk≤0.2KK, 0.2KK is the total number of labeled images in Pt. The actual labels in Pa constitute the actual label set Pa of the second training set # , the pictures in Pa constitute the picture set P in Pa, and the actual labels in the Pt samples constitute the actual label set Pt of the second test set # Pa is used to train the optimal image classification model in the training module in the image classification module, Pa # It is used to calculate the training loss when training the optimal image classification model and to correct the optimal image classification model; Pt is used to test the optimal image classification model. #Used to compare with the test output label to measure the performance of the optimal image classification model for testing.

[0160] The fourth step is to use Qa to perform supervised training on the first search module in the model search module to obtain different network weight parameters. The method is as follows:

[0161] 4.1 Initialize the weight parameters and set the parameter set W of the first feedforward neural network layer F Initialize the elements in to random numbers in [0,1]; set the parameter set W of the first linear classification layer Y The elements in are initialized to random numbers in [0,1].

[0162] 4.2 Set the network training hyperparameters, set the batch size (batch_size) to 32, the network model learning rate (learning rate) to 0.001, the maximum sequence length to 200, and the control gradient accumulation parameter (gradient_accumulation_steps) to 1.

[0163] 4.3 The first feedforward neural network layer and the first linear classification layer are trained in a supervised manner. In each round of training, the predicted label sequence set Qa′ of each batch of samples is obtained, and Pa′ and the actual label sequence set Qa of each batch of samples are calculated. ## The distribution gap is obtained, the loss value is minimized and the weight parameters of the first feedforward neural network layer and the first linear classification layer are updated until the number of iterations is met. The weight parameters of the first feedforward neural network layer and the first linear classification layer are obtained as follows:

[0164] 4.3.1 Initialize the training iteration parameter itretation = 1; initialize the iteration threshold E, where E is an integer in [1, 10]. Experiments have shown that when E is 8, the Acc value of the model search (i.e., the precision rate, which indicates the ratio of the number of correct searches to the number of all samples) is the highest.

[0165] 4.3.2 Let variable bz=1.

[0166] 4.3.3 Let o=1 and let the predicted label set Qa′ be empty.

[0167] 4.3.4 Preprocessing module reads the oth sample Qa from Qa o , will Qa o Processed as a word-level sequence G(Qa o ), G(Qa o )={[CLS],Q1,…,Q ii ,…,Q II ,[SEP],mc1,…,mcjj ,…mc JJ ,[EOS]}, where 1≤ii≤II,1≤jj≤JJ, [CLS] is a special word indicating the beginning of a sequence, Q ii Qa o User Requirements Text Q o The iith word in, II is Q o The total number of words in [SEP] is a special word that separates the user requirement text and the model information sequence, mc jj Qa o The model information sequence m o The jjth word of o The number of words in G(Qa o ) is sent to the first feed-forward neural network layer.

[0168] 4.3.5 The first feedforward neural network layer receives G(Qa o ), using the linear mapping method (see the literature "Attention is all you need.", 2017, attention is all you need) to G(Qa o ) is linearly mapped to obtain G(Qa o ) after linear mapping o ), that is, f(Qa o )=W F ×G(Qa o ), change f(Qa o ) is sent to the first linear classification layer.

[0169] 4.3.6 The first linear classification layer receives f(Qa o ), change f(Qa o ) is linearly mapped to the first two-dimensional logits vector, and the first logits vector is subjected to softmax operation to obtain Qa o The probability of each category in the corresponding MM, taking the category with the highest probability as Qa o The predicted label Qa″ is added to the predicted label set Qa′.

[0170] 4.3.7 If o≤O and o=bz×batch_size, go to 4.3.8; if o>O, it means that all O samples of Qa have participated in the training and one iteration of training has been completed, go to 4.3.9; if o≤O and o≤bz×batch_size, let o=o+1, go to 4.3.4.

[0171] 4.3.8 The first linear classification layer uses the cross loss entropy function to calculate Qa′ and Qa## The difference between them is used to obtain the loss value Loss, and the Adam optimization algorithm (see the document "Adam: A Method for Stochastic Optimization", 2015, Adam, a stochastic optimization method) is used to minimize the Loss. The weight parameter W of the first search module (including the first feedforward neural network layer and the first linear classification layer) is updated once through back propagation. F and W Y Let o=o+1, bz=bz+1. If o>O, it means that all O samples of Qa have participated in the training and one iteration of training has been completed. Go to 4.3.9. Otherwise, go to 4.3.4.

[0172] 4.3.9 Let itretation = itretation + 1. If itretation ≤ iteration threshold E, go to 4.3.2. Otherwise, it means that the training meets the iteration number requirement, and the training ends. The weight parameter W of the first feedforward neural network layer in the first search module is obtained. F , the weight parameter W of the first linear classification layer Y , go to step 5.

[0173] The fifth step is to assign the trained weight parameters to the first search model to obtain the trained first search module, that is, the trained W F As the network weight parameter of the first feedforward neural network layer, the trained W Y As the weight parameter of the first linear classification layer, the trained first search module is obtained.

[0174] Step 6: Use the trained first search module to search in M ​​according to the user input requirement text q, and obtain the 20 models that best match q. The method is as follows:

[0175] 6.1 The preprocessing module receives the user input requirement text q and compares q with the model information sequence m1,…m in M. n ,…,m N Splice them separately to form sequences qm1,…qm n ,…,qm N , qm1,…qm n ,…,qm N Put it into the requirement text-model information sequence set Set (qm n ), that is, Set(qm n )={qm1,…qm n ,…,qm N}, qm n = {q,m n}, where 1≤n≤N, and N is the number of model information sequences in M.

[0176] 6.2 Let n=1 and create an empty list list.

[0177] 6.3 Take Set(qm n ) in the nth requirement text-model information sequence qm n , qm n Transformed into word-level sequence G(qm n ), G(qm n )={[CLS],q1,q2,...,q i ,...q I ,[SEP],mc1,mc2,...,mc j ,...,mc J ,[EOS]}(where q i represents the i-th word in q, 1≤i≤I, I is the number of words in q, mc j Indicates m n The jth word in, 1≤j≤J, J is m n The number of words in G(qm), [CLS] is a special word indicating the beginning of the sequence, [SEP] is a special word that separates the requirement text sequence and the model information sequence, and [EOS] is a special word that indicates the end of the sequence). n ) is sent to the first feed-forward neural network layer.

[0178] 6.4 The first feedforward neural network layer receives G(qm n ), for G(qm n ) is linearly mapped, and the linear mapping is expressed as f(qm n ), that is, f(qm)=W F ×G(qm n ), f(qm n ) is sent to the first linear classification layer.

[0179] 6.5 The first linear classification layer receives f(qm n ), f(qm n ) is mapped to a two-dimensional original prediction vector (i.e., the first logits vector), and a Softmax operation is performed on the first logits vector to obtain qm n The probability corresponding to each category. Among them, the predicted category is 1, which represents q and m n Correspondingly, the predicted category is 0 for q and m n No correspondence.

[0180] 6.6 Get qm n The probability value pqm of the predicted category being 1 n , and m nForm a probability information binary tuple (m n , pqm n ), and store it in the list list.

[0181] 6.7 If n < N, set n = n + 1 and go to 6.3; if n ≥ N, go to 6.8.

[0182] 6.8 Sort list in descending order according to the value of pqm n , take the first 20 binary tuples to get (m1, pqm1), …, (m n , pqm n ), …, (m 20 , pqm 20 ), obtain the image classification models corresponding to m1m n , …, m 20 , and output these 20 models to the second search module.

[0183] Step 7: The second search module receives 20 image classification models from the first search module, and also receives the second training set Pa, the second test set Pt, and the category table uploaded by the user. Score and sort the 20 image classification models as follows:

[0184] 7.1 The scoring layer of the second search module receives 20 models from the first search module and scores the 20 image classification models in turn as follows:

[0185] 7.1.1 Divide the image set P of the second training set Pa into multiple batches according to the batch size, and BZ is the total number of batches, which represents rounding down 0.8KK÷batch_size. Among them, the images in one batch can be represented as the batch image set Pb = {Pb1, Pb2, …, Pb bb , …, Pb BB}, where 1 ≤ bb ≤ BB, and BB is the number of images in Pb, BB is equal to batch_size, and Pb bb is the bb-th image in Pb.

[0186] 7.1.2 Set mcnt = 1 and set the unordered set of model scores Set(score) to be empty.

[0187] 7.1.3 Set bz = 1.

[0188] 7.1.4 Take the images of the bz-th batch from P.

[0189] 7.1.5 Set the variable cnt = 1.

[0190] 7.1.6 Take the cntth image Pb from the bzth batch cnt , Pb cnt After passing through the second feedforward neural network layer of the mcnt-th image classification model, the linear mapping is obtained to represent g(Pb cnt ), namely g(Pb cnt )=ReLU(Pb cnt ), ReLU is the activation function in the second feedforward neural layer of the mcnt-th image classification model, ReLU(p)=max(0,p). The mapping of all activation functions in the second feedforward neural layer of the mcnt-th image classification model is Pb cnt Form a unique linear code u cnt , Among them, 1≤z≤N A , It is the picture Pb cnt The mapping at the zth activation unit, N A is the number of activation units in the second feedforward neural network layer in the mcnt-th image classification model.

[0191] 7.1.7 Let cnt = cnt + 1. If cnt ≤ BB, go to 7.1.6. Otherwise, it means that {u1, u2, …, u cnt ,…,u BB}, where 1≤cnt≤BB, go to 7.1.8.

[0192] 7.1.8 Using the Hamming distance d H (u a ,u b ) is used to measure the similarity of the linear codes between all the images in a batch, u a is the linear code of the ath sample, u b is the linear code of the b-th sample, 1≤a≤BB, 1≤b≤BB. Calculate the kernel matrix K H :

[0193] Calculate the score of the mcnt-th image classification model on P cnt , score cnt =log|K H |. Among them, |K H | indicates K H The value of the determinant. The model information sequence m of the mcnt-th image classification model cnt and m cnt The corresponding score cnt Composed of two tuples (m cnt ,score cnt), add it to the unordered set of model scores Set(score). Let bz=bz+1,mcnt=mcnt+1, if bz≤BZ and mcnt≤20, go to 7.1.4; if bz>BZ and mcnt≤20, go to 7.1.3; if mcnt>20, get the unordered set of model scores Set(score) containing 20 bigrams, go to 7.1.9.

[0194] 7.1.9 Send Set(score) to the sorting recommendation layer.

[0195] 7.2 The sorting recommendation layer receives Set(score) from the scoring layer, sorts Set(score) according to the scores corresponding to each model information sequence from high to low, and takes the model corresponding to the model information sequence with the highest score as the optimal image classification model, and sends it to the image classification module together with the second training set Pa, the second test set Pt, and the category table uploaded by the user.

[0196] In the eighth step, the training module in the image classification module receives the optimal image classification model and Pa, Pt, and the category table from the sorting recommendation layer, and uses Pa to train the optimal image classification model. The method is as follows:

[0197] 8.1 Initialize the weight parameters and set the parameter set W of the second feedforward neural network layer of the optimal image classification model FF Initialize the elements in to random numbers in [0,1]; Set the parameter set W of the second linear classification layer of the optimal image classification model YY The elements in are initialized to random numbers in [0,1].

[0198] 8.2 Set the network training hyperparameters, set the batch size (batch_size) to 64, and the network model learning rate (learning rate) to 0.001.

[0199] 8.3 Iteratively train the second feedforward neural network layer and the second linear classification layer of the optimal image classification model in a supervised manner. In each round of training, the predicted label set Pa′ of each batch of samples is obtained, and the difference between Pa′ and the actual label set Pa of each batch of samples is calculated. ## The distribution gap is obtained, the loss value is minimized and W is updated FF and W YY , until the number of iterations required is met, and the parameter set W of the second feedforward neural network layer is obtained. FF and the parameter set W of the second linear classification layer YY , the method is as follows:

[0200] 8.3.1 Initialize the training iteration parameter itretation = 1; initialize the second iteration threshold EE, EE is an integer in [1, 200]. Experiments show that when the training iteration is 100 (i.e. the second iteration threshold EE is 100), the Acc value (i.e. the precision rate, which indicates the proportion of the number of correctly classified images to the total number of images in the category) of image classification is the highest. Therefore, EE is preferably 100.

[0201] 8.3.2 Let bz=1.

[0202] 8.3.3 Let kk=1 and let the predicted label set Pa′ be empty.

[0203] 8.3.4 The second feedforward neural network layer of the optimal image classification model reads the kkth sample Pa from Pa kk , using the linear mapping method to kk Perform linear mapping and get Pa kk The representation after linear mapping f(Pa kk ), that is, f(Pa kk )=W FF ×Pa kk , f(Pa kk ) is sent to the second linear classification layer of the optimal image classification model.

[0204] 8.3.5 The second linear classification layer of the optimal image classification model receives f(Pa kk ), f(Pa kk ) is linearly mapped to the second logits vector of the number of categories in the category table, and the second logits vector contains the kkth sample Pa kk The unnormalized probability of each category. Perform a softmax operation on the second logits vector to obtain Pa kk The probability of each category in the corresponding category table, take the category with the highest probability as Pa kk The predicted label Pa″ is added to the predicted label set Pa′.

[0205] 8.3.6 If kk≤KK and kk=bz×batch_size, go to 8.3.7; if kk>KK, it means that all KK samples of Pa have participated in the training and one iteration of training has been completed, go to 8.3.8; if kk≤KK and kk≤bz×batch_size, let kk=kk+1, go to 8.3.4.

[0206] 8.3.7 The second linear classification layer uses the cross loss entropy function to calculate Pa′ and Pa ##The difference between them is used to obtain the loss value Loss. The SGD optimization algorithm (see the literature "A Stochastic Approximation Method", 1951, a stochastic approximation method) is used to minimize the Loss. The weight parameters of the optimal image classification model are updated through back propagation. Let kk=kk+1, bz=bz+1. If kk>KK, it means that the KK samples of Pa have participated in the training and one iteration of training has been completed. Go to 8.3.8. Otherwise, go to 8.3.4.

[0207] 8.3.8 Let itretation = itretation + 1. If itretation ≤ the second iteration threshold EE, go to 8.3.2. Otherwise, the training meets the iteration number requirement, and the training ends. The weight parameters of the second feedforward neural network layer and the second linear classification layer in the optimal image classification model are obtained, that is, the optimal W is obtained. FF and W YY , go to step 9.

[0208] In the ninth step, the training module will train the W FF As the network weight parameter of the second feedforward neural network layer of the optimal image classification model, the trained W YY As the weight parameter of the second linear classification layer of the optimal image classification model, the trained image classification model is obtained, and the trained optimal image classification model and Pa, Pt, and category table are sent to the classification module.

[0209] In the tenth step, the classification module receives the trained optimal image classification model and Pa, Pt, and category table from the training module, and receives the unlabeled image p uploaded by the user, classifies the image p, and obtains the image category of p. The method is as follows:

[0210] 10.1 The classification module receives the trained optimal image classification model from the training module;

[0211] 10.2 The second feedforward neural network layer of the trained optimal image classification model receives the unlabeled image p uploaded by the user. Linear mapping is performed on p to obtain the linear mapping representation f(p), that is, f(p) = W FF ×p, and sends f(p) to the second linear classification layer of the trained optimal image classification model.

[0212] 10.3 The second linear classification layer of the trained optimal image classification model receives f(p) from the second feedforward neural network layer, compares it with the category table uploaded by the user, obtains the number of categories, and linearly maps f(p) to the third logits vector of the category number dimension in the category table. The third logits vector contains the probability of each category of the unlabeled image p uploaded by the user without normalization. Perform a softmax operation on the third logits vector to obtain the probability of each category in the category table corresponding to p, take the category with the highest probability as the image category of p, and complete the classification of p.

[0213] To verify the effect of the present invention, the following experiment was performed, and the experimental process was:

[0214] The Cifar100 dataset (https: / / www.cs.toronto.edu / ~kriz / cifar-100-python.tar.gz, 2023-11-07) and ImageNet-1k dataset (https: / / www.image-net.org / challenges / LSVRC / 2012 / , 2012) are selected to simulate the second training set Pa and the second test set Pt. 1000 images are randomly selected as the unlabeled images p uploaded by users and deleted from the dataset. Then the dataset is processed according to the method in step 3.2. The v in the user requirement text is an integer, which is equal to all integers from 1 to 100. The value of v may be 100, that is, vt = 100; the number of models N in M ​​is 85; the iteration threshold E is 8; the second iteration threshold EE is 100. For 100 possible values ​​of v, 100 experiments were conducted, and the calculation method of the classification accuracy (Acc value) in the experimental results was: the sum of the number of correctly classified pictures in each experiment divided by the total number of pictures in 100 experiments, that is, the number of correctly classified pictures in 100 experiments / 100000*100%. The performance of the present invention is evaluated by the above examples. The software system environment of the experiment is Ubuntu22.04 version (i.e., a version of the Linux system), equipped with Intel Xeon Gold series 6154 central processing unit, with a processing frequency of 3.0Hz, and four NVIDIA Tesla V100 graphics cards, each with 5120 CUDA cores and a video memory capacity of 32GB. Under this experimental environment, the present invention is used to identify the image categories in the two data sets. The test process is consistent with the method for identifying the user-entered demand text and the uploaded pictures, and finally the results of the identification are statistically evaluated.

[0215] Table 1 compares the evaluation index comparison results of the present invention and the existing image classification methods based on model search (REA, GDAS, NASWOT) described in the background technology on the two data sets Cifar100 and ImageNet-1k. REA, GDAS, and NASWOT do not include the two-stage model search of the model search module designed by the present invention. The classification accuracy (Acc value) of image classification and the time of model search are used as the final evaluation indicators. The Cifar100 data set has 60,000 pictures and 100 categories, while ImageNet-1k has 1 million pictures and 1,000 categories. The difficulty of the model learning the features of the Cifar100 data set is less than that of learning the features of the ImageNet-1k data set, so overall, the classification accuracy of REA, GDAS, NASWOT, and the present invention on the Cifar100 data set is better than that of the ImageNet-1k data set. It can be seen from Table 1 that on the two datasets Cifar100 and ImageNet-1k, the image classification using the present invention significantly improves the classification accuracy (Acc value) on both datasets compared to the three image classification algorithms based on model search proposed by the background technology in the first column of Table 1. In Table 1, the search time of each method is listed in one column, and the method proposed by the present invention is significantly better than the REA method, the GDAS method and the NASWOT method. This experiment shows that the present invention is superior to the existing image classification methods based on model search in terms of classification accuracy and classification speed.

[0216]

[0217] Table 1 The existing image classification methods based on model search described in the present invention and the background art (REA, GDAS,

[0218] Comparison results of evaluation indicators of NASWOT) on two datasets

[0219] The above is a detailed introduction to the two-stage image classification method based on model search provided by the present invention. This article explains the principles and implementation methods of the present invention, and the above description is used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary researchers in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A two-stage image classification method based on model search, characterized by comprising the following steps: Step 1: Build an image classification model library M by: 1.1 Select N image classification models from the Paddleclas library to form an image classification model library M; 1.2 Obtain various information about the model in M, including the name of the model, the number of model parameters, the floating-point calculation amount of the model, and the source code of the model. The method is: 1.2.1 Get the model name and executable code file of each model in M ​​from the Paddleclas library. The executable code file of the model includes model definition, training and testing parts; 1.2.2 Based on the ImageNet-1k dataset, train and test all models in M ​​to obtain the classification accuracy of the models on the ImageNet-1k dataset; at the same time, record the inference speed of each model, that is, the time required for the model to output the prediction result from the input image; 1.2.3 Obtain the parameter quantity and floating-point calculation quantity of each model in M ​​through the thop library; the parameter quantity represents the number of trainable weights in the model, reflecting the complexity and storage requirements of the model; the floating-point calculation quantity represents the number of floating-point operations required for the model to perform a forward propagation, reflecting the computational overhead and energy consumption of the model; 1.2.4 Read the source code of the model from the executable code file of the model, name the source code of the model as model code Code; sort the executable code in Code; 1.2.5 The model name, classification accuracy, inference speed, parameter quantity, floating-point calculation amount and model code of each model in M ​​are formed into a model information sequence, and the model information sequence of any model is m = {Name, Accuracy, Latency, Param, Flops, Code}, where Name is the name of the model, Accuracy is the classification accuracy of the model, Latency is the inference speed of the model, Param is the number of model parameters, and Flops is the floating-point calculation amount of the model; Step 2: Build an image classification system; the image classification system consists of a model search module, an image classification module, and an image classification model library; The model search module consists of a preprocessing module, a first search module and a second search module, wherein the first search module consists of a first feedforward neural network layer and a first linear classification layer, and the second search module consists of a scoring layer and a ranking recommendation layer; The image classification module consists of a training module and a classification module; The image classification model library M stores models and model information sequences corresponding to the models, including model name, classification accuracy, inference speed, parameter quantity, floating point calculation quantity and model code; The preprocessing module is connected to the first search module and the image classification model library, receives the user's input, and uses the model information sequence in the image classification model library M to splice respectively, and processes the user's input into a word level sequence; the user's input includes the user's input demand text q or the first training set Qa; If the preprocessing module receives the requirement text q input by the user, it concatenates q with all the model information sequences in the image classification model library M to form a set Set(qM) of requirement text-model information sequences of q, where Set(qM) = {qm1, ...qm n , ..., qm N }, qm n = {q, m n }, where 1≤n≤N, N is the number of model information sequences in the image classification model library M, and m n is the nth model information sequence in M; the N demand text-model information sequences in Set(qM) are converted into word-level sequences using the sentencepiece segmentation method to obtain the word-level sequence set G of Set(qM), and G is sent to the first feedforward neural network layer; G = {G(qm1), ..., G(qm n ), ..., G(qm N )},qm n The word level sequence G(qm n )={[CLS],q1,q2,...,q i ,...q I ,[SEP],mc1,mc2,...,mc j ,...,mc J ,[EOS]}, where q i is the i-th word in q, 1≤i≤I, I is the number of words in q, mc j Indicates m n The jth word in, 1≤j≤J, J is m n The number of words in [CLS], [SEP] is a special word that separates the requirement text sequence and the model information sequence, and [EOS] is a special word that indicates the end of the sequence; The first feedforward neural network layer is connected to the preprocessing module and the first linear classification layer; the first feedforward neural network layer receives G from the preprocessing module, and processes G(qm1), ..., G(qm n ), ..., G(qm N ) is linearly mapped, and the linear mapping is expressed as f(qm1), ..., f(qm n ), ..., f(qm N ), f(qm1), ..., f(qm n ), ..., f(qm N ) is sent to the first linear classification layer; The first linear classification layer is connected to the first feedforward neural network layer and the second search module, and receives f(qm1), ..., f(qm n ), ..., f(qm N ) to perform linear mapping, and obtain f(qm1), ..., f(qm n ), ..., f(qm N ) is the original prediction vector, i.e., the first logits vector, and a Softmax operation is performed on the first logits vector to obtain the probability of matching the model information sequence of the demand text q with each model in M, and the top 20 models with the highest probability are taken as the output of the first search module, and the 20 models are sent to the second search module; The scoring layer of the second search module is connected to the first linear classification layer and the ranking recommendation layer; the scoring layer receives 20 models from the first linear classification layer, and receives the second training set Pa, the second test set Pt, and the category table uploaded by the user; the scoring layer scores the 20 models output by the first search module according to the picture set P in the second training set Pa, and obtains an unordered score set Set (score) of the 20 models, which contains the scores of the 20 models, and sends Set (score), the second training set Pa, the second test set Pt, and the category table to the ranking recommendation layer; The sorting recommendation layer is connected to the scoring layer and the image classification module, sorts the scores in Set (score) received from the scoring layer in descending order, selects the image classification model with the highest score as the optimal image classification model, and sends the optimal image classification model, the second training set Pa, the second test set Pt, and the category table to the image classification module; the optimal image classification model consists of the second feedforward neural network layer and the second linear classification layer; The training module in the image classification module is connected to the sorting recommendation layer and the classification module, and receives the optimal image classification model and the second training set Pa, the second test set Pt, and the category table from the sorting recommendation layer; Pa is input into the second feedforward neural network layer of the optimal image classification model, and the optimal image classification model is trained using Pa; the second feedforward neural network layer linearly maps the pictures in Pa one by one to obtain the representation ff(Pa) of Pa after linear mapping, and sends ff(Pa) to the second linear classification layer; the second linear classification layer linearly maps ff(Pa) to obtain the original prediction vector of ff(Pa), i.e., the second logits vector, and performs a Softmax operation on the second logits vector to obtain the probability of each category in the category table corresponding to the picture in Pa, and takes the category with the highest probability as the category of the picture; the training module calculates the loss using the cross-loss entropy function, and back-propagates the loss in the optimal image classification model to update the parameters of the optimal image classification model; the training is stopped until the loss no longer decreases; the trained optimal image classification model is sent to the classification module as the output of the training module; The classification module is connected to the training module, receives the trained optimal image classification model from the training module, and receives the unlabeled image p uploaded by the user; p is input into the trained optimal image classification model, the second feedforward neural network layer of the trained optimal image classification model performs linear mapping on p, and the linear mapping of p is represented as ff(p), and ff(p) is sent to the second linear classification layer of the trained optimal image classification model, the second linear classification layer performs linear mapping on ff(p), and obtains a prediction vector of p, namely the third logits vector, and performs Softmax operation on the third logits vector to obtain the probability of each category in the category table corresponding to p, and the category with the highest probability is taken as the category of p; Step 3: Construct the datasets for the training model search module and the image classification module as follows: 3.1 Prepare the dataset for the first search module in the training model search module as follows: 3.1.1 Describe the user requirement in natural language. The user requirement is "among the models with the highest classification accuracy in the top v% of the inference speed", where v represents the inference speed threshold required by the user, and the value range is 0<v≤100. Depending on the number of decimal places retained by v, there are a total of vt possible values, where vt is a positive integer, that is, v is a set consisting of vt possible values, v={v1,v2,...,v vvt , ..., v vt }; where vvt is a possible value of vt, 1≤vvt≤vt, and vvt is a positive integer; each possible value of vt corresponds to a user demand text, and the user demand text set Q = {Q1, Q2, ..., Q vvt , ..., Q vt }, Q vvt This is the user requirement text for Article vvt; 3.1.2 Let vvt = 1; 3.1.3 Take the vvtth user requirement text Q from Q vvt , find the one that matches Q from M vvt The model is as follows: 3.1.3.1 Sort each model in M ​​from low to high according to the latency value in the model information sequence, and select the top v vvt % of the model, let it be the screening model set MM; 3.1.3.2 Sort the models in MM from high to low according to the Accuracy value in the model information sequence, and take the model with the highest Accuracy value as the one that meets the vvtth user requirement text Q vvt Model; 3.1.4 Article vvt User Requirements Text Q vvt In line with Q vvt The model information sequence of the model of the model constitutes a demand-model pair, and the label of this demand-model pair is marked as 1; randomly select from MM that does not meet Q vvt The model of the model, and the model information sequence of the model form a demand-model pair, and the label of this demand-model pair is marked as 0; the demand-model pair and its corresponding actual label form a sample, and the form of each sample = {Q vvt ,m,label}, where Q vvt is the vvtth user requirement text, m is the model information sequence, label is the label of the requirement-model pair; put the sample into the data set; 3.1.5 Let vvt = vvt + 1. If vvt ≤ vt, go to 3.1.3; otherwise, go to 3.1.6; 3.1.6 Construct a dataset containing 2vt samples; 3.1.7 Augment the dataset to form a dataset containing 6vt samples; 3.1.8 Divide the dataset into training set and test set in a ratio of 8 to 2; the demand-model pairs of all samples in the training set constitute the first training set Qa, and the labels of all samples in the training set constitute the actual label set Qa # ; Let Qa={Qa 1 , Qa 2 ,···Qa o ,···,Qa o }, where Qa o represents the oth user requirement text-model pair in Qa, where 1≤o≤O, O is the total number of samples in Qa, O=0.8×6vt; Qa o = {Q o ,m},Q o is the oth user requirement text, m is the corresponding model information sequence; the requirement-model pairs of all samples in the test set constitute the first test set Qt, and the labels of all samples in the test set constitute the actual label set Qt of the first test set # , Qa is used to train the first search module, Qa # It is used to calculate the training loss and calibrate the model when training the first search module. Qt is used to test the first search module. # Used to compare with the labels output by the test to measure the performance of the first search module under test; 3.2 Construct the dataset for the training module in the training image classification module. This dataset is sourced from the labeled image collection uploaded by the user as follows: 3.2.1 The user searches for pictures of the corresponding category on the Internet according to the target task and makes annotations; 3.2.2 Process KK1 labeled pictures, delete duplicate and blurred pictures to obtain KK non-duplicate labeled pictures. Apply data augmentation to the KK labeled pictures through rotation, cropping, scaling, flipping, adjusting brightness and contrast to obtain KK data-augmented labeled pictures; KK is the total number of labeled pictures; KK ≤ KK1; 3.2.3 Divide the enhanced KK labeled images into training and test sets in a ratio of 8 to 2, and obtain the second training set Pa = {Pa 1 ,Pa 2 , ..., Pa kk , ..., Pa 0.8KK }, where 1≤kk≤0.8KK, 0.8KK is the total number of labeled images in Pa; and the second test set Pt is obtained, Pt={Pt 1 , Pt 2 , ..., Pt kkk , ..., Pt 0.2KK }, where 1≤kkk≤0.2KK, 0.2KK is the total number of labeled images in Pt; the actual labels in Pa constitute the actual label set Pa of the second training set # , the pictures in Pa constitute the picture set P in Pa, and the actual labels in the Pt samples constitute the actual label set Pt of the second test set # ; Pa is used to train the optimal image classification model in the training module in the image classification module, Pa # It is used to calculate the training loss when training the optimal image classification model and to correct the optimal image classification model; Pt is used to test the optimal image classification model. # Used to compare with the test output label to measure the performance of the optimal image classification model for testing; Step 4: Train the first search module in the model search module in a supervised manner using Qa to obtain different network weight parameters as follows: 4.1 Initialize the weight parameters and set the parameter set W of the first feedforward neural network layer F Initialize the elements in to random numbers in [0,1]; set the parameter set W of the first linear classification layer Y The elements in are initialized to random numbers in [0,1]; 4.2 Set the network training hyperparameters, including the batch size batch_size, the network model learning rate learningrate, the maximum sequence length, and the gradient accumulation parameter gradient_accumulation_steps; 4.3 The first feedforward neural network layer and the first linear classification layer are trained in a supervised manner. In each round of training, the predicted label sequence set Qa′ of each batch of samples is obtained, and the difference between Qa′ and the actual label sequence set Qa of each batch of samples is calculated. ## The distribution gap is obtained, the loss value is minimized and the weight parameters of the first feedforward neural network layer and the first linear classification layer are updated until the number of iterations is met. The weight parameters of the first feedforward neural network layer and the first linear classification layer are obtained as follows: 4.3.1 Initialize the training iteration parameter itretation = 1; Initialize the iteration threshold E, where E is an integer within [1, 10]; 4.3.2 Let the variable bz = 1; 4.3.3 Let o = 1, and let the predicted label set Qa' be empty; 4.3.4 Preprocessing module reads the oth sample Qa from Qa o , will Qa o Processed as a word-level sequence G(Qa o ), G(Qa o ) = {[CLS], Q1, ..., Q ii , ..., Q II , [SEP], mc1, ..., mc jj ,…mc JJ , [EOS]}, where 1≤ii≤II, 1≤jj≤JJ, Q ii Qa o User Requirements Text Q o The iith word in, II is Q o The total number of words in mc jj Qa o The model information sequence m o The jjth word of o The number of words in G(Qa o ) is sent to the first feed-forward neural network layer; 4.3.5 The first feedforward neural network layer receives G(Qa o ), using the linear mapping method to G(Qa o ) is linearly mapped to obtain G(Qa o ) after linear mapping o ), that is, f(Qa o )=W F ×G(Qa o ), change f(Qa o ) is sent to the first linear classification layer; 4.3.6 The first linear classification layer receives f(Qa o ), change f(Qa o ) is linearly mapped to the first two-dimensional logits vector, and the first logits vector is subjected to softmax operation to obtain Qa o The probability of each category in the corresponding MM, taking the category with the highest probability as Qa o The predicted label Qa″ is added to the predicted label set Qa′; 4.3.7 If o ≤ O and o = bz × batch_size, go to 4.3.8; If o > O, it means that all O samples of Qa have participated in training and one iteration of training is completed, go to 4.3.9; If o ≤ O and o < bz × batch_size, let o = o + 1 and go to 4.3.4; 4.3.8 The first linear classification layer uses the cross loss entropy function to calculate Qa′ and Qa # The difference between them is used to obtain the loss value Loss, and the Adam optimization algorithm is used to minimize the Loss. The weight parameter W of the first feedforward neural network layer is updated by back propagation. F and the weight parameter W of the first linear classification layer Y ; Let o = o + 1, bz = bz + 1. If o>O, it means that all O samples of Qa have participated in the training and one iteration of training has been completed. Go to 4.3.

9. Otherwise, go to 4.3.

4. 4.3.9 Let itretation = itretation + 1. If itretation ≤ iteration threshold E, go to 4.3.

2. Otherwise, it means that the training meets the iteration number requirement, and the training ends. The weight parameter W of the first feedforward neural network layer in the first search module is obtained. F , the weight parameter W of the first linear classification layer Y , go to step 5; The fifth step is to assign the trained weight parameters to the first search module, that is, to assign the trained W F As the network weight parameter of the first feedforward neural network layer, the trained W Y As the weight parameter of the first linear classification layer, the trained first search module is obtained; Step 6: Use the trained first search module to search in M according to the input requirement text q of the user to obtain the 20 models that best match q as follows: 6.1 The preprocessing module receives the user input requirement text q and compares q with the model information sequence m1, ...m in M. n , ..., m N Separately splice to form sequences qm1,...qm n , ..., qm N , qm1,…qm n , ..., qm N Put it into the requirement text-model information sequence set Set (qm n ), that is, Set(qm n )={qm1,...qm n , ..., qm N }, qm n = {q, m n }, where 1≤n≤N, N is the number of model information sequences in M; 6.2 Let n = 1 and create an empty list list; 6.3 Take Set(qm n ) in the nth requirement text-model information sequence qm n , qm n Transformed into word-level sequence G(qm n ), G(qm n )={[CLS],q1,q2,...,q i ,...q I ,[SEP],mc1,mc2,...,mc j ,...,mc J ,[EOS]}, where q i represents the i-th word in q, 1≤i≤I, I is the number of words in q, mc j Indicates m n The jth word in, 1≤j≤J, J is m n The number of words in G(qm n ) is sent to the first feed-forward neural network layer; 6.4 The first feedforward neural network layer receives G(qm n ), for G(qm n ) is linearly mapped, and the linear mapping is expressed as f(qm n ), that is, f(qm)=W F ×G(qm n ), f(qm n ) is sent to the first linear classification layer; 6.5 The first linear classification layer receives f(qm n ), f(qm n ) is mapped to the first logits vector of two dimensions, and the Softmax operation is performed on the first logits vector to obtain qm n The probability corresponding to each category; where the predicted category is 1, it represents q and m n Correspondingly, the predicted category is 0 for q and m n No correspondence; 6.6 Get qm n The probability value pqm of the predicted category being 1 n , and m n Composition probability information tuple (m n ,pqm n ), stored in the list list; 6.7 If n < N, let n = n + 1 and go to 6.3; If n ≥ N, go to 6.8; 6.8 List according to pqm n Sort the values ​​from high to low, take the first 20 tuples, and get (m1,pqm1),…,(m n ,pqm n ),…,(m 20 ,pqm 20 ), get m1m n , ..., m 20 The corresponding image classification model outputs these 20 models to the second search module; Step 7: The second search module receives 20 image classification models from the first search module and receives the second training set Pa, the second test set Pt, and the category table uploaded by the user. Score and rank the 20 image classification models as follows: 7.1 The scoring layer of the second search module receives 20 models from the first search module, scores the 20 image classification models in turn, obtains an unordered set of model scores Set(score), and sends Set(score) to the sorting recommendation layer; Set(score) contains 20 bigrams, the mcnt bigram is (m cnt , score cnt ), m cnt is the model information sequence of the mcnt-th image classification model, score cnt is the score of the mcnt-th image classification model on the picture set P of the second training set Pa; 1≤mcnt≤20; 7.2 The sorting and recommendation layer receives Set(score) from the scoring layer, sorts Set(score) in descending order according to the scores corresponding to each model information sequence, and sends the model corresponding to the model information sequence with the highest score as the optimal image classification model, along with the second training set Pa, the second test set Pt, and the category table uploaded by the user, to the image classification module; The eighth step: The training module in the image classification module receives the optimal image classification model, Pa, Pt, and the category table from the sorting and recommendation layer, and trains the optimal image classification model using Pa. The method is as follows: 8.1 Initialize the weight parameters and set the parameter set W of the second feedforward neural network layer of the optimal image classification model FF Initialize the elements in to random numbers in [0,1]; Set the parameter set W of the second linear classification layer of the optimal image classification model YY The elements in are initialized to random numbers in [0,1]; 8.2 Set the hyperparameters for network training, including the batch size batch_size and the learning rate learningrate of the network model; 8.3 Iteratively train the second feedforward neural network layer and the second linear classification layer of the optimal image classification model in a supervised manner. In each round of training, the predicted label set Pa′ of each batch of samples is obtained, and the difference between Pa′ and the actual label set Pa of each batch of samples is calculated. ## The distribution gap is obtained, the loss value is minimized and W is updated FF and W YY , until the number of iterations required is met, and the parameter set W of the second feedforward neural network layer is obtained. FF and the parameter set W of the second linear classification layer YY , the method is as follows: 8.3.1 Initialize the training iteration parameter itretation = 1; Initialize the second iteration threshold EE, where EE is an integer within [1, 200]; 8.3.2 Let bz = 1; 8.3.3 Let kk = 1, and let the predicted label set Pa′ be empty; 8.3.4 The second feedforward neural network layer of the optimal image classification model reads the kkth sample Pa from Pa kk , using the linear mapping method to kk Perform linear mapping and get Pa kk The representation after linear mapping f(Pa kk ), that is, f(Pa kk )=W FF ×Pa kk , f(Pa kk ) is sent to the second linear classification layer of the optimal image classification model; 8.3.5 The second linear classification layer of the optimal image classification model receives f(Pa kk ), f(Pa kk ) is linearly mapped to the second logits vector of the number of categories in the category table, and the second logits vector contains the kkth sample Pa kk The unnormalized probability of each category; perform a softmax operation on the second logits vector to obtain Pa k The probability of each category in the corresponding category table, take the category with the highest probability as Pa kk The predicted label Pa'' is added to the predicted label set Pa'; 8.3.6 If kk ≤ KK and kk = bz × batch_size, go to 8.3.7; If kk > KK, it means that all KK samples in Pa have participated in training and one iteration of training has been completed, go to 8.3.8; If kk ≤ KK and kk < bz × batch_size, let kk = kk + 1 and go to 8.3.4; 8.3.7 The second linear classification layer uses the cross loss entropy function to calculate Pa′ and Pa ## The difference between them is used to obtain the loss value Loss, and the SGD optimization algorithm is used to minimize the Loss. The weight parameters of the optimal image classification model are updated through back propagation. Let kk=kk+1, bz=bz+1. If kk>KK, it means that KK samples of Pa have participated in the training and one iteration of training has been completed. Go to 8.3.8, otherwise go to 8.3.

4. 8.3.8 Let itretation = itretation + 1. If itretation ≤ the second iteration threshold EE, go to 8.3.

2. Otherwise, the training meets the iteration number requirement, and the training ends. The weight parameters of the second feedforward neural network layer and the second linear classification layer in the optimal image classification model are obtained, that is, the optimal W is obtained. FF and W YY , go to step 9; In the ninth step, the training module will train the W FF As the network weight parameter of the second feedforward neural network layer of the optimal image classification model, the trained W YY As the weight parameter of the second linear classification layer of the optimal image classification model, the trained image classification model is obtained, and the trained optimal image classification model and Pa, Pt, and category table are sent to the classification module; The tenth step: The classification module receives the trained optimal image classification model, Pa, Pt, and the category table from the training module, and also receives the unlabeled image p uploaded by the user, and performs image classification on p to obtain the image category of p. The method is as follows: 10.1 The classification module receives the trained optimal image classification model from the training module; 10.2 The second feedforward neural network layer of the trained optimal image classification model receives the unlabeled image p uploaded by the user; linear mapping is performed on p to obtain the linear mapping representation f(p), that is, f(p) = W FF ×p, sends f(p) to the second linear classification layer of the trained optimal image classification model; 10.3 The second linear classification layer of the trained optimal image classification model receives f(p) from the second feedforward neural network layer, obtains the number of categories by referring to the category table uploaded by the user, linearly maps f(p) to a third logits vector with the dimension of the number of categories in the category table. The third logits vector contains the probabilities corresponding to each category of the unlabeled image p uploaded by the user without normalization; Perform a softmax operation on the third logits vector to obtain the probabilities of each category in the category table corresponding to p, and take the category with the highest probability as the image category of p to complete the classification of p.

2. The two-stage image classification method based on model search according to claim 1, wherein the method for sorting the executable code in Code in step 1.2.4 is: Divide the executable code of the model into three layers: The first layer is the model definition code, including the code related to the network structure and the global parameter initialization code; The second layer is the definition code of other global class names and function names; The third layer is the remaining other code in the code sequence Code; Place the first-layer code at the front of Code, the second-layer code after the first-layer code, and the third-layer code at the end.

3. The two-stage image classification method based on model search as claimed in claim 1, characterized in that the method of expanding the data set in step 3.1.7 to form a data set containing 6vt samples is: 3.1.7.1 Set variable count = 1; 3.1.7.2 Expand the data set by replacing synonyms; replace the user demand text Q in the count-th sample in the data set count From "the model with the highest classification accuracy among the top v% of models in inference speed" to Q count ′,Q count ′ is "the model with the highest prediction accuracy among the models with the top vv% in inference speed", forming a new sample = {Q count ′, m, label}, added to the data set; 3.1.7.3 Expand the dataset by changing the word positions; change the user demand text Q in the demand-model pair of the count-th sample in the dataset count From "the model with the highest classification accuracy among the top v% of models in inference speed" to Q count ″, Q count ″ is "the inference speed is in the top v% among all models, and the classification accuracy is the highest", forming a new sample = {Q count ″, m, label}, added to the data set; 3.1.7.4 Let count = count + 1. If count ≤ 2000, go to 3.1.7.

2. Otherwise, a data set containing 6vt samples is obtained.

4. The two-stage image classification method based on model search as claimed in claim 1, characterized in that 3.2.1 Step marking method is: 3.2.1.1 Determine which categories of images need to be classified, set a label for each category, and make a category table; 3.2.1.2 Collect a certain number of images for each category from the Internet, with no less than 500 images for each category, and label these images with the corresponding labels.

5. The two-stage image classification method based on model search as claimed in claim 1, characterized in that The method for setting the network training hyperparameters described in step 4.2 is: set the batch size batch_size to 32, the network model learning rate learning rate to 0.001, the maximum sequence length to 200, and the gradient accumulation parameter gradient_accumulation_steps to 1; the iteration threshold E described in step 4.3.1 is initialized to 8.

6. The two-stage image classification method based on model search as claimed in claim 1, characterized in that 7.1 The scoring layer of the second search module scores the 20 image classification models in turn in the following way: 7.1.1 Divide the image set P of the second training set Pa into multiple batches according to the batch size, BZ is the total number of batches, It means rounding down 0.8KK÷batch_size; where a batch of pictures is represented as a batch picture set Pb = {Pb1, Pb2, ..., Pb bb , ..., Pb BB }, 1≤bb≤BB, where BB is the number of pictures in Pb, BB is equal to batch_size, Pb bb It is the bbth picture in Pb; 7.1.2 Let mcnt = 1, and let the unordered set of model scores Set(score) be empty; 7.1.3 Let bz = 1; 7.1.4 Take the bzth batch of images from P; 7.1.5 Set variable cnt = 1; 7.1.6 Take the cntth image Pb from the bzth batch cnt , Pb cnt After passing through the second feedforward neural network layer of the mcnt-th image classification model, the linear mapping is obtained to represent g(Pb cnt ), namely g(Pb cnt )=ReLU(Pb cnt ), ReLU is the activation function in the second feedforward neural layer of the mcnt-th image classification model, ReLU=max(0,p); the mapping of all activation functions in the second feedforward neural layer of the mcnt-th image classification model is Pb cnt Form a unique linear code u cnt , Among them, 1≤z≤N A , It is the picture Pb cnt The mapping at the zth activation unit, N A is the number of activation units in the second feedforward neural network layer in the mcnt-th image classification model; 7.1.7 Let cnt = cnt + 1. If cnt ≤ BB, go to 7.1.

6. Otherwise, it means that {u1, u2, ..., u cnt , ..., u BB }, where 1≤cnt≤BB, go to 7.1.8; 7.1.8 Using the Hamming distance d H (u a ,u b ) measures the similarity of the linear codes between all the images in a batch, u a is the linear code of the ath sample, u b is the linear code of the b-th sample, 1≤a≤BB, 1≤b≤BB; calculate the kernel matrix K H : Calculate the score of the mcnt-th image classification model on P cnt ; Among them, |K H | indicates K H The value of the determinant; the model information sequence m of the mcnt-th image classification model cnt and m cnt The corresponding score cnt Composed of two tuples (m cnt , score cnt ), add it to the model score unordered set Set(score); let bz=bz+1, mcnt=mcnt+1, if bz≤BZ and mcnt≤20, go to 7.1.4; if bz>BZ and mcnt≤20, go to 7.1.3; if mcnt>20, get the model score unordered set Set(score) containing 20 bigrams.

7. The two-stage image classification method based on model search as claimed in claim 1, characterized in that The method for setting the network training hyperparameters in step 8.2 is: setting the batch size batch_size to 64 and the network model learning rate learning rate to 0.001; in step 8.3.1, the second iteration threshold EE is 100.

8. The two-stage image classification method based on model search as claimed in claim 1, characterized in that In the second step, the size of the unlabeled image p uploaded by the user and the KK labeled images in step 3.2.2 are both adjusted to 64×64, where KK ≥ 500.

Images