A long-tail object recognition method based on artificial synthetic data and multi-source transfer learning

Abstract

The application discloses a long-tail object recognition method based on artificial synthetic data and multi-source transfer learning, which comprises the following steps: acquiring a multi-source training image set and performing pretreatment and feature extraction, determining the class label of each training image, and constructing a training data set according to the training feature vector and the class label; initializing the weight set corresponding to the training data set respectively, setting a weight adjustment factor and an iteration number, selecting an initial weak classifier learning algorithm, training the training data set according to the weight adjustment factor, the iteration number, the initial weak classifier learning algorithm and the initialized weight set, and acquiring T weak classifiers and classifier coefficients; acquiring a target field test image set and performing pretreatment and feature extraction, predicting the class label of each test image according to the T weak classifiers, and determining the class label of each test image according to the prediction result and the classifier coefficient. The image classifier performance under the condition of absolute imbalance of image class distribution is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of small-sample long-tail object recognition, and in particular to a long-tail object recognition method based on artificially synthesized data and multi-source transfer learning. Background Technology

[0002] Object recognition is a fundamental problem in computer vision. Because real-world image datasets exhibit a long-tailed distribution of class labels, classifiers trained on imbalanced image data are prone to overfitting to the majority class (head) data, thus neglecting the minority class (tail) during classification prediction. This is especially problematic when training data is insufficient, leading to a decline in the model's generalization ability.

[0003] For the above problems, there are the following solutions: The Rare-Transfer algorithm based on single-source instance transfer learning proposed in reference [1] is the first to apply single-source instance transfer learning to solve the class label imbalance classification problem with insufficient training samples. Rare-Transfer is based on the classic TrAdaboost framework in the field of transfer learning [2], and introduces a correction factor Ct in each round of Boosting iteration to prevent the weights of the source domain samples from converging too early. The main drawback of the Rare-Transfer algorithm is that it is only suitable for handling binary classification problems and instance transfer from a single source domain. When the minority class samples in the source domain are still few, the effect of single-source transfer learning is very limited and it is easy to cause "negative transfer", that is, the knowledge contained in the "positive samples" of the source domain is harmful to solving the object recognition of the target domain. Although references [3-5] extended the TrAdaboost framework from single-source transfer to multi-source transfer learning and proposed multi-source transfer learning algorithms based on the TrAdaBoost framework, these algorithms are mainly aimed at binary classification problems with balanced class label distribution data. Reference [6] proposes a Weighted Multisource-TrAdaboost algorithm, which considers the impact of the total number of training samples in the source and target domains on the classifier performance during instance transfer. The main drawback of this algorithm is that it is applicable to binary classification problems, introduces new hyperparameters, and does not consider solving the problem of class imbalance in the target domain. Reference [7] combines the multi-class Adaboost algorithm SAMME [8] with the TrAdaboost framework to propose a multi-class single-source instance transfer learning algorithm. The main drawback of this algorithm is that it is only applicable to single-source transfer learning and standard class label distribution datasets, and cannot effectively handle the problem when the training data in the target domain is imbalanced.

[0004] In summary, existing methods have the following drawbacks: 1) Imbalanced classification algorithms based on the TrAdaboost framework only consider single-source instance transfer and binary classification problems, and are primarily applied to text classification tasks. When training samples have multiple categories, the classification accuracy is low; 2) Multi-source instance transfer algorithms based on the TrAdaboost framework only consider the classifier training problem when the class labels are in a standard distribution, and cannot effectively and accurately classify training samples with imbalanced class label distributions; 3) Algorithms based on SMOTE technology and the Boosting framework only train the model in the target domain, without considering the effective use of knowledge in the source domain, which reduces the classification accuracy. Furthermore, they mainly focus on the training algorithm design of binary classification models and cannot classify training samples with multiple categories.

[0005] References

[0006] [1]Al-Stouhi S.,&Reddy C K.Transfer learning for class imbalanceproblems with inadequate data. Knowledge&Information Systems, 2016, 48(1), 201-228.

[0007] [2]Dai W, Yang Q, Xue GR, Yu Y (2007a)Boosting for transfer learning. In: Proceedings of the international conference on machine learning, 2007, pp 193–200.

[0008] [3] Yao Y, Doretto G, Boosting for transfer learning with multiple sources, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010, 1855-1862.

[0009] [4] Zhang Qian, Li Ming, Wang Xuesong, et al. A method for instance transfer learning in multi-source domains [J]. Acta Automatica Sinica, 2014, 40(006): 1176-1183.

[0010] [5] Zhang Qian, Li Haigang, Li Ming, Cheng Yuhu. Instance transfer learning method based on multi-source dynamic TrAdaBoost [J]. Journal of China University of Mining and Technology, 2014, 43(04):713-720.

[0011] [6]Antunes,J.,Bernardino,A.,Smailagic,A.,Siewiorek,D.(2019).WeightedMultisource Tradaboost.In:Morales,A.,Fierrez,J.,Sánchez,J.,Ribeiro,B.(eds)Pattern Recognition and Image Analysis.IbPRIA 2019.Lecture Notes in ComputerScience(),vol 11867.Springer, Cham.

[0012] [7]Hanxian He,Kourosh Khoshelham,Clive Fraser,A multiclass TrAdaBoosttransfer learning algorithm for the classification of mobile lidar data,ISPRSJournal of Photogrammetry and Remote Sensing,Volume166,2020,Pages 118-127.

[0013] [8]Hastie, T., Rosset, S., Zhu, J., Zou, H., 2009. Multi-classadaboost. Stat. Interface 2, 349–360. Summary of the Invention

[0014] This invention provides a long-tail object recognition method based on artificially synthesized data and multi-source transfer learning to overcome the above-mentioned technical problems.

[0015] A method for long-tail object recognition based on artificially synthesized data and multi-source transfer learning includes,

[0016] Step 1: Obtain N source domain training image sets and one target domain training image set, and preprocess them. Then, based on the deep neural network, extract features from each training image in the preprocessed N source domain training image sets and one target domain training image set to obtain the training feature vector of each training image.

[0017] Step 2: Perform L2-z-score normalization on the training feature vector of each training image. Determine the class label of each training image based on the object categories it contains. Construct N source domain training datasets and one target domain training dataset based on the normalized training feature vectors and class labels.

[0018] Step 3: Initialize the weight sets corresponding to the N source domain training datasets and one target domain training dataset, respectively. Set the weight adjustment factor, set the number of iterations to T, select the initial weak classifier learning algorithm, and train the N source domain training datasets and one target domain training dataset based on the weight adjustment factor, the number of iterations, the initial weak classifier learning algorithm, and the initialized weight sets. Obtain T weak classifiers and their classifier coefficients.

[0019] Step 4: Obtain the target domain test image set and preprocess it. Extract features from the preprocessed target domain test image set using a deep neural network to obtain the test feature vector of each test image in the target domain test image set. Predict the class label of each test image's test feature vector using T weak classifiers. Determine the class label of each test image based on the prediction results of the T weak classifiers and the classifier coefficients. The class label is the category of the object contained in the test image.

[0020] Preferably, step three includes,

[0021] S1. Initialize the weight sets of N source domain training datasets and one target domain training dataset respectively, set the weight adjustment factor, and initialize n=1, t=1.

[0022] S2. For the t-th iteration, normalize the weight sets of the N source domain training datasets and the target domain training dataset, and manually synthesize the weight sets corresponding to the target domain training dataset and the target domain training dataset respectively, to obtain the manually synthesized weight sets corresponding to the target domain training dataset and the target domain training dataset.

[0023] S3. Merge the nth source domain training dataset, the target domain training dataset, and the artificially synthesized target domain training dataset respectively, and represent them as the nth merged training set. Merge the weight sets corresponding to the nth source domain training dataset, the target domain training dataset, and the artificially synthesized target domain training dataset respectively, and represent them as the nth merged weight set. Train the nth merged training set according to the selected initial weak classifier learning algorithm and the nth merged weight set to obtain the trained nth weak classifier. Calculate the nth training error rate of the nth weak classifier on the target domain dataset.

[0024] S4. Let n = n + 1. When n ≤ N, return to S3 to obtain N weak classifiers. Select the classifier with the lowest training error rate from the N weak classifiers as the weak classifier for the t-th iteration.

[0025] S5. Let n=1, obtain the weak classifier obtained in the t-th iteration and its corresponding training error rate, calculate the classifier coefficients of the weak classifier based on the training error rate, update the weight set of the target domain training dataset based on the classifier coefficients, calculate the correction factor based on the training error rate, and update the weight set of the source domain training dataset based on the correction factor.

[0026] S6. Let t = t + 1, return to execute S2, until t = T, and obtain T weak classifiers and their corresponding T classifier coefficients.

[0027] Preferably, the step of manually synthesizing the target domain training dataset and the weight set corresponding to the target domain training dataset includes:

[0028] S11. Determine the object category contained in each training image in the target domain training dataset and use the object category as the class label. The number of class labels is K. Divide the target domain training dataset into K sample sets according to the class labels. Calculate the number of artificially synthesized samples Nsmote(i) for the i-th sample set, where i = 1, i ≤ K.

[0029] S12. For the i-th sample set, generate Nsmote(i) synthetic sample sets based on SMOTE technology, and assign a synthetic weight and a synthetic class label to each synthetic sample in the synthetic sample set. Store the synthetic sample set, synthetic weight set, and synthetic class label set of the i-th sample set and represent it as the i-th artificial synthetic sample set.

[0030] S13. Let i = i + 1, repeat S12 until i = K, sequentially obtain the artificially synthesized sample sets of K sample sets, merge the synthetic sample sets in the artificially synthesized sample sets with the target domain training dataset, and merge the synthetic weight sets in the artificially synthesized sample sets with the weight sets corresponding to the target domain training dataset.

[0031] Preferably, the selection of the initial weak classifier learning algorithm includes randomly selecting one classifier learning algorithm from SVM, Extreme Learning Machine, and Decision Stump as the initial weak classifier learning algorithm.

[0032] Preferably, S12 includes,

[0033] S121. Let m = 1. When m ≤ Nsmote(i), perform the m-th random sampling based on the i-th sample set. Each time, obtain an anchor sample (x_anchor, y_anchor, w_anchor), where x_anchor is the feature vector of the anchor sample, y_anchor is the class label of the anchor sample, and w_anchor is the weight of the anchor sample.

[0034] S122. Use the k-nearest neighbor algorithm to search for the k nearest neighbor samples of x_anchor sample respectively. Then, select one nearest neighbor sample from the k nearest neighbor samples and assign the feature vector of the selected nearest neighbor sample to x_nearest, x_nearest is the first temporary variable. Assign the weight of the selected nearest neighbor sample to w_nearest, w_nearest is the second temporary variable.

[0035] S123. Calculate the feature vector x_syn of the synthesized sample according to formula (1).

[0036] x_syn= x_anchor+(x_nearest-x_anchor).*rand(1,Nfeatures) (1)

[0037] Where rand(1, Nfeatures) represents the random generation of an Nfeatures-dimensional feature vector with values ​​ranging from (0, 1), Nfeatures is the feature dimension of the x_anchor feature vector, and .* indicates element-wise multiplication of corresponding positions of the two feature vectors.

[0038] S 124. Calculate the synthesis weight w_syn of the synthesized sample according to formula (2).

[0039] w_syn=w_anchor*r1+w_nearest*(1-r1) (2)

[0040] Where r1 is a random number ranging from (0,1).

[0041] S125. Use the class label y_anchor of the anchor sample as the synthesis class label y_syn of the synthesized sample, and store the feature vector, synthesis weight, and synthesis class label of the synthesized sample.

[0042] S126. Let m = m + 1, return to S121, obtain Nsmote(i) synthetic samples and store them as the synthetic sample set of the i-th sample set, and represent the synthetic sample set, synthetic weight set and synthetic label set of the i-th sample set as the i-th artificial synthetic sample set.

[0043] This invention provides a long-tail object recognition method based on artificially synthesized data and multi-source transfer learning. By artificially synthesizing small-sample long-tail data using SMOTE technology, the class label distribution of training samples in the target domain can be dynamically rebalanced. This method is particularly suitable for small-sample imbalanced object recognition (image classification) tasks, improving the versatility of image classification. Furthermore, the weak classifier in each iteration can use various classifier learning algorithms such as SVM, Extreme Learning Machine, and decision stumps, increasing the flexibility of image classification. By automatically generating weighted artificially synthesized data in each iteration and performing classification training based on multiple source domain data, the performance of image classifiers under absolutely imbalanced class distribution conditions is effectively improved. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the method of the present invention;

[0046] Figure 2 This is the execution flowchart of the Weigthed_MCSMOTE algorithm of this invention;

[0047] Figure 3 This is the execution flowchart of the MSMCTraSMOTEboost algorithm of this invention;

[0048] Figure 4 This is a flowchart of the image classifier training and performance evaluation of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Figure 1 This is a flowchart of the method of the present invention, as shown below. Figure 1 As shown, the method in this embodiment may include:

[0051] A method for long-tail object recognition based on artificially synthesized data and multi-source transfer learning includes,

[0052] Step 1: Obtain N source domain training image sets and one target domain training image set, and preprocess them. The target domain training image set has insufficient samples and an imbalanced class label distribution. Using a deep neural network (ResNet18), feature extraction is performed on each training image in the preprocessed N source domain training image sets and one target domain training image set. Since ResNet18 requires an input image size of 224×224×3, but training and testing images often have different sizes, the image size needs to be preprocessed to 224×224×3 before inputting the training images into the ResNet18 network. The training feature vector for each training image is then obtained, specifically the 512-dimensional feature vector output by the global pooling layer 'pool5' at the end of the ResNet18 network.

[0053] Step 2: Perform L2-z-score normalization on the training feature vector of each training image. Determine the class label of each training image based on the object categories it contains. Construct N source domain training datasets and one target domain training dataset based on the normalized training feature vectors and class labels.

[0054] Step 3: Initialize the weight sets corresponding to the N source domain training datasets and one target domain training dataset, respectively. Set the weight adjustment factor, set the number of iterations to T, select the initial weak classifier learning algorithm, and train the N source domain training datasets and one target domain training dataset based on the weight adjustment factor, the number of iterations, the initial weak classifier learning algorithm, and the initialized weight sets. Obtain T weak classifiers and their classifier coefficients.

[0055] The initial weak classifier learning algorithm is selected by randomly choosing one classifier learning algorithm from SVM, Extreme Learning Machine, and Decision Tree.

[0056] Step three includes,

[0057] S1. Initialize the weight sets of N source domain training datasets and one target domain training dataset respectively, set the weight adjustment factor, and initialize n=1, t=1.

[0058] S2. For the t-th iteration, normalize the weight sets of the N source domain training datasets and the target domain training dataset, and manually synthesize the weight sets corresponding to the target domain training dataset and the target domain training dataset respectively, to obtain the manually synthesized weight sets corresponding to the target domain training dataset and the target domain training dataset.

[0059] The manual synthesis of the target domain training dataset and the corresponding weight set includes,

[0060] S11. Determine the source domain to which each training image in the target domain training dataset belongs and use the source domain as the class label. The number of class labels is K. Divide the target domain training dataset into K sample sets according to the class labels. Calculate the number of artificially synthesized samples Nsmote(i) for the i-th sample set, where i = 1, i ≤ K.

[0061] S12. For the i-th sample set, generate Nsmote(i) synthetic sample sets based on SMOTE technology, and assign a synthetic weight and a synthetic class label to each synthetic sample in the synthetic sample set. Store the synthetic sample set, synthetic weight set, and synthetic class label set of the i-th sample set and represent it as the i-th artificial synthetic sample set.

[0062] S12 includes,

[0063] S121. Let m = 1. When m ≤ Nsmote(i), perform the m-th random sampling based on the i-th sample set. Each time, obtain an anchor sample (x_anchor, y_anchor, w_anchor), where x_anchor is the feature vector of the anchor sample, y_anchor is the class label of the anchor sample, and w_anchor is the weight of the anchor sample.

[0064] S122. Use the k-nearest neighbor algorithm to search for the k nearest neighbor samples of x_anchor sample respectively. Then, select one nearest neighbor sample from the k nearest neighbor samples and assign the feature vector of the selected nearest neighbor sample to x_nearest, where x_nearest is the first temporary variable. Assign the weight of the selected nearest neighbor sample to w_nearest, where w_nearest is the second temporary variable.

[0065] S123. Calculate the feature vector x_syn of the synthesized sample according to formula (1).

[0066] x_syn= x_anchor+(x_nearest-x_anchor).*rand(1,Nfeatures) (1)

[0067] Where rand(1, Nfeatures) represents the random generation of an Nfeatures-dimensional feature vector with values ​​ranging from (0, 1), Nfeatures is the feature dimension of the x_anchor feature vector, and .* indicates element-wise multiplication of corresponding positions of the two feature vectors.

[0068] S124. Calculate the synthesis weight w_syn of the synthesized sample according to formula (2).

[0069] w_syn=w_anchor*r1+w_nearest*(1-r1) (2)

[0070] Where r1 is a random number that takes values ​​uniformly distributed between (0,1).

[0071] S125. Use the class label y_anchor of the anchor sample as the synthesis class label y_syn of the synthesized sample, and store the feature vector, synthesis weight, and synthesis class label of the synthesized sample.

[0072] S126, m = m + 1, return to S121, obtain Nsmote(i) synthetic samples and store them as the synthetic sample set of the i-th sample set, and represent the synthetic sample set, synthetic weight set and synthetic label set of the i-th sample set as the i-th artificial synthetic sample set;

[0073] The implementation process of S2 is represented by the Weighted-MCSMOTE algorithm. The process of the Weighted-MCSMOTE algorithm is shown in Table 1, and the flowchart of the Weighted-MCSMOTE algorithm is as follows. Figure 2 As shown.

[0074] Table 1. Process of the Weighted-MCSMOTE algorithm

[0075]

[0076]

[0077] In the algorithm Weighted-MCSMOTE, it is assumed that D T The maximum number of samples in each category is Nmax = max(classNo). The Weighted-MCSMOTE algorithm scans D T For samples with different class labels, Nmax-classNo(ii)+1 samples are artificially synthesized for the ii-th class using the SMOTE technique, so that the number of samples for each class in the training dataset reaches Nmax+1, and a weight is assigned to each artificially synthesized sample according to the formula in line 19.

[0078] S13. Let i = i + 1, repeat S12 until i = K, sequentially obtain the artificially synthesized sample sets of K sample sets, merge the synthetic sample sets in the artificially synthesized sample sets with the target domain training dataset, and merge the synthetic weight sets in the artificially synthesized sample sets with the weight sets corresponding to the target domain training dataset.

[0079] S3. Merge the nth source domain training dataset, the target domain training dataset, and the artificially synthesized target domain training dataset respectively, and represent them as the nth merged training set. Merge the weight sets corresponding to the nth source domain training dataset, the target domain training dataset, and the artificially synthesized target domain training dataset respectively, and represent them as the nth merged weight set. Train the nth merged training set according to the selected initial weak classifier and the nth merged weight set to obtain the trained nth weak classifier. Calculate the nth training error rate of the nth weak classifier on the target domain dataset.

[0080] S4. Let n = n + 1. When n ≤ N, return to S3 to obtain N weak classifiers. Select one weak classifier from the N weak classifiers as the weak classifier for the t-th iteration.

[0081] S5. Let n=1, obtain the weak classifier obtained in the t-th iteration and its corresponding training error rate, calculate the classifier coefficients of the weak classifier based on the training error rate, update the weight set of the target domain training dataset based on the classifier coefficients, calculate the correction factor based on the training error rate, and update the weight set of the source domain training dataset based on the correction factor.

[0082] S6. Let t = t + 1, return to execute S2, until t = T, and obtain the T weak classifiers and their corresponding T classifier coefficients;

[0083] Step three is represented by the MSMCTraSMOTE boost algorithm. The implementation process of the MSMCTraSMOTE boost algorithm is shown in Table 2, and the flowchart of the MSMCTraSMOTE boost algorithm is shown below. Figure 3 As shown.

[0084] Table 2 Implementation process of MSMCTraSMOTE boost algorithm

[0085]

[0086]

[0087] The first line of the MSMCTraSMOTEboost algorithm initializes the weights of the training samples. The second line sets the weight adjustment factor of the source neighborhood samples to β. s Lines 3-18 perform T rounds of iterative training to train T weak classifiers f. t(x). Line 4 normalizes the weight vector. Line 5 calls the Weighted-MCSMOTE algorithm to artificially synthesize weighted data, thereby rebalancing the class label distribution of the target domain training samples. Lines 6-9 indicate that in the t-th iteration, N source domain datasets are traversed to construct N weak classifiers, and lines 10-11 select one of them from D. T The weak classifier with the lowest weighted training error rate is used as the final weak classifier f in this iteration. t The 7th line contains the training dataset for each source domain. The dataset D output by the Weighted-MCSMOTE algorithm T ∪D SMOTE Merging, based on datasets A classifier is trained using the current weights of its samples. The 8th line calculates the classifier. Training error rate on the target domain dataset Therefore, after lines 5-11, the weak classifier f t The data stored in D is from the t-th iteration. T The classifier with the lowest error rate is trained using weighted methods. Line 12 calculates the correction factor C. t This is primarily used to prevent premature convergence of source domain sample weights during transfer learning. Line 13 is based on the training error rate ε. Tar,t Given the number of categories K, calculate the weight adjustment factor for the target domain samples. Line 14 according to Update the weights of the training samples in the target domain. Line 15 is based on the calculated... Reduce the weights of misclassified samples in the target domain training dataset, while keeping the weights of correctly classified samples unchanged. Lines 15-17 indicate that in the t-th iteration, the weights of the source domain samples are updated by traversing N source domain training datasets.

[0088] Step 4: Obtain the target domain test image set and preprocess it. Ensure the test image class label distribution is balanced. Preprocess the test image size to 224×224×3. Extract features from the preprocessed target domain test image set using a deep neural network. Obtain the test feature vector of each test image in the target domain test image set. Input the test image into the ResNet18 network and extract the 512-dimensional feature vector output by the global pooling layer 'pool5'.

[0089] Based on T weak classifiers f t For each test image, (x)(t=1,2,3,….,T) predicts the class label of its test feature vector, based on the prediction results of T weak classifiers and the classifier coefficients. Determine the class label for each test image, where the class label is the category of the object contained in the test image.

[0090] The object recognition process in this embodiment can be divided into two stages: the image classifier training stage and the classifier performance evaluation stage, such as... Figure 4 As shown.

[0091] 1. Image classifier training phase

[0092] Step 1: Load k training images from the source and target domains and preprocess them. Load and train the ResNet18 neural network and resize all images to 224×224×3.

[0093] Step 2: The ResNet18 network is used to extract features from the image. The 512-dimensional feature vector output by the global pooling layer 'pool5' at the end of the ResNet18 network is used as the feature representation of each image in the source and target domains.

[0094] Step 3: Extract the features from the k source domain training sample sets D. S1 D S2 ,…,D Sk and a target domain training sample set D T The feature matrix is ​​preprocessed using L2-z-score normalization and then used as training data input to the proposed MSMCTraSMOTEboost algorithm. T iterations are performed to train T weak classifiers f. t (x) and classifier weight coefficients Where t = 1, 2, ..., T.

[0095] 2. Image Classifier Performance Evaluation Phase

[0096] Step 1: Load the target domain test image, ensure the class label distribution of the test image is balanced, and adjust the size of the test image to 224×224×3.

[0097] Step 2: Input the test image I into the ResNet18 network and extract the 512-dimensional feature vector x output by the global pooling layer 'pool5'.

[0098] Step 3: Load the T weak classifiers f from the trained image classifier. t (x)(t=1,2,3,….,T) and classifier weight coefficients

[0099] Step 4: According to the formula Output the predicted class label for image I.

[0100] In image multi-class classification, the class labels of the training and testing data belong to the set {1,2,…,K}, where K>2.

[0101] The performance of the long-tail recognition classifier in this embodiment is measured, including the following steps:

[0102] I. Defining Indicators

[0103] 1) Definition of imbalance rate

[0104] In the field of imbalanced learning, the degree of imbalance in the distribution of training sample class labels can be measured using the imbalance rate, which is formally expressed as:

[0105]

[0106] Among them, |Y i | represents the number of samples in the i-th class in the training dataset, min i {|Y i |} represents the number of samples in the category with the fewest samples, max i {|Y i |} represents the number of samples in the category with the largest number of samples.

[0107] 1) Performance metrics for binary classification

[0108] For a binary classification problem, assuming the label of the minority class is P (Positive) and the label of the majority class is N (Negative), the confusion matrix is ​​shown in Table 3.

[0109] Table 3 Confusion Matrix for Binary Classification

[0110]

[0111]

[0112] Based on the definitions of TP, FP, TN, and FN in the confusion matrix, the classifier model performance metrics Precision, Recall, F1-measure, G-mean, and Balanced Accuracy (BAC) in long-tail classification are defined as follows:

[0113]

[0114]

[0115]

[0116]

[0117]

[0118] In the above model performance metrics, Precision measures the model's precision on minority class samples, Recall measures the model's recall on minority class samples, and F1-measure is the weighted harmonic mean of Precision and Recall. G-mean and BAC reflect the comprehensive trade-off between TPR (True Positive Rate) and TNR (True Negative Rate).

[0119] 3) Performance metrics for multi-class classification

[0120] Multi-class classification tasks can be decomposed into multiple binary classification tasks, and then a confusion matrix can be built for each category. For example, m-class classification can be decomposed into m binary classification problems, assuming that the minority class label in the i-th binary classification is C. i Most class tags are set {C} j |j=1,...m,j≠i}. In this case, macro-averaging can be used to calculate various model performance metrics.

[0121] Macro averaging refers to first calculating the performance metric on the confusion matrix of each category, and then calculating the arithmetic mean of the metrics. By calculating macro averaging, common binary classification performance metrics can be extended to multi-class classification performance metrics.

[0122] For an m-class classification task, assuming the recall, precision, F1-measure, G-mean, and BAC calculated on m binary confusion matrices are respectively represented as follows: Then we have:

[0123]

[0124]

[0125]

[0126]

[0127]

[0128] II. Constructing the Example Dataset

[0129] To verify the superiority of the proposed method in small-sample long-tail object recognition applications, this embodiment constructs a multi-class long-tail object recognition dataset, Office-Caltch10-LT, based on Office-Caltch10. This dataset contains image datasets from four domains: DSLR, Webcam, Amazon, and Caltech. The Office-Caltch10 dataset contains 10 object classes, which are the same classes as those in the Office-31 and Caltech-256 datasets. DSLR, Webcam, and Amazon are the source domain datasets, while Caltech is the target domain dataset.

[0130] [Hoffman,J.,Rodner,E.,Donahue,J.et al.Asymmetric and CategoryInvariant Feature Transformations for Domain Adaptation.Int J Comput Vis 109,28–41(2014).]

[0131] The steps to construct the long-tail object recognition dataset Office-Caltch10-LT are as follows:

[0132] Step 1: Using a pre-trained ResNet18 deep neural network, perform batch feature encoding on images from each domain in the Office-Caltech10 dataset, and assign a class label c (c∈{1,2,3,…,10}) to each feature vector according to its corresponding object category. The object categories contained in the images in each domain are represented as 'backpack', 'bike', 'calculator', 'headphones', 'calculator', 'keyboard', 'laptop_computer', 'monitor', 'mug', and 'projector', respectively, and the class labels for each object category are 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10.

[0133] Step 2: Following 5-fold Stratified Cross Validation and a 70:30 training / test data ratio, all samples in the Caltech domain were randomly divided into 5 training and test sets. DSLRs, Webcams, and Amazon images were used as the source domains. Therefore, no further random sampling was performed after feature encoding of the images.

[0134] Step 3: For each training sample set, further downsample the training samples from the five categories whose number is the last five digits. Assuming that the largest number of training samples for each object category in the Caltech domain is maxC, and the imbalance ratio is set to IR, then the number of training samples after downsampling for each selected category is reduced to floor(maxC / IRatio).

[0135] Step 4: Construct the Office-Caltch10-LT dataset based on the feature vectors and class labels of the images in the source and target domain training sets. The imbalance rate of the Office-Caltch10-LT dataset is represented by IR.

[0136] III. Main Steps of the Embodiment

[0137] Step 1: Load and preprocess training images from the source domain (DSLR, Webcam, Amazon) and the target domain (Caltech), where the target domain has insufficient training samples and an imbalanced class label distribution. Image feature extraction and encoding are performed using a deep neural network, ResNet18. Since ResNet18 requires input images to be 224×224×3, but training and test images often have different sizes, the images need to be automatically resized to 224×224×3 before being input into the ResNet18 network.

[0138] Step 2: Obtain feature sets from 3 source domain training images and 1 target domain training image. Batch processing saves the 512-dimensional feature vectors output by the global pooling layer 'pool5' at the end of the ResNet18 network as feature representations for each image in both the source and target domains. It's worth noting that since the feature extraction methods for the k source domains and 1 target domain are the same, the feature vectors output from the ResNet18 network have the same dimension.

[0139] Step 3: Combine the three source domain training sample sets D1, D2, ..., D3 after feature encoding with one target domain training sample set D... T The feature matrix is ​​preprocessed using L2-z-score normalization and then used as training data input to the proposed MSMCTraSMOTEboost algorithm. T iterations are performed to train T weak classifiers f. t (x) and classifier weight coefficients Where t = 1, 2, ..., T.

[0140] Step 4: Load the target domain test image, ensure the class label distribution of the test image is balanced, and adjust the test image size to 224×224×3.

[0141] Step 5: Input the test image I into the ResNet18 network, extract the 512-dimensional feature vector x output by the global pooling layer 'pool5', and perform l2-zscore normalization preprocessing.

[0142] Step 6: Load the T weak classifiers f from the trained image classifier. t (x)(t=1,2,3,….,T) and classifier weight coefficients

[0143] Step 7: According to the formula The predicted class label of output image I is the category of the object contained in the test image.

[0144] In Comparative Experiment 1, IR was set to 10, and the number of training and testing samples in the source and target domains are shown in Table 4. It is worth noting that all data in the source domain are training data.

[0145] Table 4. Number of training and test samples in different domains in Experiment 1 (IR=10)

[0146]

[0147]

[0148] Table 5 shows the macro-average experimental results of the comparison experiment 1. The baseline algorithms in Experiment 1 were the multi-class Adaboost algorithm SAMME without transfer learning, the single-source multi-class transfer learning algorithm (MCTrAdaboost, where the source domains were DSLR, Webcam and Amazon respectively), and the MSMCTrASMOTEboost algorithm proposed in this invention.

[0149] Table 5 Results of Comparative Experiment 1 using Macro-Average Measures

[0150]

[0151] In Comparative Experiment 1, IR was set to 20. The number of training and testing samples in the source and target domains are shown in Table 6, where all data in the source domain are training data.

[0152] Table 6 compares the number of training and testing samples in different domains in Experiment 2 (IR=10).

[0153]

[0154]

[0155] Table 7 shows the macro-average experimental results of Comparative Experiment 2. The baseline algorithms in Comparative Experiment 2 were the multi-class Adaboost algorithm SAMME without transfer learning, the single-source multi-class transfer learning algorithm (MCTrAdaboost, where the source domains were DSLR, Webcam and Amazon respectively), and the MSMCTrASMOTEboost algorithm proposed in this invention.

[0156] Table 7 Results of Comparative Experiment 2 using Macro-Average Measures

[0157]

[0158] By comparing the experimental results of Experiments 1 and 2, it can be seen that the image classifier trained by the MSMCTrASMOTEboost algorithm based on artificially synthesized data and multi-source transfer learning proposed in this invention is suitable for long-tail recognition applications and can significantly improve the overall performance of long-tail object recognition under small sample conditions.

[0159] Note:

[0160] The MCTrAdaboost algorithm comes from the literature [1] Hanxian He, Kourosh Khoshelham, Clive Fraser, A multiclass TrAdaBoost transfer learning algorithm for the classification of mobile lidar data, ISPRS Journal of Photogrammetry and Remote Sensing, Volume 166, 2020, Pages 118-127.

[0161] The SAMME algorithm comes from reference [2] Hastie, T., Rosset, S., Zhu, J., Zou, H., 2009. Multi-class adaboost.Stat.Interface 2, 349–3.

[0162] Overall beneficial effects:

[0163] This invention provides a long-tail object recognition method based on artificially synthesized data and multi-source transfer learning. By using SMOTE technology to artificially synthesize small-sample long-tail data, it is particularly suitable for small-sample imbalanced object recognition (image classification) tasks, improving the versatility of image classification. Furthermore, the weak classifier in each iteration can use various classifier learning algorithms such as SVM, Extreme Learning Machine, and decision stump, improving the flexibility of image classification. By automatically generating weighted artificially synthesized data in each iteration and performing classification training based on multiple source domain data, it effectively improves the performance of image classifiers under conditions of absolute imbalance in class distribution.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A long-tail object recognition method based on artificial synthetic data and multi-source transfer learning, characterized in that, comprises, Step one, obtaining N a source domain training image set and a target domain training image set and preprocessing, according to the deep neural network respectively to the preprocessed each training image in the source domain training image set and the target domain training image set feature extraction, obtain the training feature vector of each training image, N a source domain training image set and a target domain training image set and preprocessing, according to the deep neural network respectively to the preprocessed each training image in the source domain training image set and the target domain training image set feature extraction, obtain the training feature vector of each training image, Step two, normalizing the training feature vector of each training image l 2 -zscore determining the class label of the training image according to the object category contained in each training image, and constructing N one source domain training data set and one target domain training data set according to the normalized training feature vector and the class label respectively, Step 3: For each N Initialize the weight sets corresponding to the source domain training dataset and the target domain training dataset, set the weight adjustment factor, set the number of iterations to T, select the initial weak classifier learning algorithm, and adjust the weights according to the weight adjustment factor, the number of iterations, the initial weak classifier learning algorithm, and the initialized weight sets. N The system trains on one source domain training dataset and one target domain training dataset to obtain T weak classifiers and their coefficients. Step three includes... S1, respectively initializing a weight set of one source domain training data set and one target domain training data set, setting a weight adjustment factor, initializing N 1, n= 1, t =1, S2, for the first t The next iteration, for N The weight sets of the source domain training dataset and the target domain training dataset are normalized. Then, the weight sets corresponding to the target domain training dataset and the target domain training dataset are manually synthesized to obtain the manually synthesized weight sets. S3, respectively, the first n source domain training data set, the target domain training data set and the artificially synthesized target domain training data set are merged and represented as the first n merged training set, respectively, the weight set corresponding to the first n source domain training data set, the weight set corresponding to the target domain training data set and the weight set corresponding to the artificially synthesized target domain training data set are merged and represented as the first n merged weight set, the first n merged training set is trained according to the selected initial weak classifier learning algorithm and the first n merged weight set, the trained first n weak classifier is obtained, the first n training error rate of the first n weak classifier on the target domain data set is calculated, S4, let n = n +1 ， When n ≤ N , return to execute S3 ， Obtain N weak classifiers, select the weak classifier with the lowest training error rate from N weak classifiers as the weak classifier of the t iteration, S5, Order n= 1. Obtain the first t The weak classifiers obtained in each iteration and their corresponding training error rates are used to calculate the classifier coefficients based on the training error rates. The weight set of the target domain training dataset is then updated based on the classifier coefficients. Finally, a correction factor is calculated based on the training error rates, and the weight set of the source domain training dataset is updated based on the correction factor. S6, let t = t +1, return to execute S2 until t =T, obtain T weak classifiers and T corresponding classifier coefficients Step four, obtaining a target field test image set and preprocessing, according to the deep neural network to the target field test image set after preprocessing feature extraction, obtain the test feature vector of each test image in the target field test image set, according to the class label of each test image test feature vector belongs to T weak classifier prediction, according to the prediction result of T weak classifier and classifier coefficient determine the class label of each test image, the class label is the object category contained in the test image.

2. The long-tail object recognition method based on synthetic data and multi-source transfer learning according to claim 1, characterized in that, The artificial synthesis of the target field training data set and the weight set corresponding to the target field training data set comprises, S11, determine the object category contained in each training image in the target field training data set and take the object category as the class label, the number of class labels is K, divide the target field training data set into K sample sets according to the class label, calculate the number of samples Nsmote(i) that need to be artificially synthesized for the ith sample set, wherein i=1, i≤K, S12, for the ith sample set, generate Nsmote(i) synthetic sample sets based on the SMOTE technology, and assign a synthetic weight and a synthetic class label to each synthetic sample in the synthetic sample set, store the synthetic sample set, the synthetic weight set and the synthetic class label set of the ith sample set, and represent the ith artificial synthetic sample set, S13, let i=i+1, repeat S12 until i=K, sequentially obtain the artificial synthetic sample set of the K sample sets, and merge the synthetic sample set in the artificial synthetic sample set with the target field training data set, and merge the synthetic weight set in the artificial synthetic sample set with the weight set corresponding to the target field training data set.

3. The long-tail object recognition method based on synthetic data and multi-source transfer learning according to claim 1, characterized in that, The selection of the initial weak classifier learning algorithm comprises randomly selecting one of SVM, extreme learning machine and decision tree stump as the initial weak classifier learning algorithm.

4. The long-tail object recognition method based on synthetic data and multi-source transfer learning according to claim 2, characterized in that, The S12 comprises, S121, let m=1, when m≤Nsmote(i), based on the ith sample set, the mth random sampling is carried out, and an anchor sample (x_anchor, y_anchor, w_anchor) is obtained each time, wherein x_anchor is the feature vector of the anchor sample, y_anchor is the class label of the anchor sample, and w_anchor is the weight of the anchor sample; S122, using k nearest neighbor algorithm to search k nearest neighbor samples of x_anchor sample, and selecting one of the k nearest neighbor samples, and assigning the feature vector of the selected nearest neighbor sample to x_nearest, x_nearest is a first temporary variable, and assigning the weight of the selected nearest neighbor sample to w_nearest, w_nearest is a second temporary variable; S123, according to formula (1), the feature vector x_syn of the synthetic sample is calculated, x_syn = x_anchor + (x_nearest - x_anchor). rand(1,Nfeatures)(1) where rand(1, Nfeatures) represents a random generated NFeatures dimensional feature vector with values in the interval (0, 1), and Nfeatures is the feature dimension of the x_anchor feature vector. represents the multiplication of the elements in the corresponding positions of the two feature vectors, S 124, according to formula (2), the synthetic weight w_syn of the synthetic sample is calculated, w_syn = w_anchor r1 + w_nearest (1 - r1)(2) wherein r1 is a random number with a value ranging between (0, 1), S125, taking the class label y_anchor of the anchor sample as a synthetic class label y_syn of the synthetic sample, storing the feature vector, the synthetic weight, and the synthetic class label of the synthetic sample, S126, let m = m + 1, return to S121, obtain Nsmote(i) synthetic samples and store them as a synthetic sample set of the i-th sample set, and represent the synthetic sample set, the synthetic weight set, and the synthetic label set of the i-th sample set as the i-th artificial synthetic sample set.

Images