A data classification method based on joint transfer extreme learning machine

Abstract

The application relates to a data classification method based on a joint transfer extreme learning machine, and effectively solves the problem that an existing ELM does not have a knowledge transfer capability and thus has poor classification performance; the technical solution comprises the following steps: firstly, pre-processing data set samples; secondly, introducing cross-domain mean approximation (CDMA) into the ELM to obtain an extreme learning machine with a transfer capability, effectively minimizing the difference between the marginal and conditional distribution of two domains; thirdly, constructing another ELM model with output weight alignment and parameter approximation; applying subspace alignment technology to align the output weights of two domains, and adding an approximation term to approximate the output weight, so as to promote knowledge transfer; and finally, classifying and predicting target samples by using two learned ELMs.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a data classification method based on a joint transfer extreme learning machine. Background Art

[0002] Extreme learning machines (ELMs) are widely used in pattern recognition, computational science, and machine learning due to their fast learning speed and good generalization performance. Since ELMs do not require adjusting input weights and biases, but only optimize output weights by solving least squares, they have attracted widespread attention in fields such as industrial fault diagnosis, medical diagnosis, hyperspectral image classification, face recognition, and brain-computer interfaces.

[0003] However, like traditional machine learning models, ELM usually requires that training samples and test samples follow the same distribution to achieve good accuracy. When there are insufficient training samples, its performance deteriorates and the effect is poor. In addition, collecting training samples with the same distribution is usually time-consuming and expensive, which leads to unsatisfactory classification results of the ELM model. Transfer learning can address the above shortcomings of ELM. It uses rich labeled data samples from other fields that have different distributions but are related to the target domain (test samples) as training samples (source domain) to learn a reliable machine learning model and make predictions for the current target domain (test samples).

[0004] Currently, many scholars have designed a number of extreme learning machine models combined with transfer learning - transfer extreme learning machines (TLELM). According to the different ways of adjusting the distribution between domains, we roughly divide them into three types: 1) Target sample supervision: This type of method usually requires some labeled samples in the target domain to adjust the machine learning model learned from the source domain. Among them, the most typical method is the domain adaptive extreme learning machine (DAELM), which extends ELM to the transfer learning environment to solve the drift problem of the electric nose system in the sensor field. The subsequent online domain adaptive extreme learning machine (ODAELM) and online weighted domain transfer extreme learning machine (OWDTELM) extend DAELM to online tasks. In order to further improve DAELM, Xia et al. proposed Boosting-DAELMMOX, which integrates Boosting technology with DAELM to improve the model's knowledge transfer ability and classification performance;

[0005] 2) Output weight transformation or approximation: This method achieves cross-domain knowledge transfer by aligning or approximating output weights, such as the extreme learning machine with output weight alignment (TELM-OWA), parameter transfer ELM (PTELM), and extreme learning machine (ELM)-based domain adaptation (EDA) models.

[0006] 3) Distribution Statistic Adjustment: This type of method usually introduces statistical distribution metrics (such as MMD) into ELM to reduce domain shift. Many methods, including cross-domain extreme learning machine (CdELM), maximum weighted mean difference based extreme learning machine (ELM-MWMD), and domain space transfer ELM (DST-ELM), have applied MMD to reduce the distribution difference of output data in the hidden layer of the source and target domains.

[0007] Among the above methods, the target sample supervision method has the highest performance, but it requires labeled samples in the target domain and cannot adapt to the unsupervised learning environment. The output weight transformation or approximation method lacks the mining of data statistical distribution information, and the distribution statistics adjustment method is not as simple and direct as the output weight transformation or approximation method.

[0008] In view of the above, we provide a data classification method based on joint transfer extreme learning machine to solve the above problems. Summary of the Invention

[0009] In response to the above situation, the present invention provides a data classification method based on a joint transfer extreme learning machine, which is used to process unsupervised transfer learning tasks without labeled samples, and solves the problem that ELM does not have the ability to transfer knowledge. The method first introduces the cross-domain mean approximation (CDMA) metric into ELM to obtain a transfer ELM model, wherein CDMA can effectively minimize the marginal and conditional distribution differences between the two domains; secondly, the subspace alignment technology is used to align the output weights of the two domains, and the output weight approximation term is added to promote knowledge transfer and obtain another transfer ELM model. Finally, the two transfer ELMs are used to jointly predict labels for the target domain test samples.

[0010] A data classification method based on a joint transfer extreme learning machine, characterized by comprising the following steps:

[0011] S1: Preprocess the source domain and target domain sample data respectively to obtain the source domain and target domain datasets and

[0012] S2: Use source domain and target domain datasets D respectively S With D T Construct CDMA metric and add it to the hidden layer of extreme learning machine for model training, construct ELM model with CDMA, and use D S With D T Calculate the model parameter β1 of the migration extreme learning machine ELM-CDMA;

[0013] S3: Construct an ELM model with output weight alignment and parameter approximation by aligning β1 and the target domain output weight β T, so that the output weights between the source domain and the target domain are close to each other in the process of cross-domain knowledge transfer, and the intermediate weight β after β1 is aligned to the target domain is obtained temp ;

[0014] With the help of β temp , and approximate the regularization term through the source domain empirical risk error and the output weight between domains, construct a new transfer extreme learning machine model objective function, and obtain its output weight β2;

[0015] S4: Use β1 and β2 to analyze the target domain dataset D T Make label predictions and get their labels

[0016] S5: Loop through steps S1-S4 until No longer change, output prediction results

[0017] The beneficial effects of the above technical solution are:

[0018] (1) CDMA is introduced into the ELM model to reduce the output distribution difference of the hidden layer in the source domain and the target domain, and a transfer ELM model is designed. This model improves the knowledge transfer ability of the traditional ELM in terms of statistical split adjustment;

[0019] (2) By using output weight alignment and parameter approximation, we design another transfer ELM model. This allows our invented method JTELM to unify statistical split adjustment and parameter conversion into the same learning framework, which is more conducive to improving the performance of ELM in the transfer learning environment;

[0020] (3) JTELM uses β and β2 to jointly make label decisions for test samples, which not only helps improve its classification accuracy but also enhances its robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a block diagram of the JTELM system of the present invention;

[0022] Figure 2 The images are from Office31+Caltech256 of the present invention;

[0023] Figure 3 These are the images in MNIST+USPS of the present invention;

[0024] Figure 4 These are the images in MSRC+VOC2007 of the present invention;

[0025] Figure 5 This is a graph showing the relationship between the values ​​of parameters α1 and α2 and the classification accuracy of the present invention;

[0026] Figure 6 This is a graph showing the relationship between the values ​​of the parameters γ and λ and the classification accuracy of the present invention;

[0027] Figure 7 This is a graph showing the corresponding relationship between the number of hidden layer nodes (L), the number of iterations, and the classification accuracy of the present invention. DETAILED DESCRIPTION

[0028] The above and other technical contents, features and effects of the present invention are described below with reference to the attached Figures 1 to 7 It can be clearly presented in the detailed description of the embodiments that the structural contents mentioned in the following embodiments are all referenced to the drawings in the specification.

[0029] This paper proposes a joint transfer extreme learning machine (JTELM) method with cross-domain mean approximation and consistent output weights;

[0030] 1. The specific process is as follows:

[0031] First, the dataset samples are preprocessed;

[0032] Secondly, we introduce Cross-Domain Mean Approximation (CDMA) into ELM to obtain an extreme learning machine with transfer capability, which effectively minimizes the marginal and conditional distribution differences between the two domains.

[0033] Then, another ELM model with output weight alignment and parameter approximation is constructed;

[0034] Apply subspace alignment technology to align the output weights of the two domains, and add approximation terms to approximate the output weights to promote knowledge transfer;

[0035] Finally, the two learned ELMs are used to perform classification prediction on the target samples.

[0036] The following is combined with Figure 1-7 The technical solution of the present invention is further described in detail

[0037] As attached Figure 1 The figure shows a flow chart of the joint transfer extreme learning machine method with cross-domain mean approximation and consistent output weights, which includes the following steps:

[0038] Step 1: Preprocess the source domain and target domain datasets to obtain the source domain and target domain datasets and

[0039] Step 2: The construction process of the ELM model with CDMA is as follows:

[0040] (1) Combine the activation function to D S and DT Mapped to H S and H T ;

[0041] (2) Introducing CDMA into the loss function of ELM, the loss function of ELM with CDMA is obtained as follows:

[0042]

[0043] Where, the first two terms are the ELM loss, the third term is the CDMA loss at the output layer, and α1 is the trade-off parameter between the two losses. and Is the process of label refinement D T The pseudo label, H (c) Represents H S and H T The samples of category c in H S_av and H T_av Respectively represent H S and H T The mean vector of Is a Class C target (source) domain The mean vector of .

[0044] (3) Order We can obtain an ELM with knowledge transfer capability:

[0045]

[0046] Step 3: The construction process of the ELM model with output weight alignment and parameter approximation is:

[0047] (1) Assume that there is a weight β in the target domain T , construct the following loss function:

[0048]

[0049] Among them, ||H T β T -H av β1|| 2 represents the center loss, Indicates that the output weight is approximate, so that β1 approaches β T , to promote knowledge transfer, λ and γ are balance parameters.

[0050] (2) Using β temp Substituting β1, we get:

[0051]

[0052] (3) According to the transformation matrix make have to:

[0053]

[0054] (4) Because, The loss function in (3) is transformed as:

[0055]

[0056] Let (13) can be simplified as:

[0057]

[0058] (5) Let Then we can get:

[0059]

[0060] Step four: the prediction process based on JTELM is as follows:

[0061] (1) Use ELM with CDMA, and ELM with output weight alignment and parameter approximation. These two transfer ELMs are used to jointly train the weights and (equivalent to the optimal values of β1 and β2);

[0062] (2) Use the trained model to predict the target domain samples D T .

[0063] II. The following demonstrates the specific operation process of the above system:

[0064] First input: data sets D S and D T , balance parameters α1, α2, λ and γ.

[0065] Output: prediction results

[0066] 1: According to formula (9), use D S and D T to calculate

[0067] 2: According to formula (15), calculate with D S ;

[0068] 3: According to formula (14), solve the output weight

[0069] 4: Use and to predict the samples in D T , and get their labels

[0070] 5: Repeat 1-4 until No longer changes.

[0071] 3. The classification performance of this scheme is verified by specific experimental data

[0072] 1. Description of the dataset

[0073] In order to demonstrate the credibility and classification performance of the algorithm, the experiment selected seven commonly used image and text datasets: USPS dataset, MNIST dataset, MSRC dataset, VOC2007 dataset, Office dataset, Caltech dataset, and Reutres-21578 dataset to perform classification tasks;

[0074] (1) Office31+Caltech256 (such as Figure 2 As shown in Figure 2, it contains two domains: Office31 and Caltech256. Office31 contains 4,652 images from 31 categories, which are collected from 3 subdomains: Amazon (A), DSLR (D), and Webcam (W). Caltech (C) is also an object image dataset, including 30,607 images from 256 categories.

[0075] During the experiment, we selected 1410 images of 10 categories from office31 and 1123 images of 10 categories from Caltech. Each image was extracted using 800-dimensional SURF features. Two subdomains of A, W, D, and C were randomly selected as the source and target domain datasets, and 12 cross-domain tasks C→A, C→W, C→D, ..., D→W were constructed (as shown in Table 1).

[0076] Table 1 Description of image and text data

[0077]

[0078] (2) USPS and MNIST are two image datasets describing the numbers 0-9, so they share 10 categories but have different distributions. USPS contains 9298 images of 16×16 pixels and MNIST contains 70,000 images of 28×28 pixels. During the experiment, 1800 images from USPS and 2000 images from MNIST were randomly selected as the source domain and target domain (as shown in Table 1). Each image was converted to 16×16 pixels, and two cross-domain tasks were constructed for the transfer learning task, namely USPS vs MNIST and MNIST vs USPS.

[0079] (3)MSRC+VOC2007 (such as Figure 4 As shown in the figure): MSRC is an object image dataset consisting of 4323 images from 18 categories, and VOC2007 is an image dataset containing photos from Flickr, containing 5011 images from 18 categories. Figure 4 Similar but different distributions, in this experiment, we collect samples from the shared 6 categories of the two datasets, including airplanes, birds, cows, household cars, sheep, bicycles, etc. Then, we construct two transfer learning tasks: MSRC vs VOC and VOC vs MSRC, where 1269 images are selected from MSRC and 1530 images are selected from VOC2007. In addition, we rescale the length of all images to 256 grayscale pixels and extract 240 dimensions as new feature representations (as shown in Table 1).

[0080] (4) Reuters-21578: Reuters-21578 is a text dataset commonly used for text data mining and analysis. It has 21,577 news documents from five categories: “exchanges”, “orgs”, “people”, “places”, and “topics”. In this experiment, we selected the three largest categories: “orgs”, “people”, and “place”, and constructed six transfer learning tasks, namely, orgs vs people, people vs orgs, orgs vs place, place vs orgs, people vs place, and place vs people, as shown in Table 1.

[0081] 2. Experimental setup process:

[0082] First, we select some classifiers to compare with JTELM as follows:

[0083] 1NN: a nearest neighbor classifier;

[0084] SVM: Support vector machine with linear kernel and penalty parameter;

[0085] ELM: Standard Extreme Learning Machine;

[0086] SSELM: Semi-supervised ELM with graph regularization.

[0087] TCA1(2): TCA1 represents TCA+1NN, and TCA2 represents TCA+SVM;

[0088] JDA1(2): JDA1 represents JDA+1NN, and JDA2 represents JDA+SVM;

[0089] DAELM_S, DAELM_T: domain-adapted ELM;

[0090] AELM: ELM with feature enhancement (AELM);

[0091] ARRLS: A general transfer learning framework.

[0092] TELM-OWA: Supervisory Transmission ELM;

[0093] CdELM-C: Unsupervised Transfer ELM using MMD.

[0094] We set the penalty parameter λ∈[0.001, 0.1] in ELM, SSELM, DAELM_S, DAELM_T and TELM-OWA, and α1∈[10 -2 , 10 -1 ],α2∈[10 -4 , 10 -1 ],γ∈[10 -2 , 10 -1 ], L = 2000 and λ = 10 -4 , set L = 3000 and λ = 10 on USPS+MNIST -4 , set L = 3000 and λ = 10 on the Reuters-21578 dataset 3 and MSRC+VOC2007 datasets with L=1000 and λ=10 -1 ;

[0095] To evaluate DAELM_S, DAELM_T, and TELM-OWA in unsupervised transfer learning experiments, we selected a small number of target samples, such as 0.5% labeled target samples on the USPS+MNIST, MSRC+VOC2007, and Reuters-21578 datasets and 1% labeled target samples on the Office+Caltech dataset for training models;

[0096] 3. Experimental results and analysis

[0097] To study the performance of JTELM, we conducted classification tasks on image and text datasets including Office+Caltech, USPS+MNIST, MSRC+VOC2007, and Reuters-21578 datasets. The results are shown in Tables 2 and 3. We can see that:

[0098] (1) The overall average accuracy of JTELM is the highest among all algorithms in Table 2 and Table 3, which is improved by 8.48% and 8.73% compared with the baseline ELM in Table 2 and Table 3, respectively, which shows that our method has better knowledge transfer ability with the help of CDMA, output weight alignment and weight approximation, which enriches the theory and application of ELM.

[0099] (2) TELM-OWA and DAELM as supervised transfer learning that needs part of labeled target samples are not ideal under unsupervised learning, TCA, JDA, ARRLS and CdELM-C apply MMD to reduce the distribution difference of two domains and obtain good results, SSELM uses graph regularization to explore the information of unlabeled target samples and performs well.

[0100] (3) TCA1(2) and JDA1(2) combine the transfer feature extraction method (TCA, JDA) with the baseline classifier to realize the classification task, so they are better than 1NN and SVM.

[0101]

[0102]

[0103] Table 2. Accuracy of different algorithms on USPS+MNIST and Office+Caltech datasets

[0104]

[0105] Table 3. Accuracy of different algorithms on MSRC+VOC2007 and Reuters-21578 datasets

[0106] We study the sensitivity of JTELM to parameters α1, α2, γ, λ and the number of hidden layer nodes (L) and its convergence, and conduct experiments on org vs people, MSRC vs VOC, MNIST vs USPS, A vs D datasets, and the results are in Figures 5 to 7 The following observations are:

[0107] 1) With the increase of α1, α2, γ and λ, the accuracy of JTELM on the four datasets first increases and then decreases, as shown in Figures 5 to 6 , which shows that CDMA, error and output weight approximation are adjusted to an appropriate range, which can improve the accuracy and knowledge transfer ability of ELM in transfer learning.

[0108] 2) As Figure 7As shown in (e), as the number of L increases, on the four datasets, the accuracy first increases and then decreases slightly. When it increases, the nonlinear approximation of our network will perform well.

[0109] 3)Accuracy Figure 7 As shown in (f), the accuracy of JTELM gradually stabilizes and finally converges after 10 iterations, indicating that JTELM has strong robustness.

[0110] The above is only for illustrating the present invention. It should be understood that the present invention is not limited to the above embodiments, and various variations that conform to the concept of the present invention are within the scope of protection of the present invention.

Claims

1. A data classification method based on a joint transfer extreme learning machine, characterized in that: The following steps are involved: S1: Preprocess the source domain and target domain sample data separately to obtain the source domain dataset D S = and target domain dataset Among them, y Si is the source domain sample x Si The corresponding label, n S is the number of source domain samples; y Tj is the target domain sample x Tj The corresponding label, n T is the number of samples in the target domain; S2: Use the source domain dataset D S and the target domain dataset D T Construct a cross-domain mean approximation metric, namely CDMA metric, and add it to the hidden layer of the extreme learning machine, namely ELM, for model training. Construct an ELM model with CDMA, denoted as ELM-CDMA model, and use D S With D T Calculate the model parameter β1 of the migration extreme learning machine ELM-CDMA; S3: Construct an ELM model with output weight alignment and parameter approximation by aligning β1 and the target domain output weight β T , so that the output weights between the source domain and the target domain are close to each other in the process of cross-domain knowledge transfer, and the intermediate weight β after β1 is aligned to the target domain is obtained temp ; With the help of β temp , and approximate the regularization term through the source domain empirical risk error and the output weight between domains, construct a new transfer extreme learning machine model objective function, and obtain its output weight β2; The construction of the ELM model with output weight alignment and parameter approximation described in step S3 includes the following steps: S3-1: Assume that there is a weight β in the target domain T , construct the following loss function: in, Indicates that the output weight is approximated, forcing β1 to be close to β T To promote knowledge transfer, a2 and γ are balance parameters; S3-2: Output weight β by aligning β1 and target domain T , and obtain a transformation matrix M*=β1 T β T , let β temp =β1M*=β1β1 T β T , then replace β1 with β temp , substituting into formula (1) we get: β temp =β1β1 T β T Substituting into formula (2) we get: Because ||(I-β1β1 T )β T || 2 ≤||(I-β1β1 T )|| 2 ||β T || 2 , then formula (3) is transformed as follows: Let A = I + γ (I - β1β1 T ) T (I-β1β1 T ), H temp =H S β1β1 T , formula (4) can be simplified as: let have to: S4: Use β1 and β2 to analyze the target domain dataset D T Perform label prediction and obtain its predicted label S5: Loop through steps S1-S4 until No longer change, output prediction results 2. The data classification method based on the joint transfer extreme learning machine according to claim 1, characterized in that: The preprocessing of the source domain and target domain sample data in step S1 includes the following steps: S1-1: Source domain symbols Indicates that x Sci represents the source domain sample, y Sci Represents the source domain sample x Sci The corresponding label, n Sc is the number of source domain samples; S1-2: Target domain symbols Indicates that x Tcj represents the target domain sample, y Tcj Represents the target domain sample x Tcj The corresponding label, n Tc is the number of samples in the target domain; S1-3: D Sc With D Tc Normalize them separately and get D S To and D T ; The construction process of the ELM model with CDMA in step S2 includes the following steps: S2-1: Combine the activation function to D S and D T Mapping to H S and H T ; S2-2: Assume there are N samples x i and the corresponding label y i Labeled dataset Construct an ELM model with L hidden layer nodes as follows: ELM is based on the input sample x i Output i , w j and b j are the input weights and biases, β=[δ1,δ2,...,δ j ,...,δ L ] T ; In order to obtain the optimal β, solve the following loss function: Among them, ||β|| 2 is a parameter sparse constraint to avoid overfitting of the model; convert equation (8) into matrix form: in, H=[g(x1) T ,g(x2) T ,…,g(x N ) T ] T , According to formula (9), the optimal β Finally, ELM predicts the test sample x using the following formula Te Tags, Among them, h Te =g(x Te ); S2-3: Construct the objective function of CDMA as follows: in, μ S is the mean vector of the source domain sample data, μ T is the average vector of the target domain sample data; further considering the label information of the sample, CDMA is expressed as: in, is a sample dataset with category c in the source domain, yes The mean vector of is a sample dataset with category c in the target domain, yes The mean vector of S2-4: Introduce CDMA into the ELM loss function to reduce the distribution difference of the output layer data in the source domain and the target domain, and obtain the ELM+CDMA objective function: In formula (14), the first two terms are the ELM loss, the third term is the CDMA loss at the output layer, and α1 is the trade-off parameter between the two losses. During the label refinement process, It's D T The predicted label, H (c) Indicates H S and H T There are samples of category c in H S_av , H T_av They are H S and H T The mean vector of yes The c-class mean vector, yes The c-class mean vector of ; make Acquire ELM with knowledge transfer capabilities 3. The data classification method based on the joint transfer extreme learning machine according to claim 1, characterized in that: In step S4, β1 and β2 are used to compare the target domain dataset D T The process of label prediction is as follows: S4-1: Using ELM with CDMA and ELM with output weight alignment and parameter approximation, these two migration ELMs jointly train weights β1 and β2; S4-2: Using the trained model, the target domain dataset D T Make predictions.

Images