An unsupervised pre-adaptive image classification method based on manifold learning

Abstract

The invention discloses an unsupervised pre-adaptive image classification method based on manifold learning. The method aims to construct an image classification model, and then apply the image classification model to complete the classification of images to be classified. The method comprises the following steps: 1. constructing an image sample set; 2. constructing a feature extraction network; 3. constructing a feature vector dimension reduction network based on the manifold learning network; 4. constructing a classifier network based on a high-order information matching network; 5. constructing a discriminative loss model of source domain image samples, a clustering loss model of target domain image samples, a manifold consistency loss model, and a high-order tensor matching loss model; 6. introducing cross entropy loss to obtain a final loss model; 7. training an image classification model to be trained to obtain an image classification model; the method combines the manifold learning network and the high-order information matching network, and the extracted image features have not only good migration, but also strong discriminability, and have higher classification accuracy.

Description

Technical Field

[0001] The invention relates to an image classification method, and in particular to an unsupervised pre-adaptive image classification method based on manifold learning. Background Art

[0002] In image classification tasks, there are large differences in lighting conditions, image resolution, background color, etc. between image samples collected in laboratory environments and those collected in actual application environments. If the model trained in the laboratory is directly applied to real scenes, the detection effect will be significantly reduced due to the existence of domain differences.

[0003] The task of unsupervised domain adaptation is to learn the knowledge in the labeled source domain data and transfer it to the unlabeled target domain data to complete the task in the target domain. Many existing methods use deep neural networks to extract features with better transferability and reduce domain differences. In addition, there are some methods that use adversarial training to enable the feature extractor to extract domain-invariant features through the confrontation between the discriminator and the feature extractor. However, the above two methods will lead to reduced discriminability of features, and the high-dimensional features extracted by the deep neural network are redundant with the data itself. The present invention combines the manifold learning network with the high-order information matching network, so that the features of the image samples are both discriminative and domain-invariant, which can better classify the image samples in the target domain. Summary of the invention

[0004] The purpose of the present invention is to provide an unsupervised pre-adaptive image classification method based on manifold learning. This method combines a manifold learning network and a high-order information matching network. The extracted image features not only have good transferability, but also have strong discriminability and higher classification accuracy.

[0005] To achieve the above objectives, the present invention designs an unsupervised pre-adaptive image classification method based on manifold learning, which is characterized by: according to the following steps S1 to S7, an image classification model is obtained, and then the image classification model is applied to complete the classification of the image to be classified;

[0006] S1. Based on the preset illumination intensity, resolution, and background color, preset images of various types of objects are collected as source domain image samples, and each source domain image sample constitutes a source domain image sample set. Based on other illumination intensities, resolutions, and background colors, corresponding images of various objects are collected as target domain image samples, and each target domain image sample constitutes a target domain image sample set. According to the types of various types of objects and the preset mapping rules, the corresponding categories in the preset classifications corresponding to the source domain image samples and the target domain image samples of various types of objects are defined, and the categories corresponding to the source domain image samples and the target domain image samples of the same type of objects are the same;

[0007] S2. Based on the ResNet-50 network, a feature extraction network is constructed with image samples as input and high-dimensional feature vectors of image samples as output;

[0008] S3. Based on the manifold learning network, the high-dimensional feature vector of the image sample is used as input and the high-order statistical information of the image sample is used as output to construct a feature vector dimensionality reduction network;

[0009] S4. Based on the high-order information matching network, the high-order statistical information of the image sample is used as input and the image category corresponding to the image sample is used as output to construct a classifier network;

[0010] S5. Based on the feature extraction network, feature vector dimensionality reduction network, and classifier network, the image classification model to be trained is constructed in series. At the same time, based on the participation of the source domain image sample set and the target domain image sample set in the training, the discriminative loss model L of the source domain image sample is constructed with the output of the feature vector dimensionality reduction network. dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons , and construct a high-order tensor matching loss model L based on the output of the classifier network hom ;

[0011] S6. Discriminative loss model L based on source domain image samples dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons , high-order tensor matching loss model L hom , by introducing the cross entropy loss L ce , obtain the final loss model L;

[0012] S7. Based on the sample images in the source domain image sample set and the target domain image sample set, and each sample image corresponds to the corresponding category in the preset classification, with the image sample as input and the corresponding category corresponding to the image sample as output, combined with the final loss model L, the image classification model to be trained is trained to obtain the image classification model.

[0013] As a preferred technical solution of the invention: the discriminative loss model L of the source domain image sample in step S5 dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons The construction includes the following steps:

[0014] S51: Based on the high-dimensional feature vector f of each source domain image sample on each layer of manifold learning network i s, through source domain discriminative structure learning, the discriminative loss model L of source domain image samples is obtained dis As follows:

[0015]

[0016] Where k is the number of manifold layers, l is the number of layers of each manifold layer, i and j are the subscripts of traversal, is the similarity between the i-th category and the j-th category in the source domain image sample in the l-th manifold layer. The number of categories of the source domain image samples and the target domain image samples are both n. c ;

[0017] S52: Center matrix based on source domain image samples and the feature matrix of the target domain image sample Through target domain discriminative learning, the clustering loss model L of the target domain image samples is obtained cluster As follows:

[0018]

[0019] In the formula, the components of the soft labels of the target domain image samples are arranged from large to small according to their values, the first m components are retained, and the other components are set to 0. is the i-th component of the soft label of the j-th target domain image sample in the retained component, is the similarity between the i-th category and the j-th category in the target domain image sample in the l-th manifold layer, n t is the number of target domain image samples;

[0020] S53: The covariance matrix calculated based on the source domain image sample on the lth manifold layer through the source domain features The covariance matrix of the target domain image sample calculated on the lth manifold layer using the target domain features is Through manifold consistency structure learning, we obtain the manifold consistency loss model L cons As follows:

[0021]

[0022] In the formula, is the dimension of the lth manifold layer, is the orthogonal projection matrix of the source domain image samples in the lth manifold layer, For the matrix The transposed matrix of is the orthogonal projection matrix of the target domain image sample in the lth manifold layer, For the matrix The transposed matrix of .

[0023] As a preferred technical solution of the invention: in step S5, a high-order tensor matching loss model L is constructed. hom As follows:

[0024]

[0025] In the formula, represents the tensor product, is the third-order tensor product of the sample features of the i-th source domain image in the l-th fully connected layer, is the third-order tensor product of the j-th target domain image sample features in the l-th fully connected layer, n s is the number of source domain image samples, n t is the number of target domain image samples.

[0026] As a preferred technical solution of the invention: a discriminative loss model L based on source domain image samples dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons , high-order tensor matching loss model L hom , by introducing the cross entropy loss L ce , construct the final loss model L as follows:

[0027] L=L dis -αL cluster +βL cons +γL hom +ηL ce

[0028] Where α, β, γ, and η are hyperparameters.

[0029] As a preferred technical solution of the invention: the manifold learning network has two layers.

[0030] Beneficial effects: Compared with the prior art, the advantages of the present invention include: the present invention combines the manifold learning network and the high-order information matching network, so that the features of the image samples are both discriminative and domain invariant, and can better classify the image samples in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a framework diagram of an unsupervised pre-adaptive image classification method based on manifold learning provided according to an embodiment of the present invention;

[0032] Figure 2 It is a model diagram of an unsupervised pre-adaptive image classification method based on manifold learning provided in an embodiment of the present invention;

[0033] Figure 3It is a high-order information matching network diagram of an unsupervised pre-adaptive image classification method based on manifold learning provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0035] Reference Figure 1 , an unsupervised pre-adaptive image classification method based on manifold learning provided by an embodiment of the present invention obtains an image classification model according to the following steps S1 to S7, and then applies the image classification model to complete the classification of the image to be classified;

[0036] S1. Based on the preset illumination intensity, resolution, and background color, preset images of various types of objects are collected as source domain image samples, and each source domain image sample constitutes a source domain image sample set. Based on other illumination intensities, resolutions, and background colors, corresponding images of various objects are collected as target domain image samples, and each target domain image sample constitutes a target domain image sample set. According to the types of various types of objects and the preset mapping rules, the corresponding categories in the preset classifications corresponding to the source domain image samples and the target domain image samples of various types of objects are defined, and the categories corresponding to the source domain image samples and the target domain image samples of the same type of objects are the same. In one embodiment, a normalization method is used to preprocess the source domain image samples and the target domain image samples.

[0037] S2. Based on the ResNet-50 network, the image samples are used as input and the high-dimensional feature vectors of the image samples are used as output to construct a feature extraction network; in one embodiment, in order to avoid the long preparation time required to train the feature extraction network from scratch, a pre-trained ResNet-50 network M1 is used to extract high-dimensional feature vectors of source domain image samples and target domain image samples;

[0038] S3. Based on the manifold learning network M2, a feature vector dimensionality reduction network is constructed with the high-dimensional feature vector of the image sample as input and the high-order statistical information of the image sample as output;

[0039] The manifold learning network can not only transform the high-dimensional feature vector of the image sample through multiple manifold layers M={(M k |k=1,2,…l)}, reduce the dimension layer by layer to a low dimension, filter the redundant information, and align the structure of the source domain image sample and the target domain image sample by aligning the manifold structure, where M kis a simplified representation of the kth manifold layer; in one embodiment, a manifold learning network is used to align the marginal distribution of source domain image samples and target domain image samples, and the conditional distribution of source domain image samples and target domain image samples, respectively; when specifically deployed in a manifold learning network, two layers of manifolds are used to reduce the dimension of the feature vector of the image sample layer by layer, and the high-dimensional feature vector is reduced in dimension, so as to enhance the discriminability and transferability of the feature vector in the two-layer manifold learning network.

[0040] S4. Based on the high-order information matching network, the high-order statistical information of the image sample is used as input and the image category corresponding to the image sample is used as output to construct a classifier network M3;

[0041] S5. Based on the feature extraction network, feature vector dimensionality reduction network, and classifier network, the image classification model to be trained is constructed in series. At the same time, based on the participation of the source domain image sample set and the target domain image sample set in the training, the discriminative loss model L of the source domain image sample is constructed with the output of the feature vector dimensionality reduction network. dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons , and construct a high-order tensor matching loss model L based on the output of the classifier network hom ;

[0042] refer to Figure 2 , the discriminative loss model L of the source domain image samples dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons The construction includes the following steps:

[0043] S51: Based on the high-dimensional feature vector f of each source domain image sample on each layer of manifold learning network i s , calculate the class mean matrix and the feature mean vector in is the feature mean vector of the i-th category of the source domain image sample in the l-th manifold layer, d l is the dimension of the lth manifold layer, and the number of categories of source domain image samples and target domain image samples is n c , Indicates dimension d l ×n c ;

[0044] Feature mean vector As follows:

[0045]

[0046] In the formula Indicates dimension d l , n s is the number of source domain image samples, is the high-dimensional feature vector of the i-th source domain image sample on the l-th manifold layer;

[0047] Then adjust the class mean matrix to In the formula is a matrix of all 1s of the same dimension as the class mean matrix, is the center matrix of the source domain image samples, and calculates the similarity matrix between the source domain image samples Finally, the discriminative loss model L of the source domain image sample is obtained dis As follows:

[0048]

[0049] Where k is the number of manifold layers, l is the number of layers of each manifold layer, i and j are the subscripts of traversal, is the similarity between the i-th category and the j-th category in the source domain image sample in the l-th manifold layer. The number of categories of the source domain image samples and the target domain image samples are both n. c ;

[0050] This step is to enhance the discriminability of source domain image sample features through source domain image sample discriminative structure learning, that is, to bring target domain image samples closer to source domain image samples. Since source domain image samples contain label information, the class center of the source domain image samples is also used as the class center of the target domain image samples, and the discriminability between their features is enhanced by increasing the distance between the class centers of image samples of different item categories.

[0051] S52: Center matrix based on source domain image samples and the feature matrix of the target domain image sample Calculate the similarity matrix of various centers of target domain image samples As follows:

[0052]

[0053] In the formula is the center matrix of the source domain image sample The transposed matrix of

[0054] Then, the prediction information in the soft label of the target domain image sample predicted by the classifier is used as the confidence of each class probability. Combined with the similarity matrix calculation, the target domain image sample is pulled closer to the center of the source domain image sample class to form a cluster, and the clustering loss model L of the target domain image sample is constructed. cluster As follows:

[0055]

[0056] In the formula, the components of the soft labels of the target domain image samples are arranged from large to small according to their values, the first m components are retained, and the other components are set to 0. is the i-th component of the soft label of the j-th target domain image sample in the retained component, is the similarity between the i-th category and the j-th category in the target domain image sample in the l-th manifold layer, n t is the number of target domain image samples;

[0057] This step is target domain discriminative structure learning for enhancing the discriminability of target domain image sample features, that is, moving target domain image samples closer to their corresponding source domain image samples. Specifically, the category corresponding to the largest component in the target domain image sample soft label is used as the class label of the target domain image sample, and the distance between it and the class center corresponding to the source domain image sample is reduced. Finally, the target domain image samples will form clusters around the source domain image sample class center, thereby enhancing the discriminability of the target domain image sample features.

[0058] S53: The covariance matrix calculated based on the source domain image sample on the lth manifold layer through the source domain features The covariance matrix of the target domain image sample calculated on the lth manifold layer using the target domain features is Calculate the covariance matrix and get the following formula according to the singular value decomposition formula:

[0059]

[0060] Where U l is the projective orthogonal basis, is the matrix obtained by singular value decomposition;

[0061] Based on the projective orthogonal basis U l , construct the manifold consistency loss model L cons As follows:

[0062]

[0063] In the formula, is the dimension of the lth manifold layer, is the orthogonal projection matrix of the source domain image samples in the lth manifold layer, For the matrix The transposed matrix of is the orthogonal projection matrix of the target domain image sample in the lth manifold layer, For the matrix The transposed matrix of .

[0064] refer to Figure 3, based on the high-order statistical information in the source domain image samples and the target domain image samples, fine-grained domain alignment is performed to construct a high-order tensor matching loss model L hom As follows:

[0065]

[0066] In the formula, represents the tensor product, is the third-order tensor product of the sample features of the i-th source domain image in the l-th fully connected layer, is the third-order tensor product of the j-th target domain image sample features in the l-th fully connected layer, n s is the number of source domain image samples, n t is the number of target domain image samples.

[0067] S6. Discriminative loss model L based on source domain image samples dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons , high-order tensor matching loss model L hom , by introducing the cross entropy loss L ce , obtain the final loss model L;

[0068] The final loss model L is constructed as follows:

[0069] L=L dis -αL cluster +βL cons +γL hom +ηL ce

[0070] In the formula, α, β, γ, and η are hyperparameters;

[0071] S7. Based on the sample images in the source domain image sample set and the target domain image sample set, and each sample image corresponds to the corresponding category in the preset classification, with the image sample as input and the corresponding category corresponding to the image sample as output, combined with the final loss model L, the image classification model to be trained is trained to obtain the image classification model.

[0072] The parameters of the image classification model are updated through directional propagation, and the final loss model L is minimized to improve the classification accuracy of the image classification model for target domain image samples.

[0073] An embodiment of the present invention provides an unsupervised pre-adaptive image classification method based on manifold learning, wherein the manifold learning network has two layers.

[0074] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An unsupervised pre-adaptive image classification method based on manifold learning, characterized in that: According to the following steps S1 to S7, an image classification model is obtained, and then the image classification model is applied to complete the classification of the image to be classified; S1. Based on the preset illumination intensity, resolution, and background color, preset images of various types of objects are collected as source domain image samples, and each source domain image sample constitutes a source domain image sample set. Based on other illumination intensities, resolutions, and background colors, corresponding images of various objects are collected as target domain image samples, and each target domain image sample constitutes a target domain image sample set. According to the types of various types of objects and the preset mapping rules, the corresponding categories in the preset classifications corresponding to the source domain image samples and the target domain image samples of various types of objects are defined, and the categories corresponding to the source domain image samples and the target domain image samples of the same type of objects are the same; S2. Based on the ResNet-50 network, a feature extraction network is constructed with image samples as input and high-dimensional feature vectors of image samples as output; S3. Based on the manifold learning network, the high-dimensional feature vector of the image sample is used as input and the high-order statistical information of the image sample is used as output to construct a feature vector dimensionality reduction network; S4. Based on the high-order information matching network, the high-order statistical information of the image sample is used as input and the image category corresponding to the image sample is used as output to construct a classifier network; S5. Based on the feature extraction network, feature vector dimensionality reduction network, and classifier network, the image classification model to be trained is constructed in series. At the same time, based on the participation of the source domain image sample set and the target domain image sample set in the training, the discriminative loss model L of the source domain image sample is constructed with the output of the feature vector dimensionality reduction network. dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons , and construct a high-order tensor matching loss model L based on the output of the classifier network hom ; S6. Discriminative loss model L based on source domain image samples dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons , high-order tensor matching loss model L hom , by introducing the cross entropy loss L ce , obtain the final loss model L; S7. Based on the sample images in the source domain image sample set and the target domain image sample set, and each sample image corresponds to the corresponding category in the preset classification, with the image sample as input and the corresponding category corresponding to the image sample as output, combined with the final loss model L, the image classification model to be trained is trained to obtain the image classification model.

2. The unsupervised pre-adaptive image classification method based on manifold learning as claimed in claim 1, characterized in that: The discriminative loss model L of the source domain image sample in step S5 dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons The construction includes the following steps: S51: Based on the high-dimensional feature vector f of each source domain image sample on each layer of manifold learning network i s , through source domain discriminative structure learning, the discriminative loss model L of source domain image samples is obtained dis As follows: Where k is the number of manifold layers, l is the number of layers of each manifold layer, i and j are the subscripts of traversal, is the similarity between the i-th category and the j-th category in the source domain image sample in the l-th manifold layer. The number of categories of the source domain image samples and the target domain image samples are both n. c ; S52: Center matrix based on source domain image samples and the feature matrix of the target domain image sample Through target domain discriminative learning, the clustering loss model L of the target domain image samples is obtained cluster As follows: In the formula, the components of the soft labels of the target domain image samples are arranged from large to small according to their values, the first m components are retained, and the other components are set to 0. is the i-th component of the soft label of the j-th target domain image sample in the retained component, is the similarity between the i-th category and the j-th category in the target domain image sample in the l-th manifold layer, n t is the number of target domain image samples; S53: The covariance matrix calculated based on the source domain image sample on the lth manifold layer through the source domain features The covariance matrix of the target domain image sample calculated on the lth manifold layer using the target domain features is Through manifold consistency structure learning, we obtain the manifold consistency loss model L cons As follows: In the formula, is the dimension of the lth manifold layer, is the orthogonal projection matrix of the source domain image samples in the lth manifold layer, For the matrix The transposed matrix of is the orthogonal projection matrix of the target domain image sample in the lth manifold layer, For the matrix The transposed matrix of .

3. The unsupervised pre-adaptive image classification method based on manifold learning as claimed in claim 1, characterized in that: In step S5, a high-order tensor matching loss model L is constructed. hom As follows: In the formula, represents the tensor product, is the third-order tensor product of the sample features of the i-th source domain image in the l-th fully connected layer, is the third-order tensor product of the j-th target domain image sample features in the l-th fully connected layer, n s is the number of source domain image samples, n t is the number of target domain image samples.

4. The unsupervised pre-adaptive image classification method based on manifold learning as claimed in claim 1, characterized in that: Discriminative loss model L based on source domain image samples dis , clustering loss model L of target domain image samples cluster , manifold consistency loss model L cons , high-order tensor matching loss model L hom , by introducing the cross entropy loss L ce , construct the final loss model L as follows: L=L dis -aL cluster +βL cons +γL hom +ηL ce Where α, β, γ, and η are hyperparameters.

5. The unsupervised pre-adaptive image classification method based on manifold learning as claimed in claim 1, characterized in that: The manifold learning network has two layers.

Images