Classification algorithm based on open set recognition of cultural relics images

Abstract

The present invention discloses a classification algorithm based on open set recognition of cultural relic images, and proposes a discriminative feature extraction algorithm based on contrastive learning, which can help the network to extract richer semantic and structural information by using the enhanced information of cultural relic images, and enhance the feature extraction ability of the network in the training stage. In addition, through the open set recognition algorithm of cultural relic images based on feature transfer prior error, the difference between the features extracted by the model before and after transfer learning is used to further enhance the network's judgment ability on known class cultural relic samples and open set cultural relic samples in the testing stage. The experimental results on the open set recognition task of cultural relic show that compared with the existing open set recognition algorithm, the algorithm proposed by the present invention effectively improves the open set recognition ability of the network.

Description

Technical Field

[0001] The present invention relates to the field of cultural relic image classification in computer vision, and more specifically, to a classification algorithm for open set recognition of cultural relic images based on contrastive discriminative feature extraction. Background Art

[0002] In recent years, deep learning algorithms have achieved relatively good results in the classification of cultural relics images. However, the existing cultural relics image classification algorithms are generally trained and tested under closed set conditions, that is, the cultural relics categories in the test set are all those that appear in the training set, and do not involve cultural relics categories that have not appeared in the training set. However, in real deployment environments, such as cultural relics exploration, online museums, etc., unknown cultural relics categories that have never appeared in the training set may appear in these scenarios, which greatly weakens the robustness of existing methods and reduces the performance of existing cultural relics image classification algorithms. In addition, the existing cultural relics image classification algorithms need to rely on a large amount of cultural relics data to train the network. However, due to the rich variety of cultural relics data, the data annotation process has high requirements for professional knowledge in the field of cultural relics, so it is difficult to construct a large amount of supervised cultural relics data. Summary of the invention

[0003] In order to solve the problems of the prior art, the present invention provides a classification algorithm based on open set recognition of cultural relic images, which solves the problem that the existing cultural relic image classification algorithm only considers classification under closed set conditions and the cultural relic data is insufficient.

[0004] The present invention proposes the following technical solutions:

[0005] The method of the present invention is based on the application of convolutional neural networks in the field of cultural relic image classification. In view of the fact that existing convolutional neural networks cannot effectively extract the discriminative features required to distinguish between known category cultural relic samples and unknown category cultural relic samples, a contrastive learning method is used in the training stage to improve the network's ability to extract discriminative features. In the testing stage, a feature transfer error prior method is proposed to further improve the network's open set recognition ability of cultural relic images.

[0006] 1. Discriminative feature extraction algorithm based on contrastive learning

[0007] In the existing open-set identification algorithms for cultural relics, since the training set only contains samples of cultural relics of known categories, the network tends to extract only the features required to distinguish cultural relics of known categories. However, these features may not be sufficient to capture the differences between samples of known categories and samples of open-set cultural relics. The contrastive learning algorithm randomly selects N cultural relic images from the training set to form a batch. For a certain image x in a batch, two enhanced images are obtained through certain data augmentation operations. and Therefore, there are 2N enhanced images in a batch. The enhanced images are passed through the encoder and mapper to obtain the corresponding feature vector z i and z j Through the constraint of supervised contrast loss, for cultural relics images of the same category within a batch, the distance in their corresponding feature space is shortened; for cultural relics images of different categories within a batch, the distance in their corresponding feature space is distanced away.

[0008] The supervised contrastive learning loss function is defined as follows:

[0009]

[0010]

[0011] Among them, N is the total number of images in a batch, i represents the anchor point, Indicates that a batch has the same label as anchor i The total number of images, 1 i≠j ∈{0, 1} is an indicator function that returns 1 if and only if i≠j, otherwise it returns 0; Represents the enhanced image The feature vector obtained by the encoder E(·) and the projector P(·); z i ·z j Indicates z i and z j The inner product operation between them is used to calculate the distance between two features; τ>0 is a scalar adjustment coefficient, which is set to 0.07 in the algorithm of this paper.

[0012] Since the contrastive learning algorithm obtains two different enhanced images by performing data augmentation operations on a cultural relic image, and makes the feature spaces of the enhanced cultural relic images of the same category close to each other, and the feature spaces of cultural relic images of different categories far away from each other, the contrastive learning algorithm forces the network to further learn the inherently consistent structural information of the cultural relic images, enhances the network's feature extraction ability, and achieves the effect that the feature spaces of cultural relic images of the same category are close to each other, and the feature spaces of cultural relic images of different categories are far away from each other, so that the network can better distinguish between cultural relic samples of known categories and cultural relic samples of unknown categories.

[0013] After the encoder and mapper are trained by supervised contrast loss, the algorithm in this paper transfers the trained encoder to the classification network in the next step. A fully connected layer is added after the encoder, and the model is trained for transfer learning using the classification cross entropy loss. The number of neurons in the fully connected layer is the number of known cultural relics categories corresponding to the classification.

[0014] 2. Open set identification algorithm of cultural relics based on feature migration prior error

[0015] Through the discriminative feature extraction algorithm based on contrastive learning, the network can extract richer discriminative features of cultural relic images, thereby better distinguishing cultural relic samples of known categories from those of unknown categories. The algorithm in this paper takes into account that since the training data used before and after transfer learning is the same, a feature transfer error prior algorithm is proposed in the test phase, which uses the difference between the multi-layer features extracted in the encoder model before and after transfer learning to correct the activation vector value output by the fully connected layer.

[0016] For samples of known categories of cultural relics, the distance between the features of each layer in the encoder should be small, and for samples of open-set cultural relics, the distance between the features of each layer should be large. The activation feature value output by the classification fully connected layer is divided by the distance value between the features of each layer, so as to achieve the effect of amplifying the activation feature value of the known category cultural relic samples and reducing the activation feature value of the open-set cultural relic samples. Using the corrected activation feature value, a threshold δ is set for the result. When the activation feature value is greater than the threshold δ, the sample is judged to be the corresponding known category cultural relic sample. When the activation feature value is less than the threshold δ, the sample is judged to be an open-set cultural relic sample. The open-set recognition ability of the network's cultural relic images is further improved by an algorithm based on feature migration prior errors. The model expression is as follows:

[0017]

[0018]

[0019]

[0020]

[0021] Where i is the selected i-th layer encoder, cos(·) represents the calculation of the cosine distance between two feature vectors, x represents the input image, E(·) represents the features of the encoder output of the corresponding layer after contrastive learning training, E(·) represents the features of the encoder output of the corresponding layer after transfer learning training, f(x) represents the activation feature value output by the classification network, score(x) represents the corrected activation feature value, and δ is the set threshold.

[0022] The above-mentioned open set recognition algorithm for cultural relic images improves the classification accuracy of cultural relic images, which not only solves the shortcoming of low accuracy of existing closed set classification algorithms for cultural relic images, but also improves the open set recognition capability of cultural relic images.

[0023] Beneficial effects:

[0024] The discriminative feature extraction algorithm based on contrastive learning proposed in the present invention can help the network extract richer semantic and structural information by using the enhanced information of cultural relic images, enhance the network's feature extraction ability in the training stage, and further improve the network's ability to judge known class cultural relic samples and open set cultural relic samples in the testing stage by using the difference between the features extracted by the model before and after transfer learning through the open set recognition algorithm of cultural relic images based on feature transfer prior errors. The experimental results on the open set recognition task of cultural relic show that compared with the existing open set recognition algorithm, the algorithm proposed in the present invention effectively improves the open set recognition ability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The network structure diagram of the algorithm training phase of the present invention;

[0026] Figure 2 Network structure diagram of the algorithm testing phase of the present invention. DETAILED DESCRIPTION

[0027] The open set recognition task of cultural relic images assumes that the categories of cultural relics appearing in the training set are incomplete, and that samples of cultural relics of unknown categories that have not appeared in the training set will appear in the test set. Under this premise, the network is trained using a training set that only contains samples of cultural relics of known categories, but during testing, the network can correctly classify cultural relics of known categories and identify cultural relics categories that have not appeared in the training set as "unknown". Therefore, this paper first constructs the CIFAR-10 dataset into a dataset suitable for open set recognition problems. The present invention randomly selects 6 categories in CIFAR-10 as known categories and the remaining 4 categories as unknown categories.

[0028] When using supervised contrast loss to train the encoder and mapper, it is necessary to perform data augmentation on the input image to obtain the enhanced image. The algorithm of the present invention uses random cropping, random rotation, grayscale, Gaussian blur and other methods to enhance the image. The encoder of the network is composed of a stack of 3×3 convolutions, and each convolution layer is followed by batch normalization and a Leaky ReLU activation function with a slope of 0.2. After the enhanced image passes through the encoder, a 2048-dimensional feature vector is output. The mapper consists of two fully connected layers. The feature vector output by the encoder is a 128-dimensional feature vector after passing through the mapper. In this training stage, the Adam optimizer is used, and the initial learning rate is 3×10 -3 , every 30 rounds of training, the learning rate is set to 0.1 times the original, and a total of 100 rounds of training are performed.

[0029] After training the encoder and mapper of the network, the trained encoder is transferred to the downstream classification task by using transfer learning. The algorithm of the present invention directly connects a fully connected layer after the encoder, uses the activation feature value output by the fully connected layer, and uses the classification cross entropy loss to train the network. In this training phase, the SGD optimizer is used, the initial learning rate is 0.1, and the learning rate is set to 0.1 times the original for every 30 training rounds, and a total of 100 training rounds are performed. At this point, the training phase of the network is completed.

[0030] In the test phase, an algorithm based on feature transfer prior error is used to correct the activation feature values ​​output by the fully connected layer of the classification network using the distance between the features of the middle layer of the encoder before and after transfer learning. In the algorithm of the present invention, the features of three layers of the encoder are selected to calculate the distance between corresponding features.

[0031] In order to evaluate the performance of the algorithm, the present invention uses AUROC (Area under the ROC curve) as an evaluation indicator. The ROC curve reflects the trend of the sensitivity and accuracy of the model when different thresholds are selected. AUROC is a standard used to measure the quality of a classification model, which represents the size of the area under the ROC curve. A larger AUROC represents better performance.

[0032] The performance of the proposed algorithm in image open set recognition is verified on the CIFAR-10 dataset, and the specific experimental results are shown in Table 1. From the experimental results, it can be seen that compared with the existing algorithms, the proposed algorithm effectively improves the open set recognition effect of the network.

[0033] Table 1

[0034]

[0035] The technical solutions disclosed and proposed by the present invention can be realized by those skilled in the art by referring to the contents of this article and appropriately changing the conditions, routes and other links. Although the methods and preparation techniques of the present invention have been described through preferred embodiments, relevant technicians can obviously modify or re-combine the methods and technical routes described herein without departing from the content, spirit and scope of the present invention to achieve the final preparation technology. It is particularly important to point out that all similar substitutions and modifications are obvious to those skilled in the art, and they are all considered to be included in the spirit, scope and content of the present invention. Matters not covered in the present invention belong to the known technology.

Claims

1. A classification method based on open set recognition of cultural relics images, characterized in that: The following steps are involved: Discriminative feature extraction algorithm based on contrastive learning: In the training stage, the contrastive learning algorithm is used to expand the cultural relics data set through data enhancement to improve the network's ability to extract discriminative features; through the constraint of supervised contrastive learning loss, for cultural relics images of the same category in a batch, the distance between their corresponding feature spaces is shortened; for cultural relics images of different categories in a batch, the distance between their corresponding feature spaces is distanced; Then, an open-set cultural relic recognition algorithm based on feature transfer prior error is proposed: in the test stage, a feature transfer error prior method is proposed, and the activation feature value output by the fully connected layer is corrected by using the difference between the multi-layer features extracted in the encoder model before and after transfer learning; the trained encoder is transferred to the classification network in the next step; a fully connected layer is added after the encoder, and the model is trained for transfer learning using the classification cross entropy loss, and the number of neurons in the fully connected layer is the number of known cultural relic categories corresponding to the classification; the activation feature value output by the classification fully connected layer is divided by the distance value between the features of each layer, so as to achieve the effect of amplifying the activation feature value of the known class cultural relic sample and reducing the activation feature value of the open-set cultural relic sample; using the corrected activation feature value, a threshold δ is set for the result, when the activation feature value is greater than the threshold δ, the sample is judged to be a corresponding known class cultural relic sample, and when the activation feature value is less than the threshold δ, the sample is judged to be an open-set cultural relic sample.

2. The classification method based on open set recognition of cultural relics images according to claim 1 is characterized in that: The discriminative feature extraction algorithm based on contrastive learning refers to: By randomly selecting N cultural relics images from the training set to form a batch, for a certain image x in the batch, two enhanced images are obtained through some data enhancement operations. and Therefore, there are 2N enhanced images in a batch; the enhanced images are passed through the encoder and projector to obtain the corresponding feature vector z i and z j .

3. The classification method based on open set recognition of cultural relics images according to claim 2 is characterized in that: The supervised contrastive learning loss function is defined as follows: Among them, N is the total number of images in a batch, i represents the anchor point, Indicates that a batch has the same label as anchor i The total number of images, 1 i≠j ∈{0, 1) is an indicator function that returns 1 if and only if i≠j, otherwise it returns 0; Represents the enhanced image The feature vector obtained by the encoder E(·) and the projector P(·); z i ·z j Indicates z i and z j The inner product operation between them calculates the distance between two features, z i ·z k Indicates z i and z k The inner product operation between them; τ>0 is a scalar adjustment coefficient, set to 0.

07.

4. The classification method based on open set recognition of cultural relics images according to claim 1 is characterized in that: The open set identification algorithm of cultural relics based on feature migration prior error means: through the algorithm based on feature migration prior error, the open set identification ability of cultural relic images of the network is further improved; the model expression is as follows: Where i is the selected i-th layer encoder, cos(·) represents the calculation of the cosine distance between two feature vectors, x represents the input image, E(·) represents the features of the corresponding layer output of the encoder trained by contrastive learning, E′(·) represents the features of the corresponding layer output of the encoder trained by transfer learning, f(x) represents the activation feature value output by the classification network, score(x) represents the corrected activation feature value, δ is the set threshold, where decision represents the final judgment made by the model, max(score) represents the maximum value of the activation feature value, rejected as unknown represents that the model recognizes the image as an open set cultural relic sample, argmax score represents the category corresponding to the maximum value of the activation feature value, and class y pred Represents the known class of artifact images predicted by the model.

Images