A Transformer-based image classification method, system, device and storage medium

Abstract

The present application discloses a Transformer-based image classification method, system, device and storage medium, which can make full use of a small amount of labeled data and feature aliasing technology for model training, and quickly build a high-performance image classification model in the case of scarce labeled samples to achieve accurate classification of images. The present application method includes: obtaining a collection of labeled training images, wherein the training image collection contains training images of multiple categories; training a Transformer model and a first feature classifier according to the training image collection to improve the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier; linearly interpolating the hidden layer features of the two training images to obtain the target hidden layer features; decoding the target hidden layer features, and calculating the cross entropy loss with the category label of the training image; training the Transformer model according to the cross entropy loss to improve the third feature representation capability of the Transformer model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a Transformer-based image classification method, system, device and storage medium. Background Art

[0002] As a current-type light-emitting device, OLED has been increasingly used in high-performance display devices. Due to its self-luminous characteristics, compared with LCD, OLED has many advantages such as high contrast, ultra-lightness, wide viewing angle, fast response speed, can be used for curved display panels, wide operating temperature range, simple structure and process. However, the Mura defects of OLED displays generally have the characteristics of low contrast, blurred edges, unstable shape and difficulty in judging by the naked eye. The current mainstream detection technology and method for OLED displays is to use computer vision detection.

[0003] Computer vision inspection uses industrial cameras to obtain high-definition images of OLED screens, and then uses image preprocessing technology to highlight defect features. It then uses a trained AI neural network model and traditional visual inspection algorithms to identify defects, thereby replacing manual inspection. However, the use of neural network models requires a large amount of labeled data for training, which may make it difficult to collect defect samples or the collection cost is high. In addition, since the model is trained on a large amount of labeled data, the training model is highly dependent on a large amount of labeled data. The image recognition ability is limited when there are fewer samples, and the model generalization ability is insufficient, making it impossible to accurately classify images. Summary of the invention

[0004] The present application provides a Transformer-based image classification method, system, device and storage medium, which can make full use of a small amount of labeled data and feature aliasing technology for model training. When labeled samples are scarce, a high-performance image classification model can be quickly constructed, effectively solving the problem of requiring a large amount of labeled data for training, reducing dependence on large-scale labeled data, and further improving the generalization ability of the model to achieve accurate classification of images.

[0005] The first aspect of the present application provides a Transformer-based image classification method, comprising:

[0006] Obtaining a labeled training image collection, wherein the training image collection includes training images of multiple categories;

[0007] Training a Transformer model and a first feature classifier according to the training image collection to improve the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier;

[0008] The hidden layer features of the two training images are linearly interpolated to obtain the target hidden layer features;

[0009] After decoding the target hidden layer features, the cross entropy loss is calculated with the category labels of the training images;

[0010] Training the Transformer model according to the cross entropy loss to improve the third feature representation capability of the Transformer model;

[0011] Obtain a labeled support image collection, wherein the support image collection includes training images of the new category;

[0012] Freezing parameters of the Transformer model, and training a second feature classifier based on the support image collection, wherein the second feature classifier is used for classification based on a directional measure between features;

[0013] Constructing a classification neural network model according to the Transformer model, the first feature classifier and the second feature classifier;

[0014] The image is classified by the classification neural network model.

[0015] Optionally, training the Transformer model and the first feature classifier according to the training image collection includes:

[0016] Acquire multiple categories of training images from the training image collection;

[0017] Dividing the sample of the training image into a plurality of image blocks of fixed size;

[0018] Representing the plurality of image blocks using position coding and pixel coding respectively, wherein the position coding represents the position of the image block, and the pixel coding represents the color and brightness of the image block;

[0019] Inputting a plurality of the image blocks into the first feature classifier and the encoder of the Transformer model to obtain output content;

[0020] The output content is received by the decoder of the Transformer model, and the masked image block is predicted in combination with the category label of the image to improve the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier.

[0021] Optionally, linearly interpolating the hidden layer features of the two training images to obtain a target hidden layer includes:

[0022] After determining the first training image, randomly selecting a second training image;

[0023] Extracting first hidden layer features of the first training image and second hidden layer features of the second training image respectively;

[0024] The target hidden layer is obtained by linearly interpolating the first hidden layer features and the second hidden layer features.

[0025] Optionally, training a second feature classifier according to the supporting image collection includes:

[0026] Inputting the training image of the new category into the first feature classifier to obtain a target feature vector;

[0027] The second feature classifier is trained according to the target feature vector.

[0028] Optionally, training the second feature classifier according to the supporting image collection includes:

[0029] A cosine classifier is trained based on the support image collection.

[0030] Optionally, after classifying the image by the classification neural network model, the method further includes:

[0031] Get a collection of unlabeled query images;

[0032] The images in the query image collection are classified according to the classification neural network model to obtain classified images.

[0033] Optionally, after classifying the images in the query image collection according to the classification neural network model to obtain classified images, the method further includes:

[0034] Determining whether the classified image is an image of a new category;

[0035] If so, the image of the new category is saved to the labeled support image collection.

[0036] The second aspect of the present application provides a Transformer-based image classification system, the system comprising:

[0037] A first acquisition unit is used to acquire a labeled training image collection, wherein the training image collection includes training images of multiple categories;

[0038] A first training unit, used for training a Transformer model and a first feature classifier according to the training image collection, so as to improve a first feature representation capability of the Transformer model and a second feature representation capability of the first feature classifier;

[0039] An interpolation unit, used to obtain target hidden layer features by linearly interpolating the hidden layer features of two training images;

[0040] A calculation unit, used for decoding the target hidden layer features and calculating the cross entropy loss with the category label of the training image;

[0041] A second training unit is used to train the Transformer model according to the cross entropy loss to improve the third feature representation ability of the Transformer model;

[0042] A second acquisition unit is used to acquire a labeled support image collection, where the support image collection includes training images of a new category;

[0043] A third training unit, used for freezing the parameters of the Transformer model, and training a second feature classifier according to the support image collection, wherein the second feature classifier is used for classification by a directional measure between features;

[0044] A construction unit, configured to construct a classification neural network model according to the Transformer model, the first feature classifier and the second feature classifier;

[0045] The classification unit is used to classify the image through the classification neural network model.

[0046] A third aspect of the present application provides a Transformer-based image classification device, the device comprising:

[0047] processor, memory, input-output unit, and bus;

[0048] The processor is connected to the storage, the input and output unit, and the bus;

[0049] The storage stores a program, and the processor calls the program to perform the first aspect and any optional Transformer-based image classification in the first aspect.

[0050] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program performs the first aspect and any optional Transformer-based image classification in the first aspect.

[0051] It can be seen from the above technical solutions that this application has the following advantages:

[0052] (1) By training the Transformer model, the Transformer model can learn a more comprehensive, discriminative, and semantically rich feature representation, which can cope with complex image classification scenarios and greatly improve the classification performance.

[0053] (2) A three-stage training strategy is adopted to make full use of unlabeled data and a small amount of labeled data, which effectively alleviates the problem of scarce labeled data. In the case of scarce labeled samples, a high-performance image classification model can be quickly built, greatly reducing the dependence on large-scale labeled data.

[0054] (3) By introducing a feature aliasing mechanism and performing linear interpolation in the deep hidden layers of the Transformer, the model can learn smoother and more generalizable feature representations, broaden the inter-class decision boundary, and improve the model’s classification ability in few-sample scenarios.

[0055] (4) This application can make full use of a small amount of labeled data and feature aliasing technology for model training. When labeled samples are scarce, it can quickly build a high-performance image classification model, effectively solve the problem of requiring a large amount of labeled data for training, reduce dependence on large-scale labeled data, and further improve the generalization ability of the model to achieve accurate classification of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 This is a flowchart of an embodiment of the Transformer-based image classification method of the present application;

[0058] Figure 2 This is a flowchart of another embodiment of the Transformer-based image classification method of the present application;

[0059] Figure 3 This is a flowchart of another embodiment of the Transformer-based image classification method of the present application;

[0060] Figure 4 This is a flowchart of another embodiment of the Transformer-based image classification method of the present application;

[0061] Figure 5This is a flowchart of another embodiment of the Transformer-based image classification method of the present application;

[0062] Figure 6 This is a schematic diagram of an embodiment of the Transformer-based image classification system of the present application;

[0063] Figure 7 This is a schematic diagram of an embodiment of a Transformer-based image classification device of the present application. DETAILED DESCRIPTION

[0064] The present application provides a Transformer-based image classification method, system, device and storage medium, which can make full use of a small amount of labeled data and feature aliasing technology for model training. When labeled samples are scarce, a high-performance image classification model can be quickly constructed, effectively solving the problem of requiring a large amount of labeled data for training, reducing dependence on large-scale labeled data, and further improving the generalization ability of the model to achieve accurate classification of images.

[0065] It should be noted that the Transformer-based image classification method provided in this application can be applied to a terminal or a server. For example, the terminal can be a tablet computer, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as the execution subject for example.

[0066] See also Figure 1 , Figure 1 An embodiment of the Transformer-based image classification method provided in this application includes:

[0067] 101. Obtain a collection of labeled training images, where the collection of training images includes training images of multiple categories;

[0068] In an embodiment of the present application, before obtaining a labeled training image collection, the terminal needs to collect an image collection first. The collected image collection must ensure diversity, including image collections of different categories, different perspectives, different lighting conditions, and different backgrounds. One or more category labels are then assigned to each image in the image collection, where the label represents the category of the object in the image. Different image groups are then combined to form a labeled training image collection, which includes training images of multiple categories.

[0069] 102. Training a Transformer model and a first feature classifier according to the training image collection to improve a first feature representation capability of the Transformer model and a second feature representation capability of the first feature classifier;

[0070] In an embodiment of the present application, a previously collected and prepared training image collection is used to train the Transformer model and the first feature classifier, so that the Transformer model and the first feature classifier can extract richer and more meaningful feature representations from the image. Specifically, first, the training image collection is loaded into the training environment of the Transformer model, and the image is preprocessed, such as scaling and normalization, to ensure the consistency of the image input. At the same time, the image label is also loaded together for use in the Transformer model training process.

[0071] Transformer model training: After determining the Transformer model architecture, it is initialized, including setting parameters such as the number of model layers, the number of attention heads, and the embedding dimension. The preprocessed image collection is input into the Transformer model, and the feature representation of the image collection is gradually extracted through the self-attention mechanism and the feedforward neural network model.

[0072] First feature classifier training: While training the Transformer model, the first feature classifier is also trained simultaneously. Since the parameters of the Transformer model and the first feature classifier are jointly optimized, the feature representation extracted by the Transformer model will be suitable for the classification task of the first feature classifier.

[0073] Through continuous iterative training, the Transformer model and the first feature classifier's first feature representation capabilities are improved, and they can capture more subtle and useful information in the image.

[0074] 103. Performing linear interpolation on the hidden layer features of the two training images to obtain target hidden layer features;

[0075] In the embodiment of the present application, the training set is enriched by synthesizing new feature representations, thereby improving the generalization ability of the Transformer model. Specifically, two images are randomly selected from the training image collection, and the two images may belong to the same category or different categories, and the hidden layer features of the two images in the Transformer model are extracted. The hidden layer features are usually the output of the middle layer of the Transformer model, which contains the high-level semantic information of the image.

[0076] Linear interpolation is performed on the two selected hidden layer features. The interpolation formula can be expressed as: new feature = a * feature 1 + (1- a) * feature 2, where a is an interpolation coefficient between 0 and 1. By adjusting the value of a, multiple hidden layer features between the two hidden layer features can be generated. The new feature obtained by interpolation is used as the target hidden layer feature. The target hidden layer feature helps the Transformer model learn a smoother and more robust feature representation.

[0077] By repeating the above process between different training image pairs, a large number of diverse target hidden layer features can be generated. These target hidden layer features increase the diversity of the training set, helping the Transformer model learn more generalized feature representations, thereby improving the classification performance on new and unseen images.

[0078] 104. After decoding the target hidden layer features, a cross entropy loss is calculated with the category labels of the training images;

[0079] In an embodiment of the present application, the target hidden layer features are decoded into category predictions and compared with the actual training image category labels to calculate the loss function and optimize the model. Specifically, the decoder of the Transformer model is used to decode the target hidden layer features. The decoding process involves a series of linear transformations and nonlinear activation functions. After decoding, a vector equal to the number of categories is output, and each element corresponds to the predicted probability of a category.

[0080] The cross entropy loss measures the difference between the category probability distribution predicted by the Transformer model and the true category label. For each training image, its decoded category prediction is compared with the true category label to calculate the cross entropy loss.

[0081] Step 104 is typically repeated multiple times in an iterative training process, where in each iteration the model processes a batch of training images, calculates the loss function, and updates its parameters.

[0082] Through multiple iterative training, the model can gradually learn the mapping relationship from images to category labels and improve its classification accuracy.

[0083] 105. Training the Transformer model according to the cross entropy loss to improve the third feature representation capability of the Transformer model;

[0084] In the embodiment of the present application, the cross entropy loss provides a clear direction for the optimization of the Transforme model. By minimizing the loss function, the model can gradually adjust its parameters so that the predicted category probability distribution is closer to the true category label. After calculating the cross entropy loss, the gradient of the loss function is propagated back to the model parameters through the back propagation algorithm to update the weights and biases of the Transforme model for optimizing the Transforme model parameters. In this way, the Transforme model can learn smoother and more generalized feature representations.

[0085] 106. Obtain a labeled support image collection, wherein the support image collection includes training images of a new category;

[0086] In the embodiment of the present application, a training image collection containing multiple categories is used in the aforementioned steps to train the Transformer model and the first feature classifier. However, the training image collection may not contain all categories. In order to improve the classification ability, it is necessary to further introduce new categories and use the support image collection for training. The support training collection contains some new and previously unseen training images of categories. It should be noted that the support image collection has only 1-10 images of each type, which are used for inference prediction of new categories.

[0087] The supporting image collection should contain a sufficient number of training images of new categories. These images are of high quality and representative, and can fully demonstrate the characteristics of the new categories. Each image has a clear category label to guide the model to learn the characteristics of the new category during training.

[0088] 107. Freeze the parameters of the Transformer model, and train the second feature classifier according to the support image collection, where the second feature classifier is used for classification based on the direction measurement between features;

[0089] In an embodiment of the present application, after obtaining the training images of the new category, the parameters of the Transformer model are frozen, and the second feature classifier is trained based on the support image collection. In the aforementioned steps, the Transformer model has been fully trained with a large amount of training data, and its parameters have been optimized to a good state. In order to maintain the stability and generalization ability of the Transformer model, in this step, it is chosen to freeze its parameters, that is, these parameters will no longer be updated in the subsequent training process. Freezing parameters can prevent excessive interference to the Transformer model when training the second feature classifier, thereby maintaining its learned feature representation capabilities.

[0090] The second feature classifier is a key component for classifying new categories of images. Since the parameters of the Transformer model have been frozen, it is necessary to train the second feature classifier to learn the features of the new category. The method for training the second feature classifier is to use the support image collection for supervised learning.

[0091] The second feature classifier classifies by directional measurement between features. The directional measurement focuses on the angular relationship between feature vectors, rather than the traditional Euclidean distance, so that it can better capture the similarities and differences between features, thereby improving the accuracy of classification. During the training process, the second feature classifier will learn how to distinguish different categories based on the direction of the feature vector, so that it can better classify and identify new and unseen images. The commonly used second feature classifier is the cosine classifier, which predicts the category label by calculating the cosine similarity between feature vectors. Since the feature extractor has a strong generalization ability, it can achieve good classification results even if it is trained on a small number of labeled samples.

[0092] 108. Construct a classification neural network model according to the Transformer model, the first feature classifier, and the second feature classifier;

[0093] In the embodiment of the present application, the classification neural network model constructed according to the Transformer model, the first feature classifier and the second feature classifier plays the role of integration and assembly in the construction process of the entire image classification method. Specifically, the previously trained Transformer model, the first feature classifier and the second feature classifier are integrated. The Transformer model is responsible for extracting the features of the image, and the two feature classifiers are responsible for classifying the image according to these features. The integration process involves connecting these three components according to a specific architecture to ensure that they can work together to form a complete classification neural network model.

[0094] 109. Classify the image using the classification neural network model.

[0095] In an embodiment of the present application, the images to be classified are input into the classification neural network model. These images can be any new images from actual application scenarios, and the classification neural network model needs to classify these images. Specifically, when the image is input into the classification neural network model, it will first be subjected to feature extraction through the Transformer model. The Transformer model will use its feature representation capabilities learned during the training process to extract useful feature information from the input image. The extracted features will then be passed to the first feature classifier and the second feature classifier. The first feature classifier and the second feature classifier classify the features of the input image according to their respective learned feature representations and classification capabilities, and output the classification results.

[0096] See also Figure 2 , Figure 2 Another embodiment of the Transformer-based image classification method provided in the present application includes:

[0097] 201. Obtain a labeled training image collection, wherein the training image collection includes training images of multiple categories;

[0098] 202. Training a Transformer model and a first feature classifier according to the training image collection to improve a first feature representation capability of the Transformer model and a second feature representation capability of the first feature classifier;

[0099] 203. Performing linear interpolation on the hidden layer features of the two training images to obtain target hidden layer features;

[0100] 204. After decoding the target hidden layer features, calculate the cross entropy loss with the category label of the training image;

[0101] 205. Training the Transformer model according to the cross entropy loss to improve the third feature representation capability of the Transformer model;

[0102] 206. Obtain a labeled support image collection, wherein the support image collection includes training images of a new category;

[0103] 207. Freeze the parameters of the Transformer model, and train the second feature classifier according to the support image collection, where the second feature classifier is used for classification based on the direction measurement between features;

[0104] 208. Construct a classification neural network model according to the Transformer model, the first feature classifier and the second feature classifier;

[0105] 209. Classifying the image by using the classification neural network model;

[0106] In the embodiment of the present application, steps 201 to 209 are similar to the aforementioned steps 101 to 109 and are not described again here.

[0107] 210. Obtain a collection of unlabeled query images;

[0108] In an embodiment of the present application, the collection of unlabeled query images can come from a variety of channels, such as user uploads, web crawlers, and public data sets. These images are not accompanied by any category labels when they are acquired, so subsequent classification processing is required to check the classification ability of the classification neural network model.

[0109] After acquiring the image, the image quality needs to be checked to ensure that the image is clear and free of severe noise or damage. Poor quality images will affect the subsequent classification accuracy, so preprocessing or screening is required. In order to facilitate subsequent processing, all images need to be converted into a unified format, such as JPEG, PNG, etc. This helps ensure that the classification neural network model can correctly process all input images.

[0110] 211. Classify images in the query image collection according to the classification neural network model to obtain classified images.

[0111] In the embodiment of the present application, the image categories output by the classification neural network model are judged to determine whether these images belong to new categories outside the known categories. First, there needs to be a set containing all known categories. This set is used in the model training process and contains all category labels that the classification neural network model can recognize.

[0112] The classification results output by the classification neural network model are analyzed. Each classification result contains one or more category labels and corresponding classification probabilities. The category labels in the classification results are matched with the known category set. If a category label in the classification result exists in the known category set, then the image belongs to the known category; if a category label in the classification result does not exist in the known category set, then it can be considered that the image belongs to a new category.

[0113] 212. Determine whether the classified image is an image of a new category;

[0114] In the embodiment of the present application, the image categories output by the classification neural network model are judged to determine whether these images belong to new categories outside the known categories. In order to more accurately judge the new category, a threshold of classification probability can be set. Only when the classification probability is lower than this threshold, the image is considered to belong to the new category. When it is determined that the classified image is an image of a new category, step 213 is executed.

[0115] 213. Save the image of the new category into the labeled supporting image collection.

[0116] In the embodiment of the present application, when the classified image is determined to be a new category image, its category label and related information are recorded. At the same time, these new category images are added to the labeled support image collection to further train and update the classification neural network model.

[0117] Reference Figure 3 According to some embodiments of the present invention, training the Transformer model and the first feature classifier according to the training image collection in step 102 may specifically include, but is not limited to, the following:

[0118] 301. Acquire multiple categories of training images from the training image collection;

[0119] 302. Divide the training image sample into a plurality of image blocks of fixed sizes;

[0120] 303. Represent the plurality of image blocks using position coding and pixel coding respectively, wherein the position coding represents the position of the image block, and the pixel coding represents the color and brightness of the image block;

[0121] 304. Input the plurality of image blocks into the first feature classifier and the encoder of the Transformer model to obtain output content;

[0122] 305. Receive the output content through the decoder of the Transformer model, and predict the masked image block in combination with the category label of the image to improve the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier.

[0123] In an embodiment of the present application, for training a Transformer model and a first feature classifier according to a training image collection, the specific process is as follows: obtaining training images of multiple categories from the training image collection, and then dividing the training image samples into a number of image blocks of a fixed size, the size of which is not limited herein; after being divided into a number of image blocks of a fixed size, the image blocks are represented by position coding and pixel coding, respectively, wherein the position coding represents the position of the image block, and the pixel coding represents the color and brightness of the image block; then the image blocks are input into the encoder of the first feature classifier and the Transformer model to obtain output content; the decoder of the Transformer model receives the output content, and predicts the masked image blocks in combination with the category label of the image, so as to improve the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier.

[0124] Among them, obtaining training images of various categories is the process of preparing training data. By obtaining images of various categories from the training image collection, it can be ensured that the Transformer model can learn the features of different categories during the training process, thereby improving the generalization ability of the model. The image is divided into image blocks of fixed size to facilitate subsequent processing and encoding. This segmentation method helps the model better understand and process the local features of the image, and also helps to reduce the amount of calculation. Position encoding is used to represent the position information of the image block in the image, which helps the model understand the spatial structure of the image. Pixel encoding is used to represent the color and brightness information of the image block, which is an important feature in image classification tasks. Through these two encoding methods, the model can understand the content of the image block more comprehensively.

[0125] The encoded image blocks are input into the encoder and the first feature classifier of the Transformer model for training to obtain higher-level feature representations. The decoder of the Transformer model receives the output of the encoder and combines the category label of the image to predict the masked image blocks. This process is a self-supervised learning task, which enables the Transformer model to infer the missing parts based on known contextual information. In this way, the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier can be further improved, so that the model can perform better on image classification tasks.

[0126] Reference Figure 4 According to some embodiments of the present invention, the step 103 of linearly interpolating the hidden layer features of the two training images to obtain the target hidden layer may specifically include, but is not limited to, the following:

[0127] 401. After determining the first training image, randomly select a second training image;

[0128] 402. Extract first hidden layer features of the first training image and second hidden layer features of the second training image respectively;

[0129] 403. Perform linear interpolation on the first hidden layer features and the second hidden layer features to obtain the target hidden layer.

[0130] In an embodiment of the present application, the target hidden layer is obtained by linearly interpolating the hidden layer features of the two training images. Specifically, after determining the first training image, randomly select the second training image; extract the first hidden layer features of the first training image and the second hidden layer features of the second training image respectively; and obtain the target hidden layer by linearly interpolating the first hidden layer features and the second hidden layer features.

[0131] Specifically, after determining the first training image, the second training image is randomly selected in order to construct a pair of training images for subsequent feature extraction and interpolation. By randomly selecting the second training image, the diversity and generalization ability of the training can be increased, so that the Transformer model can learn to deal with the feature differences between different images. Furthermore, after obtaining two training images, the Transformer model extracts the feature representations of each of the two training images in a hidden layer. These hidden layer features contain high-level semantic information of the image. By using linear interpolation, linear interpolation is used to generate a new feature representation between the two hidden layer features, namely the target hidden layer. This process can simulate the feature transition between different images, which helps the Transformer model learn smoother and continuous features.

[0132] The purpose of this solution is to enhance the model's ability to understand and express image features by extracting and interpolating hidden layer features of different training images, thereby improving the generalization ability of the Transformer model and enabling it to better process unseen image data. At the same time, by simulating the feature transition between different images, it can also help the Transformer model learn more robust feature representations.

[0133] Reference Figure 5 According to some embodiments of the present invention, training the second feature classifier according to the supporting image collection in step 107 may specifically include, but is not limited to, the following:

[0134] 501. Input the training image of the new category into the first feature classifier to obtain a target feature vector;

[0135] 502. Train the second feature classifier according to the target feature vector.

[0136] In an embodiment of the present application, the second feature classifier is trained based on the supporting image collection. Specifically, the training image of the new category is input into the first feature classifier to obtain the target feature vector, and then the second feature classifier is trained based on the target feature vector.

[0137] Specifically, the purpose of inputting the training image of the new category into the first feature classifier to obtain the target feature vector is to use the trained first feature classifier to process the training image of the new category, so as to extract the feature vector of the image, that is, the target feature vector. The target feature vector contains important information of the image. After obtaining the target feature vector, these feature vectors are used to train the second feature classifier. Through training, the second feature classifier can learn the features of the new category, so that it can effectively classify or identify images of this category. In addition, the transfer and expansion of knowledge can be realized, so that the Transformer model can handle more categories, improve the generalization ability and practicality of the model, and at the same time, by continuously training and updating the feature classifier, the model can be more adapted to the changes and needs of the actual scene.

[0138] See also Figure 6 , Figure 6 In an embodiment of the Transformer-based image classification system provided in the present application, the device comprises:

[0139] A first acquisition unit 601 is used to acquire a labeled training image collection, wherein the training image collection includes training images of multiple categories;

[0140] A first training unit 602 is used to train a Transformer model and a first feature classifier according to the training image collection to improve the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier;

[0141] An interpolation unit 603 is used to obtain a target hidden layer feature by performing linear interpolation on the hidden layer features of the two training images;

[0142] A calculation unit 604 is used to decode the target hidden layer features and calculate the cross entropy loss with the category label of the training image;

[0143] A second training unit 605 is used to train the Transformer model according to the cross entropy loss to improve the third feature representation capability of the Transformer model;

[0144] A second acquisition unit 606 is used to acquire a labeled support image collection, where the support image collection includes training images of a new category;

[0145] A third training unit 607, used to freeze the parameters of the Transformer model, and train the second feature classifier according to the support image collection, wherein the second feature classifier is used to perform classification by a directional measure between features;

[0146] A construction unit 608, configured to construct a classification neural network model according to the Transformer model, the first feature classifier and the second feature classifier;

[0147] The classification unit 609 is used to classify the image through the classification neural network model.

[0148] See also Figure 7 , the present application also provides a Transformer-based image classification device, the device comprising:

[0149] Processor 701, storage 702, input and output unit 703, bus 704;

[0150] The processor 701 is connected to the storage 702, the input and output unit 703 and the bus 704;

[0151] The storage 702 stores a program, and the processor 701 calls the program to execute any of the above Transformer-based image classification methods.

[0152] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above Transformer-based image classification methods.

[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc., various media that can store program codes.

Claims

1. A Transformer-based image classification method, characterized in that: The method comprises: Obtaining a labeled training image collection, wherein the training image collection includes training images of multiple categories; Training a Transformer model and a first feature classifier according to the training image collection to improve the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier; The hidden layer features of the two training images are linearly interpolated to obtain the target hidden layer features; After decoding the target hidden layer features, the cross entropy loss is calculated with the category labels of the training images; Training the Transformer model according to the cross entropy loss to improve the third feature representation capability of the Transformer model; Obtain a labeled support image collection, wherein the support image collection includes training images of the new category; Freezing parameters of the Transformer model, and training a second feature classifier based on the support image collection, wherein the second feature classifier is used for classification based on a directional measure between features; Constructing a classification neural network model according to the Transformer model, the first feature classifier and the second feature classifier; The image is classified by the classification neural network model.

2. The Transformer-based image classification method according to claim 1, characterized in that: Training the Transformer model and the first feature classifier according to the training image collection includes: Acquire multiple categories of training images from the training image collection; Dividing the sample of the training image into a plurality of image blocks of fixed size; Representing the plurality of image blocks using position coding and pixel coding respectively, wherein the position coding represents the position of the image block, and the pixel coding represents the color and brightness of the image block; Inputting a plurality of the image blocks into the first feature classifier and the encoder of the Transformer model to obtain output content; The output content is received by the decoder of the Transformer model, and the masked image block is predicted in combination with the category label of the image to improve the first feature representation capability of the Transformer model and the second feature representation capability of the first feature classifier.

3. The Transformer-based image classification method according to claim 1, characterized in that: The target hidden layer obtained by linearly interpolating the hidden layer features of the two training images includes: After determining the first training image, randomly selecting a second training image; Extracting first hidden layer features of the first training image and second hidden layer features of the second training image respectively; The target hidden layer is obtained by linearly interpolating the first hidden layer features and the second hidden layer features.

4. The Transformer-based image classification method according to claim 1, characterized in that: Training the second feature classifier according to the supporting image collection includes: Inputting the training image of the new category into the first feature classifier to obtain a target feature vector; The second feature classifier is trained according to the target feature vector.

5. The Transformer-based image classification method according to claim 1, characterized in that: Training the second feature classifier according to the supporting image collection includes: A cosine classifier is trained based on the support image collection.

6. The Transformer-based image classification method according to claim 1, characterized in that: After classifying the image by the classification neural network model, the method further includes: Get a collection of unlabeled query images; The images in the query image collection are classified according to the classification neural network model to obtain classified images.

7. The Transformer-based image classification method according to claim 6, characterized in that: After classifying the images in the query image collection according to the classification neural network model to obtain classified images, the method further includes: Determining whether the classified image is an image of a new category; If so, the image of the new category is saved to the labeled support image collection.

8. A Transformer-based image classification system, characterized in that: The system comprises: A first acquisition unit is used to acquire a labeled training image collection, wherein the training image collection includes training images of multiple categories; A first training unit, used for training a Transformer model and a first feature classifier according to the training image collection, so as to improve a first feature representation capability of the Transformer model and a second feature representation capability of the first feature classifier; An interpolation unit, used to obtain target hidden layer features by linearly interpolating the hidden layer features of two training images; A calculation unit, used for decoding the target hidden layer features and calculating the cross entropy loss with the category label of the training image; A second training unit is used to train the Transformer model according to the cross entropy loss to improve the third feature representation ability of the Transformer model; A second acquisition unit is used to acquire a labeled support image collection, where the support image collection includes training images of a new category; A third training unit, used for freezing the parameters of the Transformer model, and training a second feature classifier according to the support image collection, wherein the second feature classifier is used for classification by a directional measure between features; A construction unit, configured to construct a classification neural network model according to the Transformer model, the first feature classifier and the second feature classifier; The classification unit is used to classify the image through the classification neural network model.

9. A Transformer-based image classification device, characterized in that: The device comprises: processor, memory, input-output unit, and bus; The processor is connected to the storage, the input and output unit, and the bus; The storage stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 7.

Images