Open set fine granularity identification method and device

Abstract

The invention provides an open set fine granularity identification method and device, and relates to the technical field of image identification. The open set fine granularity identification method comprises the following steps: constructing a sample data set; based on the sample data set, training to obtain a target Transform encoder and an open set identification classifier; dividing the acquired to-be-classified image data into a plurality of image blocks with preset sizes, and adding position codes to the image blocks; random masking is carried out on the image blocks with the position codes according to a preset proportion, all the image blocks are input into a target Transform encoder, and target features are output; and inputting the target features into an open set recognition classifier to obtain a target category to which the to-be-classified image data belongs. And the target feature is also input into a decoder corresponding to the target Transform encoder for reconstruction. According to the technical scheme of the embodiment of the invention, open-set fine-grained recognition is realized in a complex scene, the accuracy is high, and the robustness is good.

Description

Technical Field

[0001] The present application relates to the technical field of image recognition, and in particular to an open set fine-grained recognition method and device. Background Art

[0002] At present, in the field of image recognition, especially in fine-grained classification tasks, identifying objects of different categories with similar features has always been a difficult problem. Traditional fine-grained recognition methods usually rely on a large amount of labeled data to train deep learning models. When there is only a small amount of labeled data or when the data is completely unlabeled, traditional fine-grained recognition methods usually face the problem of significantly degraded performance.

[0003] In addition, in real application scenarios, fine-grained recognition problems of new or unknown categories often arise. Traditional closed-set fine-grained classification models are prone to misclassification problems and have low accuracy when encountering these new categories. Summary of the invention

[0004] Based on this, the present application provides an open set fine-grained recognition method and device to achieve open set fine-grained recognition with high accuracy and good robustness.

[0005] According to one aspect of the present application, an open set fine-grained recognition method is proposed, comprising: constructing a sample data set; wherein the sample data set comprises a plurality of groups of image data samples with category annotations; based on the sample data set, training a target Transformer encoder and an open set recognition classifier; dividing the acquired image data to be classified into a plurality of image blocks of a preset size, and adding position codes to the image blocks; randomly masking the image blocks with position codes according to a preset ratio, and inputting all the image blocks into a target Transformer encoder, and outputting target features from the target Transformer encoder; inputting the target features into the open set recognition classifier to obtain the target category to which the image data to be classified belongs.

[0006] According to some embodiments, based on a sample data set, a target Transformer encoder is trained, including: dividing an image data sample into multiple image block samples of a preset size; adding position codes to the image block samples; randomly masking the image block samples according to a preset ratio; inputting all image block samples into a preset encoder to obtain feature vectors of the image block samples; wherein the preset encoder is an encoder of a visual Transformer model based on an attention mechanism; concatenating the feature vectors of the image block samples using position codes to obtain target feature samples; inputting the target feature samples into a preset decoder to obtain a reconstructed image; wherein the preset decoder is a decoder of a visual Transformer model based on an attention mechanism; based on the difference between the reconstructed image and the corresponding image data sample, training the visual Transformer model based on the attention mechanism until a preset end condition is reached; extracting an encoder from the trained visual Transformer model based on the attention mechanism as the target Transformer encoder.

[0007] According to some embodiments, based on a sample data set, an open set recognition classifier is trained, including: performing meta-learning-based classification training and / or supervised training of known classes based on the sample data set until a preset end condition is reached to obtain a recognition classifier; integrating the recognition classifier and a preset open set recognition post-processing algorithm to obtain an open set recognition classifier.

[0008] According to some embodiments, meta-learning-based classification training is performed based on a sample data set, including: calculating a first distance between a target sample and its positive sample, and calculating a second distance between the target sample and its negative sample; wherein the target sample is each image data sample in the sample data set, the positive sample is a sample in the sample data set that has the same category as the target sample, and the negative sample is a sample in the sample data set that has a different category from the target sample; a loss function is constructed based on the first distance and the second distance to ensure that the negative sample is farther from the anchor point than the positive sample by at least one preset margin; based on the sample data set, a preset classifier is classified and trained using the loss function.

[0009] According to some embodiments, supervised training of known classes is performed based on a sample data set, including: extracting a feature vector of an image data sample as an intermediate feature vector; inputting the intermediate feature vector into a preset classifier to obtain a predicted category; and training the preset classifier based on the difference between the predicted category and the category label corresponding to the image data sample.

[0010] According to some embodiments, the intermediate feature vector is a target feature sample.

[0011] According to some embodiments, the target feature is input into an open set recognition classifier to obtain the target category to which the image data to be classified belongs, including: obtaining the output result obtained by inputting the target feature into the recognition classifier; post-processing the output result using a preset open set recognition post-processing algorithm to obtain a target score; and obtaining the target category to which the image data to be classified belongs based on the target score and a preset threshold.

[0012] According to one aspect of the present application, an open-set fine-grained recognition device includes: a data set module, used to construct a sample data set; wherein the sample data set includes multiple groups of image data samples with category annotations; a model training module, used to train a target Transformer encoder and an open-set recognition classifier based on the sample data set; a position encoding module, used to divide the acquired image data to be classified into multiple image blocks of a preset size, and add position encoding to the image blocks; a mask encoding module, used to randomly mask the image blocks with position encoding according to a preset ratio, and input all image blocks into the target Transformer encoder, and output target features from the target Transformer encoder; a recognition and classification module, used to input the target features into the open-set recognition classifier to obtain the target category to which the image data to be classified belongs.

[0013] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0014] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0015] Through the above-mentioned embodiments provided by the present application, the input image data to be classified is divided into image blocks and position encoded, so as to effectively extract local features and enable the model to better capture subtle differences; a sample data set is constructed to train the target Transformer encoder and the power-on recognition classifier. In practical applications, by using the attention ability of the target Transformer encoder, the global and local features of the image data to be classified can be quickly extracted, and the extracted features are input into the open set recognition classifier to achieve fine-grained classification of known classes and detection of unknown classes, thereby improving the accuracy and robustness of fine-grained recognition and achieving effective recognition of complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by the present application.

[0018] Figure 1 A flowchart of an open set fine-grained recognition method provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of a training target Transformer encoder provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of training an open set recognition classifier provided in an embodiment of the present application;

[0021] Figure 4 A flowchart of performing meta-learning-based classification training based on a sample data set provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a learning objective for classification training based on meta-learning provided in an embodiment of the present application;

[0023] Figure 6 A flowchart of supervised training of known classes based on a sample data set provided in an embodiment of the present application;

[0024] Figure 7 A flowchart of inputting target features into an open set recognition classifier to obtain the target category to which the image data to be classified belongs, provided in an embodiment of the present application;

[0025] Figure 8 The overall framework diagram of the open set fine-grained recognition method provided by the embodiment of the present application;

[0026] Fig. 9 A block diagram of an open set fine-grained recognition device provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0029] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0032] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0033] For specific implementation methods, please refer to the following embodiments.

[0034] Figure 1 Flow chart of the open set fine-grained recognition method provided in the embodiment of the present application. Figure 1 As shown, the method includes steps S110 to S150.

[0035] In step S110, a sample data set is constructed; wherein the sample data set includes a plurality of groups of image data samples with category annotations.

[0036] This application does not restrict the way of obtaining image data samples, and can use data from public image datasets, or collected or synthesized image data, etc. In order to further speed up the training, the image data can be representative, diverse, and sufficient in quantity so that the model can quickly learn effective features.

[0037] In addition, the category annotation label of the image data sample may be the image data sample itself, or may be manually annotated or obtained by using an annotation tool, and this application does not impose any restrictions on this.

[0038] In step S120, based on the sample data set, a target Transformer encoder and an open set recognition classifier are trained.

[0039] In the training process of the target Transformer encoder, the training goal of the target Transformer encoder can be: input new image data into the target Transformer encoder, and directly extract and output discriminative feature information based on a comprehensive understanding of the entire image. In the training process of the open set recognition classifier, the training goal of the open set recognition classifier can be: input new feature information into the open set recognition classifier to achieve fine-grained classification of known classes and detection of unknown classes.

[0040] In step S130, the acquired image data to be classified is divided into a plurality of image blocks of a preset size, and position codes are added to the image blocks.

[0041] In actual application, the image data to be classified is input image data waiting to be subjected to open-set fine-grained classification and recognition, and the image data to be classified is divided into a plurality of image blocks of preset sizes.

[0042] It should be noted that the input image data to be classified is divided into different local areas, and each area is input into the target Transformer encoder for feature extraction, so that the model can more accurately capture the subtle differences in the image data.

[0043] Further, according to the example embodiment, the image data to be classified is divided into N image blocks by using a convolution sliding window or direct segmentation. In terms of the division method, the block division generally includes overlapping block division and non-overlapping block division. Overlap means that there is a partial common area between the divided image blocks. It should be noted that in a specific embodiment, the preset size can be 16x16 or 32x32.

[0044] In addition, the present application does not limit the direction in which the image blocks are divided, and the image blocks can be divided according to the targets to be identified included in the image data to be classified. Taking the image data to be classified containing ships as an example, since ships are generally long and narrow in shape, multiple image blocks can be divided along the length direction of the ship.

[0045] Furthermore, after the image block division is completed, it can be further flattened into a one-dimensional feature vector. This step linearizes the pixel data in the image block so as to be processed in subsequent steps.

[0046] After the division is completed, a position code is added to each image block. It should be noted that the position code is used to retain the spatial position information of the image block to prevent the loss of spatial relationship when it is input into the Transformer.

[0047] Further, according to the example embodiments, the position codes are usually calculated in a specific manner, which can be fixed or learned. Fixed position codes are usually generated using different frequencies of sine and cosine functions. The advantage of this approach is that the model can handle sequences of any length because the position codes are infinitely extensible. The learned position codes are automatically optimized through the training process to better meet the needs of the specific tasks of this project. The position code is added to the vector obtained after flattening the image block to form the final input representation.

[0048] In step S140, the image blocks with position codes are randomly masked according to a preset ratio, and all the image blocks are input into a target Transformer encoder, and target features are output from the target Transformer encoder.

[0049] In order to achieve semantic reconstruction, the model adopts the mask mechanism of the MAE architecture. In actual application, the input image blocks are randomly masked according to the preset ratio.

[0050] According to an example embodiment, the preset ratio may be 75%, that is, the image data to be classified is divided into a plurality of patches (image blocks), 75% of the patches are randomly masked, and only 25% of the unmasked parts are allowed to be seen by the model.

[0051] After that, all image blocks (including masked image blocks and unmasked image blocks) are sent to the target Transformer encoder, which extracts the global features of the image by processing these input image blocks and obtains the feature vector of the Transformer encoding. The feature vectors are concatenated and the missing positions are filled with 0 values ​​to make them consistent and corresponding with the number of original image blocks, thus obtaining the target features.

[0052] In step S150, the target features are input into an open set recognition classifier to obtain the target category to which the image data to be classified belongs.

[0053] This application uses an open set recognition classifier (OSR Classifier) ​​to distinguish whether the input image belongs to a known category or an unknown category. It should be noted that the known category (Known Class) refers to the category that the model can recognize and classify, and the unknown category (Unknown Class) refers to the category that the model cannot recognize or belongs to the open category.

[0054] This application divides the input image data to be classified into image blocks and performs position encoding to effectively extract local features, so that the model can better capture subtle differences; constructs a sample data set to train the target Transformer encoder and the power-on recognition classifier. In practical applications, by using the attention ability of the target Transformer encoder, the global and local features of the image data to be classified can be quickly extracted, and the extracted features are input into the open set recognition classifier to achieve fine-grained classification of known classes and detection of unknown classes, improve the accuracy and robustness of fine-grained recognition, and achieve effective recognition of complex scenes.

[0055] According to some embodiments, reference Figure 2 In step S120, based on the sample data set, the target Transformer encoder is trained, which can be specifically implemented through steps S210 to S280.

[0056] In step S210, the image data samples are divided into a plurality of image block samples of a preset size.

[0057] The method of dividing the image data samples during the training process is the same as the method of dividing the image data to be classified during the application process, and this application will not elaborate on it here.

[0058] In step S220, position codes are added to the image block samples.

[0059] It should be noted that position encoding is used to solve the problem that the self-attention mechanism cannot capture the position information in the sequence. In the task of high-discriminative feature extraction in this application, the order of image blocks is crucial to understanding the overall features, and the Transformer's self-attention mechanism itself does not consider the position of image blocks in the sequence. Therefore, the introduction of position encoding enables the Transformer to fully understand the order information in the image block sequence.

[0060] The method of adding position coding to image block samples during the training process is the same as the method of adding position coding to image blocks during the application process, and this application will not go into details here.

[0061] The image block samples after adding position encoding are input into the model for training, so that the model can consider both the semantic information of the image block and the position information of the image block in the sequence in the self-attention mechanism.

[0062] In step S230, the image block samples are randomly masked according to a preset ratio.

[0063] The image block samples are randomly masked according to a preset ratio for subsequent self-supervised learning training. The method of randomly masking the image block samples during training is the same as the method of randomly masking the image blocks during application, and this application will not be repeated here.

[0064] In step S240, all image block samples are input into a preset encoder to obtain feature vectors of the image block samples; wherein the preset encoder is an encoder of a visual Transformer model based on an attention mechanism.

[0065] It should be noted that efficient extraction of highly discriminative features is crucial for accurately identifying small differences between target categories. These differences may come from aspects such as the target's appearance design, details, and color texture. Under the framework of deep learning, local feature extraction fusion and attention mechanism are two effective key technologies in this field, which can effectively capture general high-discriminative features. This application uses local feature extraction fusion and attention mechanism, and uses the visual Transformer model based on the attention mechanism as the basic model block for feature extraction. The MAE semantic reconstruction architecture is used to perform self-supervised learning training on the model to fully tap the attention ability of the Transformer model.

[0066] Among them, high-discrimination features are usually composed of multiple local detail features. Through the local feature extraction and fusion method based on deep learning, the fine-grained features of the target can be effectively extracted, and these features can be further combined to form efficient discriminant features. This application divides the input image data into multiple small image blocks, and then further inputs them into the neural network for local feature extraction.

[0067] In addition, in order to improve the model's perception of fine-grained features, an attention mechanism is introduced into the network structure. The model can automatically learn the importance of different regions after each convolutional layer or fully connected layer, and adjust the feature extraction process accordingly. In this way, the constructed model for feature extraction can more accurately capture the subtle differences in the image, thereby improving the accuracy and robustness of fine-grained classification tasks.

[0068] On this basis, in this embodiment, the preset encoder is an encoder that adopts a visual Transformer model based on an attention mechanism, which is composed of several layers of self-attention mechanisms.

[0069] It should be noted that the visual Transformer model based on the attention mechanism is the Transformer model (Vision Transformer, ViT) based on the visual self-attention mechanism. Vision Transformer is an innovative model that introduces the self-attention mechanism into the field of computer vision. The self-attention mechanism is a core component of the Transformer model, which enables the model to establish the association of the global context in the input sequence, so as to better understand the dependencies between different elements in the sequence. In this embodiment, each image block sample will be converted into a vector by the embedding layer, combined with the position encoding, and input into the multi-layer Transformer encoder for processing. In these encoders, the self-attention mechanism is used to calculate the attention distribution at each position, as well as the correlation between different positions. Through the self-attention mechanism, ViT can capture the complex relationship between different regions in the image, so as to better understand the image content, so that the final extracted feature vector contains both global image information and local detail features.

[0070] In step S250, the feature vectors of the image block samples are spliced ​​using position coding to obtain target feature samples.

[0071] It should be noted that this application adopts a deep learning architecture to extract local features of the image based on the method of image block splitting (PatchSplitting). In each block, a convolutional neural network (CNN) is used to mention the local detailed features of each block. Between each block, a visual attention mechanism (Vision Transformer) model is used to generate attention weights (Attention Weight) to locate feature blocks with high discriminant features. Finally, a feature fusion module is used to fuse the features of each block using position encoding to obtain the overall feature vector of the input image data sample.

[0072] In step S260, the target feature sample is input into a preset decoder to obtain a reconstructed image; wherein the preset decoder is a decoder of a visual Transformer model based on an attention mechanism.

[0073] It should be emphasized that the present application adopts an encoder-decoder architecture. In step S240, all image block samples are input into a preset encoder to obtain a feature vector of the image block sample. In fact, the preset encoder only processes unmasked image block samples, and the feature vector of the masked image block sample (masked image block sample) is 0, which can reduce the amount of calculation. Afterwards, in step S260, the decoder of the corresponding visual Transformer model based on the attention mechanism uses the output of the encoder to reconstruct the complete image, including those masked parts.

[0074] In other words, it can be understood that the masked image block samples (masked image block samples) are not sent to the Transformer encoder, but only to the decoder for restoration. For masked image block samples, the decoder needs to restore these missing image blocks based on the unmasked part and the zero value filling of the mask position, and restore the semantic information of the masked part as much as possible.

[0075] Through innovative masking strategies and asymmetric structures, efficient visual feature learning is achieved while reducing computational costs. This method not only performs well in self-supervised learning, but also shows excellent scalability and broad application prospects.

[0076] In step S270, based on the difference between the reconstructed image and the corresponding image data sample, the visual Transformer model based on the attention mechanism is trained until a preset end condition is reached.

[0077] The visual Transformer model based on the attention mechanism is self-supervised learning. By randomly masking most areas of the image and letting the model predict the masked parts, the model must infer global features from incomplete information without labeled data. This learning method prompts the model to not only focus on local details, but also learn the global semantic structure and contextual information of the image. This comprehensive feature extraction helps to improve the generalization ability of the model, making it more robust in open set recognition tasks. In addition, in open set recognition, since the model needs to be able to distinguish between unseen data and known data during training, the feature representation ability of the model trained according to the training method provided in the embodiment of the present application can better help the model capture the heterogeneity of novel and abnormal samples.

[0078] In step S280, an encoder is extracted from the trained attention-based visual Transformer model as a target Transformer encoder.

[0079] This embodiment uses a visual Transformer as a feature extraction module, which can effectively extract key features of the image; a mask block-based reconstruction self-supervised training method is used to train the model to obtain a target Transformer encoder, which can extract some task-independent features to facilitate open set recognition.

[0080] Furthermore, in the application process, after the target Transformer encoder outputs the target feature in step S140, the method further includes: inputting the target feature into the trained encoder to obtain a reconstructed image of the image data to be classified. On this basis, the image data to be classified, the reconstructed image of the image data to be classified, and the target category to which the image data to be classified output in step S150 belongs can be further used as a sample data and added to the sample data set for self-supervised learning.

[0081] According to some embodiments, reference Figure 3 In step S120, an open set recognition classifier is trained based on the sample data set, which can be specifically implemented through steps S310 to S320.

[0082] In step S310, meta-learning-based classification training and / or supervised training of known classes are performed based on the sample data set until a preset end condition is reached to obtain a recognition classifier.

[0083] It should be noted that the core goal of Open Set Recognition (OSR) is not only to accurately classify samples of known categories (i.e., categories that already exist in the training set) on the test set, but also to be able to detect samples of unknown categories (i.e., samples that do not belong to any known category in the training set). This ability is crucial for applications in the real world, because data in the real world is often open and dynamically changing. In the related art, the open set recognition method based on feature extraction is one of the early solutions. They extract specific features from the input image, such as texture features, shape features, etc., and then apply clustering or classification algorithms to process these features to achieve recognition of open set categories. The open set recognition method based on machine learning further uses machine learning algorithms to classify or regress the input image to achieve recognition of open set categories. The open set recognition method based on deep learning realizes automatic feature extraction and representation learning of data by constructing a multi-layer neural network. However, the performance of these methods may be limited when there are insufficient training samples or it is difficult to obtain annotations.

[0084] This application uses the core concept of meta-learning to perform basic training on the open set recognition classifier, so that it can quickly adapt to new tasks through a small number of samples, and work in a nearly unsupervised environment, significantly improving the model's ability to quickly learn and adapt to new categories. Meta-learning models focus on quickly extracting knowledge from limited samples, which enables them to effectively learn when encountering new categories, even with only a small amount of labeled data or without any labels at all. They can effectively cope with the constant changes in real-world data, thereby enhancing the generalization ability of the model. This ability is particularly important for application scenarios that need to quickly adapt to new situations without a large amount of re-labeling of data, and demonstrates the potential for effective open set recognition under resource-constrained conditions.

[0085] In addition, the open set recognition classifier can also be supervised for known classes. All categories are known during the training phase, and each category in the training data is representative enough so that the model can learn features that distinguish different categories.

[0086] It should be emphasized that both the classification training based on meta-learning and the supervised training of known classes are based on preset classifiers and are performed using sample data sets. In a specific embodiment, both types of training are performed simultaneously based on the sample data set. In this embodiment, the open set recognition classifier adopts meta-learning and cross-entropy loss training methods, and can simultaneously identify known class categories and discriminate unknown classes.

[0087] In step S320, the recognition classifier and the preset open set recognition post-processing algorithm are integrated to obtain an open set recognition classifier.

[0088] Since the OOD (Out-of-Distribution) detection task in open set recognition (OSR) mainly focuses on how to distinguish between data of known categories (ID, In-Distribution) and out-of-distribution data (OOD), this embodiment adds a post-processing algorithm based on the trained recognition classifier to achieve open set recognition.

[0089] Post-processing algorithms refer to further processing on the output of the recognition classifier to distinguish between ID and OOD samples. These methods are mainly based on the output of the recognition classifier (such as Softmax score, Logits, etc.), and detect OOD samples by adjusting or analyzing the output signal of the network.

[0090] According to an exemplary embodiment, the post-processing algorithms for preset open set recognition include ODIN based on maximum Softmax probability (MSP), temperature scaling and adversarial disturbance, maximum entropy, maximum Logits, Mahalanobis, Gaussian mixture model (GMM), etc. GradNorm and Class Prior may also be included. GradNorm detects OOD samples by calculating the gradient norm of the input sample to Logits, while Class Prior adjusts the classification probability using the class prior distribution. These methods process the output of the classifier in different ways to help distinguish ID from OOD samples, and can be flexibly adapted according to different task scenarios.

[0091] According to some embodiments, reference Figure 4 In step S310, classification training based on meta-learning is performed based on the sample data set, which can be specifically implemented through steps S410 to S430.

[0092] In step S410, a first distance between a target sample and its positive sample is calculated, and a second distance between a target sample and its negative sample is calculated; wherein the target sample is each image data sample in the sample data set, the positive sample is a sample in the sample data set that has the same category as the target sample, and the negative sample is a sample in the sample data set that has a different category from the target sample.

[0093] This embodiment is based on the meta-learning method of triplet loss (Triplet Loss), which further constrains the fine-grained features obtained by the fine-grained feature extraction model, making the inter-class distance of the features smaller and the inter-class distance larger. In the feature space, the unknown class samples have a larger distance from the known class samples, which helps the open set recognition module to better identify the unknown class samples.

[0094] Specifically, triplet loss is a method that uses distance metrics to achieve feature fusion. It minimizes the distance between samples of the same type and maximizes the distance between samples of different types, so that samples of the same type in the feature space are clustered together and samples of different types are dispersed. This method can effectively improve the model's discriminability for data, thereby improving the model's performance in classification tasks.

[0095] In the actual training process, a target sample x is selected from the sample data set. a , define the image data samples in the sample data set that are similar to the target samples as positive samples x p , define the image data samples in the sample data set that are different from the target samples as negative samples x n .

[0096] On this basis, the first distance D(x a ,xp ), and calculate the second distance D(x a ,x n ).

[0097] In step S420, a loss function is constructed according to the first distance and the second distance to ensure that the negative sample is farther from the anchor point than the positive sample by at least a preset margin.

[0098] After the first distance and the second distance are calculated, they meet the following conditions in the embedding space during the training process: the first distance D(x a ,x p ) is as small as possible, and the second distance D(x) between the target sample and the negative sample a ,x n ) as large as possible.

[0099] On this basis, the loss function Triplet Loss can be expressed as:

[0100] L = max(D(x a ,x p )-D(x a ,x n )+α,0),

[0101] Where D(x,y) is the distance metric between samples x and y. α is a hyperparameter called “margin” that is used to ensure that negative samples are far enough from the target samples.

[0102] It should be noted that the goal of this loss function is to ensure that the negative samples are at least a margin α farther from the anchor point than the positive samples.

[0103] In step S430, based on the sample data set, a loss function is used to perform classification training on a preset classifier.

[0104] During the training process, if the distance difference between the negative sample and the positive sample is less than α, a positive loss value will be generated, and the preset classifier will be driven to adjust the embedding space. If the distance difference is greater than or equal to α, the loss value is zero, and it will not affect the preset classifier. In this way, the training reaches the preset end condition, that is, the classification training based on meta-learning is completed.

[0105] Figure 5 The learning objective of Triplet Loss is shown, such as Figure 5 As shown in , before the network of the preset classifier is learned, the metric distance between the target sample and the positive sample may be large, and the metric distance between the target sample and the negative sample may be small, such as Figure 5As shown on the left, during the learning process of the network, the metric distance between the target sample and the positive sample will gradually decrease, while the metric distance between the target sample and the negative sample will gradually increase. The network will directly learn the separability between features: that is, the distance between features of the same class should be as small as possible, while the distance between features of different classes should be as large as possible. Through learning, the distance between classes is greater than the distance within the class, such as Figure 5 Shown on the right.

[0106] According to some embodiments, reference Figure 6 In step S310, supervised training of known classes is performed based on the sample data set, which can be specifically implemented through steps S610 to S630.

[0107] In step S610, a feature vector of the image data sample is extracted as an intermediate feature vector.

[0108] In step S620, the intermediate feature vector is input into a preset classifier to obtain a predicted category.

[0109] In step S630, a preset classifier is trained according to the difference between the predicted category and the category label corresponding to the image data sample.

[0110] The goal of supervised training of known classes is to minimize the difference between the predicted label and the true label. The supervised training of known classes is completed by inputting the feature vector of the extracted image data sample into a preset classifier, comparing the predicted result of the preset classifier with the true label of the image data sample, and minimizing their difference until the preset end condition is reached.

[0111] According to some embodiments, the intermediate feature vector is a target feature sample.

[0112] Specifically, according to the example embodiment, during the training process of the target Transformer encoder, the target feature samples extracted in step S250 are used as intermediate feature vectors in the supervised training process of known classes, that is, the training of the target Transformer encoder and the supervised training of the known classes of the open set recognition classifier are performed simultaneously, thereby speeding up the training progress while ensuring the accuracy and robustness of the trained classifier.

[0113] According to some embodiments, reference Figure 7 , input the target features into the open set recognition classifier to obtain the target category to which the image data to be classified belongs, which can be specifically implemented through steps S710 to S730.

[0114] In step S710, the target feature is obtained and input into the output result of the recognition classifier.

[0115] In step S720, the output result is post-processed using a preset open set recognition post-processing algorithm to obtain a target score.

[0116] In step S730, the target category to which the image data to be classified belongs is obtained according to the target score and a preset threshold.

[0117] In actual application, unknown categories and known categories are distinguished according to the output of the trained recognition classifier and the preset threshold. The preset threshold can be set according to the post-processing algorithm of the preset open set recognition and the actual output of the recognition classifier, and this application does not limit this.

[0118] For example, in one embodiment, the target score is based on the maximum Softmax probability, and the category with the largest probability is selected as the prediction result. If the maximum probability is lower than a set threshold, the input sample is marked as an unknown category.

[0119] The open set fine-grained recognition referred to in this application is intended to distinguish whether the input image data to be classified belongs to a threshold category or an unknown category. If it belongs to a known category, it can be further classified and identified; if it belongs to an unknown category, it can be marked as "unknown" and processed subsequently.

[0120] According to example embodiments, subsequent processing may include storage or recording, manual review, retraining models, adaptive learning, triggering specific decision processes, and the like.

[0121] In order to explain the open set fine-grained recognition method provided by the present application in more detail, taking the recognition of unknown ships as an example, a specific embodiment of the present application is given. Figure 8 As shown,

[0122] In this embodiment, the Vision Transformer (ViT) model is used as the basic module for feature extraction, and the MAE mechanism in self-supervised learning is combined to achieve semantic reconstruction of the target image. The model is designed to perform open set recognition, that is, to identify the parts of the input image that belong to known categories or unknown categories.

[0123] First, the input image data to be classified is processed. The input image data to be classified is divided into blocks to obtain multiple image blocks, and then some of the image blocks are randomly discarded (i.e., masks). The feature vector of each image block is obtained through linear projection, and the position encoding is added to it. It is sent to the target Transformer encoder of the visual Transformer to obtain the feature vector of the Transformer encoding. The feature vectors are spliced ​​together, and the missing positions are filled with 0 values ​​to make them consistent and corresponding to the number of original image blocks to obtain the target features. The spliced ​​target features are respectively input into the target Transformer encoder and the open set recognition classifier, and the decoder is used to output the reconstructed image, while the open set recognition classifier outputs the known class category and marks whether the sample is an unknown class.

[0124] This application combines the visual Transformer model with the MAE (Masked Autoencoder) reconstruction technology, and has several significant advantages for fine-grained open set recognition tasks. First, by dividing the input image into small patches (Patch) and performing position encoding, local features are effectively extracted, allowing the model to better capture subtle differences; secondly, the Transformer encoder uses the self-attention mechanism to enhance the understanding of the relationship between features and improve the ability to distinguish between known and unknown categories. In addition, with the assistance of the semantic reconstruction mechanism, the open set recognition classifier has a stronger ability to recognize unknown categories, reduces the risk of misidentification, improves the accuracy and robustness of fine-grained recognition, and achieves effective recognition of complex scenes.

[0125] The following describes an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, reference can be made to the method embodiment of the present application.

[0126] Fig. 9 A block diagram of an open set fine-grained recognition apparatus according to an exemplary embodiment is shown.

[0127] Fig. 9 The device shown can execute the aforementioned open set fine-grained recognition method according to the embodiment of the present application.

[0128] like Fig. 9 As shown, the open set fine-grained recognition device may include: a data set module 910, a model training module 920, a position encoding module 930, a mask encoding module 940 and a recognition classification module 950.

[0129] See also Fig. 9 And referring to the above description, the data set module 910 is used to construct a sample data set; wherein the sample data set includes a plurality of groups of image data samples with category annotations.

[0130] The model training module 920 is used to train a target Transformer encoder and an open set recognition classifier based on a sample data set.

[0131] The position coding module 930 is used to divide the acquired image data to be classified into a plurality of image blocks of a preset size, and add position codes to the image blocks.

[0132] The mask encoding module 940 is used to randomly mask the image blocks with position codes according to a preset ratio, input all the image blocks to the target Transformer encoder, and output the target features from the target Transformer encoder.

[0133] The recognition and classification module 950 is used to input the target features into an open set recognition classifier to obtain the target category to which the image data to be classified belongs.

[0134] The device performs functions similar to the method provided above. For other functions, please refer to the previous description and will not be repeated here.

[0135] Fig.10 An electronic device according to an exemplary embodiment of the present application is shown. Fig.10 The electronic device 1000 according to this embodiment of the present application is described. Fig. 9 The electronic device 1000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0136] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010), a display unit 1040, etc.

[0137] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 executes the methods described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 1010 can execute the above-mentioned method.

[0138] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .

[0139] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0140] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0141] The electronic device 1000 may also communicate with one or more external devices 300 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1060. The network adapter 1060 may communicate with other modules of the electronic device 1000 via the bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0142] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. The technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation method of the present application.

[0143] The software product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0144] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0145] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the aforementioned functions.

[0147] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0148] According to an embodiment of the present application, a computer program is provided, including a computer program or an instruction. When the computer program or the instruction is executed by a processor, the method described above can be executed.

[0149] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

[0150] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0151] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. An open set fine-grained recognition method, characterized in that: include: Constructing a sample data set; wherein the sample data set includes multiple groups of image data samples with category annotations; Based on the sample data set, a target Transformer encoder and an open set recognition classifier are trained; Dividing the acquired image data to be classified into a plurality of image blocks of a preset size, and adding position codes to the image blocks; Randomly masking the image blocks with the position codes according to a preset ratio, inputting all the image blocks into the target Transformer encoder, and outputting target features from the target Transformer encoder; The target feature is input into the open set recognition classifier to obtain the target category to which the image data to be classified belongs.

2. The method according to claim 1, characterized in that Based on the sample data set, the target Transformer encoder is trained, including: Dividing the image data sample into a plurality of image block samples of a preset size; Adding position codes to the image block samples; Randomly masking the image block samples according to a preset ratio; Input all image block samples into a preset encoder to obtain feature vectors of the image block samples; wherein the preset encoder is an encoder of a visual Transformer model based on an attention mechanism; Using the position code to splice the feature vectors of the image block samples to obtain a target feature sample; Inputting the target feature sample into a preset decoder to obtain a reconstructed image; wherein the preset decoder is a decoder of a visual Transformer model based on an attention mechanism; Based on the difference between the reconstructed image and the corresponding image data sample, training the visual Transformer model based on the attention mechanism until a preset end condition is reached; An encoder is extracted from the trained attention-based visual Transformer model as the target Transformer encoder.

3. The method according to claim 2, characterized in that Based on the sample data set, an open set recognition classifier is trained, including: Performing meta-learning-based classification training and / or supervised training of known classes based on the sample data set until a preset end condition is reached to obtain a recognition classifier; The recognition classifier is integrated with a post-processing algorithm of a preset open set recognition to obtain the open set recognition classifier.

4. The method according to claim 3, characterized in that Based on the sample data set, classification training based on meta-learning is performed, including: Calculate a first distance between a target sample and its positive sample, and calculate a second distance between the target sample and its negative sample; wherein the target sample is each image data sample in the sample data set, the positive sample is a sample in the sample data set with the same category as the target sample, and the negative sample is a sample in the sample data set with a different category from the target sample; constructing a loss function according to the first distance and the second distance to ensure that the negative sample is farther from the anchor point than the positive sample by at least a preset margin; Based on the sample data set, the preset classifier is trained for classification using the loss function.

5. The method according to claim 3, characterized in that: Performing supervised training of known classes based on the sample data set includes: Extracting a feature vector of the image data sample as an intermediate feature vector; Inputting the intermediate feature vector into a preset classifier to obtain a predicted category; The preset classifier is trained according to the difference between the predicted category and the category label corresponding to the image data sample.

6. The method according to claim 5, characterized in that The intermediate feature vector is the target feature sample.

7. The method according to claim 3, characterized in that Inputting the target feature into the open set recognition classifier to obtain the target category to which the image data to be classified belongs includes: Obtain the target feature and input it into the output result of the recognition classifier; Post-processing the output result using the preset open set recognition post-processing algorithm to obtain a target score; The target category to which the image data to be classified belongs is obtained according to the target score and a preset threshold.

8. An open set fine-grained recognition device, characterized in that: include: A data set module, used to construct a sample data set; wherein the sample data set includes multiple groups of image data samples with category annotations; A model training module, used for training a target Transformer encoder and an open set recognition classifier based on the sample data set; A position coding module, used for dividing the acquired image data to be classified into a plurality of image blocks of a preset size, and adding position codes to the image blocks; A mask encoding module, configured to randomly mask the image blocks with the position codes according to a preset ratio, input all the image blocks into the target Transformer encoder, and output target features from the target Transformer encoder; The recognition and classification module is used to input the target features into the open set recognition classifier to obtain the target category to which the image data to be classified belongs.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program or instruction is stored thereon, and when the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images