Open set fine-grained identification method and apparatus

Abstract

This application provides an open-set fine-grained recognition method and apparatus, relating to the field of image recognition technology. An open-set fine-grained recognition method includes: constructing a sample dataset; training a target Transformer encoder and an open-set recognition classifier based on the sample dataset; dividing the acquired image data to be classified into multiple image blocks of a preset size and adding positional encoding to the image blocks; randomly masking the positionally encoded image blocks according to a preset ratio, and inputting all image blocks into the target Transformer encoder to output target features; inputting the target features into the open-set recognition classifier to obtain the target category to which the image data to be classified belongs. The target features are also input into the decoder corresponding to the target Transformer encoder for reconstruction. According to the technical solution of this application embodiment, open-set fine-grained recognition is achieved in complex scenes with high accuracy and robustness.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, specifically to an open set fine-grained recognition method and apparatus. Background Technology

[0002] Currently, in the field of image recognition, especially in fine-grained classification tasks, identifying objects of different categories with similar features has always been a challenge. Traditional fine-grained recognition methods typically rely on large amounts of labeled data to train deep learning models. However, with only a small amount of labeled data or with no labeled data at all, traditional fine-grained recognition methods usually face a significant performance degradation problem.

[0003] Furthermore, in real-world applications, fine-grained identification of new or unknown categories often arises. Traditional closed-set fine-grained classification models are prone to misclassification when encountering these new categories, resulting in low accuracy. Summary of the Invention

[0004] Based on this, this application provides a method and apparatus for fine-grained open set recognition, which achieves high accuracy and robustness in fine-grained open set recognition.

[0005] According to one aspect of this application, an open-set fine-grained recognition method is proposed, comprising: constructing a sample dataset; wherein the sample dataset includes multiple sets of image data samples with category labels; training a target Transformer encoder and an open-set recognition classifier based on the sample dataset; dividing the acquired image data to be classified into multiple image blocks of a preset size and adding position encoding to the image blocks; randomly masking the image blocks with position encoding according to a preset ratio, inputting all image blocks into the target Transformer encoder, and outputting target features from the target Transformer encoder; inputting the target features into the open-set recognition classifier to obtain the target category to which the image data to be classified belongs.

[0006] According to some embodiments, a target Transformer encoder is trained based on a sample dataset, including: dividing the image data sample into multiple image patch samples of a preset size; adding positional encoding to the image patch samples; randomly masking the image patch samples according to a preset ratio; inputting all image patch samples into a preset encoder to obtain the feature vector of the image patch samples; wherein, the preset encoder is an encoder of a visual Transformer model based on an attention mechanism; concatenating the feature vectors of the image patch samples using positional encoding to obtain target feature samples; inputting the target feature samples into a preset decoder to obtain a reconstructed image; wherein, the preset decoder is a decoder of a visual Transformer model based on an attention mechanism; training the visual Transformer model based on an attention mechanism based on the difference between the reconstructed image and the corresponding image data sample until a preset termination condition is reached; and extracting the encoder from the trained visual Transformer model based on an attention mechanism as the target Transformer encoder.

[0007] According to some embodiments, an open set recognition classifier is trained based on a sample dataset, including: performing meta-learning-based classification training and / or supervised training of known classes based on the sample dataset until a preset termination condition is reached to obtain a recognition classifier; integrating the recognition classifier and a preset open set recognition post-processing algorithm to obtain an open set recognition classifier.

[0008] According to some embodiments, classification training based on a sample dataset includes: calculating a first distance between a target sample and its positive samples, and calculating a second distance between a target sample and its negative samples; wherein the target sample is each image data sample in the sample dataset, the positive sample is a sample in the sample dataset of the same class as the target sample, and the negative sample is a sample in the sample dataset of the different class from the target sample; constructing a loss function based on the first and second distances to ensure that the negative sample is at least one preset margin further away from the anchor point than the positive sample; and training a preset classifier for classification based on the sample dataset using the loss function.

[0009] According to some embodiments, supervised training of known classes based on sample datasets includes: extracting feature vectors from image data samples as intermediate feature vectors; inputting the intermediate feature vectors into a preset classifier to obtain predicted categories; and training the preset classifier based on the difference between the predicted categories and the category labels corresponding to the image data samples.

[0010] According to some embodiments, the intermediate feature vector is the target feature sample.

[0011] According to some embodiments, the target features are input into an open set recognition classifier to obtain the target category to which the image data to be classified belongs, including: obtaining the output result obtained by inputting the target features into the recognition classifier; performing post-processing on the output result using a preset open set recognition post-processing algorithm to obtain a target score; and obtaining the target category to which the image data to be classified belongs based on the target score and a preset threshold.

[0012] According to one aspect of this application, an open-set fine-grained recognition device includes: a dataset module for constructing a sample dataset; wherein the sample dataset includes multiple sets of image data samples with category labels; a model training module for training a target Transformer encoder and an open-set recognition classifier based on the sample dataset; a position encoding module for dividing the acquired image data to be classified into multiple image blocks of a preset size and adding position encoding to the image blocks; a mask encoding module for randomly masking the image blocks with position encoding according to a preset ratio, inputting all image blocks into the target Transformer encoder, and outputting target features from the target Transformer encoder; and a recognition and classification module for inputting the target features into the open-set recognition classifier to obtain the target category to which the image data to be classified belongs.

[0013] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0014] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0015] Through the embodiments provided in this application, the input image data to be classified is divided into image blocks and positionally encoded, thereby effectively extracting local features and enabling the model to better capture subtle differences. A sample dataset is constructed to train the target Transformer encoder and the open set recognition classifier. In practical applications, by using the attention capability of the target Transformer encoder, global and local features of the image data to be classified can be quickly extracted. The extracted features are input into the open set recognition classifier to achieve fine-grained classification of known classes and detection of unknown classes, improving the accuracy and robustness of fine-grained recognition and achieving effective recognition of complex scenes. Attached Figure Description

[0016] It should be understood that the foregoing general description and the following detailed description are merely illustrative and are not restrictive of the present application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0018] Figure 1 A flowchart of the open set fine-grained recognition method provided in the embodiments of this application;

[0019] Figure 2 A flowchart of the training target Transformer encoder provided in the embodiments of this application;

[0020] Figure 3 A flowchart illustrating the training of an open-set recognition classifier provided in this application embodiment;

[0021] Figure 4 A flowchart illustrating meta-learning-based classification training based on a sample dataset, provided for embodiments of this application;

[0022] Figure 5 A schematic diagram of the learning objective for classification training based on meta-learning provided in an embodiment of this application;

[0023] Figure 6 A flowchart illustrating supervised training of known classes based on a sample dataset, provided for embodiments of this application;

[0024] Figure 7 A flowchart illustrating how target features are input into an open-set recognition classifier to obtain the target category of the image data to be classified, provided in this embodiment of the application;

[0025] Figure 8 An overall framework diagram of the open set fine-grained recognition method provided in the embodiments of this application;

[0026] Figure 9 A block diagram of the open set fine-grained recognition device provided in the embodiments of this application;

[0027] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0032] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0033] For specific implementation details, please refer to the following examples.

[0034] Figure 1 A flowchart illustrating the open set fine-grained identification method provided in this application embodiment. Figure 1 As shown, the method includes steps S110-S150.

[0035] In step S110, a sample dataset is constructed; wherein, the sample dataset includes multiple sets of image data samples with category labels.

[0036] This application does not restrict the method of obtaining image data samples; data from publicly available image datasets, as well as collected or synthesized image data, can be used. To further accelerate training, the image data can be representative, diverse, and sufficient in quantity so that the model can quickly learn effective features.

[0037] In addition, the category labels of image data samples can be provided by the image data samples themselves, or they can be manually labeled or labeled using labeling tools. This application does not impose any restrictions on this.

[0038] In step S120, a target Transformer encoder and an open set recognition classifier are trained based on the sample dataset.

[0039] During the training of a target Transformer encoder, the training objective can be: to input new image data into the target Transformer encoder, and based on a comprehensive understanding of the entire image, directly extract and output discriminative feature information. Similarly, during the training of an open-set recognition classifier, the training objective can be: to input new feature information into the open-set recognition classifier, achieving fine-grained classification of known classes and detection of unknown classes.

[0040] In step S130, the acquired image data to be classified is divided into multiple image blocks of a preset size, and position codes are added to the image blocks.

[0041] In practical applications, the image data to be classified is the input image data waiting to be classified and recognized using open set fine-grained classification. The image data to be classified is divided into multiple image blocks of a preset size.

[0042] It should be noted that the input image data to be classified is divided into different local regions, and each region is input into the target Transformer encoder for feature extraction, which makes it easier for the model to capture the subtle differences in the image data more accurately.

[0043] Furthermore, according to the example embodiment, the image data to be classified is divided into N image blocks using a convolutional sliding window or direct segmentation. Regarding the segmentation method, block division generally includes overlapping block division and non-overlapping block division. Overlapping refers to the existence of some common areas between the divided image blocks. It should be noted that in a specific embodiment, the preset size can be 16x16 or 32x32.

[0044] In addition, this application does not restrict the direction of image block division. It can be divided according to the target to be identified included in the image data to be classified. Taking the image data to be classified containing ships as an example, since ships are generally long and narrow, multiple image blocks can be divided along the length of the ship.

[0045] Furthermore, after the image patch is divided, it can be flattened into a one-dimensional feature vector. This step linearizes the pixel data in the image patch for processing in subsequent steps.

[0046] After the image is divided, a positional encoding is added to each image patch. It should be noted that the positional encoding is used to preserve the spatial location information of the image patch and prevent the loss of spatial relationships when input into the Transformer.

[0047] Furthermore, according to the example embodiment, the positional encoding is typically calculated in a specific manner, which can be fixed or learned. Fixed positional encoding is usually generated using different frequencies of sine and cosine functions. The advantage of this approach is that the model can handle sequences of arbitrary length because the positional encoding is infinitely scalable. Learned positional encoding, on the other hand, is automatically optimized through the training process to better suit the specific task requirements of this project. The positional encoding is added to the vector obtained after flattening the image patch to form the final input representation.

[0048] In step S140, the image blocks with position encoding are randomly masked according to a preset ratio, and all image blocks are input to the target Transformer encoder, and the target features are output from the target Transformer encoder.

[0049] To achieve semantic reconstruction, the model employs a masking mechanism based on the MAE architecture. In practical applications, input image patches are randomly masked according to a preset ratio.

[0050] According to the example embodiment, the preset ratio can be 75%, that is, the image data to be classified is divided into multiple patches (image blocks), and 75% of the patches are randomly occluded, allowing the model to see only 25% of the unoccluded parts.

[0051] Next, all image patches (including masked and unmasked patches) are fed into the target Transformer encoder. The target Transformer encoder processes these input image patches to extract global features of the image, obtaining Transformer-encoded feature vectors. These feature vectors are then concatenated, and missing positions are filled with 0 values ​​to ensure they match and correspond to the number of original image patches, thus obtaining the target features.

[0052] In step S150, the target features are input into the open set recognition classifier to obtain the target category to which the image data to be classified belongs.

[0053] This application employs an open set recognition classifier (OSR Classifier) ​​to distinguish whether an input image belongs to a known category or an unknown category. It should be noted that a known category refers to a category that the model can recognize and classify, while an unknown category refers to a category that the model cannot recognize or that belongs to an open category.

[0054] This application effectively extracts local features by dividing the input image data to be classified into image patches and performing positional encoding, enabling the model to better capture subtle differences. A sample dataset is constructed to train the target Transformer encoder and the open set recognition classifier. In practical applications, by using the attention capability of the target Transformer encoder, global and local features of the image data to be classified can be quickly extracted. The extracted features are then input into the open set recognition classifier to achieve fine-grained classification of known classes and detection of unknown classes, improving the accuracy and robustness of fine-grained recognition and enabling effective recognition of complex scenes.

[0055] According to some embodiments, refer to Figure 2 In step S120, the target Transformer encoder is trained based on the sample dataset, which can be achieved through steps S210-S280.

[0056] In step S210, the image data sample is divided into multiple image block samples of a preset size.

[0057] The method for dividing image data samples during training is the same as the method for dividing image data to be classified during application, and will not be elaborated here.

[0058] In step S220, positional encoding is added to the image block samples.

[0059] It should be noted that positional encoding is used to address the problem that self-attention mechanisms cannot capture positional information within a sequence. In the high-discriminative feature extraction task of this application, the order of image patches is crucial for understanding the overall features, while the Transformer's self-attention mechanism itself does not consider the position of image patches in the sequence. Therefore, the introduction of positional encoding enables the Transformer to fully understand the order information in the sequence of image patches.

[0060] The method for adding positional encoding to image patch samples during training is the same as the method for adding positional encoding to image patches during application, and will not be described in detail here.

[0061] Image patch samples with added positional encoding are input into the model for training, enabling the model to consider both the semantic information of the image patch and its positional information in the sequence in the self-attention mechanism.

[0062] In step S230, the image block samples are randomly masked according to a preset ratio.

[0063] Image patch samples are randomly masked according to a preset ratio for subsequent self-supervised learning training. The method for randomly masking image patch samples during training is the same as the method for randomly masking image patches during application, and will not be described in detail here.

[0064] In step S240, all image patch samples are input into a preset encoder to obtain the feature vector of the image patch samples; wherein, the preset encoder is an encoder of a visual Transformer model based on an attention mechanism.

[0065] It should be noted that efficiently extracting highly discriminative features is crucial for accurately identifying subtle differences between target categories. These differences may originate from the target's appearance, details, and color texture. Within the framework of deep learning, local feature extraction fusion and attention mechanisms are two effective key technologies in this field, capable of effectively capturing highly discriminative features. This application utilizes local feature extraction fusion and attention mechanisms, using an attention-based visual Transformer model as the basic model block for feature extraction. A MAE semantic reconstruction architecture is employed to train the model through self-supervised learning, fully leveraging the attention capabilities of the Transformer model.

[0066] Highly discriminative features are typically composed of multiple local, detailed features. By employing deep learning-based local feature extraction and fusion, fine-grained features of the target can be effectively extracted, and these features can be further combined to form highly efficient discriminative features. This application divides the input image data into multiple small image patches, which are then further input into a neural network for local feature extraction.

[0067] Furthermore, to enhance the model's ability to perceive fine-grained features, an attention mechanism is introduced into the network structure. The model can automatically learn the importance of different regions after each convolutional or fully connected layer and adjust the feature extraction process accordingly. This results in a feature extraction model that more accurately captures subtle differences in images, thereby improving the accuracy and robustness of fine-grained classification tasks.

[0068] Based on this, in this embodiment, the preset encoder is an encoder that adopts a visual Transformer model based on an attention mechanism, which consists of several layers of self-attention mechanisms.

[0069] It should be noted that the attention-based visual Transformer model is specifically the Vision Transformer (ViT) model based on visual self-attention. The Vision Transformer is an innovative model that introduces self-attention mechanisms into the field of computer vision. Self-attention is a core component of the Transformer model, enabling it to establish global contextual relationships within the input sequence, thus better understanding the dependencies between different elements in the sequence. In this embodiment, each image patch sample is transformed into a vector through an embedding layer and combined with positional encoding before being input into a multi-layer Transformer encoder for processing. In these encoders, self-attention is used to compute the attention distribution at each location and the correlation between different locations. Through self-attention, ViT can capture the complex relationships between different regions in the image, thereby better understanding the image content, ensuring that the final extracted feature vector contains both global image information and local detailed features.

[0070] In step S250, the feature vectors of image block samples are spliced ​​using position encoding to obtain target feature samples.

[0071] It should be noted that this application employs a deep learning architecture, extracting local features of the image based on image patch splitting. Within each patch, a convolutional neural network (CNN) is used to extract local detailed features. Between patches, a vision transformation model is used to generate attention weights, locating feature blocks with high discriminative features. Finally, a feature fusion module is used to fuse the features of each patch using positional encoding to obtain the overall feature vector of the input image data sample.

[0072] In step S260, the target feature samples are input to a preset decoder to obtain a reconstructed image; wherein, the preset decoder is a decoder of a visual Transformer model based on an attention mechanism.

[0073] It is important to emphasize that this application employs an encoder-decoder architecture. In step S240, all image patch samples are input into a preset encoder to obtain the feature vectors of the image patch samples. In fact, the preset encoder only processes unmasked image patch samples; the feature vectors of masked image patch samples (masked image patch samples) are 0, thus reducing computational load. Subsequently, in step S260, the corresponding decoder of the attention-based visual Transformer model uses the encoder's output to reconstruct the complete image, including the masked parts.

[0074] In other words, it can be understood that the masked image patch samples (masked image patch samples) are not fed into the Transformer encoder, but only into the decoder for reconstruction. For the masked image patch samples, the decoder needs to fill in the missing image patches based on the unmasked parts and the zero values ​​at the mask positions to recover these missing image patches, and to recover the semantic information of the masked parts as much as possible.

[0075] Through innovative occlusion strategies and asymmetric structures, this method achieves efficient visual feature learning capabilities while reducing computational costs. It not only excels in self-supervised learning but also demonstrates outstanding scalability and broad application prospects.

[0076] In step S270, the visual Transformer model based on the attention mechanism is trained based on the difference between the reconstructed image and the corresponding image data sample until the preset termination condition is met.

[0077] Self-supervised learning is applied to an attention-based visual Transformer model. By randomly occluding most of an image, the model is asked to predict the occluded portion. The model must infer global features from incomplete information without labeled data. This learning approach encourages the model to not only focus on local details but also learn the global semantic structure and contextual information of the image. This comprehensive feature extraction helps improve the model's generalization ability, making it more robust in open-set recognition tasks. Furthermore, in open-set recognition, since the model needs to distinguish between unseen data and known data from training, the feature representation ability of the model trained using the method provided in this application can better help the model capture the heterogeneity of novel and anomalous samples.

[0078] In step S280, the encoder is extracted from the trained attention-based visual Transformer model as the target Transformer encoder.

[0079] This embodiment uses a visual Transformer as the feature extraction module, which can effectively extract key features of the image; it uses a self-supervised training method based on mask block reconstruction to train the model, thereby obtaining the target Transformer encoder, which can extract some task-independent features to facilitate open set recognition.

[0080] Furthermore, in the application process, after the target Transformer encoder outputs the target features in step S140, the method further includes: inputting the target features into the trained encoder to obtain the reconstructed image of the image data to be classified. Based on this, the image data to be classified, the reconstructed image of the image data to be classified, and the target category of the image data to be classified output in step S150 can be added to the sample dataset as a sample data set for self-supervised learning.

[0081] According to some embodiments, refer to Figure 3 In step S120, an open set recognition classifier is trained based on the sample dataset, which can be specifically achieved through steps S310-S320.

[0082] In step S310, meta-learning-based classification training and / or supervised training of known classes are performed based on the sample dataset until a preset termination condition is met, thus obtaining a recognition classifier.

[0083] It's important to note that the core objective of Open Set Recognition (OSR) is not only to accurately classify samples of known categories (those already existing in the training set) on the test set, but also to detect samples of unknown categories (those not belonging to any known category in the training set). This capability is crucial for real-world applications because real-world data is often open and dynamically changing. In related technologies, feature extraction-based open set recognition methods were among the early solutions. They extract specific features from the input image, such as texture and shape features, and then apply clustering or classification algorithms to process these features, achieving the recognition of open set categories. Machine learning-based open set recognition methods further utilize machine learning algorithms to classify or regress the input image to achieve open set category recognition. Deep learning-based open set recognition methods achieve automatic feature extraction and representation learning of data by constructing multi-layer neural networks. However, the performance of these methods may be limited when faced with insufficient training samples or difficulty in obtaining labeled data.

[0084] This application employs the core concept of meta-learning to train an open-set recognition classifier, enabling it to rapidly adapt to new tasks with limited samples and operate in near-unsupervised environments. This significantly improves the model's ability to quickly learn and adapt to new categories. Meta-learning models focus on rapidly extracting knowledge from limited samples, allowing them to learn effectively even with only a small amount of labeled data or no labeled data when encountering new categories. This effectively addresses the continuous changes in real-world data, thereby enhancing the model's generalization ability. This capability is particularly important for applications requiring rapid adaptation to new situations without the need for extensive relabeling, demonstrating the potential for effective open-set recognition under resource-constrained conditions.

[0085] In addition, the open set recognition classifier can be trained under the supervision of known classes. All classes are known during the training phase, and each class in the training data is sufficiently representative, enabling the model to learn the features that distinguish different classes.

[0086] It is important to emphasize that both the meta-learning-based classification training and the supervised training of known classes are based on a preset classifier and are performed using a sample dataset. In one specific embodiment, both training methods are performed simultaneously based on the sample dataset. In this embodiment, the open set recognition classifier is trained using meta-learning and cross-entropy loss, and can simultaneously identify known class categories and distinguish unknown classes.

[0087] In step S320, the recognition classifier and the preset open set recognition post-processing algorithm are integrated to obtain the open set recognition classifier.

[0088] Since the OOD (Out-of-Distribution) detection task in Open Set Recognition (OSR) mainly focuses on how to distinguish between data with known categories (ID, In-Distribution) and out-of-distribution data (OOD), this embodiment adds a post-processing algorithm based on the trained recognition classifier to achieve open set recognition.

[0089] Post-processing algorithms refer to further processing the output of the recognition classifier to distinguish between ID and OOD samples. These methods are mainly based on the output of the recognition classifier (e.g., Softmax score, Logits, etc.), and detect OOD samples by adjusting or analyzing the network's output signal.

[0090] According to an exemplary embodiment, the post-processing algorithms for predefined open set recognition include those based on maximum softmax probability (MSP), temperature scaling and adversarial perturbation ODIN, maximum entropy, maximum logits, Mahalanobis distance, and Gaussian mixture models (GMM). GradNorm and Class Prior may also be included. GradNorm detects OOD samples by calculating the gradient norm of the input sample with respect to logits, while Class Prior adjusts the classification probability using a prior distribution of categories. These methods process the classifier output in different ways, helping to distinguish between ID and OOD samples, and can be flexibly adapted to different task scenarios.

[0091] According to some embodiments, refer to Figure 4 In step S310, classification training based on meta-learning is performed on the sample dataset, which can be implemented through steps S410-S430.

[0092] In step S410, a first distance between the target sample and its positive sample is calculated, and a second distance between the target sample and its negative sample is calculated; wherein, the target sample is each image data sample in the sample dataset, the positive sample is the sample in the sample dataset that has the same category as the target sample, and the negative sample is the sample in the sample dataset that has a different category than the target sample.

[0093] This embodiment uses a meta-learning approach based on triplet loss to further constrain the fine-grained features obtained by the fine-grained feature extraction model, resulting in smaller inter-class distances and larger inter-class distances. Consequently, in the feature space, unknown class samples have a larger distance from existing class samples, helping the open set recognition module to better identify unknown class samples.

[0094] Specifically, triplet loss is a method for feature fusion that utilizes distance metrics. It minimizes the distance between similar samples while maximizing the distance between dissimilar samples, thus clustering similar samples together and dispersing dissimilar samples in the feature space. This method effectively improves the model's discriminative power, thereby enhancing its performance in classification tasks.

[0095] In the actual training process, a target sample x is selected from the sample dataset. a Define image data samples in the sample dataset that are of the same type as the target sample as positive samples x. p Image data samples in the sample dataset that belong to a different class than the target sample are defined as negative samples x. n .

[0096] Based on this, the first distance D(x) between the target sample and the positive sample is calculated. a ,xp ), and calculate the second distance D(x) between the target sample and the negative sample. a ,x n ).

[0097] In step S420, a loss function is constructed based on the first distance and the second distance to ensure that the negative sample is at least one preset margin further away from the anchor point than the positive sample.

[0098] After calculating the first and second distances, during training, they satisfy the following condition in the embedding space: the first distance D(x) between the target sample and the positive sample... a ,x p The second distance D(x) between the target sample and the negative sample should be as small as possible. a ,x n Make it as large as possible.

[0099] Based on this, the loss function Triplet Loss can be expressed as:

[0100] L = max(D(x) a ,x p )-D(x a ,x n )+α,0),

[0101] Here, D(x,y) is the distance metric between samples x and y. α is a hyperparameter called the "margin," used to ensure that negative samples are sufficiently far from the target samples.

[0102] It should be noted that the goal of this loss function is to ensure that negative samples are at least one margin α further from the anchor point than positive samples.

[0103] In step S430, a preset classifier is trained using a loss function based on the sample dataset.

[0104] During training, if the distance difference between negative and positive samples is less than α, a positive loss value is generated, driving the preset classifier to adjust the embedding space. If the distance difference is greater than or equal to α, the loss value is zero and has no effect on the preset classifier. Thus, training reaches the preset termination condition, completing the meta-learning-based classification training.

[0105] Figure 5 The learning objectives of Triplet Loss are shown, such as... Figure 5 As shown, before the network of the preset classifier has been trained, the metric distance between the target sample and the positive sample may be large, while the metric distance between the target sample and the negative sample may be small. Figure 5As shown on the left, during the network's learning process, the metric distance between the target sample and the positive sample gradually decreases, while the metric distance between the target sample and the negative sample gradually increases. The network directly learns the separability between features: that is, the distance between features of the same class should be as small as possible, while the distance between features of different classes should be as large as possible. Through learning, the distance between classes is made greater than the distance within classes, such as... Figure 5 As shown on the right.

[0106] According to some embodiments, refer to Figure 6 In step S310, supervised training of known classes is performed based on the sample dataset, which can be implemented through steps S610-S630.

[0107] In step S610, the feature vectors of the image data samples are extracted as intermediate feature vectors.

[0108] In step S620, the intermediate feature vector is input into the preset classifier to obtain the predicted category.

[0109] In step S630, a preset classifier is trained based on the difference between the predicted category and the category label corresponding to the image data sample.

[0110] The goal of supervised training for known classes is to minimize the difference between the predicted label and the true label. This is achieved by inputting the feature vector of the extracted image data sample into a pre-defined classifier, comparing the prediction result of the pre-defined classifier with the true label of the image data sample, and minimizing their difference until a pre-defined termination condition is met.

[0111] According to some embodiments, the intermediate feature vector is the target feature sample.

[0112] Specifically, according to the example embodiment, during the training process of the target Transformer encoder, the target feature samples extracted in step S250 are used as intermediate feature vectors in the supervised training process of the known class. That is, the training of the target Transformer encoder and the supervised training of the known class of the open set recognition class are carried out simultaneously, thereby accelerating the training progress while ensuring the accuracy and robustness of the trained classifier.

[0113] According to some embodiments, refer to Figure 7 The target features are input into the open set recognition classifier to obtain the target category to which the image data to be classified belongs. This can be achieved through steps S710-S730.

[0114] In step S710, the target features are input into the recognition classifier to obtain the output result.

[0115] In step S720, the output result is post-processed using a preset open set recognition post-processing algorithm to obtain the target score.

[0116] In step S730, the target category of the image data to be classified is obtained based on the target score and the preset threshold.

[0117] In practical applications, the unknown and known categories are distinguished based on the output of the trained recognition classifier and a preset threshold. The preset threshold can be set according to the preset open set recognition post-processing algorithm and the actual output of the recognition classifier; this application does not impose any restrictions on it.

[0118] For example, in one embodiment, the target score is based on the maximum softmax probability, selecting the category with the highest probability as the prediction result. If the maximum probability is lower than a set threshold, the input sample is labeled as an unknown category.

[0119] The open set fine-grained recognition referred to in this application aims to distinguish whether the input image data to be classified belongs to a threshold category or an unknown category. If it belongs to a known category, further fine-grained classification recognition can be performed; if it belongs to an unknown category, it can be marked as "unknown" and processed subsequently.

[0120] According to the example embodiment, subsequent processing may include storage or recording, manual review, retraining the model, adaptive learning, triggering specific decision-making processes, etc.

[0121] To provide a more detailed explanation of the open set fine-grained identification method provided in this application, a specific embodiment of this application is given, taking the identification of unknown types of ships as an example. Figure 8 As shown,

[0122] In this embodiment, the Vision Transformer (ViT) model is used as the basic module for feature extraction, and the MAE mechanism in self-supervised learning is combined to achieve semantic reconstruction of the target image. This model aims to perform open-set recognition, that is, to identify the parts of the input image that belong to known or unknown categories.

[0123] First, the input image data to be classified is processed by dividing it into multiple image patches. Then, some of these patches are randomly discarded (i.e., masked). The feature vector of each patch is obtained through linear projection, and positional encoding is added. This vector is then fed into the target Transformer encoder of the visual Transformer to obtain the Transformer-encoded feature vector. The feature vectors are concatenated, and missing positions are filled with 0 values ​​to ensure they correspond to the number of original image patches, thus obtaining the target features. The concatenated target features are then input into the target Transformer encoder and the open-set recognition classifier. The decoder outputs the reconstructed image, while the open-set recognition classifier outputs the known class and indicates whether the sample belongs to an unknown class.

[0124] This application combines a visual Transformer model with MAE (Masked Autoencoder) reconstruction technology, offering several significant advantages for fine-grained open-set recognition tasks. First, by dividing the input image into small patches and performing positional encoding, local features are effectively extracted, enabling the model to better capture subtle differences. Second, the Transformer encoder utilizes a self-attention mechanism to enhance the understanding of relationships between features, improving the ability to distinguish between known and unknown categories. Furthermore, with the assistance of a semantic reconstruction mechanism, the open-set recognition classifier has a stronger ability to identify unknown categories, reducing the risk of misidentification and improving the accuracy and robustness of fine-grained recognition, thus achieving effective recognition of complex scenes.

[0125] The following describes an apparatus embodiment of the present application, which can be used to perform the method embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, reference can be made to the method embodiment of the present application.

[0126] Figure 9 A block diagram of an open set fine-grained identification device according to an exemplary embodiment is shown.

[0127] Figure 9 The apparatus shown can perform the open set fine-grained identification method described above according to the embodiments of this application.

[0128] like Figure 9 As shown, the open set fine-grained recognition device may include: a dataset module 910, a model training module 920, a position encoding module 930, a mask encoding module 940, and a recognition and classification module 950.

[0129] See Figure 9 Referring to the preceding description, the dataset module 910 is used to construct a sample dataset; wherein, the sample dataset includes multiple sets of image data samples with category labels.

[0130] The model training module 920 is used to train the target Transformer encoder and open set recognition classifier based on the sample dataset.

[0131] The location encoding module 930 is used to divide the acquired image data to be classified into multiple image blocks of a preset size and add location encoding to the image blocks.

[0132] The masking module 940 is used to randomly mask the image blocks with positional encoding according to a preset ratio, input all image blocks to the target Transformer encoder, and output the target features from the target Transformer encoder.

[0133] The recognition and classification module 950 is used to input target features into the open set recognition classifier to obtain the target category to which the image data to be classified belongs.

[0134] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0135] Figure 10 An electronic device according to an exemplary embodiment of this application is shown. Reference is made below. Figure 10 To describe an electronic device 1000 according to this embodiment of the present application. Figure 9 The electronic device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0136] like Figure 10 As shown, the electronic device 1000 is presented in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010), a display unit 1040, etc.

[0137] The storage unit stores program code that can be executed by the processing unit 1010, causing the processing unit 1010 to perform the methods described in the various exemplary embodiments of this application. For example, the processing unit 1010 can perform the methods described above.

[0138] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.

[0139] Storage unit 1020 may also include a program / utility 10204 having a set (at least one) program module 10205, such program module 10205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0140] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0141] Electronic device 1000 can also communicate with one or more external devices 300 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. Network adapter 1060 can communicate with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0142] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this application.

[0143] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0144] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0145] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0147] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0148] According to an embodiment of this application, a computer program is proposed, including a computer program or instructions, which, when executed by a processor, can perform the methods described above.

[0149] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0150] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0151] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A fine-grained open set recognition method, characterized in that, include: Construct a sample dataset; wherein the sample dataset includes multiple sets of image data samples with category labels; Based on the sample dataset, a target Transformer encoder and an open set recognition classifier are trained. The acquired image data to be classified is divided into multiple image blocks of a preset size, and positional encoding is added to the image blocks; The image blocks with the location codes are randomly masked according to a preset ratio, and all the image blocks are input to the target Transformer encoder, and the target features are output from the target Transformer encoder; The target features are input into the open set recognition classifier to obtain the target category to which the image data to be classified belongs. Based on the aforementioned sample dataset, a target Transformer encoder is trained, including: The image data sample is divided into multiple image block samples of a preset size; Add positional encoding to the image patch samples; The image block samples are randomly masked according to a preset ratio; All image patch samples are input into a preset encoder to obtain the feature vector of the image patch samples; wherein, the preset encoder is an encoder of a visual Transformer model based on an attention mechanism, the feature vector of the image patch samples is the feature vector of the image patch samples without random masking, and the feature vector of the image patch samples with random masking is 0; The feature vectors of the image block samples are concatenated using the location encoding to obtain the target feature samples; The target feature samples are input into a preset decoder to obtain a reconstructed image; wherein, the preset decoder is a decoder of a visual Transformer model based on an attention mechanism; Based on the difference between the reconstructed image and the corresponding image data sample, the attention-based visual Transformer model is trained until a preset termination condition is met. The encoder is extracted from the trained attention-based visual Transformer model as the target Transformer encoder; Randomly masking the image block samples according to a preset ratio includes: randomly selecting image block samples of a preset ratio from the plurality of image block samples, and masking the selected image block samples.

2. The method according to claim 1, characterized in that, Based on the aforementioned sample dataset, an open-set recognition classifier is trained, including: Based on the sample dataset, perform meta-learning-based classification training and / or supervised training of known classes until a preset termination condition is met to obtain a recognition classifier. The open set recognition classifier is obtained by integrating the recognition classifier and the preset open set recognition post-processing algorithm.

3. The method according to claim 2, characterized in that, Meta-learning-based classification training is performed based on the aforementioned sample dataset, including: Calculate a first distance between a target sample and its positive samples, and calculate a second distance between the target sample and its negative samples; wherein, the target sample is each image data sample in the sample dataset, the positive sample is a sample in the sample dataset that is of the same category as the target sample, and the negative sample is a sample in the sample dataset that is of a different category than the target sample; A loss function is constructed based on the first distance and the second distance to ensure that negative samples are at least one preset margin further from the anchor point than positive samples. Based on the sample dataset, the preset classifier is trained using the loss function.

4. The method according to claim 2, characterized in that, Supervised training of known classes based on the sample dataset includes: Extract the feature vectors of the image data samples as intermediate feature vectors; The intermediate feature vector is input into a preset classifier to obtain the predicted category; The preset classifier is trained based on the difference between the predicted category and the category label corresponding to the image data sample.

5. The method according to claim 4, characterized in that, The intermediate feature vector is the target feature sample.

6. The method according to claim 2, characterized in that, The target features are input into the open set recognition classifier to obtain the target category to which the image data to be classified belongs, including: The target features are input into the recognition classifier to obtain the output result; The output result is post-processed using the preset open set recognition post-processing algorithm to obtain the target score; Based on the target score and the preset threshold, the target category to which the image data to be classified belongs is obtained.

7. An open-set fine-grained identification device, characterized in that, include: The dataset module is used to construct a sample dataset; wherein, the sample dataset includes multiple sets of image data samples with category labels; The model training module is used to train the target Transformer encoder and the open set recognition classifier based on the sample dataset. The location encoding module is used to divide the acquired image data to be classified into multiple image blocks of a preset size, and add location encoding to the image blocks; The masking encoding module is used to randomly mask the image blocks with the location encoding according to a preset ratio, input all the image blocks into the target Transformer encoder, and output the target features from the target Transformer encoder; The identification and classification module is used to input the target features into the open set identification classifier to obtain the target category to which the image data to be classified belongs. The model training module is further used for: The image data sample is divided into multiple image block samples of a preset size; Add positional encoding to the image patch samples; The image block samples are randomly masked according to a preset ratio; All image patch samples are input into a preset encoder to obtain the feature vector of the image patch samples; wherein, the preset encoder is an encoder of a visual Transformer model based on an attention mechanism, the feature vector of the image patch samples is the feature vector of the image patch samples without random masking, and the feature vector of the image patch samples with random masking is 0; The feature vectors of the image block samples are concatenated using the location encoding to obtain the target feature samples; The target feature samples are input into a preset decoder to obtain a reconstructed image; wherein, the preset decoder is a decoder of a visual Transformer model based on an attention mechanism; Based on the difference between the reconstructed image and the corresponding image data sample, the attention-based visual Transformer model is trained until a preset termination condition is met. The encoder is extracted from the trained attention-based visual Transformer model as the target Transformer encoder; Randomly masking the image block samples according to a preset ratio includes: randomly selecting image block samples of a preset ratio from the plurality of image block samples, and masking the selected image block samples.

8. An electronic device, characterized in that, include: one or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described in any one of claims 1-6.

Images