Open set fine-grained recognition method and apparatus

Abstract

The application provides an open set fine-grained recognition method and device, and relates to the technical field of image recognition. The method comprises the following steps: obtaining a multi-modal feature extraction model based on fine-grained recognition dataset training, and performing contrast learning on the aligned visual feature samples and text feature samples in the training process; inputting to-be-recognized image data into a preset visual feature extraction network to obtain to-be-recognized image features; inputting the to-be-recognized image features and a preset fine-grained visual feature query condition into a preset Transformer model to obtain to-be-recognized fine-grained visual features; inputting a category text description into a preset natural language feature extraction model to obtain to-be-recognized text features; inputting the to-be-recognized fine-grained visual features and the to-be-recognized text features into the multi-modal feature extraction model to perform feature alignment and obtain fusion features; and inputting the fusion features into an open set fine-grained classification model to obtain a target fine-grained category, so that open set recognition is realized and the accuracy is high.

Description

Open set fine-grained identification method and apparatus Technical Field

[0001] This application relates to the field of image recognition technology, specifically to an open set fine-grained recognition method and apparatus. Background Technology

[0002] Currently, in the field of image recognition, especially in fine-grained classification tasks, identifying objects of different categories with similar features has always been a challenge. Traditional fine-grained recognition methods typically rely on large amounts of labeled data to train deep learning models, but collecting and labeling large amounts of image data is extremely costly in practical applications.

[0003] Based on this, even with a large amount of labeled data, due to the diversity of real-world application scenarios, fine-grained identification problems of new or unknown categories still frequently arise. Traditional closed-set fine-grained classification models are prone to misclassification when encountering these new categories, resulting in low accuracy. Summary of the Invention

[0004] Based on this, this application aims to provide a method and apparatus for fine-grained open set identification, achieving high accuracy in fine-grained open set identification.

[0005] According to one aspect of this application, a fine-grained open-set recognition method is proposed, comprising: constructing a fine-grained recognition dataset; wherein the fine-grained recognition dataset includes multiple sets of image data samples with text labels; training a multimodal feature extraction model based on the fine-grained recognition dataset; inputting the image data to be recognized into a preset visual feature extraction network to obtain the image features to be recognized; inputting the image features to be recognized and preset fine-grained visual feature query conditions into a preset Transformer model to obtain the fine-grained visual features to be recognized; and inputting the category text description of the image data to be recognized into a preset... The language feature extraction model is used to obtain the text features to be identified. The fine-grained visual features and the text features to be identified are input into the multimodal feature extraction model. Modal alignment is performed based on the multimodal feature extraction model, and the modally aligned fine-grained visual features and text features to be identified are fused to obtain fused features. The fused features are input into a pre-trained open-set fine-grained classification model to obtain the target fine-grained category of the image data to be identified. The open-set fine-grained classification model is used to determine the target fine-grained category based on the feature distance between the fused features and the preset category language features, using a preset feature distance threshold.

[0006] According to some embodiments, a multimodal feature extraction model is trained based on a fine-grained recognition dataset, including: inputting image data samples into a preset visual feature extraction network to obtain image feature samples; inputting the image feature samples and preset fine-grained visual feature query conditions into a preset Transformer model to obtain fine-grained visual feature samples; inputting the text labels of the image data samples into a preset natural language feature extraction model to obtain text feature samples; constructing a loss function based on the fine-grained visual feature samples and text feature samples; and training a preset neural network model using the loss function based on the fine-grained recognition dataset until a preset termination condition is reached to obtain the multimodal feature extraction model.

[0007] According to some embodiments, a loss function is constructed based on fine-grained visual feature samples and text feature samples, including: calculating a first distance between paired fine-grained visual feature samples and text feature samples, and a second distance between unpaired fine-grained visual feature samples and text feature samples; and constructing a loss function with the training objective of minimizing the first distance and maximizing the second distance.

[0008] According to some embodiments, text labels are generated by inputting image data samples into a pre-trained large visual language model.

[0009] According to some embodiments, the preset category language features include at least some text feature samples.

[0010] According to some embodiments, the fused features are input into a pre-trained open-set fine-grained classification model to obtain the target fine-grained category of the image data to be identified, including: using the open-set fine-grained classification model to calculate the feature distance between the fused features and all preset category language features; if all feature distances are greater than a preset feature distance threshold, the target fine-grained category is determined to be an unknown category; if there is a feature distance among all feature distances that is not greater than the preset feature distance threshold, the target fine-grained category is determined to be a first category; wherein, the first category is the category of the preset category language feature corresponding to the smallest feature distance among all feature distances.

[0011] According to some embodiments, before training a multimodal feature extraction model based on a fine-grained recognition dataset, the method further includes: performing data augmentation on image data samples in the fine-grained recognition dataset.

[0012] According to one aspect of this application, an open-set fine-grained recognition device is proposed, comprising: a dataset module for constructing a fine-grained recognition dataset; wherein the fine-grained recognition dataset includes multiple sets of image data samples with text labels; a model training module for training a multimodal feature extraction model based on the fine-grained recognition dataset; a first visual feature module for inputting the image data to be recognized into a preset visual feature extraction network to obtain the image features to be recognized; a second visual feature module for inputting the image features to be recognized and preset fine-grained visual feature query conditions into a preset Transformer model to obtain the fine-grained visual features to be recognized; and a text feature module for inputting the image data to be recognized into a preset Transformer model. The text description of the category is input into a preset natural language feature extraction model to obtain the text features to be identified; the feature fusion module is used to input the fine-grained visual features and text features to be identified into a multimodal feature extraction model, perform modal alignment based on the multimodal feature extraction model, and fuse the modally aligned fine-grained visual features and text features to be identified to obtain fused features; the fine-grained classification module is used to input the fused features into a pre-trained open-set fine-grained classification model to obtain the target fine-grained category of the image data to be identified, wherein the open-set fine-grained classification model is used to determine the target fine-grained category based on the feature distance between the fused features and the preset category language features using a preset feature distance threshold.

[0013] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0014] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0015] Through the embodiments provided in this application, in practical applications, visual features are extracted by combining a visual feature extraction network and a Transformer model, text features are extracted by a natural language feature extraction model, and visual and text features are fused by a trained multimodal feature extraction model. The fused features are then used for open-set fine-grained recognition through visual-language feature matching, which can achieve open-set recognition with high accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0017] Figure 1 is a flowchart of the open set fine-grained identification method provided in an embodiment of this application;

[0018] Figure 2 is a flowchart of a multimodal feature extraction model trained on a fine-grained recognition dataset, provided in an embodiment of this application.

[0019] Figure 3 is a flowchart of the construction of the loss function provided in the embodiment of this application;

[0020] Figure 4 is a flowchart of the process of inputting fused features into a pre-trained open-set fine-grained classification model to obtain the target fine-grained category of the image data to be identified, as provided in an embodiment of this application.

[0021] Figure 5 is an overall framework diagram of the open set fine-grained identification method provided in the embodiments of this application;

[0022] Figure 6 is a block diagram of the open set fine-grained identification device provided in an embodiment of this application;

[0023] Figure 7 is a schematic diagram of the structure of the electronic device provided in the embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0028] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0029] In recent years, multimodal models based on visual and natural language features (such as CLIP) have made significant progress. These models achieve zero-shot classification of images under limited sample conditions by learning the alignment relationship between visual and natural language. However, existing methods generally do not fully utilize the semantic information described by natural language to further improve the performance of fine-grained classification and enhance the ability to identify unknown categories in open environments.

[0030] Based on this, this application provides a method and apparatus for fine-grained identification of open sets.

[0031] For specific implementation details, please refer to the following examples.

[0032] Figure 1 is a flowchart of the open set fine-grained identification method provided in an embodiment of this application. As shown in Figure 1, the method includes steps S110-S170.

[0033] In step S110, a fine-grained recognition dataset is constructed; wherein, the fine-grained recognition dataset includes multiple sets of image data samples with text labels.

[0034] The fine-grained recognition dataset is used to train a multimodal feature extraction model. In constructing the fine-grained recognition dataset, multiple sets of image data samples with text labels are obtained to form the dataset.

[0035] It should be noted that this application does not impose specific limitations on the method of obtaining image data samples. It may choose publicly available image datasets as the source, or it may use image data collected on-site or artificially synthesized.

[0036] Furthermore, in order to improve training efficiency more effectively, the selected image data samples can also be representative, diverse, and sufficient in quantity so that the model can quickly learn effective features.

[0037] In addition, the text label of the image data sample shall include at least the category to which the image data sample belongs, and may also include a text description of the characteristics of the image data sample.

[0038] Text labels can be included with the image data sample itself, or they can be obtained through manual annotation or annotation tools.

[0039] In step S120, a multimodal feature extraction model is trained based on the fine-grained recognition dataset.

[0040] During training, a multimodal feature extraction model is trained based on a fine-grained recognition dataset.

[0041] Multimodal feature extraction models combine the advantages of visual and natural language features, using the fused features for classification, thereby improving the classification model's ability to discriminate fine-grained features of known categories, while also making "unknown" judgments when unknown categories appear in images.

[0042] Furthermore, according to the example embodiment, a multimodal feature extraction model is trained based on a contrastive learning approach that matches visual and textual features, on the basis of a fine-grained recognition dataset.

[0043] In step S130, the image data to be identified is input into a preset visual feature extraction network to obtain the image features to be identified.

[0044] In practical applications, visual features are extracted from the image data to be recognized. It should be noted that visual feature extraction comprises two parts, implemented through a pre-defined visual feature extraction network and a pre-defined Transformer model.

[0045] Specifically, the preset visual feature extraction network is an image backbone network used to extract depth features from images. It should be noted that the specific type of the preset visual feature extraction network can be selected based on the actual situation, such as ResNet, ViT, etc., and this application does not impose any restrictions on this.

[0046] This application employs a pre-defined visual feature extraction network to extract hierarchical features from images, including both local details such as edges, corners, and textures, as well as higher-level structural information, thus comprehensively acquiring the visual information of the image.

[0047] In step S140, the image features to be identified and the preset fine-grained visual feature query conditions are input into the preset Transformer model to obtain the fine-grained visual features to be identified.

[0048] The preset Transformer model is used to extract a specified number of fine-grained features. Specifically, after obtaining the features of the image to be recognized, further specified features are extracted from the features of the image to be recognized.

[0049] In the specific implementation process, a pre-set fine-grained image Query (denoted as the preset fine-grained visual feature query condition) is used to assist the preset Transformer model in further extracting features from the image to be identified, and the queried fine-grained image Query is obtained, which is denoted as the fine-grained visual feature to be identified.

[0050] The pre-set fine-grained image query (i.e., preset fine-grained visual feature query conditions) is pre-set based on the open-set fine-grained recognition task associated with the image data to be recognized. Specifically, it is a task or request to query specific fine-grained image features. According to the example embodiment, the preset fine-grained visual feature query conditions may include image description, visual features, example images, and attributes, etc.

[0051] Furthermore, in some embodiments, the image description can be a detailed text description, used for fine-grained attributes of the object to be searched. Visual features can be predefined visual attributes, such as color, shape, texture, specific locations, etc. Attributes can be attribute characteristics that a specified object must possess, such as color: black, etc.

[0052] This application employs a pre-defined Transformer model to further extract specified features from the image based on the extracted hierarchical features, thereby comprehensively acquiring the visual information of the image.

[0053] In step S150, the category text description of the image data to be identified is input into a preset natural language feature extraction model to obtain the text features to be identified.

[0054] It should be noted that, in the application process, the category text description of the image data to be identified is different from the text label in the fine-grained recognition dataset. The category text description is a general description of the unknown class to be identified.

[0055] More specifically, in practical applications, a fixed teleprompter can be used to prompt users to input category text descriptions.

[0056] After obtaining the category text description of the image data to be recognized, this application needs to encode the category text description of the image to be recognized in order to convert it into a word vector space and obtain the text features to be recognized. The encoding process is implemented through a pre-set natural language feature extraction model.

[0057] The default natural language feature extraction model is a text encoder. The specific type of the default natural language feature extraction model can be selected according to the actual situation, such as BERT or GPT, and this application does not impose any restrictions on it.

[0058] In step S160, the fine-grained visual features to be identified and the text features to be identified are input into the multimodal feature extraction model. Modal alignment is performed based on the multimodal feature extraction model, and the fine-grained visual features to be identified and the text features to be identified after modal alignment are fused to obtain the fused features.

[0059] After obtaining the fine-grained visual features and text features to be identified, they are input into the trained multimodal feature extraction model. Both the fine-grained visual features and the text features to be identified are mapped to the same feature space through the multimodal feature extraction model, ultimately yielding fused features.

[0060] In step S170, the fused features are input into a pre-trained open-set fine-grained classification model to obtain the target fine-grained category of the image data to be identified. The open-set fine-grained classification model is used to determine the target fine-grained category based on the feature distance between the fused features and the preset category language features using a preset feature distance threshold.

[0061] The fused features are used for classification. Specifically, they are input into a pre-trained open-set fine-grained classification model, and the feature distance is calculated with the pre-defined category language features of known categories. The calculated feature distance is compared with a pre-defined feature distance threshold, and classification is performed based on the comparison result.

[0062] It should be noted that the preset category language features are some text features of known categories that are set for closed-set recognition based on the open-set fine-grained recognition task associated with the image data to be recognized. For example, some or all of the text features of the data in the fine-grained recognition dataset.

[0063] This application combines the advantages of visual features and natural language features through a multimodal feature extraction model, integrates visual features and natural language descriptions to create a common feature space, improves the model's ability to discriminate fine-grained features of known categories through multimodal training, and can make "unknown" judgments when unknown categories appear in images.

[0064] According to some embodiments, referring to Figure 2, in step S120, a multimodal feature extraction model is trained based on the fine-grained recognition dataset, which can be specifically implemented through steps S210-S250.

[0065] In step S210, image data samples are input into a preset visual feature extraction network to obtain image feature samples.

[0066] During the training process, it is necessary to extract features from the input image through a preset visual feature extraction network. This step is the same as step S130, and will not be described in detail here. Finally, the image feature samples of the image data samples are obtained.

[0067] In step S220, the image feature samples and preset fine-grained visual feature query conditions are input into the preset Transformer model to obtain fine-grained visual feature samples.

[0068] During the training process, it is necessary to extract further specified features from the image feature samples using a preset Transformer model. This step is the same as step S140 and will not be described in detail here. Finally, fine-grained visual feature samples of the image data samples are obtained.

[0069] In step S230, the text labels of the image data samples are input into a preset natural language feature extraction model to obtain text feature samples.

[0070] During the training process, it is necessary to extract text features from the text labels of image data samples using a pre-set natural language feature extraction model, and then convert them into word vector space to obtain text feature samples.

[0071] The preset natural language feature extraction model is the same as in step S150, and will not be described in detail here.

[0072] In step S240, a loss function is constructed based on fine-grained visual feature samples and text feature samples.

[0073] In the process of comparative learning, it is necessary to align fine-grained visual feature samples and text feature samples in the multimodal space to construct a loss function, thereby enabling the training of the multimodal feature extraction model.

[0074] To further improve training efficiency, before comparative learning, the multimodal feature extraction model maps text features and image features to the same feature space for comparison and calculation through dimensionality adjustment or feature transformation. Therefore, during training, both the visual feature samples to be refined and the text feature samples are mapped to the same feature space to facilitate comparative learning in subsequent steps.

[0075] According to the example implementation, the multimodal feature extraction model maps fine-grained visual feature samples and text feature samples to the same feature space through a visual-language cross-attention mechanism.

[0076] In step S250, based on the fine-grained recognition dataset, a preset neural network model is trained using a loss function until a preset termination condition is reached, thereby obtaining a multimodal feature extraction model.

[0077] Based on the fine-grained recognition dataset, the model aligns the fine-grained visual feature samples and text feature samples through contrastive learning using a loss function to improve the model's ability to match images and text until a preset termination condition is reached, thus obtaining a multimodal feature extraction model.

[0078] The preset termination condition can be the number of training rounds or the evaluation index reaching a threshold, and this application does not impose any restrictions on this.

[0079] According to some embodiments, referring to Figure 3, in step S240, the construction of the loss function can be specifically implemented through steps S310-S320.

[0080] In step S310, a first distance between paired fine-grained visual feature samples and text feature samples, and a second distance between unpaired fine-grained visual feature samples and text feature samples are calculated.

[0081] Specifically, the distance between a fine-grained visual feature sample and its corresponding text feature sample is calculated and denoted as the first distance; at the same time, the distance between a fine-grained visual feature sample and a text feature sample that does not match is calculated and denoted as the second distance.

[0082] For ease of computation, text features and image features can naturally be mapped to the same feature space through dimensionality adjustment or feature transformation.

[0083] Based on the above embodiments, fine-grained visual feature samples and text feature samples can be mapped to the same feature space through the visual-language cross-attention mechanism.

[0084] In step S320, a loss function is constructed with the training objective of minimizing the first distance and maximizing the second distance.

[0085] Specifically, by minimizing the distance between fine-grained features of known category images and their corresponding linguistic description features, and maximizing the distance between fine-grained features of known category images and their mismatched linguistic description features, the model can establish a connection between vision and language during training, thereby enabling the model to learn how to distinguish different fine-grained features in a multimodal feature space.

[0086] According to some embodiments, text labels are generated by inputting image data samples into a pre-trained large visual language model.

[0087] Specifically, the annotation tool used for annotating text labels can be a visual language big model, that is, using a pre-trained visual language big model to reason about image data samples, and using the text description automatically generated by the visual language big model as text labels.

[0088] This embodiment does not impose any restrictions on the large model of visual language; it can be a closed-source large model, such as ChatGPT4 or Wenxin Yiyan 4.0, or an open-source large model, such as Qwen2 or InternVL2.

[0089] According to some embodiments, the preset category language features include at least some text feature samples.

[0090] Specifically, in order to quickly determine the language features of the preset category, some or all of the text features (i.e., text feature samples) in the fine-grained recognition dataset can be directly used as the language features of the preset category.

[0091] According to some embodiments, referring to FIG4, in step S170, the fused features are input into a pre-trained open set fine-grained classification model to obtain the target fine-grained category of the image data to be identified, which can be specifically implemented through steps S410-S430.

[0092] In step S410, the feature distance between the fused feature and all preset category language features is calculated using an open set fine-grained classification model.

[0093] During the inference phase, the fused features of the image data to be identified are matched with the preset category language features of known categories. The feature distance between the fused features and all preset category language features is calculated, and the category to which the image belongs is determined by the feature distance and a preset threshold.

[0094] In step S420, if all feature distances are greater than a preset feature distance threshold, the target fine-grained category is determined to be an unknown category.

[0095] If the feature distance between the fused feature and the preset category language features of all known categories is higher than the preset feature distance threshold, it can be determined that it does not belong to a known category. Therefore, the target fine-grained category of the image data to be identified is determined to be an unknown category.

[0096] In step S430, if there is a feature distance among all feature distances that is not greater than a preset feature distance threshold, the target fine-grained category is determined as the first category; wherein, the first category is the category of the preset category language feature corresponding to the smallest feature distance among all feature distances.

[0097] Otherwise, if any of the calculated feature distances is no greater than a preset feature distance threshold, the image data to be identified is assigned to the category of the nearest preset category language feature (denoted as the first category). In this way, the model can perform fine-grained classification of known categories while also possessing the ability to recognize open sets.

[0098] During the inference phase, the open-set fine-grained classification model calculates the distance between image features and pre-defined linguistic features of known categories. Based on the calculated feature distances, samples with low matching scores (i.e., all feature distances are greater than a pre-defined feature distance threshold) are automatically identified as "unknown." For samples of known categories, fine-grained classification is automatically performed based on the degree of matching between visual and linguistic features.

[0099] Furthermore, in order to further improve the performance of the multimodal feature extraction model in fine-grained classification and its robustness in identifying unknown categories, data augmentation and contrastive learning enhancement methods can be introduced during the training of the multimodal feature extraction model.

[0100] According to some embodiments, before step S120, the method further includes:

[0101] Data augmentation is performed on image data samples in the fine-grained recognition dataset.

[0102] Specifically, data augmentation mainly refers to image data augmentation, including but not limited to random cropping, flipping, color jittering, and Gaussian blur.

[0103] Furthermore, data augmentation can be performed on the text labels in the fine-grained recognition dataset. This involves using automatically generated text descriptions to provide a more detailed description of the fine-grained categories or features of the image data samples, thereby further improving the model's ability to distinguish subtle differences.

[0104] According to some embodiments, contrastive learning enhancement can be achieved by introducing more fine-grained image queries (i.e., preset fine-grained visual feature query conditions) and detailed descriptive text to increase the contrast pairs of positive and negative samples, thereby improving the model's learning ability and convergence speed.

[0105] To further illustrate the fine-grained open set identification method provided in this application, a specific embodiment is given as shown in Figure 5.

[0106] In this embodiment, firstly, the image to be classified is input into a visual feature extraction network to extract its image feature map. Then, the preset fine-grained image query conditions and the image feature map are input into a Transformer to obtain the queried fine-grained image. Simultaneously, the text description of the image category or features is input into a natural language feature extraction module to be converted into text features.

[0107] The fine-grained image and text features after the query are fused through a cross-attention mechanism to obtain fused features.

[0108] The trained open set fine-grained classification model in the open set recognition module can match fused features with text features of known categories, and determine the category to which the image belongs based on feature distance and a preset threshold.

[0109] During training, the text comparison module aligns image features and text features through contrastive learning to improve the model's ability to match images and text. The multimodal feature extraction model is trained by minimizing the distance between image features and their corresponding text description features, while maximizing the distance between mismatched text description features.

[0110] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0111] Figure 6 shows a block diagram of an open-set fine-grained identification device according to an exemplary embodiment.

[0112] The apparatus shown in Figure 6 can perform the open set fine-grained identification method described above according to the embodiments of this application.

[0113] As shown in Figure 6, the open set fine-grained recognition device may include:

[0114] Referring to Figure 6 and the preceding description, the dataset module 610 is used to construct a fine-grained recognition dataset; wherein, the fine-grained recognition dataset includes multiple sets of image data samples with text labels.

[0115] The model training module 620 is used to train a multimodal feature extraction model based on a fine-grained recognition dataset.

[0116] The first visual feature module 630 is used to input the image data to be recognized into a preset visual feature extraction network to obtain the image features to be recognized.

[0117] The second visual feature module 640 is used to input the image features to be identified and the preset fine-grained visual feature query conditions into the preset Transformer model to obtain the fine-grained visual features to be identified.

[0118] The text feature module 650 is used to input the category text description of the image data to be identified into a preset natural language feature extraction model to obtain the text features to be identified.

[0119] The feature fusion module 660 is used to input the fine-grained visual features to be identified and the text features to be identified into the multimodal feature extraction model, perform modal alignment based on the multimodal feature extraction model, and fuse the modally aligned fine-grained visual features to be identified and the text features to be identified to obtain fused features.

[0120] The fine-grained classification module 670 is used to input the fused features into a pre-trained open-set fine-grained classification model to obtain the target fine-grained category of the image data to be identified. The open-set fine-grained classification model is used to determine the target fine-grained category based on the feature distance between the fused features and the preset category language features, using a preset feature distance threshold.

[0121] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0122] Figure 7 illustrates an electronic device according to an exemplary embodiment of this application. The electronic device 700 according to this embodiment of the present application will now be described with reference to Figure 7. The electronic device 700 shown in Figure 7 is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application.

[0123] As shown in Figure 7, the electronic device 700 is presented in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including storage unit 720 and processing unit 710), a display unit 740, etc.

[0124] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the methods described in the various exemplary embodiments of this application. For example, the processing unit 710 can perform the methods described above.

[0125] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include a read-only memory (ROM) 7203.

[0126] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0127] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0128] Electronic device 700 can also communicate with one or more external devices 300 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. Network adapter 760 can communicate with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0129] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this application.

[0130] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0131] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0132] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0134] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0135] According to an embodiment of this application, a computer program is proposed, including a computer program or instructions, which, when executed by a processor, can perform the methods described above.

[0136] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0137] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0138] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A fine-grained method for identifying open sets, characterized in that, include: A fine-grained recognition dataset is constructed, comprising multiple sets of image data samples with text labels. Based on this dataset, a multimodal feature extraction model is trained. The image data to be recognized is input into a preset visual feature extraction network to obtain image features. The image features and preset fine-grained visual feature query conditions are input into a preset Transformer model to obtain fine-grained visual features. The category text description of the image data to be recognized is input into a preset natural language feature extraction model to obtain text features, where the category text description is a description of the image to be recognized. The fine-grained visual features and the text features are input into the multimodal feature extraction model, and modality alignment is performed based on the model. The aligned fine-grained visual features and text features are then fused to obtain fused features. Finally, the fused features are input into a pre-trained open-set fine-grained classification model. The target fine-grained category of the image data to be identified is obtained, wherein the open-set fine-grained classification model is used to determine the target fine-grained category based on the feature distance between the fused feature and the preset category language feature, using a preset feature distance threshold. The target fine-grained category is one of the categories corresponding to the preset category language feature or an unknown category. The process of inputting the fused feature into a pre-trained open-set fine-grained classification model to obtain the target fine-grained category of the image data to be identified includes: using the open-set fine-grained classification model to calculate the feature distance between the fused feature and all the preset category language features; if all the feature distances are greater than the preset feature distance threshold, the target fine-grained category is determined to be an unknown category; if there is a feature distance among all the feature distances that is not greater than the preset feature distance threshold, the target fine-grained category is determined to be a first category; wherein the first category is the category of the preset category language feature corresponding to the smallest feature distance among all the feature distances.

2. The method according to claim 1, characterized in that, The step of training a multimodal feature extraction model based on the fine-grained recognition dataset includes: inputting the image data samples into the preset visual feature extraction network to obtain image feature samples; inputting the image feature samples and the preset fine-grained visual feature query conditions into the preset Transformer model to obtain fine-grained visual feature samples; inputting the text labels of the image data samples into the preset natural language feature extraction model to obtain text feature samples; constructing a loss function based on the fine-grained visual feature samples and the text feature samples; and training a preset neural network model using the loss function based on the fine-grained recognition dataset until a preset termination condition is reached to obtain the multimodal feature extraction model.

3. The method according to claim 2, characterized in that, The step of constructing a loss function based on the fine-grained visual feature samples and the text feature samples includes: calculating a first distance between paired fine-grained visual feature samples and text feature samples, and a second distance between unpaired fine-grained visual feature samples and text feature samples; and constructing the loss function with the training objective of minimizing the first distance and maximizing the second distance.

4. The method according to claim 1, characterized in that, The text labels are generated by inputting the image data samples into a pre-trained visual language large model.

5. The method according to claim 2, characterized in that, The preset category language features include at least some of the text feature samples.

6. The method according to claim 1, characterized in that, Before training the multimodal feature extraction model based on the fine-grained recognition dataset, the method further includes: performing data augmentation on the image data samples in the fine-grained recognition dataset.

7. A fine-grained open-set identification device, characterized in that, include: The system comprises the following modules: a dataset module for constructing a fine-grained recognition dataset, comprising multiple sets of image data samples with text labels; a model training module for training a multimodal feature extraction model based on the fine-grained recognition dataset; a first visual feature module for inputting the image data to be recognized into a preset visual feature extraction network to obtain the image features to be recognized; a second visual feature module for inputting the image features to be recognized and preset fine-grained visual feature query conditions into a preset Transformer model to obtain the fine-grained visual features to be recognized; a text feature module for inputting the category text description of the image data to be recognized into a preset natural language feature extraction model to obtain the text features to be recognized, wherein the category text description is a description of the image to be recognized; and a feature fusion module for inputting the fine-grained visual features to be recognized and the text features to be recognized into the multimodal feature extraction model, performing modal alignment based on the multimodal feature extraction model, and fusing the modally aligned fine-grained visual features to be recognized and the text features to be recognized. The text features to be identified are used to obtain fused features. A fine-grained classification module is used to input the fused features into a pre-trained open-set fine-grained classification model to obtain the target fine-grained category of the image data to be identified. The open-set fine-grained classification model is used to determine the target fine-grained category based on the feature distance between the fused features and the preset category language features, using a preset feature distance threshold. The target fine-grained category is one of the categories corresponding to the preset category language features or an unknown category. The fine-grained classification module is specifically used to: use the open-set fine-grained classification model to calculate the feature distance between the fused features and all the preset category language features; if all the feature distances are greater than the preset feature distance threshold, determine the target fine-grained category as an unknown category; if there is a feature distance among all the feature distances that is not greater than the preset feature distance threshold, determine the target fine-grained category as a first category; wherein the first category is the category of the preset category language feature corresponding to the smallest feature distance among all the feature distances.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-6.

Images