Open set fine granularity identification method and device

Abstract

The invention provides an open set fine granularity identification method and device, and relates to the technical field of image identification. The method comprises the steps of training based on a fine-grained recognition data set to obtain a multi-modal feature extraction model, and in the training process, performing comparative learning after a visual feature sample and a text feature sample are aligned; inputting the to-be-recognized image data into a preset visual feature extraction network to obtain to-be-recognized image features; inputting the to-be-recognized image features and a preset fine-grained visual feature query condition into a preset Transform model to obtain to-be-recognized fine-grained visual features; inputting the category text description into a preset natural language feature extraction model to obtain to-be-recognized text features; inputting the to-be-recognized fine-grained visual features and the to-be-recognized text features into a multi-modal feature extraction model, and performing feature alignment to obtain fusion features; and inputting the fusion features into the open-set fine-granularity classification model to obtain a target fine-granularity category, thereby realizing open-set identification and having high accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of image recognition, and in particular to an open set fine-grained recognition method and device. Background Art

[0002] At present, in the field of image recognition, especially in fine-grained classification tasks, identifying objects of different categories with similar features has always been a difficult problem. Traditional fine-grained recognition methods usually rely on a large amount of labeled data to train deep learning models, but in practical applications, the cost of collecting and labeling a large amount of image data is very high.

[0003] On this basis, even if a large amount of labeled data is used, due to the diversity of real application scenarios, fine-grained recognition problems of new or unknown categories still often arise. Traditional closed-set fine-grained classification models are prone to misclassification problems when encountering these new categories, and have low accuracy. Summary of the invention

[0004] Based on this, the present application aims to provide an open set fine-grained recognition method and device to achieve open set fine-grained recognition with high accuracy.

[0005] According to one aspect of the present application, an open set fine-grained recognition method is proposed, comprising: constructing a fine-grained recognition dataset; wherein the fine-grained recognition dataset comprises a plurality of sets of image data samples with text labels; based on the fine-grained recognition dataset, training a multimodal feature extraction model; inputting the image data to be recognized into a preset visual feature extraction network to obtain the image features to be recognized; inputting the image features to be recognized and the preset fine-grained visual feature query conditions into a preset Transformer model to obtain the fine-grained visual features to be recognized; inputting the category text description of the image data to be recognized into a preset automatic The method comprises the following steps: inputting the fine-grained visual features to be identified and the text features to be identified into a multimodal feature extraction model, performing modal alignment based on the multimodal feature extraction model, and fusing the fine-grained visual features to be identified and the text features to be identified after the modal alignment to obtain fused features; inputting the fused features into a pre-trained open set fine-grained classification model to obtain target fine-grained categories of the image data to be identified, wherein the open set fine-grained classification model is used to determine the target fine-grained category according to the feature distance between the fused features and the preset category language features using a preset feature distance threshold.

[0006] According to some embodiments, based on a fine-grained recognition dataset, a multimodal feature extraction model is trained, including: inputting image data samples into a preset visual feature extraction network to obtain image feature samples; inputting image feature samples and preset fine-grained visual feature query conditions into a preset Transformer model to obtain fine-grained visual feature samples; inputting text labels of image data samples into a preset natural language feature extraction model to obtain text feature samples; constructing a loss function based on the fine-grained visual feature samples and the text feature samples; based on the fine-grained recognition dataset, using the loss function to train a preset neural network model until a preset end condition is reached to obtain a multimodal feature extraction model.

[0007] According to some embodiments, a loss function is constructed based on fine-grained visual feature samples and text feature samples, including: calculating a first distance between paired fine-grained visual feature samples and text feature samples, and a second distance between unpaired fine-grained visual feature samples and text feature samples; and constructing a loss function with minimizing the first distance and maximizing the second distance as training objectives.

[0008] According to some embodiments, text labels are generated by inputting image data samples into a pre-trained visual language model.

[0009] According to some embodiments, the preset category language feature includes at least a portion of the text feature sample.

[0010] According to some embodiments, the fused features are input into a pre-trained open set fine-grained classification model to obtain a target fine-grained category of the image data to be identified, including: using the open set fine-grained classification model to calculate the feature distance between the fused features and all preset category language features; when all feature distances are greater than a preset feature distance threshold, determining the target fine-grained category to be an unknown category; when there is a feature distance that is not greater than a preset feature distance threshold among all feature distances, determining the target fine-grained category to be a first category; wherein the first category is the category of the preset category language feature corresponding to the smallest feature distance among all feature distances.

[0011] According to some embodiments, before training a multimodal feature extraction model based on a fine-grained recognition dataset, the method further includes: performing data enhancement on image data samples in the fine-grained recognition dataset.

[0012] According to one aspect of the present application, an open set fine-grained recognition device is proposed, comprising: a data set module, used to construct a fine-grained recognition data set; wherein the fine-grained recognition data set includes multiple groups of image data samples with text labels; a model training module, used to train a multimodal feature extraction model based on the fine-grained recognition data set; a first visual feature module, used to input the image data to be recognized into a preset visual feature extraction network to obtain the image features to be recognized; a second visual feature module, used to input the image features to be recognized and the preset fine-grained visual feature query conditions into a preset Transformer model to obtain the fine-grained visual features to be recognized; and a text feature module, used to input the image data to be recognized into a preset Transformer model to obtain the fine-grained visual features to be recognized. The category text description of the image data is input into a preset natural language feature extraction model to obtain the text features to be identified; a feature fusion module is used to input the fine-grained visual features to be identified and the text features to be identified into the multimodal feature extraction model, perform modal alignment based on the multimodal feature extraction model, and fuse the fine-grained visual features to be identified and the text features to be identified after the modal alignment to obtain fused features; a fine-grained classification module is used to input the fused features into a pre-trained open set fine-grained classification model to obtain the target fine-grained category of the image data to be identified, wherein the open set fine-grained classification model is used to determine the target fine-grained category according to the feature distance between the fused features and the preset category language features using a preset feature distance threshold.

[0013] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0014] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0015] Through the above embodiments provided in the present application, in practical applications, the visual feature extraction network and the Transformer model are jointly used to extract visual features, the natural language feature extraction model is used to extract the natural language features, and the trained multimodal feature extraction model is used to fuse the visual features and the text features. The fused features are used for open set fine-grained recognition through matching of visual-language features, which can achieve open set recognition with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by the present application.

[0017] Figure 1 A flowchart of an open set fine-grained recognition method provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of a multimodal feature extraction model obtained by training based on a fine-grained recognition data set is provided for an embodiment of the present application;

[0019] Figure 3 A flowchart of the construction of the loss function provided in the embodiment of the present application;

[0020] Figure 4 A flowchart of inputting fusion features into a pre-trained open set fine-grained classification model to obtain a target fine-grained category of image data to be identified provided in an embodiment of the present application;

[0021] Figure 5 The overall framework diagram of the open set fine-grained recognition method provided in the embodiment of the present application;

[0022] Figure 6 A block diagram of an open set fine-grained recognition device provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0025] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0028] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0029] In recent years, multimodal models based on visual and natural language features (such as CLIP) have made significant progress. These models achieve zero-shot classification of images under limited sample conditions by learning the alignment relationship between vision and natural language. However, existing methods generally still do not fully utilize the semantic information described in natural language to further improve the performance of fine-grained classification and improve the ability to recognize unknown categories in open environments.

[0030] On this basis, the present application provides an open set fine-grained recognition method and device.

[0031] For specific implementation methods, please refer to the following embodiments.

[0032] Figure 1 Flow chart of the open set fine-grained recognition method provided in the embodiment of the present application. Figure 1 As shown, the method includes steps S110 to S170.

[0033] In step S110, a fine-grained recognition dataset is constructed; wherein the fine-grained recognition dataset includes multiple groups of image data samples with text labels.

[0034] The fine-grained recognition dataset is used to train the multimodal feature extraction model. In the process of constructing the fine-grained recognition dataset, multiple sets of image data samples with text labels are obtained to form the fine-grained recognition dataset.

[0035] It should be noted that the present application does not impose any specific restrictions on the method of obtaining image data samples. A public image dataset may be selected as a source, or image data collected on site or artificially synthesized may be used.

[0036] Furthermore, in order to more effectively improve the training efficiency, the selected image data samples can also be representative, diverse, and sufficient in number so that the model can quickly learn effective features.

[0037] In addition, the text label of the image data sample includes at least the category to which the image data sample belongs, and may also include a text description of the features of the image data sample.

[0038] The text label may be provided by the image data sample itself, or may be obtained by manual annotation or annotation using an annotation tool.

[0039] In step S120, a multimodal feature extraction model is trained based on the fine-grained recognition data set.

[0040] During the training process, a multimodal feature extraction model is trained based on the fine-grained recognition dataset.

[0041] The multimodal feature extraction model is used to combine the advantages of visual features and natural language features, and use the fused features for classification, thereby improving the classification model's ability to discriminate fine-grained features of known categories, while being able to make "unknown" judgments when unknown categories appear in the image.

[0042] Further, according to an example embodiment, based on a fine-grained recognition dataset, a contrastive learning method based on matching of visual features and text features is adopted to train a multimodal feature extraction model.

[0043] In step S130, the image data to be identified is input into a preset visual feature extraction network to obtain the image features to be identified.

[0044] In the actual application process, visual features are extracted from the image data to be identified. It should be noted that the extraction of visual features includes two parts, which are realized by a preset visual feature extraction network and a preset Transformer model.

[0045] Specifically, the preset visual feature extraction network is an image backbone network for extracting deep features of an image. It should be noted that the specific type of the preset visual feature extraction network can be selected according to actual conditions, such as ResNet, ViT, etc., and this application does not limit this.

[0046] This application uses a preset visual feature extraction network to extract hierarchical features of the image, including both local details, such as edges, corners, textures, etc., as well as higher-level structural information, to comprehensively obtain the visual information of the image.

[0047] In step S140, the image features to be identified and the preset fine-grained visual feature query conditions are input into a preset Transformer model to obtain the fine-grained visual features to be identified.

[0048] The preset Transformer model is used to extract a specified number of fine-grained features. Specifically, after obtaining the features of the image to be identified, further specified features are extracted from the features of the image to be identified.

[0049] In the specific implementation process, a preset fine-grained image query (referred to as the preset fine-grained visual feature query condition) is used to assist the preset Transformer model in further extracting features of the image to be identified, and the fine-grained image query after the query is obtained, which is referred to as the fine-grained visual feature to be identified.

[0050] The preset fine-grained image query (i.e., the preset fine-grained visual feature query condition) is preset according to the open set fine-grained recognition task associated with the image data to be recognized, which is specifically a task or request to query specific fine-grained image features. According to an example embodiment, the preset fine-grained visual feature query condition may include image description, visual features, example images, attributes, etc.

[0051] Furthermore, in some embodiments, the image description may be a detailed text description used to determine the fine-grained attributes of the object to be found. The visual feature may be a predefined visual attribute, such as color, shape, texture, specific part, etc. The attribute may be an attribute feature that a specified object must possess, such as color: black, etc.

[0052] This application uses a preset Transformer model to further extract specified features of the image based on the extracted hierarchical features, thereby comprehensively acquiring the visual information of the image.

[0053] In step S150, the category text description of the image data to be identified is input into a preset natural language feature extraction model to obtain the text features to be identified.

[0054] It should be noted that, in the application process, the category text description of the image data to be identified is different from the text label in the fine-grained recognition dataset. The category text description is specifically a general description of the unknown class to be identified.

[0055] More specifically, in actual application, a fixed teleprompter may be used to prompt the user to input a category text description.

[0056] After obtaining the category text description of the image data to be identified, the application needs to encode the category text description text of the image to be identified to convert it into a word vector space to obtain the text features to be identified. The encoding process is implemented by a preset natural language feature extraction model.

[0057] The preset natural language feature extraction model is a text encoder. The specific type of the preset natural language feature extraction model can be selected according to actual conditions, such as BERT or GPT, and this application does not limit this.

[0058] In step S160, the fine-grained visual features to be identified and the text features to be identified are input into the multimodal feature extraction model, modal alignment is performed based on the multimodal feature extraction model, and the fine-grained visual features to be identified and the text features to be identified after modal alignment are fused to obtain fused features.

[0059] After obtaining the fine-grained visual features to be identified and the text features to be identified, they are input into the trained multimodal feature extraction model. The fine-grained visual features to be identified and the text features to be identified are mapped to the same feature space through the multimodal feature extraction model, and finally the fusion features are obtained.

[0060] In step S170, the fused features are input into a pre-trained open set fine-grained classification model to obtain a target fine-grained category of the image data to be identified, wherein the open set fine-grained classification model is used to determine the target fine-grained category based on a feature distance between the fused features and a preset category language feature using a preset feature distance threshold.

[0061] The fused features are used for classification. Specifically, they are input into a pre-trained open set fine-grained classification model, the feature distance is calculated with the preset category language features of known categories, the calculated feature distance is compared with the preset feature distance threshold, and classification is performed according to the comparison result.

[0062] It should be noted that the preset category language features are some text features of known categories for closed-set recognition set according to the open-set fine-grained recognition task associated with the image data to be recognized, such as part or all of the text features of the data in the fine-grained recognition dataset.

[0063] This application combines the advantages of visual features and natural language features through a multimodal feature extraction model, fuses visual features and natural language descriptions to create a common feature space, and improves the model's ability to discriminate fine-grained features of known categories through multimodal training methods, and is able to make "unknown" judgments when unknown categories appear in the image.

[0064] According to some embodiments, reference Figure 2 In step S120, a multimodal feature extraction model is trained based on a fine-grained recognition data set, which can be specifically implemented through steps S210 to S250.

[0065] In step S210, the image data sample is input into a preset visual feature extraction network to obtain an image feature sample.

[0066] During the training process, it is necessary to perform feature extraction on the input image through a preset visual feature extraction network. This step is the same as step S130 and will not be repeated in this application. Finally, an image feature sample of the image data sample is obtained.

[0067] In step S220, the image feature sample and the preset fine-grained visual feature query condition are input into a preset Transformer model to obtain a fine-grained visual feature sample.

[0068] During the training process, it is necessary to further extract specified features of the image feature samples through the preset Transformer model. This step is the same as step S140 and will not be repeated in this application. Finally, fine-grained visual feature samples of the image data samples are obtained.

[0069] In step S230, the text label of the image data sample is input into a preset natural language feature extraction model to obtain a text feature sample.

[0070] During the training process, it is necessary to extract text features from the text labels of image data samples through a preset natural language feature extraction model, convert them into word vector space, and obtain text feature samples.

[0071] The preset natural language feature extraction model is the same as step S150, and the present invention will not elaborate on it.

[0072] In step S240, a loss function is constructed according to the fine-grained visual feature samples and the text feature samples.

[0073] In the process of contrastive learning, it is necessary to align fine-grained visual feature samples and text feature samples in the multimodal space to construct a loss function, thereby realizing the training of the multimodal feature extraction model.

[0074] In order to further improve the training efficiency, before contrastive learning, the multimodal feature extraction model maps the text features and image features to the same feature space for comparison and calculation through dimensionality adjustment or feature conversion. Therefore, during the training process, the fine-grained visual feature samples and text feature samples are mapped to the same feature space to facilitate contrastive learning in subsequent steps.

[0075] According to an example embodiment, a multimodal feature extraction model maps fine-grained visual feature samples and text feature samples to the same feature space through a visual-linguistic cross-attention mechanism.

[0076] In step S250, based on the fine-grained recognition data set, the preset neural network model is trained using the loss function until the preset end condition is reached to obtain a multimodal feature extraction model.

[0077] Based on the fine-grained recognition dataset, the loss function is used to align the fine-grained visual feature samples and text feature samples through contrastive learning to improve the model's matching ability between images and texts until the preset end condition is reached, and a multimodal feature extraction model is obtained.

[0078] Among them, the preset end condition can be the number of training rounds or the evaluation index reaching a threshold, which is not limited in this application.

[0079] According to some embodiments, reference Figure 3 In step S240, the construction of the loss function can be specifically implemented through steps S310 to S320.

[0080] In step S310 , a first distance between paired fine-grained visual feature samples and text feature samples, and a second distance between unpaired fine-grained visual feature samples and text feature samples are calculated.

[0081] Specifically, the distance between the fine-grained visual feature sample and its corresponding text feature sample is calculated, which is recorded as the first distance; at the same time, the distance between the fine-grained visual feature sample and its mismatched text feature sample is calculated, which is recorded as the second distance.

[0082] For the convenience of calculation, naturally, text features and image features can be mapped to the same feature space through dimension adjustment or feature transformation.

[0083] Based on the above embodiments, fine-grained visual feature samples and text feature samples can be mapped to the same feature space through a visual-linguistic cross-attention mechanism.

[0084] In step S320, a loss function is constructed with minimizing the first distance and maximizing the second distance as training objectives.

[0085] Specifically, by minimizing the distance between the fine-grained features of known category images and their corresponding language description features, and maximizing the distance between the fine-grained features of known category images and their mismatched language description features, the model can establish an association between vision and language during training, so that the model can learn how to distinguish different fine-grained features in the multimodal feature space.

[0086] According to some embodiments, text labels are generated by inputting image data samples into a pre-trained visual language model.

[0087] Specifically, the annotation tool used to annotate text labels can be a visual language big model, that is, using the pre-trained visual language big model to infer the image data samples, and using the text description automatically generated by the visual language big model as the text label.

[0088] This embodiment does not limit the visual language big model, which can be a closed source big model, such as ChatGPT4, Wenxinyiyan4.0, etc., or an open source big model, such as Qwen2, InternVL2, etc.

[0089] According to some embodiments, the preset category language feature includes at least a portion of the text feature sample.

[0090] Specifically, in order to quickly determine the preset category language features, part or all of the text features (ie, text feature samples) in the fine-grained recognition dataset can be directly used as the preset category language features.

[0091] According to some embodiments, reference Figure 4 In step S170, the fused features are input into a pre-trained open set fine-grained classification model to obtain the target fine-grained category of the image data to be identified, which can be specifically achieved through steps S410 to S430.

[0092] In step S410, an open set fine-grained classification model is used to calculate the feature distance between the fusion feature and all preset category language features.

[0093] In the inference stage, the fused features of the image data to be identified are matched with the preset category language features of known categories, and the feature distance between the fused features and all the preset category language features is calculated. The image's category is determined based on the feature distance and the preset threshold.

[0094] In step S420 , when all feature distances are greater than a preset feature distance threshold, the target fine-grained category is determined to be an unknown category.

[0095] If the feature distance between the fused feature and all the preset category language features of known categories is higher than the preset feature distance threshold, it can be judged that it does not belong to a known category, so the target fine-grained category of the image data to be identified is determined to be an unknown category.

[0096] In step S430, when there is a feature distance not greater than a preset feature distance threshold among all feature distances, the target fine-grained category is determined to be the first category; wherein the first category is the category of the preset category language feature corresponding to the smallest feature distance among all feature distances.

[0097] Otherwise, that is, if there is a feature distance that is not greater than the preset feature distance threshold among all the calculated feature distances, the image data to be identified is assigned to the category of the language feature of the preset category that is closest to it (recorded as the first category). In this way, the model can perform fine-grained classification of known categories and has the ability of open set recognition.

[0098] In the inference stage, the open set fine-grained classification model calculates the distance between the image features and the preset category language features of the known categories. Based on the calculated feature distance, for samples with low matching (that is, all feature distances are greater than the preset feature distance threshold), it can automatically identify them as "unknown". For samples of known categories, fine-grained classification can be automatically completed based on the matching degree of visual-linguistic features.

[0099] Furthermore, in order to further improve the performance of the multimodal feature extraction model in fine-grained classification and the robustness of identifying unknown categories, data enhancement and contrastive learning enhancement methods can be introduced during the training of the multimodal feature extraction model.

[0100] According to some embodiments, before step S120, the method further includes:

[0101] Perform data augmentation on image data samples in fine-grained recognition datasets.

[0102] Specifically, data enhancement mainly refers to image data enhancement, including but not limited to random cropping, flipping, color jitter, Gaussian blur, etc.

[0103] Furthermore, data enhancement can be performed on the text labels in the fine-grained recognition dataset, that is, the automatically generated text description can be used to provide a more detailed text description of the fine-grained categories or features of the image data samples, further improving the model's ability to distinguish subtle differences.

[0104] According to some embodiments, contrastive learning enhancement can be achieved by introducing more fine-grained image queries (i.e., preset fine-grained visual feature query conditions) and detailed description texts to increase the contrast pairs of positive and negative samples, thereby improving the learning ability and convergence speed of the model.

[0105] In order to further illustrate the open set fine-grained recognition method provided by the present application, a specific implementation example is given. Figure 5 shown.

[0106] In this embodiment, first, the image to be classified is input into the visual feature extraction network to extract its image feature map, and then the preset fine-grained image query condition and the image feature map are combined. Figure 1 The same is input into the Transformer to obtain the fine-grained image after the query. At the same time, the text description of the image category or feature is input into the natural language feature extraction module and converted into text features.

[0107] The queried fine-grained image and text features are fused through a cross-attention mechanism to obtain fused features.

[0108] The trained open-set fine-grained classification model in the open-set recognition module can match the fused features with the text features of known categories and determine the category of the image based on the feature distance and the preset threshold.

[0109] During the training process, the text comparison module aligns image features and text features through contrastive learning to improve the model's ability to match images and texts. The multimodal feature extraction model is trained by minimizing the distance between image features and their corresponding text description features, while maximizing the distance between unmatched text description features.

[0110] The following describes an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, reference can be made to the method embodiment of the present application.

[0111] Figure 6 A block diagram of an open set fine-grained recognition apparatus according to an exemplary embodiment is shown.

[0112] Figure 6 The device shown can execute the aforementioned open set fine-grained recognition method according to the embodiment of the present application.

[0113] like Figure 6 As shown, the open set fine-grained recognition device may include:

[0114] See also Figure 6 And referring to the above description, the data set module 610 is used to construct a fine-grained recognition data set; wherein the fine-grained recognition data set includes multiple groups of image data samples with text labels.

[0115] The model training module 620 is used to train a multimodal feature extraction model based on a fine-grained recognition data set.

[0116] The first visual feature module 630 is used to input the image data to be identified into a preset visual feature extraction network to obtain the image features to be identified.

[0117] The second visual feature module 640 is used to input the image features to be identified and the preset fine-grained visual feature query conditions into the preset Transformer model to obtain the fine-grained visual features to be identified.

[0118] The text feature module 650 is used to input the category text description of the image data to be identified into a preset natural language feature extraction model to obtain the text features to be identified.

[0119] The feature fusion module 660 is used to input the fine-grained visual features to be identified and the text features to be identified into the multimodal feature extraction model, perform modal alignment based on the multimodal feature extraction model, and fuse the fine-grained visual features to be identified and the text features to be identified after modal alignment to obtain fused features.

[0120] The fine-grained classification module 670 is used to input the fused features into a pre-trained open set fine-grained classification model to obtain the target fine-grained category of the image data to be identified, wherein the open set fine-grained classification model is used to determine the target fine-grained category based on the feature distance between the fused features and the preset category language features using a preset feature distance threshold.

[0121] The device performs functions similar to the method provided above. For other functions, please refer to the previous description and will not be repeated here.

[0122] Figure 7 An electronic device according to an exemplary embodiment of the present application is shown. Figure 7 hereinafter describes an electronic device 700 according to this embodiment of the present application. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0123] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), a display unit 740, etc.

[0124] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the methods described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 710 can execute the method described above.

[0125] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0126] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0127] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0128] The electronic device 700 may also communicate with one or more external devices 300 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 760. The network adapter 760 may communicate with other modules of the electronic device 700 via the bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0129] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. The technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation method of the present application.

[0130] The software product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0131] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0132] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the aforementioned functions.

[0134] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0135] According to an embodiment of the present application, a computer program is provided, including a computer program or an instruction. When the computer program or the instruction is executed by a processor, the method described above can be executed.

[0136] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

[0137] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0138] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. An open set fine-grained recognition method, characterized in that: include: Constructing a fine-grained recognition dataset; wherein the fine-grained recognition dataset includes multiple groups of image data samples with text labels; Based on the fine-grained recognition data set, a multimodal feature extraction model is trained; Input the image data to be identified into a preset visual feature extraction network to obtain the image features to be identified; Inputting the image features to be identified and preset fine-grained visual feature query conditions into a preset Transformer model to obtain the fine-grained visual features to be identified; Inputting the category text description of the image data to be identified into a preset natural language feature extraction model to obtain the text features to be identified; Inputting the fine-grained visual features to be identified and the text features to be identified into the multimodal feature extraction model, performing modal alignment based on the multimodal feature extraction model, and fusing the fine-grained visual features to be identified and the text features to be identified after modal alignment to obtain fused features; The fused features are input into a pre-trained open-set fine-grained classification model to obtain a target fine-grained category of the image data to be identified, wherein the open-set fine-grained classification model is used to determine the target fine-grained category based on a feature distance between the fused features and a preset category language feature using a preset feature distance threshold.

2. The method according to claim 1, characterized in that The multimodal feature extraction model is trained based on the fine-grained recognition data set, including: Inputting the image data sample into the preset visual feature extraction network to obtain an image feature sample; Inputting the image feature sample and the preset fine-grained visual feature query condition into the preset Transformer model to obtain a fine-grained visual feature sample; Inputting the text label of the image data sample into the preset natural language feature extraction model to obtain a text feature sample; Constructing a loss function according to the fine-grained visual feature samples and the text feature samples; Based on the fine-grained recognition data set, the preset neural network model is trained using the loss function until a preset end condition is reached to obtain the multimodal feature extraction model.

3. The method according to claim 2, characterized in that The constructing a loss function according to the fine-grained visual feature samples and the text feature samples comprises: Calculating a first distance between paired fine-grained visual feature samples and text feature samples, and a second distance between unpaired fine-grained visual feature samples and text feature samples; The loss function is constructed with minimizing the first distance and maximizing the second distance as training objectives.

4. The method according to claim 1, characterized in that: The text label is generated by inputting the image data sample into a pre-trained visual language model.

5. The method according to claim 2, characterized in that: The preset category language features include at least part of the text feature samples.

6. The method according to claim 1 or 5, characterized in that: The step of inputting the fusion features into a pre-trained open set fine-grained classification model to obtain a target fine-grained category of the image data to be identified includes: Using the open set fine-grained classification model, calculating the feature distance between the fusion feature and all the preset category language features; When all the feature distances are greater than a preset feature distance threshold, determining that the target fine-grained category is an unknown category; When there is a feature distance that is not greater than the preset feature distance threshold among all the feature distances, the target fine-grained category is determined to be the first category; wherein the first category is the category of the preset category language feature corresponding to the smallest feature distance among all the feature distances.

7. The method according to claim 1, characterized in that Before training a multimodal feature extraction model based on the fine-grained recognition data set, the method further includes: Data enhancement is performed on the image data samples in the fine-grained recognition dataset.

8. An open set fine-grained recognition device, characterized in that: include: A data set module, used to construct a fine-grained recognition data set; wherein the fine-grained recognition data set includes multiple groups of image data samples with text labels; A model training module, used for training a multimodal feature extraction model based on the fine-grained recognition data set; A first visual feature module, used for inputting the image data to be identified into a preset visual feature extraction network to obtain the image features to be identified; A second visual feature module, used for inputting the image features to be identified and preset fine-grained visual feature query conditions into a preset Transformer model to obtain the fine-grained visual features to be identified; A text feature module, used for inputting the category text description of the image data to be identified into a preset natural language feature extraction model to obtain the text features to be identified; A feature fusion module, used for inputting the fine-grained visual features to be identified and the text features to be identified into the multimodal feature extraction model, performing modal alignment based on the multimodal feature extraction model, and fusing the fine-grained visual features to be identified and the text features to be identified after modal alignment to obtain fused features; A fine-grained classification module is used to input the fused features into a pre-trained open-set fine-grained classification model to obtain a target fine-grained category of the image data to be identified, wherein the open-set fine-grained classification model is used to determine the target fine-grained category based on a feature distance between the fused features and a preset category language feature using a preset feature distance threshold.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images