An explainable open set recognition method based on hierarchical visual semantic attribute space

Abstract

The application belongs to the technical field of open set identification, and discloses an open set identification method based on a hierarchical visual semantic attribute space. The application can not only accurately identify whether a target belongs to a seen class or an unseen class, but also can explicitly determine the coarse-grained class to which the target belongs and the specific attributes possessed by the target. Compared with the traditional open set identification technology, the open set identification method based on the hierarchical visual semantic attribute space has significant advantages, greatly improves the explainability of the model, and enables the model to not only give a simple unknown class discrimination when facing unknown targets, but also provide more valuable information, thereby providing solid technical support for the effective deployment of the model in complex real-world scenarios.

Description

Technical Field

[0001] This invention belongs to the field of open set recognition technology, and more specifically, relates to an interpretable open set recognition method based on a hierarchical visual semantic attribute space. Background Technology

[0002] Traditional deep learning models are primarily designed in closed environments, assuming that training and testing data belong to the same category and follow the independent and identically distributed (i.i.d.) assumption. However, in real-world applications, models may encounter unseen categories during training. Therefore, effectively handling these unseen categories becomes a crucial issue. To address this challenge, open-set recognition techniques have emerged, designed to identify and process samples from categories the model has never encountered before.

[0003] Existing open-set recognition methods primarily focus on single-level classification tasks, where the model receives an image as input and generates a label prediction and its confidence score. The input is then judged to belong to a known or unknown category by comparing the confidence score with a preset threshold. However, when faced with targets of unknown categories, existing methods can only classify the target as "unknown," failing to provide more information about the characteristics of the unknown target. This results in poor interpretability and makes it difficult to meet the deployment needs of models in the real world. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of existing technologies, this invention provides an interpretable open set recognition method based on hierarchical visual semantic attribute space. The purpose is to overcome the defects of existing open set recognition methods, which mainly focus on single-level classification tasks and can only output single category information, failing to explain the characteristics of unknown targets. This results in these models having insufficient interpretability and being difficult to adapt to the model deployment needs in the real world.

[0005] To achieve the above objectives, this invention provides an interpretable open set recognition method based on a hierarchical visual semantic attribute space, comprising:

[0006] Collect training set label space The binary attributes possessed by all fine-grained categories are used to construct an attribute space; in the attribute space, the attribute prototypes of all fine-grained categories form an attribute matrix; wherein, all attributes belonging to the i-th fine-grained category are called the attribute prototypes of the i-th fine-grained category;

[0007] Cluster the attribute prototypes in the attribute matrix to obtain J clusters and the attribute prototypes contained in each cluster; wherein, each cluster corresponds to a coarse-grained category, and each correspondence from the fine-grained category to the coarse-grained category constitutes a mapping function;

[0008] According to the mapping function, the frequency of all fine-grained category shared attributes in each coarse-grained category is calculated, the attribute with a higher shared frequency is taken as a coarse-grained attribute of the coarse-grained category, and the attribute with a lower shared frequency is taken as a fine-grained attribute of the coarse-grained category; wherein the shared attribute refers to an attribute existing in at least one attribute prototype;

[0009] The to-be-recognized image is input into the trained classification prediction model, and a corresponding attribute prediction result and a classifier-guided fine-grained category prediction score are output; based on the mapping function, the fine-grained category with the highest score is mapped as a coarse-grained category prediction of the to-be-recognized image, and the coarse-grained attribute contained in the attribute prediction result is shielded based on the corresponding coarse-grained attribute, to obtain a fine-grained attribute predicted for the to-be-recognized image.

[0010] Further, after obtaining the fine-grained attribute predicted for the to-be-recognized image, the following steps are further included:

[0011] According to the coarse-grained category prediction, the to-be-recognized image is assigned to a corresponding fine-grained attribute space, the similarity between the fine-grained attribute predicted for the to-be-recognized image and all fine-grained attribute prototypes in the fine-grained attribute space is calculated, and the corresponding similarity values are sequentially spliced as a category prediction score S a guided by the attribute of the to-be-recognized image; wherein all fine-grained attributes of the coarse-grained category constitute the fine-grained attribute space;

[0012] The category prediction score S a guided by the attribute is weighted with the fine-grained category prediction score S c guided by the classifier, to obtain a category prediction score S ca guided by the classifier-attribute; according to the maximum value in the category prediction score S ca , it is judged whether the to-be-recognized image belongs to a known category or an unknown category;

[0013] If it is a known category, the fine-grained category corresponding to the maximum value is taken as the category prediction guided by the classifier-attribute of the to-be-recognized image;

[0014] If it is an unknown category, the coarse-grained category prediction of the to-be-recognized image and the corresponding fine-grained attribute are output.

[0015] Further, the similarity is a cosine distance between the fine-grained attribute predicted for the to-be-recognized image and each fine-grained attribute prototype in the corresponding fine-grained attribute space;

[0016] Based on the maximum value, the maximum classifier-attribute guided open set recognition score is used to judge whether the to-be-recognized image belongs to a known category or an unknown category.

[0017] ​Further, the attribute prototypes in the attribute matrix are clustered to obtain J clusters and attribute prototypes contained in each cluster, including:

[0018] Initialize J cluster centers c j wherein J represents the number of coarse-grained categories;

[0019] For each attribute prototype in the attribute matrix, a clustering method of minimizing the total intra-cluster sum of squares is used to obtain J clusters and attribute prototypes contained in each cluster.

[0020] Further, according to the mapping function, the frequency of shared attributes of all fine-grained categories in each coarse-grained category is calculated, including:

[0021] For the jth coarse-grained category, the attribute occurrence frequency R j = {r j1 ,r j2 ,...,r jd ,...,r jD} is calculated as:

[0022]

[0023] wherein r jd represents the frequency of occurrence of attribute a d in the jth coarse-grained category, is the number of fine-grained categories contained in the jth coarse-grained category, A i represents the attribute prototype of the ith fine-grained category, and A i = {a i1 ,a i2 ,..,a id ,...,a iD}, a id represents the dth attribute of the ith fine-grained category, a id = 1 indicates that the fine-grained category i has attribute a d , a id = 0 indicates that the fine-grained category i does not have attribute a d ; and D represents the dimension of the attribute space.

[0024] Further, the attributes with higher shared frequencies are taken as coarse-grained attributes of the coarse-grained category, and the attributes with lower shared frequencies are taken as fine-grained attributes of the coarse-grained category, including:

[0025] Based on the frequency of shared attributes of all fine-grained categories in the jth coarse-grained category, the coarse-grained attribute prototype of the jth coarse-grained category is calculated

[0026]

[0027] wherein, θ is a threshold value; a coarse-grained attribute prototype of the jth coarse-grained category the contained attributes are all coarse-grained attributes of the jth coarse-grained category;

[0028] a fine-grained attribute prototype belonging to the ith fine-grained category of the jth coarse-grained category is calculated

[0029]

[0030] wherein, Mask j The element with a medium of 1 is a fine-grained attribute of the jth coarse-grained category, and a unit vector formed by all fine-grained attributes of the jth coarse-grained category spans a fine-grained attribute space; ⊙ represents a multiplication operator.

[0031] Further, a training image in the training set is used to train the classification prediction model, and a trained classification prediction model is obtained; wherein the classification prediction model comprises a data preprocessing module, a feature extractor and a double-branch mapping network;

[0032] The data preprocessing module is used to adjust the training image to an image of a suitable network size, and perform normalization processing, convert to a tensor form, and obtain a preprocessed image;

[0033] The feature extractor is used to extract visual features of the preprocessed image;

[0034] The double-branch mapping network comprises a fine-grained category mapping module and an attribute mapping module; the fine-grained category mapping module is used to map the visual features of the image to the label space obtain a classifier-guided fine-grained category prediction score p c ; the attribute mapping module is used to map the visual features of the image to the attribute space, and obtain an attribute prediction result D represents the dimension of the attribute space.

[0035] The application also provides an interpretable open set recognition system based on a hierarchical visual semantic attribute space, comprising a computer readable storage medium and a processor;

[0036] The computer readable storage medium is used to store executable instructions;

[0037] The processor is used to read the executable instructions stored in the computer readable storage medium to execute the open set recognition method described in any one of the above.

[0038] The application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the explainable open set identification method according to any one of the above.

[0039] The application further provides a computer program product, which comprises a computer program, and when the computer program is executed on a computer, the computer program enables the computer to execute the explainable open set identification method according to any one of the above.

[0040] Overall, the above technical solutions conceived by the application can achieve the following beneficial effects:

[0041] (1) The explainable open set identification method based on the hierarchical visual semantic attribute space of the application divides the fine-grained categories and the corresponding attribute prototypes into a coarse-grained hierarchical structure by hierarchical division, which realizes hierarchical division of the categories from fine-grained to coarse-grained, and the mapping function constructed reflects the mapping relationship between the coarse-grained categories, the fine-grained categories and the attribute prototypes. For each coarse-grained level, the hierarchical relationship of each attribute (whether the attribute is a coarse-grained attribute or a fine-grained attribute belonging to the corresponding coarse-grained category) is assigned according to the frequency of the shared attributes of all fine-grained categories in each coarse-grained level, thereby obtaining the coarse-grained attribute space and the fine-grained attribute space corresponding to each coarse-grained category. Based on the mapping function and the coarse-grained attribute space and the fine-grained attribute space corresponding to each coarse-grained category, for the image to be identified, whether it is a known category or an unknown category, it can be at least divided into the corresponding coarse-grained category and the corresponding fine-grained attribute. Compared with the traditional open set identification technology that can only identify the unknown category as "unknown", the method of the application can also provide the coarse-grained category and the fine-grained attribute corresponding to the unknown category, which has significant superiority compared with the traditional open set identification technology, greatly improves the explainability of the model, and can adapt to the model deployment needs in the real world.

[0042] (2) Further, by calculating the similarity between the fine-grained attribute of the image to be identified and all fine-grained attribute prototypes in the fine-grained attribute space, the attribute-guided category prediction score of the sample is obtained, the classifier-guided category prediction score and the attribute-guided category prediction score are weighted, and the maximum value in the prediction score is selected as the classifier-attribute guided category prediction of the image to be identified. That is, for the known category, the method of the application can also accurately identify the corresponding fine-grained category.

[0043] (3) As a preferred, the double-branch mapping network in the application can effectively connect after any feature extractor, and has the characteristics of simple structure and parameter reduction.

[0044] In summary, the method of this invention can not only accurately identify whether a target belongs to a known or unknown category, but also clearly define its coarse-grained category and specific attributes. This method has significant advantages over traditional open-set recognition techniques, greatly improving the interpretability of the model. When faced with unknown targets, the model can not only provide simple unknown category discrimination, but also offer more valuable information, thus providing solid technical support for the effective deployment of the model in complex and ever-changing real-world scenarios. Attached Figure Description

[0045] Figure 1 This is a flowchart of the interpretable open set recognition method based on hierarchical visual semantic attribute space in an embodiment of the present invention.

[0046] Figure 2 This is an overall framework diagram of the interpretable open set recognition method based on hierarchical visual semantic attribute space in an embodiment of the present invention.

[0047] Figure 3 This paper presents a comparison of the interpretable open set recognition method based on hierarchical visual semantic attribute space in this invention with the experimental results of open set recognition on different datasets using traditional methods. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Example 1

[0050] like Figure 1 , Figure 2 As shown in the figure, this invention discloses an interpretable open set recognition method based on a hierarchical visual semantic attribute space. The method includes the following steps:

[0051] Step 1: Collect the label space of the training set We construct an attribute space by identifying the binary attributes possessed by all fine-grained categories. Within this attribute space, the attribute prototypes of all covered fine-grained categories form an attribute matrix; where all attributes belonging to the i-th fine-grained category are called the attribute prototypes of the i-th fine-grained category.

[0052] In this embodiment of the invention, the constructed attribute space is denoted as... Among them, a nrepresents a single binary attribute in the attribute space, D represents the dimension of the attribute space, i.e., the total number of attributes, A i is located in the attribute space, i represents the index of the fine-grained category, i∈{1,2,…,K}, and A i i i1 i2 id iD is the attribute prototype of each fine-grained category, and is a subset of , a id =1 indicates that the fine-grained category i has the attribute a d , and a id =0 indicates that the fine-grained category i does not have the attribute a d . For a classification task with K fine-grained categories, the attribute prototypes A i of all fine-grained categories form an attribute matrix In this embodiment, the element A id of the attribute matrix represents the d-th attribute of the i-th fine-grained category, i∈{1,2,…,K}, d∈{1,2,…,D}.

[0053] As a preferred implementation, the binary attributes of all fine-grained categories in the training set can be collected by manual labeling or querying a large language model to ensure the accuracy and comprehensiveness of the attribute data.

[0054] Step two: clustering all attribute prototypes in the attribute matrix to obtain J clustering clusters and the attribute prototypes contained in each clustering cluster; wherein each clustering cluster corresponds to a coarse-grained category, thereby realizing hierarchical division of the categories from fine-grained to coarse-grained. Each corresponding relationship from a fine-grained category to a coarse-grained category constitutes a mapping function represents the label space of the coarse-grained category.

[0055] In the embodiment of the application, the total within-cluster sum of squares is minimized to cluster all attribute prototypes in the attribute matrix to obtain J clustering clusters and the attribute prototypes contained in each clustering cluster; specifically including: initializing J clustering centers c j representing J levels, i.e., J coarse-grained categories; for each attribute prototype (in this embodiment, each row in the attribute matrix, all attributes belonging to the i-th category), the total within-cluster sum of squares (TWCSS) is minimized, and its expression is:

[0056]

[0057] wherein J is the number of clustering clusters, C​​​​​j denotes the jth cluster, A i denotes the cluster C j denotes the attribute prototype of the ith category in

[0058] By clustering all attribute prototypes in the attribute matrix, J clusters are obtained, and the attribute prototypes contained in each cluster; each cluster in the clustering represents a coarse-grained hierarchical structure (characterizing a coarse-grained category; for example, for the fine-grained categories of Largehead gudgeon and grass carp, the corresponding coarse-grained category is fish; for the fine-grained categories of dog, cat, and horse, the corresponding coarse-grained category is mammal). For the label space , all labels (fine-grained categories) are assigned coarse-grained labels denotes the label space of the coarse-grained category, and a mapping function is obtained from each corresponding relationship of the fine-grained category to the coarse-grained category Mapping function reflects the mapping relationship between the coarse-grained category, the fine-grained category, and the attribute prototype.

[0059] Step three: according to the mapping function , the frequency of all fine-grained categories sharing attributes in each coarse-grained category is calculated, the attribute with a high sharing frequency is taken as the coarse-grained attribute of the coarse-grained category, and the attribute with a low sharing frequency is taken as the fine-grained attribute of the coarse-grained category, so as to obtain the coarse-grained attribute and fine-grained attribute space corresponding to each coarse-grained category; wherein the sharing attribute refers to an attribute existing in at least one attribute prototype.

[0060] In the embodiment of the application, according to the hierarchical division of the coarse-grained category, the frequency of all attributes appearing in each coarse-grained category is calculated. For the jth coarse-grained category, the attribute appearance frequency R j j1 j2 jd jD is calculated as:

[0061]

[0062] , wherein r jd denotes the frequency of the attribute a d appearing in the jth coarse-grained category, is the number of fine-grained categories contained in the jth coarse-grained category, A i i1 i2 id iD ​​​​​​​​}. Then, the coarse-grained attribute prototype of the jth coarse-grained category can be obtained

[0063]

[0064] where θ is a threshold value, which is empirically determined; a higher value indicates that the attribute needs to be shared by more fine-grained categories in order to be divided into a coarse-grained attribute; r jd represents the frequency of occurrence of the attribute a d in the jth coarse-grained category. The coarse-grained attribute prototype of the jth coarse-grained category contains all the coarse-grained attributes of the jth coarse-grained category.

[0065] For the fine-grained category i and its corresponding coarse-grained category j, the coarse-grained attributes of the fine-grained category i are masked to calculate the fine-grained attribute prototype of the fine-grained category i (characterizing the attributes unique to the fine-grained category i):

[0066]

[0067] where m d = 1-a jd , the unit vector formed by the elements equal to 1 in Mask j spans the fine-grained attribute space (i.e., the attributes with a value of 0 in the coarse-grained attribute prototype are all the fine-grained attributes of the jth coarse-grained category, and all the fine-grained attributes constitute the fine-grained attribute space of the jth coarse-grained category); ⊙ denotes the Hadamard product operator.

[0068] By traversing all J coarse-grained categories, the corresponding J fine-grained attribute spaces can be obtained. Each fine-grained attribute space contains attribute prototypes ( categories); is the number of categories contained in the jth coarse-grained category.

[0069] Step four: input the image to be identified into the trained classification prediction model, output the corresponding attribute prediction result and the fine-grained category prediction score guided by the classifier; based on the mapping function map the fine-grained category with the highest score to the coarse-grained category prediction of the image to be identified; based on the coarse-grained category prediction, assign the image to be identified to the corresponding fine-grained attribute space, and mask the coarse-grained attributes contained in the attribute prediction result according to the coarse-grained attributes of the coarse-grained category prediction, to obtain the fine-grained attributes of the attribute prediction result (predicted for the image to be identified)

[0070] Further, the fine-grained attributes predicted for the image to be identified The corresponding similarity values are sequentially spliced as the class prediction score S of the image attribute guided to be identified a ; the attribute guided class prediction score S a is obtained by the corresponding classifier guided fine-grained class prediction score S c is weighted and processed to obtain the classifier-attribute guided class prediction score S ca ; the fine-grained class corresponding to the maximum value in the class prediction score is taken as the classifier-attribute guided class prediction of the image to be identified, and whether the image to be identified belongs to a known class or an unknown class is judged according to the maximum value, thereby completing the open set recognition.

[0071] As a preferred implementation, in step four, the training image in the training set is used to train the classification prediction model to obtain the trained classification prediction model. In the embodiment of the present application, the classification prediction model includes a data preprocessing module, a feature extractor Φ F and a double-branch mapping network.

[0072] The data preprocessing module is used to adjust the input training image to an image of a suitable network size and perform normalization processing to convert it into a tensor form to obtain a preprocessed image. In the embodiment of the present application, the data preprocessing module adjusts the input image to a size of 224x224 and performs normalization processing to convert it into a tensor form to ensure the standardization and consistency of the data. The feature extractor Φ F is used to extract the visual features of the preprocessed image.

[0073] The double-branch mapping network includes a fine-grained class mapping module Φ C and an attribute mapping module Φ A ; the two modules share the feature extractor of the model and are each composed of a mapping layer and a fully connected layer in series to obtain the output information of the attribute prediction result and the classifier guided fine-grained class prediction score p c :

[0074]

[0075] Specifically, the fine-grained class mapping module is used to map the visual features of the image to the label space to obtain the classifier guided fine-grained class prediction score p c ; and the attribute mapping module is used to map the visual features of the image to the attribute space to obtain the attribute prediction result The double-branch mapping network in the embodiment of the present application can effectively connect to any feature extractor, and has the characteristics of simple structure and parameter reduction.

[0076] As a preferred implementation, in the embodiment of the present application, the cross-entropy loss function is used to calculate the class prediction loss of the training image And the binary cross-entropy loss function is used To calculate the attribute prediction loss of the training image. The parameters of the feature extractor and the double-branch mapping network are updated by the back propagation algorithm. Wherein, the loss function is calculated as follows:

[0077]

[0078] In the formula, λ is a hyperparameter that controls the weight; y, A represents the real fine-grained class and attribute of the current training image.

[0079] As a preferred implementation, the image to be recognized Is input to the above trained classification prediction model, and the corresponding attribute prediction result And the classifier-guided fine-grained class prediction score In this embodiment, Indicates the test sample set. The class prediction result with the highest score in the classifier-guided class prediction score is selected, and the coarse-grained mapping function Obtains the coarse-grained class prediction j of the image to be recognized:

[0080]

[0081] Wherein, argmax represents the index of the maximum value.

[0082] Based on the coarse-grained class prediction, the image to be recognized is assigned to the fine-grained attribute space corresponding to the jth coarse-grained class, and the coarse-grained attribute information contained in the attribute prediction result Is shielded to obtain the predicted fine-grained attribute of the image to be recognized

[0083] Then, the similarity between the predicted fine-grained attribute of the image to be recognized And all fine-grained attribute prototypes in the fine-grained attribute space In this embodiment, the similarity calculation method is the cosine distance between two attribute vectors. After splicing each similarity value, the attribute-guided class prediction score S of the sample is obtained a :

[0084]

[0085] Wherein, Concat[k1,k2,…,k n ] represents splicing elements k1, k2,…, k n ; Representing fine-grained attributes predicted for an image to be recognized Cosine distance between fine-grained attribute prototypes in fine-grained attribute space is used to represent similarity between and .

[0086] Assuming β is an attribute affinity factor (empirically determined), a classifier-attribute guided class prediction score S ca is obtained: ca S c = S a + βS ca .

[0087] The final predicted class is the class corresponding to the maximum value of S ca : A maximum classifier-attribute guided open-set recognition score (MCAS) is used to determine whether the image to be recognized belongs to a known class or an unknown class (argmax{S ca}):

[0088] MCAS = max(S ca ).

[0089] where MCAS represents the maximum value of S ca ; when MCAS exceeds a preset threshold, the predicted class is a known class, and the corresponding fine-grained class is Otherwise, the predicted class is an unknown class, and the coarse-grained class and the corresponding fine-grained attribute of the unknown class can be obtained.

[0090] The hierarchical visual semantic attribute space-based interpretable open-set recognition method of the present application divides fine-grained classes and corresponding attribute prototypes into a coarse-grained hierarchical structure by hierarchical division, which realizes hierarchical division of classes from fine-grained to coarse-grained, and the mapping function Reflects the mapping relationship between the coarse-grained category, the fine-grained category and the attribute prototype of each attribute. For each coarse-grained level, according to the frequency of the shared attributes of all fine-grained categories in each coarse-grained level, the hierarchical relationship of each attribute (whether the attribute is a coarse-grained attribute or a fine-grained attribute belonging to the corresponding coarse-grained category) is assigned, and thus the coarse-grained attribute and fine-grained attribute space corresponding to each coarse-grained category is obtained. Based on the mapping function and the coarse-grained attribute and fine-grained attribute space corresponding to each coarse-grained category, for the image to be identified, whether it is a known category or an unknown category, it can be at least divided into the corresponding coarse-grained category and the corresponding fine-grained attribute. Compared with the traditional method of identifying unknown categories as "unknown", the method of the present application can also provide the corresponding coarse-grained category and the corresponding fine-grained attribute of the unknown category, which has a significant advantage over the traditional open set recognition technology, greatly improving the interpretability of the model and adapting to the model deployment needs in the real world.

[0091] Further, by calculating the similarity between the fine-grained attribute of the image to be identified and all fine-grained attribute prototypes in the fine-grained attribute space, the attribute-guided class prediction score of the sample is obtained. The classifier-guided class prediction score and the attribute-guided class prediction score are weighted, and the maximum value in the prediction score is selected as the classifier-attribute-guided class prediction of the image to be identified. That is, for known categories, the method of the present application can also accurately identify the corresponding fine-grained category.

[0092] As shown in Figure 3 The open set recognition performance of the method of the present application compared with the traditional method on CUB, FGVC, LAD and AWA data sets is shown. AUROC and OSCR indicators commonly used in the field of open set recognition are adopted. On the four data sets, the method of the present application improves the AUROC indicator by 1.81% and the OSCR indicator by 2.33%.

[0093] The open set recognition method based on hierarchical visual semantic attribute space designed by the present application additionally utilizes the hierarchical information of attributes and categories for assistance in the open set recognition process. Compared with the defect that the existing method cannot provide unknown sample information, the method proposed by the present application not only can mark the unknown sample as an unknown category, but also can provide the coarse-grained category information and related fine-grained attributes. And experiments prove that compared with the existing method, the method has a significant improvement in open set recognition performance, and has the advantages of high precision and strong interpretability.

[0094] Embodiment 2

[0095] The embodiment of the present application provides a kind of based on hierarchical visual semantic attribute space's explainability open set identification system, including memory and processor, memory is stored with computer program, when processor executes computer program, it realizes the step of the explainability open set identification method based on hierarchical visual semantic attribute space in above-mentioned embodiment 1.

[0096] The related technical solutions are as above, and will not be repeated here.

[0097] Embodiment 3

[0098] The embodiment of the present application provides a kind of computer readable storage medium, it is stored with computer program on, computer program is executed when processor realizes the step of the explainability open set identification method based on hierarchical visual semantic attribute space in above-mentioned embodiment 1.

[0099] Specifically, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0100] The related technical solutions are as above, and will not be repeated here.

[0101] Embodiment 4

[0102] The embodiment of the present application provides a kind of computer program product, including computer program, when the computer program runs on computer, make computer execute the step of the explainability open set identification method based on hierarchical visual semantic attribute space in above-mentioned embodiment 1.

[0103] The related technical solutions are as above, and will not be repeated here.

[0104] Those skilled in the art readily understand that the above only describes the preferred embodiments of the present application, and does not limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A hierarchical visual semantic attribute space based explainability open set identification method, characterized in that, The method comprises the following steps: Collecting training set label space binary attributes of all fine-grained categories to construct an attribute space; in the attribute space, attribute prototypes of all fine-grained categories form an attribute matrix; wherein all attributes belonging to the i-th fine-grained category are called attribute prototypes of the i-th fine-grained category. binary attributes of all fine-grained categories to construct an attribute space; in the attribute space, attribute prototypes of all fine-grained categories form an attribute matrix; wherein all attributes belonging to the i-th fine-grained category are called attribute prototypes of the i-th fine-grained category.​ Clustering the attribute prototypes in the attribute matrix yields... There are 1 cluster and the attribute prototype contained in each cluster; wherein each cluster corresponds to a coarse-grained category, and each correspondence from the fine-grained category to the coarse-grained category constitutes a mapping function; According to the mapping function, the frequency of the shared attributes of all fine-grained categories in each coarse-grained category is calculated, the attributes with high shared frequency are taken as coarse-grained attributes of the coarse-grained category, and the attributes with low shared frequency are taken as fine-grained attributes of the coarse-grained category; wherein the shared attribute refers to an attribute existing in at least one attribute prototype; The to-be-recognized image is input into the trained classification prediction model, and the corresponding attribute prediction result and the classifier-guided fine-grained category prediction score are output; based on the mapping function, the fine-grained category with the highest score is predicted as the coarse-grained category prediction of the to-be-recognized image, and the coarse-grained attributes contained in the attribute prediction result are shielded based on the corresponding coarse-grained attributes of the coarse-grained category prediction, to obtain the fine-grained attribute prediction of the to-be-recognized image. 2.The explainable open set identification method of claim 1, wherein, After obtaining the fine-grained attribute prediction of the to-be-recognized image, the method further comprises the following steps: According to the coarse-grained category prediction, the to-be-recognized image is assigned to a corresponding fine-grained attribute space, similarities between a predicted fine-grained attribute of the to-be-recognized image and all fine-grained attribute prototypes in the fine-grained attribute space are calculated, and corresponding similarity values are sequentially spliced to serve as a category prediction score guided by an attribute of the to-be-recognized image ; wherein all fine-grained attributes of the coarse-grained category constitute the fine-grained attribute space Category prediction score guided by the attribute The fine-grained category prediction score guided by the classifier After weighting, the classifier-attribute-guided class prediction scores are obtained. Predict scores based on the categories. The maximum value in the value indicates whether the image to be identified belongs to a known category or an unknown category; If the known class, then the maximum value corresponding to the fine-grained class As a classifier-guided class prediction of an image to be recognized If it is an unknown category, the coarse-grained category prediction of the to-be-recognized image and the corresponding fine-grained attribute are output.

3. The interpretable open set identification method of claim 2, wherein, The similarity is the cosine distance between the fine-grained attribute prediction of the to-be-recognized image and each fine-grained attribute prototype in the corresponding fine-grained attribute space; Based on the maximum value, the maximum classifier-attribute-guided open set recognition score is used to determine whether the to-be-recognized image belongs to a known category or an unknown category. 4.The open set identification method of interpretability according to any one of claims 1-3, characterized in that, Clustering the attribute prototypes in the attribute matrix yields... Each cluster and the attribute prototypes contained in each cluster include: Initialization one cluster center wherein, denotes the number of coarse-grained classes; For each attribute prototype in the attribute matrix, clustering is performed by using a method of minimizing total intra-cluster sum of squares to obtain a number of clustering clusters and attribute prototypes contained in each clustering cluster.

5. The explainable open set identification method of any one of claims 1-3, wherein, According to the mapping function, the frequency of the shared attributes of all fine-grained categories in each coarse-grained category is calculated, including: For the first coarse-grained category, the attribute occurrence frequency is calculated as: wherein, denotes the number of attributes in the th coarse-grained class, the frequency of occurrence of the th attribute in the th coarse-grained class, denotes the number of fine-grained classes contained in the th coarse-grained class, , denotes the th attribute of the th fine-grained class, denotes the presence of the attribute in the th fine-grained class, , denotes the absence of the attribute in the th fine-grained class, ; denotes the dimensionality of the attribute space.

6. The interpretable open set identification method of claim 5, wherein, The attributes with higher shared frequency are taken as coarse-grained attributes of the coarse-grained category, and the attributes with lower shared frequency are taken as fine-grained attributes of the coarse-grained category, including: Based on the The frequency of attributes shared by all fine-grained categories in a coarse-grained category is calculated. Coarse-grained attribute prototypes for each coarse-grained category : wherein, is a threshold value; the coarse-grained property prototype of the first coarse-grained category all coarse-grained properties of the first coarse-grained category; Calculate the first The first coarse-grained category Fine-grained attribute prototypes for each fine-grained category : wherein, , The element with medium equal to 1 is the fine-grained attribute of the first coarse-grained category, and the unit vector spanned by all fine-grained attributes of the first coarse-grained category forms the fine-grained attribute space; represents a take product operator. 7.The explainable open set identification method of claim 1, wherein, The training image in the training set is used to train the classification prediction model, and the trained classification prediction model is obtained; wherein the classification prediction model comprises a data preprocessing module, a feature extractor, and a double-branch mapping network; The data preprocessing module is used to adjust the training image to an image with a suitable network size, perform normalization processing, convert to a tensor form, and obtain a preprocessed image; The feature extractor is used to extract the visual features of the preprocessed image; The double-branch mapping network comprises a fine-grained category mapping module and an attribute mapping module; the fine-grained category mapping module is configured to map visual features of the image to the label space , to obtain a classifier-guided fine-grained category prediction score ; the attribute mapping module is configured to map visual features of the image to the attribute space, to obtain an attribute prediction result , , wherein d represents a dimension of the attribute space.

8. A tiered visual semantic attribute space based explainable open set identification system, comprising: The computer readable storage medium and the processor are included; The computer readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer readable storage medium to execute the interpretability open set recognition method of any one of claims 1-7.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the interpretability open set recognition method of any one of claims 1-7.

10. A computer program product, characterised in that, The computer program makes the computer execute the interpretability open set recognition method of any one of claims 1-7 when the computer program runs on the computer.

Images