A zero-shot image classification method based on dictionary learning

Abstract

The present invention discloses a zero-sample image classification method based on dictionary learning, which belongs to the technical field of dictionary learning. It includes obtaining an image data set; preprocessing the image data set to obtain a training set and a test set; constructing a core dictionary learning framework, and inputting the training set into a network model based on dictionary learning to obtain picture image features, the network model based on dictionary learning includes a learning class model, structural alignment, domain adaptation, and class prototype constraint modules; by calculating the loss function of the dictionary learning network model, the parameters in the model are adjusted; the picture image features and semantic information features are jointly trained to obtain a classifier; the test set is input into the classifier for classification. Compared with the prior art, the present invention makes the generated visual image features more diversified and improves the classification accuracy by adding class prototype constraints in the network based on the dictionary learning framework.

Description

Technical Field

[0001] The invention discloses a zero-sample image classification method based on dictionary learning, and belongs to the technical field of dictionary learning. Background Art

[0002] With the continuous development of machine learning, deep learning has made great progress in image classification, target detection and other aspects. In traditional image recognition or classification models, all images that need to be distinguished are generally trained in a training set to obtain a model, and then the test images are put into the model for discrimination. The test class category has already appeared in the training set. Therefore, if you want to obtain higher accuracy in prediction, you need to spend a lot of manpower, material resources and financial resources. However, with the continuous increase in the number of recognized object categories, it is unrealistic to collect a large number of image samples for many rare species (such as reptiles that appear in the Arctic), making it difficult for these supervised learning methods to identify invisible classes that did not appear in the training stage.

[0003] To solve the above problems, Larochelle H et al. first proposed the zero-sample learning problem. Suppose you have never heard of the animal category "elk" or seen its appearance before, but you can learn a visual classifier well based on the following description: "head like a horse, horns like a deer, hooves like a cow, tail like a donkey", so even if you have never seen it, you can judge it as an "elk" based on the above description.

[0004] For zero-shot learning, it is to identify samples of categories that have never been seen during training. It is to transfer the categories that have been seen to the categories that have not been seen, so that the model trained in the training phase can learn more essential things and transfer them to the categories that have not been seen.

[0005] Traditional zero-shot learning methods mainly use manually annotated attributes as intermediate category description information to achieve zero-shot category prediction. The earliest one was the unseen class detection method based on attribute class transfer proposed by H. Lampert et al., who proposed the DAP model. This model learns a binary classifier for each attribute. However, the biggest disadvantage of this method is that it uses the attribute classifier to identify and classify the test category, resulting in a high recognition accuracy for the attribute and a low recognition accuracy for the test class. Suzuki et al. set a weight for each attribute; in order to improve the efficiency of model training, Parkash et al. used the attribute as a carrier during the training stage and used it as feedback to return more information to the model. Later, Socher et al. transformed the zero-shot learning problem into a subspace problem for the first time. They used a two-layer neural network as a cross-modal mapping model to simultaneously map images and class labels to the semantic space, and used similarity measurement methods (cosine similarity, K nearest neighbor method) to determine the class labels of the test class input images, but it caused the mapping domain offset problem; Zhang et al. proposed an idea to map semantic vectors to image feature space. This model is completely opposite to the previous idea of ​​mapping image features to semantic space. Since the input of class labels is significantly less than the number of input images, it effectively alleviates the hub problem to a certain extent. The SJE model proposed by Akata et al. utilizes a variety of auxiliary semantic information sources, combines various compatibility functions together by using convex combination, and then uses multiple views to comprehensively judge to improve the confidence of the predicted class label.

[0006] However, the traditional zero-shot learning method mainly builds a semantic space constructed by attributes or word embeddings to learn the mapping of training data to the space, and then the test sample is mapped to the semantic space through this mapping and outputs the prediction in combination with the nearest neighbor method. However, such direct training causes the problem of domain shift for the test sample. Summary of the invention

[0007] Purpose of the invention: The present invention designs a zero-shot image classification method based on dictionary learning, realizes the alignment of visual and semantic structures through a coupled dictionary learning framework; extends the structure alignment method from visible classes to invisible classes through domain adaptation methods, effectively preventing the overfitting problem; and achieves better alignment effects by retaining the geometric properties of visual and semantic class prototypes.

[0008] To achieve the above object, the present invention provides a zero-shot image classification method based on dictionary learning, the method comprising the following steps:

[0009] Step 1: Obtain a general image dataset for computer vision;

[0010] Step 2: preprocessing the image data set to obtain a training set and a test set, wherein there is no intersection between the training set and the test set, and both the training set and the test set include image information and semantic information;

[0011] Step 3: Construct a coupled dictionary learning framework to achieve visual-semantic structure alignment, use domain adaptation methods to extend the structure alignment method from visible classes to unseen classes, and preserve the geometric properties of visual and semantic class prototypes;

[0012] Step 4: Input the semantic information features and visual image features into the classifier network and train them together to obtain a classifier;

[0013] Step 5: Input the test set into the classifier to classify the test set.

[0014] Furthermore, in step 1, the general image dataset of computer vision is any one of the AWA, CUB, SUN, and aPY image datasets.

[0015] Furthermore, in step 2, the image dataset preprocessing method is as follows: ResNet101 is used to preprocess the image, and the dataset image in step 1 is sent to the ResNet101 network to output a 2048-dimensional image feature vector, and all categories are divided into training dataset image feature representation and test dataset image feature representation. The semantic dataset of the category uses attribute description information.

[0016] Furthermore, in step 3, the method for constructing a coupled dictionary learning framework is as follows:

[0017] (1) The first step in building a coupled dictionary learning framework is to learn class prototypes in the semantic space and image feature space; the visible class image features X obtained after data preprocessing are s And the semantic vector C of the visible and invisible categories s ∈R m×K / C u ∈R m×L As the training set of the model, in the semantic space, the known semantic vector C s ∈R m×K / C u ∈R m×L As the prototype of the visible class and the invisible class; in the visual space, due to the visible class image feature X s And the class label corresponding to each image is a one-hot vector It is known that by minimizing the loss function We can get the class prototype P of the visible class in the visual space s ;

[0018] (2) The second step in constructing the coupled dictionary learning framework is to structurally align the semantic space and the image feature space, and to achieve visual-semantic structure alignment through a coupled dictionary learning framework; find a new representation in the semantic space and the visual feature space respectively. To represent each class, and to force 2 new representations The same in the intermediate subspace, that is, All other parameters are known, and the dictionary D1 in the visual space and the dictionary D2 in the semantic space are learned by minimizing the loss function. The loss function is as follows:

[0019]

[0020]

[0021] Among them, D1 and D2 are dictionaries in visual space and semantic space respectively. are the values ​​of the dictionaries D1 and D2 in the i-th column, C s is the semantic vector of the visible class, P s is the class prototype of the visible class in the visual space, and λ is a parameter greater than 0;

[0022] (3) The third step in building the coupled dictionary learning framework is to use domain adaptation methods to alleviate the domain shift problem caused by using only the dictionary learned from the visible class prototypes in structure alignment: by using the class prototype C in the semantic space of the unseen class u Automatically learn class prototypes P in the visual space of unseen classes u , that is, due to the parameters D1, D2, Z u , C u It is known that P can be obtained by minimizing the loss function u :

[0023]

[0024]

[0025] Where D1 and D2 are dictionaries in visual space and semantic space respectively. are the values ​​of the dictionaries D1 and D2 in the i-th column, Z u is the representation of the unseen class in the alignment space, C u is the semantic vector of the unseen class, and λ is a parameter greater than 0;

[0026] (4) In order to preserve the geometric properties of the visual and semantic class prototypes, the class prototype constraints are added to achieve better alignment when the class prototypes in the two spaces are projected into the intermediate space; the dictionary learning framework introduces the graph Laplacian term to maintain the geometric properties of the invisible class prototypes in the visual feature space. For the invisible class, a fully connected graph G is constructed, with v vertices to represent our invisible categories. For any two categories i and j, the edge A between vertices i and j is used. ij To represent the relationship between two vertices, A ij Specifically expressed as:

[0027]

[0028] in, They represent two different types of semantic vectors respectively, and δ is a parameter not greater than 1;

[0029] Through the loss function Adjust the invisible class prototype P learned last u , They represent the class prototype vectors of the image feature space of different categories, u represents the invisible class category. If the similarity of the semantic class prototypes of two classes is greater than the preset value, then A ij And the loss function L pc The values ​​are all smaller than the preset values; since A is the edge A between the above vertices i and j ij The vector composed of D = diag(D 1 ,…D u ) Each entry in the diagonal matrix is ​​the sum of each column of A. Let L u =DA, loss function L pc It is expressed as:

[0030]

[0031] in, Respectively represent the prototype vectors of the image feature space class of two different types in the invisible class, A ij is the relationship between the two different types, L u =DA, tr represents the trace; the final objective function L is:

[0032]

[0033] in, are the class prototype representations in the visual feature space and the semantic feature space, respectively. s is the new subspace class prototype representation Z u is a new representation of the unseen classes in the alignment space, C s is the semantic vector of the visible class, Cu is the semantic vector of the unseen class, D1 and D2 are dictionaries in the visual space and the semantic space respectively, P s is the prototype of the visible image feature class, P u is the prototype representation of the unseen class in the visual feature space, X s is the visible class image feature, The class label corresponding to each image is a one-hot vector, P u ' is the prototype vector P of the invisible class u is the transpose of , λ, α, β, and γ are all weight parameters that are set.

[0034] Furthermore, in step 4, the semantic information features and the visual image features are input into the classifier network and jointly trained to obtain a classifier. The specific method is as follows:

[0035] (1) Initialization: First, initialize the parameters. K and L are the number of visible and invisible classes, respectively. Z u ∈R K×L is the class prototype cosine distance between the visible class and the invisible class in the semantic space; the semantic space class prototype C of the known invisible class u The cosine distance Z between the visible class and the invisible class in the semantic space u , through the formula The dictionary D2 in the semantic space can be obtained; the semantic space class prototype C of the known visible class s And the dictionary D2 in the semantic space obtained above, through the formula The representation Z of the visible class category in the alignment space can be obtained s ; The class prototype P of the visible class in the visual space s The initial value is set to the average value of all samples in each class; the class prototype P of the known visible class in the visual space s and the representation Z of the visible class in the alignment space s , through the formula The dictionary D1 in the visual space can be obtained; the dictionary D1 in the known visual space and the representation Z of the unseen class in the alignment space u , through the formula The class prototype P of the invisible class in the visual space can be obtained u ;

[0036] (2) Joint Optimization:

[0037] Fixed dictionary D1 in visual space and representation Z of visible class categories in alignment space s , by minimizing the formula The value of updates the class prototype P of the visible class in the visual space s, where β is the parameter set;

[0038] Fixed visible class prototype P in visual space s As well as the dictionary D1 in the visual space and the dictionary D2 in the semantic space, by minimizing the formula Update the representation Z of the visible class in the alignment space with the value of s , where λ is the parameter set;

[0039] Fixed visible and invisible class prototypes P in visual space s , P u And the representation Z of the visible class and invisible class in the alignment space s , Z u , by minimizing the formula Update the dictionary D1 in the visual space with the value of , where α is the parameter set;

[0040] Fixed visible and invisible class representations Z in the alignment space s , Z u And the semantic vectors C of known visible and invisible classes s , C u , by minimizing the formula The value of updates the dictionary D2 in the semantic space, where α is the set parameter;

[0041] Fix the class prototype P of the invisible class in the visual space u As well as the dictionary D1 in the visual space, the dictionary D2 in the semantic space, and the semantic vector C of the known unseen class u , by minimizing the formula Update the representation Z of the invisible class in the alignment space with the value of u , where λ is the parameter set;

[0042] Fix the dictionary D1 in the visual space and the representation Z of the unseen classes in the alignment space u , by minimizing the formula Update the class prototype P of the invisible class in the visual space u , where L u =DA, A represents the edge A between the above vertices i and j ij The vector composed of D = diag(D 1 ,…D u ) is the sum of each column of A.

[0043] Furthermore, in step 5, the test set tests the classification accuracy of the classifier as follows:

[0044] (1) In the visual feature space: the visual space prototype P of the known invisible classu , in order to solve the domain shift problem, modify the predicted visual space prototype P of the unseen class u , search for the prototype of the visual space class with the i-th invisible class The nearest K unseen class sample image feature vectors X t , by calculating And take this value as the class prototype P of the final invisible class in the visual space u ', directly calculate X during the prediction process i With all invisible class prototypes P u 'SimilaritySim v , i.e., cosine distance, takes the closest class prototype as the estimated value of the classification to complete the zero-sample image classification task;

[0045] (2) In the alignment space: The representation Z of the known unseen class in the alignment space u , by minimizing the formula It is possible to obtain the representation Z of the test class in the alignment space i , by calculating Z i With all invisible class prototypes Z u SimilaritySim v , i.e., cosine distance, takes the closest class prototype as the estimated value of the classification to complete the zero-sample image classification task, where D1 is the dictionary in the visual space, X i is the image feature vector of the test category, μ is a parameter not greater than 1;

[0046] (3) In the semantic space: The previous step obtains the representation Z of the test class in the alignment space i , through formula C i =D2Z i The semantic vector of the test class can be obtained, where D2 is a dictionary. By calculating the semantic vector C of the test class i and the semantic vectors C of all unseen classes u The similarity between s , that is, the cosine distance, takes the closest class prototype as the estimated value of the classification to complete the zero-sample image classification task.

[0047] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0048] The present invention improves the classification accuracy of zero-shot image classification through a zero-shot image classification method based on dictionary learning. At the same time, in order to achieve an ideal classification effect, a dictionary learning framework is introduced. The structural alignment method is extended from visible classes to invisible classes through a domain adaptation method, which effectively prevents the overfitting problem. At the same time, the geometric properties of the visual and semantic class prototypes are retained to achieve a better alignment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a zero-sample image classification method based on dictionary learning according to the present invention;

[0050] Figure 2 CDL algorithm model network structure diagram;

[0051] Figure 3 Class prototype constraint structure diagram. Specific implementation methods

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] The present invention improves the classification accuracy of zero-shot image classification through a zero-shot image classification method based on dictionary learning. At the same time, in order to achieve an ideal classification effect, a dictionary learning framework is introduced. The structural alignment method is extended from visible classes to invisible classes through a domain adaptation method, which effectively prevents the overfitting problem. At the same time, the geometric properties of the visual and semantic class prototypes are retained to achieve a better alignment effect.

[0054] See also Figure 1 As shown, the zero-sample image classification method based on dictionary learning of the present invention specifically includes:

[0055] Step 1: Get a common dataset for computer vision.

[0056] Obtain public image datasets for training the system; obtain common datasets for computer vision, namely AWA (animal dataset with attributes), FLO (flower dataset), CUB (fine-grained bird dataset), and SUN (large-scale scene dataset).

[0057] Step 2: preprocess the image data set to obtain a training set and a test set, wherein there is no intersection between the training set and the test set, and both the training set and the test set include image information and semantic information;

[0058] Step 3: Construct a coupled dictionary learning framework to achieve visual-semantic structure alignment, use domain adaptation methods to extend the structure alignment method from visible classes to invisible classes, and retain the geometric properties of visual and semantic class prototypes.

[0059] See also Figure 2 As shown in the figure, the first step is to learn the class prototypes in the semantic space and image feature space. The visible class image feature X obtained after data preprocessing in step 2 is s And the semantic vector C of the visible and invisible categories s ∈R m ×K / C u∈R m×L As the training set of the model. In the semantic space, the known semantic vector C s ∈R m×K / C u ∈R m×L As the prototype of the visible class and the invisible class; in the visual space, due to the visible class image feature X s And the class label corresponding to each image is a one-hot vector are all known, so by minimizing the loss function We can get the class prototype P of the visible class in the visual space s .

[0060] The second step is to structurally align the semantic space and the image feature space, mainly through a coupled dictionary learning framework to achieve visual-semantic structure alignment. We find a new representation in the semantic space and the visual feature space respectively. To represent each class, and to force 2 new representations The same in the intermediate subspace, that is, Since all other parameters are known, the dictionary D1 in the visual space and the dictionary D2 in the semantic space can be learned by minimizing the loss function. The loss function is as follows:

[0061]

[0062]

[0063] Where D1 and D2 are dictionaries in visual space and semantic space respectively. are the values ​​of the dictionaries D1 and D2 in the i-th column, C s is the semantic vector of the visible class, P s is the class prototype of the visible class in the visual space, and λ is a parameter greater than 0.

[0064] The third step is to alleviate the domain shift problem caused by using only the dictionary learned from the seen class prototypes in the structure alignment by using domain adaptation methods. We do this by using the class prototypes C in the semantic space of the unseen classes. u Automatically learn class prototypes P in the visual space of unseen classes u , that is, due to the parameters D1, D2, Z u , C u It is known that P can be obtained by minimizing the loss function u :

[0065]

[0066]

[0067] Where D1 and D2 are dictionaries in visual space and semantic space respectively. are the values ​​of the dictionaries D1 and D2 in the i-th column, Z u is the representation of the unseen class in the alignment space, C u is the semantic vector of the unseen class, and λ is a parameter greater than 0.

[0068] For the last step, see Figure 3 As shown, in order to preserve the geometric properties of the visual and semantic class prototypes (if the class prototype C is not visible in the semantic space i , C j are very close, then they are invisible to the prototype P in visual space i , P j The class prototype constraint is added to achieve better alignment when the class prototypes in the two spaces are projected into the intermediate space. The dictionary learning framework mainly introduces the graph Laplacian term to maintain the geometric characteristics of the invisible class prototypes in the visual feature space. For the invisible class, a fully connected graph G is constructed, with v vertices to represent our invisible categories. For any two categories i and j, we use the edge A between vertices i and j. ij To represent the relationship between two vertices, A ij Specifically expressed as:

[0069]

[0070] in They represent two different types of semantic vectors respectively, and δ is a parameter not greater than 1.

[0071] Then we use the loss function To adjust the invisible class prototype P learned last u , Represents the class prototype vectors of different categories of image feature space respectively. If the similarity of the semantic class prototypes of two classes is greater than the preset value, then A ij And the loss function L pc is less than the preset value; since A is the edge A between the above vertices i and j ij The vector composed of D = diag(D 1 ,…D u ) Each entry in the diagonal matrix is ​​the sum of each column of A. We let L u =DA, so the loss function L pc It can also be expressed as:

[0072]

[0073] in Respectively represent the prototype vectors of the image feature space class of two different types in the invisible class, A ij is the relationship between the two different types, L u =DA, tr represents the trace. The final objective function L is:

[0074]

[0075] Step 4: Input the picture image features and visual image features into the classifier network and train them together to obtain a classifier.

[0076] First, initialize the parameters, K and L are the number of visible and invisible classes respectively; Z u ∈R K×L is the class prototype cosine distance between the visible class and the invisible class in the semantic space; the semantic space class prototype C of the known invisible class u The cosine distance Z between the visible class and the invisible class in the semantic space u , through the formula The dictionary D2 in the semantic space can be obtained; the semantic space class prototype C of the known visible class s And the dictionary D2 in the semantic space obtained above, through the formula The representation Z of the visible class category in the alignment space can be obtained s ; The class prototype P of the visible class in the visual space s The initial value is set to the average value of all samples in each class; the class prototype P of the known visible class in the visual space s and the representation Z of the visible class in the alignment space s , through the formula The dictionary D1 in the visual space can be obtained; the dictionary D1 in the known visual space and the representation Z of the unseen class in the alignment space u , through the formula The class prototype P of the invisible class in the visual space can be obtained u .

[0077] Then they are jointly optimized, fixing the dictionary D1 in the visual space and the representation Z of the visible class categories in the alignment space s , by minimizing the formula The value of updates the class prototype P of the visible class in the visual space s , where β is the parameter set;

[0078] Fixed visible class prototype P in visual space s As well as the dictionary D1 in the visual space and the dictionary D2 in the semantic space, by minimizing the formula Update the representation Z of the visible class in the alignment space with the value of s, where λ is the parameter set;

[0079] Fixed visible and invisible class prototypes P in visual space s , P u And the representation Z of the visible class and invisible class in the alignment space s , Z u , by minimizing the formula Update the dictionary D1 in the visual space with the value of , where α is the parameter set;

[0080] Fixed visible and invisible class representations Z in the alignment space s , Z u And the semantic vectors C of known visible and invisible classes s , C u , by minimizing the formula The value of updates the dictionary D2 in the semantic space, where α is the set parameter;

[0081] Fix the class prototype P of the invisible class in the visual space u As well as the dictionary D1 in the visual space, the dictionary D2 in the semantic space, and the semantic vector C of the known unseen class u , by minimizing the formula Update the representation Z of the invisible class in the alignment space with the value of u , where λ is the parameter set;

[0082] Fix the dictionary D1 in the visual space and the representation Z of the unseen classes in the alignment space u , by minimizing the formula Update the class prototype P of the invisible class in the visual space u , where L u =DA, A represents the edge A between the above vertices i and j ij The vector composed of D = diag(D 1 ,…D u ) is the sum of each column of A.

[0083] Step 5: Input the test set into the classifier to classify the test set.

[0084] This part mainly uses the following three methods to obtain the final estimated category for classification:

[0085] In the visual feature space: the known invisible class visual space class prototype P u , in order to further solve the domain shift problem, the predicted visual space prototype of the unseen class P is modified u , search for the prototype of the visual space class with the i-th invisible class The nearest K unseen class sample image feature vectors X t , by calculating And take this value as the class prototype P of the final invisible class in the visual space u '. X is directly calculated during the prediction process i With all invisible class prototypes P u 'SimilaritySim v (cosine distance), and then use the closest class prototype as the estimated value of the classification to complete the zero-sample image classification task;

[0086] In the alignment space: The representation Z of the known unseen classes in the alignment space u , by minimizing the formula It is possible to obtain the representation Z of the test class in the alignment space i , by calculating Z i With all invisible class prototypes Z u SimilaritySim v (cosine distance), and then use the closest class prototype as the estimated value of the classification to complete the zero-sample image classification task. Where D1 is the dictionary in the visual space, X i is the image feature vector of the test category, μ is a parameter not greater than 1;

[0087] In the semantic space: The previous step obtained the representation Z of the test class in the alignment space i , through formula C i =D2Z i The semantic vector of the test class can be obtained, where D2 is the dictionary, and then the semantic vector C of the test class is calculated i and the semantic vectors C of all unseen classes u The similarity between s (cosine distance), and then use the closest class prototype as the estimated value of the classification to complete the zero-sample image classification task.

[0088] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A zero-shot image classification method based on dictionary learning, characterized in that: The method comprises the following steps: Step 1: Obtain a general image dataset for computer vision; Step 2: preprocessing the image data set to obtain a training set and a test set, wherein there is no intersection between the training set and the test set, and both the training set and the test set include image information and semantic information; Step 3: Construct a coupled dictionary learning framework to achieve visual-semantic structure alignment, use domain adaptation methods to extend the structure alignment method from visible classes to unseen classes, and preserve the geometric properties of visual and semantic class prototypes; Step 4: Input the semantic information features and visual image features into the classifier network and train them together to obtain a classifier; Step 5: Input the test set into the classifier to classify the test set; In step 3, the method for constructing a coupled dictionary learning framework is as follows: (1) The first step in building a coupled dictionary learning framework is to learn class prototypes in the semantic feature space and the image feature space; the visible class image features X obtained after data preprocessing are s And the semantic vector C of the visible and invisible categories s ∈R m×K / C u ∈R m×L As the training set of this learning framework, the known semantic vector C s ∈R m×K / C u ∈R m×L As the class prototype vector of the visible class and the invisible class; in the visual feature space, due to the visible class image feature X s And the class label corresponding to each image is a one-hot vector It is known that by minimizing the loss function Get the class prototype P of the visible class in the visual feature space s ; (2) The second step in constructing the coupled dictionary learning framework is to structurally align the semantic feature space and the image feature space, and to achieve visual-semantic structure alignment through a coupled dictionary learning framework; find a new representation in the semantic feature space and the visual feature space respectively. To represent each class, and to force 2 new representations The same in the aligned subspace, that is, are the class prototype representations in the visual feature space and the semantic feature space, respectively. s is the class prototype vector representation of the visible class in the aligned subspace; all other parameters are known, and the dictionary D1 in the visual feature space and the dictionary D2 in the semantic feature space are learned by minimizing the loss function. The loss function is as follows: Among them, D1 and D2 are dictionaries in the visual feature space and the semantic feature space respectively. are the values ​​of the dictionaries D1 and D2 in the i-th column, P s is the class prototype vector of the visible class in the visual feature space, and λ is a parameter greater than 0; (3) The third step in building the coupled dictionary learning framework is to use domain adaptation methods to alleviate the domain shift problem caused by using only the dictionary learned from the visible class prototypes in structure alignment: u Automatically learn the class prototype vector P in the visual feature space of unseen classes u , that is, due to the parameters D1, D2, Z u , C u All are known, and P is obtained by minimizing the loss function u : Among them, D1 and D2 are dictionaries in the visual feature space and the semantic feature space respectively. are the values ​​of the dictionaries D1 and D2 in the i-th column, Z u is the representation of the unseen class in the alignment space, and λ is a parameter greater than 0; (4) By adding class prototype constraints, the class prototypes in the two spaces can be better aligned when projected into the intermediate space; the dictionary learning framework introduces the graph Laplacian term to maintain the geometric characteristics of the invisible class prototypes in the visual feature space, and constructs a fully connected graph G for the invisible class, using v vertices to represent the invisible category. For any two invisible categories i and j, the edge A between vertices i and j is used. ij To represent the relationship between two vertices, A ij Specifically expressed as: in, Represent two different types of semantic vectors of categories i and j respectively, and δ is a parameter not greater than 1; Through the loss function Adjust the invisible class prototype P learned last u , They represent the class prototype vectors of the image feature space of different categories, u represents the number of invisible classes, and if the similarity of the semantic prototypes of two classes is greater than the preset value, then A ij And the loss function L pc The values ​​are all less than the preset value; A is the edge A between the above vertices i and j ij The vector composed of D = diag(D 1 ,…D u1 ) Each entry in the diagonal matrix is ​​the sum of each column of A. Let L u =DA, loss function L pc It is expressed as: in, Respectively represent the prototype vectors of the image feature space class of two different types in the invisible class, A ij is the relationship between the two different types, L u =DA, tr represents the trace; the final objective function L1 is: Among them, P s is the prototype of the visible image feature class, The class label corresponding to each image is a one-hot vector, P u 'It's P u The transpose of , λ, α, β, γ are all weight parameters set; In step 4, the semantic information features and visual image features are input into the classifier network and trained together to obtain a classifier. The specific method is as follows: (1) Initialization: First, initialize the parameters. K and L are the number of visible and invisible classes, respectively, and L = u. u ∈R K×L As the class prototype cosine distance between the visible class and the invisible class in the semantic feature space; it is known that C u and Z u , through the formula The dictionary D2 in the semantic feature space can be obtained; given C s And the dictionary D2 in the semantic feature space obtained above, through the formula Then we can get Z s ; The class prototype P of the visible class in the visual feature space s The initial value is set to the average value of all samples in each class; the class prototype P of the known visible class in the visual feature space is s and the representation Z of the visible class in the alignment space s , through the formula That is, the dictionary D1 in the visual feature space is obtained; the dictionary D1 in the known visual feature space and the representation Z of the invisible class in the alignment space u , through the formula The class prototype P of the invisible class in the visual feature space can be obtained u ; (2) Joint Optimization: Fixed dictionary D1 in visual feature space and representation Z of visible class categories in alignment space s , by minimizing the formula The value of updates the class prototype P of the visible class in the visual feature space s ; Fixed visible class prototype P in visual feature space s As well as the dictionary D1 in the visual feature space and the dictionary D2 in the semantic feature space, by minimizing the formula Update the representation Z of the visible class in the alignment space with the value of s ; Fixed visible and invisible class prototypes P in the visual feature space s , P u and Z s , Z u , by minimizing the formula Update the dictionary D1 in the visual feature space with the value of Fixed Z s , Z u And the known C s , C u , by minimizing the formula Update the dictionary D2 in the semantic feature space with the value of Fix the class prototype P of the invisible class in the visual feature space u As well as the dictionary D1 in the visual feature space, the dictionary D2 in the semantic feature space, and the known C u , by minimizing the formula Update the value of Z u ; Fixed dictionary D1 and Z in visual feature space u , by minimizing the formula The value of updates the class prototype P of the unseen class in the visual feature space u , where L u =DA, A represents the edge A between the above vertices i and j ij The vector composed of D = diag(D 1 ,…D u1 ) is the sum of each column of A, D 1 -D u1 Respectively represent the vectors consisting of Aij in columns 1 to u1.

2. The zero-shot image classification method based on dictionary learning according to claim 1, characterized in that: In step 1, the general image dataset of computer vision is any one of the AWA, CUB, and SUN image datasets, where AWA is an animal dataset with attributes, CUB is a fine-grained bird dataset, and SUN is a large-scale scene dataset.

3. The zero-shot image classification method based on dictionary learning according to claim 1, characterized in that: In step 2, the image dataset preprocessing method is as follows: ResNet101 is used to preprocess the image, and the dataset image in step 1 is sent to the ResNet101 network to output a 2048-dimensional image feature vector, and all categories are divided into training dataset image feature representation and test dataset image feature representation. The semantic dataset of the category uses attribute description information.

4. The zero-shot image classification method based on dictionary learning according to claim 1, characterized in that: In step 5, the test set tests the classification accuracy of the classifier as follows: (1) In the visual feature space: the visual feature space prototype P of the known invisible class u , modify the predicted unseen class visual feature space class prototype P u , search for the prototype of the visual feature space of the i-th invisible class The nearest K1 unseen class sample image feature vectors X t , by calculating And this value is used as the final prototype P of the invisible class visual feature space u ', directly calculate X during the prediction process i With all the unseen visual feature space class prototypes P u 'The cosine distance of the closest class prototype is used as the estimated value of the classification to complete the zero-sample image classification task; (2) In the alignment space: given Z u , by minimizing the formula It is possible to obtain the representation Z of the test class in the alignment space i , by calculating Z i With Z u The cosine distance of the closest class prototype is used as the estimated value of the classification to complete the zero-sample image classification task, where D1 is the dictionary in the visual feature space, X i is the image feature vector of the test category, μ is a parameter not greater than 1; (3) In the semantic feature space: The previous step obtains the representation Z of the test class in the alignment space i , through formula C i =D2Z i The semantic vector of the test class can be obtained, where D2 is a dictionary. By calculating the semantic vector C of the test class i and all C u The cosine distance between them is used to take the closest class prototype as the estimated value of the classification to complete the zero-sample image classification task.

Images