Zero sample image recognition model and training method, image recognition method, program product, medium and equipment

Abstract

The embodiment of the invention provides a zero-sample image recognition model, a training method, an image recognition method, a program product, a medium and equipment. According to the embodiment of the invention, an indirect domain adaptation method is adopted, generated features and real features of an image are mapped to an embedding space together, distribution registration between the generated features and the real features is completed, and a classifier is trained by using the registered distribution. By adopting the registered projection network, the technical problem that a zero sample identification method is inaccurate is solved, and the technical effect of more accurately identifying unseen images is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a zero-sample image recognition model and training method, an image recognition method, a program product, a medium and a device. Background Art

[0002] Traditional image classification collects a large number of image samples, and then trains a classifier containing all known categories based on the collected samples to predict the image category. During the prediction process, the classifier can usually only classify known categories and is almost helpless for samples of unknown categories.

[0003] The zero-shot recognition method aims to solve the problem of missing visual samples of the target category in real-world scenarios. In the zero-shot recognition method, the categories that have appeared in the training samples are called visible classes, and the sample categories in the test samples have never appeared in the training set, which are called unseen classes. Zero-shot recognition requires the model to predict the category to which the unseen class samples belong by learning from the visible class data. Due to the lack of visual information of the unseen class, zero-shot recognition will learn the knowledge of the unseen class with the help of cross-category semantic information. Therefore, the training data set required for zero-shot recognition contains three types of information: visual, semantic, and label. In the training phase, all information of the visible class and the semantic information of the unseen class are available information; in the testing phase, the visual samples of the unseen class are the test objects.

[0004] The generative method of zero-shot recognition is a method of establishing a semantic-to-visual mapping based on a generative model. The generative method obtains visual samples of unseen classes through a generative model, converting zero-shot learning into a normal classification problem without missing samples.

[0005] The generative method first learns the semantic-to-visual mapping information based on the data of the visible class, and then transfers this mapping relationship to the unseen class, and generates visual samples or features of the unseen class with the help of the known semantic information of the unseen class. Through this generative method, the problem of zero-shot image recognition can be converted into a conventional image classification problem.

[0006] The classifier trained by this generation method can not only classify images of known categories, but also classify images of unknown categories.

[0007] In addition, in order to avoid domain bias in the knowledge learned on the visible class, many zero-shot recognition methods also use unlabeled visual samples of unseen classes to correct the generative model. This allows the trained generative model to better generate image samples or features that are close to the real world, thereby improving the recognition accuracy of unseen classes. This correction method is called the direct zero-shot recognition method.

[0008] However, this direct domain adaptation zero-shot recognition method has drawbacks. Using unlabeled samples of unseen classes to directly adjust the generative model trained on the seen classes may destroy the mapping relationship from semantic information to visual information originally established by the generative model, thereby reducing the generation quality. Therefore, a more accurate zero-shot recognition method is needed. Summary of the invention

[0009] In view of the deficiencies in the prior art, the present invention provides a zero-sample image recognition model and training method, image recognition method, program product, medium and device, which solve the technical problem that the zero-sample recognition method in the prior art is not accurate enough.

[0010] In a first aspect, according to an embodiment of the present invention, a training method for a zero-shot image recognition model is provided, comprising the following steps: S1, initializing a model, wherein the model comprises a feature generation network, a projection network, a domain classifier and an image classifier; S2, constructing a training data set, wherein the training data set comprises visible class images and their semantics, category labels and domain labels, as well as unseen class images and their semantics and domain labels, wherein the domain label indicates that they are visible class images or unseen class images; S3, training the model based on the constructed training data set, and updating the parameters of the model based on a preset loss function, wherein the domain classification loss is used to update the parameters of the domain classifier, the image classification loss is used to update the parameters of the image classifier, the combination of the domain classification loss and the image classification loss is used to update the parameters of the projection network, and the combination of the feature generation loss and the projection loss used by the projection network is used as the total loss to update the parameters of the generator.

[0011] According to an embodiment of the present invention, the feature generation network includes a generator and a discriminator, wherein the generator generates features based on semantics, and the discriminator determines whether the input features are generated features or real features, and the real features are features extracted from the image through a convolutional neural network.

[0012] According to an embodiment of the present invention, noise is added when generating features.

[0013] According to an embodiment of the present invention, the feature generation loss is:

[0014]

[0015] in,

[0016]

[0017] α~U(0,1),

[0018] Where D(*) indicates the discriminator, E[*] indicates the expectation, λ is the weight, indicates the gradient, ‖*‖2 indicates L2 regularization, x indicates the true feature, Indicates the creation of features.

[0019] According to an embodiment of the present invention, the domain classification loss is a cross entropy loss.

[0020] According to an embodiment of the present invention, the cross entropy loss is:

[0021]

[0022] Among them, l represents the actual domain, Represents the prediction domain.

[0023] According to an embodiment of the present invention, the image classification loss is a cross entropy loss.

[0024] According to an embodiment of the present invention, the cross entropy loss is:

[0025]

[0026] Among them, y represents the actual category, Represents the predicted category.

[0027] According to an embodiment of the present invention, when updating parameters of the projection network, the domain classification loss in the projection loss is gradient inverted.

[0028] According to an embodiment of the present invention, the domain classification loss after gradient inversion is further multiplied by a scaling coefficient.

[0029] According to an embodiment of the present invention, the projection loss is:

[0030]

[0031] in, is the image classification loss, is the domain classification loss, and β is the proportionality coefficient.

[0032] According to an embodiment of the present invention, when updating parameters of the feature generation network, the projection loss in the total loss is gradient inverted.

[0033] According to an embodiment of the present invention, the projection loss after gradient inversion is further multiplied by a proportionality coefficient.

[0034] According to an embodiment of the present invention, the total loss is:

[0035]

[0036] in, indicates the feature generation loss, is the image classification loss, is the domain classification loss, and β and γ are proportional coefficients.

[0037] In a second aspect, according to another embodiment of the present invention, a zero-shot image recognition model is provided, wherein the model is configured to be trained using the method described in the first aspect, and comprises: a feature generation network, used to generate features based on semantics; a projection network, used to project the generated features into embedded features; a domain classifier, used to perform domain classification on the embedded features; and an image classifier, used to perform image classification on the embedded features.

[0038] In a third aspect, according to another embodiment of the present invention, there is provided an image recognition method, the method comprising: P1, obtaining an image to be recognized and its semantics; and P2, using a zero-sample image recognition model as described in the second aspect to classify the image to be recognized.

[0039] In a fourth aspect, according to another embodiment of the present invention, a computer program product is provided, wherein the computer program product includes a computer program code, and when the computer program code runs on an electronic device, the electronic device executes the method as described in the first aspect or the third aspect.

[0040] In the fifth aspect, according to another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program can be executed by a processor to implement the steps of the method described in the first aspect or the third aspect.

[0041] In the sixth aspect, according to another embodiment of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect or the third aspect by executing the executable instructions.

[0042] The technical principle of the present invention is: an indirect domain adaptation method is adopted to map the generated features and real features of the image together into the embedding space, complete the distribution alignment between the generated features and the real features, and use the aligned distribution to train the classifier.

[0043] Compared with the prior art, the present invention has the following beneficial effects: by adopting a registered projection network, it solves the technical problem of inaccuracy of the zero-sample recognition method, and produces a technical effect of more accurately recognizing unseen images. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The figure is a flowchart of a training method for a zero-sample image recognition model according to an embodiment of the present invention.

[0045] Figure 2Schematic diagram of a training method for a zero-sample image recognition model according to an embodiment of the present invention.

[0046] Figure 3 A schematic diagram of a feature generation network of a zero-sample image recognition model according to an embodiment of the present invention.

[0047] Figure 4 This is a schematic diagram of the image classification of the zero-sample image recognition model according to an embodiment of the present invention.

[0048] Figure 5 Schematic diagram of the indirect domain adaptation of the zero-shot image recognition model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0050] As described in the background technology, there is a technical problem in the prior art that the zero-sample recognition method is not accurate enough. Figure 1 (Training Flowchart) and Figure 2 As shown in the (model training principle diagram), the embodiment of the present invention proposes a training method for a zero-sample image recognition model, comprising the following steps: S1, initializing the model, the model comprising a feature generation network G (including a discriminator D), a projection network M, a domain classifier P and an image classifier C; S2, constructing a training data set, the training data set comprising visible class images and their semantics, category labels and domain labels, as well as unseen class images and their semantics and domain labels, wherein the domain label indicates that it is a visible class image or an unseen class image; S3, training the model based on the constructed training data set, and updating the parameters of the model based on a preset loss function, wherein the domain classification loss is used to update the parameters of the domain classifier, the image classification loss is used to update the parameters of the image classifier, the combination of the domain classification loss and the image classification loss is used to update the parameters of the projection network, and the combination of the feature generation loss and the projection loss used by the projection network is used as the total loss to update the parameters of the generator. By adopting an indirect domain adaptation method, the generated features and the real features of the image are jointly mapped to the embedding space, and the distribution alignment between the generated features and the real features is completed. The classifier is trained using the registered distribution so that it can have higher accuracy when identifying unknown categories.

[0051] According to an embodiment of the present invention, referring to Figure 2 and Figure 3, the feature generation network includes a generator and a discriminator, wherein the generator generates features based on semantics, and the discriminator determines whether the input features are generated features or real features, and the real features are features extracted from the image through the convolutional neural network. In addition, in order to increase the generalization ability of the model, noise is added when generating features.

[0052] As we all know, taking C-GAN as an example, the generator G(z,a;θ g ) uses noise and semantics as input and outputs the generated features corresponding to the semantics, that is, Z×A→X (“×” represents concatenation). Discriminator D(x,a;θ d ) Input the generated features and real features and related semantics, and train its ability to identify the authenticity of the corresponding category features, that is, to achieve X×A→(min,max). And on the basis of the discriminator's authenticity, train the generator's ability to generate real features. The training process of C-GAN can be regarded as the following maximum and minimum optimization problem:

[0053]

[0054] Where D(*) indicates the discriminator, E[*] indicates the expectation, and x indicates the true feature. indicates the generated features, and a indicates the semantics.

[0055] The discriminator uses the maximum and minimum scores as loss according to the authenticity of the input sample, while the generator uses the minimum negative score as loss. While iteratively enhancing the generator's generation ability and the discriminator's discrimination ability, it gradually achieves the approach of the generated features to the real distribution. However, although the Generative Adversarial Network (GAN) has been proven to fit complex data distributions, it is difficult to train due to its complex adversarial process.

[0056] In order to improve the problems of GAN, the embodiment of the present invention adopts the more advanced WGAN-GP (WassersteinGAN with Gradient Penalty) generative adversarial optimization model technology. WGAN-GP adds a gradient penalty loss term to the discriminator network D, and the new generative network loss is:

[0057]

[0058] in, λ is a positive penalty weight, indicates the gradient, and ‖*‖2 indicates L2 regularization. The gradient penalty term of the discriminator ensures that when the discriminator G(z,a) is aligned with the true distribution, D(G(z,a),a) will not exceed D(x,a), avoiding the collapse of the optimization process. In addition, WGAN-GP does not use logarithm as loss. For the positive scores of true samples and the negative scores of false samples, the discriminator can be directly optimized in the direction of maximization. C-WGAN-GP is also a maximum-minimum optimization problem:

[0059]

[0060] Therefore, C-WGAN-GP achieves the gradual fitting of the distribution of the generated network to the real distribution through the maximum and minimum game. Finally, the conditional generator G realizes the construction of semantic to visual feature mapping: A→X

[0061] Since H is obtained by real feature projection, it has relevant classification knowledge. (Generated features) do not have any labels. To achieve classification of unseen classes in the embedding space, it can only be done with the help of H's classification knowledge transfer.

[0062] refer to Figure 4 In order to obtain the classification information on H, we first train the classifier on the H distribution. i Send it to the classification network C to get the predicted label y i By training with a calibrated embedding feature set H, the classification network C can learn from H to the label y i According to an embodiment of the present invention, the domain classification loss is a cross entropy loss. The cross entropy loss is:

[0063]

[0064] Among them, y represents the actual category, Represents the predicted category.

[0065] In order to transfer classification knowledge, the domain adaptation task of the embedding space must be completed and the distribution of H. To achieve adaptation by coinciding with the distribution of H, we first need to obtain The difference information between the distribution and H will be and h are sent to the domain classification network P to obtain the predicted domain label And align to their corresponding domain labels respectively, so that P can capture domain difference information. According to an embodiment of the present invention, the image classification loss is a cross entropy loss. The cross entropy loss is:

[0066]

[0067] Among them, l represents the actual domain, Represents the prediction domain.

[0068] Found through the domain classification network P After the domain difference of and H, it is necessary to and H are optimized towards each other's distribution. The distributions of P and H are obtained by mapping the feature vector X in the feature space through the projection network M. The domain difference information stored in P is transmitted back to M, so that M is optimized in the direction of reducing the domain difference. The projection network M can learn to convert the domain difference in the original feature space into (unseen class generating features) and X u The distribution of (unseen class true features) is projected into the same distribution under the embedding space.

[0069] For the projection network M, the projection task is to achieve the embedding space The distribution of M must overlap with that of H, and the classification information on the H domain must not be lost. Therefore, M is optimized by the cross entropy loss of the classification network in the forward feedback and the cross entropy loss of the domain classification network in the reverse feedback. In order to achieve the domain classification loss Backward pass back to the projection network M, add a gradient transposition (inversion) layer between the output layer of the projection network M and the input layer of the domain classification network P. The specific role of the gradient transposition is: the embedded feature h output by the projection network M is passed to the domain classification network P and the classification network C without bias, but the domain classification loss When the gradient is passed back, the gradient needs to be inverted and then passed to the projection network M. In order to shield the early noise of the domain discrimination network, it is also necessary to multiply it by a weight coefficient β. The projection loss is:

[0070]

[0071] in, is the image classification loss, is the domain classification loss, and β is the proportionality coefficient.

[0072] refer to Figure 5 , after realizing the indirect domain adaptation of the embedding space, it is necessary to pass the overall loss of domain adaptation back to the generator, so that when the generator generates samples, the samples can better complete the indirect domain adaptation task. Similarly, according to an embodiment of the present invention, when updating the parameters of the feature generation network, the projection loss in the total loss is gradient inverted. Also taking into account the noise, according to an embodiment of the present invention, the projection loss after gradient inversion is also multiplied by a proportional coefficient. According to an embodiment of the present invention, the total loss is:

[0073]

[0074] in, indicates the feature generation loss, is the image classification loss, is the domain classification loss, and β and γ are proportional coefficients.

[0075] According to another embodiment of the present invention, a zero-shot image recognition model is provided. The model is configured to be trained using the training method as described above, and includes: a feature generation network for generating features based on semantics; a projection network for projecting the generated features into embedded features; a domain classifier for performing domain classification on the embedded features; and an image classifier for performing image classification on the embedded features.

[0076] According to another embodiment of the present invention, an image recognition method is provided, the method comprising: P1, obtaining an image to be recognized and its semantics; and P2, using a zero-sample image recognition model as described above to classify the image to be recognized.

[0077] According to another embodiment of the present invention, a computer program product is provided. The computer program product includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes the training method or the image recognition method.

[0078] According to another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. The computer program can be executed by a processor to implement the steps of the training method or the image recognition method.

[0079] According to another embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the training method or the image recognition method by executing the executable instructions.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A training method for a zero-shot image recognition model, characterized in that: The following steps are involved: S1. Initialize the model, wherein the model includes a feature generation network, a projection network, a domain classifier, and an image classifier; S2. constructing a training data set, wherein the training data set includes visible class images and their semantics, category labels and domain labels, and unseen class images and their semantics and domain labels, wherein the domain label indicates whether it is a visible class image or an unseen class image; S3. Training the model based on the constructed training data set, and updating the parameters of the model based on a preset loss function, wherein domain classification loss is used to update the parameters of the domain classifier, image classification loss is used to update the parameters of the image classifier, a combination of domain classification loss and image classification loss is used to update the parameters of the projection network, and a combination of feature generation loss and projection loss used by the projection network is used as the total loss to update the parameters of the generator.

2. The training method of a zero-sample image recognition model according to claim 1, characterized in that: The feature generation network includes a generator and a discriminator, wherein the generator generates features based on semantics, and the discriminator determines whether the input features are generated features or real features, and the real features are features extracted from the image through a convolutional neural network.

3. The training method of a zero-sample image recognition model according to claim 1, characterized in that: Add noise when generating features.

4. The training method of a zero-sample image recognition model according to claim 1, characterized in that: The feature generation loss is: in, α～U(0,1)， Where D(*) indicates the discriminator, E[*] indicates the expectation, λ is the weight, indicates the gradient, ‖*‖2 indicates L2 regularization, x indicates the true feature, Indicates the creation of features.

5. The training method of a zero-sample image recognition model according to claim 1, characterized in that: The domain classification loss is the cross entropy loss.

6. A training method for a zero-sample image recognition model as claimed in claim 5, characterized in that: The cross entropy loss is: Among them, l represents the actual domain, Represents the prediction domain.

7. The training method of a zero-sample image recognition model according to claim 1, characterized in that: The image classification loss is the cross entropy loss.

8. The method for training a zero-sample image recognition model according to claim 7, wherein: The cross entropy loss is: Among them, y represents the actual category, Represents the predicted category.

9. The method for training a zero-sample image recognition model according to claim 1, wherein: When updating the parameters of the projection network, the domain classification loss in the projection loss is gradient inverted.

10. The method for training a zero-sample image recognition model according to claim 9, wherein: The domain classification loss after gradient inversion is also multiplied by the scaling factor.

11. A training method for a zero-sample image recognition model as claimed in claim 10, characterized in that: The projection loss is: in, is the image classification loss, is the domain classification loss, and β is the proportionality coefficient.

12. A method for training a zero-sample image recognition model as claimed in claim 11, characterized in that: When updating the parameters of the feature generation network, the projection loss in the total loss is gradient inverted.

13. A training method for a zero-sample image recognition model as claimed in claim 12, characterized in that: The projection loss after gradient inversion is also multiplied by the scaling factor.

14. A method for training a zero-sample image recognition model as claimed in claim 13, characterized in that: The total loss is: in, indicates the feature generation loss, is the image classification loss, is the domain classification loss, and β and γ are proportional coefficients.

15. A zero-shot image recognition model, characterized in that: The model is configured to be trained using the method according to any one of claims 1 to 14, and comprises: Feature generation network, used to generate features based on semantics; Projection network, used to project generated features into embedded features; Domain classifier, which performs domain classification on embedded features; Image classifier, classifies images based on embedded features.

16. An image recognition method, characterized in that: The method comprises: P1. Obtain the image to be recognized and its semantics; and P2. Use a zero-sample image recognition model as described in claim 15 to classify the image to be recognized.

17. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device executes the method as claimed in any one of claims 1-14 and 16.

18. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1-14 and 16.

19. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-14 and 16 by executing the executable instructions.

Images