Image recognition training method and device, computer device and storage medium

Abstract

The application discloses an image recognition training method and device, computer equipment and a storage medium, wherein the image recognition training method comprises the following steps: constructing corresponding extended attributes based on entity attributes of entity types; performing semantic consistency synthesis on known class visual features of the entity types based on the entity attributes and the extended attributes of the entity types, and acquiring known class synthesized features corresponding to the entity attributes and the extended attributes; taking the known class visual features, the known class synthesized features, the entity attributes and the extended attributes as input data of a to-be-trained classification model, taking class prototypes corresponding to the known class visual features and entity types as output data of the to-be-trained classification model, training the to-be-trained classification model, and finally generating a trained classification model. The method can effectively maintain the consistency between the entity type visual features and the semantic features, guarantee the training stability, and improve the accuracy of zero sample recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification, and in particular to an image recognition training method and device, a computer device and a storage medium. BACKGROUND

[0002] Traditional image classification can only classify samples of classes that have appeared in the training set, and cannot classify samples of classes that have not appeared in the training set. However, in real applications, new classes are constantly emerging, and therefore a large amount of new class labeled data needs to be collected and the classifier needs to be retrained, which is time-consuming and laborious, and in some fields it is even impossible to obtain a sufficient number of new class labeled data. In view of this problem, zero-shot learning (ZSL) has emerged.

[0003] The purpose of zero-shot learning is to train a model using only the visual features and semantic representations of visible class samples, and then use the semantic representations of unseen classes as a bridge to enable the model to have the ability to recognize unseen class samples. In the zero-shot learning task, the main task is image classification. Zero-shot image classification can use existing image samples to label newly generated image samples, avoiding manual annotation and reducing the consumption of manpower and resources. Zero-shot learning essentially uses the idea of transfer learning to solve the problem. That is, learn an effective model from one domain / task / distribution, and then migrate to a new domain, task or distribution. Through zero-shot learning, "knowledge" migration from visible classes to unseen classes can be achieved, and these transferable knowledge is obtained from the semantic relevance of visible classes and unseen classes.

[0004] However, the semantic definition and relevance information of the user for the unseen class is not complete and lacks distinguishability. Most current methods for recognizing unseen types directly use user-defined semantic information as a constraint for generating unknown models, which makes the visual features synthesized by the model lack diversity and distinguishability. SUMMARY

[0005] The embodiments of the present application provide an image recognition training method and device, a computer device and a storage medium to solve the problem of lack of diversity and distinguishability in visual features of the synthesized model for recognizing unknown types.

[0006] An image recognition training method comprises:

[0007] Based on the entity attribute of the entity type, an extended attribute corresponding to the entity type is constructed.

[0008] The known class visual features of the entity type are semantically consistent synthesized based on the entity attribute and the extended attribute of the entity type, and known class synthesized features corresponding to the entity attribute and the extended attribute are obtained.

[0009] The known class visual features, the known class synthesized features, the entity attribute, and the extended attribute are taken as input data of a to-be-trained classification model, and the class prototype corresponding to the known class visual features and the entity type are taken as output data of the to-be-trained classification model, the to-be-trained classification model is trained, and a trained classification model is finally generated.

[0010] Preferably, based on the entity attribute of the entity type, the corresponding extended attribute is constructed, including:

[0011] The known class visual features are sequentially decomposed by the MSE loss and the CE loss to obtain visual distinguishing features;

[0012] The class prototype of the known class visual features is extracted by the triplet loss.

[0013] The entity attribute corresponding to the known class visual features is extracted, the entity attribute is trained by the MSE loss to align the entity attribute with the visual distinguishing features and the class prototype respectively, and the training result is taken as the extended attribute corresponding to the entity type.

[0014] Preferably, the to-be-trained classification model includes an encoder and a generator.

[0015] The known class visual features of the entity type are semantically consistent synthesized based on the entity attribute and the extended attribute of the entity type, and known class synthesized features corresponding to the entity attribute and the extended attribute are obtained, including:

[0016] The entity attribute, the extended attribute, and the known class visual features of the entity type are input into the encoder to obtain a noise distribution parameter.

[0017] The noise distribution parameter, the entity attribute, and the extended attribute of the entity type are input into the generator to obtain the known class synthesized features of the entity type.

[0018] Preferably, the method further includes:

[0019] The known class visual features and the known class synthesized features are compared by using the binary cross-entropy loss.

[0020] Preferably, the to-be-trained classification model includes a generator, an entity attribute decoder, and an extended attribute decoder.

[0021] After obtaining the known class synthesized features corresponding to the entity attribute and the extended attribute, the method further includes:

[0022] Input the known class visual feature and the entity attribute into the entity attribute decoder to obtain a real reconstruction attribute;

[0023] Input the known class visual feature and the extended attribute into the reconstruction attribute decoder to obtain an extended reconstruction attribute;

[0024] Adopt a cycle consistency loss to train the real reconstruction attribute and the extended reconstruction attribute, and use the real reconstruction attribute and the extended reconstruction attribute to approximate the known class visual feature.

[0025] Preferably, the known class visual feature, the known class synthetic feature, the entity attribute, and the extended attribute are used as input data of a to-be-trained classification model, the class prototype and the entity type corresponding to the known class visual feature are used as output data of the to-be-trained classification model, the to-be-trained classification model is trained, and a trained classification model is finally generated, including:

[0026] Adopt a WGAN network as the to-be-trained classification model, use the known class visual feature, the known class synthetic feature, the entity attribute, and the extended attribute as input data of the to-be-trained classification model, train the WGAN network through an optimization loss, and finally generate the trained classification model.

[0027] Preferably, after the trained classification model is finally generated, the method further includes:

[0028] Obtain an unseen class visual feature, input the unseen class visual feature into the trained classification model, and use the trained classification model to identify an identification type corresponding to the unseen class visual feature.

[0029] An image recognition training device, including:

[0030] An extended attribute module is constructed, which is used to construct corresponding extended attributes based on entity attributes of an entity type;

[0031] An obtained synthetic feature module is used to perform semantic consistency synthesis on a known class visual feature of an entity type based on entity attributes and extended attributes of the entity type, and obtain known class synthetic features corresponding to the entity attributes and the extended attributes, respectively;

[0032] A generated classification module is used to use the known class visual feature, the known class synthetic feature, the entity attribute, and the extended attribute as input data of a to-be-trained classification model, use a class prototype and an entity type corresponding to the known class visual feature as output data of the to-be-trained classification model, train the to-be-trained classification model, and finally generate a trained classification model.

[0033] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned image recognition training method when executing the computer program.

[0034] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the image recognition training method.

[0035] The image recognition training method, device, computer device and storage medium can effectively maintain consistency between the entity type visual feature and the semantic feature by adding a limitation on semantic consistency during training of the trained classification model, thereby improving the accuracy of zero-shot recognition while ensuring training stability. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0037] Figure 1 is a schematic diagram of an application environment of the image recognition training method in an embodiment of the present application;

[0038] Figure 2 is a flowchart of the image recognition training method in an embodiment of the present application;

[0039] Figure 3 is a first flowchart of the image recognition training method in another embodiment of the present application;

[0040] Figure 4 is a flowchart of the image recognition training method in an embodiment of the present application;

[0041] Figure 5 is a second flowchart of the image recognition training method in another embodiment of the present application;

[0042] Figure 6 is a schematic diagram of the whole flow implementation of the image recognition training method in an embodiment of the present application;

[0043] Figure 7 is a third flowchart of the image recognition training method in another embodiment of the present application;

[0044] Figure 8 is a fourth flowchart of the image recognition training method in another embodiment of the present application;

[0045] Figure 9 is a schematic diagram of the image recognition training device in an embodiment of the present application;

[0046] Figure 10 is a schematic diagram of the computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of the present application.

[0048] The image recognition training method provided by the embodiments of the present application can be applied in an application environment such as Figure 1 The image recognition training method is applied in an image recognition training system, and the image recognition training system includes a client and a server, wherein the client communicates with the server through a network. The client, also known as the user end, is a program that provides local services for the client corresponding to the server. The client can be installed on, but is not limited to, various computer devices such as personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0049] The scheme provided by the embodiments relates to the content of classifying visual samples of unseen classes through zero-shot learning. Early methods of zero-shot learning infer the image labels of unseen classes by using attributes. Attributes are characteristics shared by classes, and each class can be associated by using attributes, thereby realizing knowledge transfer. For example, define attributes “four legs”, “black”, “white”, “stripes”, and “grass-eating”. The identification model of these attributes can be trained by using source classes, and for a new class such as “zebra”, even if there is no any labeled image, as long as the attribute identification model identifies that an image has the above attributes, it can be judged that the image belongs to the “zebra” class. However, a kind of attribute can correspond to a large number of visual samples, and limited attribute information limits the performance of zero-shot learning.

[0050] In recent years, with the rapid development of GAN (Generative Adversarial Networks), some variants of GAN have been able to generate sufficiently real image samples, which also provides a new idea for solving the zero-shot image classification task, that is, using the visual features of known classes and some semantic information of invisible classes to generate visual features of unknown classes.

[0051] In an embodiment, as shown in Figure 2 An image recognition training method is provided, and the server in Figure 1 is taken as an example for illustration, and the method specifically includes the following steps:

[0052] S10. Construct the corresponding extended attribute based on the entity attribute of the entity type.

[0053] The entity attribute is an indispensable property of the entity itself, such as, for a horse, the entity attribute includes a furry skin, hooves, a tail, etc.

[0054] The extended attribute is an attribute obtained by processing and extending based on the entity attribute, such as, according to the attribute of a certain entity having four hooves, it can be deduced that the entity also has four legs, etc.

[0055] Specifically, in order to expand and improve the learning ability based on the zero sample model, the embodiments provided by the present application should as far as possible deduce a plurality of extended attributes related to the entity attribute, in order to improve the diversity of subsequent model determination.

[0056] S20. Based on the entity attribute and the extended attribute of the entity type, the known class visual features of the entity type are semantically consistent, and the known class synthesized features corresponding to the entity attribute and the extended attribute are obtained.

[0057] The entity type is the known type of entity, such as, monkey, chicken and dog, etc.

[0058] The semantic consistency synthesis is a method of consistent constraint on the known class visual features of the entity type through the description of the attributes corresponding to the entity type, which maintains the consistency of the attributes of the known class visual features in the model training process, and improves the accuracy of the final positioning class prototype of the model.

[0059] The known class synthesized features are a feature set with more comprehensive semantic description obtained by limiting and training processing of the known class visual features of the entity type based on the entity attribute and the extended attribute of the entity type.

[0060] S30. The known class visual features, the known class synthesized features, the entity attribute and the extended attribute are used as input data of the to-be-trained classification model, and the class prototype corresponding to the known class visual features and the entity type are used as output data of the to-be-trained classification model. The to-be-trained classification model is trained, and a trained classification model is finally generated.

[0061] In this embodiment, the WGAN (Wasserstein-Generative Adversarial Networks) network can be used as the to-be-trained classification model, so that the training process is more stable, so that the pseudo sample features output by the trained classification model are more distinguishable and accurate.

[0062] Specifically, the embodiment takes the known class visual features, the known class synthetic features, the entity attributes and the extended attributes as the input data of the WGAN network, takes the class prototypes corresponding to the known class visual features and the entity types as the output data of the to-be-trained classification model, trains the to-be-trained classification model, and finally generates the trained WGAN network.

[0063] WGAN mainly improves GAN from the perspective of loss function. The WGAN after the improvement of the loss function can obtain good performance results even on the full connection layer. The improvements of WGAN to GAN mainly include: the loss functions of the generator and the discriminator do not take log; the updated weights are forced to be truncated to a certain range, such as [-0.01, 0.01]; WGAN theoretically gives the reason for the instability of GAN training, that is, the cross entropy (Jensen-Shannon divergence) is not suitable for measuring the distance between distributions with disjoint parts, and the wassertein distance is used to measure the distance between the generated data distribution and the real data distribution, which theoretically solves the problem of training instability; the problem of collapse mode is solved, and the diversity of the generated results is richer.

[0064] It should be noted that in the machine learning algorithm, the objective function is the core orientation of the whole model optimization learning. When the objective function needs to be minimized, the objective function is also called the loss function or the cost function. For a supervised learning task, the goal is usually to make the predicted value as close as possible to the label, that is, to minimize the objective function, so in the neural network it is generally called the loss function.

[0065] The image recognition training method provided in the embodiment can effectively maintain the consistency of the entity type visual features and the semantic features by adding the limitation of semantic consistency in the process of training the trained classification model, guarantee the training stability, and improve the accuracy of zero sample recognition and the diversity of the generated results.

[0066] In a specific embodiment, as shown in FIG. 1, in step S10, the corresponding extended attributes are constructed based on the entity attributes of the entity types, and the specific steps include the following steps: Figure 3

[0067] S11. The known class visual features are decomposed by the MSE loss and the CE loss in sequence to obtain the visual distinguishing features.

[0068] S12. The class prototypes of the known class visual features are extracted by the triplet loss.

[0069] ​S13. Extracting the entity attribute corresponding to the known class visual feature, training the entity attribute through the MSE loss to align the entity attribute with the visual distinguishing feature and the class prototype respectively, and taking the training result as the extended attribute corresponding to the entity type.

[0070] The MSE (mean-square error) loss refers to the expected value of the square of the difference between the parameter estimate and the true value of the parameter.

[0071] The CE (Cross Entropy Loss) loss is used to predict the probability of an input sample belonging to a certain class.

[0072] The class prototype is a type similar to the known type. For example, for the type of cow in animals, the known type is dairy cow, and the class prototype can be derived to include buffalo, yak, and yellow cattle, etc.

[0073] The Triplet Loss is a loss function in deep learning, which is used to train samples with small differences, such as human faces. The input Feed data includes Anchor example, Positive example, and Negative example. By optimizing the distance between the Anchor example and the Positive example to be smaller than the distance between the Anchor example and the Negative example, the similarity of the samples is calculated.

[0074] The model alignment referred to in the present application refers to making the entity attribute approach the visual distinguishing feature and the class prototype respectively, which is realized by the loss function MSE, so as to make the probability distribution of the visual feature and the class prototype the same.

[0075] Specifically, as shown in Figure 4 The selection of the loss function depends on the type of input label data. If the input is a real number, an unbounded value, the loss function uses the square difference; if the input label is a bit vector (classification label), it is more appropriate to use cross-entropy. The visual distinguishing feature is the feature obtained by matrixing the known class visual feature and decomposing it into a matrix representation.

[0076] The MSE loss and the CE loss are used to improve the distinguishing feature of the visual feature x. The class prototype of the visual feature x is extracted through the Triplet loss. The difference between images is called distinguishing feature. The larger the distinguishing feature, the more accurate the classification. The attribute feature a is extracted respectively, the MSE loss is used to align the attribute feature a with the distinguishing visual feature and the class prototype respectively, and the extended attribute A is finally formed.

[0077] In a specific embodiment, the classification model to be trained includes an encoder and a generator. As shown in Figure 5As shown in step S20, the known class visual features of the entity type are semantically consistent synthesized based on the entity attributes and the extended attributes of the entity type, and known class synthesized features corresponding to the entity attributes and the extended attributes are obtained, specifically including the following steps:

[0078] S21. The entity attributes, the extended attributes, and the known class visual features of the entity type are input into an encoder to obtain noise distribution parameters.

[0079] In this embodiment, the noise distribution parameters are matched with the parameters of a zero-mean unit-variance Gaussian prior distribution using KL (Kullback-Leibler divergence) divergence, and are used to conform to the Gaussian prior distribution.

[0080] S22. The noise distribution parameters, the entity attributes, and the extended attributes of the entity type are input into a generator to obtain known class synthesized features of the entity type.

[0081] Specifically, as shown in the formula: Figure 6 The input of E is the known class visual features x, the attributes a, and the extended attributes A of the visible class, and the output of E is the noise distribution parameters z.

[0082] The input of G is the noise distribution parameters z, the entity attributes a, and the extended attributes A, and the output is the synthesized sample pseudo features (known class synthesized features).

[0083] Preferably, in step S20, the known class visual features of the entity type are semantically consistent synthesized based on the entity attributes and the extended attributes of the entity type, and known class synthesized features corresponding to the entity attributes and the extended attributes are obtained, specifically including the following steps:

[0084] S23. The known class visual features and the known class synthesized features are compared using binary cross-entropy loss.

[0085] The binary cross-entropy, also known as log loss, is a cost function used in logistic regression, and is the negative average of the logs of the corrected predicted probabilities of each instance. The log loss is the most important classification indicator based on probability. For any given problem, a lower log loss value means better prediction.

[0086] Specifically, the comparison of the known class synthesized features and the known class visual features x uses binary cross-entropy loss, so that the pseudo features generated by the generator G are closer to the real features x.

[0087] In a specific embodiment, the classification model to be trained includes a generator, an entity attribute decoder, and an extended attribute decoder. As shown in the formula: Figure 7As shown, after step S20, that is, after obtaining the known class synthetic features corresponding to the entity attribute and the extended attribute respectively, the method further comprises the following steps:

[0088] S201. Input the known class visual features and the entity attribute into an entity attribute decoder to obtain a real reconstructed attribute.

[0089] S202. Input the known class visual features and the extended attribute into a reconstructed attribute decoder to obtain an extended reconstructed attribute.

[0090] S203. Use a cycle consistency loss to train the real reconstructed attribute and the extended reconstructed attribute, so as to approximate the known class synthetic features to the known class visual features.

[0091] The real reconstructed attribute is an extended pseudo feature attribute obtained by decoding the known class visual features and the entity attribute in the entity attribute decoder.

[0092] The extended reconstructed attribute is an extended pseudo feature attribute obtained by decoding the known class visual features and the extended attribute in the reconstructed attribute decoder.

[0093] The cycle consistency loss is generally k layers of down-sampling, n layers of resblock network cells, and k layers of up-sampling. The number of layers of k is generally not large. According to the visualized output results of each network layer, the first k layers can greatly retain the original structural information. The resblock in the back has strong gradient propagation ability, so the cycle consistency loss weight is generally higher than the adversarial loss in the optimization process, so the adversarial loss will not cause the output of the intermediate layer and the generated image to have a large structural difference from the original image.

[0094] Specifically, the discriminator D inputs x or As inputs, the attribute a and the extended attribute A, and through training, the ability to distinguish between true features and pseudo features is continuously improved. The output result of the discriminator D is a W loss function.

[0095] In the joint model of the traditional VAE (variational-autoencoders, variational auto-encoder) and GAN, the generator G shares parameters with the decoders Dec and Dec2, but lacks consistency constraints on semantic embedding. Therefore, the decoder Dec is used to Visual features as input reconstructed attribute The output R is a loss function. The decoder Dec2 is used to Visual features as input reconstructed attribute And a cycle consistency loss or L R2 is used to make the generated pseudo features closer to the true features. The cycle consistency loss can be expressed as:

[0096]

[0097] where Dec(x) and Dec(a) are the attribute features of the real and fake features after reconstruction by Dec, respectively.

[0098] In a specific embodiment, in step S30, the known-class visual features, the known-class synthetic features, the entity attributes, and the extended attributes are taken as input data of the classification model to be trained, the class prototypes corresponding to the known-class visual features and the entity types are taken as output data of the classification model to be trained, the classification model to be trained is trained, and finally a trained classification model is generated, specifically including the following steps:

[0099] S31. The WGAN network is taken as the classification model to be trained, the known-class visual features, the known-class synthetic features, the entity attributes, and the extended attributes are taken as input data of the classification model to be trained, the WGAN network is trained by optimizing the loss, and finally the trained classification model is generated.

[0100] Specifically, the model is constructed based on WGAN (Wasserstein GAN), so that the training process is more stable. The generated fake sample features are more distinguishable in combination with the powerful WGAN loss and the classification loss. This model lays the foundation for most subsequent GAN model-based basic architectures. The visual features are mapped back to the semantic space, and the correspondence from the visual to the semantic is completed by constructing a cycle consistency loss function.

[0101] The WGAN network provided in this embodiment combines the advantages of the VAE and GAN joint model, as shown in Figure 6 The training is more stable, and good results are achieved in ZSL image classification. In the VAE-GAN network, an encoder E(x, a) and a generator G(z, a) are included. The encoder takes the feature x as input to generate a latent code hidden variable z conforming to a Gaussian distribution. The generator reconstructs z back to x. Both E and G are conditioned on the semantic embedding feature a, the purpose being to constrain the generated feature in the visual space, so that the attribute feature is integrated into the visual feature. The optimization function is as follows:

[0102]

[0103] where, is the optimization objective function, which is composed of the KL loss and the binary cross-entropy loss . Wherein, KL is the Kullback-Leibler divergence, and p(z|a) is assumed to be subject to ​the prior distribution of z, and logG(z, a) is the reconstruction loss. The generative adversarial network adopts WGAN, which solves the problem of training instability to a certain extent compared with the original GAN, and the optimization loss of WGAN is:

[0104]

[0105] wherein, is the synthetic feature, and λ is a penalty coefficient, and δ obeys U(0, 1).

[0106] The final loss formula is:

[0107]

[0108] wherein, α1 and α2 are both hyperparameters, and a hyperparameter is a parameter whose value is set before starting the learning process, rather than a parameter data obtained through training.

[0109] In a specific embodiment, as shown in Figure 8 after step S30, that is, after the final generation of the trained classification model, the method further specifically includes the following steps:

[0110] S301. Obtain an unseen class visual feature, and input the unseen class visual feature into the trained category recognition model to identify an identification type corresponding to the unseen class visual feature.

[0111] Specifically, the unseen class visual feature is the feature of an entity whose type is to be confirmed. The identification type is the entity type of the entity to be confirmed.

[0112] The image recognition training method provided in the embodiment can effectively maintain the consistency between the entity type visual feature and the semantic feature by adding a limitation on semantic consistency during the training of the trained classification model, thereby guaranteeing the training stability and improving the accuracy of zero-shot recognition.

[0113] As shown in Figure 6As shown, the attribute extension-based zero-shot classification network structure proposed in the application mainly consists of an encoder E, a generator G, decoders Dec and Dec2, a discriminator D, and a feedback module FM, wherein the feedback module FM is configured to feed back the intermediate layer features of D to G. The application combines a stable VAE (variational auto-encoder) and a GAN to process a ZSL (zero-shot learning) task, the encoder (Encoder) in the VAE represents input data as data conforming to a Gaussian distribution, and the decoder (Decoder) reconstructs the input by a latent variable. The two models are combined in a manner that the parameters of the VAE decoding module and the GAN generator are shared, and this combination manner can be regarded as adding the constraint of the VAE to the input of the generator of the GAN.

[0114] Further, in combination with the CVAE (Conditional AutoEncoder) conditioned on semantic attributes and the CGANS conditioned on classes and attributes, the method further uses a direct inference zero-shot learning manner to fine-tune the network by using the unlabeled invisible class data, so as to generate higher-quality features.

[0115] In order to improve the reconstruction ability of the decoding network, a feedback mechanism based on attribute regression loss is added on the basis of the encoder to generate features with semantic consistency.

[0116] The embodiments provided in the application fuse the visual-text semantic mapping, the text semantic-visual mapping, and the metric learning method in a unified framework, which correspond to the generator module, the regressor module, and the discriminator module of the proposed model respectively. The discriminator loss is constituted by a plurality of forms of pseudo data through the combination of the text semantic and the visual feature, which can help to generate more robust cross-modal corresponding relationship. In order to avoid generating unconstrained visual features, the application also constrains the cyclic consistency loss of different branches, promotes the diversity of the visual features, and maintains the consistency between the visual features and the semantic features.

[0117] The embodiments provided in the application combine the advantages of the VAE and the GAN, and can also generate more real pseudo features to directly improve the accuracy of the classification result. Secondly, the known class visual features, the known class synthetic features, the entity attributes, and the extended attributes are simultaneously used as the constraints of the generation model, which makes the visual features synthesized by the model full of diversity and distinguishability.

[0118] The embodiment proposes an attribute extension-based zero-shot classification network and an image recognition training method. Five commonly used data sets for zero-shot image classification are AWA1 (Animal with Attributes, animal label data set), AWA2, CUB (Caltech-UCSD Birds, bird data set), SUN (Scene Understanding, scene understanding), and FLO (Oxford Flowers, Oxford flower data set). As shown in Table 1 (ZSL image classification experiment results), the image recognition method provided in the embodiment verifies the calculation advantage in the above data sets, and further verifies that the model provided in the embodiment has the same advantage in other public data sets.

[0119]

[0120]

[0121] Table 1

[0122] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0123] In an embodiment, an image recognition training device is provided, which corresponds to the image recognition training method in the above embodiment. As shown in the figure, the image recognition training device includes a constructed extension attribute module 10, an obtained synthesized feature module 20, and a generated classification module 30. The functions of each module are described in detail as follows: Figure 9

[0124] The constructed extension attribute module 10 is configured to construct corresponding extension attributes based on entity attributes of entity types;

[0125] The obtained synthesized feature module 20 is configured to perform semantic consistency synthesis on known class visual features of entity types based on entity attributes and extension attributes of entity types, to obtain known class synthesized features corresponding to entity attributes and extension attributes, respectively;

[0126] The generated classification module 30 is configured to take the known class visual features, the known class synthesized features, the entity attributes, and the extension attributes as input data of a to-be-trained classification model, take class prototypes corresponding to the known class visual features and entity types as output data of the to-be-trained classification model, train the to-be-trained classification model, and finally generate a trained classification model.

[0127] ​Preferably, the constructing extended attribute module 10 comprises a obtaining partition feature submodule, an extracting class prototype submodule and an extracting entity attribute submodule. The function modules are described in detail as follows:

[0128] The obtaining partition feature submodule is configured to sequentially perform feature decomposition on the known class visual features through MSE loss and CE loss, and obtain visual distinguishing features.

[0129] The extracting class prototype submodule is configured to extract class prototypes of the known class visual features through a triplet loss.

[0130] The extracting entity attribute submodule is configured to extract entity attributes corresponding to the known class visual features, train the entity attributes through MSE loss, so that the entity attributes are respectively aligned with the visual distinguishing features and the class prototypes, and take the training results as the extended attributes corresponding to the entity types.

[0131] Preferably, the obtaining synthetic feature module 20 comprises an obtaining noise parameter submodule and an obtaining synthetic feature submodule. The function modules are described in detail as follows:

[0132] The obtaining noise parameter submodule is configured to input the entity attributes, the extended attributes of the entity types and the known class visual features into an encoder, and obtain noise distribution parameters.

[0133] The obtaining synthetic feature submodule is configured to input the noise distribution parameters, the entity attributes and the extended attributes of the entity types into a generator, and obtain known class synthetic features of the entity types.

[0134] Preferably, the image recognition training device further comprises a comparing visual feature module. The function modules are described in detail as follows:

[0135] The comparing visual feature module is configured to compare the known class visual features and the known class synthetic features through binary cross-entropy loss.

[0136] Preferably, the image recognition training device further comprises an obtaining real reconstructed attribute module, an obtaining extended reconstructed attribute module and a training extended reconstructed attribute module. The function modules are described in detail as follows:

[0137] The obtaining real reconstructed attribute module is configured to input the known class visual features and the entity attributes into an entity attribute decoder, and obtain real reconstructed attributes.

[0138] The obtaining extended reconstructed attribute module is configured to input the known class visual features and the extended attributes into a reconstructed attribute decoder, and obtain extended reconstructed attributes.

[0139] The training extended reconstructed attribute module is configured to train the real reconstructed attributes and the extended reconstructed attributes through cycle consistency loss, so as to approximate the known class synthetic features to the known class visual features.

[0140] Preferably, the generating classification module 30 comprises a training WGAN network module. The function modules are described in detail as follows:

[0141] The training WGAN network module is configured to use the WGAN network as a classification model to be trained, use the known class visual features, the known class synthetic features, the entity attributes and the extended attributes as input data of the classification model to be trained, train the WGAN network by optimizing the loss, and finally generate a trained classification model.

[0142] Preferably, the image recognition training apparatus further comprises a recognition unseen class feature module. The function modules are described in detail as follows:

[0143] The recognition unseen class feature module is configured to acquire the unseen class visual features, input the unseen class visual features into the trained class recognition model, and identify the recognition type corresponding to the unseen class visual features.

[0144] The specific limitations of the image recognition training apparatus can be referred to the limitations of the image recognition training method described above, which will not be repeated here. The modules in the above image recognition training apparatus can be realized by software, hardware and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0145] In an embodiment, a computer device is provided, which can be a server, and the internal structure diagram thereof can be as shown in Figure 10 The computer device comprises a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data related to the image recognition training method. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an image recognition training method.

[0146] In an embodiment, a computer device is provided, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor. The processor implements the image recognition training method of the above embodiment when executing the computer program, for example Figure 2The processor implements the functions of the modules 10 to 30 of the image recognition training device in the above embodiment when executing the computer program. For the sake of brevity, the functions of the modules 10 to 30 will not be repeated here. Figure 9 The processor implements the functions of the modules 10 to 30 of the image recognition training device in the above embodiment when executing the computer program. For the sake of brevity, the functions of the modules 10 to 30 will not be repeated here.

[0147] In an embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program, when executed by a processor, implements the image recognition training method in the above embodiment, for example Figure 2 The processor implements the functions of the modules 10 to 30 of the image recognition training device in the above embodiment when executing the computer program. For the sake of brevity, the functions of the modules 10 to 30 will not be repeated here. Figure 9 The processor implements the functions of the modules 10 to 30 of the image recognition training device in the above embodiment when executing the computer program. For the sake of brevity, the functions of the modules 10 to 30 will not be repeated here.

[0148] Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above embodiments can be included. Any reference to memory, storage, database or other storage medium in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0149] Those skilled in the art can clearly understand that, for the sake of brevity and simplicity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above described functions.

[0150] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An image recognition training method, characterized in that, include: S10. Based on the entity type's entity attributes, construct corresponding extended attributes; S20. Based on the entity attributes and extended attributes of the entity type, perform semantic consistency synthesis on the known class visual features of the entity type to obtain the known class synthesized features corresponding to the entity attributes and extended attributes respectively. S30. The known class visual features, known class synthetic features, entity attributes and extended attributes are used as input data for the classification model to be trained. The class prototypes corresponding to the known class visual features and the entity types are used as output data for the classification model to be trained. The classification model to be trained is trained and finally a trained classification model is generated. S10 includes: The known visual features are sequentially decomposed using MSE loss and CE loss to obtain visual distinguishing features. The class prototype is extracted using the known class visual features through triplet loss; Extract the entity attributes corresponding to the known class visual features, train the entity attributes using MSE loss, so that the entity attributes are aligned with the visual distinguishing features and the class prototype respectively, and use the training results as the extended attributes corresponding to the entity type; S30 includes: The WGAN network is used as the classification model to be trained. The known class visual features, known class synthetic features, entity attributes and extended attributes are used as the input data of the classification model to be trained. The WGAN network is trained by optimizing the loss to finally generate the trained classification model. Unseen visual features are acquired, and the unseen visual features are input into the trained classification model to identify the recognition type corresponding to the unseen visual features. The classification model to be trained includes a generator, an entity attribute decoder, and an extended attribute decoder; Following S20, the following is also included: The known class visual features and the entity attributes are input into the entity attribute decoder to obtain the true reconstructed attributes; The known class visual features and the extended attributes are input into the reconstruction attribute decoder to obtain the extended reconstruction attributes; Cyclic consistency loss is used to train the true reconstructed attributes and the extended reconstructed attributes to approximate the known class visual features with the known class synthetic features; The cycle consistency loss L R Represented as: Where Dec(x) and These are the attribute features reconstructed by Dec from true and false features, respectively, where 'a' represents the entity attribute.

2. The image recognition training method according to claim 1, characterized in that, The classification model to be trained includes an encoder and a generator; The step of performing semantic consistency synthesis on the known class visual features of the entity type based on the entity attributes and extended attributes of the entity type, and obtaining the known class synthesized features corresponding to the entity attributes and extended attributes respectively, includes: The entity attributes, extended attributes, and known class visual features of the entity type are input into the encoder to obtain noise distribution parameters; The noise distribution parameters, entity attributes of the entity type, and extended attributes are input into the generator to obtain the known class synthesis features of the entity type.

3. The image recognition training method according to claim 2, characterized in that, Also includes: The known class visual features and the known class synthetic features are compared using binary cross-entropy loss.

4. An image recognition training device, characterized in that, include: Build an extended attribute module to construct corresponding extended attributes based on entity attributes of entity type; The module for obtaining synthetic features is used to perform semantic consistency synthesis on the known class visual features of the entity type based on the entity attributes and extended attributes of the entity type, and to obtain the known class synthetic features corresponding to the entity attributes and extended attributes respectively. The classification model to be trained includes a generator, an entity attribute decoder, and an extended attribute decoder; The module for obtaining real reconstruction attributes is used to input the known class visual features and the entity attributes into the entity attribute decoder to obtain real reconstruction attributes. An extended reconstruction attribute module is used to input the known class visual features and the extended attributes into the reconstruction attribute decoder to obtain the extended reconstruction attributes; The training extended reconstruction attribute module is used to train the real reconstruction attribute and the extended reconstruction attribute using cycle consistency loss, so as to approximate the known class synthetic feature with the known class visual feature; A classification generation module is used to take the known class visual features, known class synthetic features, entity attributes, and extended attributes as input data to the classification model to be trained, take the class prototype corresponding to the known class visual features and the entity type as output data to the classification model to be trained, train the classification model to be trained, and finally generate a trained classification model. The construction of the extended attribute module includes: The partition feature acquisition submodule is used to sequentially perform feature decomposition on the known class visual features through MSE loss and CE loss to obtain visual discrimination features; The class prototype extraction submodule is used to extract the class prototype of the known class visual features through triplet loss; The entity attribute extraction submodule is used to extract entity attributes corresponding to the known class visual features, train the entity attributes using MSE loss so that the entity attributes are aligned with the visual distinguishing features and the class prototype respectively, and use the training results as the extended attributes corresponding to the entity type; The classification generation module includes: The WGAN network training module is used to use the WGAN network as the classification model to be trained. The known class visual features, known class synthetic features, entity attributes and extended attributes are used as input data for the classification model to be trained. The WGAN network is trained by optimizing the loss to finally generate the trained classification model. The module for identifying unseen class features is used to acquire unseen class visual features and input the unseen class visual features into the trained classification model to identify the recognition type corresponding to the unseen class visual features. The cycle consistency loss L R Represented as: Where Dec(x) and These are the attribute features reconstructed by Dec from true and false features, respectively, where 'a' represents the entity attribute.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image recognition training method as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image recognition training method as described in any one of claims 1 to 3.

Images