An image classification method and apparatus

Abstract

The application discloses an image classification method and device. The method comprises the following steps: performing semantic-guided attention weight processing on an image to be classified, extracting attribute visual primitive features and object visual primitive features, and processing the features by using an image classifier to obtain a target classification result corresponding to the image. The image classifier comprises fine-grained attribute prototypes and fine-grained object prototypes. Each fine-grained attribute prototype is calculated based on attribute visual primitive features extracted from an image sample combined with corresponding attribute semantic features and object semantic features, and the fine-grained object prototypes are calculated in the same way. Based on this, each attribute semantic feature or object semantic feature has a plurality of fine-grained attribute prototypes or fine-grained object prototypes corresponding thereto, the visual diversity of each attribute semantic feature and object semantic feature is improved, and the accuracy of image classification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and more particularly to an image classification method and device. BACKGROUND

[0002] In the field of computer vision, image classification can learn visual primitive features corresponding to multiple different semantic labels, and classify based on the visual primitive features of the image to be classified and the visual primitive features corresponding to the above semantic labels. Usually, the semantic information represented by an image is not only one, which can include object information and attribute information. Based on this, a trained combined zero-shot classification model can be used to identify the object semantic label and attribute semantic label corresponding to the image.

[0003] However, in the prior art, usually only one unified visual primitive feature is learned for one semantic label, ignoring the difference between the visual representations of the same semantic label in different image scenes. Based on the above visual image classification lacking diversity, the final image classification result will also have a large difference with the image itself. SUMMARY

[0004] Therefore, the present application provides an image classification method and device to solve the problem of single visual feature in the existing image recognition method.

[0005] In order to achieve the above purpose, the present scheme is as follows:

[0006] An image classification method comprises:

[0007] Obtaining an image to be classified;

[0008] Performing semantic-guided attention weight processing on the image to be classified to obtain attribute visual primitive features and object visual primitive features of the image to be classified, wherein the semantic-guided attention weight processing is a processing process of calculating attribute visual primitive features and object features based on feature maps of the image to be classified and attribute semantic features and object semantic features extracted from a training data set;

[0009] inputting the attribute visual element features of the to-be-classified image and the object visual element features of the to-be-classified image into an image classifier for classification processing to obtain a target classification result corresponding to the to-be-classified image, the image classifier being trained by image samples in the training data set, attribute semantic features and object semantic features corresponding to the image samples, and fine-grained attribute prototypes and fine-grained object prototypes, so that the image classifier classifies attribute visual element features and object visual element features of an image, wherein each fine-grained attribute prototype is calculated from attribute visual element features extracted from an image sample corresponding to the combination of the attribute semantic features and the object semantic features, and the fine-grained object prototype is calculated from object visual element features extracted from an image sample corresponding to the combination of the object semantic features and the attribute semantic features.

[0010] Preferably, the method further comprises:

[0011] obtaining a training data set, the training data set including a plurality of image samples, each of the image samples including an attribute label and an object label;

[0012] extracting feature maps from each of the image samples;

[0013] performing semantic extraction processing on the attribute label and the object label to obtain attribute semantic features and object semantic features;

[0014] based on semantic similarity of the attribute semantic features and the object semantic features, selecting a target image from the image samples for image mixing to obtain a new image sample;

[0015] extracting feature maps, attribute semantic features, and object semantic features of the new image sample;

[0016] performing semantic-guided attention weight processing on the feature maps, the attribute semantic features, and the object semantic features of all the image samples and the new image sample to obtain attribute visual element features and object visual element features corresponding to each of the image samples or the new image sample;

[0017] based on a preset classifier structure, the attribute visual element features, and the object visual element features, training to obtain an image classifier.

[0018] Preferably, the semantic-guided attention weight processing on the feature maps, the attribute semantic features, and the object semantic features of all the image samples and the new image sample to obtain attribute visual element features and object visual element features corresponding to each of the image samples or the new image sample comprises:

[0019] performing calculation on all the attribute semantic features and the feature maps to obtain attribute similarity between the feature maps and the attribute semantic features;

[0020] performing convolution calculation and integration on the attribute similarity and performing calculation on the feature maps to obtain attribute visual primitive features corresponding to the attribute semantic features of the image samples or the new image samples;

[0021] performing calculation on all the object semantic features and the feature maps to obtain object similarity between each feature map and each object semantic feature;

[0022] performing convolution calculation and integration on the object similarity and performing calculation on the feature maps to obtain object visual primitive features corresponding to the object semantic features of the image samples or the new image samples.

[0023] Preferably, the image classifier comprises a fine-grained primitive classifier, wherein the image classifier is obtained by training based on the preset classifier structure, the attribute visual primitive features and the object visual primitive features, and the fine-grained primitive classifier comprises:

[0024] performing fine-grained division on the image samples and the new image samples based on the attribute semantic features and the object semantic features of the image samples and the new image samples to obtain a plurality of fine-grained categories, each of the fine-grained categories comprising at least one of the image samples or the new image samples, and the attribute semantic features and the object semantic features of the image samples or the new image samples in the same fine-grained category being the same;

[0025] performing calculation on the attribute visual primitive features of the image samples or the new image samples included in each of the fine-grained categories to obtain a fine-grained attribute prototype of each of the fine-grained categories;

[0026] performing calculation on the object visual primitive features of the image samples or the new image samples included in each of the fine-grained categories to obtain a fine-grained object prototype of each of the fine-grained categories;

[0027] initializing the preset classifier structure by using the fine-grained attribute prototype and the fine-grained object prototype to obtain a fine-grained primitive classifier to be trained;

[0028] training based on the to-be-trained fine-grained base classifier and the image sample, to obtain a fine-grained base classifier, wherein the training process is based on the attribute visual base feature and the object visual base feature of the image sample to train, so that the fine-grained base classifier calculates a prediction score between the fine-grained attribute prototype and the fine-grained object prototype according to the attribute visual base feature and the object visual base feature respectively, and calculates and outputs a fine-grained prediction score based on the prediction score.

[0029] Preferably, the method further comprises:

[0030] mapping all of the attribute visual base feature, the object visual base feature, the attribute semantic feature, the object semantic feature, the fine-grained attribute prototype and the fine-grained object prototype to a contrast space;

[0031] in the contrast space, the attribute visual base feature, the object visual base feature, the attribute semantic feature, the object semantic feature, the fine-grained attribute prototype and the fine-grained object prototype are constrained by a loss function to obtain reinforced attribute visual base feature, object visual base feature, fine-grained attribute prototype and fine-grained object prototype, and the loss function at least includes a multi-modal contrast loss, a fine-grained prototype loss and a coarse-grained prototype loss;

[0032] initializing a preset classifier structure by using the reinforced fine-grained attribute prototype and fine-grained object prototype to obtain a to-be-trained fine-grained base classifier;

[0033] training based on the to-be-trained fine-grained base classifier and the image sample, to obtain a fine-grained base classifier, wherein the training process is based on the attribute visual base feature and the object visual base feature of the image sample to train, so that the fine-grained base classifier calculates a prediction score between the fine-grained attribute prototype and the fine-grained object prototype according to the attribute visual base feature and the object visual base feature respectively, and calculates and outputs a fine-grained prediction score based on the prediction score.

[0034] Preferably, the method further comprises:

[0035] in the image classifier, the image classifier is constrained by a classification loss function.

[0036] Preferably, the inputting the to-be-classified attribute visual base feature and the to-be-classified object visual base feature into the image classifier for classification processing to obtain a target classification result corresponding to the to-be-classified image comprises:

[0037] input the attribute visual element features to be classified and the object visual element features to be classified into the image classifier, and process them by the fine-grained element classifier and the coarse-grained element classifier respectively to obtain fine-grained prediction scores output by the fine-grained element classifier and coarse-grained prediction scores output by the coarse-grained element classifier;

[0038] calculate the fine-grained prediction scores and the coarse-grained prediction scores according to preset weights to obtain target prediction scores of the image to be classified;

[0039] determine a combined label based on the target prediction scores, and take the combined semantic features as a target classification result, wherein the combined label is a combination of attribute labels and object labels.

[0040] An image classification device comprises:

[0041] an image information acquisition unit configured to acquire an image to be classified;

[0042] an element feature calculation unit configured to perform semantic-guided attention weight processing on the image to be classified to obtain attribute visual element features to be classified and object visual element features to be classified of the image to be classified, wherein the semantic-guided attention weight processing is a processing process of calculating attribute visual element features and object features based on attribute semantic features and object semantic features extracted from a training data set and feature maps of the image to be classified;

[0043] an image classification unit configured to input the attribute visual element features to be classified and the object visual element features to be classified into an image classifier for classification processing to obtain a target classification result corresponding to the image to be classified, wherein the image classifier is trained by image samples in the training data set, attribute semantic features and object semantic features corresponding to the image samples, and fine-grained attribute prototypes and fine-grained object prototypes, so as to classify attribute visual element features and object visual element features of an image, wherein each fine-grained attribute prototype is calculated from attribute visual element features extracted from image samples corresponding to combined attribute semantic features and object semantic features, and the fine-grained object prototype is calculated from object visual element features extracted from image samples corresponding to combined attribute semantic features and object semantic features.

[0044] Preferably, the device further comprises:

[0045] a training data acquisition unit configured to acquire a training data set, wherein the training data set comprises a plurality of image samples, and each image sample comprises attribute labels and object labels;

[0046] a feature map extraction unit configured to extract feature maps from each image sample.

[0047] a semantic feature extraction unit configured to perform semantic extraction processing on the attribute label and the object label to obtain attribute semantic features and object semantic features;

[0048] a sample synthesis unit configured to select a target image from the image samples based on semantic similarity of the attribute semantic features and the object semantic features, and perform image mixing to obtain new image samples;

[0049] an information extraction unit configured to extract feature maps, attribute semantic features, and object semantic features of the new image samples;

[0050] a visual primitive feature acquisition unit configured to perform semantic-guided attention weight processing on the feature maps, the attribute semantic features, and the object semantic features of all the image samples and the new image samples to obtain attribute visual primitive features and object visual primitive features corresponding to each of the image samples or the new image samples;

[0051] a classifier training unit configured to train, based on a preset classifier structure, the attribute visual primitive features, and the object visual primitive features, to obtain an image classifier.

[0052] Preferably, the visual primitive feature acquisition unit comprises:

[0053] an attribute similarity calculation subunit configured to calculate all the attribute semantic features and the feature maps to obtain attribute similarity between the feature maps and the attribute semantic features;

[0054] an attribute visual primitive feature acquisition subunit configured to perform convolution calculation and integration on the attribute similarity and calculation with the feature maps to obtain attribute visual primitive features corresponding to the attribute semantic features of the image samples or the new image samples;

[0055] an object similarity calculation subunit configured to calculate all the object semantic features and the feature maps to obtain object similarity between each feature map and each of the object semantic features;

[0056] an object visual primitive feature calculation subunit configured to perform convolution calculation and integration on the object similarity and calculation with the feature maps to obtain object visual primitive features corresponding to the object semantic features of the image samples or the new image samples.

[0057] The image classifier of the present application comprises fine-grained attribute prototypes and fine-grained object prototypes. Taking the fine-grained attribute prototypes as an example, each fine-grained attribute prototype has an attribute semantic feature corresponding thereto, and the fine-grained attribute prototype can be calculated from the attribute semantic feature and the attribute visual primitive feature extracted from all image samples combined with the same object semantic feature. Since one attribute semantic feature can be combined with multiple object semantic features respectively, a corresponding fine-grained attribute prototype can be calculated from the attribute visual primitive feature embodied by each combination. Based on this, each attribute semantic feature or object semantic feature has multiple fine-grained attribute prototypes or fine-grained object prototypes corresponding thereto, which improves the visual diversity of each attribute semantic feature and object semantic feature. Based on this, the image is classified, which is beneficial to improving the accuracy of image classification. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0059] Figure 1 A flowchart of an image classification method provided by an embodiment of the present application is shown in the figure.

[0060] Figure 2 A flowchart of an image classifier provided by an embodiment of the present application is shown in the figure.

[0061] Figure 3 A visual diagram of an attention mechanism module provided by an embodiment of the present application is shown in the figure.

[0062] Figure 4 A structural diagram of a semantic-guided attention mechanism module provided by an embodiment of the present application is shown in the figure.

[0063] Figure 5 A structural diagram of a part of a system for implementing an image classification method provided by an embodiment of the present application is shown in the figure.

[0064] Figure 6 A structural diagram of another part of a system for implementing an image classification method provided by an embodiment of the present application is shown in the figure.

[0065] Figure 7 A structural diagram of an image classification device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0066] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0067] With reference to Figure 1 It shows a flow diagram of an image classification method provided by an embodiment of the present application. The method can be used on a server, a terminal, a mobile terminal or other electronic devices, for classifying images, and using an image classifier to classify images based on attribute visual primitive features and object visual primitive features of the images to obtain a final target classification result of the images. Specifically, the flow of the image classification method can include:

[0068] In step S110, a to-be-classified image is obtained.

[0069] In step S120, a semantic-guided attention weight processing is performed on the to-be-classified image to obtain to-be-classified attribute visual primitive features and to-be-classified object visual primitive features of the to-be-classified image.

[0070] In the embodiment of the present application, after obtaining the to-be-classified image, further processing is needed to obtain feature maps and other information corresponding to the to-be-classified image, and to calculate visual primitive features corresponding to the to-be-classified image. The visual primitive features include attribute visual primitive features and object visual primitive features. The feature map can convert the visual image of the to-be-classified image into a digital image with three-dimensional digital information, i.e., representing the visual content contained in the to-be-classified image with digital information. The visual content can include image size, image color, etc. A pre-trained ResNet18 can be used to extract a feature map based on the to-be-classified image, so as to classify the image based on the feature map subsequently.

[0071] The visual primitive features are specific representations of a certain visual region in the image determined according to the semantic features expressed by the image, which are more discriminative and less noisy than global features extracted from the image. In the embodiment of the present application, for the to-be-classified image without intuitive attribute semantic features and object semantic features, a semantic-guided attention weight processing method can be used to obtain to-be-classified attribute visual primitive features and to-be-classified object visual primitive features of the to-be-classified image.

[0072] In the embodiment of the present application, the semantic-guided attention weight processing can be based on similarity calculation of the feature map of the image to be classified with all known semantic features, and based on the calculation result, convolution, integration and other calculations are performed to obtain the attribute visual primitive features and the object visual primitive features corresponding to the image to be classified.

[0073] In step S130, the attribute visual primitive features and the object visual primitive features to be classified are input into an image classifier for classification processing to obtain the target classification result of the image to be classified.

[0074] The image classifier is pre-prepared. In the embodiment of the present application, a plurality of image samples in a training data set can be used as training samples, wherein each image sample has corresponding attribute semantic features and object semantic features. The two types of semantic features can be obtained by an encoder from image information of the image sample, wherein the image information can include attribute labels, object labels, etc. The image classifier is mainly trained based on fine-grained attribute prototypes and fine-grained object prototypes, wherein each fine-grained attribute prototype is calculated from attribute visual primitive features extracted from image samples corresponding to attribute semantic features and object semantic features, and the fine-grained object prototype is calculated from object visual primitive features extracted from image samples corresponding to object semantic features and attribute semantic features.

[0075] As known from the above, the visual primitive features are specific representations of image semantic features, and the attribute visual primitive features and the object visual primitive features corresponding to the attribute semantic features or the object semantic features can be extracted from images as classification basis. However, usually the semantic features of an image can be a combination of attribute semantic features and object semantic features, and different combinations of attribute semantic features and object semantic features have different visual contents, and the attribute visual primitive features and the object visual primitive features obtained are also different. That is, there is a visual diversity problem between attribute semantic features and object semantic features due to mutual dependence, and the same attribute semantic feature has great visual difference under different attribute-object combinations, and similarly, the same object semantic feature also has different visual performances under different attribute-object combinations. For example, the mature performance in “mature apple” is red, while the mature performance in “mature banana” is yellow. If the same “mature” prototype is used to represent such a big difference in features, the classification effect will be poor.

[0076] Therefore, to solve the problem of visual diversity, attribute visual primitive features can be extracted in images of any combination of attribute semantic features and object semantic features (or attribute-object combination) for each attribute semantic feature, and further, a plurality of fine-grained attribute prototypes can be created. Similarly, object visual primitive features can be extracted in images of any combination of object semantic features and attribute semantic features for each object semantic feature, and further, a plurality of fine-grained object prototypes can be created. Thus, each attribute semantic feature and object semantic feature can be dependent on each other while representing the visual primitive features they represent.

[0077] Specifically, in the embodiments of the present application, attribute semantic features and object semantic features corresponding to each image are extracted using a training data set. Further, attribute visual primitive features and object visual primitive features are calculated based on the attribute semantic features and object semantic features. Images with the same attribute semantic features and object semantic features are integrated and classified into a class, and attribute visual primitive features and object visual primitive features of all images in this class are calculated to obtain fine-grained attribute prototypes corresponding to the attribute semantic features and fine-grained object prototypes corresponding to the object semantic features.

[0078] The purpose of creating the fine-grained attribute prototypes or fine-grained object prototypes is to refine the visual primitive features corresponding to the attribute semantic features or object semantic features, and the fine-grained attribute prototypes and fine-grained object prototypes can be used to perform fine-grained classification on the images to be classified.

[0079] In addition, in the embodiments of the present application, the attribute visual primitive features, object visual primitive features, fine-grained attribute prototypes, and fine-grained object prototypes extracted from the training data set can be further strengthened, which can improve the intra-class compactness of fine-grained attribute prototypes (or fine-grained object prototypes) belonging to the same attribute (or object) category and the inter-class difference between fine-grained attribute prototypes (or fine-grained object prototypes) of different attribute (or object) categories.

[0080] Further, the attribute visual primitive features, object visual primitive features, fine-grained attribute prototypes, and fine-grained object prototypes can be used as classification basis of an image classifier, and based on this, the images to be classified can be classified. The attribute visual primitive features to be classified can be matched or calculated with the fine-grained attribute prototypes, and the object visual primitive features to be classified can be matched or calculated with the fine-grained object prototypes to obtain a classification result corresponding to the image to be classified. Because the fine-grained attribute prototypes and fine-grained object prototypes are refined in visual primitive features, the images to be classified are further refined when classified.

[0081] In the embodiment of the present application, the coarse-grained classifier based on the coarse-grained attribute prototype and the coarse-grained object prototype can be used simultaneously on the basis of the fine-grained classification using the fine-grained attribute prototype and the fine-grained object prototype. The coarse-grained attribute prototype is calculated from the attribute visual element features of images with the same attribute semantic features, and the coarse-grained object prototype is calculated from the object visual element features of images with the same object semantic features.

[0082] In the embodiment of the present application, the coarse-grained classification and the fine-grained classification can act together. The result of the coarse-grained classification can be used to verify the result of the fine-grained classification. Alternatively, the two classification results of the coarse-grained classification and the fine-grained classification can be combined based on a certain calculation rule to obtain the target classification result corresponding to the image to be classified.

[0083] The image classifier of the present application includes the fine-grained attribute prototype and the fine-grained object prototype. Each fine-grained attribute prototype has an attribute semantic feature corresponding thereto, and the fine-grained attribute prototype can be calculated from the attribute visual element features extracted from all image samples combined with the attribute semantic feature and the same object semantic feature. Since one attribute semantic feature can be combined with multiple object semantic features, the attribute visual element features of each combination can be calculated to obtain a corresponding fine-grained attribute prototype. Based on this, each attribute semantic feature or object semantic feature has multiple fine-grained attribute prototypes or fine-grained object prototypes corresponding thereto, which improves the visual diversity of each attribute semantic feature and object semantic feature. Based on this, the image is classified, which is beneficial to improve the accuracy of image classification.

[0084] Next, the embodiment of the present application will further introduce the image classification method.

[0085] Before classifying the image, an image classifier needs to be prepared in advance for classifying the image to be classified. The specific process of preparing the image classifier can refer to Figure 2 A flowchart for preparing an image classifier is shown in the embodiment of the present application. The specific process can include:

[0086] In step S131, the training data set and the attribute semantic features, object semantic features and feature maps of the image samples in the training data set are obtained.

[0087] The training data set includes a plurality of image samples, each of the image samples includes an attribute label and an object label, the attribute label and the object label are in a form representing attribute semantic features and object semantic features, and the attribute label and the object label can be subjected to semantic extraction processing to obtain the attribute semantic features and the object semantic features. In order to facilitate digital processing of the image samples, a feature map can be extracted from the image samples.

[0088] In the embodiments of the present application, the training data set can be obtained from a public combined zero-shot classification data set, such as MIT-States, UT-Zappos and C-GQA. The MIT-States data set includes 245 object classes, 115 attribute classes and 53K images, and includes various object labels (for example, island, camera, cat) and attribute labels (for example, brown, scratched, foggy). The word2vec can be used to initialize the attribute label and the object label to obtain semantic vectors corresponding to the attribute label and the object label respectively. The attribute semantic features corresponding to the attribute label and the object semantic features corresponding to the object label can be obtained by processing the semantic vectors through a semantic encoder. In addition, the image samples can be input to a visual encoder for processing to obtain a feature map of the image samples, or ResNet18 can be used to extract the feature map of the image samples.

[0089] In the embodiments of the present application, the training data set can be There are N image samples, and the image sample x i has a semantic label which is a combination of an attribute label and an object label, that is wherein represents the attribute label, represents the object label, and represent an attribute label set and an object label set respectively. The attribute semantic features of the image sample x i are and the object semantic features are In the training process, the label space includes a visible label combination and an unseen label combination The visible label combination is a label combination that appears in the training data set, and the unseen label combination is a label combination that does not appear in the training data set.

[0090] In step S132, based on the similarity of the attribute semantic features and the object semantic features, image mixing is performed to obtain a new image sample.

[0091] Considering that similar attributes are likely to modify the same object, or the same attributes are likely to modify similar objects, two images with high semantic similarity can be selected to synthesize a new image, thus reasonably expanding the image samples in the training dataset. Furthermore, this approach avoids situations where the synthesized image has semantically unreasonable characteristics.

[0092] For example, the combination of attribute labels and object labels for existing image samples is y1 = (ripe, apple) and y2 = (green, pencil), etc., if we do not calculate... and The semantic similarity between the corresponding semantic features of objects is directly cross-synthesized as y. j = (mature, pencil), which is clearly semantically unreasonable. To avoid this, we can first calculate the semantic features of all objects in the training dataset and their corresponding object labels. semantic features The semantic similarity between them indicates that objects with high semantic similarity are more likely to be associated with each other. For example, for fruit-type object tags, the attribute tags will be... With the Combine the labels of fruit-type objects to get y n = (ripe orange), y n-1 =Semantically reasonable labels for new image samples such as (ripe, banana).

[0093] In this embodiment, the semantic similarity between attribute semantic features or object semantic features can be calculated, and target images can be selected from the image samples for mixing based on this similarity. In this embodiment, this semantic similarity is used as the sampling probability for combination. The higher the sampling probability, the more similar the features are, the higher the probability of being selected as similar semantic features, and the higher the probability of combining images labeled with that semantic feature. If the combined semantics are unreasonable or do not exist, the sampling probability of the combination can be set to zero and it will not be selected.

[0094] Specifically, using any image x i For example, the image x in the image sample j The probability of being selected is:

[0095]

[0096] Where com_sim(.) represents cosine similarity; These are the images x i Similarly, the semantic features of attributes and the semantic features of objects, These are the images x j The semantic features of attributes and the semantic features of objects, where d is the dimension of the semantic features.

[0097] Further, after obtaining the semantic features with a similarity reaching a certain preset value, images corresponding to the two semantic features with high similarity are extracted, linear interpolation is performed on the images by using mixup, and a new image sample is synthesized at a feature level. Combinations of attribute labels and object labels of the new image sample can be summarized in the above unseen label combinations.

[0098] In step S133, the feature map, attribute semantic feature and object semantic feature of the new image sample are extracted.

[0099] The new image sample is synthesized based on the image sample and can be calculated based on the feature map of the image sample. The definition of the feature map of the new image sample is as follows:

[0100]

[0101] wherein is the feature map of the image sample, which can be extracted by a pre-trained ResNet18, C, H and W are respectively the channel number, height and width. λ is an interpolation coefficient, and the value range is [0, 1], which is sampled from a beta distribution. Similarly, the attribute semantic feature and the object semantic feature of the new image sample can be obtained by using the interpolation method:

[0102]

[0103]

[0104] The semantic feature of the combined label obtained by combining the attribute label and the object label can also be obtained by a simple splicing operation. The combined label is a two-by-two combination of the attribute label and the object label, and the set of the combined label is as follows:

[0105]

[0106] In step S134, the feature map, the attribute semantic feature and the object semantic feature of all the image samples and the new image samples are processed by using the attention weight of semantic guidance to obtain the attribute visual primitive feature and the object visual primitive feature.

[0107] The visual primitive feature is used to represent the visual content or visual area in which a certain attribute semantic feature or object semantic feature is embodied on an image. In the embodiments of the present application, the attribute semantic feature and the object semantic feature can be used to locate the most similar visual area in the image, and a greater attention weight is given to the visual area, so as to remove irrelevant visual features and noise and achieve the purpose of highlighting the attribute feature and the object feature.

[0108] For example, Figure 3The attention mechanism module provided in the embodiments of the present application is shown in the visual schematic diagram. The left image is the original image, which is represented as "sleeping kitten". The "kitten" is the visual feature corresponding to the object label. The attention mechanism is used to adjust the attention weight of each visual area of the whole image, and different colors are used to distinguish each visual area. The right image is the processed image. The kitten in the image is highlighted in black after attention adjustment. When identifying and classifying the image, the visual feature can be used to identify that the object label corresponding to the image is "kitten". However, in actual application, the image is not highlighted in black and white, but can be processed in multiple colors and multiple layers.

[0109] On the basis of mapping the attribute semantic features and the object semantic features to the visual space, the step of calculating the attribute visual primitive features and the object visual primitive features by using the attention weight can include: calculating all the attribute semantic features and the feature map to obtain the attribute similarity between the feature map and each attribute semantic feature; and performing convolution calculation and integration on the attribute similarity to obtain the attribute visual primitive features corresponding to the attribute semantic features of the image sample or the new image sample. All the object semantic features and the feature map are calculated to obtain the object similarity between the feature map and each object semantic feature; and the object similarity is convolved and integrated to obtain the object visual primitive features corresponding to the object semantic features of the image sample or the new image sample.

[0110] In the embodiments of the present application, two semantic-guided attention modules are designed to calculate the visual primitive features of the image. The semantic-guided attention module includes a semantic-guided spatial attention module and a channel-level attention module. As shown in the following formula: Figure 4 The structure schematic diagram of the semantic-guided attention mechanism module provided in the embodiments of the present application is shown. Taking the calculation of the attribute visual primitive features as an example, the semantic-guided attention mechanism module inputs the set of all attribute semantic features and the feature map X i , wherein K a is the number of all attribute labels in the training data set.

[0111] For the semantic-guided spatial attention branch, the attribute semantic features need to be first mapped to the visual space through linear transformation Φ(·), and then the attribute similarity with the feature map is calculated by matrix multiplication. Then, the activation mask corresponding to all attribute semantic features is calculated by using the activation function and the reshaping function. M=Re(σ(Φ(E a )·Re(X i). Next, the activation mask M of all attribute semantic features is integrated by a convolution operation to obtain the final spatial attention activation map The spatial attention activation map is the attention weight of the attribute semantic feature corresponding to the visual region of the image sample or the new image sample. To ensure that each activation mask can correctly identify the corresponding attribute, we can also obtain the attribute prediction through the convolution layer, global average pooling and Softmax activation function and is constrained by a cross-entropy function.

[0112] For the channel-level attention branch, the input feature map first needs to pass through a global average pooling layer (GAP), and then pass through two fully connected layers to obtain the channel-level attention activation map

[0113] The spatial attention activation map, the channel-level attention activation map and the feature map are multiplied element by element and added, and the added result is processed by a GAP layer to obtain the attribute visual primitive feature The calculation formula is as follows:

[0114]

[0115] Where, E p (·) represents an extension function that can extend the attention activation map to the same dimension as X i .

[0116] Similarly, the process of obtaining the object visual primitive feature also needs to pass through the semantic-guided spatial attention and the channel-level attention two branches to perform the same step calculation. Only the input of all attribute semantic feature sets is replaced by all object semantic feature sets, and the same calculation is performed on the all object semantic feature sets to obtain the object similarity, and further calculation can obtain the object visual primitive feature

[0117] The above method of calculating attribute visual primitive features and object visual primitive features is also applicable to the image to be classified. The feature map of the image to be classified is calculated with the attribute semantic features and object semantic features obtained from the training data set. Based on the similarity calculation result, further convolution, integration and other calculations are performed to obtain the classification attribute visual primitive feature and the classification object visual primitive feature corresponding to the image to be classified.

[0118] Step S135, based on the pre-set classifier structure, the attribute visual primitive feature, and the object visual primitive feature, training is performed to obtain an image classifier.

[0119] The image classifier comprises a fine-grained primitive classifier and a coarse-grained primitive classifier, wherein the fine-grained primitive classifier needs to create a fine-grained prototype based on the attribute visual primitive features and the object visual primitive features, so as to further create a fine-grained primitive classifier; the coarse-grained primitive classifier can be directly based on the attribute visual primitive features and the object visual primitive features, and generate a coarse-grained primitive classifier based on a preset classifier structure.

[0120] The process of making the fine-grained primitive classifier can comprise: based on the attribute semantic features and the object semantic features of the image samples and the new image samples, performing fine-grained division on the image samples and the new image samples to obtain a plurality of fine-grained categories, each of the fine-grained categories comprising at least one of the image samples or the new image samples, and the attribute semantic features and the object semantic features of the image samples or the new image samples in the same fine-grained category being the same; calculating the attribute visual primitive features of the image samples or the new image samples included in each of the fine-grained categories to obtain a fine-grained attribute prototype of each of the fine-grained categories; calculating the object visual primitive features of the image samples or the new image samples included in each of the fine-grained categories to obtain a fine-grained object prototype of each of the fine-grained categories; initializing a preset classifier structure by using the fine-grained attribute prototype and the fine-grained object prototype to obtain a fine-grained primitive classifier to be trained; training the fine-grained primitive classifier to be trained based on the image samples to obtain a fine-grained primitive classifier, wherein the training process is based on the attribute visual primitive features and the object visual primitive features of the image samples to train, so that the fine-grained primitive classifier calculates a prediction score between the fine-grained attribute prototype and the fine-grained object prototype based on the attribute visual primitive features and the object visual primitive features respectively, and calculates and outputs a fine-grained prediction score based on the prediction score.

[0121] Wherein, the making of the fine-grained attribute prototype and the fine-grained object prototype can first classify the image samples and the new image samples, which are hereinafter referred to as images, determine the images including the same attribute semantic feature and the same object semantic feature as the attribute fine-grained category corresponding to the attribute semantic feature and the object fine-grained category corresponding to the object semantic feature, that is, the combination of the attribute semantic feature and the object semantic feature of the images included in one fine-grained category is the same, and the images at least include one, and the images can be different in vision. Extract the respective attribute visual primitive features from all the images of one attribute fine-grained category, and extract the respective object visual primitive features from all the images of one object fine-grained category; calculate all the attribute visual primitive features and the object visual primitive features to obtain the fine-grained attribute prototype corresponding to the attribute semantic feature, and calculate the object visual primitive features to obtain the fine-grained object prototype corresponding to the object semantic feature.

[0122] Similarly, assuming that the corresponding attribute semantic feature in the label set is "ripe", the object semantic feature is "apple", and the corresponding reasonable semantic combination is "ripe apple", then the images with the semantic "ripe apple" can be classified into one attribute fine-grained category corresponding to "ripe" and one object fine-grained category corresponding to "apple" from the new image samples and the image samples. Based on this, the attribute visual primitive features and the object visual primitive features of all "ripe apple" images are extracted, the attribute visual primitive features are calculated to obtain the fine-grained attribute prototype corresponding to "ripe (apple)", and the object visual primitive features are calculated to obtain the fine-grained object prototype corresponding to "apple (ripe)". Further, the fine-grained attribute prototypes and the fine-grained object prototypes corresponding to "ripe banana", "sliced apple", "sliced banana", etc. can also be calculated.

[0123] Because the visual effects of the same attribute on different objects are different, the visual primitive features of the images with the same attribute combined with different objects are different, the fine-grained attribute prototypes obtained for the same attribute semantic feature are also different, and multiple fine-grained attribute prototypes for the same attribute semantic feature and multiple fine-grained object prototypes for the same object semantic feature are obtained, thereby increasing the diversity of each attribute semantic feature and object semantic feature in vision.

[0124] In the embodiments of the present application, in order to classify the to-be-classified image based on the fine-grained element classifier, the fine-grained attribute prototype and the fine-grained object prototype need to be calculated. Based on the above, all the attribute and object combination labels can be divided into unseen combinations and visible combinations. The visible combination can be understood as the original image sample in the training data set, and the unseen combination can be understood as the new image sample. The fine-grained prototypes in the unseen combination and the fine-grained prototypes in the visible combination are calculated in two different ways. Taking the fine-grained attribute prototype as an example, the fine-grained attribute prototype is the attribute visual element feature extracted from the image sample or the new image sample in a specific attribute-object combination, wherein the attribute-object combination in the image sample is a visible combination, and the attribute-object combination of the new image sample is an unseen combination.

[0125] For the fine-grained attribute prototype in the visible combination, the attribute visual element feature belonging to the same attribute fine-grained category in the visible combination can be directly averaged to obtain the fine-grained attribute prototype. For details, refer to the following method:

[0126]

[0127] wherein, represents a set of attribute visual element features corresponding to the image with the label of the kth attribute-object combination, is the size of the set.

[0128] The image in the unseen combination is the new image sample, and the fine-grained attribute prototype of the unseen combination can be estimated according to the visual element feature of the new image sample. The semantic association between the attribute semantic feature and the object semantic feature is used, and because the combination label of the new image sample is a weighted form of multiple combination labels, an attention mechanism is used to construct the fine-grained attribute prototype belonging to the attribute fine-grained category in the unseen combination according to the similarity of the semantics:

[0129]

[0130] wherein, represents a set of attribute visual element features corresponding to the new image sample containing the kth attribute-object combination; represents the similarity between the semantics of the kth combination and the semantics of the attribute-object combination corresponding to the ith new image sample, which can reflect the contribution degree of the attribute visual element feature of the ith new image sample to the fine-grained attribute prototype corresponding to the unseen combination k; W a is a parameter matrix.

[0131] In the embodiments of the present application, in order to stabilize the training process of the fine-grained base element classifier, the fine-grained attribute prototype and the fine-grained object prototype can be calculated continuously for each observed mini-batch by using a moving average strategy, and the momentum coefficient can be set to 0.7. Similarly, based on the above steps, the fine-grained object prototype can be calculated As follows:

[0132]

[0133]

[0134] After obtaining the fine-grained attribute prototype and the fine-grained object prototype, the classifier can be initialized based on the fine-grained attribute prototype and the fine-grained object prototype to obtain a to-be-trained fine-grained base element classifier. Then, the attribute visual base element features and the object visual base element features of the image sample are used to train the to-be-trained fine-grained base element classifier to obtain a fine-grained base element classifier. In the training process, the attribute visual base element features and the object visual base element features of the image sample are input into the to-be-trained fine-grained base element classifier, the attribute visual base element features and the fine-grained attribute prototype are calculated to obtain a combined prediction score of the attribute direction, and similarly, a combined prediction score of the object direction can be obtained. The two combined prediction scores are calculated to obtain a fine-grained prediction score. In the training process, the fine-grained base element classifier can be adjusted in parameters according to the accuracy of the prediction score, so that the fine-grained base element classifier can reach an optimal prediction state, and the accuracy of the fine-grained base element classifier can be improved.

[0135] In addition, in order to improve the coarse-grained element classifier and the fine-grained element classifier, especially the intra-class compactness and inter-class difference between each fine-grained prototype and visual element feature, all attribute semantic features, object semantic features, attribute visual element features, object visual element features, fine-grained attribute prototypes and fine-grained object prototypes can be mapped into a contrast space, and then all attribute semantic features, object semantic features, attribute visual element features, object visual element features, fine-grained attribute prototypes and fine-grained object prototypes in the contrast space are constrained by using a loss function to obtain reinforced attribute visual element features, object visual element features, fine-grained attribute prototypes and fine-grained object prototypes. The fine-grained attribute prototypes and fine-grained object prototypes are used to initialize a preset classifier structure to obtain a fine-grained element classifier to be trained. The fine-grained element classifier is trained based on the image sample to obtain a fine-grained element classifier. The training process is based on the attribute visual element features and the object visual element features of the image sample to train, so that the fine-grained element classifier calculates a prediction score between the reinforced fine-grained attribute prototypes and fine-grained object prototypes according to the attribute visual element features and the object visual element features respectively, and calculates and outputs a fine-grained prediction score based on the prediction score.

[0136] Specifically, the embodiments of the present application refer to a multi-modal contrast loss, a fine-grained prototype loss, and a coarse-grained prototype loss, and further include a classification loss. The classification loss is introduced into an image classifier to constrain the classifier. In the process of training the image classifier, the classifier is optimized by minimizing the loss function. The specific minimization method is as follows:

[0137]

[0138] Wherein, γ is a hyperparameter; and are the classification loss, the multi-modal contrast loss, the fine-grained prototype loss and the coarse-grained prototype loss, respectively.

[0139] The classification loss optimizes the element classifier by using a cross-entropy function CE(·). The specific calculation function is as follows:

[0140]

[0141] The multi-modal contrast loss The gap between the semantic features and the visual primitive features can be narrowed, the attribute visual primitive features and the object visual primitive features can be made more discriminative, and the multi-modal contrast loss can include a loss of visual primitive features to semantic features and a loss of semantic features to visual primitive features and semantic features to visual primitive features The following are The specific calculation formula is:

[0142]

[0143]

[0144]

[0145] wherein The cos_sim(·) function represents a cosine similarity calculation function. n represents the size of the training set. a (n o ) represents the number of samples in the training set whose attribute (object) label of the picture is the same as the attribute (object) label of the current image x i . is the semantic feature of the kth attribute label, is the semantic feature of the kth object label.

[0146] The coarse-grained prototype loss can ensure that the fine-grained attribute prototypes or the fine-grained object prototypes of the same attribute semantic feature or object semantic feature have certain commonality, and in the above contrast space, the similarity between the fine-grained attribute prototypes belonging to the same attribute semantic feature and the diversity between the fine-grained attribute prototypes corresponding to different attribute semantic features can be constrained, and the fine-grained object prototypes are the same. Thus, the commonality between the visual primitive features in the fine-grained attribute prototype or the fine-grained object prototype can be ensured, and the diversity between the fine-grained attribute prototypes or the fine-grained object prototypes is not destroyed. The following is a calculation formula of the coarse-grained prototype loss provided by the embodiments of the present application:

[0147]

[0148]

[0149] wherein, represents the semantic features of the attribute and the object corresponding to the kth combination in the image set.

[0150] In the embodiments of the present application, all the fine-grained attribute prototypes and fine-grained object prototypes obtained after the above reinforcement are used to initialize and train a preset classifier structure to obtain a fine-grained element classifier. When processing the image to be classified, the fine-grained element classifier needs to calculate the prediction scores between the attribute visual element features to be classified and all the fine-grained attribute prototypes in the fine-grained element classifier, and the prediction scores between the object visual element features to be classified and all the fine-grained object prototypes, and then select one highest prediction score corresponding to the fine-grained attribute prototype and the fine-grained object prototype respectively, and finally add the two highest prediction scores to obtain a fine-grained prediction score.

[0151] Similarly, all the attribute visual element features and object visual element features obtained after the above reinforcement can be used to initialize and train a preset classifier to obtain a coarse-grained element classifier which can classify the attribute visual element features to be classified and the object visual element features to be classified in the image to be classified respectively.

[0152] When the coarse-grained element classifier processes the image to be classified, the prediction scores between the attribute visual element features to be classified and each coarse-grained attribute prototype in the coarse-grained element classifier can be calculated, and the prediction scores between the object visual element features to be classified and each coarse-grained object prototype in the coarse-grained element classifier can be calculated, the maximum prediction scores are selected in the two calculation channels of the coarse-grained attribute prototype and the coarse-grained object prototype respectively, and the two maximum prediction scores are calculated according to a preset weight distribution to obtain a coarse-grained prediction score.

[0153] After obtaining the coarse-grained prediction score and the fine-grained prediction score, the fine-grained prediction score and the coarse-grained prediction score need to be calculated according to a preset weight distribution to obtain a target prediction score of the image to be classified, and based on the target prediction score, a corresponding combined semantic feature of the attribute semantic feature and the object semantic feature can be determined, and the attribute label and the object label corresponding to the attribute semantic feature or the object semantic feature can be combined together to obtain a target classification result.

[0154] In the embodiments of the present application, the target prediction score can reflect the corresponding attribute prediction score and object prediction score, and each prediction score can be traced back to the attribute label and the object label corresponding thereto according to the calculation process, so that the attribute label and the object label corresponding to the image to be classified can be determined, and a target combined label can be obtained.

[0155] Referring to Figure 5 , Figure 6This example illustrates a system structure diagram of an image classification method provided in an embodiment of this application. For instance... Figure 5 As shown, this is the semantically guided sample synthesis module, responsible for the image mixing described above. As shown in the figure, y s The visible label set represents the collection of labels in the training dataset, including attribute labels and object labels. Each label has its own corresponding attribute semantic features and object semantic features. All attribute and object semantic features are embedded into an image fusion mechanism. By calculating the similarity between attribute and object semantic features, images with high semantic similarity are selected and synthesized into new image samples using a linear interpolation method. The original image samples and the new image samples are then processed to extract feature maps, which, along with the semantic features, are input into attention mechanism modules guided by attributes and objects, respectively, for further processing.

[0156] like Figure 6 As shown, the attribute-guided attention mechanism module and the object-guided attention mechanism module process all the feature maps respectively to obtain attribute visual primitive features corresponding to each attribute label and object visual primitive features corresponding to each object label. Based on the attribute visual primitive features and the object visual primitive features, fine-grained attribute prototypes and fine-grained object prototypes are further created. Based on the fine-grained attribute prototypes and fine-grained object prototypes, fine-grained and coarse-grained attribute classifiers and object classifiers are created respectively.

[0157] In addition, Figure 6 This can be intuitively represented as follows: the fine-grained attribute prototypes and fine-grained object prototypes are embedded in a common cross-modal space. The loss function needs to constrain the visual primitive features and fine-grained prototypes in the space to improve the discrimination ability between each element, ensure the diversity between each prototype and visual primitive features, and preserve the commonality between each prototype and visual primitive features, which is conducive to improving the accuracy of image classification.

[0158] The image classification apparatus provided in the embodiments of this application is described below. The image classification apparatus described below can be referred to in correspondence with the image classification method described above.

[0159] Combination Figure 7 The image classification device may include:

[0160] Image information acquisition unit 100 is used to acquire images to be classified;

[0161] The primitive feature calculation unit 200 is configured to perform semantic-guided attention weight processing on the image to be classified to obtain attribute visual primitive features and object visual primitive features of the image to be classified, and the semantic-guided attention weight processing is based on attribute semantic features and object semantic features extracted from the training data set, and the attribute visual primitive features and the object features are obtained by performing calculation on feature maps of the image to be classified.

[0162] The image classification unit 300 is configured to input the attribute visual primitive features and the object visual primitive features of the image to be classified into an image classifier to perform classification processing, so as to obtain a target classification result corresponding to the image to be classified, and the image classifier is trained by image samples in the training data set, attribute semantic features and object semantic features corresponding to the image samples, and fine-grained attribute prototypes and fine-grained object prototypes, so that the image classifier is capable of classifying attribute visual primitive features and object visual primitive features of an image, wherein each fine-grained attribute prototype is calculated by attribute visual primitive features extracted from image samples combined with corresponding attribute semantic features and object semantic features, and the fine-grained object prototype is calculated by object visual primitive features extracted from image samples combined with corresponding object semantic features and attribute semantic features.

[0163] The image classifier includes fine-grained attribute prototypes and fine-grained object prototypes. Taking the fine-grained attribute prototypes as an example, each fine-grained attribute prototype has corresponding attribute semantic features, and the fine-grained attribute prototype can be calculated by attribute visual primitive features extracted from all image samples combined with the attribute semantic features and the same object semantic features. Since one attribute semantic feature can be combined with multiple object semantic features, a corresponding fine-grained attribute prototype can be obtained by calculating attribute visual primitive features embodied by each combination. Based on this, each attribute semantic feature or object semantic feature has multiple corresponding fine-grained attribute prototypes or fine-grained object prototypes, which improves the visual diversity of each attribute semantic feature and object semantic feature, and based on this, the image is classified, which is beneficial to improving the accuracy of image classification.

[0164] Optionally, the apparatus further includes:

[0165] The training data acquisition unit is configured to acquire a training data set, and the training data set includes a plurality of image samples, and each image sample includes an attribute label and an object label.

[0166] The feature map extraction unit is configured to extract a feature map from each image sample.

[0167] The semantic feature extraction unit is configured to perform semantic extraction processing on the attribute label and the object label to obtain attribute semantic features and object semantic features.

[0168] The sample synthesis unit is configured to select a target image from the image samples based on semantic similarity of the attribute semantic features and the object semantic features, and perform image mixing to obtain new image samples.

[0169] The information extraction unit is configured to extract feature maps, attribute semantic features and object semantic features of the new image samples.

[0170] The visual primitive feature acquisition unit is configured to perform semantic-guided attention weight processing on the feature maps, the attribute semantic features and the object semantic features of all the image samples and the new image samples to obtain attribute visual primitive features and object visual primitive features corresponding to each of the image samples or the new image samples.

[0171] The classifier training unit is configured to train, based on a preset classifier structure, the attribute visual primitive features and the object visual primitive features to obtain an image classifier.

[0172] Optionally, the visual primitive feature acquisition unit comprises:

[0173] The attribute similarity calculation subunit is configured to calculate all the attribute semantic features and the feature maps to obtain attribute similarity between the feature maps and the attribute semantic features.

[0174] The attribute visual primitive feature acquisition subunit is configured to perform convolution calculation and integration on the attribute similarity and calculation with the feature maps to obtain attribute visual primitive features corresponding to the attribute semantic features of the image samples or the new image samples.

[0175] The object similarity calculation subunit is configured to calculate all the object semantic features and the feature maps to obtain object similarity between each feature map and each of the object semantic features.

[0176] The object visual primitive feature calculation subunit is configured to perform convolution calculation and integration on the object similarity and calculation with the feature maps to obtain object visual primitive features corresponding to the object semantic features of the image samples or the new image samples.

[0177] Optionally, the image classifier comprises a fine-grained primitive classifier, and the classifier training unit comprises:

[0178] a sample division subunit, configured to perform fine-grained division on the image samples and the new image samples based on the attribute semantic features and the object semantic features of the image samples and the new image samples, to obtain a plurality of fine-grained categories, each of the fine-grained categories including at least one of the image samples or the new image samples, and the image samples or the new image samples in a same fine-grained category having the same attribute semantic features and the same object semantic features;

[0179] an attribute prototype creation subunit, configured to calculate a fine-grained attribute prototype of each of the fine-grained categories according to attribute visual primitive features of the image samples or the new image samples included in the fine-grained category;

[0180] an object prototype creation subunit, configured to calculate a fine-grained object prototype of each of the fine-grained categories according to object visual primitive features of the image samples or the new image samples included in the fine-grained category;

[0181] a classifier initialization first subunit, configured to initialize a preset classifier structure by using the fine-grained attribute prototypes and the fine-grained object prototypes, to obtain a fine-grained primitive classifier to be trained;

[0182] a classifier training first subunit, configured to train the fine-grained primitive classifier to be trained based on the image samples, to obtain a fine-grained primitive classifier, the training being based on attribute visual primitive features and object visual primitive features of the image samples, so that the fine-grained primitive classifier calculates a prediction score between the attribute visual primitive features and the object visual primitive features and the fine-grained attribute prototypes and the fine-grained object prototypes respectively, and outputs a fine-grained prediction score based on the prediction score.

[0183] Optionally, the classifier training unit can further include:

[0184] a feature mapping subunit, configured to map all of the attribute visual primitive features, the object visual primitive features, the attribute semantic features, the object semantic features, the fine-grained attribute prototypes, and the fine-grained object prototypes to a contrast space;

[0185] a feature enhancement subunit, configured to constrain the attribute visual primitive features, the object visual primitive features, the attribute semantic features, and the object semantic features by using a loss function in the contrast space, to obtain enhanced attribute visual primitive features, enhanced object visual primitive features, enhanced fine-grained attribute prototypes, and enhanced fine-grained object prototypes, the loss function including at least a multi-modal contrast loss, a fine-grained prototype loss, and a coarse-grained prototype loss;

[0186] The classifier initialization second subunit is configured to initialize a preset classifier structure by using the reinforced fine-grained attribute prototype and the fine-grained object prototype, to obtain a fine-grained base element classifier to be trained.

[0187] The classifier training second subunit is configured to train the fine-grained base element classifier based on the fine-grained base element classifier to be trained and the image sample, to obtain a fine-grained base element classifier. The training process is based on the attribute visual base element features and the object visual base element features of the image sample, so that the fine-grained base element classifier calculates a prediction score between the reinforced fine-grained attribute prototype and the fine-grained object prototype according to the attribute visual base element features and the object visual base element features respectively, and calculates and outputs a fine-grained prediction score based on the prediction score.

[0188] Optionally, the apparatus further comprises:

[0189] The classifier constraint unit is configured to constrain the image classifier by using a classification loss function in the image classifier.

[0190] Optionally, the image classification unit 300 comprises:

[0191] The classifier first calculation subunit is configured to input the attribute visual base element features to be classified and the object visual base element features to be classified into the image classifier, and process the attribute visual base element features to be classified and the object visual base element features to be classified by a fine-grained base element classifier and a coarse-grained base element classifier respectively, to obtain a fine-grained prediction score output by the fine-grained base element classifier and a coarse-grained prediction score output by the coarse-grained base element classifier.

[0192] The classifier second calculation subunit is configured to calculate the fine-grained prediction score and the coarse-grained prediction score according to a preset weight, to obtain a target prediction score of the image to be classified.

[0193] The classification result output subunit is configured to determine a combined label based on the target prediction score, and take the combined label as a target classification result. The combined label is a combination of an attribute label and an object label.

[0194] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and are not intended to denote the presence of any such actual relationship or order. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0195] The various embodiments in the specification are described in progressive order with reference to each embodiment, each embodiment highlighting differences from other embodiments, and the same or similar elements in the various embodiments are described with reference to each other where possible.

[0196] The above description of disclosed embodiments provides enabling concepts for making or using the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image classification method, characterized by, The method comprises the following steps: obtaining an image to be classified; performing semantic-guided attention weight processing on the image to be classified to obtain attribute visual primitive features and object visual primitive features of the image to be classified, wherein the semantic-guided attention weight processing is based on attribute semantic features and object semantic features extracted from a training data set, and the attribute visual primitive features and the object visual primitive features are obtained by performing calculation on feature maps of the image to be classified; inputting the attribute visual primitive features and the object visual primitive features into an image classifier to perform classification processing, so as to obtain a target classification result corresponding to the image to be classified, wherein the image classifier is trained by image samples in the training data set, attribute semantic features and object semantic features corresponding to the image samples, and fine-grained attribute prototypes and fine-grained object prototypes, so that the image classifier classifies attribute visual primitive features and object visual primitive features of images, wherein each fine-grained attribute prototype is calculated based on attribute visual primitive features extracted from the image samples corresponding to the attribute semantic features and the object semantic features, and the fine-grained object prototype is calculated based on object visual primitive features extracted from the image samples corresponding to the object semantic features and the attribute semantic features.

2. The method of claim 1, wherein, The method further comprises the following steps: obtaining a training data set, wherein the training data set comprises a plurality of image samples, and each image sample comprises an attribute label and an object label; extracting feature maps from each image sample; performing semantic extraction processing on the attribute label and the object label to obtain attribute semantic features and object semantic features; selecting target images from the image samples based on semantic similarity of the attribute semantic features and the object semantic features, and performing image mixing to obtain new image samples; extracting feature maps, attribute semantic features and object semantic features of the new image samples; performing semantic-guided attention weight processing on the feature maps, the attribute semantic features and the object semantic features of all the image samples and the new image samples to obtain attribute visual primitive features and object visual primitive features corresponding to each image sample or new image sample; training an image classifier based on a preset classifier structure, the attribute visual primitive features and the object visual primitive features.

3. The method of claim 2, wherein, The semantic-guided attention weight processing on the feature maps, the attribute semantic features and the object semantic features of all the image samples and the new image samples to obtain attribute visual primitive features and object visual primitive features corresponding to each image sample or new image sample comprises the following steps: performing calculation on all the attribute semantic features and the feature maps to obtain attribute similarity between the feature maps and each attribute semantic feature; performing convolution calculation and integration on the attribute similarity and performing calculation on the feature maps to obtain attribute visual primitive features corresponding to the attribute semantic features of the image sample or the new image sample. The object similarity between the feature map and each of the object semantic features is obtained by performing calculation on all the object semantic features and the feature map; The object visual primitive features corresponding to the object semantic features of the image sample or the new image sample are obtained by performing convolution calculation and integration on the object similarity and performing calculation with the feature map.

4. The method of claim 2, wherein, The image classifier includes a fine-grained primitive classifier, wherein the image classifier is obtained by training based on the preset classifier structure, the attribute visual primitive features, and the object visual primitive features, including: The image sample and the new image sample are divided into a plurality of fine-grained categories based on the attribute semantic features and the object semantic features of the image sample and the new image sample, each of the fine-grained categories includes at least one of the image sample or the new image sample, and the attribute semantic features and the object semantic features of the image sample or the new image sample in the same fine-grained category are the same; The fine-grained attribute prototype of each of the fine-grained categories is obtained by performing calculation according to the attribute visual primitive features of the image sample or the new image sample included in each of the fine-grained categories; The fine-grained object prototype of each of the fine-grained categories is obtained by performing calculation according to the object visual primitive features of the image sample or the new image sample included in each of the fine-grained categories; The fine-grained primitive classifier to be trained is obtained by initializing the preset classifier structure using the fine-grained attribute prototype and the fine-grained object prototype; The fine-grained primitive classifier is obtained by training based on the fine-grained primitive classifier to be trained and the image sample, and the training process is based on the attribute visual primitive features and the object visual primitive features of the image sample to train, so that the fine-grained primitive classifier calculates the prediction score between the fine-grained attribute prototype and the fine-grained object prototype according to the attribute visual primitive features and the object visual primitive features respectively, and outputs the fine-grained prediction score based on the prediction score.

5. The method of claim 4, wherein, Further comprising: Mapping all the attribute visual primitive features, the object visual primitive features, the attribute semantic features, the object semantic features, the fine-grained attribute prototype, and the fine-grained object prototype to a contrast space; In the contrast space, the attribute visual primitive features, the object visual primitive features, the attribute semantic features, the object semantic features, the fine-grained attribute prototype, and the fine-grained object prototype are constrained by using a loss function to obtain reinforced attribute visual primitive features, object visual primitive features, fine-grained attribute prototype, and fine-grained object prototype, and the loss function at least includes a multi-modal contrast loss, a fine-grained prototype loss, and a coarse-grained prototype loss; The fine-grained primitive classifier to be trained is obtained by initializing the preset classifier structure using the reinforced fine-grained attribute prototype and the fine-grained object prototype. training based on the to-be-trained fine-grained classifier and the image sample, to obtain a fine-grained element classifier, the training process being trained based on the attribute visual element features and the object visual element features of the image sample, so that the fine-grained element classifier respectively calculates a prediction score between the reinforced fine-grained attribute prototype and fine-grained object prototype according to the attribute visual element features and the object visual element features, and outputs a fine-grained prediction score based on the prediction score.

6. The method of claim 2, wherein, Also comprising: In the image classifier, the image classifier is constrained by a classification loss function.

7. The method of claim 1, wherein, The inputting of the to-be-classified attribute visual element features and the to-be-classified object visual element features into the image classifier for classification processing to obtain a target classification result corresponding to the to-be-classified image comprises: The inputting of the to-be-classified attribute visual element features and the to-be-classified object visual element features into the image classifier for processing by the fine-grained element classifier and the coarse-grained element classifier respectively to obtain a fine-grained prediction score output by the fine-grained element classifier and a coarse-grained prediction score output by the coarse-grained element classifier; The fine-grained prediction score and the coarse-grained prediction score are calculated according to a preset weight to obtain a target prediction score of the to-be-classified image; A combined label is determined based on the target prediction score, and the combined label is taken as the target classification result, the combined label being a combination of an attribute label and an object label.

8. An image classification apparatus characterized by comprising: Comprising: An image information acquisition unit is configured to acquire a to-be-classified image; An element feature calculation unit is configured to perform semantic-guided attention weight processing on the to-be-classified image to obtain to-be-classified attribute visual element features and to-be-classified object visual element features of the to-be-classified image, the semantic-guided attention weight processing being a processing process of calculating attribute visual element features and object features based on attribute semantic features and object semantic features extracted from a training data set and a feature map of the to-be-classified image; An image classification unit is configured to input the to-be-classified attribute visual element features and the to-be-classified object visual element features into an image classifier for classification processing to obtain a target classification result corresponding to the to-be-classified image, the image classifier being trained by image samples in the training data set, attribute semantic features and object semantic features corresponding to the image samples, and fine-grained attribute prototypes and fine-grained object prototypes, so that the image classifier classifies attribute visual element features and object visual element features of an image, wherein each fine-grained attribute prototype is calculated from attribute visual element features extracted from image samples corresponding to combined attribute semantic features and object semantic features, and the fine-grained object prototype is calculated from object visual element features extracted from image samples corresponding to combined attribute semantic features and object semantic features.

9. The apparatus of claim 8, wherein, The device further comprises: A training data acquisition unit is configured to acquire a training data set, the training data set comprising a plurality of image samples, each image sample comprising an attribute label and an object label; The feature map extraction unit is configured to extract a feature map from each of the image samples; The semantic feature extraction unit is configured to perform semantic extraction processing on the attribute label and the object label to obtain attribute semantic features and object semantic features; The sample synthesis unit is configured to select a target image from the image samples based on semantic similarity of the attribute semantic features and the object semantic features, and perform image mixing to obtain new image samples; The information extraction unit is configured to extract the feature map, the attribute semantic features, and the object semantic features of the new image samples; The visual primitive feature acquisition unit is configured to perform semantic-guided attention weight processing on the feature map, the attribute semantic features, and the object semantic features of all the image samples and the new image samples to obtain attribute visual primitive features and object visual primitive features corresponding to each of the image samples or the new image samples; The classifier training unit is configured to train an image classifier based on a preset classifier structure, the attribute visual primitive features, and the object visual primitive features.

10. The apparatus of claim 9, wherein, The visual primitive feature acquisition unit comprises: The attribute similarity calculation subunit is configured to calculate all the attribute semantic features and the feature map to obtain attribute similarity between the feature map and the attribute semantic features; The attribute visual primitive feature acquisition subunit is configured to perform convolution calculation and integration on the attribute similarity and calculation with the feature map to obtain attribute visual primitive features corresponding to the attribute semantic features of the image samples or the new image samples; The object similarity calculation subunit is configured to calculate all the object semantic features and the feature map to obtain object similarity between each feature map and each of the object semantic features; The object visual primitive feature calculation subunit is configured to perform convolution calculation and integration on the object similarity and calculation with the feature map to obtain object visual primitive features corresponding to the object semantic features of the image samples or the new image samples.

Images