Combined zero sample image classification and model training method and device

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a combined zero sample image classification and model training method and device. The training method of the combined zero sample image classification model comprises the steps of obtaining an image classification data set, and dividing the image classification data set into a training set and a test set; a neural network model is constructed, the neural network model is trained based on the training set, the combined zero sample image classification model is generated, and the neural network model at least comprises a combined recognition branch and an element recognition branch. The combined recognition branch is used for obtaining combined feature representation of each sample based on global features of training set samples, the primitive recognition branch is used for obtaining decoupled primitive features by mining a global primitive relationship of the training samples so as to recognize primitives, and the primitives comprise attributes and objects. The present disclosure improves the performance of combined zero sample image classification.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a combined zero-sample image classification and model training method and device. Background Art

[0002] Combined zero-shot image classification aims to classify images into previously unseen attribute-object pair labels by learning visual primitives (i.e., attributes or objects) in images with known pair labels. For example, based on the learned images of "ripe apples" and "sliced ​​bread", the unlearned "sliced ​​apple" image can be successfully classified.

[0003] The key to combinatorial zero-shot image classification is to model attributes and objects separately so that they can be transferred to the unseen combination. To this end, visual disentanglement technology is needed to separate the representation of attributes and objects from the image of the visible combination. The current visual disentanglement technology extracts the disentangled primitive representation by calculating the similarity of a pair of images with the same primitives.

[0004] However, this method only explores local primitive relationships with prerequisites (i.e., when another primitive is the same). For complex combinations, it is difficult for the model to obtain a comprehensive primitive representation, which easily leads to classification errors. Summary of the invention

[0005] The disclosed embodiment proposes a combined zero-shot image classification scheme based on global primitive relationships to solve the problem that the current combined zero-shot image classification scheme has poor classification effect due to being based on local primitive relationships.

[0006] A first aspect of an embodiment of the present disclosure provides a training method for a combined zero-sample image classification model, comprising: Obtain an image classification dataset and divide it into a training set and a test set; Construct a neural network model, train the neural network model based on the training set, and generate the combined zero-sample image classification model, wherein the neural network model at least includes a combined recognition branch and a primitive recognition branch, the combined recognition branch is used to obtain the combined feature representation of each sample based on the global features of the training set samples, and the primitive recognition branch is used to obtain the decoupled primitive features by mining the global primitive relations of the training samples to identify the primitives, and the primitives include attributes and objects.

[0007] In some embodiments of the present disclosure, obtaining a combined feature representation of each sample based on the global features of the training set samples includes: Extract global features of training set samples; The feature representation of the global feature is enhanced based on self-attention to obtain a combined feature representation of each sample.

[0008] In some embodiments of the present disclosure, extracting global features of training set samples includes: The ViT model pre-trained in a self-supervised manner on the ImageNet dataset is used as the image encoder to extract global features of the training set samples that contain more semantic information and are more suitable for image classification tasks.

[0009] In some embodiments of the present disclosure, obtaining decoupled primitive features by mining global primitive relationships of training samples to identify primitives includes: Learn primitive experts on the entire training set; The primitive expert is used as a query, and the global features of the training set samples are used as keys and key values, and cross-attention processing is performed to obtain decoupled primitive feature representations for the training set samples.

[0010] In some embodiments of the present disclosure, performing cross-attention processing further includes: For each sample data, two auxiliary features are randomly selected in the training set, and the decoupled primitive features are constrained at the attention level based on the EMD distance of the two auxiliary features, wherein one of the two auxiliary features has the same attributes but different objects, and the other has the same object but different attributes.

[0011] In some embodiments of the present disclosure, the neural network model is also used to extract text features for all combinations and primitive categories in the training set, and obtain the classification results of the training samples by respectively calculating the cosine similarity between the visual features in the combination recognition branch and the primitive recognition branch and the text features of each category, and using cross entropy loss for constraints.

[0012] In some embodiments of the present disclosure, the method further includes: The combined zero-shot image classification model is tested based on the test set.

[0013] A second aspect of an embodiment of the present disclosure provides a training device for a combined zero-sample image classification model, comprising: An acquisition module is used to acquire an image classification data set and divide it into a training set and a test set; A training module is used to construct a neural network model, train the neural network model based on the training set, and generate the combined zero-sample image classification model, wherein the neural network model at least includes a combined recognition branch and a primitive recognition branch, the combined recognition branch is used to obtain the combined feature representation of each sample based on the global features of the training set samples, and the primitive recognition branch is used to obtain the decoupled primitive features by mining the global primitive relations of the training samples to identify the primitives, and the primitives include attributes and objects.

[0014] A third aspect of the embodiments of the present disclosure provides a combined zero-sample image classification method, including: Get image data; The image data is input into the combined zero-sample image classification model trained according to the method described in the first aspect of the embodiment of the present disclosure, and the cosine similarity between the corresponding visual features and the text features of each category is obtained in the combined recognition branch and the primitive recognition branch respectively, and the two are weightedly fused, and the category with the highest similarity is selected as the classification result of the image data and output.

[0015] A fourth aspect of the embodiments of the present disclosure provides a combined zero-sample image classification device, including: An acquisition module, used for acquiring image data; A classification module is used to input the image data into the combined zero-sample image classification model trained according to the method described in the first aspect of the embodiment of the present disclosure, obtain the cosine similarity of the corresponding visual features and the text features of each category in the combined recognition branch and the primitive recognition branch, perform weighted fusion on the two, select the category with the highest similarity, and output it as the classification result of the image data.

[0016] In summary, the training method and device of the combined zero-shot image classification model and the combined zero-shot image classification method and device provided in the embodiments of the present disclosure, by holistically modeling two learnable primitive experts (i.e., attribute expert and object expert) on the entire training set to mine global primitive relationships, thereby avoiding the problem of the prior art of only mining local primitive relationships for limited data with prerequisites. Therefore, a more comprehensive primitive representation can be obtained, thereby improving the performance of combined zero-shot image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings: Figure 1 It is an algorithm framework of a combined zero-shot image classification method based on global primitive relations proposed in the present disclosure; Figure 2 is a flowchart of a training method for a combined zero-shot image classification model according to some embodiments of the present disclosure; Figure 3 is a network architecture of a neural network model for training a combined zero-shot image classification model in training and testing phases according to some embodiments of the present disclosure; Figure 4 is a schematic diagram of a training device for a combined zero-sample image classification model according to some embodiments of the present disclosure; Figure 5 is a flowchart of a combined zero-sample image classification method according to some embodiments of the present disclosure; Figure 6 is a schematic diagram of a combined zero-sample image classification device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.

[0019] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.

[0020] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0021] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0022] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.

[0023] Humans are naturally capable of generalizing different learned concepts (e.g., objects and attributes) into new combinations. For example, with images of "ripe apple" and "sliced ​​bread" in mind, when an unknown image of "sliced ​​apple" appears, humans can effortlessly understand such a new composite concept. Similar to language-based computer vision tasks such as image captioning and navigation, recognizing new composite concepts is crucial for visual understanding when training samples are insufficient. With this in mind, combinatorial zero-shot learning, which aims to classify images into previously unseen attribute-object pair labels by learning visual primitives (i.e., attributes or objects) from images with known pair labels, has recently received increasing attention.

[0024] Since visual primitives appear in combinations of visible images, the key to combinatorial zero-shot image classification lies in how to model attributes and objects separately so as to transfer to unseen combinations. To address this issue, there is growing interest in visual disentanglement, which aims to separate the representation of attributes and objects from the images of visible combinations. Among them, some previous studies extract disentangled primitive representations by calculating the similarity of a pair of images with the same primitives. Their success largely relies on mining different attributes under the same object / different object relationships under the same attribute. However, these methods only explore local primitive relationships with prerequisites (i.e., when another primitive is the same). For complex combinations, it is difficult for the model to obtain a comprehensive primitive representation, which easily leads to classification errors.

[0025] In order to solve the above problems, the present disclosure proposes a combined zero-shot image classification model based on global primitive relations and performs combined zero-shot image classification based on this model. The combined zero-shot image classification method proposed in the present disclosure aims to explore two global primitive relations to better solve the combined zero-shot problem. In order to model the primitive relations as a whole, the present disclosure learns two experts for all attributes and all objects respectively. In addition, in order to avoid incomplete learning of relations for limited data (single or paired samples), the present disclosure learns experts at the level of the entire data set. Figure 1 It is an algorithm framework of a combined zero-sample image classification method based on global primitive relationships proposed in the present disclosure.

[0026] In some embodiments, the training method of the combined zero-shot image classification model is as follows: Figure 2 As shown, the specific steps include: S210, obtaining an image classification data set and dividing it into a training set and a test set.

[0027] First, obtain the image classification dataset.

[0028] In some embodiments of the present disclosure, the acquired image classification datasets include three benchmark datasets: UT-Zappos50K, Clothing16K, and C-GQA, which have 16 / 12, 9 / 8, and 413 / 674 attributes and objects, respectively. Specifically, UT-Zappos50K is a fine-grained dataset that contains different types of shoes (e.g., boots, sandals) with texture attributes (e.g., canvas, cotton). Clothing16K consists of different types of clothes (e.g., shirts, trousers) with color attributes (e.g., blue, yellow). C-GQA is created for VQA tasks based on the StanfordGQA dataset, and consists of common attributes (e.g., red, dirty) and objects (e.g., pens, windows) in daily life.

[0029] Secondly, the semantic categories are divided into known combination categories and unknown combination categories, and the training set and test set are divided accordingly (including narrow and broad test set division methods).

[0030] Some embodiments of the present disclosure divide UT-Zappos50K, Clothing16K, and C-GQA into 83, 18, and 5592 training set categories, respectively.

[0031] S220, constructing a neural network model, training the neural network model based on the training set, and generating the combined zero-sample image classification model, wherein the neural network model at least includes a combined recognition branch and a primitive recognition branch, the combined recognition branch is used to obtain a combined feature representation of each sample based on the global features of the training set samples, and the primitive recognition branch is used to obtain decoupled primitive features by mining the global primitive relationships of the training samples to identify primitives, and the primitives include attributes and objects.

[0032] The complete training process of the combined zero-shot image classification model includes two stages: training and testing.

[0033] In the training phase, the neural network model extracts text features for all combinations and primitive categories in the training set, and calculates the cosine similarity between the visual features in the combination recognition branch and the primitive recognition branch and the text features of each category, and uses cross entropy loss for constraints to obtain the classification results of the training samples. In the testing phase, the combined zero-shot image classification model based on global relationship mining is tested using a test set, and the classification results are output and the model performance is evaluated. In some embodiments of the present disclosure, the network architecture of the neural network model used to train the combined zero-shot image classification model in the training and testing phases is as follows: Figure 3 shown.

[0034] The specific steps of the training phase are as follows: 1. Obtaining combined feature representation based on global features and mining global primitive relationships Specifically, the combined recognition branch first uses the ViT model pre-trained in a self-supervised manner on the ImageNet dataset as an image encoder to extract global features of the training set samples that contain more semantic information and are more suitable for image classification tasks, which can be expressed as ,in is the sample index, is the number of training set samples, For the image samples, is the image encoder, For the The global features of all samples are then enhanced through the self-attention module to obtain the combined feature representation of each sample. , specifically: ;

[0035] Among them, Q is the query, K is the key, V is the key value, and the scaling factor for and The dimension of is the sample index, is the number of training set samples, For the The global features of the image samples.

[0036] The primitive recognition branch first learns two primitive experts (i.e., attribute experts) With object experts ) Mining global primitive relationships. and is randomly initialized, with have the same dimensions. Since a single sample contains only one attribute and one object, in order to avoid incomplete learning of relationships with limited data, and The learning of is across data batches, that is, on the entire dataset.

[0037] Then, the primitive expert is used as the query and the global feature of the training sample is used as the key and key value to input into the cross attention module at the same time, which is the training sample feature Get the decoupled attributes and object features respectively and , specifically: ;

[0038] Specifically, is the sample index, is the number of training set samples, For the The global features of image samples, is an attribute expert, Is an object expert. is the training sample feature Get the decoupled attribute feature representation, is the training sample feature Obtain the decoupled object feature representation. To ensure that the primitive experts produce similar features for samples with the same primitive, for each , the present invention randomly selects two auxiliary features in the data set: one has the same attributes but different objects, denoted as , and another object with the same attributes but different properties, denoted by Since EarthMover's Distance (EMD) is often used to evaluate the dissimilarity between two multidimensional distributions in feature space, this disclosure introduces EMD to constrain the decoupled primitive features at the attention level. To represent Therefore, the EMD regularization constraint It can be expressed as:

[0039] in and They are the training sample features Decoupled attributes and object features, and They are the training sample features Decoupled properties and object features.

[0040] 2. Use word embedding Word2vec to extract text features for all combinations and primitive categories in the training set ,in , as learnable text information. It should be noted that word embedding can directly extract each and , input through a linear layer and The connected embedding of the corresponding primitive combination can be obtained .

[0041] 3. Calculate the cosine similarity between the visual features and the text features of each category in the combination and primitive branches respectively. and Through two layers of MLP, the feature dimensions are adjusted to The same results are recorded as , At this time, all visual features that have been self-attention or cross-attention are collectively referred to as , its classification probability It can be calculated as:

[0042] in, is the natural exponential function, is the matrix multiplication operation, is the temperature coefficient, The collection of all label texts for combinations, attributes, and objects.

[0043] 4. Visual features Calculate the cross entropy loss function , and use Adam optimizer to optimize the entire model, the total loss function It can be expressed as: ;

[0044] in, , , are the cross entropy loss functions on the attribute branch, combination branch, and object branch, respectively. The collection of all label texts for combinations, attributes, and objects.

[0045] The specific steps of the testing phase are as follows: 1. Use the pre-trained image encoder to extract the global features of the test samples, and enhance the feature representation through the self-attention module to obtain the combined feature representation of each sample; and use the two learned primitive experts (i.e., attribute expert and object expert) and the global features of the test samples to simultaneously input the cross-attention module to obtain the decoupled primitive feature representation for the test samples.

[0046] 2. Use the learned primitive word embeddings to obtain text features for the combined categories in all test sets.

[0047] 3. Calculate the cosine similarity between the visual features and the text features of each category in the combination and primitive branches respectively to obtain the classification results of the test samples.

[0048] In particular, during the test phase, all parameters of the model are frozen, no loss function is calculated, and no image encoder parameters are updated. Figure 4is a schematic diagram of a training device for a combined zero-sample image classification model according to some embodiments of the present disclosure. Figure 4 As shown, the training device 400 of the combined zero-sample image classification model includes a division module 410 and a training module 420, wherein: A partitioning module 410 is used to obtain an image classification data set and partition it into a training set and a test set; The training module 420 is used to construct a neural network model, train the neural network model based on the training set, and generate the combined zero-sample image classification model, wherein the neural network model at least includes a combined recognition branch and a primitive recognition branch, the combined recognition branch is used to obtain the combined feature representation of each sample based on the global features of the training set samples, and the primitive recognition branch is used to obtain the decoupled primitive features by mining the global primitive relationships of the training samples to identify the primitives, and the primitives include attributes and objects.

[0049] Figure 5 FIG. 1 is a flowchart of a combined zero-sample image classification method according to some embodiments of the present disclosure. Figure 5 As shown, the specific steps include: S510, acquiring image data.

[0050] S520, input the image data into the Figure 2 The combined zero-sample image classification model trained by the method described in s210-s220 obtains the cosine similarity of the corresponding visual features and the text features of each category in the combined recognition branch and the primitive recognition branch, respectively, performs weighted fusion on the two, selects the category with the highest similarity, and outputs it as the classification result of the image data.

[0051] Figure 6 is a schematic diagram of a combined zero-sample image classification device according to some embodiments of the present disclosure. Figure 4 As shown, the combined zero-sample image classification model device 600 includes an acquisition module 610 and a classification module 620, wherein: An acquisition module, used for acquiring image data; A classification module is used to input the image data into a Figure 2 The combined zero-sample image classification model trained by the method described in s210-s220 obtains the cosine similarity of the corresponding visual features and the text features of each category in the combined recognition branch and the primitive recognition branch, respectively, performs weighted fusion on the two, selects the category with the highest similarity, and outputs it as the classification result of the image data.

[0052] In summary, the training method and device of the combined zero-shot image classification model and the combined zero-shot image classification method and device provided in the embodiments of the present disclosure, by holistically modeling two learnable primitive experts (i.e., attribute expert and object expert) on the entire training set to mine global primitive relationships, thereby avoiding the problem of the prior art of only mining local primitive relationships for limited data with prerequisites. Therefore, a more comprehensive primitive representation can be obtained, thereby improving the performance of combined zero-shot image classification.

[0053] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0054] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0055] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A training method for a combined zero-shot image classification model, characterized in that: include: Obtain an image classification dataset and divide it into a training set and a test set; Construct a neural network model, train the neural network model based on the training set, and generate the combined zero-sample image classification model, wherein the neural network model at least includes a combined recognition branch and a primitive recognition branch, the combined recognition branch is used to obtain the combined feature representation of each sample based on the global features of the training set samples, and the primitive recognition branch is used to obtain the decoupled primitive features by mining the global primitive relations of the training samples to identify the primitives, and the primitives include attributes and objects.

2. The method according to claim 1, characterized in that: The step of obtaining a combined feature representation of each sample based on the global features of the training set samples includes: Extract global features of training set samples; The feature representation of the global feature is enhanced based on self-attention to obtain a combined feature representation of each sample.

3. The method according to claim 2, characterized in that: The global features of the training set samples are extracted including: The ViT model pre-trained in a self-supervised manner on the ImageNet dataset is used as the image encoder to extract global features of the training set samples that contain more semantic information and are more suitable for image classification tasks.

4. The method according to claim 1, characterized in that: The step of obtaining decoupled primitive features by mining the global primitive relationship of the training samples to identify the primitives includes: Learn primitive experts on the entire training set; The primitive expert is used as a query, and the global features of the training set samples are used as keys and key values, and cross-attention processing is performed to obtain decoupled primitive feature representations for the training set samples.

5. The method according to claim 4, characterized in that: The cross-attention processing further comprises: For each sample data, two auxiliary features are randomly selected in the training set, and the decoupled primitive features are constrained at the attention level based on the EMD distance of the two auxiliary features, wherein one of the two auxiliary features has the same attributes but different objects, and the other has the same object but different attributes.

6. The method according to claim 1, characterized in that: The neural network model is also used to extract text features for all combinations and primitive categories in the training set, and obtain the classification results of the training samples by respectively calculating the cosine similarity between the visual features in the combination recognition branch and the primitive recognition branch and the text features of each category, and using cross entropy loss for constraints.

7. The method according to claim 1, characterized in that: The method further comprises: The combined zero-shot image classification model is tested based on the test set.

8. A training device for a combined zero-shot image classification model, characterized in that: include: An acquisition module is used to acquire an image classification data set and divide it into a training set and a test set; A training module is used to construct a neural network model, train the neural network model based on the training set, and generate the combined zero-sample image classification model, wherein the neural network model at least includes a combined recognition branch and a primitive recognition branch, the combined recognition branch is used to obtain the combined feature representation of each sample based on the global features of the training set samples, and the primitive recognition branch is used to obtain the decoupled primitive features by mining the global primitive relations of the training samples to identify the primitives, and the primitives include attributes and objects.

9. A combined zero-shot image classification method, characterized in that: include: Get image data; The image data is input into the combined zero-sample image classification model trained according to the method according to any one of claims 1 to 7, and the cosine similarities of the corresponding visual features and the text features of each category are obtained in the combined recognition branch and the primitive recognition branch respectively, and the two are weightedly fused, and the category with the highest similarity is selected as the classification result of the image data and output.

10. A combined zero-sample image classification device, characterized in that: include: An acquisition module, used for acquiring image data; A classification module is used to input the image data into the combined zero-sample image classification model trained according to the method according to any one of claims 1 to 7, obtain the cosine similarity of the corresponding visual features and the text features of each category in the combined recognition branch and the primitive recognition branch, perform weighted fusion on the two, select the category with the highest similarity, and output it as the classification result of the image data.

Images