A combined generalized image multi-label classification method based on self-evolving conditional representation

Abstract

The present invention provides a combined generalized image multi-label classification method based on self-evolving conditional representation, which uses the cross-fusion feature matrix of the predicted other concept meta-text prompt feature matrix and the original feature matrix of the input image as a condition, and performs feature fusion with the semantic and visual representation of the concept meta to be represented and evolved, thereby realizing the instant self-evolution of each concept meta-semantic and visual representation for different input images, which not only makes the learned semantic and visual representation more adaptable to different input images, but also solves the problem of the lack of modeling of the mutual relationship between concept metas in previous methods. The classification accuracy on multiple data sets has achieved outstanding progress. It can instantly self-evolve the visual and semantic representation of a certain concept meta based on the representation of other types of concept metas and the visual representation of the input image, making the learned representation more flexible and more generalizable to new combinations.

Description

Technical Field

[0001] The present invention relates to the field of multimodal visual language, and specifically proposes a self-evolving conditional semantic representation learning method to perform more flexible representation modeling of concept elements in a combination, and to construct a generalizable combined image multi-label zero-shot classification method and system. Background Art

[0002] Combinatorial generalization of multi-label image classification involves predicting the object category and multiple attribute categories for a given object in an image, thereby achieving combinatorial prediction. During model training, the model only sees a subset of combinations consisting of conceptual elements such as objects and multiple attributes. During the inference phase, the model is required to correctly predict other combinations consisting of the same conceptual elements, achieving combinatorial generalization. This requires the model to learn to correctly model the representational information of each conceptual element from the seen combinations, learn to flexibly combine these representations, and then correctly generalize to the recognition of unseen combinations. Combinatorial generalization is a widespread phenomenon in the real world. For example, the same objective entity can exhibit multiple objective attributes, significantly varying its appearance. The same objective attribute can also appear in multiple entities, making the attribute itself abstract and difficult to describe. Achieving a model's ability to handle combinatorial generalization is a key foundational research area for achieving artificial intelligence and a crucial foundational step in realizing machine intelligence. It holds enormous practical and commercial value and has attracted considerable attention in academia and industry in recent years.

[0003] Currently, there are two main approaches to combinatorial generalized image multi-label zero-shot classification. One relies on a multi-layer perceptron or graph neural network to directly model the image and semantic representations of the combined concept in a shared feature embedding space, and then calculates the similarity between the two to complete classification. However, the disadvantage of this approach is that it is difficult for the model to directly learn how to model the visual and semantic representations of the concept elements, making it difficult to identify new combinations of these concept elements during the testing phase, and the generalization performance for new combinations is poor. The other approach independently models the visual and semantic representations of all attributes, objects, and combinations, and calculates the similarity between the image and semantic representations in independent embedding spaces. Finally, a weighted sum of the similarities is calculated to complete classification. This approach overcomes the shortcomings of the first approach, enabling the model to learn how to model the visual and semantic representations of concept elements and combinations, and thus generalizes better to new combinations. Therefore, it is the current mainstream approach.

[0004] While numerous improvements based on the second approach can improve performance on combinatorial generalized image multi-label classification tasks, these approaches fail to consider the interdependencies between the representations of different concept elements and fail to model these interdependencies, resulting in inflexible learned concept element representations. For example, in a binary concept, for the object concept element "apple," the visual appearance of "intact apple" and "sliced ​​apple" are completely different. Therefore, for the two attribute concept elements "intact" and "sliced," the visual and semantic representations of the object concept element "apple" should change with the attribute concept element. Similarly, for the attribute concept element "wet," its visual appearance is completely different for "wet cat" and "wet umbrella." Therefore, the visual and semantic representations of the attribute concept element "wet" should also change with the object concept element. However, previous methods treat the visual and semantic representations of different concept elements as static representations, meaning that no single representation is affected by the representations of other concept elements during training. This makes it difficult to capture the dependencies between multiple concept elements and, consequently, to model sufficiently flexible combinatorial representations, severely impacting the model's generalization performance to new combinations. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention provides a combined generalized image multi-label classification method based on self-evolving conditional representation, which can instantly self-evolve the visual and semantic representations of a certain concept element according to the representations of other types of concept elements to which the input image belongs and the visual representation of the input image, making the learned representation more flexible and having stronger generalization performance for new combinations.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step 1: Randomly sample a batch of input images from the dataset, the batch contains b input images, and uniformly scale all input images in the batch to H×W size using the bicubic interpolation method, where H is the image height and W is the image width. Then, with a probability of 0.5, all input images in the batch are randomly horizontally flipped, randomly rotated within ±5°, and normalized with the average values ​​of the RGB three channels as μ1, μ2, and μ3, and the standard deviations as σ1, σ2, and σ3. All input images in the batch output after image scaling, random horizontal flipping, random rotation, and image RGB channel normalization are passed through the open source pre-trained visual language model B with all parameters frozen. * Visual Transformer Model in Extract features and obtain visual feature matrix Among them, * indicates that all parameters are frozen and not updated during training, n v represents the number of visual features, dv Indicates the number of channels for each feature in the visual feature, n v The first feature among the visual features represents the global feature of the input image;

[0008] Step 2: Given N concept element sets A i , i represents the serial number of all types of concept elements, i = 1, 2, ..., N, j represents each concept element set A i The concept element number in A i Contains m i Concept element a ij , A i ={a ij |j=1,2,...,m i}, all concept elements a ij Total composition Combinations, using set representation, all combinations consisting of N concept elements are represented as combination set C = {c k |k=1,2,...,n c}, where c k represents the kth combination concept in C, and k represents the sequence number of the combination concept in the combination set C;

[0009] Step 3: Input the visual feature matrix V into N+1 visual autoencoders, and perform N+1 visual autoencodings independently. Each visual autoencoding first passes through a single layer of self-attention and then passes through a multi-layer perceptron to map V to the concept metaspace where each concept metaset is located and the combination space where the combination set C is located;

[0010] Step 4: Construct two text prompt templates i and t c :t i Used to construct N kinds of concept meta-prompts, each concept meta-prompt template t i Can constitute m i concept meta-prompts, each concept meta-prompt template t i The text contains 1 concept meta-word placeholder; t c Used to construct combined concept prompts, t c Can constitute n c Combined concept prompts, each combined concept prompt template t c The text contains N concept meta-word placeholders, and the concept meta-word placeholders are all represented by the letter "x";

[0011] Step 5: Use B * Pre-trained word vector library in t i and t cEach word in is mapped into the corresponding word feature vector, which constitutes the concept meta-prompt feature matrix and combined concept hint feature matrix Then extract all concept meta-word feature vectors For the i-th concept element, all concept element word feature vectors of this concept element Replace T i The feature vectors of all “x” in the final result are N replaced concept element prompt feature matrices Among them, each T′ i Contains m i Concept meta-prompt features, n t is the maximum number of words in the prompt text sentence, d t is the number of channels of each word feature vector. For the k-th combined concept, the N concept meta-word feature vectors constituting the combined concept are substituted into T c The feature vector of the letter "x" is replaced in the final result, and the combined concept prompt feature matrix containing nc combined prompt features is obtained.

[0012] Step 6: All T' i and T′c 输 Enter B * Pre-trained text Transformer model in Output concept meta-semantic feature matrix and combined concept semantic feature matrix

[0013] Step 7: For all P i and P c The second dimension of the average pooling is performed to eliminate the second dimension and obtain and In V c and N V i Take the first element of the second dimension and we get and N, since d is not equal to d v , so B i and B c The last dimension of is mapped to d channels, that is, and In addition, S i and S c The last dimension of is mapped with the same channel number again, in, and It's B * 2 pre-trained LNs in, and It's B* The pre-trained mapping matrix with two different values ​​in ; Finally, calculate S′ c and B′ c The cosine similarity matrix of and N S′ ai and B′ ai The cosine similarity matrix of Where ||x||2 represents the calculation of the bi-norm of the matrix or vector x, and the pre-classification results of all N concept elements are N Among them, argmax means returning the maximum value in the last dimension. Represents the sequence numbers of all concept elements to which b input images belong;

[0014] Step 8: Calculate the semantic self-evolution condition representation of the i-th concept element through formula (7) to formula (9) Through formula (13) to formula (15), the visual self-evolution condition representation of the i-th concept element is calculated

[0015] Step 9: In C i Average the third dimension and eliminate C i The third dimension of By selecting J i The first feature of the second dimension, we get Calculate the input image for the i-th concept element set A i The classification results of all concept elements in , that is, calculating J′ i and C′ i The cosine similarity matrix between

[0016] Step 10: For all Y i and D calculated in step 7 c Cross entropy loss is used for training;

[0017] Step 11: Use the linear normalization formula (1+X) / 2 to convert all similarities Y i 、D i and D c Normalize them to the closed interval [0, 1] respectively, and then integrate them to get the final classification score Y;

[0018] Step 12: Substitute the L calculated by formula (17) c , L calculated by formula (19) y And the N loss values ​​L calculated by formula (16) i Perform weighted summation and calculate the final loss L:

[0019]

[0020] Among them, α2 and α3 are the balancing factors for balancing different parts of the final loss L;

[0021] Step 13: Using the backpropagation method, starting from the final loss function L, reversely calculate the gradients of all trainable parameters, and use the Adam optimizer to perform gradient descent on the gradients of all trainable parameters to determine whether the training round is greater than or equal to the maximum training round n max If not, return to step 1; if so, end the algorithm.

[0022] In step 3, the specific operation of each visual autoencoding is as follows:

[0023]

[0024] Among them, l represents the number of the concept element type plus the combined concept, V ql 、V kl , and V vl represents the normalized query, key, and value matrices of V, Represents the result after V is self-attention calculated, V l Represents the visual feature matrix of V in the concept element space where the ,th concept element set is located or the combination space where the combination set C is located.

[0025] Among them, since the dimension of some matrices in the present invention exceeds two dimensions, the present invention only performs matrix multiplication on the last two dimensions of the two matrices participating in the multiplication. Since there is only one combination concept, each combination concept in this combination concept is composed of N concept elements, so l is at most N+1. LN (LayerNorm) represents the layer normalization operation on the last dimension commonly used in deep learning. LN vql 、LN vkl 、LN vvl and LN vml They are four different LNs in the lth visual autoencoder, where LN vql represents the layer normalization operation for the l-th query of the visual feature matrix V, LN vkl Represents the layer normalization operation for the lth key of the visual feature matrix V, LN vvl Represents the layer normalization operation for the lth value of the visual feature matrix V, MLP vl Indicates the use Multi-Layer Perceptron (MLP) vl The number of channels in the last dimension of the input matrix is ​​increased from d v Linear mapping to 4D v , GeLU activation function activation and the last dimension channel number from 4dv Mapping back to d v , LN vml Indicates that Input to the previous layer normalization operation, W ql Is V ql The number of channels is from d v Mapping matrix, W kl Is V kl The number of channels is from d v Mapping to d v The mapping matrix, W vl Is V vl The number of channels is from d v Mapping to d v The mapping matrix, W ql 、W kl and W vl The shape of d v ×d v And the element values ​​are different. Softmax(X) is a softmax normalization operation on the last dimension of the input matrix X. After V is calculated N+1 times by the visual autoencoder, N+1 outputs are obtained, that is, a combined visual feature matrix V c =V N+1 and N concept meta-visual feature matrices V i .

[0026] In step 8, the i-th concept element is removed, and the n-th concept element is obtained. i The classification result vector of b input images of concept elements The value of N-1 Extract the first dimension The concept meta-semantic features corresponding to the concept meta-number included are obtained, and N-1 Contains the nth i Among the concept elements, the concept element semantic features corresponding to the concept element to which the b input images belong are then Insert a new dimension before the second dimension and add Copy m i times, get N-1 According to n i The order from small to large is spliced ​​in the third dimension, because The number of channels in the last dimension is d v ≠d, so calculate Among them LN v and It is the LN and mapping matrix that maps V, adds a dimension to V′ before the second dimension, and copies V′ to this dimension mi times, get Finally, V" is spliced ​​onto the back of these N-1 in the third dimension to obtain the initial semantic conditional representation of the i-th concept element. Then it is fed into the semantic conditional encoder, which performs the following operations:

[0027]

[0028]

[0029] Get the semantic conditional representation of the i-th concept element Among them, LN rqi 、LN rki 、LN rvi and LN rmi They are four different layer normalization operations in the i-th concept meta-semantic conditional encoder, LN rqi It is used to calculate The query layer normalization, LN rki It is used to calculate Layer normalization of the key, LN rvi It is used to calculate Layer normalization of the value (value), MLP ri is the MLP in the i-th concept meta-semantic conditional encoder, MLP ri The input matrix is ​​linearly mapped from d to 4d in the last dimension, activated by the GeLU activation function, and the last dimension of the channel is mapped from 4d back to d, LN rmi It will Input to MLP ri The previous layer normalization, G qi Is to R qi The number of channels is mapped from d to d by the mapping matrix, G ki Is Rk i The number of channels is mapped from d to d by the mapping matrix, G vi Is to R vi The number of channels is mapped from d to d by the mapping matrix, G qi , G ki and G vi The shape of R is d×d and the element values ​​are different. qi 、R ki and R vi Respectively The normalized query matrix, key matrix, and value matrix of , Indicates that The result after self-attention calculation;

[0030] Then, in P iInsert a dimension before the first dimension of P i Copy b times and get Then R i and P′ i Input semantic self-evolving cross encoder, use single-layer cross attention and multi-layer perceptron to transform R i Fusion into P′ i , the following calculation is performed:

[0031] C qi =LN cqi (P′ i ), C ki =LN cki (R i ), C vi =LN cvi (R i ), i=1,2,...,N (7)

[0032]

[0033] Get the i-th concept element with R i Semantic self-evolution conditional representation Among them, LN cqi 、LN cki 、LN cvi and LN cmi They are four different layer normalization operations in the i-th concept meta-semantic self-evolution cross encoder, LN cqi is used to calculate P′ i The query layer normalization, LN cki It is used to calculate R i Layer normalization of the key, LN cvi It is used to calculate R i Layer normalization of the value (value), MLP ci is the MLP in the i-th concept meta-semantic self-evolving cross encoder, MLP ci The input matrix is ​​linearly mapped from d to 4d in the last dimension, activated by the GeLU activation function, and the last dimension of the channel is mapped from 4d back to d, LN cmi It will Input to MLP ci The previous layer normalization, H qi Is C qi The number of channels is mapped from d to d by the mapping matrix, H ki Is C ki The number of channels is mapped from d to d by the mapping matrix, H vi Is C viThe number of channels is mapped from d to d by the mapping matrix, H qi 、H ki and H vi The shape of C is d×d and the element values ​​are different. qi 、C ki and C vi Represent P′ i The normalized query matrix, R i The normalized bond matrix and R i The normalized value matrix of Indicates that P′ i and R i The result after cross attention calculation;

[0034] Next, we calculate the visual self-evolution conditional representation J ai , N-1 In the order of small to large, they are spliced ​​in the second dimension to obtain the initial visual condition representation of the i-th concept element Then it is fed into the visual condition encoder of the i-th concept element, which performs the following operations:

[0035] Get the visual condition representation of the i-th concept element Among them, LN uqi 、LN uki 、LN uvi and LN umi They are four different layer normalization operations in the i-th concept meta-visual conditional encoder, LN uqi It is used for calculation The query layer normalization, LN uki It is used for calculation Layer normalization of the key, LN uvi Is the layer normalization used to calculate the value (value), MLP ui is the MLP in the i-th concept meta-visual self-evolving cross encoder, MLP ui The input matrix is ​​linearly mapped from d to 4d in the last dimension, activated by GeLU activation function, and the last dimension of the channel is mapped from 4d back to d. umi It will Input to MLP ui The previous layer normalization, Q qi Is to U qi The number of channels is mapped from d to d by the mapping matrix, Q ki Is to U ki The number of channels is mapped from d to d by the mapping matrix, Q vi Is U viThe number of channels is mapped from d to d by the mapping matrix, Q qi , Q ki and Q vi The shape of U is d×d and the element values ​​are different. qi 、U ki and U vi Respectively The normalized query matrix, key matrix, and value matrix of , Indicates that The result after self-attention calculation;

[0036] Calculate U i After that, due to V i The number of channels in the last dimension d v ≠d, so calculate in Will U i and V′ i Input visual self-evolving cross encoder, use single-layer cross attention and multi-layer perceptron to transform U i Integration into V′ i , the following calculation is performed:

[0037] J qi =LN jqi (V′ i ), J ki =LN jki (U i ), J vi =LN jvi (U i ), i=1,2,...,N (13)

[0038]

[0039] Get the i-th concept element with U i Conditional representation of visual self-evolution Among them, LN jqi 、LN jki 、LN jvi and LN jmi They are four different layer normalization operations in the visual self-evolving cross encoder of the i-th concept element, LN jqi is used to calculate V′ i The query layer normalization, LN jki It is used to calculate U i Layer normalization of the key, LN jvi It is used to calculate U i Layer normalization of the value (value), MLP jiis the MLP in the i-th concept meta-visual self-evolving cross encoder, MLP ji The input matrix is ​​linearly mapped from d to 4d in the last dimension, activated by GeLU activation function, and the last dimension of the channel is mapped back to d. jmi It will Input to MLP ji The previous layer normalization, K qi It is J qi The number of channels is mapped from d to d by the mapping matrix, K ki It is J ki The number of channels is mapped from d to d by the mapping matrix, K vi It is J vi The number of channels is mapped from d to d by the mapping matrix, K qi , K ki and K vi The shape of is d×d and the element values ​​are different. Due to the different input images, the semantic condition representation R of each concept element is calculated. i and visual condition representation U i Different, so the semantic self-evolution condition representation C of each concept element i and visual self-evolution conditional representation J i All have real-time evolution characteristics, J qi 、J ki and J vi Represent V′ i The normalized query matrix, U i The normalized bond matrix and U i The normalized value matrix of Indicates that V′ i and U i The result after cross attention calculation.

[0040] In step 10, all Y i and D calculated in step 7 c Cross entropy loss is used for training, namely:

[0041] where τ is the temperature coefficient used for cosine similarity scaling.

[0042] In step 11, the final classification score Y is obtained as:

[0043]

[0044] Among them, α1 is the weighting factor, and Y is trained using cross entropy loss, that is:

[0045]

[0046] An electronic device, comprising:

[0047] one or more processors;

[0048] Memory;

[0049] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to execute the method as described above.

[0050] A computer-readable storage medium is characterized in that the computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method described above.

[0051] The beneficial effect of the present invention is to flexibly encode all kinds of concept meta-semantics and visual representations using an innovative and efficient conditional self-evolution representation calculation method. Unlike previous methods, this method uses the cross-fusion feature matrix of the predicted other concept meta-text prompt feature matrix and the original feature matrix of the input image as a condition to perform feature fusion with the concept meta-semantics and visual representation to be self-evolved, thereby achieving the instant self-evolution of each concept meta-semantics and visual representation for different input images, which not only makes the learned semantics and visual representations more adaptable to different input images, but also solves the problem of previous methods' lack of modeling of the mutual relationship between concept metas. Compared with previous methods, this method has achieved outstanding progress in classification accuracy on multiple data sets, whether for classification of visible classes or new classes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the accompanying drawings and examples.

[0054] The present invention provides a combined generalized image multi-label classification method based on self-evolving conditional representations. This method can instantly self-evolve the visual and semantic representations of a certain concept element according to the representations of other types of concept elements to which the input image belongs and the visual representation of the input image, making the learned representation more flexible and having stronger generalization performance for new combinations.

[0055] The entire system first uses a pre-trained visual backbone model to extract a set of feature embeddings for the input image, and uses a pre-trained text backbone model to extract feature embeddings containing prompts of all candidate concept meta-elements; then a multimodal self-attention mechanism is used to fuse the image with the feature embeddings of predicted other types of concept meta-elements to obtain the preconditions for self-evolution of the representation; then a multimodal feature fusion mechanism is used to fuse the preconditions with the prompts of all candidate concept meta-elements, and output the self-evolved concept meta-elements prompt representation, so that all candidate concept meta-elements prompt representations can autonomously adapt to different input samples; finally, the similarity between the image features and the concept meta-elements prompt representations of each condition and the combined prompt representation are weighted and summed to obtain the final classification probability distribution.

[0056] The technical solution of the present invention mainly includes six parts: the first part is to use pre-trained visual and language models to extract features of images and text prompts respectively; the second part is to calculate the similarity between the representation of the input image mapped into the concept meta-space and the semantic representation of the concept meta-prompt, and pre-classify the image to the concept meta-; the third part is to model the self-evolution condition based on the pre-classified concept meta-prompt and the original representation of the input image; the fourth part is the process of self-evolving the concept meta-prompt representation and the visual representation based on the self-evolution condition; the fifth part is to calculate the similarity between the self-evolved visual and semantic representations of each concept meta, as well as the similarity between the representation of the input image mapped into the combination space and the semantic representation of the combination prompt, to complete the classification of the image to the combination; the last part is to integrate all classification results.

[0057] The following is a specific embodiment of the present invention on the combined generalized image three-label classification task. The pre-trained visual Transformer and pre-trained text Transformer models in the CLIP pre-trained open source model are used as the backbone models for visual and text feature extraction, and the parameters of the backbone model are not updated, that is, B * It's CLIP. It is CLIP's pre-trained visual Transformer, It is CLIP's pre-trained text Transformer, n v =257,d v =1024, d=768. Set b=128, n t=16, τ = 0.01, and the number of sampling rounds for all images in the dataset is 4. Three-label classification training is performed on the training set of the UT-Zappos50K dataset. The three labels are: two concept elements - attributes and objects, and the combination of the two, that is, N = 2. Since the training set of the UT-Zappos50K dataset contains a total of 22,998 images, b = 128, it takes 180 image batches to sample all images in the training set. 4 rounds of sampling for all images in the training set require a total of 180 × 4 = 720 image batches, that is, n max =720.

[0058] 1. Extract input image features

[0059] Set the training round counter to 1, randomly sample a batch of b = 128 RGB input images from the training set, scale the entire image to 224 × 224 using the bicubic interpolation method, randomly flip the left and right with a probability of 0.5, and randomly rotate it within ±5°. Finally, perform a normalization operation with the average values ​​of the RGB channels being 0.48145466, 0.4578275, and 0.40821073, and the standard deviations being 0.26862954, 0.26130258, and 0.27577711 (the average values ​​and standard deviations are predefined by the CLIP open source pre-trained model). Input the output of the normalization operation into Extract visual feature matrix

[0060] 2. Calculate the visual feature matrix of the input image feature matrix in attribute, object and combination space.

[0061] Input V into the attribute, object and combined visual autoencoder, and calculate it through equations (1) to (3) to obtain V a 、V o and V c , that is, the visual feature matrix of attribute, object and combination space, which is calculated as follows:

[0062] V ql =LN vql (V), V kl =LN vkl (V), V vl =LN vvl (V), l=1, 2, 3

[0063]

[0064] Let V a =V1,V o =V2,V c =V3. V a 、V o and Vc The dimensions are 128×257×1024.

[0065] 3. Extract the attribute and object concept elements and the feature matrix of the combined prompt text.

[0066] The UT-Zappos50K dataset contains m1=16 attribute concept elements and m2=12 object concept elements. The training set contains n c = 83 combinations. Construct attribute prompt template t1 = "A photo of x object", object prompt template t2 = "A photo of x" and combination prompt template t c = "A photo of xx", and replace t1, t2 and t c Copy m1=16, m2=12 and n c =83 times. First, use B * Word vector library in Extract word vectors for all attribute and object concept metawords, and obtain 28 word vectors with d=768 dimensions. Secondly, use the "0" placeholder to expand the number of words in all prompt templates to n t =16, extract the word vectors of all words (including the "0" placeholder) of all prompt templates to obtain the word vector matrix of the attribute prompt template Word vector matrix of object hint template And the word vector matrix of the combined prompt template The sizes of the three are 16×16×768, 12×16×768 and 83×16×768 respectively. Then, the word vectors of the 16 attribute concept elements are calculated according to T a The attribute to which each row belongs, replacing T a The word vector of "x" in this row is the word vector of 12 object concept elements according to T o The object to which each row belongs, replaces T o The word vector of "x" in this row is the word vector of 83 attribute-object combination concept elements according to T c The combination to which each row belongs, replace T c The word vectors of the two "x" in this row are respectively and Finally, adopt For T′ a , T′ o and T′ c Extract features, keep the output size unchanged, and obtain the semantic feature matrix of attributes, objects and combinations and

[0067] 4. Pre-classify attributes, objects and combinations.

[0068] P a 、P o and P c The second dimension of the average pooling is performed to eliminate the second dimension and obtain and Take V a 、V o and V c The first element of the second dimension, and Then calculate and Finally calculate B′ a , B′ o and B′ c and S′ a , S′ o and S′ c The cosine similarity vector of and

[0069] The pre-classification results are and

[0070] 5. Compute semantically and visually conditioned self-evolving representations of attributes and objects.

[0071] For attribute or object concept element, according to or The value of P o or P a The characteristics of the corresponding row are obtained a or E o , E a and E o The size of E is 128×16×768. a and E o Insert a new dimension before the second dimension and change E to a and E o Replicate 16 and 12 times respectively, and you get and because The number of channels in the last dimension of is 1024≠768, so the calculation Among them LN v and is the LN and mapping matrix that maps V; add a dimension to V′ before the second dimension and replicate V′ 16 times and 12 times on this dimension respectively, to obtain and Finally, V″ a and V″ o Spliced ​​to E' a and E′ o After that, we get the initial semantic condition representation of attribute L or object concept element or Then and The semantic conditional encoders of the attribute or object concept are fed into the encoders respectively, and the following operations are performed:

[0072]

[0073] Obtain semantic conditional representations of attribute and object concept elements and After that, in P a and P o Insert a dimension before the first dimension of P a and P o Copy 128 times to get P′ a and P′ o , P′ a and P′ o The shapes are 128×16×16×768 and 128×12×16×768 respectively, let P′1=P′ a , P′2=P′ o Finally, the semantic self-evolving cross encoders of the input attributes and object concept elements of R1 and P′1, R2 and P′2 are used, that is, single-layer cross attention and multi-layer perceptron are used to fuse R1 into P′1 or R2 into P′2:

[0074] C qi =LN cqi (P′ i ), C ki =LN cki (R i ), C vi =LN cvi (R i ), i=1,2

[0075]

[0076] The semantic self-evolution conditions of attributes and objects are represented as and Next, we calculate the visual self-evolution conditional representation J of attributes and objects a =J1 and J o = J2. First, calculate the visual condition representations U1 and U2 of the attributes and objects, and set E1 = E a and E2=Eo Each input attribute and object visual condition encoder performs the following operations:

[0077] U qi =LN uqi (E i ), U ki =LN uki (E i ), U vi =LN uvi (E i ), i=1,2

[0078]

[0079] Get visually conditioned representations of attributes and objects and Then the visual self-evolved conditional representation of attributes and objects is calculated. a and V o The number of channels in the last dimension is 1024≠768, so the calculation in Let V′1=V′ a , V′2=V′ o , using single-layer cross attention and multi-layer perceptron to integrate U1 or U2 into V′1 or V′2, we have:

[0080] I qi =LN jqi (V′ i ), J ki =LN jki (U i ), J vi =LN jvi (U i ), i=1,2

[0081]

[0082] Obtaining the visual self-evolution conditional representation of attribute and object concept elements and

[0083] 6. Final classification of attributes and objects

[0084] In C a and C o Average the third dimension and eliminate C a and C o The third dimension of and Select J a and J o The first feature of the second dimension, we get and Calculate the classification results of the input image for all attributes or object concept elements, that is, calculate J′ a , C′ a and J′ o , C′ o The cosine similarity between and

[0085] 7. Model loss calculation

[0086] For all Y a 、Y o and D c Cross entropy loss is used for training, namely: Use the formula (1+X) / 2 to convert all similarities Y a 、Y o and D c Normalize to the closed interval [0, 1] and integrate all classification probabilities to get the final classification score Y:

[0087] Y=0.5D c +0.25(Y a ⊙D a +Y o ⊙D o )

[0088] Y is trained using cross entropy loss, i.e. The final loss is calculated by weighted summing of all losses, namely:

[0089] L=0.05(L a +L o )+0.05L c +L y ;

[0090] 8. Model parameter update

[0091] The back-propagation method is used to start from the final loss function L, and the gradients of all trainable parameters are calculated in reverse. The Adam optimizer is used to perform gradient descent on the gradients of all trainable parameters.

[0092] 9. Determine the training stop conditions

[0093] If the training round at this time is less than the maximum training round n max =720, then the training round counter is incremented by 1, and then the process returns to step 1, otherwise the training ends.

[0094] The above is the end-to-end training process of the present invention for the UT-Zappos50K dataset. Only one image is input to the network each time. Figure 1 A schematic diagram of the invention structure under this specific embodiment is given. Three datasets, namely UT-Zappos50K, MIT-States, and C-GQA, are used for model training and evaluation testing during formal model training. The batch size is set to 128 input images, the initial learning rate is set to 1e-4, and 1, 2, and 4 NVIDIA A100-80GB graphics cards are used, respectively, to perform different numbers of iterations on all images in the training set of each dataset. After every 3 training rounds, the learning rate is halved, and the Adam optimizer is used for gradient descent.

[0095] 8. Model Application

[0096] During the above training process, all trainable parameters in the model are saved when the model achieves better index results. During testing, the model parameters with the best index (the best test effect on the validation set) are selected for application. It is only necessary to adjust the image to 224x224 size and normalize it before using it as the input of the model. The parameters of the entire network model are fixed, as long as the input image and all the concept meta-words are propagated forward together. The model can automatically generate the conditional self-evolution semantics and visual representations of the concept meta-words based on the image and the concept meta-words, and finally obtain the multi-label classification results of the image. Based on the present invention, a higher combined generalized image classification accuracy can be obtained.

Claims

1. A combined generalized image multi-label classification method based on self-evolving conditional representation, characterized by The steps include: Step 1: Randomly sample a batch of input images from the dataset, the batch contains b input images, and uniformly scale all input images in the batch to H×W size using bicubic interpolation, where H is the image height and W is the image width. Then, with a probability of 0.5, all input images in the batch are randomly horizontally flipped, randomly rotated within ±5°, and normalized with the average values ​​of the RGB three channels as μ1, μ2, and μ3, and the standard deviations as σ1, σ2, and σ3. All input images of the batch output after image scaling, random horizontal flipping, random rotation, and image RGB channel normalization are passed through the open source pre-trained visual language model B with all parameters frozen. * Visual Transformer Model in Extract features and obtain visual feature matrix Among them, * indicates that all parameters are frozen and not updated during training, n v represents the number of visual features, d v Indicates the number of channels for each feature in the visual feature, n v The first feature among the visual features represents the global feature of the input image; Step 2: Given N concept element sets A i , i represents the serial number of all types of concept elements, i = 1, 2, ..., N, j represents each concept element set A i The concept element number in A i Contains m i Concept element a ij , A i ={α ij |j=1,2,...,m i }, all concept elements a ij Total composition Combinations, using set representation, all combinations consisting of N concept elements are represented as combination set C = {c k |k=1,2,...,n c }, where c k represents the kth combination concept in C, and k represents the sequence number of the combination concept in the combination set C; Step 3: Input the visual feature matrix V into N+1 visual autoencoders, and perform N+1 visual autoencodings independently. Each visual autoencoding first passes through a single layer of self-attention and then passes through a multi-layer perceptron to map V to the concept metaspace where each concept metaset is located and the combination space where the combination set C is located; Step 4: Construct two text prompt templates i and t c :t i Used to construct N kinds of concept meta-prompts, each concept meta-prompt template t i Can constitute m i concept meta-prompts, each concept meta-prompt template t i The text contains 1 concept meta-word placeholder; t c Used to construct combined concept prompts, t c Can constitute n c Combined concept prompts, each combined concept prompt template t c The text contains N concept meta-word placeholders, and the concept meta-word placeholders are all represented by the letter "x"; Step 5: Use B * Pre-trained word vector library in t i and t c Each word in is mapped into the corresponding word feature vector, which constitutes the concept meta-prompt feature matrix and combined concept hint feature matrix Then extract all concept meta-word feature vectors For the i-th concept element, all concept element word feature vectors of this concept element Replace T i All the feature vectors of "x" in the final result are N replaced concept element prompt feature matrices Among them, each T′ i Contains m i Concept meta-prompt features, n t is the maximum number of words in the prompt text sentence, d t is the number of channels of each word feature vector. For the k-th combined concept, the N concept meta-word feature vectors constituting the combined concept are substituted into T c The eigenvector of the letter "x" is replaced in the final result, which contains n c combined concept prompt feature matrix of combined prompt features Step 6: All T' i and T′ c Input to B * Pre-trained text Transformer model in Output concept meta-semantic feature matrix and combined concept semantic feature matrix Step 7: For all P i and P c The second dimension of the average pooling is eliminated, and the second dimension is obtained respectively. and In V c and N V i Take the first element of the second dimension and get and N Since d is not equal to d v , so B i and B c The last dimension of is mapped to d channels, that is, and In addition, S i and S c The last dimension of is mapped with the same channel number again, in, and It's B * 2 pre-trained LNs in, and It's B * The pre-trained mapping matrix with two different values ​​in ; Finally, calculate S′ c and B′ c The cosine similarity matrix of and N S′ ai and B′ ai The cosine similarity matrix of Where ||x||2 represents the calculation of the bi-norm of the matrix or vector x, and the pre-classification results of all N concept elements are N Among them, argmax means returning the maximum value in the last dimension. Represents the sequence numbers of all concept elements to which b input images belong; Step 8: Calculate the semantic self-evolution conditional representation of the i-th concept element Calculate the visual self-evolution conditional representation of the i-th concept element Step 9: In C i Average the third dimension and eliminate C i The third dimension of By selecting J i The first feature of the second dimension, we get Calculate the input image for the i-th concept element set A i The classification results of all concept elements in , that is, calculating J′ i and C′ i The cosine similarity matrix between Step 10: For all Y i and D calculated in step 7 c Cross entropy loss is used for training; Step 11: Use the linear normalization formula (1+X) / 2 to convert all similarities Y i 、D i and D c Normalize them to the closed interval [0,1] respectively, and then integrate them to get the final classification score Y: Among them, α1 is the weighting factor, and Y is trained using cross entropy loss; Step 12: Calculate the L c 、L y and N loss values ​​L i Perform weighted summation and calculate the final loss L: Among them, α2 and α3 are the balancing factors for balancing different parts of the final loss L; L i It is the cosine similarity matrix Y of the input image to the i-th concept element i The cross entropy loss function obtained after substituting into the cross entropy loss formula, L c is the cosine similarity matrix D of the input image pairs c The cross entropy loss function obtained after substituting into the cross entropy loss formula, L y It is the cross entropy loss function obtained by substituting the integrated classification score matrix Y into the cross entropy loss formula; Step 13: Using the backpropagation method, starting from the final loss function L, reversely calculate the gradients of all trainable parameters, and use the Adam optimizer to perform gradient descent on the gradients of all trainable parameters to determine whether the training round is greater than or equal to the maximum training round n max If not, return to step 1; if so, end the algorithm.

2. The combined generalized image multi-label classification method based on self-evolving conditional representation according to claim 1 is characterized by: In step 3, the specific operation of each visual autoencoding is as follows: Among them, l represents the number of the concept element type plus the combined concept, V ql 、V kl , and V vl represents the normalized query, key, and value matrices of V, Represents the result after V is self-attention calculated, V l Represents the visual feature matrix of V in the concept element space where the l-th concept element set is located or the combination space where the combination set C is located; Among them, since the dimensions of some matrices exceed two dimensions, matrix multiplication is performed only on the last two dimensions of the two matrices participating in the multiplication. Since there is only one combination concept, each combination concept in this combination concept is composed of N concept elements, so l is at most N+1. LN represents the layer normalization operation on the last dimension commonly used in deep learning. LN vql 、LN vkl 、LN vvl and LN vml They are four different LNs in the lth visual autoencoder, where LN vql Denotes the layer normalization operation for the l-th query of the visual feature matrix V, LN vkl Represents the layer normalization operation for the l-th key of the visual feature matrix V, LN vvl Represents the layer normalization operation for the l-th value of the visual feature matrix V, MLP vl Indicates the use Multilayer Perceptron, MLP vl The number of channels in the last dimension of the input matrix is ​​increased from d v Linear mapping to 4D v , GeLU activation function activation and the last dimension channel number from 4d v Mapping back to d v , LN vml Indicates that Input to the previous layer normalization operation, W ql Is V ql The number of channels is from d v Mapping matrix, W kl Is V kl The number of channels is from d v Mapping to d v The mapping matrix, W vl Is V vl The number of channels is from d v Mapping to d v The mapping matrix, W ql 、W kl and W vl The shape of d v ×d v And the element values ​​are different. Softmax(X) is a softmax normalization operation on the last dimension of the input matrix X. After V is calculated N+1 times by the visual autoencoder, N+1 outputs are obtained, that is, a combined visual feature matrix V c =V N+1 and N concept meta-visual feature matrices V i .

3. The combined generalized image multi-label classification method based on self-evolving conditional representation according to claim 1 is characterized by: In step 8, the i-th concept element is removed, and the n-th concept element is obtained. i The classification result vector of b input images of concept elements n i =1,2,...,i-1,i+1,...,N values, from N-1 Extract the first dimension The concept meta-semantic features corresponding to the concept meta-number included are obtained, and N-1 Contains the nth i Among the concept elements, the concept element semantic features corresponding to the concept element to which the b input images belong are then Insert a new dimension before the second dimension and add Copy m i times, get N-1 According to n i The order from small to large is spliced ​​in the third dimension, because The number of channels in the last dimension is d v ≠d, so calculate Among them LN v and It is the LN and mapping matrix that maps V, adds a dimension to V′ before the second dimension, and copies V′ to this dimension m i times, get Finally, V″ is spliced ​​onto the back of these N-1 in the third dimension to obtain the initial semantic conditional representation of the i-th concept element. Then it is fed into the semantic conditional encoder, which performs the following operations: Get the semantic conditional representation of the i-th concept element Among them, LN rqi 、LN rki 、LN rvi and LN rmi They are four different layer normalization operations in the i-th concept meta-semantic conditional encoder, LN rqi It is used to calculate The query layer normalization, LN rki It is used to calculate Layer normalization of the key, LN rvi It is used to calculate Layer normalization of the value of , MLP ri is the MLP in the i-th concept meta-semantic conditional encoder, MLP ri The input matrix is ​​linearly mapped from d to 4d in the last dimension, activated by the GeLU activation function, and the last dimension of the channel is mapped from 4d back to d, LN rmi It will Input to MLP ri The previous layer normalization, G qi Is to R qi The number of channels is mapped from d to d by the mapping matrix, G ki Is to R ki The number of channels is mapped from d to d by the mapping matrix, G vi Is to R vi The number of channels is mapped from d to d by the mapping matrix, G qi , G ki and G vi The shape of R is d×d and the element values ​​are different. qi 、R ki and R vi Respectively The normalized query matrix, key matrix, and value matrix of , Indicates that The result after self-attention calculation; Then, in P i Insert a dimension before the first dimension of P i Copy b times and get Then R i and P′ i Input semantic self-evolving cross encoder, use single-layer cross attention and multi-layer perceptron to transform R i Fusion into P′ i , the following calculation is performed: Get the i-th concept element with R i Semantic self-evolution conditional representation Among them, LN cqi 、LN cki 、LN cvi and LN cmi They are four different layer normalization operations in the i-th concept meta-semantic self-evolution cross encoder, LN cqi is used to calculate P′ i The query layer normalization, LN cki It is used to calculate R i Layer normalization of the key, LN cvi It is used to calculate R i Layer normalization of the value of , MLP ci is the MLP in the i-th concept meta-semantic self-evolving cross encoder, MLP ci The input matrix is ​​linearly mapped from d to 4d in the last dimension, activated by the GeLU activation function, and the last dimension of the channel is mapped from 4d back to d, LN cmi It will Input to MLP ci The previous layer normalization, H qi Is C qi The number of channels is mapped from d to d by the mapping matrix, H ki Is C ki The number of channels is mapped from d to d by the mapping matrix, H vi Is C vi The number of channels is mapped from d to d by the mapping matrix, H qi 、H ki and H vi The shape of C is d×d and the element values ​​are different. qi 、C ki and C vi Represent P′ i The normalized query matrix, R i The normalized bond matrix and R i The normalized value matrix of Indicates that P′ i and R i The result after cross attention calculation; Next, we calculate the visual self-evolution conditional representation J ai , N-1 According to n i The order from small to large is spliced ​​in the second dimension to obtain the initial visual condition representation of the i-th concept element Then it is fed into the visual condition encoder of the i-th concept element, which performs the following operations: Get the visual condition representation of the i-th concept element Among them, LN uqi 、LN uki 、LN uvi and LN umi They are four different layer normalization operations in the i-th concept meta-visual conditional encoder, LN uqi It is used for calculation The query layer normalization, LN uki It is used for calculation Layer normalization of the key, LN uvi is the layer normalization used to calculate the value of MLP ui is the MLP in the i-th concept meta-visual self-evolving cross encoder, MLP ui The input matrix is ​​linearly mapped from d to 4d in the last dimension, activated by GeLU activation function, and the last dimension of the channel is mapped from 4d back to d. umi It will Input to MLP ui The previous layer normalization, Q qi Is to U qi The number of channels is mapped from d to d by the mapping matrix, Q ki Is U ki The number of channels is mapped from d to d by the mapping matrix, Q vi Is U vi The number of channels is mapped from d to d by the mapping matrix, Q qi , Q ki and Q vi The shape of U is d×d and the element values ​​are different. qi 、U ki and U vi Respectively The normalized query matrix, key matrix, and value matrix of , Indicates that The result after self-attention calculation; Calculate U i After that, due to V i The number of channels in the last dimension d v ≠d, so calculate in Will U i and V′ i Input visual self-evolving cross encoder, use single-layer cross attention and multi-layer perceptron to transform U i Integration into V′ i , the following calculation is performed: Get the i-th concept element with U i Conditional representation of visual self-evolution Among them, LN jqi 、LN jki 、LN jvi and LN jmi They are four different layer normalization operations in the visual self-evolving cross encoder of the i-th concept element, LN jqi is used to calculate V′ i The query layer normalization, LN jki It is used to calculate U i Layer normalization of the key, LN jvi It is used to calculate U i Layer normalization of the value of , MLP ji is the MLP in the i-th concept meta-visual self-evolving cross encoder, MLP ji The input matrix is ​​linearly mapped from d to 4d in the last dimension, activated by GeLU activation function, and the last dimension of the channel is mapped back to d. jmi It will Input to MLP ji The previous layer normalization, K qi It is J qi The number of channels is mapped from d to d by the mapping matrix, K ki It is J ki The number of channels is mapped from d to d by the mapping matrix, K vi It is J vi The number of channels is mapped from d to d by the mapping matrix, K qi , K ki and K vi The shape of is d×d and the element values ​​are different. Due to the different input images, the semantic condition representation R of each concept element is calculated. i and visual condition representation U i Different, so the semantic self-evolution condition representation C of each concept element i and visual self-evolution conditional representation J i All have real-time evolution characteristics, J qi 、J ki and J vi Represent V′ i The normalized query matrix, U i The normalized bond matrix and U i The normalized value matrix of Indicates that V′ i and U i The result after cross attention calculation.

4. The combined generalized image multi-label classification method based on self-evolving conditional representation according to claim 1 is characterized by: In step 10, all Y i and D calculated in step 7 c Cross entropy loss is used for training, namely: where τ is the temperature coefficient used for cosine similarity scaling.

5. The combined generalized image multi-label classification method based on self-evolving conditional representation according to claim 1 is characterized by: In step 11, the final classification score Y is obtained as: Among them, α1 is the weighting factor, and Y is trained using cross entropy loss, that is:

6. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the method according to any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 3.

Images