Combined zero sample image classification method based on hierarchical feature fusion

Abstract

The invention belongs to the technical field of computer vision and artificial intelligence, and discloses a combined zero sample image classification method based on hierarchical feature fusion. According to the method, a solution based on hierarchical feature fusion is innovatively provided for the technical difficulty of combined zero sample image classification. According to the method, features of different depths in a CLIP model visual encoder are selected and subjected to hierarchical fusion, multi-level and multi-scale visual information is effectively extracted, the fused features are further used for visual and semantic cross-modal interaction, and the understanding ability of the model for complex combinations is remarkably improved. In addition, by introducing a specific loss item, attributes and object features are effectively separated, and the feature aliasing problem is avoided. In a combined zero sample image classification task, the method shows excellent generalization ability and recognition effect, and can significantly improve the classification performance of the model for unknown complex combined images.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and artificial intelligence, and in particular relates to a combined zero-sample image classification method based on hierarchical feature fusion. Background Art

[0002] The existing combined zero-shot image classification methods at home and abroad can be divided into four categories. The first type of method is a model based on a single concept classifier. This method regards attributes and objects as equally important concepts. It learns classifiers for each concept and combines attributes and objects to identify integrated concepts, that is, new attribute-object pairs. However, this type of method has obvious shortcomings. It generalizes attributes and objects, ignoring the fact that attributes are highly correlated with objects visually and depend on context, so they often perform poorly. The second type of method is based on an image-combination compatibility model. This method treats attribute-object pairs as a whole and directly learns the compatibility feature representation between them and the image. Due to the introduction of deep learning technology, this type of method has shown some improvements in model performance. However, this type of method still does not clearly distinguish between attributes and objects. Attributes and objects are still intertwined and affect each other. The third type of method is based on the attribute-object explicit decoupling model, which usually uses spatial embedding methods to explicitly decouple attributes and objects. Attributes and objects are no longer regarded as the same concept, but are successfully decoupled in the embedding space while emphasizing their differences and connections. This method has brought performance improvements, but information loss is inevitable in the embedding process. Some subtle but crucial features or relationships may become blurred or lost, making it difficult for the model to accurately represent all attribute-object combinations. The fourth type of method is a model based on CLIP. CLIP was proposed by the OpenAI team in 2021. It provides a new idea and paradigm for multimodal information fusion processing.

[0003] However, although CLIP shows considerable learning efficiency and generalization characteristics in the pre-training stage and can be widely adapted to general multimodal data processing requirements, its limitations become apparent when it is directly deployed and applied to specific downstream tasks. Take the combinatorial zero-shot image classification task as an example. This task requires the model to accurately identify image category combinations that have never appeared in the training set. Since the original CLIP model is not carefully optimized for such specific scenarios, it is difficult to adapt to its special data distribution and task requirements, resulting in unsatisfactory model performance.

[0004] To resolve this dilemma, a series of improvement plans have emerged in academia and industry. The core strategy is to perform targeted fine-tuning on the embedding vectors generated by CLIP. In principle, fine-tuning aims to reshape the text embeddings produced by the model to make them fit the specific task objectives, thereby improving the performance of the model in specific scenarios. However, the existing fine-tuning method has obvious defects. This method focuses too much on the optimization of the output of the visual encoder and the text encoder, and ignores the key information contained in the feature encoding process. Each level of the visual encoder contains rich and valuable intermediate features in the feature extraction and transformation links. These features have not been properly mined and effectively utilized, which directly weakens the model's ability to generalize to unknown combination scenarios. As a result, when the model encounters a new image-text combination, the classification accuracy and stability are poor, and it cannot fully meet the stringent performance standards of practical applications. Summary of the invention

[0005] The purpose of the present invention is to propose a combined zero-shot image classification method based on hierarchical feature fusion, which fully mines the key information contained in each level of the CLIP visual encoder in the feature extraction and transformation links, and uses it in the cross-modal interaction of text features and visual features, so that the generated embedded representation is more in line with the requirements of the combined zero-shot classification task, so as to improve the performance of the model in processing complex attribute-object combination scenarios.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme: A combined zero-shot image classification method based on hierarchical feature fusion includes the following steps: Step 1. Build a combined zero-shot image classification model based on hierarchical feature fusion, including CLIP text encoder, CLIP visual encoder, multi-layer visual feature fusion module, cross-modal interaction module, decoupling module and loss calculation module; Step 2. Generate preliminary embedding representations, i.e., word embeddings, for attributes, objects, and combinations. The preliminary embedding representations will be updated as parameters during the back-propagation process. Combine the preliminary embedding representations of attributes, objects, and combinations with the prompt prefix embeddings to obtain the text features of attributes, objects, and combinations through the pre-trained CLIP text encoder. Step 3. Pass the image in the training set as the input image into the CLIP visual encoder, process the features of the image layer by layer, and extract features of different levels from the CLIP visual encoder; Step 4. Filter out the initial layer features, intermediate layer features, and final layer features from the CLIP visual encoder, and calculate the weights of features at different levels, i.e., weighted coefficients, through a multi-layer visual feature fusion module, perform weighted combination of features at different levels, and construct visual fusion features; Step 5. Input the text features obtained in step 2 and the visual fusion features constructed in step 4 into the cross-modal interaction module, and achieve deep fusion of visual fusion features and text features through the cross-attention mechanism to obtain optimized attribute, object, and combined text features; Step 6. Decouple the image features output by the last layer of the CLIP visual encoder. The decoupling module uses three trainable multi-layer perceptrons (MLPs) to extract attribute visual features, object visual features, and combined visual features from the image features output by the last layer of the CLIP visual encoder. Step 7. The loss calculation module calculates the similarity of the visual features of the attributes, objects, and combinations obtained after decoupling in step 6 and the text features of the optimized attributes, objects, and combinations obtained in step 5 using cosine similarity to obtain the predicted value, and measures the difference between the predicted value and the true label through the classification loss function to form the loss of each branch. At the same time, the covariance loss term is introduced to optimize the decoupling effect, and the total loss is obtained by weighted summation; Step 8. Through the back-propagation mechanism of the total loss, the weight coefficients of the multi-layer visual feature fusion module, the parameters of the MLP for decoupling, and the preliminary embedding representations of attributes, objects, and combinations are jointly updated; Step 9. Repeat steps 3 to 8, and evaluate the combined zero-shot image classification model based on hierarchical feature fusion after each iteration on the validation set until the loss function converges. According to the validation set results, the trained combined zero-shot image classification model based on hierarchical feature fusion and the optimized embedding representations of attributes, objects, and combinations are obtained. Step 10. Use the trained combined zero-shot image classification model based on hierarchical feature fusion to classify the input image.

[0007] The present invention has the following advantages: As described above, the present invention describes a combined zero-sample image classification method based on hierarchical feature fusion. First, by deeply mining and fusing the features of each level of the CLIP visual encoder, the multi-dimensional information of the image from local details to the overall semantics is effectively integrated. The multi-level feature fusion method enables the model to understand the image features more comprehensively when facing complex image content, avoiding classification errors caused by the limitations of single-level features. When processing images containing multiple objects and objects with complex attribute associations, the fused visual features can simultaneously capture the detailed information such as the edges and textures of the objects as well as the high-level information such as the spatial relationships and semantic categories between objects, thereby providing a solid foundation for accurate classification.

[0008] Secondly, the introduction of the cross-modal interaction module greatly enhances the correlation between text features and visual features. Using the cross-attention mechanism, text features can be dynamically adjusted according to the information in the visual fusion features, so that the text embedding can better adapt to the actual visual content of the image. This not only improves the accuracy of the text's description of the image's semantics, but also enables the model to more accurately classify images of unseen categories based on text cues in combined zero-shot classification tasks.

[0009] Furthermore, the feature decoupling operation improves the adaptability of the model in zero-shot combination tasks. The three trainable multi-layer perceptrons extract attribute, object, and combination visual features respectively, ensuring the independence of these key elements in the feature representation. This helps the model to flexibly use existing feature knowledge for reasoning and classification when faced with new attribute-object combinations without being disturbed by confusion or redundant information between features.

[0010] In addition, the design of loss calculation and optimization objectives makes the model training process more scientific and effective. By measuring the difference between the predicted value and the true label through the classification loss function, the model can be directly guided to optimize in the direction of improving classification accuracy. At the same time, the introduction of covariance loss optimization decoupling effect further improves the independence and distinguishability of features, thereby improving the overall performance of the model. During the model backpropagation and parameter update process, the parameters of multiple key modules are jointly optimized to ensure the collaborative work between the various parts of the model and the improvement of overall performance.

[0011] Finally, the strategy of repeatedly iteratively optimizing the embedded representation ensures that the model can gradually converge to the optimal state. By continuously evaluating the model performance and adjusting the parameters on the validation set, the final model has excellent generalization ability and classification accuracy in the combined zero-shot image classification task. The method of the present invention is not only suitable for combined zero-shot image classification scenarios, but also plays an important role in special fields such as rare object recognition and emerging concept image classification, providing strong support for the application of image classification technology in more complex and diverse scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Flow chart of a combined zero-shot image classification method based on hierarchical feature fusion in an embodiment of the present invention; Figure 2 It is a system block diagram of a combined zero-shot image classification model based on hierarchical feature fusion in an embodiment of the present invention; Figure 3 Schematic diagram of the structure of a multi-layer visual feature fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments: In this embodiment, a combined zero-sample image classification method based on hierarchical feature fusion is proposed, which innovatively proposes a solution based on hierarchical feature fusion for the technical difficulties of combined zero-sample image classification. By selecting features of different depths in the CLIP model visual encoder and performing hierarchical fusion on them, multi-level and multi-scale visual information is effectively extracted, and the fused features are further used for cross-modal interaction of vision and semantics, which significantly improves the model's ability to understand complex combinations. In addition, the present invention effectively separates attributes and object features by introducing specific loss terms to avoid feature aliasing problems. In the combined zero-sample image classification task, the method of the present invention exhibits excellent generalization ability and recognition effect, and can significantly improve the model's classification performance for unknown complex combination images.

[0014] like Figure 1 As shown, a combined zero-shot image classification method based on hierarchical feature fusion specifically includes the following steps: Step 1. Build a combined zero-shot image classification model based on hierarchical feature fusion, including CLIP text encoder, CLIP visual encoder, multi-layer visual feature fusion module, cross-modal interaction module, decoupling module and loss calculation module.

[0015] Step 2. Preliminary construction of embedding representation: Generate preliminary embedding representations, i.e., word embeddings, for attributes, objects, and combinations. The preliminary embedding representations will be updated as parameters during the back-propagation process. Combine the preliminary embedding representations of attributes, objects, and combinations with the prompt prefix embedding, and obtain the text features of attributes, objects, and combinations through the pre-trained CLIP text encoder to capture semantic information and enhance the expressiveness of text features.

[0016] Step 2.1. For the attribute collection , object collection and combined collections , first through the word embedding function of CLIP Obtain preliminary embedding representations for each attribute, object, and their combination; in, , to Represents a collection of attributes 1st to 2nd properties, is the total number of attributes in the training set; , to Represents a collection of objects 1st to 2nd objects, represents the total number of objects in the training set; , Represents a collection of attributes Middle properties, Represents a collection of objects Middle objects, Representation attributes and objects combination.

[0017] The word embedding vectors of each attribute, object and their combination are represented as: , , ; in, , and The word embeddings representing attributes, objects and their combinations are respectively, that is, the preliminary embedding representations of attributes, objects and their combinations. The preliminary embedding representations of attributes, objects and their combinations will be updated as parameters in the subsequent back-propagation. In the process of training the combined zero-shot image classification model based on hierarchical feature fusion, the preliminary embedding representations of attributes, objects and their combinations will be iteratively updated and do not need to be regenerated.

[0018] Step 2.2. In order to enhance the semantic expression ability of embedding, a hint prefix mechanism is introduced.

[0019] Process each word in the prompt prefix using the word embedding function to obtain the set of prompt prefix embeddings ,in , to Respectively represent sets The embeddings of the 1st to kth context words in , Indicates the total number of context words in the prompt prefix.

[0020] The word embedding of the prompt prefix, i.e., the prompt prefix embedding, is concatenated with the word embedding of each attribute, object, and their combination to construct the complete input text embedding with the prompt prefix, which is expressed as: ; ; ; in, Word embeddings and attributes representing the hint prefix Word embedding The complete input text constructed after concatenation, Word embeddings and objects representing the hint prefix Word embedding The complete input text constructed after concatenation, Word embeddings and attributes representing the hint prefix and objects Combination of word embeddings The complete input text constructed after concatenation.

[0021] Step 2.3. Using CLIP text encoder Encode the complete input text embedding with the prompt prefix to obtain the text feature , text features Include attribute text features , object text features and combined text features , respectively expressed as: ; ; ; in, , and are the text features of attributes, objects and their combinations in the encoder output space, namely Figure 2 The attribute text vector, object text vector and combination text vector are shown. , and Contextual cues, namely contextual vocabulary, are combined to enhance the expression of complex semantics.

[0022] Step 3. Image feature extraction and multi-layer feature representation: The images in the training set are passed into the CLIP visual encoder as input images. The image features are processed layer by layer through a multi-level transformer structure to capture the visual information of the image at different scales and semantic levels, and features of different levels are extracted from the CLIP visual encoder.

[0023] Specifically, the images in the training set are passed as input images to the CLIP visual encoder. The visual encoder gradually extracts the features of the image through multiple levels of transformer models. In these levels, the low-level layers mainly focus on details such as edges, textures, etc., while the high-level layers focus on more abstract semantic information such as object categories, scene content, etc. In this way, the model can fully understand the input image at multiple scales and semantic levels, providing rich visual representations for subsequent feature fusion and cross-modal interaction.

[0024] Step 4. Multi-level feature fusion: Filter out the initial layer features, intermediate layer features and final layer features from the CLIP visual encoder, and calculate the weights of features at different levels, i.e. weighting coefficients, through the multi-layer visual feature fusion module, perform weighted combination of features at different levels, and construct visual fusion features.

[0025] The multi-layer visual feature fusion module adopts a four-layer network architecture, including two fully connected layers, a Relu activation layer and a Softmax normalization layer. It uses a multi-head attention module to perform weighted combination of features at different levels to ensure that the fused features can retain both low-level detail information and high-level abstract semantics.

[0026] Step 4.1. Based on the features of different levels extracted from the CLIP visual encoder in step 3, define the initial layer features output by the CLIP visual encoder , middle layer features and the final layer features They are: ; ; ; in, , , Represent the initial layer features The feature map height, width, and number of channels; , , Represent the middle layer features The feature map height, width, and number of channels; , , Represent the final layer features The feature map height, width, and number of channels.

[0027] Step 4.2. Resize each feature map to a uniform spatial size by upsampling or pooling : ; in, represents the initial layer features after reshaping, represents the reshaped intermediate layer features, represents the final layer features after reshaping, represents the height of the reshaped feature map, represents the width of the reshaped feature map, Indicates the number of feature map channels after reshaping.

[0028] Step 4.3. Flatten and concatenate the reshaped features of each layer into a long vector: ; in, .

[0029] The dimension of the concatenated feature vector is: .

[0030] in, .

[0031] Step 4.4. In order to learn the weight coefficients of each feature, first pass the concatenated feature vector through the fully connected layer Map to an intermediate dimension: ; .

[0032] Then through the ReLU activation function we get: ; in, .

[0033] Another fully connected layer is used to map the activated features to three weighting coefficients: ; in, represents the weighting coefficient, , represents the weight matrix, Represents the bias term.

[0034] Then the weighted coefficients are normalized by the Softmax function to ensure that the sum of the weighted coefficients is 1: ; in, is the weight coefficient of the initial layer feature, is the weighting coefficient of the final layer feature.

[0035] Finally, according to the weighted coefficients obtained by learning , , The features of each layer are weighted and summed to obtain the visual fusion features : .

[0036] Through this weighted fusion method, the combined zero-shot image classification model based on hierarchical feature fusion can adaptively learn how to combine features at different levels, thereby improving the classification performance of the model for downstream tasks.

[0037] Step 5. Cross-modal interaction to optimize text features: Input the text features obtained in step 2 and the visual fusion features constructed in step 4 into the cross-modal interaction module. Through the cross-attention mechanism, deep fusion of visual fusion features and text features is achieved to generate optimized text features that can better reflect visual information, improve the precise description of image semantics, and obtain optimized text features of attributes, objects, and combinations.

[0038] Step 5.1. Text features obtained from step 2 And the visual fusion features obtained in step 4 , using cross-attention mechanism for interaction.

[0039] First, generate the query vector, key vector, and value vector.

[0040] The original text features Perform linear mapping to obtain the query vector : ; in, .

[0041] Visual fusion features Perform linear mapping to obtain the key vector : ; in, Represents the visual fusion feature The weight matrix for linear mapping.

[0042] Calculate the cross attention between text features and visual fusion features : ; in, represents the normalized exponential function; Represents the dimension of the key, usually scaling the result to avoid numerical instability.

[0043] After the cross attention calculation, the original text features are connected through the residual connection The output of the cross attention is fused, and then the cross-modal optimized text features are obtained through normalization and multi-layer perceptron mapping. .

[0044] Step 5.2. After obtaining the cross-modal optimized text features Then, through weighted fusion, With the original text features Fusion : ; in, represents the weight parameter, is an adjustable weight parameter used for cross-modal optimized text features With the original text features Relative importance in fusion.

[0045] Fusion text features Normalize by L2 norm to get the final optimization result, that is, optimize the text features : .

[0046] Optimizing text features It includes optimized attribute text features, optimized object text features and optimized combined text features, such as Figure 2 The optimized attribute text vector, optimized object text vector and optimized combination text vector are shown.

[0047] Step 6. Feature decoupling and multimodal representation: Decouple the image features output by the last layer of the CLIP visual encoder. The decoupling module uses three trainable multi-layer perceptrons (MLPs) to extract attribute visual features, object visual features, and combined visual features from the image features output by the last layer of the CLIP visual encoder. Figure 2 The attribute visual vector, object visual vector, and combination visual vector are shown. These decoupled features ensure the independence of attributes, objects, and their combination to better adapt to zero-shot combination tasks.

[0048] Specifically, the image feature output by the last layer of the CLIP visual encoder obtained in step 3 is the image embedding vector Based on three trainable multi-layer perceptrons , , Extract attribute visual features respectively , object visual features and combined visual features : ; ; .

[0049] Step 7. The loss calculation module calculates the similarity of the visual features of the attributes, objects, and combinations obtained after decoupling in step 6 and the text features of the optimized attributes, objects, and combinations obtained in step 5 using cosine similarity to obtain the predicted value, and measures the difference between the predicted value and the true label through the classification loss function to form the loss of each branch. At the same time, the covariance loss term is introduced to optimize the decoupling effect, and the total loss is obtained by weighted summation.

[0050] Step 7.1. Attribute visual features obtained from step 6 , object visual features and combined visual features , and the optimized attribute, object, and combined text features obtained in step 5, respectively calculate the cosine similarity of the attribute visual features, object visual features, combined visual features and the optimized text features of the corresponding branches of all candidate pairs : ; in, and Represent the visual features and text features of the same branch respectively. The visual features Including attribute visual features, object visual features and combined visual features, optimized text features Including optimized attribute text features, object text features and combined text features, represents the dot product, , Respectively, vector , The norm of is the modulus length.

[0051] Step 7.2. After the cosine similarity calculation is completed, it is normalized by the softmax function to obtain the predicted values ​​of the attribute branch, object branch and combination branch respectively.

[0052] In order to measure the difference between the predicted value and the true label, the classification loss function is introduced , which is expressed as: ; Among them, the xth input image is the xth sample, represents the predicted value of the xth sample, Represents the true label of the xth sample, through the classification loss function The cross entropy loss of each branch will be used as part of the total loss.

[0053] In addition, in order to optimize the effect of the decoupling module and promote better separation of attributes and objects, the present invention introduces covariance loss. Through covariance loss, the model can reduce the similarity between the visual features of attributes and objects to strengthen their decoupling, so that attributes and objects can learn more independent representations respectively.

[0054] For the x-th input image, the visual features , using the decoupled attribute visual features of the x-th input image obtained in step 5 and object visual features ,get and The covariance loss between It is expressed as: ; Among them, s is the number of samples of the category corresponding to the x-th input image, is the mean value of the attribute visual features in the samples of the category corresponding to the x-th input image, is the mean of the object visual features in the samples of the category corresponding to the x-th input image. By minimizing the covariance loss, the network will be guided to learn to make the attribute visual features and the object visual features as unrelated as possible, thereby achieving a better decoupling effect.

[0055] Step 7.3. Calculate the losses of the attribute branch, object branch, and combination branch through the classification loss function, introduce the attribute-object covariance loss, and obtain the total loss through weighted summation.

[0056] Total loss It is expressed as: ; in, , and are the losses of the attribute branch, object branch, and combination branch, respectively. is the attribute-object covariance loss; , , , The weight coefficients corresponding to each loss item are used to balance the impact of different loss items.

[0057] Step 8. Model back propagation and parameter update: Through the back propagation mechanism of the total loss, the weight coefficients of the multi-layer visual feature fusion module, the parameters of the MLP used for decoupling, and the preliminary embedding representations of attributes, objects, and combinations are jointly updated, thereby gradually improving the representation and generalization capabilities of the model.

[0058] Specifically, based on the total loss obtained in step 7, through the back-propagation mechanism, the trainable parameters of each module, including the weighted coefficients of the multi-layer visual feature fusion module, the parameters in the MLP used for decoupling, and the preliminary embedding representations of attributes, objects, and combinations, will be jointly updated according to the loss function to improve the representation and generalization capabilities of the combined zero-shot image classification model based on hierarchical feature fusion.

[0059] Back propagation starts with the total loss, first calculating the gradient of the output layer, and propagating these gradients forward layer by layer to the weights and bias parameters of each layer. For each layer, the gradient is used to update the weight parameters of the layer, with the goal of minimizing the loss function.

[0060] Specifically, for the multi-layer visual feature fusion module, the loss will be reflected in the gradient calculation through the weighted coefficient, so as to adjust the weighted coefficient to make the fusion of visual features more reasonable, thereby improving the feature expression ability of the model.

[0061] For the MLP network in the decoupling module, by calculating the impact of covariance loss on MLP weights, back propagation will adjust the parameters in the MLP so that the correlation between attributes and object features is further reduced, thereby better achieving the decoupling of attributes and objects.

[0062] Most importantly, the initial embedding representations of attributes, objects, and combinations are also affected by the loss and updated in reverse. Through these updates, the model gradually fine-tunes and optimizes the initial embedding representations of attributes, objects, and combinations, namely word embeddings, to make them more in line with the requirements of downstream combined zero-shot image classification tasks.

[0063] Step 9. Iteratively optimize the embedded representation: By repeatedly iterating steps 3 to 8, the embedded representations and parameters of attributes, objects, and combinations in the combined zero-shot image classification model based on hierarchical feature fusion are optimized in each round of training; in each iteration, the model performs forward propagation, prediction and calculates the loss based on the training data, and updates the parameters of each module through back propagation of the loss, including the weighting coefficients of the multi-layer visual feature fusion module, the MLP weights for decoupling, and the preliminary embedded representations of attributes, objects, and combinations; compared with the original model, the fine-tuned model can have stronger generalization ability and higher classification accuracy in the combined zero-shot image classification task.

[0064] Specifically, by iteratively executing steps 3 to 8, the preliminary embedding representations and related parameters of attributes, objects, and combinations will be continuously optimized in each round of training; in each iteration, the model performs forward propagation and prediction based on the training data, and then calculates the total loss and backpropagates to update the parameters of each module, including the weighting coefficients of the multi-layer visual feature fusion module, the MLP weights for decoupling, and the preliminary embedding representations of attributes, objects, and combinations.

[0065] After each iteration, the combined zero-shot image classification model based on hierarchical feature fusion is evaluated on the validation set to monitor its performance on unseen categories; as the loss function gradually converges, the performance of the model on the validation set continues to improve, and the initial embedding representations of attributes, objects, and combinations are continuously updated as parameters in each back propagation, and finally the optimized embedding representations of attributes, objects, and combinations, as well as the trained combined zero-shot image classification model based on hierarchical feature fusion, are obtained, which is the optimal model. Compared with the original model, the fine-tuned model has stronger generalization ability and higher classification accuracy in the combined zero-shot image classification task.

[0066] Step 10. Use the trained combined zero-shot image classification model based on hierarchical feature fusion to classify the input image.

[0067] In addition, in order to verify the effectiveness of the method proposed in the present invention, the combined zero-sample image classification model based on hierarchical feature fusion proposed in the present invention is also used for image classification on the ut-zappos data set. After experimental verification, the image classification method of the present invention has a visible combination accuracy, unknown combination accuracy, best harmonic mean and AUC value of 69.99%, 76.41%, 55.98% and 0.4511, respectively, which are much higher than the best levels of 66.8%, 73.8%, 54.6% and 0.4177 of all open source models at home and abroad. The image classification method of the present invention shows excellent generalization ability and recognition effect, and can significantly improve the classification performance of the model for unknown complex combination images.

[0068] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A combined zero-shot image classification method based on hierarchical feature fusion, characterized in that: The steps include: Step 1. Build a combined zero-shot image classification model based on hierarchical feature fusion, including CLIP text encoder, CLIP visual encoder, multi-layer visual feature fusion module, cross-modal interaction module, decoupling module and loss calculation module; Step 2. Generate preliminary embedding representations, i.e., word embeddings, for attributes, objects, and combinations. The preliminary embedding representations will be updated as parameters during the back-propagation process. Combine the preliminary embedding representations of attributes, objects, and combinations with the prompt prefix embeddings to obtain the text features of attributes, objects, and combinations through the pre-trained CLIP text encoder. Step 3. Pass the image in the training set as the input image into the CLIP visual encoder, process the features of the image layer by layer, and extract features of different levels from the CLIP visual encoder; Step 4. Filter out the initial layer features, intermediate layer features, and final layer features from the CLIP visual encoder, and calculate the weights of features at different levels, i.e., weighted coefficients, through a multi-layer visual feature fusion module, perform weighted combination of features at different levels, and construct visual fusion features; Step 5. Input the text features obtained in step 2 and the visual fusion features constructed in step 4 into the cross-modal interaction module, and achieve deep fusion of visual fusion features and text features through the cross-attention mechanism to obtain optimized attribute, object, and combined text features; Step 6. Decouple the image features output by the last layer of the CLIP visual encoder. The decoupling module uses three trainable multi-layer perceptrons (MLPs) to extract attribute visual features, object visual features, and combined visual features from the image features output by the last layer of the CLIP visual encoder. Step 7. The loss calculation module calculates the similarity of the visual features of the attributes, objects, and combinations obtained after decoupling in step 6 and the text features of the optimized attributes, objects, and combinations obtained in step 5 using cosine similarity to obtain the predicted value, and measures the difference between the predicted value and the true label through the classification loss function to form the loss of each branch. At the same time, the covariance loss term is introduced to optimize the decoupling effect, and the total loss is obtained by weighted summation; Step 8. Through the back-propagation mechanism of the total loss, the weight coefficients of the multi-layer visual feature fusion module, the parameters of the MLP for decoupling, and the preliminary embedding representations of attributes, objects, and combinations are jointly updated; Step 9. Repeat steps 3 to 8, and evaluate the combined zero-shot image classification model based on hierarchical feature fusion after each iteration on the validation set until the loss function converges. According to the validation set results, the trained combined zero-shot image classification model based on hierarchical feature fusion and the optimized embedding representations of attributes, objects, and combinations are obtained. Step 10. Use the trained combined zero-shot image classification model based on hierarchical feature fusion to classify the input image.

2. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.

1. For the attribute collection , object collection and combined collections , first through the word embedding function of CLIP Obtain preliminary embedding representations for each attribute, object, and their combination; The preliminary embedding representations of attributes, objects, and combinations will be updated as parameters during the back-propagation process; in, , to Represents a collection of attributes 1st to 2nd properties, is the total number of attributes in the training set; , to Represents a collection of objects 1st to 2nd objects, represents the total number of objects in the training set; , Represents a collection of attributes Middle properties, Represents a collection of objects Middle objects, Representation attributes and objects combination of; The word embedding vectors of each attribute, object and their combination are represented as: ， ， ； in, , and Word embeddings representing attributes, objects, and their combinations, i.e., preliminary embedding representations of attributes, objects, and combinations; Step 2.

2. In order to enhance the semantic expression ability of embedding, a hint prefix mechanism is introduced; Process each word in the prompt prefix using the word embedding function to obtain the set of prompt prefix embeddings ,in , to Respectively represent sets The embeddings of the 1st to kth context words in , represents the total number of context words in the prompt prefix; The word embedding of the prompt prefix, i.e., the prompt prefix embedding, is concatenated with the word embedding of each attribute, object, and their combination to construct the complete input text embedding with the prompt prefix, which is expressed as: ； ； ； in, Word embeddings and attributes representing the hint prefix Word embedding The complete input text constructed after concatenation, Word embeddings and objects representing the hint prefix Word embedding The complete input text constructed after concatenation, Word embeddings and attributes representing the hint prefix and objects Combination of word embeddings The complete input text constructed after splicing; Step 2.

3. Using CLIP text encoder Encode the complete input text embedding with the prompt prefix to obtain the text feature , text features Include attribute text features , object text features and combined text features , respectively expressed as: ； ； ； in, , and are the text features of attributes, objects and their combination in the encoder output space, respectively.

3. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: The step 3 is specifically as follows: Pass the images in the training set as input images into the CLIP visual encoder; The CLIP visual encoder gradually extracts features of the image through multiple levels of transformer models, where low-level layers focus on details and high-level layers focus on more abstract semantic information.

4. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: In step 4, the multi-layer visual feature fusion module adopts a four-layer network architecture; The multi-layer visual feature fusion module consists of two fully connected layers, a Relu activation layer and a Softmax normalization layer.

5. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: The step 4 is specifically as follows: Step 4.

1. Define the initial layer features of the CLIP visual encoder output , middle layer features and the final layer features They are: ； ； ； in, , , Represent the initial layer features The feature map height, width, and number of channels; , , Represent the middle layer features The feature map height, width, and number of channels; , , Represent the final layer features The feature map height, width, and number of channels; Step 4.

2. , and The feature maps are adjusted to a uniform spatial size by upsampling or pooling : ； in, represents the initial layer features after reshaping, represents the reshaped intermediate layer features, represents the final layer features after reshaping, represents the height of the reshaped feature map, represents the width of the reshaped feature map, Indicates the number of feature map channels after reshaping; Step 4.

3. Reshape the features of each layer , and Flatten and concatenate into one long vector : ； in, Indicates connection, Indicates flattening processing; The concatenated feature vector The dimensions are: ; in, Superscript Represents the feature vector The number of concatenated features in ; Step 4.

4. To learn the weight coefficients of features at different levels, first pass the concatenated feature vector through a fully connected layer Map to an intermediate dimension: ； in, represents the features after mapping, Represents the weight matrix of the mapping process, represents the bias term; Then through the ReLU activation function we get: ； in, ; Use another fully connected layer to transform the activated features Mapped to three weighting coefficients: ； in, represents the weighting coefficient, , represents the weight matrix, represents the bias term; Then the weighted coefficients are normalized by the Softmax function to ensure that the sum of the weighted coefficients is 1: ； in, is the weight coefficient of the initial layer feature, is the weight coefficient of the final layer feature; Finally, according to the weighted coefficients obtained by learning , , The features of each layer are weighted and summed to obtain the visual fusion features : 。 6. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: The step 5 is specifically as follows: Step 5.

1. Text features obtained from step 2 And the visual fusion features obtained in step 4 , using cross-attention mechanism for interaction; First, generate the query vector, key vector, and value vector; The original text features Perform linear mapping to obtain the query vector : ； in, Represents the original text features The weight matrix for linear mapping; Visual fusion features Perform linear mapping to obtain the key vector : ； in, Represents the visual fusion feature The weight matrix for linear mapping; Calculate the cross attention between text features and visual fusion features : ； in, represents the normalized exponential function; Indicates the dimension of the key, used to avoid numerical instability and scale the results; After the cross attention calculation, the original text features are connected through the residual connection The output of the cross attention is fused, and then the cross-modal optimized text features are obtained through normalization and multi-layer perceptron mapping. ; Step 5.

2. After obtaining the cross-modal optimized text features Then, through weighted fusion, With the original text features Fusion : ； in, represents the weight parameter; Fusion text features Normalize by L2 norm to get optimized text features : 。 7. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: The step 6 is specifically as follows: The image feature obtained in step 3, output by the last layer of the CLIP visual encoder, is the image embedding vector Based on three trainable multi-layer perceptrons , , Extract attribute visual features respectively , object visual features and combined visual features : ； ； 。 8. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: The step 7 is specifically as follows: Step 7.

1. Attribute visual features obtained from step 6 , object visual features and combined visual features , and the optimized attribute, object, and combined text features obtained in step 5, respectively calculate the cosine similarity of the attribute visual features, object visual features, combined visual features and the optimized text features of the corresponding branches of all candidate pairs : ； in, and Represent the visual features and text features of the same branch respectively. The visual features Including attribute visual features, object visual features and combined visual features, text features Including optimized attribute text features, object text features and combined text features, represents the dot product, , Respectively, vector , The norm of is the modulus length; Step 7.

2. After the cosine similarity is calculated, the predicted values ​​of the attribute branch, object branch, and combination branch are obtained by normalization through the softmax function; In order to measure the difference between the predicted value and the true label, the classification loss function is introduced , expressed as: ； Among them, the xth input image is the xth sample, represents the predicted value of the xth sample, Represents the true label of the xth sample; through the classification loss function The cross entropy loss of each branch obtained will be used as part of the total loss; Introducing covariance loss, for the visual features of the xth input image , using the decoupled attribute visual features of the x-th input image obtained in step 5 and object visual features ,get and The covariance loss between It is expressed as: ； Among them, s is the number of samples of the category corresponding to the x-th input image, is the mean value of the attribute visual features in the samples of the category corresponding to the x-th input image, is the mean value of the visual features of the objects in the samples of the category corresponding to the x-th input image; Step 7.

3. Calculate the losses of the attribute branch, object branch, and combination branch through the classification loss function, introduce the attribute-object covariance loss, and obtain the total loss through weighted summation; Total loss It is expressed as: ； in, , and Represent the losses of attribute branch, object branch and combination branch respectively, is the attribute-object covariance loss; , , , The weight coefficients corresponding to each loss item are used to balance the impact of different loss items.

9. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: The step 8 is specifically as follows: Based on the total loss obtained in step 7, through the back-propagation mechanism, the trainable parameters of each module, including the weight coefficients of the multi-layer visual feature fusion module, the parameters in the MLP used for decoupling, and the preliminary embedding representations of attributes, objects, and combinations, will be jointly updated according to the loss function to improve the representation and generalization capabilities of the combined zero-shot image classification model based on hierarchical feature fusion; Back propagation starts from the total loss, first calculating the gradient of the output layer, and propagating the gradient forward layer by layer to the weights and bias parameters of each layer; for each layer, the gradient is used to update the weight parameters of the layer to reduce the loss function; For the multi-layer visual feature fusion module, the loss will be reflected in the gradient calculation through the weighted coefficient, so the weighted coefficient is adjusted to improve the feature expression ability of the model; For the MLP in the decoupling module, by calculating the impact of the covariance loss on the MLP weights, back-propagation adjusts the parameters in the MLP so that the correlation between attributes and object features is further reduced to achieve the decoupling of attributes and objects; The preliminary embedding representations of attributes, objects, and combinations are also affected by the loss and are gradually updated during the back-propagation process to meet the requirements of the downstream combinatorial zero-shot image classification task.

10. The combined zero-shot image classification method based on hierarchical feature fusion according to claim 1, characterized in that: The step 9 is specifically as follows: By repeatedly iterating steps 3 to 8, the embedded representations and parameters of attributes, objects, and combinations in the combined zero-shot image classification model based on hierarchical feature fusion are optimized in each round of training; In each iteration, the model performs forward propagation based on the training data, makes predictions and calculates the loss. The parameters of each module are updated through the back propagation of the loss, including the weight coefficients of the multi-layer visual feature fusion module, the MLP weights for decoupling, and the preliminary embedding representations of attributes, objects, and combinations. After each iteration, the combined zero-shot image classification model based on hierarchical feature fusion is evaluated on the validation set to monitor its performance on unseen categories; As the loss function gradually converges, the performance of the model on the validation set continues to improve. When the loss of the model on the validation set converges, we will eventually obtain the optimized embedding representation of attributes, objects, and combinations and the trained combined zero-shot image classification model based on hierarchical feature fusion.

Images