A generalized zero-shot image recognition method and system fusing attention feature aggregation and probability modeling

Abstract

The application discloses a generalized zero-shot image recognition method and system fusing attention feature aggregation and probability modeling. The method comprises the following steps: extracting image features by using a pre-trained visual backbone network, obtaining image representation considering global semantics and local details by adopting a splicing mode of global semantic labeling and attention weighted image block aggregation, and projecting the image representation to a semantic attribute space corresponding to the image through a learnable mapping module; explicitly parameterizing the conditional probability of the labeled state of samples, deriving a joint likelihood objective function of the training set and the mixed test set samples, and constructing a probability modeling framework for mixed distribution; and through end-to-end training, the mapping model simultaneously completes visual-semantic alignment and probability decoupling of seen classes and unseen classes. The application starts from the essential perspective of label incompleteness, alleviates the seen class bias problem in generalized zero-shot learning through mixed distribution probability modeling, and does not need to train with unseen class data, and has the characteristics of flexibility and practicality.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and weakly supervised learning technology, specifically to a generalized zero-shot image recognition method and system that integrates attention feature aggregation and probabilistic modeling. Background Technology

[0002] In image recognition tasks, traditional supervised learning methods rely on a large number of manually labeled samples for training, and completely lose the ability to recognize new categories that have not appeared during the training phase. To solve this problem, Zero-Shot Learning (ZSL) has emerged. Its core idea is to establish a mapping relationship between the visual feature space and the semantic space by using auxiliary knowledge such as semantic attributes and word vectors. This allows the model to recognize categories that have never been seen during training based on the semantic description of the categories, thereby eliminating the dependence on labeled data for unseen categories.

[0003] However, standard zero-shot learning only classifies unseen samples during evaluation, which differs significantly from real-world applications. In practical deployments, test samples often come from a mixed distribution of seen and unseen samples, requiring the model to simultaneously identify both classes. To address this, Generalized Zero-Shot Learning (GZSL) was proposed, requiring the model to correctly classify both seen and unseen samples within a unified label space. However, because only seen samples carry labeled information during training, the model develops a strong bias towards seen classes. During testing, samples from unseen classes are easily misassigned to seen classes, a problem known as "seen class bias," which remains a core challenge in the GZSL field. Fundamentally, GZSL is a learning problem with incomplete labels. During training, only seen samples carry labels, while test samples come from a mixed distribution of seen and unseen classes. Existing methods mostly address this issue from the perspective of feature enhancement or data generation, rarely explicitly modeling this problem from the essential perspective of mixed distribution probability modeling.

[0004] Meanwhile, the rapid development of large-scale pre-trained visual backbone networks, represented by DINOv2, has provided visual recognition tasks with a far superior general visual representation capability compared to traditional convolutional networks. DINOv2, based on the Vision Transformer (ViT) architecture, is self-supervised pre-trained on large-scale unlabeled data. Its output features exhibit significant semantic structure and cross-class transferability, demonstrating strong generalization capabilities across various downstream visual tasks. However, existing GZSL schemes based on DINOv2 lack explicit probabilistic modeling mechanisms for mixed test distributions, failing to theoretically guarantee balanced recognition of seen and unseen class samples.

[0005] Therefore, it is necessary to propose a new approach that combines a probabilistic modeling framework for mixed distributions with a powerful pre-trained visual backbone network to alleviate the seen-class bias problem. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a generalized zero-shot image recognition method and system that integrates attention feature aggregation and probabilistic modeling, which effectively alleviates the bias problem of seen classes and improves the overall performance of generalized zero-shot image recognition without relying on unseen class training data.

[0007] Technical solution: A generalized zero-shot image recognition method that integrates attention feature aggregation and probabilistic modeling, comprising the following steps:

[0008] S1. Construct a training set from the labeled images of the known categories. The test set consists of unlabeled images containing a mixture of seen and unseen samples. Get all The prototype vectors of semantic attributes for each category are arranged in order of seen classes first and unseen classes last, forming a prototype matrix. ,in The number of seen classes, The number of unseen classes. For semantic attribute dimensions;

[0009] S2, for the input image Visual features are extracted through a pre-trained visual backbone network and then projected onto a learnable mapping module. In a 3D semantic space, we obtain a semantic mapping vector. ,in These are the learnable network parameters for the mapping module;

[0010] S3. Based on the similarity between semantic mapping vectors and category prototype vectors Based on this, parameterize the label conditional probability of each sample for the known class:

[0011]

[0012]

[0013] For unseen classes:

[0014]

[0015]

[0016] in This indicates the labeled status of the sample; 1 represents a labeled sample, and -1 represents an unlabeled sample. This indicates that the true category of the sample is the first. kind, For the Sigmoid function;

[0017] Based on similarity Introducing a learnable categorical temperature coefficient Calculate the total number of samples for each sample. Softmax classification probabilities on semantic prototypes of each category ;

[0018] By maximizing the joint log-likelihood of the training and test sets, the end-to-end optimization objective is derived. Based on the label conditional probability and classification probability, a total loss function containing the probabilistic modeling main loss and multiple auxiliary losses is constructed for end-to-end training.

[0019] S4. Input the image to be recognized into the trained model and calculate the image to be recognized. Normalized semantic mapping vectors and all categories The cosine similarity between normalized semantic prototypes is used, and the category corresponding to the maximum value is taken as the predicted category.

[0020] Furthermore, the pre-trained visual backbone network is the DINOv2 ViT-L / 14 model, which processes the input image... The visual feature extraction employs a method combining global semantic labeling and attention-weighted image patch aggregation and concatenation, specifically including:

[0021] By forward propagating through the backbone network, the global semantic labeling features of the last layer are extracted. and the feature sequences of all N image patch labels N is the number of image patches;

[0022] Utilizing the query projection weight matrix frozen in the last attention layer of the backbone network With key projection weight matrix Manually calculate the attention weights of the global semantic tags for each image patch tag:

[0023]

[0024] Where H represents the number of attention heads. In order to target the The query vector with globally semantically labeled data extracted by head attention. In order to target the The key matrix of all image patches extracted by head attention; the fusion weights are obtained by averaging the attention weights of all heads. Utilizing fusion weights Weighted aggregation of image patch marker feature sequences yields image patch aggregated features. Finally, the global semantic tagging features and image patch aggregation features are concatenated along the channel dimension to obtain the final visual features. , as input to the mapping module;

[0025] The main parameters of the backbone network are kept frozen during training, and only the parameters of the last layer (LayerNorm) are unfrozen to accommodate the drift of the downstream semantic distribution.

[0026] Furthermore, the mapping module consists of an input normalization layer and a multilayer perceptron connected in series, organized in the following order:

[0027] Input normalization layer: normalizes the input visual features Apply layer normalization to eliminate feature scale differences;

[0028] The first linear transformation layer: The fully connected layer compresses the feature dimension from the input dimension to 1024. It is followed by a normalization layer, a LeakyReLU activation layer with a specified negative slope, and a Dropout layer to complete the dimension alignment and non-linear transformation.

[0029] The residual block contains a fully connected layer that maintains 1024 dimensions, followed by a normalization layer, a LeakyReLU activation layer, and a Dropout layer. The block output is added to the block input element by element to preserve the feature orientation of the previous layer and to mine fine-grained semantic associations during semantic compression.

[0030] Output linear transformation layer: The fully connected layer compresses the feature dimension from 1024 to the target semantic attribute dimension. Output the original semantic mapping vector;

[0031] All parameters of the mapping module participate in end-to-end gradient optimization, and the output vector is subjected to... Normalization process.

[0032] Furthermore, the total loss function, which includes the probabilistic modeling main loss and multiple auxiliary losses, is as follows:

[0033]

[0034] in, The main loss is composed of the training set loss term. and test set loss terms constitute, and To mitigate losses, For semantic anchoring loss, For conditional prototype alignment loss, and For the corresponding weights;

[0035] by Indicates training samples The true category labels, and the training set loss term are:

[0036]

[0037] The test set loss term is expanded based on the law of total probability and uses logarithmic-summary-exponential operations:

[0038]

[0039] The semantic anchoring loss constrains the cosine distance between the semantic mapping vectors of seen class samples in the training set and their true class semantic prototypes:

[0040]

[0041] The conditional prototype alignment loss models the unseen class posterior weights estimated by the framework itself using a mixture distribution probability model. To adapt the weights, a weighted soft classification loss within the unseen class subspace is applied to the test set samples:

[0042]

[0043] in The most recently unseen class prototype; For the posterior weights of the mixed distribution, by The intermediate quantities during the forward propagation process are obtained directly, among which... This indicates the probability mass of a test sample belonging to a known class that has not been labeled. This indicates the probability mass of the test sample belonging to the unseen class; when the test sample actually belongs to the seen class, Approaching zero Automatically approaches zero.

[0044] Furthermore, a dual-stream balanced sampling strategy is adopted during the training phase to optimize the training set. With test set The samples are mixed in equal numbers in each batch, where each batch is drawn from two independent, randomly shuffled index pools. Each sample was spliced ​​together and then randomly shuffled again within the batch. Batch size; when the number of available samples in an index pool is insufficient. The pool is then reused cyclically to replenish the required quantity, ensuring a consistent batch size for each training round; each sample in a batch carries a source identifier. 0 represents a test set sample, and 1 represents a training set sample, used to distinguish which sample should participate during training. still The calculation.

[0045] Furthermore, the optimization strategy for end-to-end training specifically includes: adjusting the mapping module parameters, the normalized parameters of the unfrozen layers in the backbone network, and the learnable temperature coefficients. The learning rate is set independently for three groups, with the learning rate for the layer normalization parameters being two orders of magnitude lower than that for the mapping module; a learning rate scheduling strategy combining linear preheating and cosine annealing is employed; the AdamW optimizer is used, and the mapping module parameters are adjusted accordingly. Weight decay; after each backpropagation step, the gradients of all trainable parameters are clipped to a fixed threshold according to the global norm.

[0046] Furthermore, in calculating the classification probability At that time, a margin penalty is applied to the similarity component corresponding to the seen class to suppress the bias of the seen class:

[0047]

[0048] in To fix the cosine margin hyperparameter; Let be the similarity components after applying margin penalties; then the Softmax classification probability is... And satisfy , for Normalized cosine similarity function.

[0049] A generalized zero-shot image recognition system that integrates attention feature aggregation and probabilistic modeling includes:

[0050] The data preparation module is used to construct a training set from labeled images of known categories. The test set consists of unlabeled images containing a mixture of seen and unseen samples. Get all The prototype vectors of semantic attributes for each category are arranged in order of seen classes first and unseen classes last, forming a prototype matrix. ,in The number of seen classes, The number of unseen classes. For semantic attribute dimensions;

[0051] The feature extraction and mapping module is used for the input image. Visual features are extracted through a pre-trained visual backbone network and then projected onto a learnable mapping module. In a 3D semantic space, we obtain a semantic mapping vector. ,in These are the learnable network parameters for the mapping module;

[0052] The end-to-end training module is used to measure the similarity between semantic mapping vectors and category prototype vectors. Based on this, parameterize the label conditional probability of each sample for the known class:

[0053]

[0054]

[0055] For unseen classes:

[0056]

[0057]

[0058] in This indicates the labeled status of the sample; 1 represents a labeled sample, and -1 represents an unlabeled sample. This indicates that the true category of the sample is the first. kind, For the Sigmoid function;

[0059] Based on similarity Introducing a learnable categorical temperature coefficient Calculate the total number of samples for each sample. Softmax classification probabilities on semantic prototypes of each category By maximizing the joint log-likelihood of the training and test sets, the end-to-end optimization objective is derived. Based on the label conditional probability and classification probability, a total loss function containing the probabilistic modeling main loss and multiple auxiliary losses is constructed, and end-to-end training is performed.

[0060] The inference module is used to input the image to be recognized into the trained model and calculate the recognition value of the image. Normalized semantic mapping vectors and all categories The cosine similarity between normalized semantic prototypes is used, and the category corresponding to the maximum value is taken as the predicted category.

[0061] The present invention also provides an electronic device, including one or more processors, a memory, and one or more programs stored in the memory, wherein when the program is executed by the processor, it implements the above-described generalized zero-shot image recognition method that integrates attention feature aggregation and probabilistic modeling.

[0062] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described generalized zero-shot image recognition method that integrates attention feature aggregation and probabilistic modeling.

[0063] Beneficial effects: (1) Starting from the essence of the label incompleteness problem in generalized zero-shot learning, this invention constructs a mixed distribution probability modeling framework, explicitly parameterizes the conditional probability of the sample label state, and directly models the training-test mixed distribution with maximum likelihood estimation, providing a clear probabilistic basis for mitigating the bias of the seen class at the theoretical level. (2) This method combines the mixed distribution probability modeling framework with a pre-trained visual backbone network, enabling the mapping module to directly learn the semantic alignment mode adapted to the mixed test distribution under the constraint of probability modeling loss, without the need for an independent post-processing calibration stage. The framework is simple and easy to extend to different backbone networks. (3) This invention achieves excellent harmonic mean indices on three generalized zero-shot learning standard benchmark datasets: AWA2, CUB, and SUN. The harmonic mean of AWA2 reaches 80.95%, the harmonic mean of CUB reaches 72.04%, and the harmonic mean of SUN reaches 59.55%, verifying the effectiveness and cross-dataset generalization ability of the proposed method. Attached Figure Description

[0064] Figure 1 The flowchart shows the overall framework of the generalized zero-shot image recognition method that integrates attention feature aggregation and probabilistic modeling provided by this invention.

[0065] Figure 2 This is a schematic diagram of the dual-stream balanced sampling strategy used in data preprocessing according to the present invention.

[0066] Figure 3 This is a thermal diagram of the CLS-Patch attention weights generated on several test images on the DINOv2 visual backbone network according to the present invention.

[0067] Figure 4 This is a visualization of the t-SNE dimensionality reduction distribution of semantic mapping vectors in this invention on the CUB and AWA2 test sets. Detailed Implementation

[0068] This invention proposes a generalized zero-shot image recognition framework that integrates attention feature aggregation and probabilistic modeling. The core idea is to address the label incompleteness problem between the training and test sets in generalized zero-shot learning by constructing a probabilistic modeling framework oriented towards mixed distributions. It utilizes a pre-trained visual backbone network to extract image representations that consider both global semantics and local details. A learnable mapping module projects visual features onto a semantic attribute space, and semantic similarity directly parameterizes the label conditional probability of samples, constructing an end-to-end optimization objective. During training, the joint distribution of the training and mixed test sets is explicitly modeled, thereby mitigating seen-class bias. Figure 1 As shown, the method of the present invention includes the following steps:

[0069] Step S1: Construct the data set and semantic attribute prototype matrix.

[0070] The training set is constructed from the labeled images of the known categories. The test set consists of unlabeled images containing a mixture of seen and unseen samples. In this embodiment of the invention, experiments were conducted on three generalized zero-shot learning benchmark datasets: AWA2, CUB, and SUN. All datasets followed the xlsa17 standard segmentation commonly used in the ZLS field. (Proposed) The Split (PS) protocol is used to divide the datasets, with the following specific configurations: The AWA2 dataset contains 50 animal categories, including 40 seen categories and 10 unseen categories, with a semantic attribute dimension of 85. The training set contains 23,527 images, and the test samples contain 5,882 images of seen categories and 7,913 images of unseen categories. The CUB dataset contains 200 fine-grained bird categories, including 150 seen categories and 50 unseen categories, with a semantic attribute dimension of 312. The training set contains 7,057 images, and the test samples contain 1,764 images of seen categories and 2,967 images of unseen categories. The SUN dataset contains 717 scene categories, including 645 seen categories and 72 unseen categories, with a semantic attribute dimension of 102. The training set contains 10,320 images, and the test samples contain 2,580 images of seen categories and 1,440 images of unseen categories. Table 1 shows the basic configuration of the zero-sample standard datasets used in this method.

[0071] Table 1 Basic Configuration of the Generalized Zero-Shot Standard Dataset

[0072] CUB 200（150 / 50） 11,788 312 Fine-grained bird species with high semantic differentiation SUN 717（645 / 72） 14,340 102 Scene recognition shows significant inter-class differences. AWA2 50（40 / 10） 37,322 85 Animal classification, semantically compact

[0073] set up For categories that have already been seen, For unseen categories, the semantic attribute prototype vectors corresponding to each category are denoted as follows: and All data are derived from manually labeled attribute matrices provided by the dataset. Arranged in the order of seen classes first, followed by unseen classes, a prototype matrix is ​​constructed to obtain... ,in This represents the semantic attribute dimension. This arrangement ensures that the splitting operation of the seen class logit column in subsequent loss calculations strictly corresponds to the actual seen class prototype.

[0074] like Figure 2 As shown, a dual-stream balanced sampling strategy is used during the training phase, with the training set and the data stream being sampled separately. (Categories with labeled samples already seen) and the test set (For unlabeled samples of no class) Establish independent index pools. At the beginning of each training round, randomly shuffle both index pools, and extract samples from each pool in each batch. After merging the samples, they are randomly shuffled again within the batch, batch size. Image preprocessing uniformly adopts the DINOv2 standard workflow: training set images are sequentially and randomly cropped to... After random horizontal flipping and enhanced color jitter, the average value is calculated. Standard deviation Normalization is performed; the test set images are first scaled to... Cut to center The normalization parameters are the same.

[0075] Step S2: Extract visual features from the image using a pre-trained visual backbone network.

[0076] This method uses DINOv2 ViT-L / 14 as the visual backbone network and loads the pre-trained weights provided by the official documentation. For the input image... The backbone network divides it into Image patches of pixels, along with a learnable global semantic token (CLS token), are fed into a multi-layer Transformer encoder to produce the final output. Normalized global semantic tagging features and all N image patch token features Where N=256. The DINO model is based on the Transformer encoder. In the Transformer, the CLS token is a set of learnable vectors that aggregate all information within an image patch to represent the global features of the entire image. In DINO V2, it is input into the Transformer encoder along with all image patches as a representation of the overall image. In this paper, the CLS token input to the encoder is called the global semantic token, and the corresponding output is called the global semantic token feature or global semantic feature. A patch refers to a local image patch after the image has been segmented. The patch token is the sequence of feature vectors obtained after linear projection of each image patch, storing the local information of each image patch. In this paper, the patch token input to the encoder is called the image patch marker, and the corresponding output is the image patch marker feature or image patch feature.

[0077] Using only global semantic tagging results in the loss of local detail information, which is particularly detrimental to fine-grained datasets (such as CUB). Therefore, this invention utilizes the query projection weight matrix frozen in the last attention layer of the backbone network. With key projection weight matrix We manually calculate the attention weights of global semantic tagging for each image patch tag. Let the number of attention heads be H, and for each head... Extract query vectors from global semantic tags respectively Key matrix of all image patches Perform a scaled dot product:

[0078]

[0079] The fusion weights are obtained by averaging the attention weights of all heads. Then, the feature sequences of the Patch are weighted and aggregated:

[0080]

[0081] Features obtained from global semantic tag output Features obtained by aggregation with image patch labels By stitching along the channel dimension, the final visual features are obtained. It takes into account both the global semantics and local details of the image.

[0082] The main parameters of the backbone network are frozen throughout training, with only the scaling and bias parameters of the last LayerNorm being unfrozen to adapt to downstream semantic distribution shifts at minimal cost. Trainable parameters account for approximately 1% of the total backbone parameters, significantly reducing the risk of overfitting while preserving the general visual priors of DINOv2.

[0083] Step S3: Project the visual features into the semantic space using a learnable mapping module. The mapping module uses the visual features obtained in step S2. For input, organize in the following order:

[0084] First, the visual features are processed through an input normalization layer (LayerNorm). Scale normalization is performed to eliminate the amplitude difference between global semantic features and image patch features after concatenation. Then, a first fully connected layer compresses the dimension from 2048 to 1024, followed by LayerNorm, LeakyReLU activation with a negative slope of 0.2, and Dropout to complete dimensionality reduction and nonlinear transformation. Next, a residual block that maintains the 1024-dimensionality is passed through it. Its internal structure is fully connected layer → LayerNorm → LeakyReLU → Dropout, where the block output is added element-wise to the block input. , The input features for the residual block are... This is the output obtained from the transformation from the fully connected layer to the Dropout layer within the block. The final output of the residual block preserves the feature orientation of the previous layer while compressing the semantic dimension, preventing excessive distortion of the visual representation by deep transformations; finally, the output fully connected layer maps the 1024-dimensional features to the target semantic attribute dimension. Output the original semantic mapping vector.

[0085] This invention is based on Indicates the original input image End-to-end mapping to the semantic space, i.e., sequential visual feature extraction via step S2 (to obtain intermediate features). The semantic mapping vector obtained after projection of the mapping module in step S3, where For end-to-end learnable network parameters.

[0086] All parameters of the mapping module participate in end-to-end gradient optimization, and the output vector is first processed before participating in loss calculation. Normalization process, and the same process Performing cosine similarity calculations on normalized semantic prototype vectors ensures that the similarity range is constrained within a certain range. Within the interval, it provides a stable input for subsequent mixed distribution probability modeling.

[0087] Step S4: Construct a probability model and end-to-end loss function for mixed distributions.

[0088] This invention addresses the training set in generalized zero-shot learning. (Categories with labeled samples already seen) and the test set To address the label incompleteness problem among mixed unlabeled samples, a mixed distribution probability modeling framework is established, and the conditional probability of the labeled state of the sample is explicitly parameterized.

[0089] First, the sample The conditional probabilities of the labeled data are parameterized. For the previously seen classes... The cosine similarity between the semantic mapping vector and the prototype of that category, expressed by the Sigmoid function, is used as the label probability:

[0090]

[0091]

[0092] For unseen categories Based on the objective fact that no unlabeled samples of any class were seen during the training phase, the following is stipulated:

[0093]

[0094]

[0095] in Equivalent to , The labeling status is 1 (1 indicates a labeled sample, -1 indicates an unlabeled sample). For the Sigmoid function, for Normalized cosine similarity.

[0096] At the same time in all Softmax classification probabilities are defined on the semantic prototypes of each category. For seen classes, a fixed cosine margin penalty (CosFace margin penalty) is applied before calculating the similarity score to suppress the model's bias towards seen classes; no penalty is applied to unseen classes. The specific logit calculation is as follows:

[0097]

[0098]

[0099]

[0100] in For learnable classification temperature coefficients, To fix the cosine margin hyperparameter, the margin penalty only applies to the softmax classification path and does not affect the input of the sigmoid in the label conditional probability.

[0101] Using the idea of ​​maximum likelihood estimation, we maximize the joint probability function of samples from the training set S and the test set T:

[0102]

[0103] The above equation is equivalent to minimizing the following objective:

[0104]

[0105] According to the conditional probability formula, the training set terms are expanded as follows:

[0106]

[0107] Substituting the conditional probability formula for the known class labels into the Softmax classification probability formula, we obtain the training set loss term. :

[0108]

[0109] For each item in the test set, expand according to the law of total probability:

[0110]

[0111] By applying the unlabeled conditional probability formula for seen classes and the Softmax classification probability formula, and considering the rule that the labeled probability of unseen classes is zero, substituting these into the above total probability expansion, we obtain:

[0112]

[0113] Thus, the test set loss term is obtained. The summation within the logarithm is performed using logarithmic-summation-exponential operations to ensure numerical stability.

[0114]

[0115] and Together they constitute the main loss in probabilistic modeling .

[0116] exist Based on this, two auxiliary losses are introduced, and the complete optimization objective is:

[0117]

[0118] Among them semantic anchoring loss Constrain the cosine distance between the semantic mapping vectors of known class samples in the training set and their true class semantic prototypes:

[0119]

[0120] Conditional Prototype Alignment Loss The unseen class posterior weights estimated by the modeling framework itself using mixed distribution probability. To adapt the weights, a weighted soft classification loss within the unseen class subspace is applied to the test set samples:

[0121]

[0122] in The most recently unseen class prototype; For the posterior weights of the mixed distribution, by The intermediate quantities during the forward propagation process are obtained directly, among which... This indicates the probability mass of a test sample belonging to a known class that has not been labeled. This indicates the probability mass that the test sample belongs to the unseen class. When the test sample actually belongs to the seen class, Approaching zero Automatic zero-reaching makes Automatic elimination of gradient contributions from previously seen test samples to avoid interference with... This creates a contradiction with the modeling objectives. Weight coefficients Starting from the initial maximum value using the cosine scheduling strategy As training cycles progress, the performance monotonically decays to zero; initially, training involves significant reduction in intensity. Establish a stable semantic alignment foundation, and gradually hand over the initiative to the probabilistic modeling main loss in the later stages.

[0123] Step S5: End-to-end optimization training.

[0124] Divide all trainable parameters into three groups and set independent learning rates: mapping module parameters. (Including all parameters of the input normalization layer and the multilayer perceptron) The learning rate is set to The learning rate of the last LayerNorm parameter in the backbone network is set to [value missing]. Learnable temperature coefficient The learning rate is set to The AdamW optimizer is used, with a weight decay factor of . It only applies to the mapping module parameters; the backbone LayerNorm parameters and temperature coefficients are not subject to weight decay.

[0125] The learning rate scheduling employs a composite strategy combining linear preheating and cosine annealing, with preheating rounds... Total training rounds After each backpropagation step, the gradients of all trainable parameters are clipped to a threshold of 1.0 according to the global norm to prevent gradient explosion.

[0126] The main hyperparameter configurations for each dataset are as follows: batch size is 64 for all three datasets, and cosine margin is... Mark the temperature coefficient of the conditional probability path. , , The maximum value is 1.0, and the minimum value decays to 0. The experiment was conducted on a single NVIDIA RTX 4090 GPU and implemented using the PyTorch framework.

[0127] This invention employs a lightweight fine-tuning strategy that freezes the backbone and only unfreezes the last LayerNorm layer. The number of trainable parameters for each dataset in the embodiments is as follows: In the AWA2 dataset, the mapping module contains 3,243,093 parameters, and the backbone LayerNorm contains 2,048 parameters, totaling 3,245,141 trainable parameters, accounting for 1.05% of the total number of parameters in DINOv2 ViT-L / 14 (307,611,733); In the CUB dataset, due to its higher semantic attribute dimension (312 dimensions), the number of parameters in the mapping module increases to 3,475,768, with a total of 3,477,816 trainable parameters, accounting for 1.13% of the total number of parameters (307,844,408); In the SUN dataset, the total number of trainable parameters is 3,262,566, accounting for 1.06% of the total number of parameters (307,629,158). The proportion of trainable parameters in the three datasets is between 1.05% and 1.13%, which fully demonstrates the advantages of the lightweight design of this invention. While effectively preserving the visual prior of DINOv2 pre-trained, it significantly reduces the risk of overfitting and memory overhead.

[0128] Regarding training efficiency, the training time per round varies across the datasets. The AWA2 dataset contains approximately 735 batches per round, with a single round taking about 12 minutes; the CUB dataset contains 220 batches per round, with a single round taking about 3-4 minutes; and the SUN dataset contains 322 batches per round, with a single round taking about 2 minutes. All datasets underwent 30 rounds of training on a single NVIDIA RTX 4090 GPU, with the total training time for all three datasets kept under 10 hours, demonstrating good practicality.

[0129] Step S6, reasoning and classification.

[0130] Load the trained model parameters, extract the semantic mapping vector from the image to be recognized according to steps S2 and S3, and then... Normalization, compared with all categories Normalized semantic prototypes are cosine similarity calculated and then multiplied by the current... Obtain the classification prediction value.

[0131] During the inference phase, a biased scanning strategy is employed to search for the optimal decision equilibrium point on the pre-defined validation set within the xlsa17 protocol dataset. Since the model has only seen labeled samples of known classes during training, it naturally biases towards these known classes during inference, making it prone to misclassifying unseen samples as known classes. Therefore, a non-negative bias needs to be applied to the predicted values ​​of known classes. Compensation should be provided. The larger the value, the more the competitive advantage of seen classes is suppressed, the higher the accuracy of unseen classes becomes, and the lower the accuracy of seen classes becomes. Therefore, it is necessary to search on the validation set. The value of is chosen to optimize the overall performance of both. Specifically, within the interval... The bias values ​​are enumerated uniformly with a step size of 0.2. For each The previous class corresponding to the seen class Column predicted value minus The argmax statistical precision is calculated on both the seen class test set and the unseen class test set, and the harmonic mean is calculated.

[0132]

[0133] Take The largest As the optimal bias, then Used for evaluation on the test set, , and The accuracy of the classes already seen in the evaluation metrics of generalized zero-shot learning are respectively used as the evaluation metric. No class precision found and harmonic mean , Indicates at the bias Below, the model's prediction accuracy for known class samples in the validation set; Indicates at the bias Below, the model's prediction accuracy for samples of classes not seen in the validation set; This represents the harmonic mean of the two precision values ​​mentioned above, used to evaluate the overall accuracy of the model on the validation set.

[0134] As shown in Table 2, the observed class accuracy of this invention on the three standard benchmark datasets AWA2, CUB, and SUN is as follows: No class precision found and harmonic mean The experimental results were compared with existing representative inductive and transductive methods, among which... The accuracy of the model's predictions for categories already seen in the training set is measured. Measure the model's prediction accuracy for unseen categories. The harmonic mean, representing the overall balanced performance of the model on a mixed test set of seen and unseen classes, is a core evaluation metric in the field of generalized zero-shot learning. This invention employs a transductive approach, achieving optimal harmonic means on the three datasets: 80.95% (S=96.98%, U=69.46%) for AWA2, 72.04% (S=87.87%, U=61.04%) for CUB, and 59.55% (S=55.93%, U=63.68%) for SUN. Comparing the results across the three datasets, this invention achieves the best harmonic mean among all compared methods on the SUN dataset, the second-best harmonic mean on the AWA2 dataset, and a harmonic mean on the CUB dataset that is within 2.2 percentage points of the current best method. This demonstrates that this invention is highly competitive on multiple standard benchmark datasets.

[0135] AWA2 is a coarse-grained animal attribute dataset with clear semantic boundaries between classes. The visual features extracted by DINOv2 align well with the 85-dimensional attribute prototypes. The probabilistic modeling framework demonstrates strong ability to distinguish between seen and unseen classes, and the model converges and remains stable around the 24th epoch. CUB is a fine-grained bird dataset with subtle differences between the 150 seen classes and a high semantic attribute dimension of 312, making feature-attribute alignment challenging. The model was still in a slow improvement phase after 30 epochs of training, indicating room for further optimization in fine-grained scenarios. SUN is a large-scale scene dataset with relatively blurred boundaries in the classification space composed of 645 seen classes, while the 72 unseen classes are relatively sparsely distributed in the semantic space. The probabilistic modeling loss has a more significant effect on driving test samples to cluster towards the unseen class prototypes.

[0136] Table 2 Performance Comparison of Generalized Zero-Shot Image Recognition Methods (%)

[0137]

[0138] Note: Inductive / directive methods indicate the setting category; bold and underlined text indicates the best metric for each dataset; italic and underlined text indicates the second-best metric.

[0139] like Figure 3 As shown, the attention heatmap visually demonstrates the effect of the CLS-Patch attention-weighted aggregation design. Each dataset contains ten sets of images, and each set consists of the following rows from top to bottom: original image, attention overlay map, and attention distribution map. It can be seen that the model focuses on the body contour in animal images from AWA2, on key parts such as beaks and wings in bird images from CUB, and on discriminative texture regions in scene images from SUN, demonstrating good interpretability. Figure 4 As shown, the t-SNE visualization results indicate that after end-to-end training, the samples of each category in the semantic space exhibit a relatively compact intra-class clustering structure, maintaining a reasonable distribution spacing between seen and unseen classes. The overall structure of the semantic space is highly consistent with the semantic topology of the attribute prototype. To verify the necessity of the CLS-Patch attention-weighted aggregation design in this invention, this experiment compares three visual feature extraction methods under the premise of fixing the loss function and training hyperparameters: using only the global features of the image ( The proposed solutions include: Scheme A (1024 dimensions), concatenation of global features and equally weighted mean image patch features (Scheme B, 2048 dimensions), and concatenation of global features and attention-weighted image patch features (Scheme C, 2048 dimensions). As shown in Table 3, Scheme A yields very low accuracy for the unseen class, with U only reaching 13.45% on CUB. The harmonic mean H on all three datasets lags significantly, indicating that simple global features lack sufficient local discriminative information. Scheme B improves H by introducing the mean of image patch features, but the equally weighted processing introduces a large amount of background noise, and the accuracy for the unseen class remains significantly insufficient. Scheme C uses attention weights to focus the model on semantically relevant local regions, further improving the H index on CUB, SUN, and AWA2 by 4.93, 17.46, and 9.74 percentage points respectively compared to Scheme B, verifying the effectiveness and necessity of the attention-weighted aggregation design of this invention.

[0140] Table 3. Comparison of visual feature extraction methods using DINOv2 (%)

[0141]

[0142] This invention also provides a generalized zero-shot image recognition system that integrates attention feature aggregation and probabilistic modeling, comprising:

[0143] The data preparation module is used to construct a training set from labeled images of known categories. The test set consists of unlabeled images containing a mixture of seen and unseen samples. Get all The prototype vectors of semantic attributes for each category are arranged in order of seen classes first and unseen classes last, forming a prototype matrix. ,in The number of seen classes, The number of unseen classes. For semantic attribute dimensions;

[0144] The feature extraction and mapping module is used for the input image. Visual features are extracted through a pre-trained visual backbone network and then projected onto a learnable mapping module. In a 3D semantic space, we obtain a semantic mapping vector. ,in These are the learnable network parameters for the mapping module;

[0145] The end-to-end training module is used to measure the similarity between semantic mapping vectors and category prototype vectors. Based on this, parameterize the label conditional probability of each sample for the known class:

[0146]

[0147]

[0148] For unseen classes:

[0149]

[0150]

[0151] in This indicates the labeled status of the sample; 1 represents a labeled sample, and -1 represents an unlabeled sample. This indicates that the true category of the sample is the first. kind, For the Sigmoid function;

[0152] Based on similarity Introducing a learnable categorical temperature coefficient Calculate the total number of samples for each sample. Softmax classification probabilities on semantic prototypes of each category By maximizing the joint log-likelihood of the training and test sets, the end-to-end optimization objective is derived. Based on the label conditional probability and classification probability, a total loss function containing the probabilistic modeling main loss and multiple auxiliary losses is constructed, and end-to-end training is performed.

[0153] The inference module is used to input the image to be recognized into the trained model and calculate the recognition value of the image. Normalized semantic mapping vectors and all categories The cosine similarity between normalized semantic prototypes is used, and the category corresponding to the maximum value is taken as the predicted category.

[0154] It should be understood that the generalized zero-shot image recognition system that integrates attention feature aggregation and probabilistic modeling in the embodiments of the present invention can realize all the technical solutions in the above method embodiments. The functions of each functional module can be specifically implemented according to the steps of the above method, and the specific implementation process will not be described here.

[0155] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the generalized zero-shot image recognition method that integrates attention feature aggregation and probabilistic modeling as described above.

[0156] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the generalized zero-shot image recognition method as described above, which integrates attention feature aggregation and probabilistic modeling.

Claims

1. A generalized zero-shot image recognition method integrating attention feature aggregation and probabilistic modeling, characterized in that, Includes the following steps: S1. Construct a training set from the labeled images of the known categories. The test set consists of unlabeled images containing a mixture of seen and unseen samples. Get all The prototype vectors of semantic attributes for each category are arranged in order of seen classes first and unseen classes last, forming a prototype matrix. ,in The number of seen classes, The number of unseen classes. For semantic attribute dimensions; S2, for the input image Visual features are extracted through a pre-trained visual backbone network and then projected onto a learnable mapping module. In a dimensional semantic space, we obtain a semantic mapping vector. ,in These are the learnable network parameters for the mapping module; S3. Based on the similarity between semantic mapping vectors and category prototype vectors Based on this, parameterize the label conditional probability of each sample for the known class: For unseen classes: in This indicates the labeled status of the sample; 1 represents a labeled sample, and -1 represents an unlabeled sample. This indicates that the true category of the sample is the first. kind, For the Sigmoid function; Based on similarity Introducing a learnable categorical temperature coefficient Calculate the total number of samples for each sample. Softmax classification probabilities on semantic prototypes of each category ; By maximizing the joint log-likelihood of the training and test sets, the end-to-end optimization objective is derived. Based on the label conditional probability and classification probability, a total loss function containing the probabilistic modeling main loss and multiple auxiliary losses is constructed for end-to-end training. S4. Input the image to be recognized into the trained model and calculate the image to be recognized. Normalized semantic mapping vectors and all categories The cosine similarity between normalized semantic prototypes is used, and the category corresponding to the maximum value is taken as the predicted category.

2. The method according to claim 1, characterized in that, The pre-trained visual backbone network is the DINOv2 ViT-L / 14 model, which processes the input image. The visual feature extraction employs a method combining global semantic labeling and attention-weighted image patch aggregation and concatenation, specifically including: By forward propagating through the backbone network, the global semantic labeling features of the last layer are extracted. and the feature sequences of all N image patch labels N is the number of image patches; Utilizing the query projection weight matrix frozen in the last attention layer of the backbone network With key projection weight matrix Manually calculate the attention weights of the global semantic tags for each image patch tag: Where H represents the number of attention heads. In order to target the The query vector with globally semantically labeled data extracted by head attention. In order to target the The key matrix of all image patches extracted by head attention; the fusion weights are obtained by averaging the attention weights of all heads. Utilizing fusion weights Weighted aggregation of image patch marker feature sequences yields image patch aggregated features. Finally, the global semantic tagging features and image patch aggregation features are concatenated along the channel dimension to obtain the final visual features. , as input to the mapping module; The main parameters of the backbone network are kept frozen during training, and only the parameters of the last layer (LayerNorm) are unfrozen to accommodate the drift of the downstream semantic distribution.

3. The method according to claim 1, characterized in that, The mapping module consists of an input normalization layer and a multilayer perceptron connected in series, organized in the following order: Input normalization layer: normalizes the input visual features Apply layer normalization to eliminate feature scale differences; The first linear transformation layer: The fully connected layer compresses the feature dimension from the input dimension to 1024. It is followed by a normalization layer, a LeakyReLU activation layer with a specified negative slope, and a Dropout layer to complete the dimension alignment and non-linear transformation. The residual block contains a fully connected layer that maintains 1024 dimensions, followed by a normalization layer, a LeakyReLU activation layer, and a Dropout layer. The block output is added to the block input element by element to preserve the feature orientation of the previous layer and to mine fine-grained semantic associations during semantic compression. Output linear transformation layer: The fully connected layer compresses the feature dimension from 1024 to the target semantic attribute dimension. Output the original semantic mapping vector; All parameters of the mapping module participate in end-to-end gradient optimization, and the output vector is subjected to... Normalization process.

4. The method according to claim 1, characterized in that, The total loss function, which includes the probabilistic modeling main loss and multiple auxiliary losses, is as follows: in, The main loss is composed of the training set loss term. and test set loss terms constitute, and To mitigate losses, For semantic anchoring loss, For conditional prototype alignment loss, and For the corresponding weights; by Indicates training samples The true category labels, and the training set loss term are: The test set loss term is expanded based on the law of total probability and uses logarithmic-summary-exponential operations: The semantic anchoring loss constrains the cosine distance between the semantic mapping vectors of seen class samples in the training set and their true class semantic prototypes: The conditional prototype alignment loss models the unseen class posterior weights estimated by the framework itself using a mixture distribution probability model. To adapt the weights, a weighted soft classification loss within the unseen class subspace is applied to the test set samples: in The most recently unseen class prototype; For the posterior weights of the mixed distribution, by The intermediate quantities during the forward propagation process are obtained directly, among which... This indicates the probability mass of a test sample belonging to a known class that has not been labeled. This indicates the probability mass of the test sample belonging to the unseen class; when the test sample actually belongs to the seen class, Approaching zero Automatically approaches zero.

5. The method according to claim 1, characterized in that, During the training phase, a two-stream balanced sampling strategy is used to sample the training set. With test set The samples are mixed in equal numbers in each batch, where each batch is drawn from two independent, randomly shuffled index pools. Each sample was spliced ​​together and then randomly shuffled again within the batch. Batch size; when the number of available samples in an index pool is insufficient. The pool is then reused cyclically to replenish the required quantity, ensuring a consistent batch size for each training round; each sample in a batch carries a source identifier. 0 represents a test set sample, and 1 represents a training set sample, used to distinguish which sample should participate during training. still The calculation.

6. The method according to claim 1, characterized in that, The optimization strategies for end-to-end training specifically include: adjusting the mapping module parameters, the normalized parameters of the unfrozen layers in the backbone network, and the learnable temperature coefficients. The learning rate is set independently for three groups, with the learning rate for the layer normalization parameters being two orders of magnitude lower than that for the mapping module; a learning rate scheduling strategy combining linear preheating and cosine annealing is employed; the AdamW optimizer is used, and the mapping module parameters are adjusted accordingly. Weight decay; after each backpropagation step, the gradients of all trainable parameters are clipped to a fixed threshold according to the global norm.

7. The method according to claim 1, characterized in that, In calculating classification probability At that time, a margin penalty is applied to the similarity component corresponding to the seen class to suppress the bias of the seen class: in To fix the cosine margin hyperparameter; Let be the similarity components after applying margin penalties; then the Softmax classification probability is... And satisfy , for Normalized cosine similarity function.

8. A generalized zero-shot image recognition system that integrates attention feature aggregation and probabilistic modeling, characterized in that, include: The data preparation module is used to construct a training set from labeled images of known categories. The test set consists of unlabeled images containing a mixture of seen and unseen samples. Get all The prototype vectors of semantic attributes for each category are arranged in order of seen classes first and unseen classes last, forming a prototype matrix. ,in The number of seen classes, The number of unseen classes. For semantic attribute dimensions; The feature extraction and mapping module is used for the input image. Visual features are extracted through a pre-trained visual backbone network and then projected onto a learnable mapping module. In a dimensional semantic space, we obtain a semantic mapping vector. ,in These are the learnable network parameters for the mapping module; The end-to-end training module is used to measure the similarity between semantic mapping vectors and category prototype vectors. Based on this, parameterize the label conditional probability of each sample for the known class: For unseen classes: in This indicates the labeled status of the sample; 1 represents a labeled sample, and -1 represents an unlabeled sample. This indicates that the true category of the sample is the first. kind, For the Sigmoid function; Based on similarity Introducing a learnable categorical temperature coefficient Calculate the total number of samples for each sample. Softmax classification probabilities on semantic prototypes of each category By maximizing the joint log-likelihood of the training and test sets, the end-to-end optimization objective is derived. Based on the label conditional probability and classification probability, a total loss function containing the probabilistic modeling main loss and multiple auxiliary losses is constructed, and end-to-end training is performed. The inference module is used to input the image to be recognized into the trained model and calculate the recognition value of the image. Normalized semantic mapping vectors and all categories The cosine similarity between normalized semantic prototypes is used, and the category corresponding to the maximum value is taken as the predicted category.

9. An electronic device, characterized in that, It includes one or more processors, a memory, and one or more programs stored in the memory, wherein when the programs are executed by the processor, they implement the generalized zero-shot image recognition method that integrates attention feature aggregation and probabilistic modeling as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the generalized zero-shot image recognition method that integrates attention feature aggregation and probabilistic modeling as described in any one of claims 1 to 7.

Images