Training method of image classification model, image classification method and device

Abstract

The application belongs to the technical field of image processing, and discloses a training method and a device of an image classification model, and an image classification method and device. The image classification model and a prototype library are trained by using an input image. The input image is subjected to data enhancement processing to obtain a plurality of views. A shared encoder is used to extract features of the plurality of views to obtain a plurality of target feature representations. A nonlinear projection head is used to map the plurality of target feature representations to a unit hypersphere to obtain a plurality of feature embeddings. A normalized cosine classifier is used to map the target feature representations of at least one view to a category space to obtain a category probability. Based on the plurality of feature embeddings and the prototype library, an exchange loss value is calculated by using a view exchange prediction mechanism, and a cross-entropy loss value is calculated based on the category probability and a category label. Parameters of the image classification model and prototypes in the prototype library are updated based on the loss values. The embodiment can improve the generalization capability of the model on tail categories.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a training method for an image classification model, an image classification method, and an apparatus. Background Technology

[0002] Multimedia data in the real world inherently exhibits an extreme long-tail distribution: a few head classes occupy the vast majority of samples, while a large number of tail classes have only a small number of samples. In standard supervised learning paradigms, driven by minimizing global empirical risk, models often allocate the vast majority of the feature space to head classes. This unbalanced allocation leads to severe feature collapse in tail classes: tail samples are extremely compressed in the embedding space, even submerged within the decision boundaries of head classes, ultimately resulting in a significant decrease in the model's generalization ability on tail classes. Summary of the Invention

[0003] The technical problem to be solved by the present invention is that the models trained by the methods commonly used in the prior art have poor generalization ability in the tail category, resulting in low accuracy of image classification results. In order to solve the above problem, the present invention provides a training method for an image classification model, an image classification method and apparatus.

[0004] The content of this invention includes: In a first aspect, embodiments of the present invention provide a method for training an image classification model, comprising: The image classification model and prototype library are trained using a long-tail training dataset until a preset convergence condition is met, resulting in a trained image classification model. The long-tail training dataset includes multiple input images with category labels. These input images can be medical images or animal / plant images. When the input image is a medical image, the category label is "pathological state"; when the input image is an animal / plant image, the category label is "species category". The image classification model includes a shared encoder, a nonlinear projection head, and a normalized cosine classifier. The prototype library includes... A trainable prototype, It is a positive integer; The process of training the image classification model and prototype library using the input image includes: The input image is subjected to data augmentation processing to obtain... One view, It is a positive integer greater than 1; The shared encoder was used to respectively Feature extraction is performed on the views to obtain... Each target feature is represented; Through the nonlinear projection head Each of the target features is mapped onto a unit hypersphere, resulting in... The system embeds at least one feature and maps the target feature representation of at least one view to the category space through the normalized cosine classifier to obtain the category probability. based on The feature embeddings and the prototype library are used to calculate the exchange loss value using the view exchange prediction mechanism, and the cross-entropy loss value is calculated based on the class probability and the class label; Calculate the total loss value based on the exchange loss value and the cross-entropy loss value; The parameters of the image classification model and the prototypes in the prototype library are updated based on the total loss value, the exchange loss value, and the cross-entropy loss value.

[0005] Optionally, the The view includes two global views and A partial view, For positive integers greater than 2, the step of mapping the target feature representation of at least one of the views to the category space through the normalized cosine classifier to obtain the category probability includes: The normalized cosine classifier maps the target feature representation of one of the global views to the category space to obtain the category probability.

[0006] Optionally, the basis The feature embeddings and the prototype library are used to calculate the exchange loss value using an inter-view exchange prediction mechanism, including: Calculate the similarity score matrix between the embedding matrix and the prototype matrix, where the embedding matrix is... The matrix formed by the aforementioned feature embeddings, wherein the prototype matrix is... A matrix composed of the aforementioned prototypes; An allocation coding matrix is ​​calculated based on the similarity score matrix, and the allocation coding matrix contains... The nth row vector, the th The row vector represents the first row vector. Each view in The allocation code on the prototype. ; The exchange loss value is calculated using the allocation encoding and prediction distribution between different views; wherein the first view and the second view are any two different views, and the exchange loss value is calculated based on the cross-entropy between the prediction distribution of the first view and the allocation encoding of the second view, and the prediction distribution is determined after normalization based on the similarity score between the feature embedding of the first view and the prototype.

[0007] Optionally, the exchange loss value for: ; ; in, A collection used to represent the global view. for The first in A global view A collection used to represent all views. for Except Any view outside, for Feature embedding, for The allocation code, for The first in One portion, for The corresponding predicted distribution, for The first in One portion, for and Cross-entropy between them.

[0008] Optionally, updating the parameters of the input image classification model and the prototype in the prototype library based on the total loss value, the exchange loss value, and the cross-entropy loss value includes: The parameters of the nonlinear projection head and the prototypes in the prototype library are updated based on the exchange loss value, the parameters of the normalized cosine classifier are updated based on the cross-entropy loss value, and the parameters of the shared encoder are updated based on the total loss value.

[0009] Secondly, embodiments of the present invention provide an image classification method, including: The target image is input into the target encoder for encoding processing to obtain the feature representation; The feature representation is input into a pre-trained classification head for classification processing to obtain a classification result, wherein the target encoder is a shared encoder in a trained image classification model obtained by training the image classification model as described in the first aspect.

[0010] Thirdly, embodiments of the present invention provide a training apparatus for an image classification model, comprising: The training module is used to train the image classification model and the prototype library using a long-tail training dataset until a preset convergence condition is met, resulting in a trained image classification model. The long-tail training dataset includes multiple input images with category labels. These input images can be medical images or animal / plant images. When the input image is a medical image, the category label is "pathological state"; when the input image is an animal / plant image, the category label is "species category". The image classification model includes a shared encoder, a nonlinear projection head, and a normalized cosine classifier. The prototype library includes... A trainable prototype, It is a positive integer; The process of training the image classification model and prototype library using the input image includes: The input image is subjected to data augmentation processing to obtain... One view, It is a positive integer greater than 1; The shared encoder was used to respectively Feature extraction is performed on the views to obtain... Each target feature is represented; Through the nonlinear projection head Each of the target features is mapped onto a unit hypersphere, resulting in... The system embeds at least one feature and maps the target feature representation of at least one view to the category space through the normalized cosine classifier to obtain the category probability. based on The feature embeddings and the prototype library are used to calculate the exchange loss value using the view exchange prediction mechanism, and the cross-entropy loss value is calculated based on the class probability and the class label; Calculate the total loss value based on the exchange loss value and the cross-entropy loss value; The parameters of the image classification model and the prototypes in the prototype library are updated based on the total loss value, the exchange loss value, and the cross-entropy loss value.

[0011] Fourthly, embodiments of the present invention provide an image classification apparatus, comprising: The first processing module is used to input the target image into the target encoder for encoding processing to obtain feature representation; The second processing module is used to input the feature representation into a pre-trained classification head for classification processing to obtain a classification result, wherein the target encoder is a shared encoder in a trained image classification model obtained by training the image classification model as described in the first aspect.

[0012] Fifthly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor is configured to read the program in the memory to implement steps in the training method of the image classification model as described in the first aspect, or to implement steps in the image classification method as described in the second aspect.

[0013] In a sixth aspect, embodiments of the present invention provide a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the training method of the image classification model as described in the first aspect, or implements the steps in the image classification method as described in the second aspect.

[0014] The beneficial effects of this invention are as follows: In its embodiments, on the one hand, by introducing multiple independent, trainable prototypes, the model no longer relies entirely on the statistical mean of a small number of samples to represent the class center. The prototype library can be corrected through global optimization during training, acting as a more robust class anchor, thereby significantly improving the model's ability to represent tail-class features and enhancing generalization. On the other hand, cross-entropy loss ensures basic classification accuracy, while exchange loss utilizes the consistency between views to supplement the supervision signal missing for tail-classes due to the small number of samples. By combining cross-entropy loss and exchange loss, geometric uniformity and semantic discriminability are simultaneously achieved in the end-to-end feature learning method, overcoming the structural limitations of existing methods. Through the above training method, the overfitting and feature drift problems caused by the scarcity of tail-class samples in long-tail distributions are effectively overcome, significantly improving the generalization ability and robustness of image classification models on long-tail categories. Attached Figure Description

[0015] Figure 1 A schematic flowchart illustrating the training method for the image classification model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of training an image classification model and a prototype library using an input image, as provided in an embodiment of the present invention. Figure 3 A schematic diagram of the data augmentation and feature extraction parts provided in an embodiment of the present invention; Figure 4 A schematic diagram of the prototype and clustering parts, as well as the loss and model optimization parts, provided for embodiments of the present invention; Figure 5 A schematic flowchart of the image classification method provided in an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a training device for an image classification model provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the image classification device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] In the embodiments of this application, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar. The terms "first," "second," etc., in the specification of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such terms can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are usually of the same class, without limiting the number of objects. For example, the first object can be one or multiple.

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] This application provides a training method, an image classification method, and an apparatus for an image classification model, aiming to improve the generalization ability of the image classification model on tail categories, solve the representation learning problem in long-tail scenarios, and train an image classification model with better classification performance.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating the training method for an image classification model provided in an embodiment of the present invention. The method specifically includes the following steps: Step 101: Train the image classification model and prototype library using the long-tail training dataset until a preset convergence condition is met to obtain the trained image classification model. The long-tail training dataset includes multiple input images with category labels. The input images are medical images or animal / plant images. When the input image is a medical image, the category label is pathological state; when the input image is an animal / plant image, the category label is species category. The image classification model includes a shared encoder, a nonlinear projection head, and a normalized cosine classifier. The prototype library includes... A trainable prototype, It is a positive integer.

[0021] This invention can be applied to the field of medical imaging diagnosis. The input images in the long-tail training dataset are human organ imaging data acquired by X-ray, computed tomography (CT), or magnetic resonance imaging (MRI) equipment, with category labels corresponding to different pathological states. Specifically, clinically acquired medical imaging data naturally exhibits a long-tail distribution, with common diseases (such as ordinary pneumonia) accounting for an absolute majority of the sample size, while the number of samples for rare diseases, early small lesions (such as millimeter-sized pulmonary nodules), or novel variant diseases (such as the imaging manifestations of emerging infectious diseases) is extremely small. Traditional deep learning models are easily affected by the uneven distribution of samples during training, tending to identify common head categories with high confidence, while lacking sufficient feature representation ability for rare tail categories. This leads to technical defects in clinical auxiliary diagnosis, such as misclassifying rare diseases as common diseases or missing early lesions. The method provided by this invention enables the model to dynamically balance the gradient contributions of head and tail categories during the learning process, enhancing the sensitivity to fine-grained features of rare tail lesions. The above methods can capture minute spiculation signs of early lung cancer and atypical enhancement patterns of rare brain tumors in CT images. This can significantly improve the accuracy and recall of the auxiliary diagnostic system for rare and early lesions without increasing the cost of additional clinical data collection, reduce the risk of false negatives, and provide more reliable image interpretation support for clinical practice.

[0022] This invention can also be applied to the field of plant and animal identification. The input images in the long-tail training dataset are plant and animal images automatically collected by outdoor infrared cameras or monitoring equipment, with category labels corresponding to different species categories. Specifically, data in the field of plant and animal identification naturally exhibits a significant long-tail distribution: in ecosystems, common species (such as sparrows, gray squirrels, and common herbaceous plants) appear very frequently in field monitoring data due to their large populations and wide range of activity, forming the head of the data distribution; while rare and endangered species (such as snow leopards, pangolins, and wild orchids) or seasonally migratory species, due to their small populations, elusive activities, and difficulty in monitoring, have very few effective image samples collected, often only a few scattered blurry images or brief video clips, constituting the tail of the data distribution. Furthermore, the differences in the probability of species occurrence in different regions and seasons further exacerbate this imbalance. Through the method provided by this invention, even in cases of extremely scarce samples, the identification confidence can be improved through reinforcement learning of local features. Meanwhile, the normalized cosine classification mechanism eliminates the interference of feature vector magnitude on classification results, preventing common species at the top from dominating the classification boundary due to their large sample size, and ensuring that species at the bottom obtain a fair discrimination region in the feature space. This method can significantly improve the identification recall rate of rare species in field monitoring systems and reduce false positives and false negatives.

[0023] In this embodiment, the long-tail training dataset is , For the first An input image, which can be simply referred to as an image. , for Corresponding category tags, The number of input images, This represents the number of class labels. Long-tailed training datasets follow a long-tailed distribution, with class frequencies exhibiting exponential or power-law decay. ,in, For the first The number of input images corresponding to each category label .

[0024] In this embodiment, the image classification model includes a shared encoder, a nonlinear projection head, and a normalized cosine classifier. The shared encoder maps the input image to a high-dimensional feature representation to obtain the target feature representation. The specific structure of the shared encoder is not limited here; in some embodiments, the parameters of the shared encoder are randomly initialized. In some embodiments, the shared encoder is a ResNet, consisting of one initial convolutional layer (7×7 convolution, stride 2, followed by batch normalization and ReLU), one max-pooling layer (3×3, stride 2), and three consecutive residual stages, totaling 15 residual blocks (each residual block contains two convolutional layers, i.e., 3 stages × 5 residual blocks × 2 layers = 30 convolutional layers, plus the initial independent convolutional layer, for a total of 32 trainable weight layers).

[0025] The nonlinear projection head, as an unsupervised branch, is used to map the target feature representation output by the shared encoder to a normalized embedding on a unit hypersphere. In some embodiments, the nonlinear projection head is a 2-layer multi-layer perceptron (MLP) with a hidden dimension of 2048 and an output dimension of 128.

[0026] The normalized cosine classifier, as a supervised branch, maps the target feature representation output by the shared encoder to class probabilities. In some embodiments, the normalized cosine classifier performs L2 normalization on the input feature vector and the weight vector for each class, making their magnitudes equal to 1, and then calculates the cosine similarity (i.e., the vector dot product) as the classification score, instead of directly calculating the unnormalized dot product.

[0027] In this embodiment, a set of trainable prototypes is also maintained. This is called a prototype library. Indicates the number of prototypes. For the first A prototype, which can be simply referred to as a prototype. , Each prototype can be viewed as a cluster center in the feature space, and the prototype is not fixed, but rather the parameters are continuously updated and optimized through backpropagation during model training.

[0028] In some embodiments, setting This strategy (called hyperclustering) ensures sufficient geometric capacity to accommodate tail classes and avoids overcrowding. By setting a large number of prototypes, the feature space is divided into very fine segments. Even if a tail class has few samples, it can find its own or shared prototype regions without being completely covered by the prototypes of the head classes. This ensures the geometric uniformity of the feature space.

[0029] like Figure 2As shown, taking any input image as an example, the image classification model and prototype library are trained using the input image, specifically including the following steps: Step 201: Perform data augmentation processing on the input image to obtain... One view, It is a positive integer greater than 1; Step 202, using the shared encoder to respectively... Feature extraction is performed on the views to obtain... Each target feature is represented; Step 203, using the nonlinear projection head to... Each of the target features is mapped onto a unit hypersphere, resulting in... The system embeds at least one feature and maps the target feature representation of at least one view to the category space through the normalized cosine classifier to obtain the category probability. Step 204, based on The feature embeddings and the prototype library are used to calculate the exchange loss value using the view exchange prediction mechanism, and the cross-entropy loss value is calculated based on the class probability and the class label; Step 205: Calculate the total loss value based on the exchange loss value and the cross-entropy loss value; Step 206: Update the parameters of the image classification model and the prototypes in the prototype library based on the total loss value, the exchange loss value, and the cross-entropy loss value.

[0030] First, process the input image. Perform data augmentation to obtain the corresponding data from the input image. In some embodiments, the input image is data augmented to obtain two global views corresponding to the input image. The global views are used to capture the semantic structure of the input image, which helps the model learn the overall category features of the image. In other embodiments, the input image is data augmented to obtain one global view and multiple local views corresponding to the input image. The local views are used to capture fine-grained details of the input image, preventing the model from relying solely on the background or overall shape.

[0031] As a specific embodiment, the The view includes two global views and In practical implementation, local views can be obtained by using different cropping ratios to obtain the global and local views corresponding to the input image. For example, using a larger cropping ratio (e.g., covering 80%-100% of the original image) preserves the overall context of the image, resulting in the global view; while using a smaller cropping ratio (e.g., covering 10%-50% of the original image) and performing high-resolution scaling yields the local view. Each local view randomly covers different regions of the input image.

[0032] In this embodiment, by acquiring a local view, the shared encoder can be forced to focus on local discriminative regions of the image (such as specific parts of an object or texture details), rather than relying solely on the background or overall outline. For the tail category in a long-tailed distribution, due to the limited number of samples, the model is prone to overfitting to simple global features. Introducing a local view can effectively improve the model's ability to distinguish subtle differences.

[0033] Each view is input into a shared encoder for processing to obtain a target feature representation for each view. For each target feature representation, this representation is then input into a nonlinear projection head for processing to obtain the corresponding feature embedding. Based on... The feature embeddings and prototype libraries corresponding to each view are used to calculate the exchange loss value using the view exchange prediction mechanism.

[0034] For any view , for A set of views, whose normalized feature embeddings are obtained through a shared encoder and a nonlinear projection head. : ; in, Used to characterize the shared encoder, its parameters are: , The parameters used to characterize the nonlinear projection head are: , Used to characterize the L2 norm.

[0035] Optionally, in some embodiments, the basis The feature embeddings and the prototype library are used to calculate the exchange loss value using an inter-view exchange prediction mechanism, including: Calculate the similarity score matrix between the embedding matrix and the prototype matrix, where the embedding matrix is... The matrix formed by the aforementioned feature embeddings, wherein the prototype matrix is... A matrix composed of the aforementioned prototypes; An allocation coding matrix is ​​calculated based on the similarity score matrix, and the allocation coding matrix contains... The nth row vector, the th The row vector represents the first row vector. Each view in The allocation code on the prototype. ; The exchange loss value is calculated using the allocation encoding and prediction distribution between different views; wherein the first view and the second view are any two different views, and the exchange loss value is calculated based on the cross-entropy between the prediction distribution of the first view and the allocation encoding of the second view, and the prediction distribution is determined after normalization based on the similarity score between the feature embedding of the first view and the prototype.

[0036] First, calculate the embedding matrix. With prototype matrix Similarity score matrix between : ; in, For prototype matrix, For feature dimension, For the first A prototype, for transpose, For embedding matrix, For the first Embedded features corresponding to each view for transpose, for The Middle Line 1 The elements of the column.

[0037] In this embodiment, instead of directly using the original similarity scores for prediction, the similarity score matrix is ​​used instead. Mapping to a robust allocation encoding matrix This serves as a pseudo-label. Specifically, it is calculated using the Sinkhorn-Knopp algorithm. This yields a row and column normalized matrix. From a mathematical perspective, the aim is to find a... To maximize the similarity to the score while satisfying the equal division constraint, and to ensure that the prototype is used evenly within the batch, the optimization objective function can be expressed as: ; in, For entropy regularization, Here is the entropy regularization coefficient. For the allocation matrix, for transpose, Used to measure the consistency between the assignment matrix and the similarity matrix. This is used to indicate maximizing this consistency.

[0038] By solving the above optimization objective function, the allocation encoding matrix can be obtained, and then the global view can be utilized. The generated allocation code This requires other views to predict the encoding. It should be understood that the view... Corresponding prediction distribution Based on its embedding features The calculated prototype prediction probability distribution vector is equivalent to the model's current instantaneous prediction.

[0039] For example, for another feature with embedding characteristics view It was assigned to the first The probability of a prototype is denoted as: and through Distribution modeling: ; in, For temperature coefficient, for transpose, For the first A prototype, For the first A prototype, Used to represent exponentiation.

[0040] Global view Corresponding allocation code With View Corresponding prediction distribution Cross-entropy between for: ; in, for The first in Each component, i.e., the global view Assigned to the The probability value on each prototype for The first in Each component, i.e., the model's predicted view. Belongs to the The probability of a prototype, Used to represent logarithmic operations.

[0041] Ultimately, the swap loss value is calculated through the swap allocation among all view pairs, that is, by using the global view to guide all other views, swapping the loss value. for: ; in, , The set used to represent the global view. By minimizing the exchange loss, the model learns a geometrically uniform feature space that is robust to data imbalance.

[0042] from From the given views, one view (usually a global view) is selected. The target feature representation corresponding to this view is input into a normalized cosine classifier to obtain the class probability. Optionally, in some embodiments, the... The view includes two global views and A partial view, For positive integers greater than 2, the step of mapping the target feature representation of at least one of the views to the category space through the normalized cosine classifier to obtain the category probability includes: The normalized cosine classifier maps the target feature representation of one of the global views to the category space to obtain the category probability.

[0043] Because the global view preserves the complete object structure and background context, in this embodiment, a normalized cosine classifier maps the target feature representation of one of the global views to the category space to obtain category probabilities. This forces the model to make classification decisions based on complete semantic information, ensuring that the features learned by the model are object-level, rather than fragmented texture-level or component-level features. This approach ensures a strict match between the semantic integrity of the target feature representation and the globality of the category label. This allows the model to make judgments based on the overall structure, shape, and contextual relationships of the object, thereby learning global semantic features that truly represent the essence of the category.

[0044] The cross-entropy loss value can be calculated using the class probabilities and the class labels corresponding to the input image. For example, in some embodiments, the cross-entropy loss value... for: ; In the above formula, Batch size refers to the number of samples contained in a training batch currently used for loss calculation. The total number of categories, For the first Each sample in category The real labels on To predict the first Each sample belongs to category The probability value.

[0045] In this embodiment, the total loss value is determined based on the exchange loss value and the cross-entropy loss value. In some embodiments, the total loss value is determined based on a weighted sum of the exchange loss value and the cross-entropy loss value. : ; in, The weights are predetermined values, and their specific values ​​can be set and adjusted according to actual conditions; no specific limitations are specified here. During training, end-to-end training is performed by minimizing a joint objective function that integrates geometric constraints and semantic guidance. ; in, For parameters of the shared encoder, These are the parameters of the nonlinear projection head. The parameters of the normalized cosine classifier, Used to characterize the optimization of model parameters by minimizing empirical risk.

[0046] Optionally, in some embodiments, updating the parameters of the input image classification model and the prototype in the prototype library based on the total loss value, the exchange loss value, and the cross-entropy loss value includes: The parameters of the nonlinear projection head and the prototypes in the prototype library are updated based on the exchange loss value, the parameters of the normalized cosine classifier are updated based on the cross-entropy loss value, and the parameters of the shared encoder are updated based on the total loss value.

[0047] In this embodiment, the shared encoder receives data from... and The gradient is updated through cooperative force to ensure that the learned representation simultaneously satisfies geometric homogeneity and semantic discriminability. Prototypes and nonlinear projection heads in the prototype library are only obtained through... The update focuses on mapping features to a hypersphere for prototype assignment; the normalized cosine classifier only... The update focuses on establishing decision boundaries based on semantic labels.

[0048] In deep learning frameworks, the encoder is a fundamental feature extractor, and the model's generalization ability largely depends on the quality and robustness of the learned representations. Therefore, enhancing the backbone network's ability to learn discriminative features is crucial for improving long-tail recognition performance. In this embodiment, the shared encoder simultaneously receives data from... and gradient, This is used to guide the encoder to focus on the intrinsic structure and fine-grained details of the data, preventing overfitting to the head class. The encoder is guided to align with the true semantic labels. In this way, the feature representation it learns contains accurate semantic information and is robust to long-tail data in terms of feature space distribution.

[0049] Please see Figure 3 and Figure 4The following description uses a specific embodiment as an example. In this embodiment, for a given input image... Data augmentation is performed using a multi-pruning strategy to generate a set of views. This includes two global views (for capturing semantic structure) and two local views (for capturing fine-grained details). The core objective of data augmentation is to ensure consistency across different views of the same sample by using features from one view to predict the clustering assignment codes generated by another global view.

[0050] Then, a randomly initialized shared encoder is used. Each view is mapped to a target feature representation. The target feature representation is then processed through two collaborative branches. This is done via a non-linear projection head. Mapping the target feature representations of the four views to a normalized embedding on a unit hypersphere To obtain a global view Figure 1 Corresponding embedding features Global view Figure 2 Corresponding embedding features Local view Figure 1 Corresponding embedding features and local view Figure 2 Corresponding embedding features By using a normalized cosine classifier global view Figure 1 The corresponding target feature is represented as a mapping to class probability. .

[0051] Based on embedding matrix With prototype matrix The allocation coding matrix is ​​calculated using the Sinkhorn-Knopp algorithm. The exchange loss value was further calculated. for: ; ; in, A collection used to represent the global view. for The first in A global view A collection used to represent all views. for Except Any view outside, for Feature embedding, for The allocation code is calculated using the Sinkhorn-Knopp algorithm. for The first in One portion, for The corresponding predicted distribution, for The first in One portion, for and Cross-entropy between them.

[0052] Based on class probability and input image Corresponding category tags Calculate the cross-entropy loss value Then, the total loss (Total Loss) is calculated based on the swap loss (Swap-Loss) and cross-entropy loss (CE-Loss). The parameters of the nonlinear projection head and the prototype in the prototype library are updated based on the swap loss, the parameters of the normalized cosine classifier are updated based on the CE-Loss, and the parameters of the shared encoder are updated based on the total loss.

[0053] In this embodiment, the nonlinear projection head constitutes an unsupervised branch based on long-tailed multi-view super-clustering. This unsupervised super-clustering branch provides a continuous gradient for geometric expansion and preserves the prototype space for tail categories. The normalized cosine classifier constitutes a traditional supervised branch that provides semantic guidance. This supervised branch provides necessary semantic anchoring to prevent cluster misalignment. By combining online multi-view super-clustering with supervised semantic guidance in this way, the feature space is effectively reconstructed, tail category collapse is prevented, and supervised signals are used as semantic anchors to correct clustering bias, overcoming the limitations of existing methods.

[0054] From an optimization perspective, the dominant negative gradient of the head category causes the effective gradient in the direction of the true tail category to vanish. The method provided in this embodiment can guarantee a positive lower bound on the effective gradient of the tail category, ensuring continuous constructive updates. The effectiveness of this embodiment is explained in detail below.

[0055] set up This indicates that there is a supervised gradient for the true tail category. For the sample, the gradient derivation can be found in Equation 1 below: ; in, express The weight, The model predicts the current sample as the true tail class. The probability, The model predicts the current sample as a category. The probability, True tail category The weight vector, For category The weight vector, For gradient.

[0056] In long-tail scenarios, tail samples are often misclassified as head samples. It is worth noting that, due to data scarcity, the weight vector of the tail category... Insufficient updates, extremely small norm Therefore, the first term (semantic pull) in Equation 1 disappears. Although the second term (repulsive push) still exists, it merely blindly pushes features away from the head region, lacking a defined target direction. Therefore, the effective gradient projected onto the true tail category direction disappears. .

[0057] in, Used to characterize the projection in the target direction.

[0058] In contrast, unsupervised gradients drive Aligned with the target prototype, the gradient is approximated as: ; in, It is an allocation matrix The target prototype to be assigned. After row and column normalization and the imposition of equal partition constraints, it is assumed that each prototype in the feature space is assigned a fixed and equal capacity. Therefore, even if the classifier ignores tail samples, the allocation mechanism must still assign them to specific reserved prototypes. To meet the requirement of equal division. Due to the prototype and characteristics All are located on the unit hypersphere, and when the tail category is misclassified... (far from its ideal location) With the target prototype Given the considerable distance, we can conclude that: ; in, This indicates a non-vanishing force that actively pulls features toward the reserved prototype. This contributes positively to the effective gradient in the target direction. The norm is used to represent a vector.

[0059] The total gradient is The effective gradient is defined as the component that contributes to convergence towards the tail-class objective. Since the constructive force of the supervised components is negligible, the lower bound of the total effective gradient is determined by the unsupervised term: ; in, Used to characterize the projection of the gradient in the target direction. This represents a preset minimum value, the specific value of which can be set based on experience, and will not be elaborated here.

[0060] As can be seen from the above analysis, the method provided by the embodiments of the present invention can ensure that the effective gradient of the tail category is positive. Supervised learning only provides blind rejection, while this embodiment can ensure that there is a continuous directional force to expand the feature space towards the reserved tail prototype.

[0061] like Figure 5 As shown, this embodiment of the invention also provides an image classification method, including: Step 501: Input the target image into the target encoder for encoding processing to obtain the feature representation; Step 502: Input the feature representation into the pre-trained classification head for classification processing to obtain the classification result, wherein the target encoder is the shared encoder in the trained image classification model obtained by the above-described image classification model training method.

[0062] It should be understood that, in some embodiments, the pre-trained classification head is the normalized cosine classifier in the trained image classification model obtained using the training method described above. In other embodiments, the pre-trained classification head is a classification head pre-trained using other datasets, and the specific method is not limited here.

[0063] In this embodiment, the shared encoder in the trained image classification model obtained by the above-mentioned image classification model training method is used for encoding processing. As a result, the trained shared encoder can obtain a higher quality feature representation, thereby better representing the features of the target image and obtaining a more accurate image classification result.

[0064] like Figure 6 As shown, this embodiment of the invention also provides a training device 600 for an image classification model, comprising: Training module 601 is used to train the image classification model and prototype library using a long-tail training dataset until a preset convergence condition is met, resulting in a trained image classification model. The long-tail training dataset includes multiple input images with category labels. These input images are either medical images or animal / plant images. When the input image is a medical image, the category label is "pathological state"; when the input image is an animal / plant image, the category label is "species category". The image classification model includes a shared encoder, a nonlinear projection head, and a normalized cosine classifier. The prototype library includes... A trainable prototype, It is a positive integer; The process of training the image classification model and prototype library using the input image includes: The input image is subjected to data augmentation processing to obtain... One view, It is a positive integer greater than 1; The shared encoder was used to respectively Feature extraction is performed on the views to obtain... Each target feature is represented; Through the nonlinear projection head Each of the target features is mapped onto a unit hypersphere, resulting in... The system embeds at least one feature and maps the target feature representation of at least one view to the category space using the normalized cosine classifier to obtain the category probability. based on The feature embeddings and the prototype library are used to calculate the exchange loss value using the view exchange prediction mechanism, and the cross-entropy loss value is calculated based on the class probability and the class label; Calculate the total loss value based on the exchange loss value and the cross-entropy loss value; The parameters of the image classification model and the prototypes in the prototype library are updated based on the total loss value, the exchange loss value, and the cross-entropy loss value.

[0065] Optionally, the The view includes two global views and A partial view, For positive integers greater than 2, the step of mapping the target feature representation of at least one of the views to the category space through the normalized cosine classifier to obtain the category probability includes: The normalized cosine classifier maps the target feature representation of one of the global views to the category space to obtain the category probability.

[0066] Optionally, the basis The feature embeddings and the prototype library are used to calculate the exchange loss value using an inter-view exchange prediction mechanism, including: Calculate the similarity score matrix between the embedding matrix and the prototype matrix, where the embedding matrix is... The matrix formed by the aforementioned feature embeddings, wherein the prototype matrix is... A matrix composed of the aforementioned prototypes; An allocation coding matrix is ​​calculated based on the similarity score matrix, and the allocation coding matrix contains... The nth row vector, the th The row vector represents the first row vector. A view in The allocation code on the prototype. ; The exchange loss value is calculated using the allocation encoding and prediction distribution between different views; wherein the first view and the second view are any two different views, and the exchange loss value is calculated based on the cross-entropy between the prediction distribution of the first view and the allocation encoding of the second view, and the prediction distribution is determined after normalization based on the similarity score between the feature embedding of the first view and the prototype.

[0067] Optionally, the exchange loss value for: ; ; in, A collection used to represent the global view. for The first in A global view A collection used to represent all views. for Except Any view outside, for Feature embedding, for The allocation code, for The first in One portion, for The corresponding predicted distribution, for The first in One portion, for and Cross-entropy between them.

[0068] Optionally, updating the parameters of the input image classification model and the prototype in the prototype library based on the total loss value, the exchange loss value, and the cross-entropy loss value includes: The parameters of the nonlinear projection head and the prototypes in the prototype library are updated based on the exchange loss value, the parameters of the normalized cosine classifier are updated based on the cross-entropy loss value, and the parameters of the shared encoder are updated based on the total loss value.

[0069] The image classification model training device 600 provided in this application embodiment can execute the above-described image classification model training method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0070] like Figure 7 As shown, this embodiment of the invention also provides an image classification device 700, comprising: The first processing module 701 is used to input the target image into the target encoder for encoding processing to obtain feature representation; The second processing module 702 is used to process the feature representation input into a pre-trained classification head to obtain a classification result, wherein the target encoder is a shared encoder in a trained image classification model obtained by the above-described image classification model training method.

[0071] The image classification device 700 provided in this application embodiment can execute the above-described image classification method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0072] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0073] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] like Figure 8As shown, this application embodiment provides an electronic device 800, including: a memory 802, a processor 801, and a program stored in the memory 802 and executable on the processor 801. The processor 801 is used to read the program in the memory 802 to implement the training method of the image classification model or the steps in the image classification method as described above.

[0075] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the various processes of the image classification model training method or the image classification method embodiment described above, and achieves the same technical effect. To avoid repetition, it will not be described again here. The readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (such as compact discs (CDs), digital video discs (DVDs), Blu-ray discs (BDs), high-definition versatile discs (HVDs), etc.), and semiconductor storage (such as read-only memory (ROMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), non-volatile memory (NAND flash), solid-state disks (SSDs), etc.).

[0076] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0078] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of this application without departing from its spirit, and all of these modifications are within the scope of protection of this application.

Claims

1. A training method for an image classification model, characterized in that, include: The image classification model and prototype library are trained using a long-tail training dataset until a preset convergence condition is met, resulting in a trained image classification model. The long-tail training dataset includes multiple input images with category labels. These input images can be medical images or animal / plant images. When the input image is a medical image, the category label is "pathological state"; when the input image is an animal / plant image, the category label is "species category". The image classification model includes a shared encoder, a nonlinear projection head, and a normalized cosine classifier. The prototype library includes... A trainable prototype, It is a positive integer; The process of training the image classification model and prototype library using the input image includes: The input image is subjected to data augmentation processing to obtain... One view; The shared encoder was used to respectively Feature extraction is performed on each of the views to obtain... Each target feature is represented; Through the nonlinear projection head Each of the target features is mapped onto a unit hypersphere, resulting in... The system embeds at least one feature and maps the target feature representation of at least one view to the category space using the normalized cosine classifier to obtain the category probability. based on The feature embeddings and the prototype library are used to calculate the exchange loss value using the view exchange prediction mechanism, and the cross-entropy loss value is calculated based on the class probability and the class label; Calculate the total loss value based on the exchange loss value and the cross-entropy loss value; The parameters of the image classification model and the prototypes in the prototype library are updated based on the total loss value, the exchange loss value, and the cross-entropy loss value. Among them, the The view includes two global views and A partial view, For positive integers greater than 2, the step of mapping the target feature representation of at least one of the views to the category space through the normalized cosine classifier to obtain the category probability includes: The normalized cosine classifier maps the target feature representation of one of the global views to the category space to obtain the category probability; The step of updating the parameters of the image classification model and the prototype in the prototype library based on the total loss value, the exchange loss value, and the cross-entropy loss value includes: The parameters of the nonlinear projection head and the prototypes in the prototype library are updated based on the exchange loss value; the parameters of the normalized cosine classifier are updated based on the cross-entropy loss value; and the parameters of the shared encoder are updated based on the total loss value. Wherein, the basis The feature embeddings and the prototype library are used to calculate the exchange loss value using an inter-view exchange prediction mechanism, including: Calculate the similarity score matrix between the embedding matrix and the prototype matrix, where the embedding matrix is... The matrix formed by the aforementioned feature embeddings, wherein the prototype matrix is... A matrix composed of the aforementioned prototypes; An allocation coding matrix is ​​calculated based on the similarity score matrix, and the allocation coding matrix contains... The nth row vector, the th The row vector represents the first row vector. Each view in The allocation code on the prototype. ; The exchange loss value is calculated using the allocation encoding and prediction distribution between different views; wherein the first view and the second view are any two different views, and the exchange loss value is calculated based on the cross-entropy between the prediction distribution of the first view and the allocation encoding of the second view, and the prediction distribution is determined after normalization based on the similarity score between the feature embedding of the first view and the prototype.

2. The training method for the image classification model according to claim 1, characterized in that, The exchange loss value for: ； ； in, A collection used to represent the global view. for The first in A global view A collection used to represent all views. for Except Any view outside, for Feature embedding, for The allocation code, for The first in One portion, for The corresponding predicted distribution, for The first in One portion, for and Cross-entropy between them.

3. An image classification method, characterized in that, include: The target image is input into the target encoder for encoding processing to obtain the feature representation; The feature representation is input into a pre-trained classification head for classification processing to obtain a classification result, wherein the target encoder is a shared encoder in a trained image classification model obtained by training the image classification model as described in claim 1 or 2.

4. A training device for an image classification model, characterized in that, include: The training module is used to train the image classification model and the prototype library using a long-tail training dataset until a preset convergence condition is met, resulting in a trained image classification model. The long-tail training dataset includes multiple input images with category labels. These input images can be medical images or animal / plant images. When the input image is a medical image, the category label is "pathological state"; when the input image is an animal / plant image, the category label is "species category". The image classification model includes a shared encoder, a nonlinear projection head, and a normalized cosine classifier. The prototype library includes... A trainable prototype, It is a positive integer; The process of training the image classification model and prototype library using the input image includes: The input image is subjected to data augmentation processing to obtain... One view; The shared encoder was used to respectively Feature extraction is performed on each of the views to obtain... Each target feature is represented; Through the nonlinear projection head Each of the target features is mapped onto a unit hypersphere, resulting in... The system embeds at least one feature and maps the target feature representation of at least one view to the category space using the normalized cosine classifier to obtain the category probability. based on The feature embeddings and the prototype library are used to calculate the exchange loss value using the view exchange prediction mechanism, and the cross-entropy loss value is calculated based on the class probability and the class label; Calculate the total loss value based on the exchange loss value and the cross-entropy loss value; The parameters of the image classification model and the prototypes in the prototype library are updated based on the total loss value, the exchange loss value, and the cross-entropy loss value. Among them, the The view includes two global views and A partial view, For positive integers greater than 2, the step of mapping the target feature representation of at least one of the views to the category space through the normalized cosine classifier to obtain the category probability includes: The normalized cosine classifier maps the target feature representation of one of the global views to the category space to obtain the category probability; The step of updating the parameters of the image classification model and the prototype in the prototype library based on the total loss value, the exchange loss value, and the cross-entropy loss value includes: The parameters of the nonlinear projection head and the prototypes in the prototype library are updated based on the exchange loss value; the parameters of the normalized cosine classifier are updated based on the cross-entropy loss value; and the parameters of the shared encoder are updated based on the total loss value. Wherein, the basis The feature embeddings and the prototype library are used to calculate the exchange loss value using an inter-view exchange prediction mechanism, including: Calculate the similarity score matrix between the embedding matrix and the prototype matrix, where the embedding matrix is... The matrix formed by the aforementioned feature embeddings, wherein the prototype matrix is... A matrix composed of the aforementioned prototypes; An allocation coding matrix is ​​calculated based on the similarity score matrix, and the allocation coding matrix contains... The nth row vector, the th The row vector represents the first row vector. Each view in The allocation code on the prototype. ; The exchange loss value is calculated using the allocation encoding and prediction distribution between different views; wherein the first view and the second view are any two different views, and the exchange loss value is calculated based on the cross-entropy between the prediction distribution of the first view and the allocation encoding of the second view, and the prediction distribution is determined after normalization based on the similarity score between the feature embedding of the first view and the prototype.

5. An image classification device, characterized in that, include: The first processing module is used to input the target image into the target encoder for encoding processing to obtain feature representation; The second processing module is used to input the feature representation into a pre-trained classification head for classification processing to obtain a classification result, wherein the target encoder is a shared encoder in a trained image classification model obtained by training the image classification model as described in claim 1 or 2.

6. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor, characterized in that the processor is configured to read the program from the memory to implement the steps in the training method of the image classification model as described in claim 1 or 2, or to implement the steps in the image classification method as described in claim 3.

7. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps in the training method of the image classification model as described in claim 1 or 2, or the steps in the image classification method as described in claim 3.

Images