Image classification model training method and device based on cross-domain small sample learning

Abstract

The application relates to an image classification model training method and device based on cross-domain small sample learning. The method comprises the following steps: obtaining a target domain support set; the target domain support set comprises a small amount of first sample images; the first sample images are input into an initial image classification model pre-trained by source domain sample images to obtain first sample features; the initial image classification model comprises a head classifier and a tail classifier; the first sample features corresponding to the respective first sample images are subjected to head-tail category division to obtain first head category sample features and first tail category sample features; the first head category sample features are input into the head classifier to calculate first prototypes of each head category in the target domain, and the head classifier is updated based on the first prototypes; the first tail category sample features are input into the tail classifier to calculate second prototypes of each tail category in the target domain, and the tail classifier is updated based on the second prototypes to obtain a target image classification model suitable for the target domain. The application can train an image classification model with higher detection performance.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for training an image classification model based on cross-domain few-shot learning. Background Technology

[0002] Cross-domain few-shot learning has become a research hotspot in the field of visual recognition in recent years. Traditional few-shot learning methods assume that the data in the source and target domains follow the same distribution, but this assumption is often not met in real-world scenarios, leading to a significant drop in classification performance after distribution shift. Because of this domain distribution mismatch problem, how to achieve robust few-shot classification capabilities in cross-domain scenarios has become a key research issue.

[0003] Some studies typically attempt to transfer visual features learned in the source domain to the target domain using transfer learning and meta-learning frameworks. Currently, most cross-domain few-shot learning methods primarily focus on eliminating distributional differences between the source and target domains. However, these methods are only applicable to scenarios where the source and target domains have relatively similar distributions, and cannot be directly applied to cross-domain scenarios with significant distributional differences. For example, real-world data often exhibits severe long-tail imbalance, resulting in large differences in data distribution across domains. Using traditional methods for cross-domain few-shot learning leads to poor detection performance of the trained image classification model. Therefore, there is an urgent need to propose a cross-domain few-shot learning method that can improve the detection performance of image classification models. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for training an image classification model based on cross-domain few-shot learning, in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for training an image classification model based on cross-domain few-shot learning. The method includes: obtaining a target domain support set; the target domain support set includes a small number of first sample images; inputting the first sample images into an initial image classification model pre-trained from source domain sample images to obtain first sample features; the initial image classification model includes a head classifier and a tail classifier; performing head and tail category division on the first sample features corresponding to each first sample image to obtain first head class sample features and first tail class sample features; inputting the first head class sample features into the head classifier to calculate the first prototype of each head class in the target domain, and updating the head classifier based on the first prototype; inputting the first tail class sample features into the tail classifier to calculate the second prototype of each tail class in the target domain, and updating the tail classifier based on the second prototype to obtain a target image classification model suitable for the target domain.

[0006] Secondly, this application provides a training device for an image classification model based on cross-domain few-shot learning, comprising: The feature acquisition module is used to acquire the target domain support set; the target domain support set includes a small number of first sample images; the first sample images are input into an initial image classification model pre-trained from the source domain sample images to obtain the first sample features; the initial image classification model includes a head classifier and a tail classifier; The head and tail segmentation module is used to perform head and tail category segmentation on the first sample features corresponding to each first sample image, and obtain the first head class sample features and the first tail class sample features. The classifier update module is used to input the first head class sample features into the head classifier to calculate the first prototype of each head class category in the target domain, and update the head classifier based on the first prototype; input the first tail class sample features into the tail classifier to calculate the second prototype of each tail class category in the target domain, and update the tail classifier based on the second prototype to obtain a target image classification model suitable for the target domain.

[0007] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in the first aspect.

[0008] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0009] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0010] The aforementioned image classification model training method, apparatus, computer device, computer-readable storage medium, and computer program product based on cross-domain few-shot learning obtain a target domain support set. This target domain support set includes a small number of first sample images. The first sample images are input into an initial image classification model pre-trained from source domain sample images to obtain first sample features. The initial image classification model includes a head classifier and a tail classifier. The first sample features corresponding to each first sample image are divided into head and tail categories to obtain first head class sample features and first tail class sample features. The first head class sample features are input into the head classifier to calculate the first prototype of each head class in the target domain, and the head classifier is updated based on the first prototype. The first tail class sample features are input into the tail classifier to calculate the second prototype of each tail class in the target domain, and the tail classifier is updated based on the second prototype to obtain a target image classification model suitable for the target domain. In other words, by constructing independent classifiers for head and tail samples respectively, and combining head and tail category division processing, fine-grained learning of different category sub-distributions can be achieved, optimizing the effect of cross-domain few-shot learning. This allows for the training of an image classification model with higher detection performance. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating an image classification model training method based on cross-domain few-shot learning in one embodiment. Figure 2 This is a schematic diagram illustrating the principle of initial image classification model pre-training in one embodiment; Figure 3 This is a structural block diagram of an image classification model training device based on cross-domain few-shot learning in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] It should be understood that, unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing specific examples only and are not intended to limit the scope of this application.

[0014] like Figure 1 As shown, in some embodiments, a method for training an image classification model based on cross-domain few-shot learning is provided. This method is applied to a computer device and includes the following steps: S11, Obtain the target domain support set; the target domain support set includes a small number of first sample graphs.

[0015] S12, input the first sample image into the initial image classification model pre-trained from the source domain sample image to obtain the first sample features; the initial image classification model includes a head classifier and a tail classifier.

[0016] It should be understood that an initial image classification model was pre-trained using source domain sample images through multiple iterations. This initial image classification model includes a head classifier and a tail classifier that had been initially trained using the source domain sample images. The head and tail classifiers were specifically trained based on a unique concept in this case. Compared to the traditional approach of using the same general classifier for all samples, this approach essentially constructs independent classifiers for head and tail samples, thereby achieving refined learning of different class sub-distributions. This can, to some extent, solve the problem of inaccurate classification caused by the imbalance between head and tail distributions, and improve detection accuracy.

[0017] The source domain sample map includes the second sample map of the source domain support set and the third sample map of the source domain query set; the image classification model to be trained includes the head classifier and the tail classifier to be trained.

[0018] In some embodiments, during each training iteration, the image classification model to be trained extracts features from each second sample image to obtain corresponding second sample features. For example, the image classification model to be trained includes a trained visual feature extractor, which extracts features from each second sample image, resulting in second sample features. During iterative training, parameter tuning of the visual feature extractor is not required. The second sample features can be directly extracted features or features resulting from further processing of the extracted features (such as enhanced features). Further feature processing is described below and will not be discussed here.

[0019] Furthermore, the second sample features of each second sample image are divided into second head class sample features and second tail class sample features. Second head class sample features refer to second sample features belonging to the head class. Second tail class sample features refer to second sample features belonging to the tail class. Samples of the head class (i.e., second sample images) constitute a large proportion of the source domain support set, while samples of the tail class constitute a small proportion.

[0020] Furthermore, temporary model parameters for the head classifier and tail classifier are determined based on the features of the second head class samples and the features of the second tail class samples, respectively. For example, the features of the second head class samples are input into the head classifier to be trained, and the features of the second tail class samples are input into the tail classifier to be trained, obtaining the class prediction results for each second sample image. Based on the difference between the class prediction results of each second sample image and the corresponding class labels (it should be understood that each sample in the source domain support set—i.e., the second sample image—has a corresponding class label), temporary model parameters for the head classifier and tail classifier are generated. This is equivalent to completing the inner loop processing of this training based on the source domain support set.

[0021] Furthermore, the third-sample features of each third-sample graph in the source domain query set are divided into third-head class sample features and third-tail class sample features. Third-head class sample features refer to third-sample features belonging to the head class. Third-tail class sample features refer to third-sample features belonging to the tail class. Samples in the head class (i.e., third-sample graphs) constitute a large proportion of the source domain query set, while samples in the tail class constitute a small proportion.

[0022] Then, based on the head classifier using temporary model parameters, the class of the third head class sample features is predicted (i.e., the third head class sample features are input into the head classifier, and the head classifier uses the corresponding temporary model parameters to predict the class of the input third head class sample features), a first class prediction result is obtained. A first head classification loss is generated based on the first class prediction result, that is, the first head classification loss is generated according to the difference between the first class prediction result and its corresponding class label. Furthermore, based on the tail classifier using temporary model parameters, the class of the third tail class sample features is predicted, and a first tail classification loss is generated based on the second class prediction result, that is, the first tail classification loss is generated according to the difference between the second class prediction result and its corresponding class label. Then, based on the first head classification loss and the first tail classification loss, the model parameters of the head classifier and tail classifier to be trained (i.e., the original model parameters of the head classifier and tail classifier to be trained) are updated. Thus, the outer loop processing is completed based on the source domain query set, thus completing this training. Based on this training, the next training is performed until the iteration stopping condition is met, obtaining the initial image classification model pre-trained from the source domain sample images.

[0023] In some embodiments, the initial image classification model includes a trained visual feature extractor. Based on the visual feature extractor, features are extracted from a first sample image in the target domain support set to obtain first sample features. The first sample features can be features directly extracted from the first sample image by the visual feature extractor, or features resulting from further processing of the initially extracted features from the first sample image. For example, the initial image classification model also includes a trained conditional generator for feature enhancement. The first sample features can be features enhanced using the conditional generator; the first initial sample features are features extracted from the first sample image using the visual feature extractor. The conditional generator is trained during the iterative training of the image classification model; the specific training process will be described below and will not be discussed here.

[0024] S13, perform head and tail category division on the first sample features corresponding to each first sample image to obtain the first head class sample features and the first tail class sample features.

[0025] Here, the first head class sample feature refers to the first sample feature belonging to the head class category of the target domain. The first tail class sample feature refers to the first sample feature belonging to the tail class category of the target domain. The head class samples (i.e., the first sample image) account for a large proportion of the number of samples in the target domain support set, while the tail class samples account for a small proportion of the number of samples in the target domain support set.

[0026] In some embodiments, the first sample features can be divided into head and tail categories using a power-law function with known power-law parameters to obtain first head class sample features and first tail class sample features. Specifically, the class ranking and power-law parameters of each category in the target domain support set are substituted into the power-law function to obtain the prediction frequency of each category; based on the prediction frequency of each category and a smoothing threshold function, a smoothing threshold factor for each category is determined; and based on the smoothing threshold factor for each category, the first sample features corresponding to each first sample image are divided into head and tail categories to obtain first head class sample features and first tail class sample features.

[0027] The power-law parameter is obtained through statistical calculation of the source domain sample graph. For example, the source domain sample graph is statistically analyzed to obtain the class frequency sequence of the source domain; the power-law parameter is obtained by performing maximum likelihood estimation based on the class frequency sequence.

[0028] For example, the class frequency sequence of the source domain The frequency of a class follows a power-law function. , It is a class ranking (class frequency in descending order). It is a constant, a power-law parameter. Indicates tail strength. The larger the size, the more pronounced the long tail. It is the ranked sample set in the source domain (including the support set and query set). Number of samples in the class This is the total number of classes (the total number of classes in the source domain sample set). Taking the logarithm of the power-law function yields the formula... This involves transforming the problem into a linear regression problem, and then performing maximum likelihood estimation based on the transformed formula and class frequency sequence to solve for the power-law parameters. Solve for the calculated power-law parameters. ,in, This represents the total number of samples participating in the statistics within the source domain sample set, i.e., the sum of the frequency sequences. The detailed process of maximum likelihood estimation calculation will be described below and will not be discussed here.

[0029] In some embodiments, the class ranking of any category c in the target domain support set is implemented. and the power-law parameters to be solved Substitute into the power-law function Calculate the prediction frequency of category c We introduce the sigmoid function for smoothing thresholding, with the following formula:

[0030] in, For prediction frequency The median threshold, For smoothness hyperparameters ( Smooth width, (Approaching the hard threshold). Indicates head category (high frequency). This indicates the tail category (low frequency). That is, if the smoothing threshold factor for a certain category c... Then, the first sample feature of the first sample image belonging to that category in the target domain support set is determined to be a first head class sample feature; if the smoothing threshold factor of a certain category c is... The first sample feature of the first sample image belonging to the first category in the support set of the target domain is determined to be a feature of the first tail class sample. That is, a soft threshold is applied to the class boundary to achieve differentiated enhancement training for different classes and avoid the performance loss caused by hard partitioning.

[0031] In some embodiments, the class ranking of each category in the target domain support set can be determined based on the class frequency of each category or based on the class frequency reliability score.

[0032] In some examples, for each category in the target domain support set, the intra-class divergence and inter-class interval for that category are calculated. Based on the difference between the class interval and the intra-class divergence, the class frequency reliability score corresponding to that category is calculated. Then, the class frequency reliability scores of each category are sorted to obtain the class ranking of each category.

[0033] For example, in the target domain support set, the intra-class divergence of any class c The calculation formula is: ; in, This represents the set of first sample graphs belonging to class c in the target domain support set; i is the sample index. It is a set The first sample feature of the i-th first sample image. It is a set The first sample feature of the j-th first sample image. Characterizing the features of the first sample Features of the first sample The distance between them, where K is the number of samples in class c.

[0034] In some examples, the inter-class spacing of any class c The calculation formula is: ; Where K is the number of samples in category c. It is a category The number of samples in Represents a set The first sample feature of the i-th first sample image. Represents a set The first sample feature of the j-th first sample image. Characterizing the features of the first sample Features of the first sample The distance between them.

[0035] In some examples, the class frequency reliability score of category c The calculation formula is: ; In some examples, the class frequency reliability scores of each category are sorted from largest to smallest to obtain the class ranking for each category. This is achieved using the following formula: ;in, This indicates the class ranking of category c.

[0036] S14, input the features of the first head class sample into the head classifier to calculate the first prototype of each head class in the target domain, and update the head classifier based on the first prototype.

[0037] Specifically, the features of the first head class sample are input into the head classifier trained in the initial image classification model. Based on the head classifier, the prototype of each head class category in the target domain, i.e., the first prototype, is calculated. Based on the first prototype, the prototype of each head class category in the source domain (i.e., the third prototype) is replaced, resulting in an updated head classifier that is adapted to the head class sample features of the target domain (i.e., a classifier that can accurately classify the head class sample features of the target domain).

[0038] S15, input the features of the first tail class sample into the tail classifier to calculate the second prototype of each tail class in the target domain, update the tail classifier based on the second prototype, and obtain the target image classification model applicable to the target domain.

[0039] Specifically, the first tail class sample features are input into the tail classifier trained in the initial image classification model. Based on the tail classifier, the prototype of each tail class category in the target domain, i.e. the second prototype, is calculated. Based on the second prototype, the prototype of each tail class category in the source domain (i.e. the fourth prototype) is replaced, resulting in an updated tail classifier that is adapted to the tail class sample features in the target domain (i.e., a classifier that can accurately classify the tail class sample features in the target domain).

[0040] The second prototype for each tail class can be the mean of the features of each first tail class sample under that tail class, or it can be the mean of the features of each first tail class sample under that tail class and the tail class sample features under that tail class expanded by the data augmentation module in the tail classifier.

[0041] Thus, the resulting target image classification model has a head classifier that adapts to the head class sample features of the target domain and a tail classifier that adapts to the tail class sample features of the target domain. Therefore, it is suitable for image classification in the target domain and can accurately classify the input image to be classified in the target domain based on the head classifier or the tail classifier.

[0042] In some embodiments, for any current image to be classified within the target domain, the current image is input into a target image classification model applicable to the target domain to extract features from the current image and obtain target features. For example, features are extracted from the current image based on the visual feature extractor in the target image classification model, and the initially extracted image is enhanced based on the conditional generator in the target image classification model to obtain target features. Target features can also be features extracted directly from the current image; this is not limited.

[0043] Furthermore, based on the distance between the reference prototype and the target feature, the target head and tail categories of the target feature are identified. The reference prototype is a prototype used as a reference in the process of identifying the head and tail categories of the target features in the image to be classified. The reference prototype can be either a head class prototype or a tail class prototype. Specifically, the head class prototype is determined by a first prototype that integrates all head classes in the target domain (e.g., by averaging the first prototypes of all head classes in the target domain); the tail class prototype is determined by a second prototype that integrates all tail classes in the target domain (e.g., by averaging the second prototypes of all tail classes in the target domain). The target head and tail categories are used to characterize whether the target feature is a head class feature or a tail class feature.

[0044] In some examples, taking the head class prototype as a reference prototype, the probability of the target feature belonging to the head class set is identified based on the distance between the head class prototype and the target feature of the current image. Based on this probability, it is determined whether the target feature belongs to the head class or the tail class. For example, the probability of the head class set is compared with a preset probability threshold (such as 0.5). If it is greater than the preset probability threshold, it is determined to be a head class; if it is less than or equal to the preset probability threshold, it is determined to be a tail class.

[0045] It should be understood that any image to be classified in the target domain can be an image in the target domain query set, or an image to be classified with an unknown category after the model is deployed.

[0046] Taking the current image as an example that is an image in the target domain query set, the target features of the image in the target domain query set can be calculated according to the following formula. Probability of belonging to the head class set : ; in, The header class prototype represents the target domain. Representing target features With head class prototype The distance between them.

[0047] Furthermore, a target classifier corresponding to the target head and tail categories can be selected from the head and tail classifiers in the target image classification model. The target features are then input into the target classifier to obtain the category to which the current image belongs. For example, if the target head and tail category indicates that the target features are tail features, then the target features are input into the tail classifier in the target image classification model.

[0048] It should be understood that the scheme of this application trains a tail classifier suitable for small sample features, which can classify small sample features more accurately, making up for the problem that traditional methods use a fixed classifier with poor ability to detect and recognize small sample features. Therefore, in this case, the target features are input into the target classifier corresponding to the target head and tail categories. Even if the image to be classified belongs to a small number of sample categories (that is, its target head and tail categories represent that the target features are tail features), it can be accurately classified by the tail classifier.

[0049] In some embodiments, during each training iteration using the second sample map of the source domain support set, after classifying each second sample feature into second head class sample features and second tail class sample features, the second head class sample features can be input into the head classifier to be trained. The head classifier then uses standard meta-learning to compute prototypes; that is, prototypes are computed for second head class sample features of the same category, resulting in the third prototypes for each head class in the source domain. For example, the second sample map corresponding to the second head class sample features can be denoted as the second head class sample map. The third prototype of any head class c in the source domain... Calculated using the following formula: ; in, It is the set of second sample graphs belonging to class c in the head class set H of the source domain support set (i.e., the set of second head class sample graphs belonging to class c). It is a set The number of samples in It is a set The i-th second sample image The second sample feature. For example, the second sample feature... These could be enhanced features, for example, visual feature extractors from second sample images. After extracting the initial features, they are input into the condition generator for feature enhancement to obtain the second sample features. .

[0050] Furthermore, class prediction of the third head class sample features based on the head classifier using temporary model parameters includes: based on the head classifier using temporary model parameters, predicting the head class corresponding to each third head class sample feature based on the distance between each third head class sample feature and the third prototype of each head class, and obtaining the first class prediction result.

[0051] It should be understood that the first category prediction result for each third head class sample feature includes the set of predicted probabilities that the third head class sample feature belongs to each head class.

[0052] For example, the predicted probability that each third head class sample feature belongs to any head class c can be calculated using the following formula: ; in, This represents a subset of header-type queries in the source domain query set. Any third sample image in The predicted probability of belonging to category c, where category c is any head class in the head class set H; It is the third-head class sample feature (i.e., the third sample image) (the third sample feature) It is the third prototype of head class c; Indicates the features of the third-head class samples With the third prototype The distance between them.

[0053] In some embodiments, a first-head classification loss is generated based on the difference between the first-class prediction result corresponding to the features of each third-head class sample and the corresponding class label.

[0054] For example, the first-head classification loss is calculated using the following formula. : in, It is a header query set in the source domain query set. Any third sample image in the image, It is the third sample image The actual category labels, H represents the head class set (i.e., the head class query set). The set of third prototypes for each head class in ) This represents the third sample in the first category prediction results. Belongs to the head category The predicted probability.

[0055] In some embodiments, during each training iteration using the source domain support set, the computer device can expand and enhance the features of the second tail class samples to obtain enhanced tail class sample features; a target tail class sample feature set is then formed based on the enhanced tail class sample features and the second tail class sample features. In this way, tail class enhancement sampling achieves generalization of scarce samples, which can mitigate long tail domain shifting to some extent, thereby improving the recognition ability of tail class samples.

[0056] For example, the data augmentation module using the tail classifier generates augmented tail class sample features using the following formula. :

[0057] in, This represents the set of second-tailed class sample graphs supporting class c in the source domain. and Sets The second-tailed class sample features of the i-th and j-th second-tailed class sample images are used. The second-tailed class sample features are... and Weighted fusion generates enhanced tail class sample features for category c. .

[0058] Enhanced tail class sample features for each category in the source domain The features of the target tail class samples are combined with the features of each second tail class sample to form the target tail class sample feature set.

[0059] Furthermore, the features of each tail class sample in the target tail class sample feature set (including each original second tail class sample feature and each enhanced tail class sample feature) can be denoted as: High-frequency noise filtering is performed to obtain the corresponding target tail class sample features.

[0060] For example, for each tail class sample feature in the target tail class sample feature set Perform a Fourier transform to decompose the sample features. From the low-frequency semantics and high-frequency noise components, the high-frequency noise is filtered out to obtain the target tail class sample features. This is to enhance local discriminative features. The specific calculation formula is as follows: . This represents a low-pass filter used to filter high-frequency noise; FFT() represents Fourier transform. This represents the inverse Fourier transform.

[0061] Furthermore, the fourth prototype of each tail category in the source domain is calculated based on the features of each target tail class sample. The fourth prototype of any category c in the source domain is calculated using the following formula. : ; in, It is the set of tail class sample features belonging to category c in the target tail class sample feature set. It is a set The total number of tail class sample features, It is a set Features of the i-th target tail class sample.

[0062] In this embodiment, the category prediction of the third tail class sample features based on the tail classifier using temporary model parameters includes: based on the tail classifier using temporary model parameters, predicting the tail class of each third tail class sample feature based on the distance between each third tail class sample feature and the fourth prototype of each tail class, and obtaining the second category prediction result.

[0063] It should be understood that the second category prediction result for each third tail sample feature includes the set of predicted probabilities that the third tail sample feature belongs to each tail category.

[0064] The third-tailed sample features can be either raw features extracted directly from the image or enhanced features. Taking the enhanced features as an example, high-frequency noise can be filtered out first to obtain the target tailed sample features. Based on the characteristics of the target tail class samples The distance between the third tail class and the fourth prototype of each tail class is used to predict the corresponding tail class. For example, the predicted probability that each third tail class sample feature belongs to any tail class c is calculated using the following formula: ; in, This represents the third sample graph in the tail class set T of the source domain query set. The predicted probability of belonging to category c, where category c is any tail class in the tail class set T; It is the third sample image The target tail class sample features, It is the fourth prototype of tail category c; Representing the characteristics of the target tail class samples With the fourth prototype The distance between them.

[0065] In some embodiments, generating a first tail classification loss based on the second category prediction result includes: determining an initial tail classification loss based on the difference between the second category prediction result and the corresponding category label; generating a first tail classification loss based on the initial tail classification loss and the tail class adjustment coefficients of each tail class in the source domain; wherein the tail class adjustment coefficient of each tail class is determined based on the power law parameter and the class ranking of the tail class in the source domain, and is used to compensate for the head-tail imbalance.

[0066] For example, the first-tail classification loss is calculated using the following formula. : ; in, It is the tail class adjustment factor of the tail class c in the source domain. It is the tail-type query set in the source domain query set. Any third sample image in the image; It is the third sample image The actual tail category; Represents the tail class set T (i.e., the tail class query set) The set of fourth prototypes for each tail category in ) This represents the third sample plot in the second category prediction results. Belongs to the real category The predicted probability.

[0067] In some embodiments, the image classification model to be trained further includes a trained visual feature extractor and a text feature extractor, as well as a condition generator and a discriminator to be trained.

[0068] It should be understood that the process of performing one iteration of training using the source domain sample graph includes inner loop processing using the source domain support set and outer loop processing using the source domain query set. For example... Figure 2 As shown, in each iteration of training, the backbone network of the feature extraction layer... (i.e., the visual feature extractor) extracts visual features from the second sample image in the source domain support set of the input layer. By backbone network (i.e., the text feature extractor) extracts semantic features from the textual description information of the second sample image. Furthermore, semantic features Projecting onto the visual space yields projection features. Projection features .in, Let b be the projection weight matrix, and b be the bias term. Representation layer normalization.

[0069] Furthermore, based on visual features and projection features Generate enhanced features This enhanced feature can be used as a second sample feature. Specifically, through a condition generator... With projection features As Conditional visual features Feature enhancement is performed, dynamically focusing on visual texture to generate second-sample features. This reduces texture shift, bridges domain displacement, provides hierarchical guidance for categories, and compensates for insufficient tail-class visual information.

[0070] like Figure 2 As shown, in the feature enhancement layer, the condition generator Combined with mapping matrix , and visual features and projection features Processing is performed to generate enhanced features. .

[0071] For example, enhanced features (i.e., second sample features) are generated using the following formula. :

[0072] in, It is a learnable mapping matrix; The key / query vector dimension is used to scale the attention score to avoid excessive inner product leading to Softmax saturation and to improve training stability. It preserves visual texture under semantic guidance while reducing overlap offset.

[0073] Furthermore, the condition generator G is based on enhanced features. and noise Generate offset sample features ,Right now .

[0074] Please continue reading. Figure 2 This will enhance the features (i.e., true sample features, label y=1) and offset sample features (i.e., fake sample features, label y=0) are input into the discriminator D to be trained for generative adversarial training. Discriminator This is a binary classification network used to determine the probability that an input sample feature belongs to a true sample feature. Thus, an adversarial loss (generated by...) can be generated. Figure 2 (The adversarial generation loss and adversarial discrimination loss are composed of...)

[0075] In some embodiments, during adversarial training, the condition generator To minimize discriminator misclassification, hence the adversarial generation loss. The calculation formula is as follows:

[0076] Discriminator To maximize the features that distinguish between real and fake samples, hence the adversarial discriminative loss. The calculation formula is as follows: ; in, The discriminator believes It is the probability of a feature of a true sample. The discriminator believes It is the probability that it is a feature of a true sample.

[0077] It should be understood that condition generators and discriminator By constructing a zero-sum game and using adversarial learning to jointly improve performance, the adversarial generation loss is minimized. Let the generator Generate more realistic features and maximize adversarial discriminative loss Let the discriminator To better distinguish between real and fake sample features, specifically, the overall adversarial loss can be represented by the following formula: .

[0078] It should be understood that in iterative training, Continuously improve alignment quality. By continuously improving the ability to discriminate, the Nash equilibrium is eventually reached. Unable to distinguish between real and fake The generated features are highly aligned.

[0079] In some embodiments, the original hard label (category label) y is softened to generate a smooth soft label. Based on soft tags The difference between the discrimination result and the discriminator's result generates a smoothing loss. For example, it is calculated using the following formula:

[0080] ; in, It is a smoothing factor, and y is a hard label. The discriminator believes It represents the probability of features from genuine samples. In this embodiment, the smooth loss can smooth gradients during adversarial training and prevent the discriminator from becoming overconfident and causing pattern collapse, thus avoiding instability during training with few samples.

[0081] Please continue reading. Figure 2 After discrimination by discriminator D, in the classification layer, each true augmented feature is separated by a power-law function (whose power-law parameter is a known value). The second sample features are divided into head class features and tail class features. The head class features (i.e., the second head class features) are input into the head classifier to determine the head classification loss (i.e., the second head classification loss); the tail class features (i.e., the second tail class features) are input into the tail classifier to determine the tail classification loss (i.e., the second tail classification loss). Then, based on the adversarial loss (composed of adversarial generation loss and adversarial discriminant loss), smoothing loss, the second head classification loss, and the second tail classification loss, the parameters of the conditional generator and discriminator to be trained are updated, and the temporary model parameters of the head and tail classifiers are determined to complete the inner loop processing of this training. In other embodiments, the smoothing loss may not be calculated, and this is not limited.

[0082] It should be understood that the parameters of the condition generator and discriminator are only updated and adjusted using the source domain support set, while the final parameters of the head classifier and tail classifier are updated and adjusted using the source domain query set. For example, please refer to [link to relevant documentation]. Figure 2 After the inner loop processing in the source domain is completed, the third sample graphs in the source domain query set are input into the backbone network. Extracting visual features z Q v The parameters are then input into the condition generator G, which has been updated in the inner loop (for example, the updated parameters of the condition generator G after this inner loop processing can be directly copied—i.e., parameter sharing; or the condition generator G can be shared with the inner loop processing—i.e., model sharing; this is not limited here). The output is the enhanced feature, i.e., the third sample feature. Based on the power-law function, each third sample feature is divided into head and tail categories. The head class sample feature (i.e., the third head class sample feature) is input into the head classifier using the temporary model parameters determined in the inner loop processing to determine the head classification loss (i.e., the first head classification loss); the tail class sample feature (i.e., the third tail class sample feature) is input into the tail classifier using the temporary model parameters determined in the inner loop processing to determine the tail classification loss (i.e., the first tail classification loss); then, based on the first head classification loss and the first tail classification loss, the model parameters of the head classifier and tail classifier to be trained are updated (i.e., updated based on the initial model parameters before the start of this training). This completes the outer loop processing for this training. The data augmentation module in the tail classifier is only used for processing the support set (source and target domains) to improve the accuracy of the second and fourth prototypes. It is not used for processing the source and target query sets.

[0083] The above scheme adopts a cross-modal conditional adversarial generation mechanism. In addition to the visual semantic features of the image, it also uses the textual semantic features of the image's textual description information as the conditional input of the generator. Combined with gradient smoothing loss, it achieves the alignment of visual features between the source domain and the target domain to alleviate the problem of domain distribution misalignment.

[0084] For example, each batch of training uses of The update ratio (i.e., in a batch of training, the conditional generator G is updated m times and the discriminator D is updated once) is used to prevent the discriminator from becoming too strong and causing the generator to overfit, while maintaining training stability.

[0085] It should be understood that traditional power-law models often use hard thresholds such as the median or KS statistics to distinguish between head and tail classes, forcing the classification of boundary classes and ignoring gradual changes, leading to fragmentation and amplifying domain shift bias in long-tailed CD-FSL. This paper constructs a smooth-boundary power-law distribution, making the classification continuous. The weights are based on the power-law prediction frequency weighted sample contribution, which can better handle boundary ambiguity and improve tail class generalization. The calculation process of the power-law parameter α will be described in detail below.

[0086] 1. Given the class frequency sequence of the source domain (Sorted in descending order of frequency) (Ranked by class), which follows a power-law distribution: Here, α is the power-law parameter to be solved, and A is a constant. It is a ranking category.

[0087] 2. Construct the likelihood function, based on the general hypothesis for count data, assuming the observation frequency. Independently obeys the parameter is The Poisson distribution, in which The likelihood function is then:

[0088] 3. Take the negative log-likelihood:

[0089] Will Substituting, we get

[0090] right Optimize, fix Find the partial derivative and let :

[0091] Solving Estimate : (1) Substitute equation (1) into ,have ; The original negative log-likelihood then simplifies to: ; right Differentiate:

[0092] in .Order Then there is (8) 4. Due to the scoring equation yes For complex nonlinear functions without analytical solutions, approximate analytical solutions can be calculated using variance stability transformation. Definition: (9) Equation (1) Rewritten as:

[0093] 5. Taylor expansion approximation for real but unknown power-law parameters. Nearby Taylor expansion (first order):

[0094] Where the derivative is: ; Substitute (3) into equation (2):

[0095] Solving for:

[0096] 6. To solve right Dependencies, define the construction of variance-stable transformation (VST):

[0097] Fisher's information content: ; in, In the power-law model with parameters of At that time, the "ranking random variable" induced by this model — Under the distribution of "class ranking", random variables The variance.

[0098] choose (Due to long-tail distribution) ),but:

[0099] when : therefore:

[0100] 7. The limit of information content behavior. Definition of Riemann function:

[0101] when Using Riemann Function properties:

[0102] Therefore, integral of equation (5) is:

[0103] 8. Inverse Transform and Closed-Form Solution. Define the sample mean: ; Will Applying this to equation (4), we can obtain the following by transforming the left side:

[0104] Real expectations: According to the Central Limit Theorem: Transform the standardization on the right side to Rewritten as:

[0105] when ,have Divergent, but if we assume The limiting behavior and transformation matching can be approximated as follows: ; 9. Combining equations (6) and (7), we can obtain:

[0106] when hour: , Exponential approximation: ,because : The population size is approximated by sample statistics. In implementation, the observation frequency is used. Calculate sample statistics and sample variance Used to replace unknown theoretical quantities and Combining formulas (9) and (2), we can obtain:

[0107] Then there is .

[0108] in, These are the power-law parameters that are calculated.

[0109] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. At least some of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0110] Based on the same inventive concept, this application also provides an image classification model training device for implementing the aforementioned cross-domain few-shot learning-based method. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the image classification model training device based on cross-domain few-shot learning provided below can be found in the limitations of the image classification model training method based on cross-domain few-shot learning described above, and will not be repeated here.

[0111] In one embodiment, such as Figure 3 As shown, an image classification model training device based on cross-domain few-shot learning is provided. The device includes: The feature acquisition module 302 is used to acquire the target domain support set; the target domain support set includes a small number of first sample images; the first sample images are input into an initial image classification model pre-trained from the source domain sample images to obtain the first sample features; the initial image classification model includes a head classifier and a tail classifier; The head and tail feature segmentation module 304 is used to segment the head and tail categories of the first sample features corresponding to each first sample image, so as to obtain the first head class sample features and the first tail class sample features. The head and tail classifier update module 306 is used to input the first head class sample features into the head classifier to calculate the first prototype of each head class category in the target domain, and update the head classifier based on the first prototype; input the first tail class sample features into the tail classifier to calculate the second prototype of each tail class category in the target domain, and update the tail classifier based on the second prototype, so as to obtain a target image classification model applicable to the target domain.

[0112] It should be understood that the device can also be used to implement other steps of the image classification model training method based on cross-domain few-shot learning in the embodiments of this application.

[0113] The modules in the image classification model training device based on cross-domain few-shot learning described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0114] In one embodiment, a computer device is provided, which may be a terminal or a server. The internal structure diagram of the computer device may be as follows: Figure 4As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a training method for an image classification model based on cross-domain few-shot learning.

[0115] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the embodiments of this application.

[0117] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the embodiments of this application.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited thereto.

[0119] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above embodiments are merely illustrative of several implementation methods of this application and should not be construed as limiting the scope of this patent application. Those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training an image classification model based on cross-domain few-shot learning, characterized in that, The method includes: Obtain the target domain support set; the target domain support set includes a small number of first sample images; The first sample image is input into the initial image classification model pre-trained from the source domain sample image to obtain the first sample features; the initial image classification model includes a head classifier and a tail classifier; The first sample features corresponding to each first sample image are divided into head and tail categories to obtain the first head class sample features and the first tail class sample features. The first head class sample features are input into the head classifier to calculate the first prototype of each head class category in the target domain, and the head classifier is updated based on the first prototype. The first tail class sample features are input into the tail classifier to calculate the second prototype of each tail class in the target domain. The tail classifier is updated based on the second prototype to obtain a target image classification model applicable to the target domain.

2. The method according to claim 1, characterized in that, The initial image classification model is obtained by iteratively training the image classification model to be trained using source domain sample maps; the source domain sample maps include the second sample map of the source domain support set and the third sample map of the source domain query set; The image classification model to be trained includes a head classifier and a tail classifier. The method further includes: In each iteration of training, the second sample features of each second sample image are divided into second head class sample features and second tail class sample features; the temporary model parameters of the head classifier and the tail classifier are determined based on the second head class sample features and the second tail class sample features, respectively. The third sample features of each third sample image are divided into third head class sample features and third tail class sample features; The head classifier using temporary model parameters is used to predict the category of the third head class sample features, and the first head classification loss is generated based on the first category prediction result. The tail classifier using temporary model parameters is used to predict the category of the third tail class sample features, and the first tail classification loss is generated based on the second category prediction result. The model parameters of the head classifier and tail classifier to be trained are updated based on the first head classification loss and the first tail classification loss.

3. The method according to claim 2, characterized in that, The method further includes: The prototype is calculated for the features of the second head class samples of the same category, and the third prototype of each head class in the source domain is obtained. The method of predicting the category of third-head class samples based on the head classifier using temporary model parameters includes: Based on a head classifier using temporary model parameters, the head class corresponding to each third head class sample feature is predicted according to the distance between each third head class sample feature and the third prototype of each head class category, thus obtaining the first category prediction result.

4. The method according to claim 2, characterized in that, The method further includes: The features of the second tail class sample are expanded and enhanced to obtain the enhanced tail class sample features; a target tail class sample feature set is formed based on the enhanced tail class sample features and the second tail class sample features. High-frequency noise is filtered on the features of each tail class in the target tail class sample feature set to obtain the features of each target tail class sample; the fourth prototype of each tail class in the source domain is calculated based on the features of each target tail class sample. The method of predicting the category of third-tailed class samples based on the tail classifier using temporary model parameters includes: Based on a tail classifier using temporary model parameters, the tail class corresponding to each third tail class sample feature is predicted according to the distance between each third tail class sample feature and the fourth prototype of each tail class category, thus obtaining the second category prediction result.

5. The method according to claim 4, characterized in that, The generation of the first-tail classification loss based on the second-class prediction result includes: The initial tail classification loss is determined based on the difference between the second category prediction result and the corresponding category label; The first tail classification loss is generated based on the initial tail classification loss and the tail class adjustment coefficients of each tail class in the source domain; wherein the tail class adjustment coefficient of each tail class is determined based on the power law parameter and the class ranking of the tail class in the source domain.

6. The method according to claim 2, characterized in that, The initial image classification model includes a trained visual feature extractor and a conditional generator for feature enhancement; the first sample feature is the feature enhanced by the conditional generator on the first initial sample feature. The first initial sample features are extracted from the first sample image using the visual feature extractor.

7. The method according to claim 6, characterized in that, The image classification model to be trained also includes a trained visual feature extractor and a text feature extractor, as well as a conditional generator and a discriminator to be trained; before dividing the second sample features of each second sample image into second head class sample features and second tail class sample features, the method further includes: In each training iteration, the visual feature extractor extracts visual features from the second sample image; the text feature extractor extracts semantic features from the text description information of the second sample image. The semantic features are projected onto the visual space to obtain the projected features; The enhanced second sample features are generated based on visual and projection features using the conditional generator to be trained; offset sample features are generated based on the second sample features and noise. The second sample features and the offset sample features are input into the discriminator to be trained for generative adversarial training to obtain the adversarial loss; The temporary model parameters for the head classifier and tail classifier, determined based on the features of the second head class samples and the features of the second tail class samples, respectively, include: The features of the second head class sample are input into the head classifier to determine the second head classification loss; The features of the second tail class sample are input into the tail classifier to determine the second tail classification loss; Based on the adversarial loss, the second head classification loss, and the second tail classification loss, the parameters of the condition generator and discriminator to be trained are updated, and the temporary model parameters of the head classifier and tail classifier are determined.

8. The method according to claim 1, characterized in that, The method further includes: Statistical analysis is performed on the source domain sample graph to obtain the class frequency sequence of the source domain; Based on the aforementioned frequency sequence, maximum likelihood estimation is performed to obtain the power-law parameters. The step of dividing the first sample features corresponding to each first sample image into head and tail categories to obtain the first head class sample features and the first tail class sample features includes: Substitute the class rankings of each category in the target domain support set and the power law parameters into the power law function to obtain the prediction frequency of each category; Based on the prediction frequency and smoothing threshold function of each category, the smoothing threshold factor of each category is determined. Based on the smoothing threshold factor of each category, the first sample features corresponding to each first sample image are divided into head and tail categories to obtain the first head class sample features and the first tail class sample features.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Any current image to be classified within the target domain is input into the target image classification model to extract features from the current image and obtain target features; Based on the distance between the reference prototype and the target feature, the target head and tail categories of the target feature are identified; the reference prototype is a head category prototype or a tail category prototype; the head category prototype is determined by a first prototype that integrates all head categories in the target domain; the tail category prototype is determined by a second prototype that integrates all tail categories in the target domain; the target head and tail categories are used to characterize whether the target feature is a head category feature or a tail category feature. Select the target classifier corresponding to the target head and tail categories from the head classifier and tail classifier in the target image classification model; The target features are input into the target classifier to obtain the category to which the current image belongs.

10. A training device for an image classification model based on cross-domain few-shot learning, characterized in that, The device includes: The feature acquisition module is used to acquire the target domain support set; the target domain support set includes a small number of first sample images; the first sample images are input into an initial image classification model pre-trained from the source domain sample images to obtain the first sample features; the initial image classification model includes a head classifier and a tail classifier; The head and tail segmentation module is used to perform head and tail category segmentation on the first sample features corresponding to each first sample image, and obtain the first head class sample features and the first tail class sample features. The classifier update module is used to input the first head class sample features into the head classifier to calculate the first prototype of each head class category in the target domain, and update the head classifier based on the first prototype; input the first tail class sample features into the tail classifier to calculate the second prototype of each tail class category in the target domain, and update the tail classifier based on the second prototype to obtain a target image classification model suitable for the target domain.

Images