Cross-domain small sample image classification method based on deep frequency domain prior

Abstract

The application relates to a cross-domain small sample image classification method based on a deep frequency domain prior, which is composed of an image decomposition module and a prior regular module. The image decomposition module takes an original image as input and is used for decomposing the original image into a high-frequency image part and a low-frequency image part; the prior regular module is composed of a three-branch prototype network and is used for processing the original image, the high-frequency image and the low-frequency image respectively. In the structure, each branch comprises a feature extractor and a prototype classifier, which are respectively used for feature extraction and category membership prediction. Between the three branches, the application designs a feature reconstruction prior regular term and a prediction consistency prior regular term to relieve the overfitting problem of the model in the source domain training process, so as to improve the generalization effect of the model on the target domain.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a cross-domain few-sample image classification method based on depth frequency domain prior. Background Technology

[0002] Image classification is a core foundation of image processing technology. Although significant progress has been made in image classification algorithms, existing methods typically rely on training with large-scale labeled data. However, in some extreme task domains, there are often practical problems such as the difficulty in obtaining and labeling data, resulting in image classification algorithms being trained with only a small amount of labeled data. To alleviate the problem of insufficient data, utilizing source domain knowledge, which is easy to collect, to help solve few-shot tasks in real-world cross-domain scenarios is a feasible approach. However, due to the distributional differences between source and target domain data, the knowledge learned in the source domain is difficult to effectively apply to the target domain task in real-world scenarios. Therefore, research on few-shot image classification techniques for cross-domain scenarios has significant application value. The literature "Snell J, Swersky K, Zemel R. Prototypical networks for few shot learning[C] / / Advances in Neural Information Processing Systems.2017:4077-4087." proposes a few-shot image classification method based on prototype networks. This method uses a meta-learning algorithm to mine cross-task shared knowledge from a series of few-shot tasks sampled from the source domain to help the model generalize to the target task. However, in real-world applications, there are significant distributional differences between the target domain task and the source domain task, causing meta-learning algorithms to tend to overfit to the source domain task, thus limiting their generalization performance on cross-domain few-sample tasks. Summary of the Invention

[0003] Technical problems to be solved

[0004] To avoid the shortcomings of existing technologies, this invention proposes a cross-domain few-sample image classification method based on depth frequency domain prior.

[0005] Technical solution

[0006] A cross-domain few-shot image classification method based on deep frequency domain priors, characterized by using source domain datasets... Learning cross-domain transferable knowledge to solve problems in the target domain. Small sample task The classification steps are as follows:

[0007] Step 1: Source Domain Small sample task By support set and query set Composition, in which support set and query set Each includes N categories and supports sets. Each category has K supervised samples, abbreviated as N-way K-shot; query set Each category has M unlabeled samples;

[0008] Step 2: For small sample tasks Each image X in the dataset is decomposed to obtain the low-frequency task. and high-frequency tasks

[0009] Step 3: Prior Regularization Module Calculation: A prior regularization module consisting of a main branch, a high-frequency branch, and a low-frequency branch is used to process the original small sample task. High-frequency tasks and low-frequency tasks Obtaining the original small sample task Category prediction score High-frequency tasks Category prediction score low-frequency tasks Category prediction score

[0010] Step 4, Distribution Alignment Regularization: Regularize the main branch network by aligning the high- and low-frequency prediction distributions with the original prediction distributions.

[0011]

[0012] in, For distribution alignment loss, D KL This indicates the calculation of the Kullback-Leibler divergence loss between two distributions;

[0013] Step 5, Feature Reconstruction Regularization Terms:

[0014] A projection network is used to combine the original image X and the low-frequency image X. low and high-frequency image X high The corresponding features are mapped to a low-dimensional space, and then the mean squared error loss between the high- and low-frequency fused features and the original features is calculated in the low-dimensional space:

[0015] z X =g η (f θ (X))

[0016]

[0017]

[0018] in, Let g be the feature reconstruction loss for image X, MSE be the mean squared error loss function, and g be the mean squared error loss function. η For a feature projection network, f θ f φ and These are feature extraction networks in the main branch, low-frequency branch, and high-frequency branch, respectively.

[0019] Step 6: Based on the cross-entropy loss calculated in Step 3, the distribution alignment loss calculated in Step 4, and the feature reconstruction loss calculated in Step 5, the total loss for this small sample task is:

[0020]

[0021] in, This represents the set of query images in a few-shot task. This represents the set of all images in this few-sample task;

[0022] Step 7: Based on the loss calculated in Step 6, use stochastic gradient descent to train the network parameters θ of the main branch and the projected network parameters η end-to-end.

[0023] For the network parameters φ of the low-frequency branch and the network parameters of the high-frequency branch Update using the exponential moving average of the main branch network parameters:

[0024] φ←m1φ+(1-m1)θ

[0025]

[0026] Where m1 and m2 represent the momentum coefficients in the exponential moving average update;

[0027] After training, the image decomposition module, high-frequency branch, and low-frequency branch are discarded, and only the main branch is retained to process the few-sample image classification task in the target domain. The calculation process for each few-sample image classification task in the target domain is the same as in step 3.

[0028] The image X is decomposed as follows:

[0029] First, a Fast Fourier Transform is used to map the image to the frequency domain:

[0030] in, Indicates Fourier transform, and These represent the low-frequency and high-frequency components of the original image X in the frequency domain, respectively.

[0031] The inverse Fourier transform is used to map the high-frequency and low-frequency components in the frequency domain to the image space respectively:

[0032]

[0033]

[0034] in, Represents the inverse Fourier transform, X low and X high These represent the low-frequency and high-frequency image components of the original image X, respectively.

[0035] For small sample tasks Each image in the dataset is decomposed to form a low-frequency task. and high-frequency tasks

[0036] The original small sample task Category prediction score The calculation is:

[0037] Based on the support set in step 1 Building a small sample task Prototype representation:

[0038]

[0039] in, Indicates support set The k-th sample of the n-th class, f θ C represents the feature extraction network in the main branch. n The prototype representation of the nth category;

[0040] Then, based on the prototype of the appeal, the query set For each sample in the dataset, predict its class affiliation:

[0041]

[0042] in, Represents query set The i-th query sample in the data. This represents the category prediction score of the sample; matching(·) is the similarity calculation function;

[0043] The similarity calculation function uses the inverse of the Euclidean distance between two vectors as the similarity calculation function.

[0044] The high-frequency task Category prediction score The calculation is:

[0045] First, based on the support set in high-frequency tasks Building high-frequency tasks Prototype representation:

[0046]

[0047] in, Indicates support set The k-th sample of the n-th category, This represents the feature extraction network in the high-frequency branch. The prototype representation of the nth category;

[0048] Then, based on the prototype of the appeal, the query set For each sample in the dataset, predict its class affiliation:

[0049]

[0050] in, Represents query set The i-th query sample in the data. This represents the category prediction score of the sample; matching(·) is the similarity calculation function;

[0051] The similarity calculation function uses the inverse of the Euclidean distance between two vectors as the similarity calculation function.

[0052] The low-frequency task Category prediction score The calculation is:

[0053] First, based on the support set in low-frequency tasks Building low-frequency tasks Prototype representation:

[0054]

[0055] in, Indicates support set The k-th sample of the n-th class, f φ This represents the feature extraction network in the low-frequency branch. The prototype representation of the nth category;

[0056] Then, based on the prototype of the appeal, the query set For each sample in the dataset, predict its class affiliation:

[0057]

[0058] in, Represents query set The i-th query sample in the data. This represents the category prediction score of the sample; matching(·) is the similarity calculation function;

[0059] The similarity calculation function uses the inverse of the Euclidean distance between two vectors as the similarity calculation function.

[0060] The source and target domain categories do not overlap.

[0061] The momentum coefficient m1 in the exponential moving average update is set to 0.997.

[0062] The momentum coefficient m2 in the exponential moving average update is set to 0.999.

[0063] The feature extraction networks in the main branch, high-frequency branch, and low-frequency branch are all ResNet-10 structures.

[0064] The projection network is a single-layer fully connected neural network.

[0065] Beneficial effects

[0066] This invention proposes a cross-domain few-sample image classification method based on deep frequency domain priors, consisting of an image decomposition module and a prior regularization module. The image decomposition module takes the original image as input and decomposes it into high-frequency and low-frequency image components. The prior regularization module consists of a three-branch prototype network, used to process the original image, high-frequency image, and low-frequency image respectively. Structurally, each branch includes a feature extractor and a prototype classifier, used for feature extraction and class membership prediction, respectively. Between the three branches, this invention designs a feature reconstruction prior regularization term and a prediction consistency prior regularization term to alleviate the overfitting problem during model training in the source domain, thereby improving the model's generalization performance in the target domain.

[0067] The beneficial effects of this invention are: by constructing a frequency prior regularization term, it alleviates the problem of model overfitting during the source domain training stage, thereby improving the model's cross-domain generalization potential. The method of this invention achieves good classification performance in cross-domain few-shot image classification. For example, using the miniImageNet dataset as the source domain and testing it on the bird fine-grained classification dataset CUB and the agricultural dataset CropDisease as the target domain, the method of this invention achieves classification accuracies of 51.55% and 71.47% respectively on the 5-way 1-shot task. Compared to prototype-based few-shot image classification methods, these represent improvements of 4.65% and 1.58% respectively. Detailed Implementation

[0068] The present invention will now be further described with reference to the embodiments:

[0069] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0070] Step 1, Data Preprocessing:

[0071] Cross-domain few-shot image classification task description: Cross-domain few-shot image classification aims to classify images from source domain datasets... Learning cross-domain transferable knowledge to solve problems in the target domain. Small sample task The source and target domain categories do not overlap. Few-sample tasks are performed on each target domain. By support set and query set Composition. Among them... It includes N categories, each with K supervised samples, abbreviated as N-way K-shot. This also includes N categories, each with M unlabeled samples. A task-specific classifier is constructed based on the support set for each task to classify the samples in the query set, and the classification accuracy on the query set is used as the performance metric for that task. Typically, 600 few-shot tasks are randomly selected in each target domain, and the average accuracy on these tasks is used as the evaluation metric for the algorithm in that target domain. To simulate the few-shot tasks that continuously arrive during the testing phase, few-shot tasks are constructed in the same way for model training during the training phase.

[0072] Step 2: Image decomposition. For the source domain For each image X in the small sample task, the image is first mapped to the frequency domain using a Fast Fourier Transform:

[0073]

[0074] in, Indicates Fourier transform, and These represent the low-frequency and high-frequency components of the original image X in the frequency domain, respectively. Then, an inverse Fourier transform is used to map the high-frequency and low-frequency components in the frequency domain to the image space, respectively.

[0075]

[0076]

[0077] in, Represents the inverse Fourier transform, X low and X highThese represent the low-frequency and high-frequency image components of the original image X, respectively. Following the same approach, this is applied to the few-sample task. Each image in the process is decomposed to obtain the low-frequency task. and high-frequency tasks

[0078] Step 3: Calculation of the prior regularization module. The prior regularization module designed in this invention consists of a main branch, a high-frequency branch, and a low-frequency branch, which are used to process the original task, respectively. High-frequency tasks and low-frequency tasks For the main branch, firstly, based on the support set in step 1... Building a small sample task Prototype representation:

[0079]

[0080] in, Indicates support set The k-th sample of the n-th class, f θ C represents the feature extraction network in the main branch. n This represents the prototype representation of the nth category. Then, the query set is processed based on the aforementioned prototype. For each sample in the dataset, predict its class affiliation:

[0081]

[0082] in, Represents query set The i-th query sample in the data. This represents the predicted category score for the sample. `matching(·)` is the similarity calculation function; this invention uses the negative of the Euclidean distance between two vectors as the similarity calculation function. Finally, based on the predicted score... The category corresponding to the maximum similarity in the data is used as the query sample. Predicted labels And calculate the cross-entropy loss based on the predicted label and the true label of the query sample:

[0083]

[0084] Where H(·) represents the cross-entropy loss function, This represents the actual label corresponding to the query sample.

[0085] For the low-frequency and high-frequency branches, the same procedure as for the main branch is followed to calculate the prediction scores for the query samples in the low-frequency and high-frequency tasks, respectively. and

[0086] The high-frequency task Category prediction score The calculation is:

[0087] First, based on the support set in high-frequency tasks Building high-frequency tasks Prototype representation:

[0088]

[0089] in, Indicates support set The k-th sample of the n-th category, This represents the feature extraction network in the high-frequency branch. The prototype representation of the nth category;

[0090] Then, based on the prototype of the appeal, the query set For each sample in the dataset, predict its class affiliation:

[0091]

[0092] in, Represents query set The i-th query sample in the data. This represents the category prediction score of the sample; matching(·) is the similarity calculation function;

[0093] The similarity calculation function uses the inverse of the Euclidean distance between two vectors as the similarity calculation function.

[0094] The low-frequency task Category prediction score The calculation is:

[0095] First, based on the support set in low-frequency tasks Building low-frequency tasks Prototype representation:

[0096]

[0097] in, Indicates support set The k-th sample of the n-th class, f φ This represents the feature extraction network in the low-frequency branch. The prototype representation of the nth category;

[0098] Then, based on the prototype of the appeal, the query set For each sample in the dataset, predict its class affiliation:

[0099]

[0100] in, Represents query set The i-th query sample in the data. This represents the category prediction score of the sample; matching(·) is the similarity calculation function;

[0101] The similarity calculation function uses the inverse of the Euclidean distance between two vectors as the similarity calculation function.

[0102] Step 4: Align the distribution with regularization terms.

[0103] For the query samples in the original small sample task Yes, after the calculations in steps 2 and 3, the original prediction corresponding to the query sample is obtained. High-frequency prediction and low-frequency prediction This invention regularizes the main branch network by aligning the high- and low-frequency prediction distributions with the original prediction distribution, thereby alleviating the overfitting problem of the model. The distribution alignment regularization term is calculated as follows:

[0104]

[0105] in, For distribution alignment loss, D KL This indicates the calculation of the Kullback-Leibler divergence loss between two distributions.

[0106] Step 5: Feature reconstruction regularization terms.

[0107] This invention designs a feature reconstruction regularization term to further enhance the consistency between the original branch and the high- and low-frequency branches. First, a projection network is used to reconstruct the original image X and the low-frequency image X... low and high-frequency image X high The corresponding features are mapped to a low-dimensional space, and then the mean squared error loss between the high- and low-frequency fused features and the original features is calculated in the low-dimensional space:

[0108] z X =g η (f θ (X)) (8)

[0109]

[0110]

[0111] in, Let g be the feature reconstruction loss for image X, MSE be the mean squared error loss function, and g be the mean squared error loss function. η For a feature projection network, fθ f φ and These are feature extraction networks in the main branch, low-frequency branch, and high-frequency branch, respectively.

[0112] Step 6: Model training.

[0113] Based on the cross-entropy loss calculated in step 3, the distribution alignment loss calculated in step 4, and the feature reconstruction loss calculated in step 5, the total loss for this small sample task can be derived as follows:

[0114]

[0115] in, This represents the set of query images in a few-shot task. This represents the set of all images in this few-sample task.

[0116] Step 7: Based on the loss calculated in Step 6, use stochastic gradient descent to train the network parameters θ of the main branch and the projected network parameters η end-to-end. For the network parameters φ of the low-frequency branch and the network parameters η of the high-frequency branch... Update using the exponential moving average of the main branch network parameters:

[0117] φ←m1φ+(1-m1)θ (12)

[0118]

[0119] Wherein, m1 and m2 represent the momentum coefficients in the exponential moving average update, and in this invention, m1 and m2 are set to 0.997 and 0.999, respectively.

[0120] The feature extraction networks in the main branch, high-frequency branch, and low-frequency branch are all ResNet-10 structures, and the projection network is a one-layer fully connected neural network.

[0121] Once the model training is complete, this invention discards the image decomposition module, high-frequency branches, and low-frequency branches, retaining only the main branch to process the few-sample image classification task in the target domain. The computation process for each few-sample image classification task in the target domain is the same as in step 3.

[0122] This invention achieves superior classification performance in cross-domain few-shot image classification tasks. For example, using the mini-ImageNet dataset as the source domain and testing it on the bird fine-grained classification dataset CUB and the agricultural dataset CropDisease as the target domains, the method of this invention achieves classification accuracies of 51.55% and 71.47% respectively on the 5-way 1-shot task. Compared to prototype-based few-shot image classification methods, these represent improvements of 4.65% and 1.58% respectively.

Claims

1. A cross-domain few-sample image classification method based on depth frequency domain prior, characterized in that... From source domain dataset Learn cross-domain transferable knowledge to solve problems in the target domain. Small sample task The classification steps are as follows: Step 1: Source Domain Small sample task By support set and query set Composition, in which support set and query set Each includes N categories and supports sets. Each category has K supervised samples, abbreviated as N-way K-shot; query set Each category has M unlabeled samples; Step 2: For small sample tasks Each image X in the dataset is decomposed to obtain the low-frequency task. and high-frequency tasks Step 3: Prior Regularization Module Calculation: A prior regularization module consisting of a main branch, a high-frequency branch, and a low-frequency branch is used to process the original small sample task. High-frequency tasks and low-frequency tasks Obtaining the original small sample task Category prediction score High-frequency tasks Category prediction score low-frequency tasks Category prediction score Step 4, Distribution Alignment Regularization: Regularize the main branch network by aligning the high- and low-frequency prediction distributions with the original prediction distributions. in, For distribution alignment loss, D KL This indicates the calculation of the Kullback-Leibler divergence loss between two distributions; Step 5, Feature Reconstruction Regularization Terms: A projection network is used to combine the original image X and the low-frequency image X. low and high-frequency image X high The corresponding features are mapped to a low-dimensional space, and then the mean squared error loss between the high- and low-frequency fused features and the original features is calculated in the low-dimensional space: z X =g η (f θ (X)) in, Let g be the feature reconstruction loss for image X, MSE be the mean squared error loss function, and g be the mean squared error loss function. η For a feature projection network, f θ f φ and These are feature extraction networks in the main branch, low-frequency branch, and high-frequency branch, respectively. Step 6: Based on the cross-entropy loss calculated in Step 3, the distribution alignment loss calculated in Step 4, and the feature reconstruction loss calculated in Step 5, the total loss for this small sample task is: in, This represents the set of query images in a few-shot task. This represents the set of all images in this few-sample task; Step 7: Based on the loss calculated in Step 6, use stochastic gradient descent to train the network parameters θ of the main branch and the projected network parameters η end-to-end. For the network parameters φ of the low-frequency branch and the network parameters of the high-frequency branch Update using the exponential moving average of the main branch network parameters: Where m1 and m2 represent the momentum coefficients in the exponential moving average update; After training, the image decomposition module, high-frequency branch, and low-frequency branch are discarded, and only the main branch is retained to process the few-sample image classification task in the target domain. The calculation process for each few-sample image classification task in the target domain is the same as in step 3.

2. The cross-domain few-sample image classification method based on depth frequency domain prior as described in claim 1, characterized in that: The image X is decomposed as follows: First, a Fast Fourier Transform is used to map the image to the frequency domain: in, Indicates Fourier transform, and These represent the low-frequency and high-frequency components of the original image X in the frequency domain, respectively. The inverse Fourier transform is used to map the high-frequency and low-frequency components in the frequency domain to the image space respectively: in, Represents the inverse Fourier transform, X low and X high These represent the low-frequency and high-frequency image components of the original image X, respectively. For small sample tasks Each image in the dataset is decomposed to form a low-frequency task. and high-frequency tasks 3. The cross-domain few-sample image classification method based on depth frequency domain prior as described in claim 1, characterized in that: The original small sample task Category prediction score The calculation is: Based on the support set in step 1 Building a small sample task Prototype representation: in, Indicates support set The k-th sample of the n-th class, f θ C represents the feature extraction network in the main branch. n The prototype representation of the nth category; Then, based on the above prototype, the query set For each sample in the dataset, predict its class affiliation: in, Represents query set The i-th query sample in the data. This represents the category prediction score of the sample; matching(·) is the similarity calculation function; The similarity calculation function uses the inverse of the Euclidean distance between two vectors as the similarity calculation function.

4. The cross-domain few-sample image classification method based on depth frequency domain prior as described in claim 1, characterized in that: The high-frequency task Category prediction score The calculation is: First, based on the support set in high-frequency tasks Building high-frequency tasks Prototype representation: in, Indicates support set The k-th sample of the n-th category, This represents the feature extraction network in the high-frequency branch. The prototype representation of the nth category; Then, based on the above prototype, the query set For each sample in the dataset, predict its class affiliation: in, Represents query set The i-th query sample in the data. This represents the category prediction score of the sample; matching(·) is the similarity calculation function; The similarity calculation function uses the inverse of the Euclidean distance between two vectors as the similarity calculation function.

5. The cross-domain few-sample image classification method based on depth frequency domain prior as described in claim 1, characterized in that: The low-frequency task Category prediction score The calculation is: First, based on the support set in low-frequency tasks Building low-frequency tasks Prototype representation: in, Indicates support set The k-th sample of the n-th class, f φ This represents the feature extraction network in the low-frequency branch. The prototype representation of the nth category; Then, based on the above prototype, the query set For each sample in the dataset, predict its class affiliation: in, Represents query set The i-th query sample in the data. This represents the category prediction score of the sample; matching(·) is the similarity calculation function; The similarity calculation function uses the inverse of the Euclidean distance between two vectors as the similarity calculation function.

6. The cross-domain few-sample image classification method based on depth frequency domain prior as described in claim 1, characterized in that: The source and target domain categories do not overlap.

7. The cross-domain few-sample image classification method based on depth frequency domain prior as described in claim 1, characterized in that: The momentum coefficient m1 in the exponential moving average update is set to 0.

997.

8. The cross-domain few-sample image classification method based on depth frequency domain prior as described in claim 1, characterized in that: The momentum coefficient m2 in the exponential moving average update is set to 0.

999.

9. The cross-domain few-sample image classification method based on depth frequency domain prior as described in claim 1, characterized in that: The feature extraction networks in the main branch, high-frequency branch, and low-frequency branch are all ResNet-10 structures.

10. The cross-domain few-sample image classification method based on depth frequency domain prior according to claim 1, characterized in that: The projection network is a single-layer fully connected neural network.

Images