Long tail learning method based on fourier transform and contrastive learning

Abstract

The application discloses a long-tail learning method based on Fourier transform and contrastive learning, comprising the following steps: extracting samples from a data set by adopting two different sampling strategies, and dividing the samples into head category samples and tail category samples; inputting the extracted samples into a backbone network, adding a Fourier transform module after each residual block of the backbone network, and migrating style information of the head category data to tail category data; adding a contrastive learning module, performing contrastive learning on feature maps of the tail category after style migration and original tail category, and maximizing similarity between the tail category data after style migration and original data; and inputting the feature maps of the head category and the tail category into a classifier to obtain a classification result. According to the scheme, the diversity of the tail category features can be enhanced by using the style information of the head category features while keeping the feature semantic information of the tail category unchanged, so that the generalization capability of the model on the tail category is improved.

Description

Technical Field

[0001] This invention relates to image classification, specifically to a long-tail learning method based on Fourier transform and contrastive learning, a long-tail data distribution method, and a computer-readable storage medium. Background Technology

[0002] The long-tail data distribution problem refers to a situation in a dataset where the number of samples in each category is imbalanced, meaning that a few categories (head classes) constitute the majority of the samples, while the majority categories (tail classes) constitute only a small portion. This data imbalance poses challenges to the training and testing of deep learning models, especially for the recognition performance of tail classes. For example, in the field of medical image disease diagnosis, due to the varying incidence rates of different diseases, common diseases have a large number of samples, while rare diseases have a small number of samples, thus exhibiting a long-tail distribution problem, which increases the difficulty of training and testing deep learning models.

[0003] To address the problem of long-tailed data distribution, several methods have been proposed, which can be mainly divided into two categories: methods based on resampling or reweighting and methods based on feature learning. For example, K. Cao et al. (K. Cao, C. Wei, A. Gaidon, N. Arechiga, T. Ma, Learning imbalanced datasets with label-distribution-aware margin loss, Advances in neural information processing systems 32 (2019)) proposed a theoretical label-distribution-aware margin loss. However, this method is still based on reweighting or resampling, which does not effectively increase the diversity of tail classes but simply reuses existing samples, which may lead to overfitting or model bias. Another example is X. Chen et al. (X. Chen, Y. Zhou, D. Wu, W. Zhang, Y. Zhou, B. Li, W. Wang, Imagine by reasoning: A reasoning-based implicit semantic dataaugmentation for long-tailed classification, in: Proceedings of the AAAIConference on Artificial Intelligence, Vol. 36, 2022, pp. 356-364.) proposes an implicit semantic data augmentation method based on reasoning, which adaptively augments tail class samples. However, this method ignores the feature correlation between head class and tail class, which often contains more diverse feature information than tail class.

[0004] Therefore, the purpose of this invention is to provide a new solution to the long-tail data distribution problem. This method can enhance the diversity of tail category features by utilizing the style information of head category features while keeping the feature semantic information of tail category unchanged, thereby improving the model's generalization ability on tail category. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a long-tail learning method based on Fourier transform and contrastive learning. This method can enhance the diversity of tail category features by utilizing the style information of head category features while maintaining the semantic information of tail category features, thereby improving the model's generalization ability on tail categories.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a long-tail learning method based on Fourier transform and contrastive learning, comprising the following steps.

[0007] Two different sampling strategies, uniform sampling and flip sampling, are used to extract samples from the dataset. Uniform sampling is used to obtain head category samples, and flip sampling is used to obtain tail category samples. The number of head category samples is greater than the number of tail category samples.

[0008] The extracted samples are input into the backbone network, and a Fourier transform module is added after each residual block of the backbone network to transfer the style information of the head category features to the tail category features.

[0009] A contrastive learning module is added to the last layer of the backbone network to compare and learn the features of the tail category after style transfer and the original tail category. SimSiam Loss is used to maximize the similarity between the tail category data after style transfer and the original data.

[0010] The feature maps of the head category and the style-transferred tail category are input into a classifier to obtain their classification results.

[0011] As a further improvement of the present invention, the Fourier transform module includes the following execution steps:

[0012] The feature maps of the head category and the tail category are obtained from the output of the previous layer of the network, and a certain proportion of the feature maps of the head category and the tail category are selected.

[0013] A two-dimensional Fourier transform is performed on the selected feature map to obtain its complex frequency domain representation;

[0014] The frequency domain representation is decomposed into two parts: amplitude and phase.

[0015] Define a mask to obtain the low-frequency part of the amplitude;

[0016] Style transfer is achieved by replacing the low-frequency component of the amplitude of the tail category with the low-frequency component of the amplitude of the head category.

[0017] The replaced amplitude and the phase of the original tail category are merged, and then an inverse Fourier transform is performed to obtain the feature map of the tail category after style transfer.

[0018] The feature maps of the selected style-transferred tail categories are concatenated with the original feature maps of the unselected categories;

[0019] The tail category data after style transfer is residually connected with the original tail category data to obtain the feature map of the next layer.

[0020] As a further improvement of the present invention, in the Fourier transform module, the proportion of the selected feature map is γ∈(0,1);

[0021] In the Fourier transform module, the size of the mask is βH×βW, where β∈(0,0.5), and H and W represent the height and width of the feature map, respectively.

[0022] The weight of the residual connection is λ. residual , where λ residual ∈(0,1).

[0023] As a further improvement of the present invention, the comparison learning module includes the following execution steps:

[0024] The style-transferred tail category feature map obtained after the last layer of the backbone network is processed by the Fourier transform module, and the original tail category feature map are respectively input into two perceptrons to obtain their prediction vectors.

[0025] SimSiam Loss is used to calculate the similarity between tail category data, i.e., negative cosine similarity;

[0026] The gradients of the feature maps of the tail categories that have not undergone style transfer are blocked, so that the feature maps of the tail categories after style transfer are not affected by the gradients of the feature maps of the tail categories that have not undergone style transfer.

[0027] As a further improvement of the present invention, the two perceptrons are a mapping perceptron based on a fully connected layer and a predictive perceptron based on a fully connected layer. The mapping perceptron transforms the input data and maps the image to a low-dimensional vector space, while the predictive perceptron maps the projection vector to an even lower-dimensional vector space for predicting the category.

[0028] The beneficial effects of this invention are:

[0029] (1) By introducing a Fourier transform module, this invention can enhance the diversity of tail category data by utilizing the style information of head category data while maintaining the semantic information of tail category data, thereby improving the classification accuracy of the model on all categories, especially improving the generalization ability on tail category.

[0030] (2) By introducing a contrastive learning module, this invention can effectively use contrastive learning to narrow the data distribution between the tail category data after style transfer and the original data, making the data distribution of the tail category more compact, reducing the difference in data distribution, thereby improving the stability and convergence of the model;

[0031] (3) This invention is a general module that can be added to any long-tail learning method with ResNet and its derivative networks as the backbone network. Without changing the original sampler, optimizer or hyperparameters, it can improve the recognition performance of tail categories. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0033] Figure 2 This is a schematic diagram of the Fourier transform module of the present invention;

[0034] Figure 3 This is a schematic diagram of the contrast learning module of the present invention;

[0035] Figure 4 This is a schematic diagram of the basic steps of the present invention. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0037] Reference Figure 1-4 As shown,

[0038] Figure 1 This is a schematic diagram of the overall framework of the present invention. Figure 4 This is a schematic diagram of the basic steps of the present invention. As shown in the figure, the present invention includes the following main components:

[0039] Step S1: Two different sampling strategies are used to extract samples from the dataset, which are divided into head category samples and tail category samples.

[0040] Step S2: Input the extracted samples into the backbone network. A Fourier transform module is added after each residual block of the backbone network to transfer the style information of the head category data to the tail category data.

[0041] Step S3: Add a contrastive learning module to the last layer of the backbone network to perform contrastive learning on the feature maps of the style-transferred tail category and the original tail category, and use SimSiam Loss to maximize the similarity between the style-transferred tail category data and the original data.

[0042] Step S4: Input the feature maps of the head category and the style-transferred tail category into a classifier to obtain their classification results.

[0043] In step S1, two different sampling strategies are employed: uniform sampling and flipped sampling. Uniform sampling follows the original data distribution, while flipped sampling uses the inverse ratio of the sample size to the sampling probability; that is, the more samples a category has, the lower its sampling probability. Uniform sampling yields samples for the head category, and flipped sampling yields samples for the tail category. h and I t The input images represent the head category and the tail category, respectively.

[0044] In step S2, combined Figure 2 The extracted samples are input into the backbone network. A Fourier transform module is added after each residual block of the backbone network to transfer the style information of the head category data to the tail category data. The specific implementation steps are as follows:

[0045] Step S2.1: Let the output of the i-th residual block be... and in and Let represent the feature maps obtained by uniform sampling and flip sampling, respectively. B represents the batch size, C represents the number of channels, and H and W represent the height and width of the feature map, respectively. γB feature maps are randomly selected for style transfer, where γ∈(0,1), and in this example, γ=0.5. The remaining 1-γB feature maps are not subjected to style transfer. The selected feature maps are represented as follows: and in This represents the selected head category feature map. This represents the selected tail category feature map.

[0046] Step S2.2, for and Performing two-dimensional Fourier transforms on each part yields its complex frequency domain representation. The formula for calculating the two-dimensional Fourier transform is as follows:

[0047]

[0048] in, h and w represent the coordinates in the spatial domain, and m and n represent the coordinates in the frequency domain. This represents the frequency domain representation of S after Fourier transform.

[0049] Step S2.3: Decompose the frequency domain representation into two parts: amplitude and phase. The calculation formula is as follows:

[0050]

[0051] in, This represents the amplitude of S after its Fourier transform. This represents the phase of S after its Fourier transform.

[0052] Step S2.4: Define a mask M β β∈(0,1), the size of the mask is βH×βW, and the calculation formula is as follows:

[0053]

[0054] in It is an indicator function, [-βH:βH, -βW:βW] represents a rectangular region, and in this example, β = 0.1;

[0055] Step S2.5: Replace the low-frequency component of the amplitude of the tail category with the low-frequency component of the amplitude of the head category. The calculation formula is as follows:

[0056]

[0057] Where ⊙ represents element-wise multiplication, i.e., the Hadamard product. This represents the tail category features after style transfer.

[0058] Step S2.6: Combine the amplitude of the tail category after replacement with the phase of the original tail category, and then perform an inverse Fourier transform to obtain the tail category data after style transfer. The calculation formula is as follows:

[0059]

[0060] in This represents the inverse Fourier transform.

[0061] Step S2.7: Concatenate the γB feature maps of the style-transferred tail category with the 1-γB feature maps of the original tail category. The calculation formula is as follows:

[0062]

[0063] in, This represents the feature map of the tail category obtained after concatenation, and [:] indicates the concatenation operation according to the channel dimension.

[0064] Step S2.8: Combine the concatenated tail category features Compared with the original tail category features Perform residual connections to obtain the feature map of the next layer. The calculation formula is as follows:

[0065]

[0066] Where, λ residual The weights of the residual connections are λ, which in this example are λ and λ.residual =0.1;

[0067] In step S3, combined Figure 3 A contrastive learning module is added to the last layer of the backbone network to perform comparative learning on the feature maps of the style-transferred tail categories and the original tail categories. SimSiam Loss is used to maximize the similarity between the style-transferred tail category data and the original data. The specific implementation steps are as follows:

[0068] Step S3.1: Take the output of the last layer of the backbone network and pass it through the Fourier transform module to obtain the style-transferred tail category feature map X. t ′ and the feature map X of the original tail category t The data are input into two perceptrons respectively, and their prediction vectors p are obtained. t ′ and p t :

[0069] z t =f(X) t ), z t ′=f(X t ′)

[0070] Where f(·) is a fully connected layer-based mapping perceptron that transforms the input data and maps the image to a low-dimensional vector space;

[0071] p t =h(z) t ), p t ′=h(z t ′)

[0072] Where h(·) is a predictive perceptron based on a fully connected layer, used to map the projected vector to a lower-dimensional vector space for predicting the category;

[0073] Step S3.2: Use SimSiam Loss to calculate the similarity between tail category data, i.e.

[0074]

[0075] sim(·) is the negative cosine similarity, and sg(·) is the function that prevents gradient backpropagation.

[0076] The formula for calculating the negative cosine similarity sim(·) is as follows:

[0077]

[0078] Where ||·|| represents the L2 norm.

[0079] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A long-tail learning method based on Fourier transform and contrastive learning, characterized in that, Includes the following steps Two different sampling strategies, uniform sampling and flip sampling, are used to extract samples from the dataset. Uniform sampling is used to obtain head category samples, and flip sampling is used to obtain tail category samples. The number of head category samples is greater than the number of tail category samples. Both head category samples and tail category samples are image samples. The extracted samples are input into the backbone network, and a Fourier transform module is added after each residual block of the backbone network to transfer the style information of the head category features to the tail category features. A contrastive learning module is added to the last layer of the backbone network to compare and learn the features of the tail category after style transfer and the original tail category. SimSiam Loss is used to maximize the similarity between the tail category data after style transfer and the original data. The feature maps of the head category and the style-transferred tail category are input into a classifier to obtain their classification results; The Fourier transform module includes the following execution steps: The feature maps of the head category and the tail category are obtained from the output of the previous layer of the network, and a certain proportion of the feature maps of the head category and the tail category are selected. A two-dimensional Fourier transform is performed on the selected feature map to obtain its complex frequency domain representation; The frequency domain representation is decomposed into two parts: amplitude and phase. Define a mask to obtain the low-frequency part of the amplitude; Style transfer is achieved by replacing the low-frequency component of the amplitude of the tail category with the low-frequency component of the amplitude of the head category. The replaced amplitude and the phase of the original tail category are merged, and then an inverse Fourier transform is performed to obtain the feature map of the tail category after style transfer. The feature maps of the selected style-transferred tail categories are concatenated with the original feature maps of the unselected categories; The tail category data after style transfer is residually connected with the original tail category data to obtain the feature map of the next layer.

2. The long-tail learning method according to claim 1, characterized in that, In the Fourier transform module, the proportion of the selected feature map is... ; and / or In the Fourier transform module, the size of the mask is ,in , and These represent the height and width of the feature map, respectively. and / or The weights of the residual connections are ,in .

3. The long-tail learning method according to claim 2, characterized in that, The comparative learning module includes the following execution steps: The style-transferred tail category feature map obtained after the last layer of the backbone network is processed by the Fourier transform module, and the original tail category feature map are respectively input into two perceptrons to obtain their prediction vectors. SimSiam Loss is used to calculate the similarity between tail category data, i.e., negative cosine similarity; The gradients of the feature maps of the tail categories that have not undergone style transfer are blocked, so that the feature maps of the tail categories after style transfer are not affected by the gradients of the feature maps of the tail categories that have not undergone style transfer.

4. The long-tail learning method according to claim 3, characterized in that, The two perceptrons are a mapping perceptron based on a fully connected layer and a predictive perceptron based on a fully connected layer. The mapping perceptron transforms the input data, mapping the image to a low-dimensional vector space, while the predictive perceptron maps the projected vector to an even lower-dimensional vector space for predicting the category.

Images