A method for generating an unbalanced data image with a characteristic interpolation of cyclical nature

Abstract

The application discloses a feature interpolation unbalanced data image generation method with cyclic properties, comprising: data set preprocessing, which is used for artificially generating an unbalanced data set; building a self-encoder neural network, wherein a convolution is used in the encoder stage to extract features from the image and reduce dimensions, and a deconvolution is used in the decoder stage to restore features of the image; new samples are constructed by using linear interpolation of the features, and image generation is carried out through the decoder; in order to ensure the semantic attributes of the new samples, the encoder is used to form a cycle at the feature level, and the cycle consistency is constrained; meanwhile, the cross-domain distance consistency is adopted to ensure the mutual corresponding relationship between the features and the consistency between the images. The application alleviates the problems that the existing generation model falls into local optimization, the generated samples lack diversity, and the visual fidelity is poor, and the generated samples are helpful for a classification model to obtain higher performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and artificial intelligence, and particularly relates to an unbalanced data image generation method with feature interpolation of cyclic properties. BACKGROUND

[0002] With the development of artificial intelligence technology, deep learning models have been widely applied in academia and industry. However, many existing studies show that deep learning models face the challenge of class imbalance. More specifically, the vast training data often presents an imbalanced or long-tailed label distribution, which makes the model pay more attention to the classes with more samples and ignore the classes with fewer samples.

[0003] To alleviate the class imbalance problem, following recent research, methods can be simply divided into algorithm-level and data-level methods. The idea of algorithm-level methods is to encourage the model to focus on minority classes through some cost-sensitive strategies; the data-level method is to balance the dataset by increasing the minority class samples or reducing the majority class samples. In recent years, generative adversarial networks have been used to generate images with high visual fidelity, and researchers have proven that these images can be used as additional training data to improve classification or other applications. Unfortunately, on imbalanced datasets, conventional image generation methods often fail, and more specifically, image generation models have difficulty in ensuring the reliability and diversity of augmented samples. Therefore, it is of great significance to design an image generation method that can adapt to imbalanced distribution under the widespread imbalance phenomenon.

[0004] Existing generation models have difficulty in ensuring the diversity of generated samples, and this phenomenon is more prominent and obvious in the case of a small number of samples. Existing generation models usually use changes in input features to control image generation changes, but this artificial input parameter control often cannot achieve the desired effect, as generative adversarial networks often have mode collapse problems. Unlike existing generation methods, in the present application, the problem of insufficient sample diversity is improved by constructing the cyclic consistency of features and the cross-domain distribution consistency, in addition, to enhance the authenticity of generated images and the robustness of the image generation network, the network of the present application additionally trains pictures other than the target image set to learn general knowledge. SUMMARY

[0005] The present application provides an unbalanced data image generation method with feature interpolation of cyclic properties to address the mode collapse phenomenon often occurring in existing image generation methods and the difficulty in adjusting the changes in features.

[0006] Technical solution: The unbalanced data image generation method with feature interpolation of cyclic properties provided by the present application comprises the following steps:

[0007] (1) Manually unbalance the pre-acquired balanced dataset, select some classes as minority classes, and divide them into a training set and a test set;

[0008] (2) Build an image generation network model based on feature interpolation, use a fully convolutional autoencoder network as the backbone network, and reuse the encoder structure to form a cycle at the feature level;

[0009] (3) Determine the loss function of the image generation network model and train the image generation network model on the training set;

[0010] (4) Use the trained image generation network model to oversample the minority classes in the training set to balance the dataset;

[0011] (5) Use a unified benchmark classifier to verify the improvement of the proposed image generation method on the unbalanced dataset.

[0012] Further, the step (1) is implemented as follows:

[0013] Randomly select half of the classes in the dataset as minority classes, and downsample the number of selected class samples to two specifications, 20 and 150.

[0014] Further, the first half of the autoencoder network in step (2) is an encoder network, which uses a 4x4 convolution kernel size and a step size of 4 for feature dimension reduction. After each convolution operation, batch normalization and Leaky Relu activation function are performed. The second half is a decoder network, which uses a 4x4 deconvolution module with a step size of 4. After each convolution operation, batch normalization and Relu activation function are performed. The last activation function is tanh, which restores the compressed high-level features to the original image.

[0015] Further, the loss function of the image generation network model in step (3) is:

[0016] The loss is optimized by mean square error loss and KL divergence loss. The mean square error loss measures the difference between two samples, while the KL divergence loss measures the difference between two distributions. The total loss consists of three losses:

[0017] Loss is the basic reconstruction loss of the autoencoder structure, which is used to train the basic autoencoder module to reconstruct the image data. The purpose is to learn the most basic dimension reduction mapping function and image generation mapping function. The loss is as follows:

[0018]

[0019] where x nFor the original sample, E is the encoder module, D is the decoder module, and the loss measures the pixel distance between the original image and its restored version;

[0020] Loss The cycle-consistent loss for feature interpolation coding is used to measure the restoration effect of the interpolation feature, deepen the correspondence between the feature space and the original image space, and distinguish the generation effect of different features:

[0021]

[0022] wherein, The interpolation sample is artificially constructed, E and D are the encoder module and the decoder module respectively, and the loss measures the Euclidean distance between the interpolation feature and its restored version.

[0023] Loss The cross-space consistency loss is used to further strengthen the correlation between the feature space and the original space, and alleviate the mode collapse phenomenon, and the loss is as follows:

[0024]

[0025] wherein, G f is the sample relationship of the feature level arranged in order, and G x is the sample relationship of the image level arranged in order, sim(·) represents similarity calculation, and cosine similarity is usually used to measure the similarity relationship between samples; finally, the correlation of the two sets of normalized graph relationships is calculated using KL divergence; the total loss is represented as follows:

[0026]

[0027] wherein, λ is the cross-space consistency loss weight.

[0028] Further, the interpolation sample is constructed as follows:

[0029]

[0030] wherein, f1 and f2 are the encoded versions of the real samples x1 and x2 respectively, and α is the interpolation ratio of the two encodings, and the value range is between 0 and 1.

[0031] Beneficial effects: compared with the prior art, in the application scene of unbalanced data, the uniqueness of the generated sample is guaranteed by constructing the cycle-consistent loss of the feature and the cross-domain consistency loss of the feature space and the image space, the stability of the feature interpolation generated image is improved, and the mode collapse phenomenon of the conventional image generation method is alleviated; compared with the latest image generation method, the invention can more stably improve the classification effect of the deep model in the unbalanced scene. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Flowchart of the present application

[0033] Figure 2 Network structure of the autoencoder and Implementation of the loss

[0034] Figure 3 Improved network structure of the present application and Implementation of the loss

[0035] Figure 4 Legend explanation of the cross-domain distribution consistency Loss

[0036] Figure 5 Comparison chart of the present application and the generation results of different generation models DETAILED DESCRIPTION

[0037] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0038] The present application provides an unbalanced data image generation method with cyclic characteristic interpolation, as shown in Figure 1 , specifically comprising the following steps:

[0039] Step 1: Prepare the data sets MNIST and E-MNIST, and artificially unbalance the training data set. Specifically, select half of the categories as the minority categories, and reduce the sample quantity of the minority categories to 20 and 150.

[0040] Step 2: Build an image generation network model based on feature interpolation, taking a fully convolutional autoencoder network as the backbone network, and reusing the encoder structure to form a cycle at the feature level.

[0041] A cascaded structure of the encoder and the decoder is constructed to learn the mutual transformation function between the image sample and its low-dimensional encoding format. The overall network structure is a fully convolutional autoencoder network, as shown in Figure 2 The first half of the autoencoder network is the encoder network, which can reduce the input image to a lower-dimensional but higher semantic high-level feature; the second half is the decoder network, which can restore the compressed high-level feature to the original image. Unlike conventional autoencoder networks, the present application additionally increases the encoder module after the decoder, so that the feature and the image can achieve a double-loop structure, as shown in Figure 3The encoder and the decoder structure are in full convolution form. The encoder uses a 4x4 convolution kernel size and a step size of 4 for feature dimension reduction. After each convolution operation, batch normalization and Leaky Relu activation function are performed. The decoder is also a 4x4 deconvolution module with a step size of 4. After each convolution operation, batch normalization and Relu activation function are performed. It should be noted that the activation function of the last layer is tanh.

[0042] Step 3: Determine the loss function of the image generation network model, and train the image generation network model on the training set.

[0043] The unbalanced MNIST and E-MNIST training data sets prepared in step 1 are trained using the network model defined in step 2. The input image size is set to 32x32. During training, the model's training objects are not only the minority class samples, but also the majority class samples as auxiliary samples to optimize model parameter learning. The purpose is to learn general pixel knowledge from the majority class samples. It should be noted that in one batch of data during training, the number of target minority class images and auxiliary sample images controlled by the application is the same, which ensures that the majority class samples do not encroach on the position information of the minority samples in the feature space. The network model parameters are optimized by mean square error loss and KL divergence loss, where the mean square error loss measures the difference between two samples, and the KL divergence loss measures the difference between two distributions. More specifically, the total loss consists of three losses:

[0044] Loss 1 is the basic reconstruction loss of the autoencoder structure, as shown in Figure 2 , which is used to measure the reconstruction ability of the model for training image data. The purpose is to learn the basic dimension reduction mapping function and image generation mapping function. The loss is as follows:

[0045]

[0046] where x n is the original sample, E is the encoder module, and D is the decoder module. This loss measures the pixel distance between the original image and its restored version.

[0047] Loss 2 is the core loss of the application, which is the cyclic consistency loss of feature interpolation coding. It is used to measure the restoration effect of interpolated features and deepen the correspondence between the feature space and the original image space, as shown in Figure 3 . Specifically, the spatial mapping of a conventional image model is biased, and the image generator tends to generate similar samples, which is called mode collapse. To alleviate this problem, the application uses the following loss to distinguish the generation effect of different features:

[0048]

[0049] where, is the artificial constructed interpolated sample, E and D are the encoder module and decoder module respectively, and the loss measures the Euclidean distance between the interpolated feature and its recovered version. The interpolated sample is constructed as follows:

[0050]

[0051] where, f1 and f2 are the encoded versions of the real samples x1 and x2 respectively, and a is the interpolation ratio of the two encodings, which ranges from 0 to 1.

[0052] Loss 3 is the cross-space consistency loss, which is used to further strengthen the correlation between the feature space and the original space, as shown in Figure 4 . Specifically, by constraining the consistency between each group of feature similarity graphs and image similarity graphs, the goal is to help the interpolated feature find its corresponding position in the original image space, which can to some extent alleviate the mode collapse phenomenon. The loss is as follows:

[0053]

[0054] where, G f is the sample relationship of the ordered feature level, and G x is the sample relationship of the ordered image level, sim(·) represents the similarity calculation, which is usually cosine similarity to measure the similarity relationship between samples. Finally, the correlation between the two sets of normalized graph relationships is calculated using KL divergence, and the total loss is represented as follows:

[0055]

[0056] where, λ is the cross-space consistency loss weight, which is empirically set to 10 3 .

[0057] Step 4: Use the trained image generation network model to oversample the minority classes in the training set to balance the dataset; use the unified benchmark classifier to verify the improvement of the proposed image generation method for imbalanced datasets.

[0058] The Adam optimizer is selected, and the initial learning rate is set to 1e-4, a total of 1000 epochs are performed, and the exponential decay rates β1 and β2 are set to 0.9 and 0.999 respectively. Update the network parameters through backpropagation, and save the model parameters after all rounds of training are completed.

[0059] Track the latest image generation methods, and select several typical methods as comparison algorithms. Among them, ACGAN, BAGAN, VAE and the traditional method SMOTE are included.

[0060] The generated results of different generation methods are randomly sampled for subjective visual angle analysis, including visual fidelity of generated images and generation diversity. The same convolutional neural network is used as a classification network, and classification tests are performed on the original unbalanced data set and the balanced data set augmented by different generation models, respectively, and the test results are systematically analyzed. The evaluation indexes for analysis are four kinds of classification accuracy of the whole test data set, F1-measure index, G-Mean index and classification accuracy of the minority class. Among them, F1-measure and G-Mean index are functions related to the confusion matrix, and the calculation method is as follows:

[0061] F1-measure=2TP / (2TP+FP+FN)

[0062]

[0063] The confusion matrix is shown in Table 1:

[0064] Table 1: Confusion matrix

[0065] Predicted positive class Predicted negative class Actual positive class TP(True Positive) FN(False Negative) Actual negative class FP(False Positive) TN(True Negative)

[0066] The present application is realized under the pytorch framework. The processor used in the experiment is Intel(R) Core(TM) i7-12700KF processor, the memory is 64GB, and the graphics card is NVIDIAGeForce RTX 3060. In the generator model training and image classification, the size of the input image is uniformly set to 32x32, the batch size is set to 128, and the training period is set to 1000. In the experiment, the Adam optimizer is used, and the initial learning rate is set to 1e-4.

[0067] Subjective visual analysis: the image generation model of the present application and ACGAN, BAGAN, VAE, SMOTE method are compared by visual analysis, and the generated results are shown in Figure 5 The present application evaluates the generation effect in the following three parts.

[0068] Distribution fidelity: distribution fidelity is represented as checking whether the class information of the generated sample is consistent with the label of the target class. This is the key to providing supplements for minority groups. Overall, ACGAN tends to generate some images of other classes instead of in-class images. Figure 5 The second column of shows this situation. BAGAN improves this situation. However, there are also some scattered outliers, for example, it generates some samples of other classes for SVHN. In contrast, VAE and the model of the present application show better situations, in which the generated samples almost conform to the label of the target class.

[0069] Visual fidelity: GAN-based techniques usually pay more attention to image details and can generate some samples that look realistic. VAE tends to produce relatively blurry generated samples, which is an inevitable phenomenon due to the L2 norm metric. In contrast, the model of the present application can change this situation, although it still uses the same L2-based loss. Intuitively, it does improve the visual fidelity of the generated images.

[0070] Visual diversity: To solve the problem of class imbalance, it should be emphasized that visual diversity is an important guarantee to improve the generalization performance of the classification model. For ACGAN, there is a mode collapse phenomenon, for example, it produces some similar samples "O" on the E-MNIST dataset. For VAE, the generated samples usually present some common appearance and lack of variation. This can be due to the sampling strategy of the normal distribution in the encoding space, which can only provide some very narrow degrees of freedom for the variation of the generated samples. In contrast, the interpolation of the intra-class input in the present application overcomes this problem in the figure proposed.

[0071] Objective classification result analysis: The evaluation indicators used for analysis are the classification accuracy of the whole test dataset, the F1-measure indicator, the G-Mean indicator and the classification accuracy of the minority class, four kinds. Table 2 shows the classification results of the augmented dataset of each model under the same test setting. It can be seen that the present application has more accurate classification results, which further illustrates the superiority of the present application relative to other benchmark models.

[0072] Table 2: Classification performance of different generation methods on handwritten dataset

[0073]

[0074]

[0075] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for generating imbalanced data images with cyclic feature interpolation, characterized in that, Includes the following steps: (1) Manually imbalance the pre-acquired balanced dataset, select some classes as minority classes, and divide them into training set and test set; (2) Construct an image generation network model based on feature interpolation, using a fully convolutional autoencoder network as the backbone network, and reusing the encoder structure to form a feature-level loop; (3) Determine the loss function of the image generation network model and train the image generation network model on the training set; (4) Use the trained image generation network model to oversample and balance the minority classes of the training set; (5) The improvement of the proposed image generation method on imbalanced datasets is verified using a unified benchmark classifier; The loss function of the image generation network model in step (3) is: Optimization is achieved using mean squared error loss and KL divergence loss. The mean squared error loss measures the difference between two samples, while the KL divergence loss measures the difference between two distributions. The total loss consists of three losses: loss The basic reconstruction loss of the autoencoder structure is used to train the basic autoencoder module's ability to reconstruct image data. The goal is to learn the most basic dimensionality reduction mapping function and image generation mapping function. The loss is as follows: Where, x n For the original sample, E is the encoder module, and D is the decoder module. The loss measures the pixel distance between the original image and its reconstructed version. loss Cyclic consistency loss, used for feature interpolation encoding, is employed to measure the recovery effect of interpolated features, deepen the correspondence between the feature space and the original image space, and distinguish the generation effects of different features. in, For artificially constructed interpolated samples, E and D are the encoder module and decoder module, respectively. This loss measures the Euclidean distance between the interpolated features and their recovered versions. loss The cross-space consistency loss is used to further strengthen the correlation between the feature space and the original space, and alleviate the pattern collapse phenomenon. The loss is as follows: Among them, G f G represents the ordered relationship between sample levels at the feature level. x For the ordered image-level sample relationships, sim(·) represents the similarity calculation, usually cosine similarity is used to measure the similarity between samples; finally, KL divergence is used to calculate the correlation between the two sets of normalized graph relationships; the total loss is expressed as follows: Where λ is the weight of cross-space consistency loss.

2. The method for generating unbalanced data images with cyclic feature interpolation according to claim 1, characterized in that, The implementation process of step (1) is as follows: Half of the classes in the dataset are randomly selected as the minority class, and the number of samples in the selected class is downsampled to two specifications: 20 and 150.

3. The method for generating unbalanced data images with cyclic feature interpolation according to claim 1, characterized in that, The first half of the autoencoder network described in step (2) is the encoder network, which uses a 4×4 convolution kernel size and a stride of 4 to perform feature dimensionality reduction. After each convolution operation, batch normalization and Leaky ReLU activation function are applied. The second half is the decoder network, which uses a 4×4 deconvolution module with a stride of 4. After each convolution operation, batch normalization and ReLU activation function are applied. The last activation function is tanh, which restores the compressed high-level features to the original image.

4. The method for generating unbalanced data images with cyclic feature interpolation according to claim 1, characterized in that, The interpolation sample The construction method is as follows: Where f1 and f2 are the encoded versions of the real samples x1 and x2, respectively, and α is the interpolation ratio between the two codes, with a value between 0 and 1.

Images