An implicit semantic data enhancement method

Abstract

The present invention discloses an implicit semantic data enhancement method. In the first stage of the present invention, the original data is used to train a feature extractor and a classifier, and then the classifier is used to construct a knowledge graph, and the class center and covariance matrix of each category are calculated based on the extracted features; in the second stage, the constructed knowledge graph is used to identify similar categories of the tail category, and then the feature transformation direction of the similar category is migrated to each tail sample for feature change. By mining the correlation between categories, the diversity of the tail categories is greatly enriched. Finally, infinite samples can be obtained by sampling each sample with an infinite number of Gaussian distributions along the direction of the migration covariance matrix. By optimizing the upper bound of the infinite sample loss function, a new implicit semantic data enhancement loss based on reasoning is obtained for model training.

Description

Technical Field

[0001] The invention relates to an implicit semantic data enhancement method based on reasoning for solving long-tail classification, and belongs to the technical field of computer software. Background Art

[0002] With the rapid development of data acquisition technology, deep neural networks can achieve excellent performance on large-scale evenly distributed training data. However, in real scenarios, data is usually distributed in a long-tail distribution, that is, most categories belong to the tail category and only have a small number of samples, while a few head categories have most samples. In recent years, image classification for long-tail distributed data is an important task in the image field. Effectively utilizing unbalanced long-tail data and training good classifiers has received increasing attention. Existing methods generally improve the classification accuracy of tail categories by balancing training data and data enhancement techniques. The strategy of balancing training data alleviates the phenomenon that the classification hyperplane is seriously tilted toward the tail category to a certain extent. However, due to the small number of samples in the tail category, it is difficult to improve the intra-class diversity of the tail category using this strategy, which makes the algorithm very easy to overfit on the tail category. Data enhancement methods can improve the diversity of samples. Traditional data enhancement methods obtain new samples by rotating, cropping or linear interpolating samples within the class, but the diversity of these new samples is strictly limited. Therefore, in the face of extremely unbalanced long-tail data, it is a huge challenge to enrich the diversity of tail categories based on limited samples.

[0003] Traditional solutions have the following defects:

[0004] 1. Data balancing method:

[0005] The existing data balancing strategies mainly include resampling and reweighting. The resampling method obtains training data with uniform data distribution by oversampling the tail categories or undersampling the head categories. The reweighting method gives a greater penalty to the tail samples when calculating the loss function while keeping the original data distribution unchanged to increase the importance of the tail samples. Although these two methods ensure that the tail samples can effectively participate in model training, the limited intra-class diversity will cause the model to overfit on the tail categories.

[0006] 2. Data enhancement methods:

[0007] Traditional data augmentation methods include flipping, translation, cropping, etc. The diversity of new samples generated in this way is limited to a single sample. The mixup operation generates new samples by linearly combining two samples. Although this method breaks the limitation of a single sample and can generate new samples that are very different from existing samples, the linear combination of any two samples is likely to generate meaningless new samples. In order to ensure the quality of new samples, the generative adversarial network is introduced. However, the generative adversarial network is difficult to train, and as the number of augmented samples increases, the convergence speed of the model drops sharply. Therefore, how to improve the training speed of the model while ensuring the quality of new samples is still an important issue. Summary of the invention

[0008] In view of the technical problems existing in the prior art, the purpose of the present invention is to design an implicit data enhancement method based on reasoning, which improves the richness of the tail class by simulating human reasoning and imagination. The method is divided into two stages. In the first stage, the basic feature extractor and classifier are first trained on the original data distribution, and then the knowledge graph is constructed using the classifier, and the class center and covariance matrix of each category are calculated based on the extracted features. Since the covariance matrix represents the feature change direction of each category, the similar categories of the tail category are identified in the second stage using the constructed knowledge graph, and then the feature transformation direction of the similar category is migrated to each tail sample for feature change. By mining the correlation between categories, the diversity of the tail category is greatly enriched. Finally, infinite samples can be obtained by sampling each sample along the migrated covariance matrix for an infinite number of times in the direction of change, and a new implicit semantic data enhancement loss based on reasoning is obtained for model training by optimizing the upper bound of the infinite sample loss function.

[0009] This method transfers the semantic feature transformation direction from other categories to the tail category to enhance the diversity of the tail category samples. The present invention constructs a learnable knowledge graph, so that the method can adaptively select similar categories for reasonable migration for different samples, and on the other hand, it also expands the scope of application of the method. The method shows good performance on long-tail datasets (such as CIFAR-100-LT, ImageNet-LT and iNaturelist 2018).

[0010] The technical solution of the present invention is:

[0011] An implicit semantic data enhancement method, the steps of which include:

[0012] 1) For a long-tailed distributed data set, a neural network model is trained using samples in the data set to obtain a feature extractor F and a classifier H;

[0013] 2) The feature extractor F extracts features from the samples in the data set, and uses the extracted features to calculate the covariance matrix and class mean of each sample category, where the covariance matrix represents the semantic transformation direction of all features of each category, and the class mean represents the feature vector of each category;

[0014] 3) Using the classifier H to classify the samples in the data set, and constructing a knowledge graph based on the classification results to store the similarity between any two categories;

[0015] 4) extracting features of the tail samples in the data set using a feature extractor F to obtain semantic features of the tail samples; and enhancing the tail samples using semantic transformations in a covariance matrix of the tail sample category;

[0016] 5) Using the covariance matrix of similar categories defined in the knowledge graph to expand the semantic change direction of the enhanced tail sample category, to obtain an updated covariance matrix;

[0017] 6) using the class mean of the category similar to the tail sample category to supplement the semantic features of the tail sample obtained in step 4) to obtain enhanced sample features;

[0018] 7) Using the enhanced sample features obtained in step 6) and the updated covariance matrix of the tail samples obtained in step 5), a Gaussian distribution is established for the corresponding tail samples, and samples with infinite semantic conversion are sampled from the Gaussian distribution to achieve infinite enhancement of the tail samples.

[0019] Furthermore, the method for calculating the covariance matrix of each sample category is as follows: first calculate the class center of each sample category Among them, N c is the cth category l c The number of samples in f i Indicates l c The characteristics of the i-th sample in Calculate the category l c The covariance matrix of ; where D represents the feature dimension.

[0020] Furthermore, the method for constructing the knowledge graph is: using the classifier H to classify the samples in the data set, and obtaining the classification prediction result H (f i ), then according to H(f i ) and the true labels of the corresponding samples to build a knowledge graph<V,ε> ; Where V is the category node, each element ε in the matrix ε ij Indicates category l i is predicted to be l j The sample of category li The proportion of all samples.

[0021] Further, in step 6), using Get the enhanced sample features; among them, represents the class center of the cth class, ε c，i represents the similarity between the cth category and the ith category, μ i is the class center of the ith category.

[0022] Furthermore, in step 6), the feature f of the i-th sample is i Along from Random sampling is performed to generate sample features of various semantic transformations of the i-th sample, and the sample features after enhancement of the i-th sample are obtained; wherein α and β are the set strength coefficients.

[0023] Furthermore, the features of each tail category sample are enhanced M times, and each feature after feature enhancement generates a sample. Then the data set corresponding to each tail category sample is Among them, y i is the category label of the i-th tail sample, is the kth enhanced sample corresponding to the i-th tail sample, k=1~M.

[0024] A training method for a neural network model, characterized in that the data set and loss function obtained by the above method are used Train a neural network model; where Represents the neural network model for the i-th sample x in the data set i The j-th output result of c is the total number of samples in the cth category, and the weight γ is a hyperparameter, is the i-th sample in the y-th i The output result derived after the class makes an inference transformation, is the corresponding weight vector of the jth category in the fully connected layer of the classifier, For the yth i The corresponding weight vector of each category in the fully connected layer of the classifier, is the inference class center obtained by inference for the jth category, The new covariance is obtained by inference for the i-th sample in the j-th category.

[0025] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the above method.

[0026] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0027] The advantages of the present invention are as follows:

[0028] This method increases the intra-class diversity of tail categories by simulating human reasoning and imagination. Specifically, without relying on any external knowledge, the present invention constructs an adaptive knowledge graph to fully explore the correlation between different categories as prior knowledge, and greatly improves the intra-class richness of tail categories by transferring the semantic transformation direction of similar categories. In addition, the present invention is the first to discover that the migration method of transferring the semantic transformation direction of similar categories to the tail category is effective for enriching the intra-class diversity of the tail category, and the designed loss function not only fully expands each category but also reduces the amount of calculation because there is no need to explicitly generate additional samples. Experiments show that the classification performance of the present invention on existing long-tail datasets has been significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a network structure diagram of the present invention.

[0030] Figure 2 This is a sensitivity analysis chart;

[0031] (a) is the sensitivity analysis diagram of hyperparameters α and β on CIFAR-LT-100 when λ = 200 and β = 1.

[0032] (b) is the sensitivity analysis diagram of hyperparameters α and β on CIFAR-LT-100 when λ = 200 and α = 0.5.

[0033] (c) is the sensitivity analysis diagram of hyperparameters α and β on CIFAR-LT-100 when λ = 100 and β = 0.75.

[0034] (d) Sensitivity analysis of hyperparameters α and β on CIFAR-LT-100 when λ = 100 and α = 0.5.

[0035] Figure 3 The effect of the number of head categories on CIFAR-LT-100 when λ is different;

[0036] (a) shows the effect of the number of head categories when λ=200, and (b) shows the effect of the number of head categories when λ=100. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0038] like Figure 1 As shown, the reasoning-based implicit semantic data augmentation method (RISDA) proposed in the present invention is divided into two stages. In the first stage, all samples are used to train feature extractors and classifiers. Subsequently, the trained feature extractor is used to extract semantic features from the samples in the data set, and the covariance matrix and class mean of each category are calculated using the extracted features, wherein the covariance matrix represents the semantic transformation direction of all features of each category, and the class mean represents the feature vector of each category. Then, the trained classifier is used to classify the samples of the training set, and a knowledge graph is constructed based on the classification results of all training data to store the similarity of any two categories. In the second stage, the tail sample is first passed through the feature extractor to obtain semantic features. Then, the present invention will enhance the tail sample using the semantic transformation in the covariance matrix of the tail category (reference Wang Y, PanX, Song S, et al. Implicit semantic data augmentation for deep networks [J]. Advances in Neural Information Processing Systems, 2019, 32: 12635-12644). However, the transformation direction in the tail category covariance matrix is ​​limited, so the present invention uses the covariance matrix of similar categories defined in the knowledge graph to expand the category change direction of the enhanced tail sample. At the same time, the tail sample may lose some important features. In order to ensure that it contains the features required for all change directions, the present invention uses the class mean of the category similar to the tail category to supplement the features extracted in the first stage of the tail sample to obtain the enhanced sample features. Through the above processing, the present invention can obtain the enhanced features of each tail sample in the original data set and the corresponding expanded and updated covariance matrix, and establish a Gaussian distribution with the feature as the mean and the enhanced feature change direction as the covariance for each tail sample. Samples with infinite semantic conversion are sampled from the distribution to achieve infinite enhancement of the tail sample. Finally, in order to realize the loss calculation under infinite samples, the present invention derives the upper bound of the cross entropy loss under infinite data conditions and trains the network.

[0039] (I) Implicit semantic data enhancement:

[0040] Given a long-tailed distribution data set , where y i ∈L={l 1 , l 2 , ..., l C}, C is the total number of categories. Since training the network on the original distribution of the dataset can learn feature representations with stronger generalization ability, we use the original training dataset in the first stage to train a neural network model by minimizing the cross entropy loss to obtain the feature extractor F and classifier H.

[0041] First, for each sample x in the training set i , we use f i =F(x i ) to extract its features. Then, the semantic transformation direction of each category is counted. The semantic transformation direction of each category can be represented by the covariance matrix of all sample features of the category. In order to obtain the covariance matrix, we first need to calculate the class center of each class, as shown in formula (1):

[0042]

[0043] Where c∈[1,C],N c Is class l c The number of samples in the class, f i Indicates l c The features of the i-th sample in . Then, we use formula (2) to calculate each element in the covariance matrix:

[0044]

[0045] Where D represents the feature dimension, f i m and f i n Represent the values ​​of the i-th sample feature in the m-th dimension and the n-th dimension respectively. and Represent the values ​​of the mth and nth dimensions of the class center of the cth class respectively. Using formula (2), we can obtain the covariance matrix of all classes ∑={∑ 1 ,∑ 2 ,…,∑ C Then the covariance matrix of the tail category is used to expand the tail samples, but the covariance matrix of the tail category has limited change directions. To alleviate this problem, we propose a new reasoning module that allows the tail category to increase the diversity of samples by migrating the feature change directions of other similar categories.

[0046] (II) Implicit Data Augmentation Based on Reasoning:

[0047] In this section, we will introduce in detail the process of transferring knowledge to increase the diversity of tail samples. By constructing a knowledge graph, we can infer the tail class covariance matrix with rich variations.

[0048] Knowledge graph construction: Current feature enhancement methods based on transfer learning usually adopt the transfer from head category to tail category, but our method attempts to transfer knowledge from similar categories more reasonably. To achieve this, we use classifier H to predict H(f i ), then the predicted value of the classifier and the true label of the sample construct a knowledge graph<V,ε> , where V is a category node, ε∈R C×C , each element of the matrix ε ij Indicates category l i is predicted to be l j The sample is in l i The proportion of all samples is calculated as follows:

[0049]

[0050] Among them, function 1() is an indicator function, which means that when the condition in the brackets is met, the value is 1, otherwise the value is 0.

[0051] Reasoning-based semantic transformation: We propose to optimize the covariance matrix of the tail category by the covariance matrix of other similar categories. Because similar categories have similar appearance variations and living environments, for example, because "kingfisher" and "parrot" are very similar, when we see a flying "parrot", we can imagine a flying "kingfisher", even if we have never seen a flying "kingfisher". Therefore, we can infer better intra-class covariance of the tail category by transferring similar classes:

[0052]

[0053] Generally, the network is trained by batch iterative optimization. Ideally, we should use all training samples in each iterative training to calculate the covariance matrix of each category using formula (2), but this operation takes a long time. Therefore, we use an online batch update strategy, that is, use each batch of training samples to calculate a new covariance matrix, and update the new covariance matrix weighted with the historically stored covariance matrix. Using the inference-based covariance matrix, we can transform a sample with richer semantic changes. However, the sample may miss some features that are necessary for the inference-based transformation direction. For example, we can only change the posture of its mouth when the mouth is detected in "kingfisher". Therefore, in order to ensure that the tail sample contains all the features to be transformed, we use the features of similar categories to obtain the inference features to supplement them. The inference feature calculation formula is as follows:

[0054]

[0055] in represents the class center obtained by reasoning, μ i represents the class center of the i-th category, ε c，i represents the similarity relationship between the cth category and the ith category. Then, through reasoning, each tail sample obtains a new covariance matrix and enhanced features through reasoning of similar categories. i You can follow from Random sampling generates sample features of various semantic transformations of the i-th sample to enhance the i-th sample, where N() is a Gaussian distribution function, and formula (6) is used to generate enhanced sample features during training

[0056]

[0057] Where α and β are coefficients that control the strength of semantic data enhancement based on reasoning. In the experiment, α and β are both decayed by t / T, where t and T represent the current number of iterations and the total number of iterations, respectively.

[0058] (III) Optimization:

[0059] Based on our data augmentation method, a simple way to train the classifier is to generate samples for the tail class using formula (6) until the tail class has a comparable or equal number of samples to the head class. Assuming that each sample in the tail class is augmented M times, we can obtain a new dataset in The kth augmented sample generated. Then, we can use the traditional cross entropy loss to train the classifier:

[0060]

[0061] Among them, TC represents the tail category set, HC represents the head category set, W = [w 1 , w 2 , ..., w C ] T and b = [b 1 , b 2 , ..., b C ] T Corresponding to the weight matrix and bias vector of the last fully connected layer respectively.

[0062] However, when the tail category is sampled M times, the semantic direction of the sampling is limited and unstable. The ideal situation is to generate as many samples as possible (i.e., set M as large as possible), but increasing M will increase the computational effort. To simplify the computation while generating more data, we propose to implicitly generate infinite sampling data. When M→∞, we can derive a simple upper bound loss based on the law of large numbers. Specifically, when M→∞, the loss of the tail category can be defined as:

[0063]

[0064] However, the above loss is difficult to calculate accurately. With the help of Jensen's inequality E[logX]≤logE[X], we can derive its upper bound as follows:

[0065]

[0066] because We can get is also a Gaussian variable, subject to: in After that, according to the moment generating function:

[0067]

[0068] Finally, we get the optimized upper bound function as follows:

[0069]

[0070] in is the i-th sample in the y-th i The class makes inferences and then derives the output results, Represents x i The jth output result of is the corresponding weight vector of the jth category in the fully connected layer of the classifier, For the yth i The corresponding weight vector of each category in the fully connected layer of the classifier, is the inference class center obtained by inference for the jth category, The new covariance is obtained by inference for the i-th sample in the j-th category, and α is a hyperparameter.

[0071] But when we generate infinite samples for the tail class, the data distribution is still unbalanced because the head class becomes the "tail class". Therefore, similar to the tail class, we also perform infinite enhancement on the head class. However, since the enhancement result depends on the training samples and the head class has most samples, the head class still dominates the loss. To solve this problem, we introduce a reweighting strategy, and the weight is defined as Where γ is a hyperparameter. Through the reweighting strategy, the loss function obtained is as follows:

[0072]

[0073] Experiments show that our reasoning module can generate more diverse samples for tail categories by transferring features and transformation directions from other classes.

[0074] Datasets: We conduct extensive experiments on large-scale long-tail datasets to evaluate the effectiveness of the reasoning-based implicit data augmentation method (RISDA). Our model is trained and tested on the datasets CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018. (1) CIFAR-100 is a balanced dataset containing 50,000 training images and 10,000 validation images. In our experiments, the imbalance factor λ, i.e., the ratio of the number of samples of the most frequent and least frequent classes, is used to generate different training sets for CIFAR-100-LT. Five training sets can be obtained by setting λ to different values ​​in {200, 100, 50, 20, 10}. (2) The ImageNet-LT dataset is constructed by removing some training samples from the ImageNet dataset. It has 115,846 training images, 20,000 validation images, and 50,000 test images. The class with the largest number of samples in the training set contains 1,280 samples, and the class with the smallest number of samples has only 5 samples. (3) iNaturalist 2018 is a large-scale species dataset with an imbalance factor of 500 and a severely unbalanced sample distribution. It contains 437,513 training images and 24,426 validation images from 8,142 classes.

[0075] Evaluation indicators: We use the standard indicator commonly used in classification: error rate as the evaluation indicator, which represents the ratio of wrong classification of images in all data sets. The smaller the value, the better.

[0076] Experimental Results: Under the same settings and data partitioning, we compare the proposed RISDA method with some of the best existing methods to verify its effectiveness. Tables 1, 2, and 3 show the experimental results on the datasets CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018, respectively.

[0077] As shown in Table 1, we can draw the following conclusions: First, our method can achieve the best performance compared with the best existing methods. Second, the more unbalanced the dataset is, the more the error rate of our method is reduced. For example, when λ = 200 and 100, the error rate is reduced by 4.82% and 3.29% respectively compared with MetaSAug. When λ = 20, the error rate is reduced by 0.82%. When λ = 10, the error rate is only reduced by 0.65%. This is because the more unbalanced, the worse the covariance matrix learned by the tail category, and the changes transferred from similar categories can better improve the diversity of the tail category feature transformation. Therefore, it can be concluded that our method is more suitable for processing extremely unbalanced data.

[0078] From the experimental results in Table 2, we can see that the error rate of the RISDA method is reduced to 49.31%. Compared with the MetaSAug method, the error rate is reduced by 3.3%. This further shows that the diversity contained in the tail category is extremely limited, and the diversity of the tail category can be greatly enriched by borrowing the semantic features of other classes to change the direction. Similarly, as shown in Table 3, our method RISDA has a lower error rate of 30.85% on iNaturalist 2018, achieving good results.

[0079] In summary, RISDA has obvious advantages over other best methods, indicating that our proposed inference-based implicit enhancement is more suitable for processing extremely imbalanced datasets, and our inference module can greatly improve the feature diversity of tail categories.

[0080] Model Analysis: We conduct detailed model analysis experiments on the CIFAR-100-LT dataset to study the impact of different modules on the performance of RISDA and discuss the sensitivity of parameters in RISDA. In addition, we visualize the implicitly generated samples.

[0081] (1) Ablation Experiments: Our RISDA consists of two parts: the reasoning module and the re-weighting strategy. To further explore the importance of different modules, we designed ablation experiments of RISDA (w / o re-weighting) and RISDA (w / o reasoning). Specifically, RISDA (w / o re-weighting) and RISDA (w / o reasoning) respectively denote the removal of the re-weighting module and the reasoning module from the complete RISDA. The results shown in Table 4 show that: 1) Re-weighting is very important in our RISDA. Through the re-weighting operation, the error rate of RISDA is reduced by 18.24% and 7.08% when λ = 200 and λ = 100, respectively. Although our goal is to generate an infinite number of samples through Gaussian distribution, enhancement is achieved in each sample. Therefore, re-weighting plays an important role in achieving fairness in training. 2) The reasoning module can effectively transfer the feature change direction of the head category to the tail category, thereby reducing the classification error rate. When λ=200, the inference module reduces the error rate by 1.73%, and when λ=10, the inference module reduces the error rate by 0.81%, which once again proves that the inference module is more suitable for processing data with extremely unbalanced sample distribution.

[0082] (2) α and β parameter sensitivity analysis: α and β are coefficients that control the strength of semantic data augmentation based on reasoning. We further explore the impact of both hyperparameters from {0.25, 0.5, 0.75, 1.0, 1.25, 1.5}. On the CIFAR-100-LT dataset: 1) When α = 0.5, the model can achieve the best performance under λ = 100 and 200. When α increases, the performance decreases. It indicates that the characteristics of other classes can help supplement the characteristics of tail samples. 2) When λ = 200, the model achieves the best results when β = 1.0, and when λ = 100, the model performs best when β = 0.75. We can conclude that the more unbalanced the dataset is, the better the β value is.

[0083] (3) Analysis of the number of head categories: The head category itself has rich intra-class diversity. Transferring feature transformations from other categories may damage their original intra-class diversity, thereby reducing the overall performance. Therefore, the number of head categories will also affect the performance of our model. We select the number of head categories from {10, 20, 30, 40} to explore its impact. The experimental results are shown in Figure 2. Figure 3As shown in the figure, the left and right bar graphs show the results of different values ​​of the number of head categories at λ = 200 and λ = 100, respectively. It can be seen that when the top 20 categories with the largest number of samples are selected as the head categories, the performance is the best. When the number of head categories increases, the model performance decreases due to the lack of diversity in some head categories. However, when the number of head categories decreases, the feature representation of some rich categories will be damaged by the migration results of other categories, thereby reducing the performance.

[0084] (4) Visualization Analysis: To intuitively show that our RISDA can generate more diverse samples, we implemented visualization of some implicitly generated samples. For example, “car” changes the background, perspective, and color, and “sandpiper” changes the posture, background, and perspective. These changes are not present in the training data, and we believe they come from similar categories. Therefore, it can be concluded that our RISDA can obtain meaningful transformations from similar categories.

[0085] Table 1 Error rate of ResNet-32 on CIFAR-LT-100 under different imbalance factors

[0086] Imbalance Factor 200 100 50 20 10 Cross-entropy traning 65.30 61.54 55.98 48.94 44.27 Mixup - 60.46 55.01 - 41.98 LqCy 67.00 61.10 56.83 49.25 47.88 Class-balanced cross-entropy loss 61.34 58.50 53.78 47.70 42.43 Meta-weight net 63.38 58.39 54.34 46.96 41.09 Meta-class-weight with CE loss 60.69 56.65 51.47 44.38 40.42 BBN - 57.44 52.98 - 40.88 MetaSAug with cross-entropy loss 60.06 53.13 48.10 42.15 38.27 Hybrid-PSC - 55.03 51.07 - 37.63 Bag of Tricks - 52.17 48.31 - - RISDA 55.24 49.84 46.16 41.33 37.62

[0087] Table 2 Performance comparison of different methods on the ImageNet-LT dataset

[0088] method Error rate (%) CE training 61.12 Class-balanced CE 59.15 OLTR 59.64 LDAM 58.14 LDAM-DRW 54.26 Meta-class-weight 55.08 MetaSAug 52.61 Bag of Tricks 56.87 RISDA 49.31

[0089] Table 3 Performance comparison of different methods on the iNaturalist 2018 dataset

[0090]

[0091]

[0092] Table 4 Results of ablation experiments on CIFAR-LT-100 under different imbalance factors

[0093] Imbalance Factor 200 100 50 20 10 w / o re-weighting 36.82 42.76 46.98 54.4 58.6 w / o reasoning 56.97 51.09 47.6 42.6 38.43 RISDA 55.24 49.84 46.16 41.33 37.62

[0094] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A method for implicit semantic data enhancement, comprising the steps of: 1) For a data set with a long tail distribution, a neural network model is trained using samples in the data set to obtain a feature extractor F and a classifier H; the data set includes training images, verification images, and test images; 2) The feature extractor F extracts features from the samples in the data set, and uses the extracted features to calculate the covariance matrix and class mean of each sample category, where the covariance matrix represents the semantic transformation direction of all features of each category, and the class mean represents the feature vector of each category; 3) Using the classifier H to classify the samples in the data set, and constructing a knowledge graph based on the classification results to store the similarity between any two categories; the method of constructing the knowledge graph is: using the classifier H to classify the samples in the data set, and obtaining the classification prediction result H (f i ), then according to H(f i ) and the true labels of the corresponding samples to build a knowledge graph<V,ε> ; Where V is the category node, each element ε in the matrix ε ij Indicates category l i is predicted to be l j The sample of category l i The proportion of all samples; 4) Using the feature extractor F to extract features from the tail samples in the data set to obtain semantic features of the tail samples; The tail samples are enhanced by using the semantic transformation in the covariance matrix of the tail sample category; 5) Using the covariance matrix of similar categories defined in the knowledge graph to expand the semantic change direction of the enhanced tail sample category, to obtain an updated covariance matrix; 6) using the class mean of the category similar to the tail sample category to supplement the semantic features of the tail sample obtained in step 4) to obtain enhanced sample features; 7) Using the enhanced sample features obtained in step 6) and the updated covariance matrix of the tail samples obtained in step 5), a Gaussian distribution is established for the corresponding tail samples, and samples with infinite semantic transformation are sampled from the Gaussian distribution to achieve infinite enhancement of the tail samples.

2. The method according to claim 1, characterized in that The method for calculating the covariance matrix of each sample category is: first calculate the class center of each sample category Among them, N c is the cth category l c The number of samples in f i Indicates l c The characteristics of the i-th sample in Calculate the category l c The covariance matrix of ; where D represents the feature dimension.

3. The method according to claim 1, characterized in that In step 6), use Get the enhanced sample features; among them, represents the class center of the cth class, ε c，i represents the similarity between the cth category and the ith category, μ i is the class center of the ith category.

4. The method according to claim 3, characterized in that In step 6), the feature f of the i-th sample is i Along from Random sampling is performed to generate sample features of various semantic transformations of the i-th sample, and the sample features after enhancement of the i-th sample are obtained; wherein α and β are the set strength coefficients.

5. The method according to claim 4, characterized in that The features of each tail category sample are enhanced M times, and each feature after feature enhancement generates a sample. Then the data set corresponding to each tail category sample is Among them, y i is the category label of the i-th tail sample, is the kth enhanced sample corresponding to the i-th tail sample, k=1~M.

6. A training method for a neural network model, characterized in that: The data set and loss function obtained by the method described in claim 1 Train a neural network model; where Represents the neural network model for the i-th sample x in the data set i The j-th output result of c is the total number of samples in the cth category, and the weight γ is a hyperparameter, is the i-th sample in the y-th i The output result derived after the class makes an inference transformation, is the corresponding weight vector of the jth category in the fully connected layer of the classifier, For the yth i The corresponding weight vector of each category in the fully connected layer of the classifier, is the inference class center obtained by inference for the jth category, The new covariance is obtained by inference for the i-th sample in the j-th category.

7. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images