Two-path data augmentation method based on out-of-distribution sample detection

Abstract

The application discloses a two-path data augmentation method based on out-of-distribution sample detection, and relates to the field of data augmentation.The application solves the problem that the existing data augmentation method cannot simultaneously increase sample diversity and reduce out-of-distribution data.The original image array is augmented by two different augmentation intensities, and the two-path data augmentation is realized;in the meanwhile, the augmented images from the basic augmentation branch and the strong augmentation branch are intelligently fused by relying on a data mixing strategy;in the fusion process, the samples in the strong augmentation branch image array are screened by a threshold tau, the out-of-distribution samples are screened out, and the out-of-distribution samples are replaced by the in-distribution samples in the basic augmentation branch image array with the same index to generate a mixed augmented image array, so that the two-path data augmentation of the original image array is realized.The application is mainly used for enhancing the diversity of the samples used in neural network training, so as to improve the performance of the neural network.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data augmentation. BACKGROUND

[0002] Deep neural networks have played an important role in computer vision tasks such as image classification, object detection and semantic segmentation. The training of deep neural networks usually relies on data augmentation to alleviate the problem of overfitting. Data augmentation increases the number and diversity of data by randomly transforming the original data. Generally, the stronger the data augmentation, the better. However, excessive strong data augmentation sometimes destroys the semantic context of the original data, resulting in poor performance of the neural network. These samples without practical significance can be considered as out-of-distribution samples.

[0003] The existing data augmentation methods mainly have the following two problems: first, the diversity of the samples after data augmentation is good, but it is easy to produce out-of-distribution data, and second, the samples after data augmentation are mostly in-distribution data, but the diversity of the samples is not good. Most of the existing data augmentation methods tend to choose the second conservative scheme, which uses a weak data augmentation strength to avoid producing data that exceeds the distribution range (out-of-distribution samples). Although this method can avoid producing out-of-distribution samples (i.e. abnormal samples), it cannot guarantee that the data is fully augmented. Therefore, the problem of simultaneously considering sample diversity and reducing out-of-distribution data in the existing data augmentation method needs to be solved. SUMMARY

[0004] The purpose of the present application is to solve the problem that the existing data augmentation method cannot simultaneously consider increasing sample diversity and reducing out-of-distribution data in the data augmentation problem. The present application provides a dual-path data augmentation method based on out-of-distribution sample detection.

[0005] The dual-path data augmentation method based on out-of-distribution sample detection is used to perform data augmentation on each sample in the original image array to obtain a mixed augmented image array, each sample being an image, and specifically comprising the following steps:

[0006] S1, after data augmentation of the original image array with two different augmentation strengths, a basic augmented branch image array and a strong augmented branch image array are obtained;

[0007] S2, the confidence score of each sample in the basic augmented branch image array is detected by a first confidence detection network At the same time, the confidence score of each sample in the strong augmented branch image array is detected by a second confidence detection network i = 1, 2, … N; N is the total number of samples;

[0008] a confidence score of an i-th sample in the base augmented branch image array;

[0009] a confidence score of an i-th sample in the strong augmented branch image array;

[0010] S3, according to the confidence score of each sample in the base augmented branch image array obtain a first confidence score set S of the base augmented branch image array base

[0011] S4, the first confidence score set Sb ase is considered to conform to a normal distribution; according to S base calculate a threshold τ for screening samples;

[0012] S5, judge whether the distribution detection score of each sample is less than the threshold τ, and the result is yes, then determine that the sample is an out-of-distribution sample, and the result is no, then determine that the sample is an in-distribution sample;

[0013] S6, according to the index corresponding to each out-of-distribution sample in the strong augmented branch image array, replace each out-of-distribution sample in the strong augmented branch image array with the corresponding sample in the base augmented branch image array under the same index to obtain a mixed augmented image array, thereby completing the data augmentation of each sample in the original image array.

[0014] Preferably, in step S2,

[0015] wherein f θ (·) is the output of the image feature extraction network, is an i-th sample in the base augmented branch image array, is a j-th sample in the base augmented branch image array, and T is a hyperparameter for controlling the smoothness of the output distribution, and exp(·) is an exponential function.

[0016] Preferably, in step S2,

[0017] wherein f θ (·) is the output of the image feature extraction network, is an i-th sample in the strong augmented branch image array, is a j-th sample in the strong augmented branch image array, and T is a hyperparameter for controlling the smoothness of the output distribution, and exp(·) is an exponential function.

[0018] Preferably, in step S4, the implementation manner of calculating the threshold τ includes:

[0019] ​S41, calculate a first confidence score set S base of the mean μ(S base ) and variance σ(S base ) of the first confidence score set S base ;

[0020] S42, obtain a threshold τ according to the mean μ(S base ) and variance σ(S base ).

[0021] Preferably, τ = μ(S base )-λ*σ(S base );

[0022] Wherein, λ is the hyperparameter of 3σ principle.

[0023] Preferably, in step S1, the implementation manner of data augmentation on each sample in the original image array by using the first augmentation intensity is:

[0024]

[0025] Wherein, is the i-th sample in the basic augmentation branch image array, x i is the i-th sample in the original image array, Φ(·) is an automatic augmentation operation, K base is the first augmentation intensity, and the K base is a set composed of different transformation operation parameters.

[0026] Preferably,

[0027] is the m-th transformation operation parameter, m = 1, 2, …, P.

[0028] Preferably, in step S1, the implementation manner of data augmentation on each sample in the original image array by using the second augmentation intensity is:

[0029]

[0030] Wherein, is the i-th sample in the strong augmentation branch image array, x i is the i-th sample in the original image array, Φ(·) is an automatic augmentation operation, K heavy is the second augmentation intensity, and the K heavy is a set composed of different transformation operation parameters.

[0031] Preferably, in step S1,

[0032]

[0033] For the n-th transformation operation parameter, n = 1, 2, …, q.

[0034] Preferably, And Both consist of three hyperparameters, and the three hyperparameters are transformation type, probability of applying transformation and amplitude of transformation respectively.

[0035] Principle analysis:

[0036] The original image array is data augmented by two different augmentation intensities, respectively. The relatively weak augmentation intensity is used to data augment each image in the original image array to obtain a basic augmented branch image array. The relatively strong augmentation intensity is used to data augment each image in the original image array to obtain a strong augmented branch image array. Then, the threshold τ for screening samples in the strong augmented branch image array is determined according to the confidence scores of each sample in the basic augmented branch image array. The samples outside the distribution are screened out from the strong augmented branch image array by the threshold τ, and are replaced by samples inside the distribution to generate a hybrid augmented image array. Each image in the hybrid augmented image array is a sample inside the distribution, so that data augmentation of the original image array is realized, and the problems of sample diversity and reduction of out-of-distribution data are considered.

[0037] The beneficial effects brought by the application are:

[0038] The application designs a double-path data augmentation, relies on a data mixing strategy, and intelligently fuses augmented data from the basic augmented branch and the strong augmented branch. The application finds a compromise solution to enhance the strength of data augmentation while avoiding out-of-distribution data. The application can consider sample diversity and reduce out-of-distribution data at the same time.

[0039] The application can maintain data distribution as much as possible within a reasonable time and computing cost.

[0040] Depending on each sample image in the hybrid augmented image array of the application, the feature extraction and classification ability of the deep neural network can be significantly enhanced during network training. Specifically, the recognition accuracy on the ImageNet / CIFAR-10 / CIFAR-100 data set is significantly improved, and the robustness and domain generalization ability of the neural network are significantly enhanced.

[0041] The application relates to a method for reducing neural network overfitting and improving its generalization ability through data augmentation, and realizes optimization in a specific application scenario.

[0042] Hybridization process: Out-of-distribution sample data are downgraded to basic augmentation in a strongly augmented branch image array. Using the above method, dual-path data augmentation can adequately augment all data while preserving as much in-distribution data as possible. Attached Figure Description

[0043] Figure 1 This is a conceptual diagram for dual-path data augmentation; among which, Figure 1 (a) A schematic diagram of augmenting data using basic augmentation methods. Figure 1 (b) A schematic diagram illustrating the use of strong data augmentation to augment data. Figure 1 (c) A schematic diagram showing that all data are fully augmented and all augmented data are within the distribution area;

[0044] Figure 2 This is a schematic diagram illustrating the principle of the dual-path data augmentation method based on out-of-distribution sample detection described in this invention.

[0045] Figure 3 This is a visualization of the data augmentation results achieved by applying the dual-path data augmentation method based on out-of-distribution sample detection described in this invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0048] To address the issue that existing data augmentation methods cannot simultaneously balance sample diversity and reduce out-of-distribution data, the motivation for dual-path data augmentation is as follows: Figure 1 As shown, Figure 1 A conceptual diagram for two-way data augmentation; circles, squares, and crosses represent different data points. Current automatic augmentation methods using conservative parameters can keep the data within the distribution, but this is insufficient for circles and squares, such as... Figure 1 As shown in (a). Strong data augmentation is used, but only circles and squares are needed, while crosses will reach out-of-distribution sample regions, such as... Figure 1 As shown in (b), we need to modify the out-of-distribution sample data (crosses) to properly transform it into ID data (i.e., non-outlier data), as follows: Figure 1(c) are shown. All data have been well augmented and in the in-distribution region. This implies that we propose a two-path data augmentation method by detecting out-of-distribution samples and replacing them with their base augmented versions to maintain the distribution of augmented results as much as possible while preserving in-distribution data so that all data can be well augmented.

[0049] DETAILED DESCRIPTION Figure 2 To illustrate the present embodiment, the two-path data augmentation method based on out-of-distribution sample detection is described. The method is used to perform data augmentation on each sample in the original image array to obtain a mixed augmented image array, each sample being an image, and specifically includes the following steps:

[0050] S1, after data augmentation of the original image array with two different augmentation intensities, a base augmented branch image array and a strong augmented branch image array are obtained;

[0051] S2, the confidence score of each sample in the base augmented branch image array is detected by a first confidence detection network Meanwhile, the confidence score of each sample in the strong augmented branch image array is detected by a second confidence detection network i = 1, 2, …, N; N is the number of sample species;

[0052] is the confidence score of the i-th sample in the base augmented branch image array;

[0053] is the confidence score of the i-th sample in the strong augmented branch image array;

[0054] S3, according to the confidence score of each sample in the base augmented branch image array a first confidence score set S base of the base augmented branch image array is obtained;

[0055] S4, the first confidence score set S base is considered to conform to a normal distribution; according to S base , a threshold τ for screening samples is calculated; τ = μ(S base ) - λ * σ(S base ); where λ is a hyperparameter of the 3σ principle, μ(S base ) is the mean of the first confidence score set S base , and σ(S base ) is the variance of the first confidence score set S base ;

[0056] S5, the distribution detection score of each sample is determined whether less than a threshold value τ, the result is yes, it is determined that the sample is out-of-distribution sample, the result is no, it is determined that the sample is in-distribution sample;

[0057] S6, according to the index corresponding to each out-of-distribution sample in the strong augmented branch image array, each out-of-distribution sample in the strong augmented branch image array is replaced into the corresponding sample in the basic augmented branch image array with the same index of the out-of-distribution sample, a mixed augmented image array is obtained, and the data augmentation of each sample in the original image array is completed.

[0058] Figure 2 In the S heavy is a set composed of the second confidence scores of all samples of the strong augmented branch image array; S base is considered to conform to a normal distribution;

[0059] The embodiment performs two-way data augmentation while relying on a data mixing strategy to intelligently integrate augmented data from the basic augmented branch and the strong augmented branch, finds a compromise solution to increase the strength of augmented data augmentation while avoiding the generation of out-of-distribution data; and makes the method of the present application take into account both sample diversity and reduced out-of-distribution data.

[0060] The mixing process: downgrading out-of-distribution sample data in the strong augmented branch image array to the basic augmented branch. Through the above method, two-way data augmentation can fully augment all data and retain as much in-distribution data as possible.

[0061] As an example, the data augmentation method for the original image array in step S1 can be implemented by AutoAugment, RandAugment or TrivialAugment in the existing automatic augmentation method.

[0062] Further, in step S2,

[0063] wherein f θ is the output of the image feature extraction network, is the i-th sample in the basic augmented branch image array, is the j-th sample in the basic augmented branch image array, T is a hyperparameter for controlling the smoothness of the output distribution, and exp(·) is an exponential function.

[0064] Further, in step S2,

[0065] wherein f θ is the output of the image feature extraction network, is the i-th sample in the strong augmented branch image array, is the jth sample in the strong augmented branch image array, T is a hyperparameter that controls the degree of smoothing of the output distribution, and exp(·) is the exponential function.

[0066] Further, in step S1, the implementation manner of data augmentation of each sample in the original image array by using the first augmentation intensity is as follows:

[0067]

[0068] wherein, is the ith sample in the basic augmented branch image array, x i is the ith sample in the original image array, Φ(·) is an automatic augmentation operation, K base is the first augmentation intensity, and the K base is a set composed of different transformation operation parameters. Specifically, is the mth transformation operation parameter, m = 1, 2, …, P.

[0069] Further, in step S1, the implementation manner of data augmentation of each sample in the original image array by using the second augmentation intensity is as follows:

[0070]

[0071] wherein, is the ith sample in the strong augmented branch image array, x i is the ith sample in the original image array, Φ(·) is an automatic augmentation operation, K heavy is the second augmentation intensity, and the K heavy is a set composed of different transformation operation parameters. Specifically, is the nth transformation operation parameter, n = 1, 2, …, q. q is greater than or equal to P, that is, the number of transformation operations in the strong augmentation process is greater than or equal to the number of transformation operations in the basic augmentation.

[0072] Specifically, the K and the K each consist of three hyperparameters, and the three hyperparameters are transformation type, probability of applying transformation, and transformation amplitude. In application, since the augmentation intensity corresponding to the strong augmented branch image array is better than the augmentation intensity corresponding to the basic augmented branch image array, that is, the second augmentation intensity K heavy is stronger than the first augmentation intensity K base .

[0073] In specific application, the transformation type, the probability of applying transformation, and the transformation amplitude are randomly selected. As an example:

[0074]

[0075]

[0076] As can be seen from the above, in specific applications, one or more operation methods can be randomly selected from the table above to perform image augmentation;

[0077] Verification experiment:

[0078] right Figure 3 The original image in the image is subjected to occlusion, translation, and rotation operations, and both basic augmentation and strong augmentation are subject to occlusion operations. Figure 3 The image shows the visualization results of the original image array consisting of the original images, the basic augmented branch image array after basic augmented data, the strong augmented branch image array after strong augmented data, and the dual-path augmented images.

[0079] in, Figure 3 The images in the row containing the strong augmented data are sorted from smallest to largest according to their confidence scores. Strong augmented images that are out of the distribution range are downgraded to basic augmented images. Furthermore, the dual-path augmented images are a mixture of strong augmented images and basic augmented images, so that the mixed images are all samples within the distribution. This verifies that the present invention can enhance sample diversity while also reducing out-of-distribution data, thus enabling the neural network to have better performance.

[0080] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A two-path data augmentation method based on out-of-distribution sample detection, characterized in that, The method is used for data augmentation of each sample in the original image array to obtain a mixed augmented image array, each sample being an image, and specifically comprising the following steps: S1, after data augmentation of the original image array by two different augmentation intensities, a basic augmented branch image array and a strong augmented branch image array are obtained; The implementation mode of data augmentation of each sample in the original image array by the first augmentation intensity is as follows: wherein, is the i-th sample of the basis augmented branch image array, x i is the i-th sample of the original image array, Φ(·) is an automatic augmentation operation, K base is the first augmentation strength, and this K base is a set of different transformation operation parameters; The implementation mode of data augmentation of each sample in the original image array by the second augmentation intensity is as follows: wherein, is the i-th sample of the strong augmented branch image array, x i is the i-th sample of the original image array, Φ(·) is the auto-augmentation operation, K heavy is the second augmentation strength, and this K heaby is a set of different transformation operation parameters; S2, detecting the confidence score of each sample in the base augmented branch image array through the first confidence detection network At the same time, detecting the confidence score of each sample in the strong augmented branch image array through the second confidence detection network N is the total number of samples; augmenting the base branch image array with the i-th sample; and is the confidence score for the i-th sample in the augmented strong branch image array; S3, obtaining a first set of confidence scores S for each sample in the base augmented branch image array obtaining a first set of confidence scores S for each sample in the base augmented branch image array base ; S4, a first set of confidence scores S base is considered to be normally distributed; according to S base calculating a threshold value τ for screening the sample; S5, judging the distribution detection score of each sample whether it is less than the threshold τ, and if the result is yes, determining that the sample is an out-of-distribution sample, and if the result is no, determining that the sample is an in-distribution sample; S6, according to the index corresponding to each out-of-distribution sample in the strong augmented branch image array, each out-of-distribution sample in the strong augmented branch image array is replaced by the corresponding sample in the basic augmented branch image array with the same index as the out-of-distribution sample, to obtain a mixed augmented image array, thereby completing the data augmentation of each sample in the original image array.

2. The two-path data augmentation method based on out-of-distribution sample detection according to claim 1, characterized in that, In step S2, where f θ (·) is the output of the image feature extraction network, is the i-th sample in the base augmented branch image array, is the j-th sample in the base augmented branch image array, and T is a hyperparameter that controls the degree of smoothing of the output distribution, and exp(·) is the exponential function.

3. The two-path data augmentation method based on out-of-distribution sample detection according to claim 1, characterized in that, In step S2, where f θ (·) is the output of the image feature extraction network, is the i-th sample in the strong augmented branch image array, is the j-th sample in the strong augmented branch image array, and T is a hyperparameter that controls the degree of smoothing of the output distribution, and exp(·) is the exponential function.

4. The two-path data augmentation method based on out-of-distribution sample detection according to claim 1, wherein, The implementation mode of calculating the threshold τ in step S4 includes: S41、calculating a first confidence score set S base the mean μ(s base ) and variance σ(s base ) of the set S S42、According to the mean μ(S base ) and variance σ(S base ), the threshold τ is obtained.

5. The two-path data augmentation method based on out-of-distribution sample detection according to claim 4, characterized in that, τ = μ(S base ) - λ * σ(S base ) ; Where λ is a hyperparameter of the 3σ principle.

6. The two-way data augmentation method based on out-of-distribution sample detection according to claim 1, characterized in that, for the mth transformation operation parameter, m = 1, 2,..., P.

7. The two-path data augmentation method based on out-of-distribution sample detection according to claim 1, characterized in that, In step S1, for the nth transformation operation parameter, n = 1, 2,..., q.

8. The two-path data augmentation method based on out-of-distribution sample detection according to claim 6 or 7, characterized in that, and Each of the two is composed of three hyperparameters, and the three hyperparameters are transform type, probability of applying transform, and amplitude of transform, respectively.

Images