A machine learning data augmentation method based on feature sets

Abstract

A machine learning data enhancement method based on feature set, comprising the following steps: S1, feature extraction: acquiring an image data set, each picture being cut to a region of interest corresponding to an aspect of a component under inspection; S2, feature set collection: collecting features of labeled data and unlabeled data, respectively generating feature set target functions of the labeled data and the unlabeled data; S3, target training: generating a training target model according to the feature set target functions of the labeled data and the unlabeled data and completing training; S4, experiment and test: selecting a data set containing the same number of normal and defective aspect images for verification and test; the application simultaneously uses feature set collection of labeled data and unlabeled data, solves the problems of unbalanced normal and defective image sample quantity and difficult data labeling, and effectively improves the precision of image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic visual inspection, and particularly to a machine learning data enhancement method based on feature sets. BACKGROUND

[0002] Automatic visual inspection is the key to production efficiency, and a convolutional neural network (CNN) is superior to previous visual task processing methods and has become a standard method for processing visual inspection images in the industry. However, the use of CNN in visual inspection is still limited by the availability of training data. First, there is an imbalance between the number of normal sample images and defective sample images; second, data labeling is difficult, and obtaining consistent labels requires professional inspectors. It is easy to obtain a large number of unlabeled detection images, but the labeling cost is high. This increases the difficulty of image classification, so it is urgent to develop an effective method to address the problem of image quantity imbalance and data labeling difficulty affecting image classification accuracy.

[0003] Most technical researchers solve the problem of image quantity imbalance and data labeling difficulty affecting image classification accuracy by using oversampling and undersampling in CNN, as well as their combination, and also by using supervised and semi-supervised methods. However, these methods have the following problems: low image classification accuracy.

[0004] To overcome the above shortcomings, the present application provides a machine learning data enhancement method based on feature sets, which can be combined with or replace some existing methods. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application provides a machine learning data enhancement method based on feature sets to solve the technical problems described in the background.

[0006] A machine learning data enhancement method based on feature sets, characterized by comprising the following steps:

[0007] S1, feature extraction: obtain an image data set, and cut each picture to a region of interest corresponding to an aspect of the inspected component;

[0008] S2, feature set: feature set the labeled data and unlabeled data, respectively generate a feature set target function of the labeled data and the unlabeled data;

[0009] S3, target training: generate a training target model according to the feature set target function of the labeled data and the unlabeled data and complete the training;

[0010] S4, experiment and test: select a data set containing the same number of normal and defective aspect images for verification and testing.

[0011] Further, the generation method of the objective function in the feature set of the labeled data is: setting that the image data set is unbalanced, the class with more training samples in the image data set is the majority class, the class with less training samples is the minority class, the majority class contains normal samples, and the minority class contains abnormal samples, taking the label information of the labeled data as a centralized direction, and taking the centers of the majority feature clustering and the minority feature clustering as reference points; when the iteration number of the training process is t, the feature center may be expressed as the exponential moving average of the data features, and is updated constantly as the training proceeds, the majority feature center and the minority feature center are defined as follows:

[0012]

[0013]

[0014] In the formula, and are feature extractors, is the parameter of the feature extractor, t is the iteration number of the training, and are input data, and are the smoothing factors of the majority class and the minority class, respectively;

[0015] The objective functions in the majority class and the minority class feature set are respectively expressed as:

[0016]

[0017]

[0018] The objective function in the feature set of the labeled data can be expressed as:

[0019]

[0020] In the formula, is the number of majority samples, is a boundary value.

[0021] Preferably, the generation method of the objective function in the feature set of the unlabeled data is: on the basis of the feature center, the feature F u associated with the input unlabeled data x u is moved towards or away from the majority feature center, the consistency regularization in the feature space is realized by making the response of the feature extractor consistent between the relocated feature and the original feature, and the change of the feature space is reflected into the input space; the input unlabeled data x u ​​​​​A perturbation Δ is applied, and then the gradient of Δ is determined according to the difference between g(x u + Δ) and f int , and a back propagation process is performed to optimize the solution. The perturbation Δ can be moved in two directions: Δ + and , Δ + means moving the feature towards the main feature center c + , means moving the feature away from the main feature center c + , and the iterative process formula is as follows:

[0022] - ;

[0023] - ;

[0024] In the formula, and are the outputs of g( ) and g( ) at the kth iteration in the back propagation process, λ is the rate of gradient update, and is the gradient parameter.

[0025] After perturbing the input in each direction to obtain and , the corresponding features g( ) and g( ) are centralized based on the original input feature g(x u ).

[0026] The objective function in the unlabeled feature set can be expressed as:

[0027] .

[0028] Preferably, the training target model is:

[0029] ;

[0030] In the formula, is the cross-entropy loss of the softmax function at the output layer of the CNN, and are the losses in the labeled data feature set and the unlabeled data feature set, respectively, and w(t) is the ramp-up function at the tth training iteration.

[0031] Preferably, the value of w(t) is linearly increased from 0 to 1 until a predefined iteration, and then kept at 1, both labeled data and unlabeled data are used in the same mini-batch training process, when there are two classes with large density difference, the cross-entropy loss of the output layer softmax function is expressed as:

[0032] ;

[0033] In the formula, represent the output of the majority class; the threshold value τ used is determined empirically.

[0034] Further, the images in the image data set are captured by three cameras, and the three cameras capture the same inspection position at the same time.

[0035] Further, the labeled data is marked by the production factory inspector through the assigned label.

[0036] Further, the training samples in the image data set account for 50%, and the validation samples and test samples account for 25% respectively.

[0037] Compared with the prior art, the beneficial effects of the present application are: the present application simultaneously uses the feature sets of labeled data and unlabeled data, solves the problems of unbalanced number of normal and abnormal image samples and difficult data labeling, and effectively improves the precision of image classification. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The image classification flowchart of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described by the description of specific embodiments, but this is not a limitation of the present application, those skilled in the art can make various modifications or improvements according to the basic idea of the present application, as long as they do not deviate from the basic idea of the present application, they are within the protection scope of the present application.

[0040] Referring to Figure 1 , the present application provides one embodiment as follows:

[0041] A machine learning data enhancement method based on feature set, comprising the following steps:

[0042] S1, feature extraction: using the image data set obtained when detecting the metal frame automatically, eight features of the metal frame are checked, the images at each inspection position are captured by three cameras, which are located above, below and on the side of the metal frame respectively, and each picture is cut to the region of interest corresponding to the aspect of the component being inspected;

[0043] 80517 Eight product aspects shown in the images were used for training, validation and testing, of which 951 cases (1.18%) showed defects, and each aspect had a different defect rate ranging from 0.23% to 2.54%. In order to label the images, the labels assigned by the production plant inspectors were used, and approximately 2500 parts were inspected per day, and data was collected for one month.

[0044] S2, feature set: feature set of labeled data and unlabeled data, respectively, to generate the feature set target function of labeled data and unlabeled data;

[0045] The generation method of the feature set target function of the labeled data is: set the image data set to be unbalanced, the class with more training samples in the image data set is the majority class, and the class with fewer training samples is the minority class. The majority class contains normal samples, and the minority class contains abnormal samples. The label information of the labeled data is taken as a centralized direction, and the centers of the majority feature clustering and the minority feature clustering are taken as reference points; when the number of iterations in the training process is t, the feature center Can be expressed as the exponential moving average of the data features, and is updated continuously as the training progresses, the majority feature center And the minority feature center Is defined as follows:

[0046] ;

[0047] ;

[0048] In the formula, And is the feature extractor, is the parameter of the feature extractor, t is the number of training iterations, And is the input data, And are the smoothing factors of the majority class and the minority class, respectively;

[0049] The target functions in the majority class and the minority class feature set are respectively expressed as:

[0050] ;

[0051] ;

[0052] The target function of the labeled data feature set can be expressed as:

[0053] ;

[0054] In the formula, is the number of majority samples, is the boundary value.

[0055] The method for generating the objective function in the feature set of unlabeled data is: on the basis of the feature center, the target function is generated by moving the input unlabeled data x u The associated feature F u towards or away from the majority of feature centers, by making the response of the feature extractor consistent between the relocated features and the original features, achieving consistency regularization in the feature space, and reflecting the changes in the feature space to the input space; for the input unlabeled data x u Apply perturbation ∆, then determine the gradient of ∆ according to the difference between g(x u +∆) and f int , and perform the backpropagation process to optimize the solution, the perturbation ∆ can be moved in two directions: ∆ + and , ∆ + means moving the feature towards the main feature center c + , means moving the feature away from the main feature center c + , the iterative process formula is as follows:

[0056] - ;

[0057] - ;

[0058] In the formula, and are the outputs of g( ) and g( ) at iteration k in the backpropagation process, λ is the rate of gradient update, is the gradient parameter;

[0059] After perturbing the input in each direction to obtain and , based on the original input feature g(x u ), the corresponding features g( ) and g( ) will be concentrated;

[0060] The objective function in the unlabeled feature set can be expressed as:

[0061] .

[0062] S3, target training: generating a training target model according to the objective function in the feature set of labeled data and unlabeled data and completing the training;

[0063] The training target model is:

[0064] ;

[0065] where, is the cross-entropy loss of the softmax function at the output layer of the CNN, and are the loss in the labeled data and the loss in the unlabeled data in the feature set, respectively, w(t) is the ramp-up function at the training iteration number t;

[0066] The value of w(t) increases linearly from 0 to 1 until a predefined iteration, and then remains at 1, both the labeled data and the unlabeled data are used in the same mini-batch training process, when there are two classes with a large density difference, the cross-entropy loss of the softmax function at the output layer is expressed as:

[0067] ;

[0068] where, represents the output of the majority class; the threshold value τ used is determined empirically.

[0069] 50% of the dataset is used for training, 25% for validation, and 25% for testing. A dataset containing the same number of normal and defective aspect images is selected for validation and testing, and the remaining dataset is used for training. In the supervised setting, all training data are labeled. In the three semi-supervised settings, 5%, 10%, and 15% of the training data are labeled. Table 1 shows how the data is divided when the annotation supervision rate is 10%;

[0070] Table 1 Number of images used for training, validation, and testing under 10% supervision

[0071]

[0072] In all experiments, the ADAM optimizer is used for training, and in the supervised and semi-supervised settings the values are 0.9 and 0.999, respectively, and the weight decay is set to 0.0001, the initial learning rate is 0.0001, which is reduced to 0.00001 after the 30th cycle of 60 training cycles, the smoothing factor and are 0.0001 and 0.001, respectively, and the boundary value m is 2, all experiments are repeated 5 times with different seeds, and the average accuracy is taken.

[0073] S4, Experiment and Test:

[0074] Supervised experiment: three groups of experiments were carried out in the following way in ResNet-18, VGGNet-19, DenseNet-121 networks respectively, the first group used single CNN, the second group used CNN plus entropy minimization method, and the third group used CNN plus feature set method, and those skilled in the art can obtain the best performance in ResNet-18 network through a limited number of experiments.

[0075] Semi-supervised experiment: in semi-supervised classification, the network is trained with a small amount of labeled data and a large amount of unlabeled data, and the feature set method is compared with pseudo-labeling, "mean teacher" and virtual adversarial training (VAT), the specific steps include:

[0076] When using pseudo-labeling, the threshold τ is set to 0.95 and the learning rate is 0.0001; when using "mean teacher", the learning rate is set to 0.00001, the exponential moving average decay is 0.99, and the maximum consistency coefficient is 5; when using virtual adversarial training, the learning rate is set to 0.0001, 8.0, ξ is set to 10 −6 , and entropy minimization method is used;

[0077] ADAM optimizer and ResNet-18 are used to implement all these methods, the experiment is divided into six groups, the first group is only ResNet-18, the second group is ResNet-18 plus pseudo-labeling, the third group is ResNet-18 plus "mean teacher", the fourth group is ResNet-18 plus virtual adversarial training and entropy minimization, the fifth group is ResNet-18 plus feature set of labeled data, and the sixth group is ResNet-18 plus feature set of labeled and unlabeled data, the above six groups of experiments are carried out at a supervision ratio of 5%, 10% and 15%, and the influence of this ratio on the accuracy is observed, those skilled in the art can obtain that the feature set of labeled and unlabeled data is more conducive to improving the classification accuracy through a limited number of experiments.

[0078] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application, therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical idea disclosed by the present application should be covered by the claims of the present application.

Claims

1. A machine learning data augmentation method based on a feature set, characterized by: The method comprises the following steps: S1, feature extraction: obtain an image dataset, and each picture is cut to a region of interest corresponding to an aspect of a component under inspection; S2, feature set collection: collect features of labeled data and unlabeled data, and generate a feature set target function of the labeled data and the unlabeled data respectively; S3, target training: generate a training target model according to the feature set target functions of the labeled data and the unlabeled data and complete training; S4, experiment and test: select a dataset containing the same number of normal and defective aspect images for verification and testing; The method for generating the target function in the feature set of the marked data is: setting that the image data set is unbalanced, the class with more training samples in the image data set is the majority class, the class with less training samples is the minority class, the majority class contains normal samples, and the minority class contains abnormal samples, taking the label information of the marked data as a centralized direction, and taking the centers of the majority feature clustering and the minority feature clustering as reference points; when the iteration number of the training process is t, the feature centers are expressed as the exponential moving average of the data features, and are updated constantly as the training proceeds, the majority feature center and the minority feature center are defined as follows: ； ； wherein, and is a feature extractor, is a parameter of the feature extractor, t is the number of training iterations, and are input data, and are the smoothing factors for the majority and minority classes, respectively; The target functions in the majority class and the minority class feature sets are respectively represented as: ； ； The target function in the labeled data feature set is represented as: ； wherein is the number of most samples, is the boundary value.

2. The method of claim 1, wherein the method is based on a machine learning data augmentation method in a feature set. The method for generating the objective function in the feature set of the unlabeled data is: on the basis of the feature center, the input unlabeled data x u The associated feature F u Moving towards or away from the majority of feature centers, by making the response of the feature extractor consistent between the relocated features and the original features, achieving consistency regularization in the feature space, and reflecting the changes in the feature space to the input space; for the input unlabeled data x u Apply perturbation ∆, then determine the gradient of ∆ according to the difference between g(x u +∆) and f int , and carry out the back propagation process to optimize the solution, the perturbation ∆ moves in two directions: ∆ + and , ∆ + means moving the feature towards the main feature center c + , means moving the feature away from the main feature center c + , the iterative process formula is as follows: - ； - ； wherein, and are the outputs of g( ) and g( ) at iteration k, respectively, during backpropagation, λ is the rate of gradient update, is the gradient parameter; The input in each direction is perturbed to obtain and Based on the original input feature g(x u ), the corresponding features g( ) and g( ) are concentrated; The target function in the unlabeled feature set is represented as: 。 3.The machine learning data augmentation method based on feature set according to claim 2, characterized in that: The training target model is: ； wherein is the cross-entropy loss of the softmax function at the output layer of the CNN, and are the loss in the labeled data feature set, the loss in the unlabeled data feature set, respectively, and w(t) is a ramp-up function at the training iteration number t.

4. The method of claim 3, wherein: The value of w(t) linearly increases from 0 to 1 until a predefined iteration, and then is kept at 1, the labeled data and the unlabeled data are used in the same mini-batch training process, when there are two classes with a large density difference, the cross-entropy loss of the softmax function of the output layer is expressed as: ； where representing the output of most classes; the threshold value τ used is determined empirically.

5. The method of claim 1, wherein: The images in the image dataset are captured by three cameras, and the three cameras capture the same inspection position at the same time.

6. The machine learning data augmentation method based on feature set according to claim 1, characterized in that: The labeled data is marked by a production plant inspector through an assigned label.

7. The method of claim 1, wherein: The training samples in the image dataset account for 50%, and the verification samples and the test samples account for 25% respectively.

Images