A wheat imperfect kernel identification method combining an adversarial sample and a weak classifier

Abstract

For the task of wheat imperfect kernel recognition, the application designs a method based on the combination of adversarial samples and weak classifiers. In actual application scenarios, the detection equipment of wheat often needs to be transported, which will introduce complex noise to the input image. In order to enhance the robustness of the model to noise in complex environment, the adversarial technology is used to generate adversarial samples for wheat image, and the generalization of the model is enhanced by expanding the data. After generating the adversarial samples, a weak classifier is cascaded to the existing model to integrate the model, so as to improve the accuracy of the model in detecting real samples in actual detection. The application proposes a novel and practical solution for the task of wheat imperfect kernel recognition, which has wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to the application of adversarial samples and weak classifiers in deep neural networks. BACKGROUND

[0002] The task of wheat imperfect kernel recognition aims to identify the imperfect kernels in the image. In addition to perfect kernels, the general imperfect kernel types can be divided into broken kernels, worm-eaten kernels, disease spot kernels, sprout kernels and mildewed kernels. Current research mainly uses deep neural networks to classify wheat images. However, in actual application scenarios, wheat detection equipment often needs to be moved, reassembled, etc., which will introduce complex uncontrollable noise to the input image, and deep neural networks are not robust to small perturbations. Most of the current research does not consider the influence of uncontrollable noise caused by the movement of wheat instruments on the image. SUMMARY

[0003] The present application proposes a wheat imperfect kernel recognition method combining adversarial samples and weak classifiers based on the noise caused by the movement of wheat instruments. Based on the classical convolutional neural network EfficientNet-b0 model, the wheat dataset is trained on the EfficientNet-b0 network, and then the adversarial attack technology is used to generate adversarial samples for the wheat image dataset, and the generalization ability of the model is enhanced by expanding the data. After generating the adversarial samples, according to the data distribution of the model being misled to misidentify each type of wheat image, a cascaded weak classifier method is used to cascade a weak classifier to the existing model to improve the accuracy of the model on real samples during actual detection. The present application realizes the above-mentioned purposes through the following technical solutions:

[0004] A wheat imperfect kernel recognition method combining adversarial samples and weak classifiers, comprising the following steps:

[0005] Step 1: Establish a dataset;

[0006] Step 2: Generate wheat adversarial samples;

[0007] Step 3: Cascade weak classifiers.

[0008] As the image preprocessing and dataset establishment in step 1 are as follows:

[0009] 1. The wheat kernel image collected in the actual background is black background, however, the proportion of wheat kernel in the whole image is not large, so the image is cropped with the maximum contour as the boundary.

[0010] 2. For the convenience of neural network input, the image is adjusted to a fixed size (227*227) and the foreground is centered.

[0011] 3. Wheat professional quality inspection personnel look at the picture, mark its class for all pictures, and establish a data set.

[0012] 4. The data set contains 6000 images of each category, of which 5000 are used as a training set to train the neural network, and the remaining 1000 images are used as a test set to verify the generalization performance of the neural network.

[0013] As step two, the generated wheat adversarial samples are described as follows:

[0014] Adversarial attack algorithm:

[0015] Let F(·) represent a deep neural network, x ∈ R D D-dimensional features of normal samples, y ∈ R C C categories of real labels, then the adversarial sample without target attack can be defined as:

[0016] x * ∈ R D ∧ F(x * ) ≠ y ∧ d(x, x * ) ≤ δ (1-1)

[0017] Where δ is a very small constant, the condition d(x, x * ) ≤ ε restricts the difference between the adversarial sample and the normal sample to be small, and the commonly used distance metric d is commonly used l0, l2, l ∞ norm, for images, l0norm restricts the maximum number of pixel points that can be modified on the image, and does not require the modification amount of a single pixel; l2norm is the Euclidean distance, which mainly limits the total amount of perturbation so that the modification is not easily detected; l ∞ norm is mainly used to modify the perturbation range of a single pixel, without limiting the number of modified pixels. For general non-target attacks, formula 1-1 is often converted to a maximization problem for solution:

[0018] max L(F(x * ), y) subject to d(x, x * ) ≤ δ (1-2)

[0019] FGSM algorithm generates adversarial samples from the perspective of fast generation, and its method for generating adversarial samples is:

[0020]

[0021] Where sign is a sign function, taking values from {0, 1, -1}, and denotes the derivative of x, and x* is updated according to formula 1-3 in each iteration to generate the adversarial sample to deceive the neural network. Similarly, the adversarial sample with targeted attack of the target class y' can be defined as:

[0022] x * ∈R D ∧F(x * )=y'∧d(x,x * )≤δ (1-4)

[0023] Formula 1-4 is converted into an optimization problem under the distance metric L:

[0024] minL(F(x * ),y')subject to d(x,x * )≤δ (1-5)

[0025] The solution of formula 1-5 by FGSM algorithm is:

[0026]

[0027] In actual generation, x* is generated according to formula 1-6: * If F(x * ) = y', the attack is successful, otherwise x * is taken as the new x for iteration, which can almost always be successful. However, with the increase of the number of iterations, the introduced noise will be superimposed, and when the iteration is n times, the noise amplitude of the generated adversarial sample is nδ. Therefore, the number of iterations n should not be too large, otherwise visible noise will be generated.

[0028] Generation of wheat adversarial samples:

[0029] (1) Initialize the network F as EfficientNet-b0 network, the loss function L as cross-entropy loss, δ = 0.01, and the first target class y' = 0;

[0030] (2) Encode the labels in the order of normal grains, damaged grains, worm-eaten grains, diseased grains, sprouting grains, and moldy grains as 0, 1, 2, 3, 4, and 5 respectively, and input the data set wheat image x and label y;

[0031] (3) If y = y', then y' = y' + 1 and go to the next step, otherwise go to the next step directly;

[0032] (4) If y' < 6, initialize the count value i = 1 and go to the next step, otherwise it is indicated that all five target classes have been tried and the process is exited;

[0033] (5) Calculate x* according to formula 1-6, if F(x * ) = y', then y, y', i, and F(x* ) the value of the adversarial sample x* is saved as part of the file name and y' = y' + 1 is updated and then step (3) is entered, otherwise x = x * , i = i + 1 is entered into the next step;

[0034] (6) if i > 10, go to step (3), otherwise go to step (5).

[0035] As a cascaded weak classifier of step three, the description is as follows:

[0036] When the model misjudges due to the addition of noise to the input data, the model is more likely to divide it into normal grains, damaged grains and diseased spot grains. The normal grains, damaged grains and diseased spot grains divided by the model may be of other categories, but because of the introduction of noise caused by changes in the collection environment, it is difficult to detect, so the model judges it as these categories. Therefore, a weak classifier that performs quickly and accurately can be cascaded to assist in the judgment of the normal grain, damaged grain and diseased spot grain samples, so as to obtain more reliable judgment results without affecting the real-time performance of the system.

[0037] The distribution of real-world wheat grains shows that the number of normal grain samples accounts for an absolute advantage. According to the experience of grain practitioners, the number of normal grain samples accounts for more than 90% of the total. This leads to the fact that when misclassification occurs, the cost of misclassifying normal grain samples into other categories is the largest, and the cost of misclassifying other categories into normal categories is relatively small. The requirement in the project that the recognition rate of normal wheat reaches more than 90% and the recognition rate of imperfect grain wheat reaches more than 80% is also based on this consideration. Therefore, in actual application, we choose to trust the normal grain determined by EfficientNet-b0, and the diseased spot grain and damaged grain determined by EfficientNet-b0 are decided by cascading a classifier that has faster reasoning speed and acceptable accuracy. Here we use Res-Net18 as the cascaded weak classifier. The parameters of the main classification network and the auxiliary classification network of the entire classification network are shown in Table 1.

[0038] Table 1 Parameters of the classification network after cascading the classifier

[0039] BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is the six-class wheat grain image of the data set of the application.

[0041] Figure 2 is the original image of the six-class wheat grain, the generated adversarial sample and the added noise image. DETAILED DESCRIPTION

[0042] The application will be further described below in conjunction with the drawings:

[0043] Figure 2 is the attack target of EfficientNet-b0 model, and the FSGM algorithm is used to generate Wheat Images of the specified target attack. Since the generated adversarial samples are generated iteratively by image, they cannot be processed in large batches like training neural networks, and the entire process is very time-consuming when traversing the entire dataset. Therefore, 5000 adversarial samples are randomly selected from each class in the established dataset. Each sample is attacked with five incorrect classes as target classes in turn, that is, for each input image, five adversarial samples are generated, and a total of 150,000 adversarial samples are expected to be generated for 30,000 images of six classes. Set δ to 0.01, specify the maximum number of iterations to 10, and give up if the attack is not successful within 10 iterations. Therefore, the maximum perturbation added is 0.1. If the attack is successful, the generated adversarial sample is saved, and the target class, the current iteration number, and the confidence of the neural network output at this time are recorded.

[0044] Since the maximum number of iterations is specified, the number of generated adversarial samples is not the expected 150,000. In fact, only about half of the attacks were successful within 10 iterations, and a total of 78,900 adversarial samples were generated. The success rate of cross-generation of adversarial samples between classes is shown in Table 2, where the first list represents the original class, and the first row represents the target class. For example, the element 1.0 in the second row and the third column indicates that the source class is normal grain, and the target is damaged grain. All 5000 samples are attacked successfully.

[0045] Table 2: Success rate of cross-attack of six classes of wheat images

[0046]

[0047] (1) Regardless of the original class, it is very easy to attack the network to judge the input with a small perturbation as normal grain, damaged grain, and diseased grain.

[0048] (2) Regardless of the original class, it is very difficult to attack the network to judge the input with a small perturbation as worm-eaten grain, sprout grain, and mold grain.

[0049] From this we can infer that for this network model, when the input image is superimposed with noise that cannot be detected by the naked eye, it tends to be judged as normal grain, damaged grain, and diseased grain. The original image of the six classes of wheat grains, the generated adversarial samples, and the added noise are as follows: Figure 2, in each group of subgraphs, the leftmost is the original image, and in the five columns of images on the right, the first row is the generated adversarial sample, and the second row is the perturbation image obtained by directly subtracting the original image from the adversarial sample. Each perturbation image is annotated with the target class of the corresponding adversarial sample, the number of iterations used to generate the adversarial sample, and the confidence with which the model determines the adversarial sample to be the target class. For example Figure 2 In the second column of the first group of subgraphs, the annotation "broken_1_0.69" indicates that the first iteration successfully changed the image class, causing the model to determine the adversarial sample as broken wheat with a confidence of 0.69. After cascading the weak classifiers, the two models were deployed in a wheat quality automatic detection system and tested again using five groups of samples. The error comparison of the system output for the five groups of samples in this test is shown in Table 3.

[0050] Table 3. Error comparison before and after cascading classifiers

[0051]

[0052] As can be seen from Table 3, after cascading the weak classifiers, the imperfect rates of the detected samples are reduced, but the human-machine error of some samples (samples 1, 3, and 4) is higher than before the cascading classifier. This is because the imperfect rates of these samples after adversarial training are already very close to or even lower than the true values. The re-inspection of imperfect grains by the cascading classifier further reduces the imperfect rate of detection, which in turn increases the human-machine error of some samples. However, according to the experience of wheat quality inspectors, the true imperfect rate of most samples is very low, which leads to the fact that from the perspective of large samples, the decision error occurs more often when perfect grains are determined to be imperfect, resulting in a higher imperfect rate of system detection than the true imperfect rate of manual detection. The data in Table 3 also confirms this conclusion. Overall, the cascading classifier is more beneficial than harmful.

Claims

1. A method of identifying imperfect grains of wheat by combining an adversarial sample and a weak classifier, characterized by Comprising the following steps: Step one: In order to enhance the robustness of the model to uncontrollable noise in complex environment, the adversarial attack technology is used to generate adversarial samples for wheat data set, and the generalization of the model is improved; F(·) represents a deep neural network, x∈R D represents the normal sample with D-dimensional characteristics, y∈R C represents the real label with C categories, then the adversarial sample without target attack can be defined as: x * ∈R D ∧F(x * )≠y∧d(x,x * )≤δ; Step two: for general non-targeted attack, convert step one into a maximum problem solution: max L(F(x * ),y) subject to d(x,x * )≤δ, FGSM algorithm is generated from the fast generated adversarial samples, and the method for generating the generated adversarial samples is: Step three: convert step two into an optimization problem solution under distance metric L: min L(F(x * ),y') subject to d(x,x * )≤δ, the solution of FGSM algorithm for this equation is: Step four: for step three, the FSGM algorithm is used to expand the wheat data set for adversarial attack, which will cause the input data to be misjudged due to the addition of noise. When the model is misjudged due to the addition of noise to the input data, the model is more likely to classify it into normal kernels, damaged kernels, and diseased kernels. By cascading a weak classifier to assist in decision-making, the accuracy of the decision result can be improved.

2. The method according to claim 1, wherein the method is characterized by In step one, the generation of a wheat dataset using adversarial samples is proposed, where δ is a small constant, and the condition d(x, x * )≤ε restricts the difference between the adversarial sample and the normal sample to be small. Common distance metrics d commonly use l0, l2, and l ∞ norms. For images, the l0norm restricts the maximum number of pixel points that can be modified, without requiring the modification amount of a single pixel. The l2norm is the Euclidean distance, which mainly limits the total amount of perturbation to make the modification not easily detected. The l ∞ norm is mainly used to modify the perturbation range of a single pixel, without limiting the number of modified pixels. 3.The method according to claim 1, wherein In step two, the fast adversarial sample is generated by FGSM algorithm, wherein sign is a sign function, taking a value of one of {0, 1, -1}, and is a gradient of x. x , which represents derivation of x, and x* is updated according to step two in iteration, so that the adversarial sample can be generated to deceive the neural network, and the adversarial sample with target attack of the specified target category y' can be defined as: x * ∈R D ∧F(x * )=y'∧d(x,x * )≤δ。 4. The method according to claim 1, wherein the method is characterized by In step three, actually generate x according to step three * , if the model is successfully misled so that F(x * ) = y', the attack is successful, otherwise, continue iteration with x * as the new x, almost always successful, but as the number of iterations increases, the introduced noise will be superimposed, when iterating n times, the noise amplitude of the generated adversarial samples is nδ, so generally the number of iterations n should not be too large, otherwise visible noise will be generated.

5. The method according to claim 1, wherein the method is characterized by In step four, the normal kernels, damaged kernels, and diseased kernels classified by the model may actually belong to other categories. However, due to changes in the collection environment, noise that is difficult to detect is introduced, causing the model to classify them as these categories. Therefore, a weak classifier that performs quickly and accurately can be cascaded to assist in decision-making for the normal kernels, damaged kernels, and diseased kernels classified by the model. This can obtain more reliable decision results without affecting the real-time performance of the system. Multiple classifiers are used to make the final decision, which belongs to the category of model integration. Model integration is a common strategy in image classification competitions and can effectively avoid overfitting.

Images