A Wheat Seed Classification Method Based on Data Augmentation and Attention Mechanism

Abstract

The present application discloses a wheat seed classification method based on data augmentation and attention mechanism, which relates to the field of nondestructive testing. This method can quickly, accurately and nondestructively classify and identify wheat seeds by using hyperspectral images in combination with a wheat seed classification model. In this method, image preprocessing operations are adopted to achieve data augmentation, which can achieve the dual purposes of expanding the sample quantity and reducing the dimensionality of bands for sample wheat seeds. It not only solves the problems of insufficient data volume caused by difficult annotation of hyperspectral images, easy overfitting of the model, and poor generalization ability, but also reduces the redundancy of band information and saves computational consumption. The wheat seed classification model is based on a deep learning model to learn the internal laws of sample data, and different band weights are assigned to different bands according to the importance of the band information of hyperspectral images, making full use of the morphological dimension and spectral dimension of hyperspectral images, greatly improving the stability, inference speed and accuracy of the model.

Description

Technical Field

[0001] This application relates to the field of non-destructive testing, and particularly to a wheat seed classification method based on data augmentation and attention mechanism. Background Art

[0002] As a major food crop in China, the yield and quality of wheat are related to the good life of the people and economic development. However, the mixing of different wheat varieties may lead to low yield and quality during the breeding and planting processes. Therefore, accurate classification of wheat seeds is of great significance.

[0003] Traditional seed variety detection methods are mainly biochemical detection methods. Although these methods can achieve high accuracy in the detection of crop seeds, biochemical detection methods are highly dependent on manual labor, have a long detection time, and are prone to cause irreversible damage to the inside of the seeds.

[0004] Hyperspectral imaging technology can obtain the physical and chemical information of seeds without damaging the inside of the seeds. Currently, it has broad prospects in the detection and classification of agricultural products. Hyperspectral images contain the external morphological information and internal spectral information of samples. However, when using conventional statistical methods to process hyperspectral data, problems such as low accuracy and poor timeliness will be faced, and manual feature extraction is required, which is time-consuming and laborious. Summary of the Invention

[0005] In view of the above problems and technical requirements, the applicant of this application proposes a wheat seed classification method based on data augmentation and attention mechanism. The technical solution of this application is as follows:

[0006] A wheat seed classification method based on data augmentation and attention mechanism, characterized in that the method includes:

[0007] Obtain the sample hyperspectral images of several sample wheat seeds of different varieties;

[0008] Perform image preprocessing operations on all the obtained sample hyperspectral images to obtain several training sample images. Compared with the sample hyperspectral images, the training sample images achieve an increase in the number of samples and a reduction in the number of bands;

[0009] Train a wheat seed classification model using the training sample images. The wheat seed classification model includes a per-channel convolutional layer, an attention mechanism module, a convolutional module, a fully connected module, and a classifier cascaded in sequence from input to output. The per-channel convolutional layer extracts the feature maps of each band of the input image respectively, and the attention mechanism module outputs the feature maps containing band weights for each band;

[0010] Obtain the variety classification result based on the hyperspectral image of the wheat seeds to be classified using the trained wheat seed classification model.

[0011] A further technical solution is that the number of convolutional kernels included in the per-channel convolutional layer is equal to the number of bands included in each training sample image. Each convolutional kernel in the per-channel convolutional layer performs convolutional processing on the corresponding band in the input training sample image to achieve feature extraction, and a feature map of the corresponding band is obtained.

[0012] A further technical solution is that the attention mechanism module includes a global average pooling layer, a first fully connected layer, a first activation function layer, a second fully connected layer, and a second activation function layer that are cascaded in sequence from input to output:

[0013] The global average pooling layer performs global average pooling on the feature maps of all bands extracted by the per-channel convolutional layer to compress them into a one-dimensional feature vector. Each value in the one-dimensional feature vector is the average value of all pixel points in the feature map of each band and has a global receptive field of the corresponding band;

[0014] After the one-dimensional feature vector passes through the first fully connected layer, the first activation function layer, the second fully connected layer, and the second activation function layer in sequence, the band weights of each band are output and multiplied by the feature maps of all bands extracted by the per-channel convolutional layer, and the feature maps of each band containing the band weights are output.

[0015] A further technical solution is that the ratio of the number of neurons in the first fully connected layer to the number of bands included in each training sample image is 1:M, where M>1; the first activation function layer uses the ReLU function; the number of neurons in the second fully connected layer is equal to the number of bands included in each training sample image; the second activation function layer uses the Sigmoid function to fix the value between 0 and 1 to obtain the band weights of each band.

[0016] A further technical solution is to perform image preprocessing operations on all obtained sample hyperspectral images, including for each sample hyperspectral image:

[0017] Calculate the band mean of each original band included in the sample hyperspectral image;

[0018] Cluster all original bands according to the band means of each original band to obtain several band categories, and each band category includes one or more original bands;

[0019] Randomly extract several original bands from each band category respectively, and use the original bands in each band category to form a new training sample image of the sample hyperspectral image. The number of bands included in the training sample image is less than the number of bands included in the sample hyperspectral image, and band dimensionality reduction is achieved;

[0020] Repeat the step of randomly extracting a number of original bands from each band category respectively, and construct multiple training sample images from a sample hyperspectral image to achieve the expansion of the number of samples.

[0021] A further technical solution thereof is that the convolution module in the wheat seed classification model includes two cascaded convolution units with the same structure. Each convolution unit includes a two-dimensional convolution layer, a batch normalization layer, an activation function layer, and a max pooling layer cascaded in sequence from input to output.

[0022] A further technical solution thereof is that the two-dimensional convolution layer in the first convolution unit contains 32 convolution kernels, the two-dimensional convolution layer in the second convolution unit contains 64 convolution kernels, and the ReLU function is adopted for the activation function layer in both convolution units.

[0023] A further technical solution thereof is that the size of each convolution kernel in the two-dimensional convolution layer of the first convolution unit is 3×3, the stride is 1, and the padding method is same; the size of each convolution kernel in the two-dimensional convolution layer of the second convolution unit is 3×3, the stride is 1, and the padding method is same; the size of the convolution kernel in the max pooling layer of both convolution units is 5×5, the stride is 2, and the padding method is same.

[0024] A further technical solution thereof is that the fully connected module in the wheat seed classification model includes three fully connected layers, and the ReLU activation function is adopted between every two fully connected layers; the classifier in the wheat seed classification model adopts a Softmax classifier; the wheat seed classification model adopts cross-entropy loss as the loss function and the Adam optimization method.

[0025] A further technical solution thereof is that the variety classification result is obtained based on the hyperspectral image of the wheat seeds to be classified by using the trained wheat seed classification model, including:

[0026] Perform image preprocessing operations on the hyperspectral image of the wheat seeds to be classified to obtain multiple images to be classified with reduced number of bands.

[0027] Input the multiple images to be classified into the wheat seed classification model to obtain the predicted probabilities of each variety for each image to be classified respectively.

[0028] Sum up the predicted probabilities of all images to be classified belonging to the same variety, and take the category with the highest final predicted probability as the variety classification result.

[0029] The beneficial technical effects of this application are:

[0030] The present application discloses a wheat seed classification method based on data augmentation and attention mechanism. This method can quickly and accurately identify and classify wheat seeds by using hyperspectral images in combination with a wheat seed classification model, without causing damage to the interior of the seeds, greatly reducing manpower and material resources, being efficient and convenient. The wheat seed classification model is based on a deep learning model such as a convolutional neural network to learn the internal laws of sample data, automatically extract the deep features of the data, and adopt image preprocessing operations to achieve data augmentation, which can achieve the dual purposes of expanding the sample quantity and reducing the dimensionality of the bands of the sample wheat seeds. It not only solves the problems of insufficient data volume caused by difficult annotation of hyperspectral images, easy overfitting of the model, and poor generalization ability, but also reduces the redundancy of band information and saves computational consumption.

[0031] In addition, the wheat seed classification model can assign different band weights to different bands according to the importance of the band information of the hyperspectral image for classification, reduce the band differences between the expanded samples, and combine with a two-dimensional convolutional neural network to fully utilize the morphological dimension and spectral dimension of the hyperspectral image of wheat seeds. Compared with a single two-dimensional convolution operation, it greatly improves the stability, inference speed and accuracy of the model. Brief Description of the Drawings

[0032] Figure 1 is the method flow chart of the wheat seed classification method according to an embodiment of the present application.

[0033] Figure 2 is a schematic diagram of obtaining several training sample images by performing image preprocessing operations on the sample hyperspectral image in an example.

[0034] Figure 3 is the model structure diagram of the wheat seed classification model. Detailed Embodiments

[0035] The following further describes the detailed embodiments of the present application with reference to the drawings.

[0036] The present application discloses a wheat seed classification method based on data augmentation and attention mechanism. Please refer to Figure 1 , this method includes the following steps:

[0037] Step 110, obtain sample hyperspectral images of several sample wheat seeds of different varieties. It includes the following steps:

[0038] (1)Collect the original hyperspectral images of each sample wheat seed. The general collection method is to place the sample wheat seeds in the seed plate of the hyperspectral image collection system and use the line scanning method to collect and obtain the original hyperspectral images of the sample wheat seeds. Therefore, the obtained original hyperspectral images often contain background images of other irrelevant regions in addition to the images of the regions of interest where the sample wheat seeds are located. In one embodiment, the obtained original hyperspectral images contain 94 original bands, and the spatial resolution under each band is 1392×1000.

[0039] (2)In order to extract the regions of interest from the original hyperspectral images, first perform black and white board correction on the obtained original hyperspectral images according to to obtain the corrected images, making the images as accurate as possible. Among them, I represents the original hyperspectral image before correction, and I cablibration represents the corrected image obtained after correction, I Black represents the all-black image of the black board, and I White is the white board image.

[0040] (3)Construct a pseudo-RGB image from the corrected images, convert it to the YCbCr color space, select the Cb component as the grayscale image, perform K-means clustering segmentation to obtain the binary image after clustering segmentation, and perform opening and closing operations on the binary image to obtain the binary mask of the sample wheat seeds.

[0041] (4)Use the binary mask to cut the original hyperspectral images to obtain the sample hyperspectral images of the sample wheat seeds. At this time, the obtained sample hyperspectral images remove the background images of the irrelevant regions and only contain the images of the regions of interest where the sample wheat seeds are located. In one embodiment, the obtained sample hyperspectral images also contain 94 original bands, and the spatial resolution under each band is 60×80.

[0042] Step 120, perform image preprocessing operations on all the obtained sample hyperspectral images to obtain a number of training sample images. The training sample images achieve an expansion of the sample quantity and a reduction in the number of bands compared to the sample hyperspectral images. Specifically:

[0043] (1)Calculate the band mean of each original band included in the sample hyperspectral images.

[0044] (2)Cluster all the original bands according to the band means of the respective original bands to obtain a number of band categories, and each band category contains one or more original bands. The information correlation among the original bands in the same band category is high and the difference is small, while the information correlation among the original bands in different band categories is small and the difference is large.

[0045] (3) Randomly extract a number of original bands from each band category, and use the original bands in each band category to form a new training sample image of the sample hyperspectral image. The number of bands included in the training sample image is less than the number of bands included in the sample hyperspectral image, so as to achieve band dimensionality reduction.

[0046] (4) Repeat the step of randomly extracting a number of original bands from each band category, and construct multiple training sample images from a sample hyperspectral image to achieve sample number expansion. The number of original bands extracted from each band category can be equal or unequal.

[0047] That is, through the above operations, the dual purposes of data expansion and band dimensionality reduction are achieved at the same time, which is beneficial to solving the problem of overfitting in model training due to insufficient samples in the subsequent model training process, and is also beneficial to solving problems such as high computational consumption caused by high band redundancy.

[0048] For example, in an instance, as Figure 2 shown, the sample hyperspectral image contains 94 original bands. Perform K-means clustering on the 94 original bands, where K is selected as 5, that is, cluster the 94 original bands into 5 band categories, denoted as band categories 1 to 5 respectively. Then randomly select a number of original bands from band category 1, and also select a number of original bands from band categories 2 to 5 respectively. Then use the original bands extracted from the 5 band categories to construct a training sample image. The constructed training sample image contains 25 original bands, achieving dimensionality reduction compared to the sample hyperspectral image. 4 training sample images are formed from the original bands included in each sample hyperspectral image according to the above method. Thus, the sample number changes from the sample hyperspectral image to the training sample image by 4 times, achieving sample number expansion. Figure 2 It only shows a schematic diagram of each training sample image selecting original bands from each band category, and does not represent the specific number of original bands selected from each band category.

[0049] Step 130, train a wheat seed classification model using the training sample image.

[0050] Please refer to Figure 3 , the wheat seed classification model includes a per-channel convolutional layer, an attention mechanism module, a convolutional module, a fully connected module, and a classifier cascaded in sequence from input to output. Among them:

[0051] (1) The per-channel convolutional layer separately extracts the feature maps of each band of the input image and outputs them to the attention mechanism module: The number of convolutional kernels included in the per-channel convolutional layer is equal to the number of bands included in each training sample image. Each convolutional kernel in the per-channel convolutional layer performs convolutional processing on the corresponding band in the input training sample image to achieve feature extraction and obtain the feature map of the corresponding band.

[0052] In one embodiment, each training sample image includes 25 bands, so the per-channel convolutional layer includes 25 convolutional kernels. Each convolutional kernel performs a convolutional operation on one band to extract the feature map of that band, and finally 25 feature maps of 25 bands are extracted. The size of each convolutional kernel in the per-channel convolutional layer is 3×3, the stride is 1, and the padding method is same (padding 0 around the input data to make the output image size the same as the input size).

[0053] (2) The attention mechanism module obtains the feature maps of each band extracted by the per-channel convolutional layer and outputs the feature map containing the band weights for each band. The band weight of each band reflects the contribution rate of that band. Specifically, the attention mechanism module includes a global average pooling layer, a first fully connected layer, a first activation function layer, a second fully connected layer, and a second activation function layer cascaded in sequence from input to output:

[0054] The global average pooling layer performs global average pooling on the feature maps of all bands extracted by the per-channel convolutional layer to compress them into a one-dimensional feature vector. Each value in the one-dimensional feature vector is the average value of all pixel points in the feature map of each band and has the global receptive field of the corresponding band.

[0055] The one-dimensional feature vector is input into the first fully connected layer. The ratio of the number of neurons in the first fully connected layer to the number of bands included in each training sample image is 1:M, where M>1, that is, the first fully connected layer has fewer neurons. In one embodiment, M = 4.

[0056] The output of the first fully connected layer enters the first activation function layer, and the first activation function layer uses the ReLU function.

[0057] The output of the first activation function layer enters the second fully connected layer, and the number of neurons in the second fully connected layer is equal to the number of bands included in each training sample image.

[0058] The output of the second fully connected layer enters the second activation function layer, and the second activation function layer uses the Sigmoid function to fix the value between 0 and 1 to obtain the band weights of each band.

[0059] The output of the second activation function layer is multiplied by the feature maps of all bands extracted by the per-channel convolution layer, and the feature maps of each band containing band weights are output.

[0060] (3) The convolution module includes two cascaded convolution units with the same structure. Each convolution unit includes a two-dimensional convolution layer, a batch normalization layer, an activation function layer, and a max pooling layer cascaded in sequence from input to output.

[0061] The two-dimensional convolution layer in the first convolution unit contains 32 convolution kernels. The size of each convolution kernel in the two-dimensional convolution layer of the first convolution unit is 3×3, the stride is 1, and the padding method is same.

[0062] The two-dimensional convolution layer in the second convolution unit contains 64 convolution kernels. The size of each convolution kernel in the two-dimensional convolution layer of the second convolution unit is 3×3, the stride is 1, and the padding method is same.

[0063] The activation function layers in both convolution units adopt the ReLU function to prevent gradient vanishing and accelerate the convergence speed.

[0064] The max pooling layers in both convolution units are used to compress parameters. The size of the convolution kernels in the max pooling layers is 5×5, the stride is 2, and the padding method is same.

[0065] (4) The fully connected module includes three fully connected layers, and the ReLU activation function is adopted between every two fully connected layers. The classifier in the wheat seed classification model adopts the Softmax classifier.

[0066] The wheat seed classification model adopts cross-entropy loss as the loss function and the Adam optimization method.

[0067] Step 140: Use the trained wheat seed classification model to obtain the variety classification result based on the hyperspectral image of the wheat seeds to be classified.

[0068] Similarly, perform image preprocessing operations on the hyperspectral image of the wheat seeds to be classified to obtain multiple images to be classified with reduced band dimensions. The hyperspectral image of the wheat seeds to be classified in this step also refers to the image of the region of interest after removing the background image, and generally needs to be extracted from the original hyperspectral image as in step 110. The specific method of the image preprocessing operation performed on the hyperspectral image of the wheat seeds to be classified is the same as that in step 120. For example, if the hyperspectral image of the wheat seeds to be classified contains 94 bands, then 4 images to be classified can be obtained through the image preprocessing operation, and each image to be classified contains 25 bands.

[0069] Multiple images to be classified are input into the wheat seed classification model to obtain the predicted probability of each variety of each image to be classified. Then the predicted probabilities of all images to be classified belonging to the same variety are summed up, and the category with the highest final predicted probability is taken as the variety classification result.

[0070] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A wheat seed classification method based on data augmentation and attention mechanism, characterized in that, The method includes: Obtaining sample hyperspectral images of a number of sample wheat seeds of different varieties; Performing image preprocessing operations on all the obtained sample hyperspectral images to obtain a number of to-be-trained sample images, where the to-be-trained sample images achieve an expansion in the number of samples and a reduction in the number of bands compared to the sample hyperspectral images; the performing of image preprocessing operations on all the obtained sample hyperspectral images includes, for each sample hyperspectral image: obtaining the band mean of each original band included in the sample hyperspectral image, clustering all the original bands according to the band means of the respective original bands to obtain a number of band categories, with each band category including one or more original bands; randomly extracting a number of original bands from each band category respectively, and using the original bands in each band category to form a new to-be-trained sample image of the sample hyperspectral image, where the number of bands included in the to-be-trained sample image is less than the number of bands included in the sample hyperspectral image, achieving a reduction in the number of bands; repeating the step of randomly extracting a number of original bands from each band category respectively, and constructing multiple to-be-trained sample images from one sample hyperspectral image, achieving an expansion in the number of samples; Training a wheat seed classification model using the to-be-trained sample images, where the wheat seed classification model includes, cascaded in sequence from input to output, a per-channel convolutional layer, an attention mechanism module, a convolutional module, a fully connected module, and a classifier, the per-channel convolutional layer respectively extracting feature maps of each band of the input image, and the attention mechanism module outputting a feature map containing band weights for each band; the attention mechanism module includes, cascaded in sequence from input to output, a global average pooling layer, a first fully connected layer, a first activation function layer, a second fully connected layer, and a second activation function layer: the global average pooling layer performing global average pooling on the feature maps of all the bands extracted by the per-channel convolutional layer to compress them into a one-dimensional feature vector, where each value in the one-dimensional feature vector is the average value of all the pixel points in the feature map of each band and has a global receptive field corresponding to the band; after the one-dimensional feature vector passes through the first fully connected layer, the first activation function layer, the second fully connected layer, and the second activation function layer in sequence, outputting the band weights of each band and multiplying them with the feature maps of all the bands extracted by the per-channel convolutional layer, and outputting a feature map containing band weights for each band; Obtaining a variety classification result based on the hyperspectral image of the wheat seed to be classified using the trained wheat seed classification model.

2. The method according to claim 1, wherein The number of convolutional kernels included in the per-channel convolutional layer is equal to the number of bands included in each to-be-trained sample image, and each convolutional kernel of the per-channel convolutional layer performs convolutional processing on the corresponding band in the input training sample image to achieve feature extraction and obtain a feature map of the corresponding band.

3. The method according to claim 1, wherein The ratio of the number of neurons in the first fully connected layer to the number of bands included in each image of the sample to be trained is 1:M, where M>1; the ReLU function is used in the first activation function layer; the number of neurons in the second fully connected layer is equal to the number of bands included in each image of the sample to be trained; the Sigmoid function is used in the second activation function layer, and the values are fixed between 0 and 1 to obtain the band weights for each band.

4. The method according to claim 1, wherein The convolutional module in the wheat seed classification model includes two cascaded convolutional units with the same structure. Each convolutional unit includes a two-dimensional convolutional layer, a batch normalization layer, an activation function layer, and a max pooling layer cascaded in sequence from input to output.

5. The method according to claim 4, characterized in that The two-dimensional convolutional layer in the first convolutional unit contains 32 convolutional kernels, and the two-dimensional convolutional layer in the second convolutional unit contains 64 convolutional kernels. The ReLU function is used in the activation function layers of both convolutional units.

6. The method according to claim 5, wherein The size of each convolutional kernel in the two-dimensional convolutional layer of the first convolutional unit is 3×3, the stride is 1, and the padding method is same; the size of each convolutional kernel in the two-dimensional convolutional layer of the second convolutional unit is 3×3, the stride is 1, and the padding method is same; the size of the convolutional kernel in the max pooling layer of both convolutional units is 5×5, the stride is 2, and the padding method is same.

7. The method according to claim 1, characterized in that, The fully connected module in the wheat seed classification model includes three fully connected layers, and the ReLU activation function is used between every two fully connected layers; the Softmax classifier is used as the classifier in the wheat seed classification model; the cross-entropy loss is used as the loss function in the wheat seed classification model, and the Adam optimization method is used.

8. The method according to claim 1, characterized in that, Using the trained wheat seed classification model to obtain the variety classification result based on the hyperspectral image of the wheat seeds to be classified, including: Performing image preprocessing operations on the hyperspectral image of the wheat seeds to be classified to obtain multiple images to be classified with reduced band dimensions; Inputting the multiple images to be classified into the wheat seed classification model to obtain the prediction probabilities of each variety for each image to be classified respectively; Summing up the prediction probabilities of all images to be classified belonging to the same variety, and taking the category with the highest final prediction probability as the variety classification result.

Images