A CBAM-HRNet model based on convolution attention mechanism is used for wheat ear grain segmentation and counting

Abstract

The application discloses a CBAM-HRNet model wheat ear grain segmentation and counting method based on a convolution attention mechanism, relates to the technical field of segmentation and counting, and comprises the following steps: firstly, collecting dataset images, and preprocessing the images to form an ear dataset; then, using a deep learning segmentation network to perform image segmentation on the wheat ear grains, obtaining a prediction model through training, testing a test set by calling the prediction model, and outputting a prediction result; and finally, constructing an ear grain counting model by combining the output prediction result with an image processing technology, so as to realize accurate prediction and counting of the wheat ear grains. The application constructs a CBAM-HRNet wheat ear grain segmentation and counting deep learning model based on a convolution attention mechanism, uses an image processing algorithm and a wheat ear grain texture feature to construct an ear grain counting model, and the method has the advantages of better segmentation effect, better robustness, further improved segmentation precision and strong generalization ability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of segmentation and counting, and particularly relates to a CBAM-HRNet model wheat ear grain segmentation and counting method based on a convolution attention mechanism. BACKGROUND

[0002] Scientific and accurate prediction of wheat yield helps to ensure food supply security and social stability; traditional wheat yield estimation is obtained by manually investigating the number of ears per unit area and the number of grains per ear before wheat harvesting, and multiplying the conventional thousand-grain weight, and the estimated yield is often affected by human factors, and meanwhile, it is time-consuming, labor-intensive and low in efficiency, thereby restricting the timeliness and accuracy of wheat yield estimation; the number of ears and grains is an important parameter in crop yield estimation, and has become a key scientific problem in intelligent yield estimation;

[0003] With the development of image processing and machine learning technologies, an important monitoring means is provided for the segmentation and identification of the number of wheat ears and the number of grains per ear; although image processing technology is widely used in the identification of the number of wheat ears and the number of grains per ear, these methods focus on the extraction of texture features, color features and morphological features, and there are still problems in efficiency and practical application;

[0004] Fernandez-Gallego et al. used the local maximum peak method to calculate the number of wheat ears from a field RGB color image, and the success rate of counting was higher than 90%. Current image processing technology requires a large amount of manual image feature extraction, which puts high requirements on the environment and technology, and machine learning has been proven to have significant advantages in the field of image segmentation and identification. Liu Zhe et al. proposed a wheat ear counting algorithm based on color feature K-means clustering, and the recognition accuracy reached 94%. Xu X et al. automatically extracted the contour features of the wheat ear based on the K-means clustering algorithm, and then constructed a CNN model to improve the wheat ear recognition accuracy to 98.3%. However, traditional image processing technology and machine learning methods still have problems such as long recognition and segmentation time, low efficiency, poor recognition and segmentation effect of complex images, etc.

[0005] Existing deep learning methods are widely used to identify the number of wheat ears and the number of grains per ear, which can achieve high-precision image segmentation and identification, but there are still problems such as the need for professional equipment such as CMOS cameras to obtain wheat ear grain images, which is difficult to apply in complex production, and the problem of image recognition and segmentation of dense small targets, which is prone to target adhesion and difficult to improve accuracy. SUMMARY

[0006] In view of the defects and problems that in the existing recognition, segmentation and counting of wheat ear grains, the recognition and segmentation time is long, the efficiency is low, the recognition and segmentation effect of complex images is poor, the problem of image recognition and segmentation of dense small targets, the targets are easy to be connected, and the accuracy is difficult to improve, the present application provides a CBAM-HRNet model wheat ear grain segmentation and counting method based on convolution attention mechanism.

[0007] The technical problems solved by the present application are as follows: a CBAM-HRNet model wheat ear grain segmentation and counting method based on convolution attention mechanism, comprising the following steps:

[0008] Step one, data acquisition: select multiple different varieties of wheat, and obtain multiple original wheat ear images of each type, create a wheat ear information table based on different varieties;

[0009] Step two, pre-processing of the wheat ear images in step one: including data normalization and data enhancement, forming a wheat ear dataset;

[0010] Step three, using a deep learning segmentation network to perform image segmentation on the wheat ear grains, obtaining a prediction model after training, and calling the prediction model to test the test set, outputting the prediction result: constructing CBAM-HRNet, HRNet, PSPNet, DeeplabV3+ segmentation model and U-Net based on convolution attention mechanism to respectively segment the wheat ear grains and compare; the CBAM-HRNet is connected in parallel with multiple resolution branches, and the information interaction between different branches is continuously carried out, so as to achieve the purpose of strong semantic information and accurate position information, and avoid the loss of a large amount of effective information in the continuous up-sampling process; the convolution attention mechanism is added to realize the up-sampling process of the representation branch;

[0011] Step four, constructing a grain counting model by combining the prediction result output by step three with image processing technology to realize accurate prediction and counting of wheat ear grains: after the wheat ear samples of each variety selected are predicted by the deep learning segmentation model, some of the grains will be connected, and the image processing method is needed to eliminate the overlapping and connected parts, including the following steps:

[0012] S401, the prediction result of step three is subjected to gray scale processing, and the color space is converted from RGB to GRAY;

[0013] S402, the image threshold is binarized to remove the overlapping part; the binarized image needs to be eroded to eliminate noise points, and the distance from the pixel points in the image to the nearest zero pixel points is calculated, and the skeleton of the contour is obtained after distance transformation;

[0014] S403, using normalization to convert the dimensional expression into a dimensionless expression after the binary processing of the image, becoming a scalar, and the gray value of the image obtained by normalization is between 0 and 1.0;

[0015] S404, the gray image is processed into a binary image by binaryzation and opening operation;

[0016] S405, the contour is extracted according to the boundary point provided by the binary image, and the extracted contour is the number of grains on the wheat ear.

[0017] The CBAM-HRNet model based on the convolution attention mechanism is used for wheat ear grain segmentation and counting method, and the image acquisition device is parallel to the wheat ear in step one, the object distance is changed by adjusting the vertical height, and the wheat ear completely appears in the lens field of view of the mobile device, and a clear ear image is displayed.

[0018] The CBAM-HRNet model based on the convolution attention mechanism is used for wheat ear grain segmentation and counting method, and the wheat ear data table created in step one includes nitrogen fertilizer treatment, shooting background, shooting date, weather, resolution, image size, shooting device, focal length and image quantity information of wheat ear.

[0019] The CBAM-HRNet model based on the convolution attention mechanism is used for wheat ear grain segmentation and counting method, and the image size is normalized to 480x80 in step two, so as to reduce the model operation amount and reduce the risk of over fitting.

[0020] The CBAM-HRNet model based on the convolution attention mechanism is used for wheat ear grain segmentation and counting method, and the data augmentation uses the images in the original data set to expand, so as to solve the problem of insufficient image data; at the same time, Gaussian blur is used to reduce image noise and reduce detail level, so as to enhance the image effect of the image under different scale sizes.

[0021] The CBAM-HRNet model based on the convolution attention mechanism is used for wheat ear grain segmentation and counting method, and the two kinds of segmentation objects of wheat ear grain and background are manually marked by Labelme image labeling tool in step two, and the marking information is converted into a mask image; the wheat ear image and its mask image constitute the data set required by the deep learning segmentation model; the number of images of each different treatment in the data set is uniformly distributed, and the wheat ear data set is divided into training set and verification set according to the proportion.

[0022] The CBAM-HRNet model based on the convolution attention mechanism is used for wheat ear grain segmentation and counting, and the network body of the CBAM-HRNet comprises four stages and four parallel convolution branches, and the resolutions are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 respectively; the first stage comprises 4 bottleneck layer residual units, and each unit is followed by a 3*3 convolution, so that the number of feature maps becomes 32, and the other stages are the same; each module comprises 4 residual units, each unit provides two 3*3 convolutions for each resolution, followed by a BN layer and a nonlinear activation function ReLU, and a multi-resolution fusion module is arranged at the end of each stage.

[0023] In the fourth step, the prediction result is read in and gray processed by OpenCV and NumPy; and the 120 threshold value is set for binary processing.

[0024] In the fourth step, the prediction result is read in and gray processed by OpenCV and NumPy; and the 120 threshold value is set for binary processing.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] The CBAM-HRNet wheat ear grain segmentation and counting deep learning model based on the convolution attention mechanism is constructed, the image processing algorithm and the wheat ear grain texture feature are used to construct the ear grain counting model, the prediction and counting of the wheat ear grain are realized, compared with the traditional HRNet, PSPNet, DeeplabV3+ segmentation model and U-Net method, the segmentation effect of the wheat ear grain is better, and the robustness is better, the segmentation precision is further improved, and the generalization ability is strong, more rich semantic information is obtained, the problems of small target image segmentation difficulty and training underfitting can be solved, and the ear grain counting model can more quickly and accurately predict the number of wheat grains, and provides algorithm support for efficient and intelligent yield estimation of wheat.

[0027] In view of the problems that the semantic information of the wheat ear image is complex, and the adhesion and coverage between the ear grains are serious, the convolution attention mechanism is added on the basis of the original HRNet model, the efficiency of feature extraction is improved, the weight is prevented from being too random, and the training efficiency is accelerated; the CBAM-HRNet model based on the convolution attention mechanism has a better effect than the compared network model, and it is proved that the model has better robustness.

[0028] The method can be used for estimating the grain number of wheat spikes, improving the estimation efficiency of wheat yield, and providing a rapid and automatic high-throughput wheat spike grain counting system for agricultural workers, thereby improving work efficiency. The method is suitable for the segmentation and counting of wheat spike grains and can also be applied to the segmentation and counting of other plants, and has a wide range of applications. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 It is the overall workflow diagram of the application;

[0030] Figure 2 It is the wheat spike grain segmentation network structure diagram based on CBAM-HRNet;

[0031] Figure 3 It is the wheat spike grain segmentation network structure diagram based on PSPNet;

[0032] Figure 4 It is the wheat spike grain segmentation network structure diagram based on DeeplabV3+;

[0033] Figure 5 It is the wheat spike grain segmentation network structure diagram based on U-Net;

[0034] Figure 6 It is the CBAM-HRNet model training process diagram of the application;

[0035] Figure 7 It is the HRNet training process diagram;

[0036] Figure 8 It is the PSPNet training process diagram;

[0037] Figure 9 It is the DeeplabV3+ training process diagram;

[0038] Figure 10 It is the U-Net training process diagram;

[0039] Figure 11 It is the fourth step spike grain counting model flow chart of the application;

[0040] Figure 12 It is the segmentation effect diagram of different models in the test set of the application;

[0041] Figure 13 It is the counting result analysis diagram of the first method of the application;

[0042] Figure 14 It is the counting result analysis diagram of the second method of the application. DETAILED DESCRIPTION

[0043] The application will be further described below in combination with the drawings and examples.

[0044] Referring to Figures 1-14 The application provides a CBAM-HRNet model wheat ear grain segmentation and counting method based on a convolution attention mechanism, which comprises the following steps:

[0045] Embodiment one

[0046] The embodiment provides a CBAM-HRNet model wheat ear grain segmentation and counting method based on a convolution attention mechanism, which comprises the following steps:

[0047] Step one, data acquisition: multiple different varieties of wheat are selected by an image acquisition device, and in order to increase the complexity and diversity of the data set and improve the generalization level of the model, two image acquisition methods are adopted, one is ex vivo sampling, and the image of the ear is acquired in a laboratory environment; the other is in situ sampling, that is, the image of the ear is acquired in a field environment.

[0048] When the image is acquired, the image acquisition mobile device is parallel to the ear, the object distance is changed by adjusting the vertical height until the ear appears completely in the mobile device lens field of view, and a clear ear image is displayed, and the original ear image is obtained by this method; and in order to exclude the influence of different shooting distances on the number of wheat ear grains, the distance between the mobile device and the ear is basically kept consistent during image acquisition, so as to ensure the consistency of the wheat ear image; 30-40 plants of wheat of each nitrogen treatment and different shooting backgrounds are selected in each bottle during acquisition, and a total of 660 original images are obtained, and an ear information table as shown in Table 1 is constructed.

[0049] Table 1: Ear information table

[0050]

[0051]

[0052] Step two, the ear image collected in step one is preprocessed and an ear data set is constructed: the image preprocessing includes data normalization and data enhancement, specifically,

[0053] (1) Data normalization

[0054] Data normalization is to convert all images to a uniform size, so as to facilitate model training; because the size of the original ear image is too large, the requirement for the device is too high, therefore, in order to balance the operation capacity of the device and the number and quality of the image, it is necessary to normalize the original image, and the image size in the data set is normalized to 480x480 before model training, so as to reduce the model operation amount and reduce the risk of overfitting.

[0055] (2) Data augmentation

[0056] Because the number of images will affect the training of the model, it is easy to cause the test accuracy of the model to decrease, therefore, through data augmentation, the images in the original data set are expanded to solve the problem of insufficient number of images; the data set is enhanced by rotating 90°, 180° and 270°, and horizontal and vertical flipping; at the same time, Gaussian blur is used to reduce image noise and reduce detail levels, and the most appropriate Gaussian blur processing is found by continuously adjusting the size of the Gaussian convolution kernel. Through comparison, the size of the Gaussian convolution kernel is set to 5x5, so as to enhance the image effect of the image under different proportional sizes.

[0057] After processing the image, the data set is constructed, and the embodiment adopts a supervised learning mode, that is, the deep learning model is trained by using the artificially labeled data samples, and then a network model with certain generalization ability is obtained, so as to realize computer vision tasks such as target classification, target detection and image segmentation.

[0058] When the data sample is labeled, the wheat ear and background two types of segmentation objects are manually marked by using the Labelme image labeling tool, and the mask image is converted from the labeling information. The ear image and the mask image constitute the data set required by the deep learning segmentation model; the number of images processed for each of the three varieties in the data set is uniformly distributed, and the ear data set is divided into a training set and a verification set in a ratio of 9:1, wherein the training set has 612 images, the verification set has 62 images, and the test set has 56 images.

[0059] Step three, using a deep learning segmentation network to perform image segmentation on the wheat ear, obtaining a prediction model through training, and calling the prediction model to test the test set and output a prediction result: the segmentation of the wheat ear is a two-classification task in a complex background, the pixel points between the ears are similar, and the image is seriously adhered, therefore, the resolving power of the deep learning model and the global information acquisition ability are required to be high. In order to ensure the segmentation accuracy and the calculation efficiency, the embodiment constructs CBAM-HRNet, HRNet, PSPNet, DeeplabV3+ segmentation model and U-Net based on convolution attention mechanism to segment the wheat ear,

[0060] It should be noted that when training the model, the network starts from 0 due to too random weights, which will result in poor feature extraction effect. Therefore, according to the idea of transfer learning, a freeze-thaw mechanism is constructed to improve the training effect.

[0061] The above model is further introduced as follows,

[0062] 1) about CBAM-HRNet

[0063] CBAM-HRNet achieves strong semantic information and precise location information by parallelizing multiple resolution branches and continuously exchanging information between different branches, avoiding the loss of a large amount of effective information in the continuous up-sampling process.

[0064] The network body of CBAM-HRNet contains four stages and four parallel convolution branches with resolutions of 1 / 4, 1 / 8, 1 / 16 and 1 / 32. The first stage contains 4 bottleneck residual units, each followed by a 3x3 convolution to change the number of feature maps to 32, and the other stages are similar. Each module contains 4 residual units, each providing two 3x3 convolutions for each resolution, followed by a BN layer (Batch Normalization) and a nonlinear activation function ReLU, and a multi-resolution fusion module at the end of each stage.

[0065] Due to different resolutions, low-resolution representations need to be set to reduce resolution when receiving high-resolution information, and similarly high-resolution representations need to be set to increase resolution and channel number through bilinear up-sampling and 1x1 convolution when receiving low-resolution information. For semantic segmentation tasks, the structure of the representation branch is that the low-resolution part is up-sampled by bilinear up-sampling and stacked together, and then fused by 1x1 convolution.

[0066] During the up-sampling process of the representation branch, a convolution attention mechanism (CBAM) is added. CBAM is a combination of channel attention mechanism and spatial attention mechanism, which can achieve better results than attention mechanisms that only focus on channels or spaces.

[0067] CBAM will process the input feature layer through channel attention mechanism and spatial attention mechanism respectively. Channel attention mechanism will perform global average pooling and global maximum pooling on each input feature layer. Then the results of the two are processed using a shared fully connected layer, and the processed results are added together, and then passed through a Sigmoid activation function to obtain the weight (0-1) of each channel of the input feature layer.

[0068] After obtaining this weight, multiply the original input feature layer. The spatial attention mechanism takes the maximum value and the average value on the channel of each feature point of the input feature layer. Then stack the results, adjust the channel number with a convolution with a channel number of 1, and then pass through the Sigmoid activation function to obtain the weight (0-1) of each feature point of the input feature layer.

[0069] After obtaining this weight, multiply the original input feature layer. The input feature map first passes through the channel attention mechanism, and then the normalized weight and the input feature map of the spatial attention mechanism are multiplied to obtain the final feature map.

[0070] Referring to Figure 2 The network structure of CBAM-HRNet, CBAM-HRNet can keep high resolution from head to tail, and the information interaction of different branches can make up for the information loss caused by the reduction of channel number, and can realize the adaptive attention of the network. This network architecture design has significant effect on the position-sensitive semantic segmentation task.

[0071] 2) About PSPNet

[0072] PSPNet is improved on the basis of FCN. The input image is extracted by a feature extraction network. In order to increase the receptive field, the feature extraction network adopts ResNet network with added atrous convolution, and the extracted features are used as the input of the pyramid pooling module.

[0073] A feature pyramid with a depth of 4 is constructed in this module. Different depth features are obtained based on input features through different scale pooling operations, and then a 1x1 convolution layer is used to reduce the feature dimension to 1 / 4 of the original. Finally, these pyramid features are directly upsampled to the same size as the input features, and then merged with the input features to obtain the final output feature map.

[0074] Referring to Figure 3 The network structure diagram of PSPNet, and the above feature merging process is the process of fusing the detailed features (shallow features) and global features (deep features, i.e. context information) of the target.

[0075] 3) DeeplabV3+ segmentation model

[0076] As Figure 4The network structure of the DeeplabV3+ segmentation model is shown, and the DeeplabV3+ segmentation model is an encoder-decoder structure. In the encoder part, it mainly includes Backbone and Atrous Spatial Pyramid Pooling (ASPP) two parts. From the feature map output by Backbone, part of the feature map is output by the last convolution, and the other part is the feature map of the intermediate low-level feature. The ASPP module accepts the first part output of Backbone as input, uses four different dilated rate convolution blocks and a global average pooling block to obtain a total of five groups of feature maps, fuses them, and then passes them through a 1x1 convolution block. Finally, it is sent to the Decoder module.

[0077] In the decoder part, the low-level feature map from the intermediate layer of Backbone and the output from the ASPP module are received as input. First, the low-level feature map is reduced in channel dimension using 1x1 convolution; then, the feature map from the ASPP module is linearly interpolated and upsampled to obtain a feature map with the same size as the low-level feature map; then, the channel-reduced low-level feature map and the linearly interpolated and upsampled feature map are spliced and sent to a group of 3x3 convolution blocks for processing; finally, linear interpolation and up-sampling are performed again to obtain a prediction map with the same resolution size as the original image.

[0078] 4) U-Net

[0079] The network structure of U-Net is shown as Figure 5 ; U-Net is also an encoder-decoder structure. In the encoder part, convolution modules are used for feature extraction; each convolution module includes two convolution layers, followed by a BN layer. After each convolution module, a pooling layer is used to reduce the dimension of the obtained features, and the size is reduced to half of the previous convolution module while maintaining the number of channels. After passing through 5 convolution modules, the high-level feature vector of the input image is obtained, which is transmitted to the decoder part.

[0080] In the decoder part, the combination of up-sampling and convolution is used to realize deconvolution and increase the resolution of the feature image. The extracted features are decoded through 4 decoding modules. The low-level features obtained by the encoder are fused to ensure the full use of low-level features and improve the segmentation effect of the network on small targets. After the features are restored to the size of the input image, the feature map with the same size as the input image is sent to the softmax layer to obtain the probability of each pixel belonging to each class. The probability threshold of each class is determined, and if it is higher than the threshold, it is considered that the pixel belongs to the specified class. Finally, the image segmentation result is obtained.

[0081] The same wheat ear training set is trained by the CBAM-HRNet model based on the convolution attention mechanism, the HRNet, the PSPNet, the DeeplabV3+ segmentation model and the U-Net, and the mIoU and loss value comparison effects in the training process are as shown in Figures 6-10 The results show that

[0082] In the process of continuous iteration, the mIoU values of the four models are in a steady upward trend, and gradually tend to be stable convergence with the increase of the number of iterations. The mIoU value of the CBAM-HRNet model based on the convolution attention mechanism is basically stable at about 0.85, and the segmentation effect of the model on the wheat ear data set is good. And the loss value of the model in the training set and the validation set decreases rapidly and gradually converges to about 0.021, and the network converges quickly, not only without sudden increase of error, but also the error change is very gentle. The error gap between the two data sets is very small, which shows that in the process of gradient calculation, the model can quickly and accurately find the appropriate gradient direction, and the performance is stable, and the learning effect is good. The change trend of the loss value of the training set and the validation set is basically the same, which shows that the model has good generalization ability.

[0083] Step four, a spike grain counting model is constructed by combining the prediction results output by the above step three with image processing technology to realize accurate prediction and counting of wheat spike grains: see Figure 11 After the wheat ear samples selected from each variety are predicted by the deep learning segmentation model, some of the grains in each ear will be bonded, and the overlapping and bonded parts need to be removed by image processing method; including the following steps:

[0084] S401, read the prediction results by OpenCV and NumPy and perform grayscale processing on the prediction results, and convert the color space from RGB to GRAY. After processing, binarization can be performed with 120 as the threshold, and pixel points greater than the threshold are 255 (i.e. white), and pixel points less than the threshold are 0 (i.e. black);

[0085] S402, remove the noise points in the binarized image by erosion transformation, and calculate the distance from the pixel points in the image to the nearest zero pixel point, and obtain the skeleton of the contour after distance transformation; effectively remove the overlapping parts by binarization;

[0086] S403, normalize the image after the above binarization processing to convert the dimensional expression into a dimensionless expression, become a scalar, and obtain the gray value of the image between 0 and 1.0 after normalization;

[0087] S404, process the gray image into a binarized image by binarization and opening operation;

[0088] S405, contour extraction is performed according to the boundary points provided by the binary image, and the extracted contour is the number of grains on one side of the wheat ear;

[0089] According to the geometric and texture characteristics of the wheat, the grains of the wheat ear generally exist in pairs on both sides of the ear axis, so two methods can be used to count the grains; method one is that the number of grains on one side of the wheat ear is doubled to be regarded as the total number of grains; method two is that the sum of the number of grains on both sides is the total number of grains; therefore, the number of grains of the wheat ear can be easily counted according to the contour of the number of grains on one side of the wheat ear extracted by the above two methods. The grain counting model can quickly and accurately predict the number of grains of the wheat, and provide algorithm support for efficient and intelligent yield estimation of the wheat.

[0090] Embodiment two:

[0091] On the basis of embodiment one, the embodiment introduces the segmentation precision evaluation of the segmentation model in embodiment one:

[0092] The segmentation model mainly uses accuracy, recall rate, class average pixel accuracy and average intersection over union to evaluate the segmentation precision. The evaluation indexes are calculated from the parameters in the confusion matrix. In the model precision evaluation, the confusion matrix is mainly used to compare the predicted value and the true value, which is calculated by comparing the position of each real pixel and the position of the predicted pixel, specifically,

[0093] The accuracy refers to the proportion of the number of samples with the predicted value being the true value in the total number of samples, and the formula is as follows:

[0094]

[0095] The recall rate refers to the proportion of the number of samples with the predicted value being the true value in the total number of positive examples, and the formula is as follows:

[0096]

[0097] The intersection over union is a standard measure for evaluating the precision of semantic segmentation, and the formula is as follows:

[0098]

[0099] The class average pixel accuracy refers to the proportion of the number of pixels correctly classified in each class, and the formula is as follows:

[0100]

[0101] The average intersection over union refers to the average value of all classes IoU, and the formula is as follows:

[0102]

[0103] In the formula, TP is a positive example of correct classification; TN is a negative example of correct classification; FP is a negative example of misclassification as a positive example; FN is a positive example of misclassification as a negative example; k is the total number of partitions; P k Pixel accuracy of each class.

[0104] The artificial counting of the number of spike grains of each sample is taken as the true value, and the number of spike grains of each sample obtained by the image segmentation algorithm and the spike grain counting model is taken as the predicted value. The indicators for quantifying the accuracy of the counting model include root mean square error, mean absolute error, mean relative error, and determination coefficient.

[0105] Example Three

[0106] This embodiment introduces the freeze-thaw mechanism described in Example One:

[0107] The pre-trained weights of the model are universal for different data sets, because the features extracted by the neural network backbone feature extraction part are universal. The pre-trained weights must be used in most cases, otherwise the weights of the backbone part are too random, the feature extraction effect is not obvious, and the network training result is not good. Freezing and training can speed up the training efficiency and prevent the weights from being damaged.

[0108] In the freezing stage, the backbone of the model is frozen, the feature extraction network does not change, the occupied video memory is small, only the network is fine-tuned, and the training needs of different machine performances are met. In the thawing stage, the backbone of the model is not frozen, the feature extraction network will change, the occupied video memory is larger, and all the parameters of the network will change.

[0109] The training parameters of the freezing stage include the current starting training generation of the model (Init_Epoch=0), the iteration number of the model freezing training (Freeze_Epoch=50), and the batch size of the model freezing training (Freeze_batch_size=16);

[0110] The training parameters of the thawing stage include the total iteration number of the model training (UnFreeze_Epoch=300) and the batch size of the model after thawing (Unfreeze_batch_size=8).

[0111] Example Four

[0112] In order to measure the effectiveness of the model described in the application, different backbone networks, optimizers and learning rates are used according to different segmentation models, and a variety of evaluation indicators are used to compare the results and performance of model training, as shown in the following table:

[0113]

[0114]

[0115] According to the results of the above table, it can be obtained that the CBAM-HRNet model based on the convolution attention mechanism with hrnetv2_w32 as the backbone network and Adam as the optimizer achieves the best segmentation accuracy (mIoU = 0.8521), the HRNet model with hrnetv2_w32 as the backbone network and Adam as the optimizer is second (mIoU = 0.851), and the PSPNet segmentation accuracy with MobileNetv2 as the backbone network and SGD as the optimizer is the lowest (mIoU = 0.7718); the training results of the four models using the Adam optimizer are better than those using the SGD optimizer, because Adam can adaptively learn the learning rate by combining the first-order momentum and the second-order momentum, solving the problem of slow descent speed and easy to obtain local optimal solution of the SGD optimizer.

[0116] Example Five

[0117] On the basis of example four, under the condition that the optimal backbone network and the optimizer are Adam, the segmentation ability of the CBAM-HRNet model, the HRNet model, the U-Net model, the PSPNet model and the DeeplabV3+ segmentation model based on the convolution attention mechanism is compared and analyzed using the wheat ear test set, and the segmentation effect is as shown in Figure 12

[0118] It can be found from the figure that for the input wheat ear image, the segmentation effect of the PSPNet and DeeplabV3+ segmentation models is not very ideal, although the wheat ear grains and the background can be divided, but the ear grains are seriously adhered, because the gray scale features between the ear grains are similar, and the pixel point values are close, which are easy to be divided as a whole, while the gray scale value of the background is far away from the wheat ear, which is easy to be divided out.

[0119] While the segmentation effect of the HRNet and U-Net is better, but some details are lost in complex environment. In contrast, the CBAM-HRNet model based on the convolution attention mechanism with hrnetv2_w32 as the backbone network and Adam as the optimizer has strong segmentation ability for the wheat ear image, is not easily affected by other noise, and can accurately segment the wheat ear grains and calculate the number of wheat ear grains based on this.

[0120] Example Six

[0121] ​In order to further verify the accuracy of the method of the present application, 30 sample images are selected for each wheat variety, and the CBAM-HRNet model based on convolution attention mechanism and image processing algorithm are used to segment and count the grains of the wheat sample images, and the grain number of each ear is calculated according to the two counting methods described in Embodiment 1, and compared with the artificial counting; as shown in Figs. 1 and 2, which are the counting results of the ear grain counting model under the two methods, and the following table shows the counting accuracy evaluation table of the wheat ear grains under the two methods: Figure 13 and Figure 14 As shown in Figs. 1 and 2, which are the counting results of the ear grain counting model under the two methods, and the following table shows the counting accuracy evaluation table of the wheat ear grains under the two methods:

[0122]

[0123] The above results show that the method of the present application has higher counting accuracy for wheat ear grains than the traditional image processing algorithm, and the average absolute error and the average relative error are lower, and the fitting degree of the predicted value and the true value is better. Therefore, using the method to process different varieties of wheat ears can greatly improve the counting accuracy of the ear grain number, and realize the automatic counting of the ear grain with high precision.

[0124] The above description is only a preferred embodiment of the present application, and does not limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A CBAM-HRNet model wheat ear grain segmentation and counting method based on a convolution attention mechanism, characterized in that: The method comprises the following steps: Step one, data collection: select multiple different varieties of wheat, and obtain multiple original wheat ear images of each type to create a wheat ear information table based on different varieties; Step two, pre-processing of the wheat ear images in step one: including data normalization processing and data enhancement to form a wheat ear dataset; Step three, using a deep learning segmentation network to segment the wheat ear image, obtaining a prediction model through training, and testing the test set by calling the prediction model to output the prediction result: constructing CBAM-HRNet, HRNet, PSPNet, DeeplabV3+ segmentation model and U-Net based on convolution attention mechanism to segment the wheat ear and compare the results; the network main body of the CBAM-HRNet includes four stages and four parallel convolution branches, and the resolutions are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 respectively; the first stage includes 4 bottleneck layer residual units, and each unit is followed by a 3x3 convolution to change the number of feature maps to 32, and the other stages are the same; each module includes 4 residual units, each unit provides two 3x3 convolutions for each resolution, followed by a BN layer and a nonlinear activation function ReLU, and there is a multi-resolution fusion module at the end of each stage; The CBAM-HRNet achieves the purpose of strong semantic information and accurate position information by parallelizing multiple resolution branches and continuously exchanging information between different branches, avoiding the loss of a large amount of effective information in the continuous upsampling process; the convolution attention mechanism CBAM is added to realize the upsampling process of the representation branch; Step four, constructing a spike grain counting model by combining the prediction results output by step three with image processing technology to realize accurate prediction and counting of wheat spike grains: after the wheat ear samples of each variety are predicted by the deep learning segmentation model, some of the grains will be stuck together, and image processing methods are needed to remove the overlapping and sticking parts, including the following steps: S401, gray processing of the prediction results in step three, color space conversion from RGB to GRAY; S402, threshold binarization processing of the image to remove the overlapping part; the binarized image needs to be eroded to eliminate noise points, and the distance from the pixel points in the image to the nearest zero pixel point is calculated, and the skeleton of the contour is obtained after distance transformation; S403, using normalization to convert the dimensional expression into a dimensionless expression, which becomes a scalar, and the gray value of the image obtained by normalization is between 0 and 1.0; S404, process the gray image into a binary image through binarization and opening operation; S405, according to the boundary points provided by the above binary image, draw the shape to extract the contour, and the extracted contour is the number of spike grains on one side of the wheat.

2. The CBAM-HRNet model based on a convolution attention mechanism for wheat ear grain segmentation and counting method according to claim 1, characterized in that: When collecting the wheat ear image in step one, the image acquisition device is parallel to the wheat ear, the object distance is changed by adjusting the vertical height, and the wheat ear is completely displayed in the field of view of the mobile device lens, showing a clear ear image.

3. The CBAM-HRNet model based on a convolution attention mechanism for wheat ear grain segmentation and counting method according to claim 1, characterized in that: The ear data table created in step one includes nitrogen treatment, shooting background, shooting date, weather, resolution, image size, shooting device, focal length, and image quantity information of the ear.

4. The CBAM-HRNet model based on a convolution attention mechanism for wheat ear grain segmentation and counting method according to claim 1, characterized in that: In step two, the image size is normalized to 480x80 to reduce the model operation amount and the risk of overfitting.

5. The CBAM-HRNet model based on a convolution attention mechanism for wheat ear grain segmentation and counting method according to claim 1, characterized in that: In step two, data augmentation is used to expand the images in the original data set to solve the problem of insufficient image data; at the same time, Gaussian blur is used to reduce image noise and reduce the level of detail, thereby enhancing the image effect at different scales.

6. The CBAM-HRNet model based on a convolution attention mechanism for wheat ear grain segmentation and counting method according to claim 1, characterized in that: In step two, the wheat ear grains and background are manually marked as two types of segmentation objects by using the Labelme image labeling tool, and the marking information is converted into a mask image; the ear image and its mask image together constitute the data set required by the deep learning segmentation model; the number of images of each different treatment in the data set is evenly distributed, and the ear data set is divided into training set and validation set according to the proportion.

7. The CBAM-HRNet model based on a convolution attention mechanism for wheat ear grain segmentation and counting method according to claim 1, characterized in that: In step four, the prediction results are read in by OpenCV and NumPy, and the prediction results are subjected to grayscale processing; a threshold of 120 is set for binary processing.

8. The CBAM-HRNet model based on a convolution attention mechanism for wheat ear grain segmentation and counting method according to claim 1, characterized in that: The total number of grains is counted in two ways: one is that the total number of grains is twice the number of grains on one side of the ear; the other is that the total number of grains is the sum of the number of grains on both sides of the ear.

Images