A machine vision-based method and system for corn classification

Abstract

The application discloses a corn grading and classifying method and system based on machine vision, comprising the following steps: collecting photos of defective kernels in the corn storage process, performing image enhancement, manual annotation and image augmentation on the photos, and generating a training set; constructing a corn grading neural network model and training the network model using the training set; collecting real-time photos in the corn storage process using an imaging device, performing image enhancement on the real-time photos; inputting the image-enhanced real-time photos into the trained network model to obtain the category and position of the defective kernels; calculating the weight proportion of each category of defective kernels in a batch of corn images according to the category and position of the defective kernels, and combining the bulk density and moisture content of the corn to grade a batch of stored corn. The improved YOLO network model can accurately locate the target in a high-density scene, and the recognition ability of imperfect kernels, moldy kernels and impurities in the corn kernels is obviously improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine vision, and particularly relates to a corn grading and classifying method and system based on machine vision. BACKGROUND

[0002] China is a large agricultural country, and corn is one of the most important food crops in China, closely related to food security and food production, and has a very important strategic position. The national standard GB 1353-2018 Corn is the general standard for bulk corn in China, and is widely applicable to the purchase, storage, transportation, processing and sales of commodity corn. The national standard for corn takes the bulk density, imperfect kernel content, moldy kernel content, impurity content, moisture content and color and odor as the main indicators for measuring the quality of corn, which are widely representative and authoritative. Among them, the measurement of the bulk density of corn is determined according to GB / T 5498-2013 Determination of bulk density for grain and oil detection using the specified bulk density container; the measurement of the moisture content is determined according to the provisions of GB 5009.3-2016 National food safety standard Determination of moisture in food using the direct drying method.

[0003] For the detection of imperfect kernel content, moldy kernel content and impurity content, first, the large sample impurities are screened out by an electric or manual screen, and then the small sample impurities, moldy kernels and imperfect kernels are screened out by manual screening. Impurities are divided into three types according to their size: large impurities (generally referring to impurities remaining above a 14mm diameter circular screen hole), shoulder impurities (remaining above a 3mm diameter circular screen hole after passing through a 14mm diameter circular screen hole), and small impurities (screened under a 3mm diameter circular screen hole). Among them, shoulder impurities, moldy kernels and imperfect kernels cannot be distinguished from ordinary corn kernels by screen holes due to their similar size, and must be picked by hand. Manual picking has the problems of low picking efficiency and insufficient accuracy.

[0004] Machine vision technology began to develop gradually after achieving good results in two applications of biomedical images and remote sensing image processing in the early 1970s. With the rapid development of computer technology, the cost performance and processing speed of computers are continuously improved, laying a solid foundation for the research and application of machine vision related technologies. Compared with manual detection technology, machine vision detection technology has high detection accuracy, fast speed, large amount of information and good repeatability, and has a very broad development and application prospect in the field of corn grading detection. SUMMARY

[0005] The application provides a corn grading and classifying method based on machine vision, which aims to solve the problems of low picking efficiency and insufficient accuracy in the process of manually picking corn in the prior art.

[0006] To solve the above technical problems, the present application provides a corn grading and classification method based on machine vision, comprising the following steps:

[0007] S1: Collecting photos of defective kernels in the corn storage process, image enhancement is performed on the photos, the defective kernels in the enhanced photos are manually labeled, and the labeled photos are image augmented to generate a training set.

[0008] S2: Constructing a corn grading and classification neural network model, training the neural network model using the training set, and using an improved YOLO network model.

[0009] S3: Using an imaging device to collect real-time photos in the corn storage process, and performing image enhancement on the real-time photos.

[0010] S4: Inputting the image-enhanced real-time photos into the trained neural network model to obtain the category and position of the defective kernels.

[0011] S5: Calculating the weight proportion of each category of defective kernels in a batch of corn images according to the category and position of the defective kernels, and combining the bulk density and moisture content parameters of the corn to grade a batch of stored corn.

[0012] The improved YOLO network model uses a Transformer encoder module to replace part of the convolution blocks and CSP network of the detection head module in the conventional network model, and uses a convolution block attention module to replace part of the convolution blocks in the conventional network model.

[0013] Preferably, the image enhancement includes median filtering and Mosaic enhancement, the Mosaic enhancement randomly uses 4 pictures, randomly scales, and randomly distributes for splicing.

[0014] Preferably, the training set includes a test set and a validation set divided according to a set proportion.

[0015] Preferably, the defective kernels include imperfect kernels, moldy kernels and impurities.

[0016] Preferably, the improved YOLO network model includes a Backbone layer, an intermediate layer and a detection layer. The intermediate layer uses a convolution block attention module to replace part of the convolution blocks in the conventional network model, and the detection layer uses a Transformer encoder module to replace part of the convolution blocks and CSP network of the detection head module in the conventional network model.

[0017] Preferably, the convolution block attention module further comprises a channel attention submodule and a spatial attention submodule, the input feature map is input into the channel attention submodule as a first original picture, a channel attention output result is obtained, the channel attention output result is multiplied by the first original picture to obtain a second original picture, and the second original picture is input into the spatial attention submodule to obtain a spatial attention output result, and the spatial attention output result is multiplied by the second original picture to obtain an output result of the convolution block attention module.

[0018] Preferably, the channel attention submodule inputs the input feature map into two parallel maximum pooling layers and average pooling layers, changes the size of the feature map from CxHxW to Cx1x1, then inputs the feature map into a multi-layer perception module, and respectively inputs the output of the multi-layer perception module into a ReLU activation function to obtain two activated results, element-wise adds the two activated results, inputs the added result into a sigmoid activation function to obtain an output result, and multiplies the output result by the first original picture to obtain a channel attention output result with a size of CxHxW.

[0019] Preferably, the spatial attention submodule inputs the channel attention output result into a maximum pooling layer and an average pooling layer to obtain two 1xHxW feature maps, inputs the two feature maps into a Concat operation to splice the two feature maps, inputs the spliced result into a 7x7 convolution to change the spliced result into a 1-channel feature map, inputs the 1-channel feature map into a sigmoid activation function to obtain a spatial attention feature map, and multiplies the spatial attention feature map by the second original picture to obtain a spatial attention output result with a size of CxHxW.

[0020] Preferably, the Transformer encoder module comprises two sublayers arranged in sequence, the first sublayer is a multi-head attention layer, and the second sublayer is a fully connected layer of a multi-layer perception.

[0021] Correspondingly, the application further provides a corn grading and classifying system based on machine vision, comprising:

[0022] An imaging module is configured to acquire real-time photos in a corn storage process and perform image enhancement on the real-time photos.

[0023] An improved YOLO network model is configured to detect defective kernels in the real-time photos in the corn storage process, wherein the defective kernels include imperfect kernels, moldy kernels and impurities.

[0024] A grading calculation module is configured to grade a batch of stored corn according to a detection result of the improved YOLO network model and in combination with a bulk density and a moisture content of the corn.

[0025] The imaging module, the improved YOLO network model and the hierarchical calculation module are configured to perform the machine vision-based corn grading and classification method according to any one of claims 1-9.

[0026] Compared with the prior art, the present application has the following technical effects:

[0027] 1. The machine vision-based corn grading and classification method adopts an improved YOLO network model, uses a Transformer encoder module to replace some convolution blocks and a CSP network (Cross Stage Partial) in a detection head part of a conventional network model, so that the improved YOLO network model can capture global information and rich context information, accurately locate targets in a high-density scene, and enhance the recognition ability of the neural network model for small targets such as imperfect grains, moldy grains and impurities in the corn grains.

[0028] 2. The machine vision-based corn grading and classification method adopts an improved YOLO network model, uses a CBAM (Convolutional Block Attention Module) to replace part of the convolution blocks in the conventional network model, so that the entire neural network model can cover more parts of the object to be identified, making it easier for the model to find the region of interest in an image with a large area coverage, and the probability of finally identifying the feature is also higher, effectively reducing the missed detection rate of the model for defective grains in the corn image.

[0029] 3. The machine vision-based corn grading and classification method performs image augmentation on the samples in the training set to meet the demand for large data volume in the neural network model training process, changes the original image randomly to reduce the dependence of the model on certain features, thereby improving the generalization ability and robustness of the model, and avoiding overfitting in the training process.

[0030] 4. The machine vision-based corn grading and classification method performs Mosaic image enhancement on the pictures to be processed, which can effectively enrich the detection data set, randomly scale a large number of small targets to enhance the robustness of the network model, and calculate the data of four pictures to reduce the size of each batch while maintaining the detection effect, which helps to improve the training speed of the model and the accuracy of the network in the training process, and reduces the demand for GPU and video memory resources in the detection process. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart of the machine vision-based corn grading and classification method according to the present application;

[0032] Figure 2 is an improved YOLO network model structure diagram of a corn grading and classifying method based on machine vision according to the present application;

[0033] Figure 3 is a convolution block attention module structure diagram of the improved YOLO network model of the corn grading and classifying method based on machine vision according to the present application;

[0034] Figure 4 is a Transformer encoder structure diagram of the improved YOLO network model of the corn grading and classifying method based on machine vision according to the present application.

[0035] The figure mark: 1, Backbone layer; 2, middle layer; 3, detection layer. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the present application will be described clearly and completely below by combining specific embodiments of the present application and referring to the drawings.

[0037] A corn grading and classifying method based on machine vision, as shown in Figure 1 is a flow chart of the corn grading and classifying method based on machine vision according to the present application, including the following steps:

[0038] S1: Collecting photos of defective kernels in the corn storage process, image enhancement is performed on the photos, manual labeling is performed on the defective kernels in the enhanced photos, image augmentation is performed on the labeled photos, and a training set is generated. For all photos in the training set, the present embodiment divides them into a test set and a validation set in a ratio of 8:2, the test set is used to input the neural network model for training of the model, and the validation set is used to verify the output result of the training. The defective kernels of corn specifically include imperfect kernels, moldy kernels and impurities.

[0039] The imperfect grain refers to the corn grain with defects or damage but still has use value, including worm-eaten grain, diseased spot grain, broken grain, sprouting grain, moldy grain and heat-damaged grain. The worm-eaten grain is the corn grain which is damaged by worms and forms holes or tunnels; the diseased spot grain is the corn grain with diseased spots on the grain surface, which damages the embryo or endosperm; the broken grain is the corn grain with a broken kernel of more than one fifth (including) of the volume of the grain; the sprouting grain is the corn grain with the sprout or root breaking through the epidermis, or the corn grain with the sprout or root not breaking through the epidermis but the epidermis of the embryo part being broken or obviously raised, which has sprouting traces; the moldy grain is the corn grain with mold on the grain surface; the heat-damaged grain is the corn grain which is significantly discolored or damaged due to heat after heating or drying, including natural heat-damaged grain and drying heat-damaged grain; the natural heat-damaged grain is the corn grain which is significantly discolored due to excessive respiration during storage, and the drying heat-damaged grain is the corn grain which is significantly discolored, deformed or swelled due to the epidermis or embryo or endosperm being significantly discolored during heating and drying.

[0040] The moldy grain refers to the corn grain with obvious mold on the grain surface and damaging the embryo and endosperm, which has no edible value.

[0041] The impurities refer to other substances in addition to the corn grain and the corn grain with no use value, including undersize, inorganic impurities and organic impurities.

[0042] The content of each type of defective grain is an important indicator for corn grading and classification, so the specific content of each type of defective grain in the batch of corn grains must be counted, and the specific content of each type of defective grain is taken as a specific indicator for corn grading and classification.

[0043] S2: constructing a corn grading and classification neural network model, training the neural network model using the training set, and using an improved YOLO network model.

[0044] S3: using an imaging device to collect real-time photos in the corn storage process, and performing image enhancement on the real-time photos.

[0045] S4: inputting the real-time photos after image enhancement into the trained neural network model to obtain the category and position of the defective grain.

[0046] S5: calculating the weight proportion of each type of defective grain in a batch of corn images according to the category and position of the defective grain, and combining the bulk density and moisture content parameters of the corn to grade a batch of stored corn.

[0047] The corn grading standards are executed according to the provisions of the national standard GB 1353-2018 Corn, and the bulk density, imperfect kernel content, impurity content, moisture content and color and odor of corn are used as specific indexes for corn grading and classification. Among them, the measurement of the bulk density of corn is determined according to GB / T 5498-2013 Determination of bulk density for grain and oil detection using the specified bulk density meter; the measurement of the moisture content is determined according to the provisions of GB 5009.3-2016 National food safety standard Determination of moisture in food, using the direct drying method to determine the moisture content of corn. The corn grading and classification method provided in the example is based on machine vision, and the content of imperfect kernels, moldy kernels and impurities in the corn grading and classification index is determined, combined with the corn bulk density and moisture content measured by other methods, and according to the classification standard of Table 1, each batch of corn is graded.

[0048] Table 1 is the national standard corn quality index classification standard:

[0049]

[0050] Table 1

[0051] In step S1, the image enhancement includes median filtering, Mosaic enhancement, the Mosaic enhancement randomly uses 4 pictures, randomly scales, and randomly distributes for splicing. The Mosaic enhancement can effectively enrich the detection data set, the random scaling increases a lot of small targets, and enhances the robustness of the network model; by calculating the data of 4 pictures, the size of each batch can be reduced under the premise of maintaining the detection effect, which is helpful to improve the training speed of the model and the accuracy of the network during the training process of the network model, and reduce the demand for GPU and video memory resources during the detection process. The image augmentation technology refers to generating some training samples similar to but different from the original image by a series of random changes to the original image, thereby expanding the size of the training set. Another role of image augmentation is to change the original image randomly to reduce the dependence of the model on certain features, thereby improving the generalization ability of the model.

[0052] The application of this technology is mainly based on the urgent demand for a large amount of data set for the construction of a neural network model. In order to improve the generalization ability and robustness of the model and avoid overfitting, it is usually required to input sufficient data amount, but it is difficult to obtain effective corn images containing defective kernels at present, and at this time the image augmentation technology can be used to transform and expand the data.

[0053] The image augmentation method includes: image transposition, rotating the original image clockwise or counterclockwise according to a random angle to change the orientation of the image; horizontal mirror image, flipping the original image along the horizontal or vertical direction; image rotation, rotating a certain angle around a point of the image to form a new image; image translation, moving the original image along the x-axis or y-axis direction, and the translation range and translation step can be random or artificially defined to change the position of the image content; scaling transformation, enlarging or reducing the image according to a certain proportion; cropping, randomly selecting a part of the image, then cropping out the part of the image, and then adjusting to the size of the original image; scale transformation, enlarging or reducing the original image according to a specified scale factor; or using a specified scale factor to change the size or blur degree of the image content; contrast transformation, changing the saturation of the image to increase the light change; noise disturbance, overfitting usually occurs when the neural network learns high-frequency features, in order to optimize the image, noise data needs to be randomly added to eliminate high-frequency features, generally using the method of randomly disturbing the RGB value of each pixel in the image. Through image augmentation, the training set meets the demand for large data volume in the neural network model training process, the original image is randomly changed to reduce the dependence of the model on certain features, thereby improving the generalization ability and robustness of the model, and avoiding overfitting in the training process.

[0054] For the improved YOLO network model described in step S2, the embodiment selects a YOLOv5 neural network model, including Backbone layer 1, middle layer 2 and detection layer 3, and improves it: the middle layer 2 uses a convolution block attention module (CBAM, Convolutional Block Attention Module) to replace part of the convolution blocks in the conventional network model, and the detection layer 3 uses a Transformer encoder module to replace part of the convolution blocks and the CSP network of the detection head module in the conventional network model; compared with the conventional YOLO network model, an additional detection head is used for the detection of small objects to improve the detection ability of the improved YOLO network model for small corn defect kernels. Combined with the other three detection heads, the structure of the four detection heads can alleviate the negative effects caused by the dramatic change of target size. As shown in Figure 2 The added detection head (TPH1) is generated from a high-resolution feature map, and the high-resolution characteristics of the feature map make TPH1 more sensitive to small objects. The above improvement makes the improved YOLO network model can capture global information and rich context information, can accurately locate the target in a high-density scene, and enhances the recognition ability of the neural network model for small targets such as imperfect kernels, moldy kernels and impurities in the corn kernels.

[0055] As Figure 3 shown, it is a convolution block attention module structure diagram of an improved YOLO network model of a corn grading and classification method based on machine vision according to the present application, the convolution block attention module further comprises a channel attention (Channel Attention) submodule and a spatial attention (Spatial Attention) submodule, the input feature map is taken as a first original picture and passes through the channel attention submodule to obtain a channel attention output result, the channel attention output result is multiplied by the first original picture to obtain a second original picture. The second original picture passes through the spatial attention submodule to obtain a spatial attention output result, the spatial attention output result is multiplied by the second original picture to obtain an output result of the convolution block attention module. The use of the convolution block attention module enables the entire neural network model to cover more parts of the object to be recognized, so that the model can more easily find the region of interest in an image with a large area coverage, and the probability of finally discriminating the feature is also higher, effectively reducing the missed detection rate of the model for defective kernels in the corn image.

[0056] The channel attention submodule passes the input feature map (Feature Map) through two parallel maximum pooling (MaxPool) layers and average pooling (AvgPool) layers, changes the size of the feature map from CxHxW to Cx1x1, then passes through a multilayer perceptron (MLP) module, and then respectively passes through a ReLU activation function to obtain two activated results. The two activated results are added element by element, and then an output result is obtained through a sigmoid activation function, and then the output result is multiplied by the first original picture to generate a channel attention output result with a size of CxHxW.

[0057] The spatial attention submodule, as a supplement to the channel attention submodule, passes the channel attention output result through maximum pooling and average pooling to obtain two 1xHxW feature maps, then performs a Concat operation to splice the two feature maps, changes the feature map to 1 channel through a 7x7 convolution, and then obtains a spatial attention feature map through a sigmoid activation function. Finally, the spatial attention feature map is multiplied by the second original picture to generate a spatial attention output result with a size of CxHxW.

[0058] As Figure 4 shown, it is a Transformer encoder structure diagram of an improved YOLO network model of a corn grading and classification method based on machine vision according to the present application, the Transformer encoder module comprises two sublayers arranged in sequence, the first sublayer is a multi-head attention layer, and the second sublayer is a fully connected layer of a multilayer perceptron. Residual connection is used between the two sublayers.

[0059] Accordingly, the application also provides a corn grading and sorting system based on machine vision, comprising:

[0060] An imaging module is configured to collect real-time photos during corn storage and perform image enhancement on the real-time photos.

[0061] An improved YOLO network model is configured to detect defective kernels in the real-time photos, wherein the defective kernels include imperfect kernels, moldy kernels and impurities.

[0062] A grading calculation module is configured to grade a batch of corns in storage according to the detection results of the improved YOLO network model and in combination with the bulk density and moisture content parameters of the corns.

[0063] The imaging module, the improved YOLO network model and the grading calculation module are configured to perform the corn grading and sorting method based on machine vision as claimed in any one of claims 1-9.

[0064] The above description is only preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, without departing from the inventive concept, a number of modifications and improvements can be made, which are all within the scope of the present application.

Claims

1. A machine vision-based method of classifying and sorting corn, characterized by, The method comprises the following steps: S1: collecting photos of defective kernels in the corn storage process, performing image enhancement on the photos, manually labeling the defective kernels in the enhanced photos, and generating a training set by image augmentation on the labeled photos; S2: constructing a corn grading and classification neural network model, training the neural network model using the training set, and using an improved YOLO network model; S3: using an imaging device to collect real-time photos in the corn storage process, and performing image enhancement on the real-time photos; S4: inputting the image-enhanced real-time photos into the trained neural network model to obtain the category and position of the defective kernels; S5: calculating the weight proportion of each category of defective kernels in a batch of corn images according to the category and position of the defective kernels, and combining the bulk density and moisture content parameters of the corn to grade a batch of stored corn. The improved YOLO network model uses a Transformer encoder module to replace part of the convolution blocks and CSP networks of the detection head module in the conventional network model, and uses a convolution block attention module to replace part of the convolution blocks in the conventional network model.

2. The method for corn classification based on machine vision according to claim 1, characterized in that, The image enhancement includes median filtering and Mosaic enhancement, which randomly uses four images, randomly scales, and randomly distributes for splicing.

3. The method of claim 1, wherein the method further comprises: The training set includes a test set and a validation set divided according to a set proportion.

4. The method of claim 1, wherein, The defective kernels include imperfect kernels, moldy kernels, and impurities.

5. The machine vision-based method for classifying and sorting corn according to claim 1, wherein, The improved YOLO network model includes a Backbone layer, an intermediate layer, and a detection layer; the intermediate layer uses a convolution block attention module to replace part of the convolution blocks in the conventional network model, and the detection layer uses a Transformer encoder module to replace part of the convolution blocks and CSP networks of the detection head module in the conventional network model.

6. The machine vision-based method for classifying and sorting corn according to claim 5, wherein, The convolution block attention module further includes a channel attention submodule and a spatial attention submodule; the input feature map is taken as a first original image through the channel attention submodule to obtain a channel attention output result, the channel attention output result is multiplied by the first original image to obtain a second original image; the second original image is input into the spatial attention submodule to obtain a spatial attention output result, and the spatial attention output result is multiplied by the second original image to obtain an output result of the convolution block attention module.

7. The machine vision-based method of classifying corn according to claim 6, wherein, The channel attention submodule inputs the feature map through two parallel maximum pooling layers and average pooling layers, changes the size of the feature map from CxHxW to Cx1x1, then passes through a multilayer perceptron module, and then respectively passes through a ReLU activation function to obtain two activated results; the two activated results are added element by element, then an output result is obtained through a sigmoid activation function, and then the output result is multiplied by the first original image to generate a channel attention output result with a size of CxHxW.

8. The machine vision-based method of classifying corn according to claim 6, wherein, The spatial attention submodule obtains two 1xHxW feature maps through max-pooling and average-pooling on the channel attention output result, then splices the two feature maps through a Concat operation, converts the two feature maps into a 1-channel feature map through a 7x7 convolution, and then obtains a spatial attention feature map through a sigmoid activation function, and finally multiplies the spatial attention feature map with the second original image to generate a spatial attention output result with a size of CxHxW.

9. The machine vision-based method for classifying and sorting corn according to claim 5, wherein, The Transformer encoder module comprises two sub-layers arranged in sequence, the first sub-layer is a multi-head attention layer, and the second sub-layer is a fully connected layer of a multi-layer perception machine; residual connection is used between the two sub-layers.

10. A machine vision-based corn classification system, comprising: The method comprises the following steps: an imaging module is configured to collect real-time photos in the corn storage process and perform image enhancement on the real-time photos; an improved YOLO network model is configured to detect defective kernels in the real-time photos in the corn storage process, wherein the defective kernels include imperfect kernels, moldy kernels and impurities; a hierarchical calculation module is configured to perform hierarchical classification on a batch of stored corn according to the detection results of the improved YOLO network model in combination with the bulk density and moisture content parameters of the corn. The imaging module, the improved YOLO network model and the hierarchical calculation module are configured to perform the machine vision-based corn hierarchical classification method according to any one of claims 1-9.

Images