A soybean pod counting method based on machine vision

Through the machine vision-based FasterR-CNN and CSPResNet-50 network combined with kmeans clustering and feature pyramid FPN structure, the real-time, accurate and high-throughput problems of soybean pod phenotype survey in traditional methods are solved, and automated identification and counting are achieved. Reduces the time and cost of manual counting, meeting the needs of high throughput and high accuracy.

CN114724141BActive Publication Date: 2024-11-01NORTHEAST AGRICULTURAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210355852.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-06
Publication Date
2024-11-01
Estimated Expiration
2042-04-06

AI Technical Summary

Technical Problem

Traditional methods have artificial subjective errors in soybean pod phenotype surveys, the inability to track time changes in real time, the cost is high, and the inability to conduct high-throughput phenotype surveys, resulting in an urgent need for automated, real-time, and high-throughput acquisition of pod information.

Method used

Using a machine vision-based method, the FasterR-CNN model and the CSPResNet-50 network are combined with kmeans clustering and feature pyramid FPN structure to identify and count soybean pod images, use smartphones to capture images and perform image enhancement processing, and realize automation Identify and count.

Benefits of technology

It greatly reduces the time and cost of manual counting, improves recognition accuracy and efficiency, can quickly and accurately obtain soybean pod count, supports high-throughput phenotypic surveys, and meets the needs of real-time tracking and high accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114724141B_ABST
    Figure CN114724141B_ABST
Patent Text Reader

Abstract

The present invention discloses a soybean pod number counting method based on machine vision, including: collecting soybean pod images, using LabelImg to annotate the soybean pod images, using ResNet-50 as the basic network to build a CSPResNet-50 network, and extracting multi-scale features in combination with a feature pyramid FPN structure, and finally forming a new FasterR-CNN model, wherein a kmeans clustering algorithm is added to a region proposal network RPN to adjust the number and size of anchorboxes in the region proposal network RPN, the extracted multi-scale features are input into the FastR-CNN model, the recognized soybean pod images are output, and finally the recognized pod images are input into a computer program to count the number of soybean pods. The method can improve the speed of soybean pod counting and better serve agricultural research.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and more specifically to a soybean pod quantity counting method based on machine vision. Background Art

[0002] As an important food crop, soybean has many fine phenotypes that cannot be obtained due to its complex plant structure and severe shading during the growth period. The lack of real-time, accurate and batch phenotyping support has caused the research on its laws and genetic mechanisms to stagnate. Traditional manual methods have been gradually replaced by some instruments with embedded models and algorithms due to time-consuming and labor-intensive methods and high error rates. Automated phenotyping has become a goal pursued by people at this stage. In recent years, with the rapid development of artificial intelligence algorithms, deep learning technology has achieved great success in many fields, especially in computer vision. Compared with traditional algorithms, deep learning can automatically extract image features.

[0003] In the past few decades, researchers have used manual timed observation or dyeing to mark pod phenotypes. These methods mainly rely on manual investigations, so there are many problems: 1. There are human subjective errors and low accuracy. 2. Only the total number of pods that fall in stages can be counted, and the temporal changes cannot be paid attention to. 3. The source, location and target of pod formation cannot be tracked. 4. The cost of field statistics is high, and high-throughput phenotypic surveys cannot be conducted. The existence of many problems makes our demand for automated, real-time, and high-throughput acquisition of pod information particularly urgent. In addition, research on the temporal changes in soybean pod falling, the tracing and tracking of soybean pod formation pathways, and other issues will rely on the automatic identification of the number of pods. High-throughput, highly accurate pod identification and counting have become the key to solving these problems.

[0004] Object detection algorithm is one of the important research fields in computer vision. It can not only classify the objects of interest in the image, but also get the specific location of the object in the image. Object detection is mainly divided into one-stage detection and two-stage detection. The one-stage detection algorithm directly predicts the category and regresses the features extracted by the feature extraction network. The two-stage detection first generates the first-stage candidate regions for the extracted features, selects the regions that may contain objects, and classifies and regresses the objects in the candidate regions in the second stage. Common one-stage detection algorithms include SSD, YOLO series, CornerNet, etc., and two-stage detection algorithms include R-CNN, FastR-CNN, FasterR-CNN, Cascade R-CNN, etc. Compared with one-stage detection, two-stage detection has higher detection accuracy but slower speed. Summary of the invention

[0005] The embodiment of the present invention provides a soybean pod quantity counting method based on machine vision, comprising:

[0006] Collect images of whole soybean pods and partial pods;

[0007] Use LabelImg to manually annotate the whole soybean pod images and partial pod images;

[0008] In the FasterR-CNN model, the kmeans clustering algorithm is used to cluster the width and height of the manually annotated bounding boxes, so as to adjust the aspect ratio of the anchor boxes in the RPN module and obtain the preset value of the anchor box aspect ratio that is more consistent with the pod;

[0009] The feature pyramid FPN structure is introduced into the CSPResNet-50 network to detect multi-scale feature images containing small target pods, which are input into the FasterR-CNN model to obtain the recognized soybean pod images.

[0010] The identified soybean pod images are input into a computer program to count the number of soybean pods.

[0011] Preferably, the method further includes processing the whole soybean pod image and the local soybean pod image, which includes:

[0012] Random cropping and random horizontal flipping are used to enhance the original image. The cropping size is random during random cropping, and the probability of flipping the image during horizontal flipping is 0.5.

[0013] Preferably, when LabelImg is used to manually annotate the whole soybean pod images and the partial pod images, each image contains a maximum of 11 pods and a minimum of 1 pod, and each image contains an average of 3.4 pods.

[0014] Preferably, inputting the identified soybean pod image into a computer program to count the number of soybean pods comprises:

[0015] Run the prediction program to automatically identify and count the input images, and display the number of soybean pods in each image.

[0016] Preferably, the CSPResNet-50 network includes:

[0017] ResNet-50 is used as the basic network;

[0018] Multiple ResNet-50s provide the output features of the previous ResNet-50 as part of the input features to the subsequent ResNet-50 in a stage-by-stage iterative manner;

[0019] Use the feature map of the last ResNet-50 for object detection.

[0020] The embodiment of the present invention provides a soybean pod number counting method based on machine vision. Compared with the prior art, the method has the following beneficial effects:

[0021] In previous rice breeding yield research, the number of soybean pods was often calculated manually. Although this method can obtain a relatively accurate number of pods, it is time-consuming and labor-intensive, which brings great inconvenience to researchers. An automatic acquisition method for soybean pod recognition and counting based on machine vision greatly saves time, reduces labor costs, and helps promote soybean yield research. Moreover, we only need to take pictures with our smartphones, without the need for expensive shooting equipment. The recognition and detection method is simple to operate and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a method for identifying and counting soybean pods using a target detection method in deep learning according to the present invention;

[0023] Figure 2 The present invention captures soybean images under different environments;

[0024] Figure 3 The present invention performs image enhancement processing on soybean images;

[0025] Figure 4 It is the detailed network structure of the field pod counting model Faster R-CNN adopted by the present invention;

[0026] Figure 5 is a distribution diagram of the pod area and the percentage of the image area occupied by the bounding box according to the present invention;

[0027] Figure 6 The present invention uses the kmeans algorithm to perform unsupervised clustering on the bounding boxes annotated in the training set, and clusters the number of anchorboxes and aspect ratios;

[0028] Figure 7 It is the detection result of the bean pod image using the deep learning network Faster R-CNN in the present invention;

[0029] Figure 8 It is the detection result of the pod image by Faster R-CNN before and after adjusting the number and aspect ratio of anchorboxes in the present invention;

[0030] Fig. 9 It is a correlation test diagram of the soybean pod detection and counting results and the true results of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] See also Figures 1 to 9 The embodiment of the present invention provides a soybean pod number counting method based on machine vision, the method comprising:

[0033] First embodiment

[0034] See also Figure 1 Flowchart of Figures 2 to 9 The present invention provides a method for soybean pod recognition and counting based on machine vision, which is carried out according to the following steps: 1. Acquisition of soybean materials: From June 2019 to October 2019, soybean experimental materials were planted in the experimental field base of Northeast Agricultural University. The experimental varieties were DN252 and HN51 soybeans, both of which had sub-limited pod-setting habits.

[0035] 2. Acquisition of soybean pod images: All images were taken with an iPhone 8 in natural daytime conditions. The shooting angles were horizontal and overhead. For some nodes where the main stem and pods were seriously blocked from each other, overhead shooting was used. A total of 2,693 images were collected, with an image size of 3024×4032 pixels.

[0036] 3. Data preparation: The images were resized to 400×400 pixels and annotated using LabelImg. Each image contained a maximum of 11 pods and a minimum of 1 pod, with an average of 3.4 pods per image. The dataset format was prepared according to the COCO dataset format, with a 7:3 ratio between the training set and the test set.

[0037] 4. Preprocess the image: Use image enhancement methods to increase the diversity of input images by cropping, rotating, and scaling the original image, so that the model has better robustness and generalization ability when facing images in different scenes. This paper uses random cropping and random horizontal flipping to enhance the image. Random cropping will randomly crop the original image, and the probability of randomly flipping the image horizontally is 0.5.

[0038] 5. Configure the environment: Here we use Python as our programming language. In order to perform detection faster, we recommend installing the computing platform CUDA and the supporting cuDNN launched by the graphics card manufacturer NVIDIA, which will greatly improve the detection efficiency. Of course, this is not necessary. Secondly, we recommend installing Anaconda, which contains a large number of packages we need, which can save the tedious installation time later. In addition, in order to operate the code more conveniently, we need a compiler that can compile Python. Here we choose PyCharm as our compiler. Since we are using the PaddlePaddle framework for deep learning, after all the preparatory work is completed, we also need to install PaddlePaddle. If CUDA and cuDNN can be installed and used correctly, the GPU version of PaddlePaddle can be installed here. If CUDA and cuDNN are not installed, the CPU version of PaddlePaddle is provided for use.

[0039] 6. Model establishment: We selected the Faster R-CNN model as our target model and optimized it in the following way. First, we analyzed the area of ​​soybean pods. There was a relatively obvious scale change, and some of the annotated bounding boxes were in the small target range. To solve this problem, we added the feature pyramid network FPN module, which can extract multi-scale features from images, enhance semantic information, and enhance the detection ability of small targets and multi-scale objects. Secondly, the kmeans method was used to perform unsupervised clustering on the normalized annotated bounding boxes in the training set to adjust the number and size of anchorboxes and generate anchor boxes that are more in line with the pod shape. Finally, after comparing multiple feature extraction networks, CSPResNet-50 was selected as the backbone.

[0040] 7. Model training: Run the training program and start training. Through continuous training and optimization, we get the optimal training results and save the optimal weights in the logs folder.

[0041] 8. Recognition and counting of soybean pods: Running the prediction program can automatically recognize and count the input images, and finally display the number of soybean pods in each image. The mAP of the test set reaches 91%, and the correlation coefficient between the predicted number and the actual number reaches 90.46%.

[0042] Second embodiment

[0043] The method for identifying and counting soybean pods based on machine vision of the present invention is different from any implementation in the specific embodiment 1 in that a method of different feature extraction networks is used for training and testing.

[0044] Third embodiment

[0045] The method for identifying and counting soybean pods based on machine vision of the present invention is different from the first and second specific embodiments in that we use the number and aspect ratio of anchor boxes that are not adjusted. Specific embodiment 1

[0047] 1. Acquisition of soybean materials: From June 2019 to October 2019, soybean experimental materials were planted in the experimental field base of Northeast Agricultural University. The experimental varieties were two varieties of soybeans, DN252 and HN51, both of which had sub-determinate pod-setting habits.

[0048] 2. Acquisition of soybean pod images: All images were taken with an iPhone 8 in natural daytime conditions. The shooting angles were horizontal and overhead. For some nodes where the main stem and pods were seriously blocked from each other, overhead shooting was used. A total of 2,693 images were collected, with an image size of 3024×4032 pixels.

[0049] 3. Data preparation: The images were resized to 400×400 pixels and annotated using LabelImg. Each image contained a maximum of 11 pods and a minimum of 1 pod, with an average of 3.4 pods per image. The dataset format was prepared according to the COCO dataset format, with a 7:3 ratio between the training set and the test set.

[0050] 4. Preprocess the image: Use image enhancement methods to increase the diversity of input images by cropping, rotating, and scaling the original image, so that the model has better robustness and generalization ability when facing images in different scenes. This paper uses random cropping and random horizontal flipping to enhance the image. Random cropping will randomly crop the original image, and the probability of randomly flipping the image horizontally is 0.5.

[0051] 5. Configure the environment: Here we use Python as our programming language. In order to perform detection faster, we recommend installing the computing platform CUDA and the supporting cuDNN launched by the graphics card manufacturer NVIDIA, which will greatly improve the detection efficiency. Of course, this is not necessary. Secondly, we recommend installing Anaconda, which contains a large number of packages we need, which can save the tedious installation time later. In addition, in order to operate the code more conveniently, we need a compiler that can compile Python. Here we choose PyCharm as our compiler. Since we are using the PaddlePaddle framework for deep learning, after all the preparatory work is completed, we also need to install PaddlePaddle. If CUDA and cuDNN can be installed and used correctly, the GPU version of PaddlePaddle can be installed here. If CUDA and cuDNN are not installed, the CPU version of PaddlePaddle is provided for use.

[0052] 6. Model establishment: We selected the FasterR-CNN model as our target model and optimized it in the following way. First, we analyzed the area of ​​soybean pods. There were obvious scale changes, and some of the annotated bounding boxes were in the small target range. To solve this problem, we added the feature pyramid network FPN module, which can extract multi-scale features from images, enhance semantic information, and enhance the detection ability of small targets and multi-scale objects. Secondly, the kmeans method was used to perform unsupervised clustering on the normalized annotated bounding boxes in the training set to adjust the number and size of anchorboxes and generate anchor boxes that are more in line with the pod shape. Finally, after comparing multiple feature extraction networks, CSPResNet-50 was selected as the backbone.

[0053] 7. Model training: Run the training program and start training. Through continuous training and optimization, we get the optimal training results and save the optimal weights in the logs folder.

[0054] 8. Recognition and counting of soybean pods: Running the prediction program can automatically recognize and count the input images, and finally display the number of soybean pods in each image. The mAP of the test set reaches 91%, and the correlation coefficient between the predicted number and the actual number reaches 90.46%.

[0055] The specific embodiment 1 is subjected to deep learning operation training to obtain Table 1, Table 2, Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9By observing Table 1, we can see that the deep learning network FasterR-CNN has converged after 47 iterations. At this time, the accuracy of the model with CSPResNet-50+FPN as the backbone network is greater than that of the model composed of other backbone networks. Table 2 shows that after adjusting the aspect ratio, the overall detection accuracy of the model is improved by 0.1%, and the detection ability of small targets is improved by 0.5%. Figure 5 It is the result of analyzing the pixel area occupied by the annotated bounding boxes in the training set. The results show that the percentage of the area occupied by the annotated bounding boxes in the image is between 0-0.25, there is a relatively obvious scale change, and a large part of the annotated bounding boxes are in the small target range. FPN is added to solve this problem. Figure 6 Cluster diagrams under different K values ​​and the relationship between the number of anchorboxes and MeanIOU. Finally, 18 anchorboxes were selected, and then the ones with similar aspectratios were added and then the average was calculated. Finally, aspectratios of [0.75, 1.8, 3.2] were selected for the experiment. Figure 7 It shows that the detection effect of pods in different environments is good, such as pods with similar colors, dense and overlapping pods. Figure 8 (a) is the model trained after modifying the number and aspect ratio of anchorboxes. Figure 4-8 (b) is the model trained with default values. From the local magnified image, as indicated by the red arrow, the adjusted predicted bounding box is more consistent with the actual shape of the pod and has better detail processing. Fig. 9 (a) shows the frequency percentage of the error between the model prediction value and the true value, and the prediction error is small (the prediction error of 0 and the prediction error of 1 account for 96%). Fig. 9 (b) is a bubble chart showing a linear correlation between the true value and the predicted value. The size of the bubble represents the number of overlapping points. The square of the Pearson correlation coefficient between the true value and the predicted value of the proposed model is 0.9046, which indicates that the present invention has a good counting ability for pods in different scenarios.

[0056] Table 1 Comparison of different backbones as feature extraction networks

[0057]

[0058] Table 2 Comparison of test results of adjusting the aspect ratio of anchorboxes

[0059]

[0060] Disclosed above are only a few specific embodiments of the present invention. Those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the present invention. However, the embodiments of the present invention are not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A soybean pod number counting method based on machine vision, characterized in that: include: Collect images of whole soybean pods and partial pods; Use LabelImg to manually annotate the whole soybean pod images and partial pod images; In the Faster R-CNN model, the kmeans clustering algorithm is used to cluster the width and height of the manually annotated bounding boxes. The aspect ratio of the anchor boxes in the RPN module is adjusted according to the width and height of the clustered bounding boxes to obtain the preset value of the anchorboxes aspect ratio of the pod. The feature pyramid FPN structure is introduced into the CSPResNet-50 network to detect the multi-scale feature image containing the small target pods. The multi-scale feature image is input into the FasterR-CNN model to obtain the recognized soybean pod image. According to the recognized soybean pod images, the number of soybean pods is counted.

2. The method for counting the number of soybean pods based on machine vision according to claim 1, characterized in that: It also includes processing of whole soybean pod images and partial soybean pod images, including: Random cropping and random horizontal flipping are used to enhance the original image. The cropping size is random during random cropping, and the probability of flipping the image during horizontal flipping is 0.

5.

3. The method for counting soybean pods based on machine vision according to claim 1, characterized in that: When LabelImg is used to manually annotate the whole soybean pod image and the partial pod image, each image contains a maximum of 11 pods and a minimum of 1 pod, and each image contains an average of 3.4 pods.

4. The method for counting soybean pods based on machine vision according to claim 1, characterized in that: The CSPResNet-50 network includes: ResNet-50 is used as the basic network; Multiple ResNet-50s provide the output features of the previous ResNet-50 as part of the input features to the subsequent ResNet-50 in a stage-by-stage iterative manner; Use the feature map of the last ResNet-50 for object detection.