Straw chopping length detection method and system fusing neural network and image processing

Abstract

The present application relates to the technical field of intelligent corn straw harvesting, and specifically discloses a straw chopping length detection method and system combining a neural network and image processing, the method comprising: acquiring original images with various morphological characteristics; constructing a YOLOv5 neural network model; outputting the boundary box of the straw target and the reference object; and generating a straw region of interest image; performing image processing on the straw region of interest image and extracting the straw edge; determining the pixel area occupied by the straw based on the straw edge; calculating the pixel length of the straw based on the least-enclosing-rectangle fitting; converting the pixel length of the straw into actual physical dimensions based on the reference object calibration; correcting the actual physical dimensions based on the curvature compensation model; and accurately measuring the length of the chopped corn straw based on YOLOv5 and image processing. The straw region of interest is located by YOLOv5, the target is accurately segmented by traditional image processing, and the original image is processed by the length calculation algorithm, so that the length of the straw can be accurately predicted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent corn stalk harvesting, in particular to a stalk chopping length detection method and system fusing neural networks and image processing. BACKGROUND

[0002] China is an important agricultural producer in the world, and corn crops are widely planted across the country, with annual production ranking among the world's top. Crop stalks, as an important biomass resource, have an annual total output of over 800 million tons, of which corn stalks account for the largest proportion, with an annual output of about 300 million tons, accounting for 32.5% of the total. Corn stalks have the characteristics of small density, large volume, and scattered distribution, and their collection and comprehensive utilization face many challenges. For a long time, the main methods of stalk disposal have been direct return to the field, burning, or composting, but these traditional methods have significant drawbacks: direct return to the field decomposes slowly, affecting the subsequent sowing of crops; burning stalks produces a large amount of smoke and harmful gases, seriously polluting the environment. With the advancement of the "double carbon" goal and the implementation of the "straw into meat" policy, the feed use of straw has gradually become the main direction of resource utilization. After chopping and bundling, corn stalks can be used as high-quality forage, with rich nutritional content and good palatability, which is of great significance in alleviating the "grain for people and livestock" contradiction and filling the gap of 500 million tons of high-quality forage. Stalk bundling and harvesting technology can effectively improve the storage and transportation efficiency of stalks, thereby promoting more rational utilization of crop stalk resources.

[0003] In the process of corn stalk bundling, the chopping process is a key link connecting stalk collection and subsequent utilization. Chopped stalks can optimize the density of the bale to reduce storage and transportation costs, and can also improve the suitability of animal feeding. Therefore, the chopping length of the stalks is a key indicator of the stability of subsequent forage storage and the convenience of animal feeding. However, the existing corn stalk bundling machines still have the following significant problems: 1) inconsistent chopping length, affecting forage quality; 2) low level of intelligence, unable to monitor and adjust operating parameters in real time; 3) high operating loss rate, leading to resource waste. These problems seriously hinder the development of the straw feed industry. Therefore, how to detect the chopping length of the stalks in real time has become a key technical requirement for improving the utilization efficiency of stalks and promoting the upgrading of agricultural mechanization. SUMMARY

[0004] The purpose of the present application is to provide a stalk chopping length detection method and system fusing neural networks and image processing to solve the problems raised in the background.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] The stalk chopping length detection method fusing neural networks and image processing comprises:

[0007] An original image with multiple morphological characteristics is acquired, a YOLOv5 neural network model is constructed, and the YOLOv5 neural network model is trained based on the original image;

[0008] An image to be measured is acquired, the image to be measured is input into the pre-trained YOLOv5 neural network model, the bounding box of the straw target and the reference object is output, and a straw region of interest image is generated;

[0009] The straw region of interest image is processed, and the straw edge is extracted;

[0010] The pixel area occupied by the straw is determined based on the straw edge, and the pixel length of the straw is calculated based on the minimum circumscribed rectangle fitting;

[0011] The pixel length of the straw is converted into an actual physical scale based on the reference object calibration, the actual physical scale is corrected based on a curvature compensation model, and a corrected actual length of the straw is generated.

[0012] As a further scheme of the present application, the multiple morphological characteristics include single flat, multiple stacked, curved or polyline.

[0013] As a further scheme of the present application, the image processing includes grayscale, image filtering, binarization and morphological operation in sequence.

[0014] As a further scheme of the present application, the grayscale method is weighted average method, the image filtering method is Gaussian filtering, the binarization method is adaptive threshold segmentation, and the morphological operation is closing operation.

[0015] As a further scheme of the present application, the closing operation includes: first using a 3x3 rectangular kernel to perform 3 times of dilation operation on the binary image, and then performing 1 time of erosion operation on the dilated image.

[0016] As a further scheme of the present application, the minimum circumscribed rectangle fitting is realized by the minAreaRect function of the OpenCV library, and specifically includes:

[0017] The connected regions in the binary image are recognized, and the convex hull is calculated for each connected domain;

[0018] Based on the convex hull vertex set, the minimum area rectangle enclosing the convex hull is found as the minimum circumscribed rectangle by rotating the rectangular frame.

[0019] As a further scheme of the present application, the step of converting the pixel length of the straw into an actual physical scale based on the reference object calibration specifically includes:

[0020] The minimum circumscribed rectangle of the gasket is detected and extracted by YOLOv5, and the pixel width of the inner diameter in the image is calculated ;

[0021] According to the actual inner diameter size of the gasket The calibration coefficient k is calculated according to the ratio of the pixel width ; ;

[0022] The minimum circumscribed rectangle length axis pixel value of the straw is multiplied by the calibration coefficient k, and the actual length of the straw is obtained: .

[0023] As a further scheme of the present application, the correction of the actual physical scale based on the curvature compensation model specifically includes:

[0024] The aspect ratio r of the minimum circumscribed rectangle of the straw is calculated;

[0025] If r>3, the bending compensation factor calculation formula is: ;

[0026] The corrected straw length is: .

[0027] The present application also provides a straw chopping length detection system fusing neural networks and image processing, for realizing a straw chopping length detection method fusing neural networks and image processing, the system comprising:

[0028] A model construction module is configured to acquire original images with various morphological characteristics, construct a YOLOv5 neural network model, and train the YOLOv5 neural network model based on the original images;

[0029] A region of interest image generation module is configured to acquire a to-be-detected image, input the to-be-detected image into a pre-trained YOLOv5 neural network model, output the boundary box of a straw target and a reference object, and generate a straw region of interest image;

[0030] An edge extraction module is configured to perform image processing on the straw region of interest image and extract the edge of the straw;

[0031] A pixel length calculation module is configured to determine the pixel area occupied by the straw based on the edge of the straw, and calculate the pixel length of the straw based on minimum circumscribed rectangle fitting calculation;

[0032] A correction module is configured to convert the pixel length of the straw into an actual physical scale based on reference object calibration, correct the actual physical scale based on a curvature compensation model, and generate a corrected actual length of the straw.

[0033] ​Compared with the prior art, the beneficial effects of the present application are: based on YOLOv5 and image processing, the length of chopped corn straw is accurately measured, by obtaining images of chopped straw in multiple forms, the data set is enriched, the model robustness is improved, through YOLOv5 straw region of interest positioning, traditional image processing target accurate segmentation and length calculation algorithm processing original image, the length of the measured straw can be accurately predicted. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application.

[0035] Figure 1 The flow chart of the straw chopping length detection method provided by the fusion neural network and image processing in the embodiments of the present application.

[0036] Figure 2 The specific flow chart of YOLOv5 neural network training provided by the embodiments of the present application.

[0037] Figure 3 The chopped straw images of multiple morphological characteristics provided by the embodiments of the present application, wherein (a) is a single flat straw image; (b) is a multiple stacked straw image; (c) is a curved straw image; (d) is a polyline straw image.

[0038] Figure 4 The detection result schematic diagram output by the YOLOv5 neural network provided by the embodiments of the present application, wherein (a) is a single flat straw detection result; (b) is a multiple stacked straw detection result; (c) is a curved straw detection result; (d) is a polyline straw detection result. DETAILED DESCRIPTION

[0039] In order to make the technical problems to be solved by the present application, the technical solutions and beneficial effects more clear and obvious, the present application will be further described in detail in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0040] In the embodiments of the present application, the straw chopping length detection method of fusion neural network and image processing, the method comprises:

[0041] An original image with multiple morphological characteristics is obtained, a YOLOv5 neural network model is constructed, and the YOLOv5 neural network model is trained based on the original image;

[0042] An image to be measured is acquired, and the image to be measured is input into a pre-trained YOLOv5 neural network model to output a boundary box of a straw target and a reference object, and a straw region of interest image is generated;

[0043] The straw region of interest image is subjected to image processing, and a straw edge is extracted;

[0044] A pixel region occupied by the straw is determined based on the straw edge, and a pixel length of the straw is calculated based on a minimum circumscribed rectangle fitting;

[0045] The pixel length of the straw is converted into an actual physical scale based on a reference object calibration, and the actual physical scale is corrected based on a curvature compensation model to generate a corrected actual length of the straw.

[0046] As Figures 1 to 4 shown, in the embodiment, the multiple morphological features include single flat laying, multiple stacking, bending or polyline, by shooting corn chopped straw with multiple morphological features, an original image of the corn straw is obtained, the original image is subjected to data labeling, and a data set is constructed.

[0047] The data labeling process specifically includes that all images are labeled by LabelImg software, a YOLO standard is used for labeling format, two types of labels are defined, i.e., “straw” (straw main body) and “washer” (reference washer), after labeling is completed, a txt file containing target position and category information is generated, and is used as basic data for model training;

[0048] The image to be measured is input into a pre-trained YOLOv5 neural network to realize initial positioning of a target region, and a straw region of interest is obtained;

[0049] Further, the training process of the pre-trained YOLOv5 neural network specifically includes:

[0050] The data set is configured in a format required by YOLOv5;

[0051] Training is started, train.py is run through a command line, model type, data set configuration, input image size and other parameters are specified, pre-trained weights yolov5s.pt are loaded to accelerate convergence;

[0052] Further, the data set configuration includes a directory structure and a configuration file, and a specific process includes that a straw_data folder is created, images (for storing training / verification / testing images) and labels (for storing corresponding labeling files) are arranged under the straw_data folder, a subfolder is established for each subset (train / val / test), a corn_stalk.yaml file is written, and data set paths, a number of categories and category names are defined.

[0053] The parameter adjustment optimization is performed, and the superparameters to be adjusted include a learning rate, a batch size and a training round, and after the optimization, the learning rate is set to 0.001, the batch size is set to 2, and the training round is set to 200.

[0054] The image to be detected is input into the trained YOLOv5 neural network, the YOLOv5 neural network rapidly detects the straw target and the reference pad in the image, and outputs the boundary box as a region of interest, so as to realize initial positioning of the straw target region.

[0055] As a preferred embodiment of the present application, the image processing comprises, in sequence, gray-scale processing, image filtering, binaryzation and morphological operation.

[0056] The gray-scale processing method is a weighted average method, the image filtering method is Gaussian filtering, the binaryzation method is adaptive threshold segmentation, and the morphological operation is a closing operation.

[0057] The closing operation comprises: first performing 3 times of expansion operation on the binary image using a 3*3 rectangular kernel, and then performing 1 time of erosion operation on the expanded image.

[0058] In the present embodiment, the 3 times of expansion operation using the 3*3 rectangular kernel increases the average connection length of the straw broken region by 18%, and significantly improves the target continuity; and the 1 time of erosion operation on the expanded image removes the noise points with an area less than 10 pixels in the image, and greatly reduces the noise density.

[0059] The gray-scale processing converts the input color image into a gray-scale image by using the function cv2.cvtColor(image, cv2.COLOR_BGR2GRAY), the image filtering operation performs Gaussian filtering on the gray-scale image by using the function cv2.GaussianBlur(gray, (5, 5), 1.5) to suppress high-frequency noise, the binaryzation operation performs adaptive threshold segmentation on the filtered image by using the function cv2.adaptiveThreshold(blurred, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 11, 10), and the morphological operation performs 3 times of expansion on the binaryzation image by using a 3*3 rectangular kernel and the function cv2.dilate(binary, kernel, iterations=3) to connect the broken regions, and performs 1 time of erosion operation on the image after the expansion operation by using a 3*3 rectangular kernel and the function cv2.erode(dilated, kernel, iterations=1) to remove small noise points.

[0060] As a preferred embodiment of the present application, the minimum circumscribed rectangle fitting is implemented by the minAreaRect function of the OpenCV library, specifically including:

[0061] Identifying the connected regions in the binary image, calculating the convex hull for each connected domain;

[0062] Based on the convex hull vertex set, rotating the rectangular frame to find the minimum area rectangle enclosing the convex hull as the minimum circumscribed rectangle.

[0063] In this embodiment, the minAreaRect function in the OpenCV library is used to implement the rotating calibrator algorithm, and the steps are as follows: traversing the binary image, marking all connected regions to ensure that each straw target is independently identified; calculating the convex hull of each connected domain, i.e. the minimum convex polygon enclosing the target, and the convex hull is generated by the Graham scan algorithm; based on the convex hull vertex set, rotating the rectangular frame (angle step 1°), calculating the minimum area rectangle enclosing the convex hull in all possible directions, and finally selecting the smallest rectangle as the minimum circumscribed rectangle.

[0064] As a preferred embodiment of the present application, the step of converting the pixel length of the straw into actual physical scale based on the reference object calibration specifically includes:

[0065] Detecting and extracting the minimum circumscribed rectangle of the gasket by YOLOv5, calculating the pixel width of the gasket inner diameter in the image ;

[0066] According to the ratio of the actual inner diameter size of the gasket to the pixel width , the calibration coefficient k is calculated, i.e. ;

[0067] Multiply the minimum circumscribed rectangle length axis pixel value of the straw by the calibration coefficient k to obtain the actual length of the straw: .

[0068] In this embodiment, the standard metal gasket with an inner diameter of 10 mm is selected for reference object calibration.

[0069] As a preferred embodiment of the present application, the step of correcting the actual physical scale based on the curvature compensation model specifically includes:

[0070] Calculating the aspect ratio r of the minimum circumscribed rectangle of the straw;

[0071] If r>3, the bending compensation factor calculation formula is: ;

[0072] The corrected straw length is: .

[0073] In the present embodiment, a curvature compensation model is introduced for the curved straw, and the specific calculation process is as follows: it is judged whether the aspect ratio r of the minimum circumscribed rectangle is greater than 3, if greater than 3, the curvature compensation factor is calculated, and the product value of the curvature compensation factor and the long axis pixel value of the minimum circumscribed rectangle of the straw to be corrected is calculated, to obtain the corrected straw length.

[0074] If it is not greater than 3, it indicates that the degree of bending of the straw is small, and the length value calculated is not corrected.

[0075] The measured straw length data is saved, and the target straw and the calibration reference are framed in the output result of the YOLOv5 neural network, and the calculated corresponding length of the straw is labeled, as shown in Figure 4

[0076] In the present example, the test set contains 50 independent images, and the average error is 5.55 mm after calculation, and the error of more than 70% of the straw detection results is controlled within ±6.0 mm, and the final prediction accuracy is 93.1%, realizing the accurate calculation of the length of the chopped corn straw, and meeting the actual application requirements.

[0077] In the embodiment of the present application, a straw chopped length detection system is also provided, which fuses a neural network and image processing, and the system comprises:

[0078] A model construction module is configured to acquire original images with various morphological characteristics, construct a YOLOv5 neural network model, and train the YOLOv5 neural network model based on the original images;

[0079] A region of interest image generation module is configured to acquire a to-be-detected image, input the to-be-detected image into a pre-trained YOLOv5 neural network model, output the boundary frame of the straw target and the reference, and generate a straw region of interest image;

[0080] An edge extraction module is configured to perform image processing on the straw region of interest image and extract the edge of the straw;

[0081] A pixel length calculation module is configured to determine the pixel region occupied by the straw based on the edge of the straw, and calculate the pixel length of the straw based on the fitting of the minimum circumscribed rectangle;

[0082] A correction module is configured to convert the pixel length of the straw into an actual physical scale based on the calibration of the reference, correct the actual physical scale based on a curvature compensation model, and generate a corrected actual length of the straw.

[0083] The above only describes the preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.​

Claims

1. A method for detecting the length of chopped straw by integrating neural networks and image processing, characterized in that, The method includes: Obtain original images with various morphological features, construct a YOLOv5 neural network model, and train the YOLOv5 neural network model based on the original images; The image to be tested is acquired and input into a pre-trained YOLOv5 neural network model, which outputs the bounding boxes of the straw target and reference objects to generate the straw region of interest image. Image processing is performed on the region of interest image of straw, and the straw edges are extracted; The pixel area occupied by the straw is determined based on the straw edge, and the pixel length of the straw is calculated based on the minimum bounding rectangle fitting. Based on the reference calibration, the pixel length of the straw is converted into the actual physical scale, and the actual physical scale is corrected based on the curvature compensation model to generate the corrected actual length of the straw. The minimum bounding rectangle fitting is implemented using the minAreaRect function in the OpenCV library, specifically including: Identify connected regions in a binary image and compute the convex hull for each connected region; Based on the vertex set of the convex hull, rotate the rectangle to find the rectangle with the smallest area that encloses the convex hull as the minimum bounding rectangle; The step of converting the pixel length of the straw into actual physical scale based on reference calibration specifically includes: The minimum bounding rectangle of the pad is detected and extracted using YOLOv5, and its inner diameter in pixels is calculated. ; Based on the actual inner diameter of the gasket With pixel width The ratio is used to calculate the calibration coefficient k, i.e. ; The major axis pixel value of the minimum bounding rectangle of the straw Multiplying by the calibration coefficient k, we obtain the actual length of the straw: ; The correction of the actual physical scale based on the curvature compensation model specifically includes: Calculate the aspect ratio r of the minimum bounding rectangle of the straw. If r>3, the formula for calculating the bending compensation factor is: ; Corrected straw length: .

2. The straw chopping length detection method integrating neural networks and image processing according to claim 1, characterized in that, The various morphological features include single strands laid flat, multiple strands stacked, curved, or zigzag-shaped.

3. The straw chopping length detection method integrating neural networks and image processing according to claim 1, characterized in that, The image processing includes grayscale conversion, image filtering, binarization, and morphological operations in sequence.

4. The straw chopping length detection method integrating neural networks and image processing according to claim 3, characterized in that, The grayscale conversion method is weighted average, the image filtering method is Gaussian filtering, the binarization method is adaptive threshold segmentation, and the morphological operation is closing operation.

5. The straw chopping length detection method integrating neural networks and image processing according to claim 4, characterized in that, The closing operation includes: first, performing three dilation operations on the binary image using a 3×3 rectangular kernel, and then performing one erosion operation on the dilated image.

6. A straw chopping length detection system integrating neural networks and image processing, used to implement the straw chopping length detection method integrating neural networks and image processing as described in any one of claims 1-5, characterized in that, The system includes: The model building module is used to acquire original images with various morphological features, build a YOLOv5 neural network model, and train the YOLOv5 neural network model based on the original images. The region of interest image generation module is used to acquire the image to be tested, input the image to be tested into the pre-trained YOLOv5 neural network model, output the bounding box of the straw target and the reference object, and generate the straw region of interest image; The edge extraction module is used to perform image processing on the region of interest image of straw and extract the straw edges; The pixel length calculation module is used to determine the pixel area occupied by the straw based on the straw edge, and calculate the pixel length of the straw based on the minimum bounding rectangle fitting. The correction module is used to convert the pixel length of the straw into the actual physical scale based on the reference calibration, and to correct the actual physical scale based on the curvature compensation model to generate the corrected actual length of the straw.

Images