A Deep Learning-Based Method and System for Predicting Harvest Loss in Corn Ear Harvesters

Abstract

This invention relates to the field of computer vision technology, specifically disclosing a method and system for predicting harvest loss in corn ear harvesters based on deep learning. The method includes real-time acquisition of images of ears within the corn harvester's ear collection box, cropping the ear images into several sub-images of preset sizes; performing pixel-level classification on the sub-images based on a pre-trained pixel-level classification model to generate a three-channel label image for each sub-image; stitching together several three-channel label images to form an overall label image with the same size as the original image; calculating the percentage of damaged corn ear pixels based on the overall label image; and outputting a predicted value of the corn ear harvest loss quality based on the pre-trained harvest loss prediction model when the ear collection box is full. The deep learning-based method can more accurately detect the proportion of corn damage during harvesting under different lighting conditions, while also solving the problem of low model recognition of damaged pixels due to a small number of damaged ear pixel samples.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for predicting harvest loss in corn ear harvesters based on deep learning. Background Technology

[0002] Corn is an important food and feed crop in my country. Corn ear harvesters are widely used in many major producing areas due to their advantages in harvesting corn in hilly and mountainous areas, small plots, and high moisture content corn. Compared with direct harvesting of corn kernels, ear harvesting does not immediately thresh the kernels during the operation. Instead, it collects the whole ears with husks into a storage box, and then processes them through subsequent threshing and drying. This operation mode has advantages in reducing kernel breakage and improving storage quality, so its promotion area has continued to grow in recent years. However, in the complex field environment, ear harvesters still face the following problems during operation: improper settings of parameters such as header height, chain speed, and peeling roller speed can easily cause ear kernel damage or ear drop, all of which will result in harvesting losses. At present, the loss quality detection of ear harvesters mostly relies on manual sampling at the field. This involves manually picking up fallen ears, observing the appearance of the ears in the collection box, weighing and recording, and then estimating the loss rate. While manual inspection has a certain degree of accuracy, it suffers from significant drawbacks, including low efficiency, poor real-time performance, and the need to interrupt operations, making it difficult to quickly adjust operating parameters under different field conditions. Therefore, developing an intelligent inspection system that can monitor the percentage of damaged ears and assess harvest loss in real time during operation is of great significance for improving the quality of ear harvester operations and reducing loss rates.

[0003] With the development of computer vision and deep learning technologies, methods based on high-frequency camera acquisition and intelligent image recognition are gradually being introduced into the quality monitoring of corn harvesting operations. Chinese invention patents CN 119399205 B and CN 119399564 B, by installing cameras at locations such as ear collection boxes to continuously acquire images of ears during the harvesting process, both provide pixel-level classification and damage ratio calculation methods for corn ear damage identification. However, neither addresses the accuracy degradation caused by differences in field lighting, nor the problem that models with a small proportion of damaged ears are more likely to identify images as intact ears. Furthermore, neither combines pixel-level damage in the image with the actual loss quality. Therefore, how to accurately detect the ear damage ratio under different lighting conditions in real time and predict the loss quality is crucial for ensuring a high-quality and efficient harvest of silage corn. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for predicting harvest loss of corn ear harvesters based on deep learning, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A deep learning-based method for predicting harvesting loss in corn ear harvesters, the method comprising:

[0007] Real-time acquisition of images of ears of corn inside the ear collection box of a corn harvester, and cropping of the ear images into several sub-images of preset sizes;

[0008] Build a pixel-level classification model, perform pixel-level classification on sub-images based on the pre-trained pixel-level classification model, and generate a three-channel label map for each sub-image;

[0009] Several three-channel label images are stitched together to form a unified label image with the same size as the original image;

[0010] The percentage of pixels with damaged corn ears is calculated based on the overall label image.

[0011] A harvest loss prediction model is constructed. When the ear collection box is full, the pre-trained harvest loss prediction model outputs the predicted value of the corn ear harvest loss quality based on the overall label map collected multiple times and its corresponding damage ratio.

[0012] As a further embodiment of the present invention, the pixel-level classification model is a fused inverted bottleneck attention UNet model, whose encoder is based on an improved EfficientNet b3 model, wherein the improved EfficientNet b3 model is: the first five layers of the EfficientNet b3 model are replaced with a fused inverted bottleneck convolutional structure.

[0013] As a further embodiment of the present invention, the harvest loss prediction model is a twin dual-channel Gram angle field difference-summation fusion network model.

[0014] As a further embodiment of the present invention, the step of the pre-trained harvest loss prediction model outputting the predicted value of corn ear harvest loss quality based on the overall label map collected multiple times and its corresponding damage ratio specifically includes:

[0015] Obtain the overall label map and its corresponding damage percentage from multiple acquisitions, and fuse the features of each overall label map and its corresponding damage percentage;

[0016] The fused data is transformed to generate a first type of feature map representing the global distribution structure of the damage and a second type of feature map representing the differences in local details of the damage.

[0017] Feature extraction is performed on the global feature map and the detail feature map using a dual-channel approach to generate a feature map.

[0018] The feature map is fused by adaptive average pooling, flattening and full connection to output the loss quality prediction value.

[0019] As a further embodiment of the present invention, the transformation is a multi-channel Gram angle field transformation, wherein the first type of feature map is a Gram angle field summation map, and the second type of feature map is a Gram angle field difference map.

[0020] As a further aspect of the present invention, the difference between processing the channel of the first type of feature map and processing the channel of the second type of feature map is that the channel of processing the first type of feature map includes a multi-dilated convolutional multi-scale feature module, which is used to capture global patterns of different receptive field ranges.

[0021] This invention also provides a deep learning-based corn ear harvester harvest loss prediction system, used to implement a deep learning-based corn ear harvester harvest loss prediction method, the system comprising:

[0022] The acquisition module is used to acquire images of ears of corn in the ear collection box of the corn harvester in real time and crop the ear images into several sub-images of preset sizes.

[0023] The classification module is used to build a pixel-level classification model. Based on the pre-trained pixel-level classification model, it performs pixel-level classification on sub-images and generates a three-channel label map for each sub-image.

[0024] The stitching module is used to stitch together several three-channel label images to form a whole label image with the same size as the original image.

[0025] The percentage calculation module is used to calculate the percentage of pixels with damaged corn ears based on the overall label image.

[0026] The prediction output module is used to build a harvest loss prediction model. When the ear collection box is full, the pre-trained harvest loss prediction model outputs the predicted value of the corn ear harvest loss quality based on the overall label map collected multiple times and its corresponding damage ratio.

[0027] Compared with the prior art, the beneficial effects of the present invention are: the deep learning-based method can more accurately detect the proportion of corn damage during corn harvesting by corn harvesters under different lighting conditions, and at the same time solve the problem that the model has low recognition of damaged pixels due to the small number of damaged ear pixel samples.

[0028] Linking the proportion of corn harvest damage with the actual harvest loss quality allows for rapid acquisition of harvest loss quality after harvest, solving the problem of time-consuming and labor-intensive manual re-inspection after harvest. It can also provide guidance for parameters of the next harvest operation. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0030] Figure 1 The flowchart illustrates the specific process of the deep learning-based method for predicting harvest loss in corn ear harvesters, as provided in this embodiment of the invention.

[0031] Figure 2 The diagram shows the structure of the FIBA ​​Unet pixel-level damage classification network model provided in this embodiment of the invention.

[0032] Figure 3 The flowchart below shows the Siamese MGAD-Net loss quality prediction model provided in this embodiment of the invention.

[0033] Figure 4 The images provided in this embodiment of the invention are the original images and corresponding segmented three-channel label images of the corn ear harvester at the beginning, middle and end of a single harvest. Detailed Implementation

[0034] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0035] like Figures 1 to 4 As shown in the embodiment of the present invention, a method for predicting harvest loss of corn ear harvesters based on deep learning is provided, the method comprising:

[0036] Real-time acquisition of images of ears of corn inside the ear collection box of a corn harvester, and cropping of the ear images into several sub-images of preset sizes;

[0037] Build a pixel-level classification model, perform pixel-level classification on sub-images based on the pre-trained pixel-level classification model, and generate a three-channel label map for each sub-image;

[0038] Several three-channel label images are stitched together to form a unified label image with the same size as the original image;

[0039] The percentage of pixels with damaged corn ears is calculated based on the overall label image.

[0040] A harvest loss prediction model is constructed. When the ear collection box is full, the pre-trained harvest loss prediction model outputs the predicted value of the corn ear harvest loss quality based on the overall label map collected multiple times and its corresponding damage ratio.

[0041] In this embodiment, a high-definition camera acquires real-time images of corn ears in the collection box while the harvester is working, and transmits the acquired images to a small AI computer. The image preprocessing module in the small AI computer reads the corn ear images transmitted by the camera and crops the images into multiple 265×256 small images. These multiple 265×256 images are used as input to a trained pixel-level classification model to generate three-channel label images for all images. The predicted three-channel label images are then reassembled into a single three-channel label image with the same size as the original image, following the cropping order in step two. The area of ​​damaged pixels on the corn ears in the three-channel label image is calculated in pixels, and the proportion of damaged pixels on the corn ears relative to the corn husks, intact corn ears, and damaged corn ears in the entire image is also calculated. The spliced ​​three-channel label image of damaged corn ears is saved, and the proportion of ear damage in this image is saved sequentially to an Excel spreadsheet. The above steps are repeated until the corn collection box is full of corn.

[0042] The three-channel label image of corn ear damage is used as input to the trained harvest loss prediction model, which then predicts the quality of harvest loss.

[0043] The training steps for the pixel-level classification model are as follows:

[0044] First, we collected sample images of corn harvester ear collection boxes. We then created training samples based on different light intensities and different ear stacking conditions in the collection boxes. The samples were labeled with four categories: corn husks, ear collection box background, intact corn ears, and damaged corn ears. The corresponding label colors were green, black, yellow, and red, respectively.

[0045] The samples were preprocessed on a high-performance computer, including image cropping and enhancement, with random transformations of contrast, saturation, mirroring, and brightness. This expanded the training samples to 3881 images, improving the model's generalizability. Several images were also set up as a test set.

[0046] The FIBA ​​Unet model was trained using preprocessed sample images.

[0047] The trained FIBA ​​Unet model performs pixel-level classification on the test image pairs to determine whether a pixel belongs to an intact corn part, a damaged ear part, a husk, or the background.

[0048] The recognition accuracy, cross-union ratio, average cross-union ratio, and damage percentage of each part are calculated and compared with the manually labeled image to calculate evaluation indicators. Based on these evaluation indicators, parameters are optimized to find the optimal hyperparameter settings.

[0049] By cropping captured images into multiple 256×256 pixel images and applying pixel-level segmentation algorithms, the images of corn ear collection boxes are processed to achieve real-time and accurate detection of corn ear damage. Sampling stops once the collection box is full. The classified images and detection data are used as input to a loss prediction module. The loss prediction model predicts the corn harvest loss quality for this operation, filling the gap in existing methods for pixel-level detection and loss quality prediction of corn ears, and providing an automated loss detection method for corn ear harvesting.

[0050] The reason for cropping the captured image into multiple 256×256 pixel images is that the corn ear harvester has limited computing resources when operating in the field, and the collection box also contains many useless background elements when it is full. In order to save computing resources and improve response time, the original large image is cropped into multiple small images and used as input to the FIBA ​​Unet (Inverted Bottleneck Attention) model for image segmentation. The cropping function uses cv2.imread in Python to read the image, sets the step size to 256 pixels, and uses math.ceil() to set the height and width of the image to 256 pixels. Multiple 256×256 pixel images are extracted by sliding within the set range in the image captured by the installed camera.

[0051] like Figure 2 As shown, in a preferred embodiment of the present invention, the pixel-level classification model is a fused inverted bottleneck attention UNet model, whose encoder is based on an improved EfficientNet b3 model. The improved EfficientNet b3 model is characterized by replacing the first five layers of the EfficientNet b3 model with a fused inverted bottleneck convolutional structure.

[0052] In this embodiment, the pixel-level classification model uses the fused inverted bottleneck attention Unet (FIBAUnet) model, and the model structure is as follows: Figure 2 As shown, the model design uses the improved Efficientnet b3 model as the encoder of FIBAUnet, where the first 5 layers of Efficientnet b3 are replaced with fused inverted bottleneck convolutions to improve the overall feature extraction capability of the model and significantly improve the detection accuracy of the model.

[0053] The 256×265 pixel image is downsampled 8 times and feature enhanced 7 times to generate a 1×1×384 feature vector, which is then passed to the decoder. Compared with the initial Unet combined with Efficientnet b3 structure, this model structure can achieve higher detection accuracy while only slightly increasing the computational cost. To improve the real-time recognition speed of the model, it is processed by TensorRT and INT8 and then deployed on a small AI computer.

[0054] like Figure 3 As shown, in a preferred embodiment of the present invention, the harvest loss prediction model is a twin-channel Gram angle field difference-summation fusion network model. The specific steps of the pre-trained harvest loss prediction model outputting the predicted value of corn ear harvest loss quality based on multiple collected overall label maps and their corresponding damage percentages include:

[0055] Obtain the overall label map and its corresponding damage percentage from multiple acquisitions, and fuse the features of each overall label map and its corresponding damage percentage;

[0056] The fused data is transformed to generate a first type of feature map representing the global distribution structure of the damage and a second type of feature map representing the differences in local details of the damage.

[0057] Feature extraction is performed on the global feature map and the detail feature map using a dual-channel approach to generate a feature map.

[0058] The feature map is fused by adaptive average pooling, flattening and full connection to output the loss quality prediction value.

[0059] In this embodiment, the harvest loss prediction model employs a Siamese dual-channel Gram angle field difference-summation fusion network (Siamese MGAD-Net). The segmentation results of each image are fused with its corresponding damage percentage feature. The fused data undergoes multi-channel Gram angle field (MGA) transformation to generate two types of two-dimensional feature maps: one retains the global structural information of the overall damage distribution during the harvesting process, and the other highlights the detailed differences and local variations between damaged areas. These two types of feature maps are used as inputs to the MGASF and MGADF channels of the Siamese MGAD-Net model for feature extraction. After dual-channel feature extraction, the resulting feature maps undergo adaptive average pooling, flattening, and fully connected mapping. Finally, feature fusion is completed in the fully connected layer, outputting the predicted harvest loss value for the corn ear harvester. The process of processing the input three-channel label map and the corresponding damage percentage is as follows: Figure 3 As shown.

[0060] The training process for the harvest loss prediction model is as follows:

[0061] During the corn harvester operation, images of the inside of the ear collection box were taken at multiple points from the start of operation until the box was full, capturing multiple images of ear damage under different light intensities, ear accumulation morphologies, and harvesting conditions. Simultaneously, at the end of each harvest, ears and kernels scattered on the machine header, conveyor, and in the field were manually inspected and weighed. The total weight of these samples, along with the kernels in the kernel collection box, was used as the regression-predicted true loss value.

[0062] The acquired raw images are cropped and input into a pre-trained FIBA ​​U-Net pixel-level corn ear damage classification network to generate segmentation results for each image. Intact ear regions, damaged ear regions, husks, and background are classified pixel by pixel, and the ear damage percentage for each image is calculated. The obtained segmentation results, along with the corresponding damage percentage features and the loss quality of each harvest, are used as training and testing sets, divided in an 8:2 ratio.

[0063] The training set is fused and used as the basic input data for the Siamese MGAD-Net network. Then, the fused data is subjected to multi-channel Gram-Field (MGAF) transformation to generate global feature maps (MGASF) and detail feature maps (MGADF), which are used as dual-channel inputs to the Siamese network.

[0064] A Siamese MGAD-Net model structure was constructed, in which the feature maps extracted from the two channels were fused using adaptive average pooling, flattening, and full connectivity to output a predicted value for the quality loss during corn harvesting in a single batch. During training, mean squared error (MSE) was used as the loss function, and the parameters were iteratively updated using either the Adam or AdamW optimizer.

[0065] For the validation set, the mean squared error (MSE), mean absolute error (MAE), and coefficient of determination (R²) are calculated between the predicted values ​​and the actual measured values. Based on these evaluation indicators, the network structure and hyperparameters are tuned until the model's prediction accuracy reaches its optimal state.

[0066] In a preferred embodiment of the present invention, the transformation is a multi-channel Gram angle field transformation, the first type of feature map is a Gram angle field summation map, and the second type of feature map is a Gram angle field difference map.

[0067] As a preferred embodiment of the present invention, the difference between the channel processing the first type of feature map and the channel processing the second type of feature map is that the channel processing the first type of feature map includes a multi-dilated convolutional multi-scale feature module, which is used to capture global patterns with different receptive field ranges.

[0068] In this embodiment, the MGASF channel in the Siamese MGAD-Net model structure passes through the Feature Pyramid Network (FPN), the Position Attention Module (PAM), and the Channel Attention Module (CAM) in sequence, and introduces the Multi-Dilated Convolutional Multi-Scale Feature Module (MDCDS) to capture global patterns of different receptive field ranges; the MGASF channel adopts the same FPN, PAM, and CAM structure, but does not contain the MDCDS module, and is used to supplement the detailed information in the global features.

[0069] Corn damage percentage D p The number of damaged corn pixels can be determined by N d Number of pixels in the intact ear of fruit area N w Number of pixels in the bract region N l These three parameters are calculated using the following formula:

[0070] ;

[0071] This invention also provides a deep learning-based corn ear harvester harvest loss prediction system, used to implement the deep learning-based corn ear harvester harvest loss prediction method, the system comprising:

[0072] The acquisition module is used to acquire images of ears of corn in the ear collection box of the corn harvester in real time and crop the ear images into several sub-images of preset sizes.

[0073] The classification module is used to build a pixel-level classification model. Based on the pre-trained pixel-level classification model, it performs pixel-level classification on sub-images and generates a three-channel label map for each sub-image.

[0074] The stitching module is used to stitch together several three-channel label images to form a whole label image with the same size as the original image.

[0075] The percentage calculation module is used to calculate the percentage of pixels with damaged corn ears based on the overall label image.

[0076] The prediction output module is used to build a harvest loss prediction model. When the ear collection box is full, the pre-trained harvest loss prediction model outputs the predicted value of the corn ear harvest loss quality based on the overall label map collected multiple times and its corresponding damage ratio.

[0077] In this embodiment, the system also includes a high-definition camera and a small AI computer. The high-definition camera is used to acquire images of the ear collection box during corn harvesting operations at high frequency, and the small AI computer includes an image pixel-level classification model and a harvest loss prediction model.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting the harvesting loss of a corn ear harvester based on deep learning, characterized by, The method comprises: real-time collection of ear images in a corn harvester ear box, and cutting the ear images into a plurality of sub-images of a preset size; constructing a pixel-level classification model, pixel-level classification of the sub-images based on the pre-trained pixel-level classification model, and generation of a three-channel label map for each sub-image; splicing a plurality of three-channel label maps to form an overall label map consistent with the size of the original image; calculating the proportion of damaged corn ear pixels based on the overall label map; constructing a harvest loss prediction model, and when the ear box is full, outputting a predicted value of the quality of the corn ear harvest loss based on the pre-trained harvest loss prediction model according to the overall label maps collected multiple times and the corresponding damage proportions; the pixel-level classification model is a UNet model with inverted bottle neck attention fusion, and the encoder thereof is based on an improved EfficientNet b3 model, wherein the first five layers of the improved EfficientNet b3 model are replaced with an inverted bottle neck convolution structure with fusion; the harvest loss prediction model is a twin two-channel Gram angle field difference-sum fusion network model.

2. The deep learning-based ear loss prediction method for a corn ear harvester according to claim 1, characterized by, The step of outputting a predicted value of the quality of the corn ear harvest loss based on the pre-trained harvest loss prediction model according to the overall label maps collected multiple times and the corresponding damage proportions comprises: obtaining the overall label maps collected multiple times and the corresponding damage proportions, and fusing the features of each overall label map and the corresponding damage proportion; transforming the fused data to generate a first type of feature map representing the global distribution structure of damage and a second type of feature map representing the local detail difference of damage; extracting features based on the global feature map and the detail feature map to generate a feature mapping; after adaptive average pooling, flattening and full connection fusion, outputting a loss quality prediction value.

3. The deep learning-based ear loss prediction method for a corn ear harvester according to claim 2, characterized by, The transformation is a multi-channel Gram angle field transformation, the first type of feature map is a Gram angle field sum map, and the second type of feature map is a Gram angle field difference map.

4. The deep learning-based ear loss prediction method for a corn ear harvester according to claim 3, characterized by, The system comprises:

5. A deep learning-based ear loss prediction system for a corn ear harvester, for implementing the deep learning-based ear loss prediction method of any one of claims 1-4, characterized in that, an acquisition module for real-time collection of ear images in a corn harvester ear box, and cutting the ear images into a plurality of sub-images of a preset size; a classification module for constructing a pixel-level classification model, pixel-level classification of the sub-images based on the pre-trained pixel-level classification model, and generation of a three-channel label map for each sub-image; a splicing module for splicing a plurality of three-channel label maps to form an overall label map consistent with the size of the original image; a proportion calculation module for calculating the proportion of damaged corn ear pixels based on the overall label map; a prediction value output module for constructing a harvest loss prediction model, and when the ear box is full, outputting a predicted value of the quality of the corn ear harvest loss based on the pre-trained harvest loss prediction model according to the overall label maps collected multiple times and the corresponding damage proportions. ​

Images