A deep learning-based blockage detection method for corn harvester cleaning mechanism

Abstract

The present invention discloses a deep learning-based method for detecting blockage in the cleaning mechanism of a corn harvester, which relates to the field of intelligent detection technology for agricultural machinery. The method comprises the following steps: obtaining a target image of the cleaning mechanism of the corn harvester to be detected, inputting the target image into a pre-trained model to obtain a detection result corresponding to the target image, and determining the blockage status of the corn harvester to be detected based on the detection result; the training process of the pre-trained model comprises the following steps: pre-processing training samples to obtain target training samples; processing the target training samples using a Fast R-CNN model, optimizing the parameters of the Fast R-CNN model based on test samples and a loss function to generate a target model; and outputting the target model as a pre-trained model when the target model meets preset requirements. Combining image preprocessing with a target detection algorithm, the present invention can quickly identify blocked areas and trigger an alarm, effectively improving the intelligent level of machine operation.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection technology for agricultural machinery, and in particular to a deep learning-based blockage detection method for a cleaning mechanism of a corn harvester. Background Art

[0002] With the rapid development of agricultural mechanization and intelligentization, corn harvesters are playing an increasingly important role in modern agricultural production. However, during the corn harvest process, clogging of the cleaning mechanism remains a major challenge for agricultural machinery operators. This is particularly true in high-yield, densely planted fields, where corn stalks and kernels accumulate, making it highly susceptible to clogging. This clogging not only disrupts the continuity of harvesting operations but can also lead to machine wear and tear, increasing repair and maintenance costs, and severely impacting the efficiency and quality of the corn harvest.

[0003] Currently, corn harvesters typically use screens, fans, and other devices to clean corn kernels, such as screens, transmission components, and impurity removal devices. However, these cleaning mechanisms often rely on fixed operating parameters. While some machines have basic cleaning functions, clogging, screen blockage, and high power consumption remain difficult to avoid in complex and changing operating environments. Furthermore, traditional machine designs lack the ability to monitor operating status in real time, making it difficult to respond and adjust dynamically when a blockage occurs, failing to meet the demands of modern agriculture for intelligent and efficient operations.

[0004] In recent years, deep learning technology has achieved breakthroughs in image recognition and intelligent monitoring, providing technical support for intelligent detection and early warning systems for agricultural machinery. A deep learning-based congestion detection system uses image acquisition devices to monitor the operating status of the cleaning mechanism in real time, accurately identifying and locating congestion areas, and providing operators with effective early warning alerts. Applying deep learning models to corn harvester cleaning mechanism congestion detection not only improves the accuracy and real-time performance of congestion detection but also effectively reduces downtime caused by congestion, thereby improving machine operating efficiency, further enhancing harvest quality, and further enhancing the intelligence level of harvesting machinery.

[0005] Currently, Chinese invention patent application CN 221152085 U proposes a cleaning mechanism comprising a screen plate and a cleaning element. This mechanism aims to improve the screen plate's impurity removal efficiency through a transmission assembly and impurity removal device. However, this design focuses on improving cleaning efficiency and does not address intelligent processing for blockage detection. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and specifically provide a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning, as follows:

[0007] 1) In the first aspect, the present invention provides a method for detecting blockage in a cleaning mechanism of a corn harvester based on deep learning. The specific technical solution is as follows:

[0008] Obtain a target image of a cleaning mechanism of a corn harvester to be inspected, input the target image into a pre-trained model to obtain a detection result corresponding to the target image, and determine a blockage condition of the corn harvester to be inspected based on the detection result;

[0009] The training process of the pre-trained model is as follows:

[0010] Obtaining training samples, and preprocessing the training samples to obtain target training samples;

[0011] The target training samples are processed by the Fast R-CNN model, and the parameters of the Fast R-CNN model are optimized in combination with the test samples and the loss function to generate a target model. When the target model meets the preset requirements, the target model is output as the pre-trained model.

[0012] The beneficial effects of the deep learning-based corn harvester cleaning mechanism blockage detection method provided by the present invention are as follows:

[0013] First, the use of the Fast R-CNN model significantly improves detection efficiency and accuracy, rapidly processing images and accurately locating blocked areas within the sorting mechanism. This approach delivers faster detection and higher accuracy than traditional methods. Second, this solution automates detection, reducing manual intervention and labor costs. It also enables real-time monitoring of the sorting mechanism's operational status, enabling timely detection and resolution of blockages. Furthermore, the use of a pre-trained model leverages the feature representation capabilities of large-scale image data. Combined with a large number of training samples and optimized parameters, the model possesses strong generalization capabilities, adapting to images from diverse environments and operating conditions, and reducing detection errors. Furthermore, by accurately detecting and promptly addressing blockages, it effectively reduces sorting loss and trash content, improves corn harvester performance, optimizes control strategies, and enhances overall operational quality. Finally, this solution demonstrates excellent technical versatility and scalability. The Fast R-CNN model can be extended to other agricultural machinery fault detection or object recognition tasks, adapting to a wide range of scenarios and possessing broad application prospects.

[0014] Based on the above solution, the present invention can also be improved as follows.

[0015] Furthermore, the pre-processing includes:

[0016] Grayscale processing is performed on the training images in the training sample to obtain a grayscale image corresponding to each training image;

[0017] Performing smoothing filtering on each grayscale image to obtain a first image corresponding to each grayscale image;

[0018] Perform histogram equalization processing on each first image to obtain a second image corresponding to each first image.

[0019] Furthermore, the construction process of the Fast R-CNN model is as follows:

[0020] Adjust the convolutional layer structure of the Fast R-CNN basic model and set the filter size of each convolutional layer according to the requirements of blockage detection;

[0021] A pooling layer, a region proposal network, and a classification layer are added to the Fast R-CNN base model to generate the Fast R-CNN model.

[0022] Furthermore, when the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows:

[0023] The detection accuracy of the target model is calculated, and when the detection accuracy reaches a preset accuracy, it is determined that the target model meets the preset requirements.

[0024] 2) In a second aspect, the present invention also provides a corn harvester cleaning mechanism blockage detection system based on deep learning. The specific technical solution is as follows:

[0025] The detection module is used to: obtain a target image of the cleaning mechanism of the corn harvester to be detected, input the target image into the pre-trained model to obtain a detection result corresponding to the target image, and determine the blockage condition of the corn harvester to be detected based on the detection result;

[0026] The training process of the pre-trained model is as follows:

[0027] Obtaining training samples, and preprocessing the training samples to obtain target training samples;

[0028] The target training samples are processed by the Fast R-CNN model, and the parameters of the Fast R-CNN model are optimized in combination with the test samples and the loss function to generate a target model. When the target model meets the preset requirements, the target model is output as the pre-trained model.

[0029] Based on the above solution, the present invention can also be improved as follows.

[0030] Furthermore, the pre-processing includes:

[0031] Grayscale processing is performed on the training images in the training sample to obtain a grayscale image corresponding to each training image;

[0032] Performing smoothing filtering on each grayscale image to obtain a first image corresponding to each grayscale image;

[0033] Perform histogram equalization processing on each first image to obtain a second image corresponding to each first image.

[0034] Furthermore, the construction process of the Fast R-CNN model is as follows:

[0035] Adjust the convolutional layer structure of the Fast R-CNN basic model and set the filter size of each convolutional layer according to the requirements of blockage detection;

[0036] A pooling layer, a region proposal network, and a classification layer are added to the Fast R-CNN base model to generate the Fast R-CNN model.

[0037] Furthermore, when the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows:

[0038] The detection accuracy of the target model is calculated, and when the detection accuracy reaches a preset accuracy, it is determined that the target model meets the preset requirements.

[0039] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0040] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0041] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0043] Figure 1This is a flow chart of a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention;

[0044] Figure 2 This is a second flow chart of a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention;

[0045] Figure 3 Schematic diagram of a Faster R-CNN neural network model based on ResNet-50 for a corn harvester cleaning mechanism blockage detection method based on deep learning in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram showing a corn cob blockage condition on a fish-scale screen surface during a cleaning process of a corn harvester cleaning mechanism blockage detection method based on deep learning according to an embodiment of the present invention;

[0047] Figure 5 This is a second schematic diagram of a corn cob shaft blockage on a fish-scale screen surface during a cleaning process of a corn harvester cleaning mechanism blockage detection method based on deep learning according to an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the installation position of an image acquisition system for a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention;

[0049] Figure 7 This is a schematic diagram of a blockage detection system for a corn harvester cleaning mechanism blockage detection method based on deep learning according to an embodiment of the present invention;

[0050] Figure 8 (a) is a schematic diagram of an original image of a mechanism blockage in a method for detecting blockage of a corn harvester cleaning mechanism based on deep learning according to an embodiment of the present invention;

[0051] Figure 8 (b) is a schematic diagram of an annotated image of a mechanism blockage in a method for detecting blockage of a corn harvester cleaning mechanism based on deep learning according to an embodiment of the present invention;

[0052] Figure 8 (c) is a grayscale image schematic diagram of a method for detecting blockage of a corn harvester cleaning mechanism based on deep learning according to an embodiment of the present invention;

[0053] Figure 8 (d) is a schematic diagram of an image after nonlinear grayscale transformation of a corn harvester cleaning mechanism blockage detection method based on deep learning according to an embodiment of the present invention;

[0054] Figure 8(e) is a schematic diagram of an image after smoothing filtering of a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention;

[0055] Figure 8 (f) is a schematic diagram of an image after histogram equalization of a corn harvester cleaning mechanism blockage detection method based on deep learning according to an embodiment of the present invention;

[0056] Figure 9 This is a schematic diagram of the results of the test phase of a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention;

[0057] Figure 10 This is a structural framework diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0059] like Figure 1 As shown, a method for detecting blockage of a corn harvester cleaning mechanism based on deep learning according to an embodiment of the present invention includes the following steps:

[0060] S1, obtaining a target image of a cleaning mechanism of a corn harvester to be inspected, inputting the target image into a pre-trained model to obtain a detection result corresponding to the target image, and determining a blockage condition of the corn harvester to be inspected based on the detection result;

[0061] The training process of the pre-trained model is as follows:

[0062] Obtaining training samples, and preprocessing the training samples to obtain target training samples;

[0063] The target training samples are processed by the Fast R-CNN model, and the parameters of the Fast R-CNN model are optimized in combination with the test samples and the loss function to generate a target model. When the target model meets the preset requirements, the target model is output as the pre-trained model.

[0064] The beneficial effects of the deep learning-based corn harvester cleaning mechanism blockage detection method provided by the present invention are as follows:

[0065] First, the use of the Fast R-CNN model significantly improves detection efficiency and accuracy, rapidly processing images and accurately locating blocked areas within the sorting mechanism. This approach delivers faster detection and higher accuracy than traditional methods. Second, this solution automates detection, reducing manual intervention and labor costs. It also enables real-time monitoring of the sorting mechanism's operational status, enabling timely detection and resolution of blockages. Furthermore, the use of a pre-trained model leverages the feature representation capabilities of large-scale image data. Combined with a large number of training samples and optimized parameters, the model possesses strong generalization capabilities, adapting to images from diverse environments and operating conditions, and reducing detection errors. Furthermore, by accurately detecting and promptly addressing blockages, it effectively reduces sorting loss and trash content, improves corn harvester performance, optimizes control strategies, and enhances overall operational quality. Finally, this solution demonstrates excellent technical versatility and scalability. The Fast R-CNN model can be extended to other agricultural machinery fault detection or object recognition tasks, adapting to a wide range of scenarios and possessing broad application prospects.

[0066] It should be noted that if Figures 2 to 9 As shown in Figure 2, the process of obtaining training samples is as follows:

[0067] (1) Install the camera at a key position of the cleaning mechanism. The location should be selected so that the key operating area can be directly observed, such as the cleaning fan outlet, screen area or conveying channel. Ensure that the camera's field of view can fully cover the key components of the cleaning mechanism and avoid blind spots. Select a high-resolution industrial camera to ensure that the image clarity meets the detection requirements. Use a stable bracket or mounting assembly to fix the camera to prevent displacement during mechanical vibration.

[0068] (2) Use the camera to collect real-time images. Perform image collection at different stages, covering normal operating conditions and various possible blocking conditions, to ensure that the collection process covers different environmental conditions and mechanical operating parameters. To solve the lighting problem, the camera is equipped with a fill light to ensure that clear images can still be obtained in low-light environments. The collected images are stored in real time in the local storage device and classified and managed by status and timestamp for subsequent processing;

[0069] (3) Divide the collected image data into training samples and test sets in a ratio of 8:2;

[0070] (4) In the training dataset, the blocked areas were manually labeled using the Image Labeler tool in MATLAB to ensure that the labeled boxes accurately covered the blocked areas. Normal images did not need to be labeled and were treated as unblocked samples.

[0071] After determining the training samples, they need to be preprocessed. The specific process is as follows:

[0072] (1) The grayscale operation is used to convert the original RGB image into a grayscale image in order to reduce the computational complexity of the data and highlight the features of the blocked area. The grayscale operation can be implemented by the function convert_to_gray(). The calculation formula is:

[0073]

[0074] Among them, R(i,j)R(i,j)R(i,j), G(i,j)G(i,j)G(i,j), and B(i,j)B(i,j)B(i,j) respectively represent the red, green, and blue channel values ​​of the image at position (i,j)(i,j)(i,j).

[0075] (2) The grayscale transformation operation is used to adjust the grayscale value of the grayscale image according to nonlinear rules to increase the dynamic range of the image and enhance the contrast. The grayscale transformation can be implemented by the function nonlinear_gray_transform(), and the calculation formula is:

[0076]

[0077] This formula adjusts pixel values ​​to increase contrast and highlight features in occluded areas.

[0078] (3) The smoothing filter operation is used to denoise the image after grayscale transformation to reduce noise interference and improve image clarity. The smoothing filter can be implemented by the function apply_smoothing_filter(), using a 3x3 mean filter, and its formula is:

[0079]

[0080] (4) This filter takes the average value of the 3x3 neighborhood of each pixel in the image to smooth the image and reduce noise. The histogram equalization operation is used to perform global contrast enhancement on the smoothed filtered image. The equalization operation can be implemented by the function enhance_contrast_histogram(). The calculation steps are as follows:

[0081] 1. Calculate the cumulative distribution function (CDF) of the grayscale image and normalize it.

[0082] 2. Use the normalized CDF to map each pixel value. The calculation formula is:

[0083]

[0084] Wherein, CDF is the cumulative distribution function, min CDF is the minimum value of CDF, which is used to normalize the pixel value to the range of 0-255.

[0085] The above steps implement image preprocessing in sequence through different image processing functions, providing clear image input for the target detection module and ensuring efficient recognition of blocked areas.

[0086] It should be further explained that before grayscale processing, it is necessary to first count the amount of training sample data and the clarity and brightness diversity of each training image in the training sample. The specific process is as follows:

[0087] Since the image acquisition area in this solution includes multiple locations such as the cleaning fan outlet, screen, and conveyor channel, the sample sizes corresponding to each type of location in the cleaning fan outlet, screen, and conveyor channel are similar or equal when performing data volume statistics for the training samples. In other words, it is determined whether the first sample size corresponding to each type of location is within the preset sample size range. If so, there is no need to expand the training samples, and the clarity and brightness diversity can be directly judged. If not, it is necessary to expand the training samples corresponding to the target locations that are not within the preset sample size range.

[0088] The specific process of expanding the training samples is as follows:

[0089] Determine a target clarity range (preset range) and a target brightness range (preset range) corresponding to this type of position, divide the target clarity range and the target brightness range to obtain a plurality of sub-clarity ranges and a plurality of sub-brightness ranges, arrange and combine the plurality of sub-clarity ranges and the plurality of sub-brightness ranges to generate a range pair consisting of a sub-clarity range and a sub-brightness range, assign a weight to each range pair, determine a first preset sample size corresponding to each range pair based on the median of the preset sample size range corresponding to this type of position, calculate the clarity and brightness of each sample image in the first sample size corresponding to this type of position, and correspond any sample image to a target sub-clarity range based on the clarity and brightness of each sample image. and a target sub-brightness range, and determine the target range pair corresponding to the sample image according to the target sub-clarity range and the target sub-brightness range, determine the target number of samples in each range pair according to the first sample amount and the weight corresponding to each range pair, determine the standard number of samples in each range pair according to the standard sample amount corresponding to the type of position and the weight corresponding to each range pair, determine the difference between the standard number and the target number under the same range pair, and randomly select random image parameters consisting of random clarity and random brightness corresponding to the number of differences within the range pair according to the difference, and randomly extract a training sample from the first sample amount, change the clarity and brightness of the training sample to random image parameters, generate a new training image and put it into the training sample.

[0090] The process of dividing the target clarity range and the target brightness range is specifically as follows:

[0091] From all training images corresponding to this type of position, a first image set is selected, and features are extracted from the first image set by manual annotation. A detection frame corresponding to each training image in the first image set is generated. By adjusting the clarity, the clarity corresponding to more than 60% of the detection frames is determined, and this clarity is determined as the first reference clarity for dividing the target clarity range. Based on the first reference clarity, the position of the first reference clarity within the target clarity range is determined. When this position is between the minimum value and the median of the target clarity range, a second reference clarity is determined at half the difference between the first reference clarity and the maximum value of the target clarity range. The target clarity range is divided based on the first reference clarity and the second reference clarity. The brightness division method is the same as the clarity division method and will not be repeated here.

[0092] The specific process of directly judging the diversity of clarity and brightness is as follows:

[0093] It is determined whether all training images corresponding to any type of position fall within all range pairs. If not, the sample diversity processing of this type of position is improved by expanding the training samples.

[0094] The training process of the pre-trained model is as follows:

[0095] (1) The Faster R-CNN model based on ResNet-50 was selected as the base model, and the weights pre-trained on the COCO dataset were loaded to improve the initial feature extraction capability and training efficiency. The first four convolutional residual blocks of the base network ResNet-50 remained unchanged, and the fifth residual block was adjusted to enhance the feature extraction capability;

[0096] (2) Adjust the convolutional layer structure and add five convolutional layers. Add convolutional units to improve the model's ability to capture blocked area features. Change the CONV5 residual block of ResNet-50 to a five-layer convolutional structure, arranged as conv5_1, relu5_1, conv5_2, relu5_2, conv5_3, relu5_3, conv5_4, relu5_4, pool5; set the filter sizes of convolutional units conv5_1, conv5_2, conv5_3, and conv5_4 to [3,3,3,64], [3,3,64,64], [3,3,64,128], [3,3,128,256], and [3,3,256,512], respectively, with a step size of 1.

[0097] (3) Add a pooling layer after each convolution module, and arrange the pooling layer into pool5_1, pool5_2, pool5_3, pool5_4, pool5_5; set the pooling size of pool5 to [2, 2] and the step size to 2 to reduce the feature map size and extract key features;

[0098] (4) Add the region proposal network (RPN) to generate candidate region boxes to identify possible blocked areas. RPN operates after the output of the convolution layer, generates candidate boxes through sliding window convolution and anchor boxes, and further screens and adjusts the positions of these boxes through classification and regression. The filter size of the RPN sliding window convolution layer is set to [3,3,512,512] with a step size of 1. The convolution layer generates multiple candidate boxes for locating possible blocked areas; the size of the anchor box is set to [32,64,128,256,512] with an aspect ratio of [0.5, 1.0, 2.0]. Anchor boxes of various sizes and ratios are used to capture blocked areas of different scales and shapes; the IoU threshold for screening candidate boxes is IoU ≥ 0.7 for positive samples and IoU < 0.3 for negative samples. Positive and negative samples are screened based on the overlap with the true box.

[0099] (5) Add a classification layer to the RPN backend to classify the generated candidate boxes into blocked areas and non-blocked areas, and output the final detection results. After the RPN output, a classification layer and a bounding box regression layer are added: the classification layer is used to classify each candidate box as "blocked area" or "non-blocked area", the fully connected layer is set to fc_cls, and the filter size is [7,7,512,2]. This layer outputs two types of labels (blocked and non-blocked); the bounding box regression layer is used to adjust the position of the candidate box so that it can more accurately frame the real blocked area. The bounding box regression layer is used to adjust the position of the candidate box. The fully connected layer is set to fc_bbox and the filter size is [7,7,512,4]. This layer outputs four regression values, which represent the adjustment amount of the boundary coordinates of the candidate box (left, right, top, and bottom). After classification and regression processing, the model outputs the final detection result, including the detection category ("blocked area" or "non-blocked area") and bounding box coordinates (providing accurate bounding box coordinates for each candidate box to ensure the correct positioning of the blocked area). This process completes the accurate detection of the blocked area, helping the real-time recognition and early warning system to improve work efficiency and accuracy.

[0100] The optimization process of the pre-trained model is as follows:

[0101] (1) Initialize the model weights and biases, using the parameters of the pre-trained model to speed up convergence. The convolutional and fully connected layers of the Faster R-CNN model use the weights pre-trained on the ImageNet dataset; the convolutional layer weights are initialized with He (for the ReLU activation function) to avoid gradient explosion or gradient vanishing; the bias is initialized to 0; this allows the model to converge quickly; the He initialization calculation formula is:

[0102] , n in is the number of input neurons;

[0103] (2) The labeled training data set is input into the target detection model, and candidate boxes are generated and labeled by RPN. The labeled model is input into the target detection model after image data preprocessing (grayscale, grayscale transformation, smoothing filtering and histogram equalization) in step 2; RPN generates candidate boxes from the input image, generates multiple candidate regions of different sizes based on the feature map of the image (extracted by the convolution layer), and evaluates whether the region contains the target; the candidate box is labeled according to the degree of overlap (IoU) with the actual target position, and IoU ≥ 0.7 is a positive sample, indicating inclusion, and IoU < 0.3 is a negative sample, indicating exclusion;

[0104] (3) Using the cross entropy loss function, the output results are compared with the labeled results through the Softmax classification algorithm. The cross entropy loss function is used to measure the gap between the category predicted by the model and the true category. The formula is: ,

[0105] y i is the true label, p i is the probability distribution predicted by the model, and N is the number of samples;

[0106] (4) Use batch stochastic gradient descent to perform error backpropagation and gradually adjust the weights and biases of the model. The core of the training process is error backpropagation. By transferring the gradient information of the loss function from the output layer to the input layer, the model can gradually adjust its weights and biases so that the error is gradually reduced. The batch stochastic gradient descent (SGD) method is used to optimize the model parameters. In each iteration, the model randomly selects a batch of data from the training sample for training, calculates the gradient based on the error of this batch, and updates the weights and biases of the model. The update rule is:

[0107] ,

[0108] θ is the model parameter, η is the learning rate, ∇ θ J(θ) is the gradient of the loss function with respect to the parameters;

[0109] (5) Repeat steps (2)-(4) for multiple iterations of the training samples until the model's detection accuracy meets the requirements. In order to allow the model to fully learn the characteristics of the training data, the number of training epochs is set to 100. Each round of training traverses the entire training sample. The model updates the parameters after each round to improve the accuracy. The training stop conditions are set, where the specific condition parameters are: mAP ≥ 0.7, Precision ≥ 0.85, Recall ≥ 0.80, F1-score ≥ 0.80, and IoU ≥ 0.7.

[0110] The testing and verification process of the pre-trained model is as follows:

[0111] (1) Input the images in the test dataset into the trained object detection model to verify the detection effect of the model. The test set data is input into the model, and the test set data is not involved in the training; through the input, the model generates prediction results based on the trained weights and parameters, including the location and category of each candidate box (blocked area or non-blocked area); generate bounding boxes on the test image and mark the classification results of each candidate box. For the blockage detection task, the model should be able to effectively identify the blockage area and accurately mark its location;

[0112] (2) Generate candidate boxes in RPN and identify blocked areas based on model output. RPN uses a sliding window mechanism to operate on each test image to generate multiple candidate area boxes, representing the area that the model believes may contain the target (blocked area); Based on the candidate boxes, filter the candidate boxes according to the IoU threshold (IoU ≥ 0.7) and Anchor box size ([32, 64, 128, 256, 512], aspect ratio [0.5, 1.0, 2.0]) set in step 3.4. Valid candidate boxes will enter the next classification process; the classification layer will classify each candidate box to determine whether it is a blocked area. If a blocked area is detected, the model will provide the location information and category label of the area;

[0113] (3) Calculate the detection accuracy and recall rate of the model, and judge the model performance based on the test results. Calculate the accuracy (Precision): The accuracy measures the proportion of candidate boxes predicted by the model as blocked areas that are actually blocked. Calculate the recall rate (Recall): The recall rate indicates the proportion of actual blocked areas that the model can identify. The formula is:

[0114]

[0115] Use mAP (Mean Average Precision) to comprehensively evaluate the model's detection performance on different categories to ensure a balance between detection precision and recall. If the mAP value is low, further model optimization is required.

[0116] (4) If the detection accuracy is insufficient, return to step 4 for further optimization until the detection requirements are met. By calculating the precision and recall rate of the test data, determine whether the model has reached the set performance standard. The standard reference step 4.5, mAP ≥ 0.7, Precision ≥ 0.85, Recall ≥ 0.80, F1-score ≥ 0.80, IoU ≥ 0.7, if the precision and recall rate do not reach the expected target, it means that the model has not yet achieved the ideal effect; further optimization of the model includes: adjusting hyperparameters, data enhancement, improving network structure, increasing training samples, etc.; repeat training and optimization: return to step 4, retrain, adjust the model parameters, and repeatedly verify the performance of the model until the precision and recall rate of the model meet expectations, and training can be stopped.

[0117] This solution also includes: embedding the trained blockage detection model into the corn harvester's embedded system, using the embedded system's computing module to analyze the images captured by the camera in real time to identify the blockage status of the cleaning mechanism; triggering an alarm signal when the number of blockages reaches the set alarm threshold, providing real-time warning to the operator. The specific process is as follows:

[0118] (1) Model integration into embedded systems. Convert the trained object detection model into a format compatible with embedded systems using PyTorch Mobile; deploy the model to the embedded computing device of the corn harvester; configure the hardware interface of the embedded system to ensure seamless connection with the camera and alarm system;

[0119] (2) Real-time image input and detection. The embedded system receives real-time images captured by the camera, pre-processes them and inputs them into the detection model. The model is used to identify the blockage status of the cleaning mechanism and detect the existence and location of the blockage area in real time.

[0120] (3) Real-time output and alarm. The embedded system computing module generates detection results in real time and judges the congestion status. When a congestion area is detected, the embedded system makes a real-time judgment based on the set alarm threshold and triggers the corresponding alarm signal: 1) Mild congestion: When the congestion area detected per minute reaches more than 1.5%, the system triggers a warning signal and prompts the operator to check the cleaning mechanism through the display screen to prevent further congestion. 2) Moderate congestion: When the congestion area detected per minute reaches more than 6%, the system triggers a warning signal to prevent serious congestion. 3) Severe congestion: When the congestion area detected reaches more than 14%, the system issues a high-priority alarm to remind the operator to deal with the congestion problem immediately to avoid serious impact on the normal operation of the cleaning mechanism. 4) Proportion of congestion area: When the proportion of the detected congestion area in the image area exceeds 20%, the system triggers an emergency alarm and requires the operator to immediately stop the machine for cleaning to prevent further damage or failure of the mechanical equipment.

[0121] The present invention has the following beneficial effects:

[0122] This invention uses a deep learning-based image processing method to accurately detect blockages in the cleaning mechanism of a corn harvester in real time. By capturing operational images with a camera and combining image preprocessing with target detection algorithms, it can quickly identify blocked areas and trigger an alarm, effectively improving the intelligent operation of the machine.

[0123] Based on the proposed detection method, this paper developed a blockage detection system. The system supports flexible adjustment of algorithm parameters and adapts to various complex operating scenarios. This system can be embedded in the embedded platform of corn harvesters and deployed in real time for production operations, providing technical support for improving the efficiency and reliability of harvesting machinery and reducing downtime losses caused by blockages.

[0124] Furthermore, the pre-processing includes:

[0125] Grayscale processing is performed on the training images in the training sample to obtain a grayscale image corresponding to each training image;

[0126] Performing smoothing filtering on each grayscale image to obtain a first image corresponding to each grayscale image;

[0127] Perform histogram equalization processing on each first image to obtain a second image corresponding to each first image.

[0128] Furthermore, the construction process of the Fast R-CNN model is as follows:

[0129] Adjust the convolutional layer structure of the Fast R-CNN basic model and set the filter size of each convolutional layer according to the requirements of blockage detection;

[0130] A pooling layer, a region proposal network, and a classification layer are added to the Fast R-CNN base model to generate the Fast R-CNN model.

[0131] Furthermore, when the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows:

[0132] The detection accuracy of the target model is calculated, and when the detection accuracy reaches a preset accuracy, it is determined that the target model meets the preset requirements.

[0133] Example 1, as Figure 2 and Figure 3 As shown, the specific content includes the following steps:

[0134] Step 1: Image acquisition and classification. Collect images of the cleaning mechanism in normal and blocked states, and divide the image data into training samples and test sets.

[0135] Step 2: Image data preprocessing: Perform grayscale conversion, grayscale transformation, smoothing filtering, and histogram equalization on the images in the training sample to improve the image clarity and contrast.

[0136] Step 3: Build an object detection model. Use Fast R-CNN as the object detection model to mark and detect blocked areas in the image.

[0137] Step 4: Model training and optimization: Use the labeled training samples to train the Fast R-CNN model and adjust the model parameters to optimize detection performance.

[0138] Step 5: Model testing and verification. Use the test set to verify the trained model to ensure the detection accuracy and stability of the model.

[0139] Step 6: Integrate real-time monitoring and early warning functions. Deploy the trained model in the cleaning system to achieve real-time detection and alarm of congestion.

[0140] Furthermore, in step 1, a high-resolution industrial camera installed at a key position of the cleaning mechanism is used to collect images of the cleaning mechanism in normal operation and blocked state in real time, such as Figure 8 As shown in (a). The collected images are divided into training dataset and test dataset, and the blocked areas in the training dataset are manually annotated to provide supervised learning data for subsequent model training;

[0141] Furthermore, the camera should be fixed at the key position of the cleaning mechanism; the real-time collected images should cover a variety of states and environments to improve the generalization ability of the model; manual labeling is completed using the Image Labeler tool in MATLAB, and the labeling box accurately covers the blocked area, such as Figure 8 (b)

[0142] Furthermore, in step 2, the image in the training sample is input into the image preprocessing module, and a series of enhancement operations are performed on it to improve the image quality and recognizability, providing high-quality input data for subsequent target detection; the preprocessing module includes: first, grayscale conversion: through the function convert_to_gray(), the RGB image is converted into a grayscale image to reduce data complexity, such as Figure 8 (c) As shown; Secondly, grayscale transformation: Through the function nonlinear_gray_transform(), the grayscale value is nonlinearly adjusted to expand the dynamic range of the image, as shown in Figure 8 (d) As shown; Then, smoothing filtering: Use the function apply_smoothing_filter() to perform 3×3 mean filtering on the image to reduce noise interference, as shown in Figure 8 (e) As shown; Finally, histogram equalization: through the function enhance_contrast_histogram(), the gray value distribution of the image is redistributed to improve the contrast, as shown in Figure 8 (f)

[0143] Furthermore, the target detection model constructed by the present invention in step 3 is based on Faster R-CNN and is optimized as follows: first, the pre-trained model is loaded: the Faster R-CNN pre-trained on the COCO dataset is used as the base model to improve the initial feature extraction capability and training efficiency; second, the convolution layer is adjusted: the convolution layer of the base model is adjusted, and a 5-layer convolution structure (conv5_1 to conv5_4 and pool5) is added to enhance the feature extraction capability of the blocked area; then, the region proposal network (RPN): the candidate region box is generated by sliding the window, and the candidate box is screened by classification and regression, with IoU ≥ 0.7 as a positive sample and IoU < 0.3 as a negative sample; finally, the classification and regression layer: a fully connected classification layer (fc_cls) and a bounding box regression layer (fc_bbox) are added to classify the candidate box (blocked / non-blocked) and adjust its position;

[0144] Furthermore, the model training and optimization process in step 4 mainly includes: first, initializing the model parameters: using pre-trained weights and He initialization of the convolutional layer to avoid gradient explosion or vanishing; second, supervised learning: using the cross-entropy loss function to calculate the classification error, and optimizing the model through the Softmax classifier; finally, gradient descent optimization: using the batch stochastic gradient descent (SGD) method to adjust the weights and biases and iteratively optimize the model performance; after multiple rounds of iterative training (epochs set to 100), the model performance is stable and meets the expected accuracy requirements;

[0145] Furthermore, step 5 inputs the test data set into the trained model for performance verification, specifically including: first, inputting the test data: generating candidate boxes through RPN and identifying possible blocked areas; second, performance evaluation: calculating the model's precision (Precision), recall (Recall), and mean average precision (mAP); finally, performance optimization: if the detection performance does not meet the requirements, return to step 4 to adjust the model parameters until the detection accuracy (Precision ≥ 0.85) and recall (Recall ≥ 0.80) meet the requirements;

[0146] Furthermore, step six embeds the optimized object detection model into the corn harvester's embedded system, enabling real-time monitoring and alarming of blockages. The trained model is converted to an embedded system-compatible format using PyTorch Mobile and deployed to the harvester's embedded computing device. The system receives real-time camera images and analyzes the blockage status of the cleaning mechanism. When a blockage is detected, the embedded system triggers an audible and visual alarm based on the set alarm threshold, providing the operator with a real-time warning prompt to inspect and address the blockage, ensuring efficient operation of the harvester.

[0147] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the scope of protection of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0148] The present invention also provides a corn harvester cleaning mechanism blockage detection system based on deep learning, and the specific technical solution is as follows:

[0149] The detection module is used to: obtain a target image of the cleaning mechanism of the corn harvester to be detected, input the target image into the pre-trained model to obtain a detection result corresponding to the target image, and determine the blockage condition of the corn harvester to be detected based on the detection result;

[0150] The training process of the pre-trained model is as follows:

[0151] Obtaining training samples, and preprocessing the training samples to obtain target training samples;

[0152] The target training samples are processed by the Fast R-CNN model, and the parameters of the Fast R-CNN model are optimized in combination with the test samples and the loss function to generate a target model. When the target model meets the preset requirements, the target model is output as the pre-trained model.

[0153] It should be noted that the beneficial effects of the corn harvester cleaning mechanism blockage detection system based on deep learning provided by the above embodiment are the same as the beneficial effects of the corn harvester cleaning mechanism blockage detection method based on deep learning, which will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0154] like Figure 10 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, which is coupled to a memory 310. The memory 310 stores at least one computer program 330. The at least one computer program 330 is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above methods. Specifically:

[0155] The electronic device 300 may vary significantly due to different configurations or performance, and may include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, which is loaded and executed by the one or more processors 320 to enable the electronic device 300 to implement the deep learning-based corn harvester cleaning mechanism blockage detection method provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and input / output interfaces for input and output. The electronic device 300 may also include other components for implementing device functions, which are not detailed here.

[0156] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0157] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0158] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the above methods.

[0159] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and to define a specific order or precedence. Where appropriate, the order used for similar objects may be interchanged, such that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0160] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented as a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0161] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0162] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A corn harvester cleaning mechanism blockage detection method based on deep learning, characterized in that: include: Obtain a target image of a cleaning mechanism of a corn harvester to be inspected, input the target image into a pre-trained model to obtain a detection result corresponding to the target image, and determine a blockage condition of the corn harvester to be inspected based on the detection result; The training process of the pre-trained model is as follows: Obtaining training samples, and preprocessing the training samples to obtain target training samples; The target training sample is processed by the Fast R-CNN model, and the parameters of the Fast R-CNN model are optimized in combination with the test sample and the loss function to generate a target model. When the target model meets the preset requirements, the target model is output as the pre-trained model; The construction process of the Fast R-CNN model is as follows: According to the requirements of blockage detection, the convolutional layer structure of the Fast R-CNN basic model was adjusted. The Faster R-CNN model based on ResNet-50 was selected as the basic model. The convolutional layer structure was adjusted, and the CONV5 residual block of ResNet-50 was changed to a five-layer structure, arranged as conv5_1, relu5_1, conv5_2, relu5_2, conv5_3, relu5_3, conv5_4, relu5_4, pool5; and the filter size of each convolutional layer was set; The Fast R-CNN model is generated by adding a pooling layer, a region proposal network, and a classification layer to the Fast R-CNN base model.

2. The method for detecting blockage of a corn harvester cleaning mechanism based on deep learning according to claim 1, characterized in that: The pretreatment includes: Grayscale processing is performed on the training images in the training sample to obtain a grayscale image corresponding to each training image; Performing smoothing filtering on each grayscale image to obtain a first image corresponding to each grayscale image; Perform histogram equalization processing on each first image to obtain a second image corresponding to each first image.

3. The method for detecting blockage of a corn harvester cleaning mechanism based on deep learning according to claim 1, characterized in that: When the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows: The detection accuracy of the target model is calculated, and when the detection accuracy reaches a preset accuracy, it is determined that the target model meets the preset requirements.

4. A corn harvester cleaning mechanism blockage detection system based on deep learning, using the corn harvester cleaning mechanism blockage detection method based on deep learning as claimed in claim 1, characterized in that: The system comprises: The detection module is used to: obtain a target image of the cleaning mechanism of the corn harvester to be detected, input the target image into the pre-trained model to obtain a detection result corresponding to the target image, and determine the blockage condition of the corn harvester to be detected based on the detection result; The training process of the pre-trained model is as follows: Obtaining training samples, and preprocessing the training samples to obtain target training samples; The target training samples are processed by the Fast R-CNN model, and the parameters of the Fast R-CNN model are optimized in combination with the test samples and the loss function to generate a target model. When the target model meets the preset requirements, the target model is output as the pre-trained model.

5. The corn harvester cleaning mechanism blockage detection system based on deep learning according to claim 4 is characterized in that: The pretreatment includes: Grayscale processing is performed on the training images in the training sample to obtain a grayscale image corresponding to each training image; Performing smoothing filtering on each grayscale image to obtain a first image corresponding to each grayscale image; Perform histogram equalization processing on each first image to obtain a second image corresponding to each first image.

6. The corn harvester cleaning mechanism blockage detection system based on deep learning according to claim 4, characterized in that: When the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows: The detection accuracy of the target model is calculated, and when the detection accuracy reaches a preset accuracy, it is determined that the target model meets the preset requirements.

7. An electronic device, characterized in that: The electronic device includes a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement the method according to any one of claims 1 to 3.

Images