Corn harvester cleaning mechanism blockage detection method based on deep learning

Abstract

The invention discloses a corn harvester cleaning mechanism blockage detection method based on deep learning, and relates to the technical field of agricultural machinery intelligent detection, and the method comprises the steps: obtaining a target image of a cleaning mechanism of a to-be-detected corn harvester, inputting the target image into a pre-training model, and obtaining a detection result corresponding to the target image, judging the blockage condition of the to-be-detected corn harvester based on the detection result; the training process of the pre-training model comprises the following steps: preprocessing a training sample to obtain a target training sample; and processing the target training sample through the Fast R-CNN model, performing parameter optimization on the Fast R-CNN model in combination with the test sample and the loss function, generating a target model, and when the target model meets a preset requirement, outputting the target model as a pre-training model. According to the method, image preprocessing and a target detection algorithm are combined, a blocked area can be quickly identified, an alarm is triggered, and the intelligent level of mechanical operation is effectively improved.

Description

Background Art

[0002] With the rapid development of agricultural mechanization and intelligentization, corn harvesters are playing an increasingly important role in modern agricultural production. However, during the corn harvest process, clogging of the cleaning mechanism remains a major challenge for agricultural machinery operators. This is particularly true in high-yield, densely planted fields, where corn stalks and kernels accumulate, making it highly susceptible to clogging. This clogging not only disrupts the continuity of harvesting operations but can also lead to machine wear and tear, increasing repair and maintenance costs, and severely impacting the efficiency and quality of the corn harvest.

[0003] Currently, corn harvesters typically use screens, fans, and other devices to clean corn kernels, such as screens, transmission components, and impurity removal devices. However, these cleaning mechanisms often rely on fixed operating parameters. While some machines have basic cleaning functions, clogging, screen blockage, and high power consumption remain difficult to avoid in complex and changing operating environments. Furthermore, traditional machine designs lack the ability to monitor operating status in real time, making it difficult to respond and adjust dynamically when a blockage occurs, failing to meet the demands of modern agriculture for intelligent and efficient operations.

[0004] In recent years, deep learning technology has achieved breakthroughs in image recognition and intelligent monitoring, providing technical support for intelligent detection and early warning systems for agricultural machinery. A deep learning-based congestion detection system uses image acquisition devices to monitor the operating status of the cleaning mechanism in real time, accurately identifying and locating congestion areas, and providing operators with effective early warning alerts. Applying deep learning models to corn harvester cleaning mechanism congestion detection not only improves the accuracy and real-time performance of congestion detection but also effectively reduces downtime caused by congestion, thereby improving machine operating efficiency, further enhancing harvest quality, and further enhancing the intelligence level of harvesting machinery.

[0005] Currently, Chinese invention patent application CN 221152085 U proposes a cleaning mechanism comprising a screen plate and a cleaning element. This mechanism aims to improve the screen plate's impurity removal efficiency through a transmission assembly and impurity removal device. However, this design focuses on improving cleaning efficiency and does not address intelligent processing for blockage detection. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and specifically provide a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning, as follows: 1) In the first aspect, the present invention provides a method for detecting blockage in a cleaning mechanism of a corn harvester based on deep learning. The specific technical solution is as follows: Obtain the target image of the cleaning mechanism of the corn harvester to be detected, input the target image into the pre-trained model to obtain the detection result corresponding to the target image, and judge the blockage situation of the corn harvester to be detected based on the detection result; The training process of the pre-trained model is as follows: Obtain training samples, preprocess the training samples to obtain target training samples; Process the target training samples through the Fast R-CNN model, combine the test samples and the loss function, optimize the parameters of the Fast R-CNN model to generate a target model, and when the target model meets the preset requirements, output the target model as the pre-trained model.

[0007] The beneficial effects of a method for detecting blockage of the cleaning mechanism of a corn harvester based on deep learning provided by the present invention are as follows: First of all, by using the Fast R-CNN model, the detection efficiency and accuracy can be significantly improved, the image can be processed quickly and the blockage area of the cleaning mechanism can be accurately located. Compared with traditional methods, the detection speed is faster and the accuracy is higher. Secondly, this solution realizes automatic detection, reduces manual intervention, reduces labor costs, and can monitor the operating status of the cleaning mechanism in real time, discover and handle blockage problems in a timely manner. In addition, the use of the pre-trained model makes full use of the feature representation ability of large-scale image data. Combining a large number of training samples and parameter optimization, the model has strong generalization ability, can adapt to images in different environments and working conditions, and reduces detection errors. At the same time, by accurately detecting the blockage situation and dealing with it in a timely manner, the cleaning loss rate and impurity content rate can be effectively reduced, the cleaning performance of the corn harvester can be improved, the control strategy can be optimized, and the overall operation quality can be improved. Finally, this solution has good technical versatility and scalability. The Fast R-CNN model can be extended to fault detection or target recognition tasks of other agricultural machinery, adapt to a variety of scenarios, and has broad application prospects.

[0008] On the basis of the above solution, the present invention can also be improved as follows.

[0009] Further, the preprocessing includes: Grayscale the training images in the training samples to obtain the grayscale image corresponding to each training image; Perform smoothing filtering on each grayscale image to obtain the first image corresponding to each grayscale image; Perform histogram equalization on each first image to obtain the second image corresponding to each first image.

[0010] Further, the construction process of the Fast R-CNN model is as follows: Adjust the convolutional layer structure of the Fast R-CNN base model according to the clogging detection requirements, and set the filter size for each convolutional layer; Add a pooling layer, a region proposal network, and a classification layer to the Fast R-CNN base model to generate the Fast R-CNN model.

[0011] Further, when the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows: Calculate the detection accuracy of the target model, and when the detection accuracy reaches the preset accuracy, determine that the target model meets the preset requirements.

[0012] 2) On the second aspect, the present invention also provides a clogging detection system for the cleaning mechanism of a corn harvester based on deep learning, and the specific technical solution is as follows: The detection module is used to: obtain the target image of the cleaning mechanism of the corn harvester to be detected, input the target image into the pre-trained model to obtain the detection result corresponding to the target image, and judge the clogging situation of the corn harvester to be detected based on the detection result; The training process of the pre-trained model is as follows: Obtain training samples, preprocess the training samples to obtain target training samples; Process the target training samples through the Fast R-CNN model, combine the test samples and the loss function, optimize the parameters of the Fast R-CNN model to generate a target model, and when the target model meets the preset requirements, output the target model as the pre-trained model.

[0013] On the basis of the above solution, the present invention can also be improved as follows.

[0014] Further, the preprocessing includes: Grayscale the training images in the training samples to obtain the grayscale image corresponding to each training image; Perform smoothing filtering on each grayscale image to obtain the first image corresponding to each grayscale image; Perform histogram equalization on each first image to obtain the second image corresponding to each first image.

[0015] Further, the construction process of the Fast R-CNN model is as follows: Adjust the convolutional layer structure of the Fast R-CNN base model according to the clogging detection requirements, and set the filter size for each convolutional layer; Add a pooling layer, a region proposal network, and a classification layer to the Fast R-CNN base model to generate the Fast R-CNN model.

[0016] Further, when the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows: Calculate the detection accuracy of the target model, and when the detection accuracy reaches the preset accuracy, determine that the target model meets the preset requirements.

[0017] 3) Thirdly, the present invention also provides an electronic device. The electronic device includes a processor, the processor is coupled to a memory, and at least one computer program is stored in the memory. The at least one computer program is loaded and executed by the processor so that the electronic device implements any one of the above methods.

[0018] 4) Fourthly, the present invention also provides a computer-readable storage medium. At least one computer program is stored in the computer-readable storage medium. The at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0019] It should be noted that for the beneficial effects obtained by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation manners, reference may be made to the technical effects of the first aspect and its corresponding possible implementation manners described above, and details are not described herein again. Description of the Drawings

[0020] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present invention will become more obvious: Figure 1 It is one of the flow diagrams of a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention; Figure 2 It is another flow diagram of a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention; Figure 3 It is a schematic diagram of a Faster R-CNN neural network model based on ResNet-50 for a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention; Figure 4 It is one of the schematic diagrams of the blockage situation of the corn core shaft on the scale sieve surface during the cleaning process of a method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to an embodiment of the present invention; Figure 5Schematic diagram II of the blockage situation of the corn core shaft on the fish-scale sieve surface during the cleaning process of a cleaning mechanism for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 6 Schematic diagram of the installation position of an image acquisition system for a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 7 Physical diagram of a blockage detection system for a cleaning mechanism for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 8 (a) Schematic diagram of the original image of the mechanism blockage of a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 8 (b) Schematic diagram of the labeled image of the mechanism blockage of a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 8 (c) Schematic diagram of the grayscale image of a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 8 (d) Schematic diagram of the image after non-linear grayscale transformation of a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 8 (e) Schematic diagram of the image after smoothing filtering of a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 8 (f) Schematic diagram of the image after histogram equalization of a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 9 Schematic diagram of the result in the test stage of a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention; Figure 10 Structural framework diagram of an electronic device according to the present invention. Detailed implementation manners

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0022] As Figure 1 shown, a cleaning mechanism blockage detection method for a corn harvester based on deep learning according to an embodiment of the present invention includes the following steps: S1. Obtain the target image of the cleaning mechanism of the corn harvester to be detected, input the target image into the pre-trained model to obtain the detection result corresponding to the target image, and judge the blockage situation of the corn harvester to be detected based on the detection result; The training process of the pre-trained model is as follows: Obtain training samples, preprocess the training samples to obtain target training samples; Process the target training samples through the Fast R-CNN model, combine the test samples and the loss function, optimize the parameters of the Fast R-CNN model to generate a target model, and when the target model meets the preset requirements, output the target model as the pre-trained model.

[0023] The beneficial effects of a method for detecting blockage of the cleaning mechanism of a corn harvester based on deep learning provided by the present invention are as follows: First of all, by using the Fast R-CNN model, the detection efficiency and accuracy can be significantly improved, the image can be processed quickly and the blockage area of the cleaning mechanism can be accurately located. Compared with traditional methods, the detection speed is faster and the accuracy is higher. Secondly, this solution realizes automatic detection, reduces manual intervention, reduces labor costs, and can monitor the operating status of the cleaning mechanism in real time, discover and handle blockage problems in a timely manner. In addition, the use of the pre-trained model makes full use of the feature representation ability of large-scale image data. Combined with a large number of training samples and parameter optimization, the model has strong generalization ability, can adapt to images in different environments and working conditions, and reduces detection errors. At the same time, by accurately detecting the blockage situation and dealing with it in a timely manner, the cleaning loss rate and impurity content rate can be effectively reduced, the cleaning performance of the corn harvester can be improved, the control strategy can be optimized, and the overall operation quality can be improved. Finally, this solution has good technical versatility and scalability. The Fast R-CNN model can be extended to fault detection or target recognition tasks of other agricultural machinery, adapt to a variety of scenarios, and has a wide range of application prospects.

[0024] It should be noted that as Figures 2 to 9 shown, the process of obtaining training samples is specifically as follows: (1) Install a camera at a key position of the cleaning mechanism. The position is selected at a position where the key operation area can be directly observed, such as the outlet of the cleaning fan, the screen area or the conveying channel, to ensure that the camera's view can completely cover the key components of the cleaning mechanism and avoid blind spots; select a high-resolution industrial camera to ensure that the image clarity meets the detection requirements; use a stable bracket or mounting component to fix the camera to prevent displacement during mechanical vibration; (2) Use a camera to collect real-time images. Image collection is carried out at different stages, covering normal operating states and various possible blockage states, ensuring that the collection process covers different environmental conditions and mechanical operating parameters. To solve the lighting problem, the camera is equipped with a fill light to ensure clear images can still be obtained in low-light environments. The collected images are stored in a local storage device in real time and classified and managed according to status and timestamp for subsequent processing; (3) Divide the collected image data into training samples and test sets according to a ratio of 8:2; (4) In the training dataset, use the Image Labeler tool in MATLAB to manually label the blocked areas, ensuring that the bounding boxes accurately cover the blocked areas. Normal images do not need to be labeled and are used as unblocked samples.

[0025] After determining the training samples, preprocessing of the training samples is required. The specific process is as follows: (1) The grayscale operation is used to convert the original RGB image into a grayscale image to reduce the computational complexity of the data and highlight the characteristics of the blocked areas. The grayscale operation can be implemented through the function convert_to_gray(), and the calculation formula is: Among them, R(i,j), G(i,j), and B(i,j) respectively represent the red, green, and blue channel values of the image at position (i,j).

[0026] (2) The gray-scale transformation operation is used to adjust the gray-scale values of the grayscale image according to non-linear rules to increase the dynamic range of the image and enhance the contrast. The gray-scale transformation can be implemented through the function nonlinear_gray_transform(), and the calculation formula is: This formula adjusts the pixel values to increase the contrast and highlight the characteristics of the blocked areas.

[0027] (3) The smoothing filter operation is used to denoise the image after gray-scale transformation to reduce noise interference and improve image clarity. The smoothing filter can be implemented through the function apply_smoothing_filter(), and a 3x3 mean filter is used. The formula is: (4)The filter takes the average value of the 3x3 neighborhood of each pixel in the image to smooth the image and reduce noise. The histogram equalization operation is used to enhance the global contrast of the smoothed image, and the equalization operation can be implemented by the function enhance_contrast_histogram(). The calculation steps are as follows: 1. Calculate the cumulative distribution function (CDF) of the grayscale image and normalize it.

[0028] 2. Map each pixel value using the normalized CDF. The calculation formula is: where CDF is the cumulative distribution function, and min CDF is the minimum value of the CDF, which is used to normalize the pixel value to the range of 0 - 255.

[0029] The above steps sequentially implement image preprocessing through different image processing functions, providing a clear image input for the target detection module and ensuring the efficient recognition of the blocked area.

[0030] It should be further noted that before performing grayscale processing, it is necessary to first count the data volume of the training samples and the clarity and brightness diversity of each training image in the training samples. The specific process is as follows: Since in this solution, the image acquisition area includes multiple positions such as the outlet of the cleaning fan, the sieve, and the conveying channel, during the statistics of the data volume of the training samples, in the outlet of the cleaning fan, the sieve, and the conveying channel, the sample volume corresponding to each type of position is similar or equal. That is, it is judged whether the first sample volume corresponding to each type of position is within the preset sample volume range. If it is, there is no need to expand the training samples, and the clarity and brightness diversity can be directly judged. If not, it is necessary to expand the training samples corresponding to the target positions that are not within the preset sample volume range.

[0031] The process of expanding the training samples is specifically as follows: Determine the target clarity range (preset range) and the target brightness range (preset range) corresponding to this type of position. Divide the target clarity range and the target brightness range to obtain multiple sub-clarity ranges and multiple sub-brightness ranges. Perform permutations and combinations on the multiple sub-clarity ranges and multiple sub-brightness ranges to generate range pairs composed of one sub-clarity range and one sub-brightness range. Assign weights to each range pair. According to the median of the preset sample size range corresponding to this type of position, determine the first preset sample size corresponding to each range pair. Calculate the clarity and brightness of each sample image in the first sample size corresponding to this type of position. Based on the clarity and brightness of each sample image, place any sample image within a target sub-clarity range and a target sub-brightness range, and determine the target range pair corresponding to this sample image according to the target sub-clarity range and the target sub-brightness range. According to the first sample size and the weight corresponding to each range pair, determine the target number of sample sizes within each range pair. According to the standard sample size corresponding to this type of position and the weight corresponding to each range pair, determine the standard number of sample sizes within each range pair. Determine the difference between the standard number and the target number corresponding to the same range pair. According to the difference, randomly select random clarity and random brightness corresponding to the number of differences within this range pair to form random image parameters, and randomly select a training sample from the first sample size. Change the clarity and brightness of this training sample to the random image parameters to generate a new training image and put it into the training sample.

[0032] Among them, the process of dividing the target clarity range and the target brightness range is specifically as follows: Among all the training images corresponding to this type of position, select a group of first image sets. Extract features from the first image sets through manual annotation, and generate detection frames corresponding to each training image in the first image sets. By adjusting the clarity, determine the clarity corresponding to more than 60% of the detected frames, and determine this clarity as the first reference clarity for dividing the target clarity range. Based on the first reference clarity, determine the position of the first reference clarity within the target clarity range. When this position is between the minimum value and the median of the target clarity range, determine the second reference clarity at half of the difference between the first reference clarity and the maximum value of the target clarity range. Divide the target clarity range based on the first reference clarity and the second reference clarity. The brightness division method is the same as the clarity division method and will not be elaborated here.

[0033] The process of directly judging the diversity of clarity and brightness is specifically as follows: Determine whether all training images corresponding to any type of position fall within all range pairs. If not, improve the sample diversity processing for this type of position through the process of augmenting the training samples.

[0034] The training process of the pre-trained model is specifically as follows: (1) Select the Faster R-CNN model based on ResNet-50 as the basic model and load the weights pre-trained on the COCO dataset to improve the initial feature extraction ability and training efficiency. The first four convolutional residual blocks of the basic network ResNet-50 remain unchanged, and the fifth residual block is adjusted to enhance the feature extraction ability; (2) Adjust the convolutional layer structure and add five new convolutional layers. The newly added convolutional units are used to improve the model's ability to capture the features of the blocked area. Change the CONV5 residual block of ResNet-50 to a five-layer convolutional structure, arranged as conv5_1, relu5_1, conv5_2, relu5_2, conv5_3, relu5_3, conv5_4, relu5_4, pool5; set the filter sizes of the convolutional units conv5_1, conv5_2, conv5_3, conv5_4 to [3,3,3,64], [3,3,64,64], [3,3,64,128], [3,3,128,256], [3,3,256,512] respectively, and the stride is 1; (3) Add pooling layers, and add pooling layers after each convolutional module, arranged as pool_ , pool5_2, pool5_3, pool5_4, pool5_5; set the pooling size of the pooling unit pool5 to [2,2] and the stride to 2 to reduce the feature map size and extract key features; (4) Incorporate the Region Proposal Network (RPN) to generate candidate region boxes to identify possible blocked areas. The RPN operates after the output of the convolutional layer, generates candidate boxes through sliding window convolution and Anchor boxes, and further filters and adjusts the positions of these boxes through classification and regression. The filter size of the RPN sliding window convolutional layer is set to [3,3,512,512] and the stride is 1. This convolutional layer generates multiple candidate boxes for locating possible blocked areas; the sizes of the Anchor boxes are set to [32,64,128,256,512], and the aspect ratios are [0.5, 1.0, 2.0]. Multiple sizes and ratios of Anchor boxes are used to capture blocked areas of different scales and shapes; the IoU threshold for screening candidate boxes is IoU≥0.7 for positive samples and IoU<0.3 for negative samples, and positive and negative samples are screened according to the overlap degree with the ground truth box; (5) Add a classification layer to the RPN backend to classify the generated candidate boxes into blocked areas and non-blocked areas, and output the final detection results. Add a classification layer and a bounding box regression layer after the RPN output: the classification layer is used to classify each candidate box as a "blocked area" or a "non-blocked area", set the fully connected layer as fc_cls, and the filter size as [7,7,512,2]. This layer outputs two types of labels (blocked and non-blocked); the bounding box regression layer is used to adjust the position of the candidate box to more precisely frame the actual blocked area. The bounding box regression layer is used to adjust the position of the candidate box, set the fully connected layer as fc_bbox, and the filter size as [7,7,512,4]. This layer outputs four regression values, which respectively represent the adjustment amounts of the boundary coordinates of the candidate box (left, right, up, down); after classification and regression processing, the model outputs the final detection results, including the detection category ("blocked area" or "non-blocked area") and the bounding box coordinates (providing accurate bounding box coordinates for each candidate box to ensure correct positioning of the blocked area). This process completes the accurate detection of the blocked area and helps the real-time identification and warning system improve work efficiency and accuracy.

[0035] The optimization process of the pre-trained model is specifically as follows: (1) Initialize the weights and biases of the model, and adopt the parameters of the pre-trained model to accelerate the convergence speed. The convolutional layer and fully connected layer of the Faster R-CNN model use the weights pre-trained on the ImageNet dataset; the weights of the convolutional layer are initialized with He (for the ReLU activation function) to avoid gradient explosion or gradient disappearance; the biases are initialized to 0; by doing so, the model converges quickly; the calculation formula for He initialization is: , n in is the number of input neurons; (2) Input the labeled training dataset into the object detection model, generate candidate boxes through the RPN and perform annotation. Input the pre-processed (grayed, gray-transformed, smoothed filtered, and histogram equalized) image data of the model in step two into the object detection model; the RPN generates candidate boxes from the input image, generates multiple candidate regions of different sizes according to the feature map of the image (extracted by the convolutional layer), and evaluates whether the region contains the target; the candidate boxes are annotated according to the overlap degree (IoU) with the actual target position. IoU ≥ 0.7 is a positive sample, indicating inclusion, and IoU < 0.3 is a negative sample, indicating non-inclusion; (3) Adopt the cross-entropy loss function and compare the output result with the annotation result through the Softmax classification algorithm. Use the cross-entropy loss function to measure the gap between the category predicted by the model and the true category. The formula is: , y i is the true label, pi is the probability distribution predicted by the model, and N is the number of samples; (4) Use the batch stochastic gradient descent method to perform error backpropagation and gradually adjust the weights and biases of the model. The core of the training process is error backpropagation. By passing the gradient information of the loss function from the output layer to the input layer, the model can gradually adjust its weights and biases, reducing the error step by step. The batch stochastic gradient descent (SGD) method is used to optimize the model parameters. In each iteration, the model randomly selects a batch of data from the training samples for training, calculates the gradient based on the error of this batch, and updates the weights and biases of the model. The update rule is: , θ is the model parameter, η is the learning rate, and ∇ θ J(θ) is the gradient of the loss function with respect to the parameter; (5) Repeat steps (2)-(4) and perform multiple iterations on the training samples until the detection accuracy of the model meets the requirements. To allow the model to fully learn the features of the training data, the number of training epochs is set to 100. In each epoch, the entire training sample is traversed. The model updates its parameters after each epoch to improve the accuracy. And set the training stop conditions, where the specific condition parameters are: mAP ≥ 0.7, Precision ≥ 0.85, Recall ≥ 0.80, F1-score ≥ 0.80, IoU ≥ 0.7.

[0036] The testing and validation process of the pre-trained model is as follows: (1) Input the images in the test dataset into the trained object detection model to verify the detection effect of the model. Input the test set into the model. The test set data was not involved in training. Through the input, the model generates prediction results based on the trained weights and parameters, including the position and category (blocked area or non-blocked area) of each candidate box. Generate bounding boxes on the test images and label the classification results of each candidate box. For the blocked detection task, the model should be able to effectively identify the blocked area and accurately label its position; (2) Generate candidate boxes in the RPN and identify the blocked areas based on the model output. The RPN uses a sliding window mechanism to operate on each test image, generating multiple candidate region boxes, which represent the regions that the model believes may contain the target (blocked areas); based on the candidate boxes, filter the candidate boxes according to the IoU threshold (IoU ≥ 0.7) and Anchor box sizes ([32, 64, 128, 256, 512], aspect ratios of [0.5, 1.0, 2.0]) set in step 3.4. The valid candidate boxes will enter the next classification process; the classification layer will classify each candidate box to determine whether it is a blocked area. If a blocked area is detected, the model will give the location information and class label of this area; (3) Calculate the detection precision and recall rate of the model, and judge the model performance according to the test results. Calculate Precision: Precision measures the proportion of candidate boxes predicted by the model as blocked areas that are actually blocked. Calculate Recall: Recall represents the proportion of actual blocked areas that the model can identify. The formulas are: Use mAP (Mean Average Precision) to comprehensively evaluate the detection performance of the model on different classes, and ensure the balance of detection precision and recall rate. If the mAP value is low, the model needs to be further optimized.

[0037] (4) If the detection precision is insufficient, return to step four for further optimization until the detection requirements are met. By calculating the precision and recall rate of the test data, judge whether the model has reached the set performance standard. The standard refers to step 4.5, mAP ≥ 0.7, Precision ≥ 0.85, Recall ≥ 0.80, F1 - score ≥ 0.80, IoU ≥ 0.7. If both the precision and recall rate do not reach the expected target, it indicates that the model has not achieved the ideal effect; among them, further optimizing the model includes: adjusting hyperparameters, data augmentation, improving the network structure, increasing the training samples, etc.; repeat training and optimization: return to step four, retrain, adjust the parameters of the model, and repeatedly verify the performance of the model until the precision and recall rate of the model meet the expectations, and the training can be stopped.

[0038] This solution also includes: embedding the trained blockage detection model into the embedded system of the corn harvester, and analyzing the images collected by the camera in real time through the computing module of the embedded system to identify the blockage status of the cleaning mechanism; triggering an alarm signal when the number of detected blockages reaches the set alarm threshold to provide real - time warning for the operator. The specific process is as follows: (1)Model integration into the embedded system. Convert the trained object detection model into a format compatible with the embedded system using PyTorch Mobile; deploy the model to the embedded computing device of the corn harvester; configure the hardware interface of the embedded system to ensure seamless connection with the camera and alarm system; (2)Real-time image input and detection. The embedded system receives the real-time images collected by the camera, preprocesses them and inputs them into the detection model; uses the model to identify the clogging status of the cleaning mechanism and detect the presence and location of the clogging area in real time; (3)Real-time output and alarm. The computing module of the embedded system generates detection results in real time and judges the clogging status; when a clogging area is detected, the embedded system makes a real-time judgment according to the set alarm threshold and triggers the corresponding alarm signal: 1) Mild clogging: When the clogging area detected per minute reaches more than 1.5%, the system triggers a warning signal to prompt the operator to check the cleaning mechanism to prevent further clogging. 2) Moderate clogging: When the clogging area detected per minute reaches more than 6%, the system triggers a warning signal to prevent serious clogging. 3) Severe clogging: When the detected clogging area reaches more than 14%, the system issues a higher-priority alarm to remind the operator to handle the clogging problem immediately to avoid seriously affecting the normal operation of the cleaning mechanism. 4) Proportion of clogging area: When the proportion of the detected clogging area in the image area exceeds 20%, the system triggers an emergency alarm, requiring the operator to stop the machine immediately for cleaning to prevent further damage or failure of the mechanical equipment.

[0039] The present invention has the following beneficial effects: Based on the deep learning image processing method, the present invention realizes the real-time and accurate detection of the clogging status of the cleaning mechanism of the corn harvester. By collecting operation images through the camera, combining image preprocessing and object detection algorithms, it can quickly identify the clogging area and trigger an alarm, effectively improving the intelligent level of mechanical operation; Based on the proposed detection method, the present invention develops a clogging detection system. The system supports flexible adjustment of algorithm parameters to adapt to various complex operation scenarios. This system can be embedded in the embedded platform of the corn harvester and deployed in real time for production operations, providing technical support for improving the operation efficiency and reliability of the harvesting machinery and reducing the downtime loss caused by clogging.

[0040] Furthermore, the preprocessing includes: Grayscale the training images in the training samples to obtain the grayscale map corresponding to each training image; Perform smoothing filtering on each grayscale map to obtain the first image corresponding to each grayscale image; Perform histogram equalization on each first image to obtain the second image corresponding to each first image.

[0041] Further, the construction process of the Fast R-CNN model is as follows: According to the clogging detection requirements, adjust the convolutional layer structure of the Fast R-CNN base model and set the filter size for each convolutional layer; Add a pooling layer, a region proposal network, and a classification layer to the Fast R-CNN base model to generate the Fast R-CNN model.

[0042] Further, when the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows: Calculate the detection accuracy of the target model, and when the detection accuracy reaches the preset accuracy, determine that the target model meets the preset requirements.

[0043] Example 1, as Figure 2 and Figure 3 shown, the specific content includes the following steps: Step 1: Image acquisition and diversification. Collect images of the cleaning mechanism in normal and clogged states, and divide the image data into training samples and a test set.

[0044] Step 2: Image data preprocessing. Perform grayscale conversion, gray-scale transformation, smoothing filtering, and histogram equalization operations on the images in the training samples to improve the clarity and contrast of the images.

[0045] Step 3: Build a target detection model. Use Fast R-CNN as the target detection model to label and detect the clogged areas in the images.

[0046] Step 4: Model training and optimization. Use the labeled training samples to train the Fast R-CNN model and adjust the model parameters to optimize the detection performance.

[0047] Step 5: Model testing and verification. Use the test set to verify the trained model to ensure the detection accuracy and stability of the model.

[0048] Step 6: Integration of real-time monitoring and warning functions. Deploy the trained model in the cleaning mechanism system to achieve real-time detection and alarm of the clogged state.

[0049] Further, in Step 1, high-resolution industrial cameras installed at key positions of the cleaning mechanism are used to collect images of the cleaning mechanism in normal operation and clogged states in real time, as Figure 8 (a) shown. Divide the collected images into a training data set and a test data set, and manually label the clogged areas in the training data set to provide supervised learning data for subsequent model training; Furthermore, the camera should be fixed at the key position of the cleaning mechanism; the real-time collected images should cover a variety of states and environments to improve the generalization ability of the model; manual labeling is completed using the Image Labeler tool in MATLAB, and the labeling box accurately covers the blocked area, such as Figure 8 (b) Furthermore, in step 2, the image in the training sample is input into the image preprocessing module, and a series of enhancement operations are performed on it to improve the image quality and recognizability, providing high-quality input data for subsequent target detection; the preprocessing module includes: first, grayscale conversion: through the function convert_to_gray(), the RGB image is converted into a grayscale image to reduce data complexity, such as Figure 8 (c) As shown; Secondly, grayscale transformation: Through the function nonlinear_gray_transform(), the grayscale value is nonlinearly adjusted to expand the dynamic range of the image, as shown in Figure 8 (d) As shown; Then, smoothing filtering: Use the function apply_smoothing_filter() to perform 3×3 mean filtering on the image to reduce noise interference, as shown in Figure 8 (e) As shown; Finally, histogram equalization: through the function enhance_contrast_histogram(), the gray value distribution of the image is redistributed to improve the contrast, as shown in Figure 8 (f) Furthermore, the target detection model constructed by the present invention in step 3 is based on Faster R-CNN and is optimized as follows: first, the pre-trained model is loaded: the Faster R-CNN pre-trained on the COCO dataset is used as the base model to improve the initial feature extraction capability and training efficiency; second, the convolution layer is adjusted: the convolution layer of the base model is adjusted, and a 5-layer convolution structure (conv5_1 to conv5_4 and pool5) is added to enhance the feature extraction capability of the blocked area; then, the region proposal network (RPN): the candidate region box is generated by sliding the window, and the candidate box is screened by classification and regression, with IoU ≥ 0.7 as a positive sample and IoU < 0.3 as a negative sample; finally, the classification and regression layer: a fully connected classification layer (fc_cls) and a bounding box regression layer (fc_bbox) are added to classify the candidate box (blocked / non-blocked) and adjust its position; Further, in the model training and optimization in Step 4, the training process mainly includes: First, initialize the model parameters: adopt pre-trained weights, and use He initialization for the convolutional layer to avoid gradient explosion or vanishing; Second, supervised learning: use the cross-entropy loss function to calculate the classification error, and optimize the model through the Softmax classifier; Finally, gradient descent optimization: use the batch stochastic gradient descent (SGD) method to adjust the weights and biases, and iteratively optimize the model performance; After multiple rounds of iterative training (Epochs is set to 100), the model performance is stable and meets the expected accuracy requirements; Further, in Step 5, input the test data set into the trained model for performance verification, which specifically includes: First, input the test data: generate candidate boxes through RPN and identify possible blocked areas; Second, performance evaluation: calculate the precision, recall, and mean average precision (mAP) of the model; Finally, performance optimization: if the detection performance does not meet the standard, return to Step 4 to adjust the model parameters until the detection precision (Precision≥0.85) and recall (Recall≥0.80) meet the requirements; Further, in Step 6, embed the optimized object detection model into the embedded system of the corn harvester to achieve the function of real-time monitoring and alarming of the blocked state. Convert the trained model into a format compatible with the embedded system through PyTorch Mobile and deploy it to the embedded computing device of the harvester. The system receives the images collected by the camera in real time and analyzes the blocked state of the cleaning mechanism; when a blockage is detected, the embedded system triggers an audible and visual alarm according to the set alarm threshold, providing real-time warning prompts for the operator, and timely reminding the operator to check and handle the blockage problem to ensure the efficient operation of the harvester.

[0050] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, and this is also within the protection scope of the present invention. It can be understood that in some embodiments, it may include some or all of the above embodiments.

[0051] The present invention also provides a blocked state detection system for the cleaning mechanism of a corn harvester based on deep learning, and the specific technical solution is as follows: The detection module is used to: obtain the target image of the cleaning mechanism of the corn harvester to be detected, input the target image into the pre-trained model to obtain the detection result corresponding to the target image, and judge the blocked situation of the corn harvester to be detected based on the detection result; The training process of the pre-trained model is: Obtain training samples, preprocess the training samples to obtain target training samples; Process the target training samples through the Fast R-CNN model, and optimize the parameters of the Fast R-CNN model by combining the test samples and the loss function to generate a target model. When the target model meets the preset requirements, output the target model as the pre-trained model.

[0052] It should be noted that the beneficial effects of the corn harvester cleaning mechanism blockage detection system based on deep learning provided in the above embodiments are the same as those of the corn harvester cleaning mechanism blockage detection method based on deep learning, which will not be elaborated here. In addition, when the system provided in the above embodiments implements its functions, only the division of the above functional modules is used as an example for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system provided in the above embodiments and the method embodiments belong to the same concept, and the specific implementation process can be seen in the method embodiments, which will not be elaborated here.

[0053] As Figure 10 shown, an electronic device 300 according to an embodiment of the present invention, the electronic device 300 includes a processor 320, the processor 320 is coupled to a memory 310, and at least one computer program 330 is stored in the memory 310. The at least one computer program 330 is loaded and executed by the processor 320 to enable the electronic device 300 to implement any one of the above methods. Specifically: The electronic device 300 may have relatively large differences due to different configurations or performances. It may include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310. Among them, at least one computer program 330 is stored in the one or more memories 310. The at least one computer program 330 is loaded and executed by the one or more processors 320 to enable the electronic device 300 to implement a corn harvester cleaning mechanism blockage detection method based on deep learning provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input / output. The electronic device 300 may also include other components for implementing the functions of the device, which will not be elaborated here.

[0054] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any one of the above methods.

[0055] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, an optical data storage device, or the like.

[0056] In an exemplary embodiment, a computer program product or a computer program is further provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes any one of the above methods.

[0057] It should be noted that the terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects, and do not represent a specific order or sequence. In appropriate cases, the order of use of similar objects may be interchanged, so that the embodiments of the present application described herein can be implemented in an order other than the illustrated or described order.

[0058] Those skilled in the art know that the present invention can be implemented as a system, a method, or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms: it can be completely hardware, can be completely software (including firmware, resident software, microcode, etc.), or can be a combination of hardware and software, which is generally referred to as "circuit", "module", or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable media contain computer-readable program code.

[0059] Any combination of one or more computer-readable media may be employed. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present document, a computer-readable storage medium may be any tangible medium that contains or stores a program which can be used by or in connection with an instruction execution system, apparatus, or device.

[0060] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting blockage of the cleaning mechanism of a corn harvester based on deep learning, characterized in that, Including: Obtain a target image of the cleaning mechanism of the corn harvester to be detected, input the target image into a pre-trained model to obtain a detection result corresponding to the target image, and judge the blockage condition of the corn harvester to be detected based on the detection result; The training process of the pre-trained model is as follows: Obtain training samples, preprocess the training samples to obtain target training samples; Process the target training samples through the Fast R-CNN model, combine test samples and a loss function, optimize the parameters of the Fast R-CNN model to generate a target model, and when the target model meets the preset requirements, output the target model as the pre-trained model.

2. The clogging detection method for the cleaning mechanism of a corn harvester based on deep learning according to claim 1, characterized in that, The preprocessing includes: Grayscale the training images in the training samples to obtain a grayscale image corresponding to each training image; Perform smoothing filtering on each grayscale image to obtain a first image corresponding to each grayscale image; Perform histogram equalization on each first image to obtain a second image corresponding to each first image.

3. A clogging detection method for the cleaning mechanism of a corn harvester based on deep learning according to claim 1, characterized in that, The construction process of the Fast R-CNN model is as follows: According to the blockage detection requirements, adjust the convolutional layer structure of the Fast R-CNN basic model and set the filter size of each convolutional layer; Add a pooling layer, a region proposal network, and a classification layer to the Fast R-CNN basic model to generate the Fast R-CNN model.

4. A method for detecting blockage of a cleaning mechanism of a corn harvester based on deep learning according to claim 1, characterized in that, When the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically: Calculate the detection accuracy of the target model, and when the detection accuracy reaches the preset accuracy, judge that the target model meets the preset requirements.

5. A clogging detection system for the cleaning mechanism of a corn harvester based on deep learning, characterized in that, Including: The detection module is used to: obtain a target image of the cleaning mechanism of the corn harvester to be detected, input the target image into a pre-trained model to obtain a detection result corresponding to the target image, and judge the blockage condition of the corn harvester to be detected based on the detection result; The training process of the pre-trained model is as follows: Obtain training samples, preprocess the training samples to obtain target training samples; Process the target training samples through the Fast R-CNN model, combine test samples and a loss function, optimize the parameters of the Fast R-CNN model to generate a target model, and when the target model meets the preset requirements, output the target model as the pre-trained model.

6. The clogging detection system for the cleaning mechanism of a corn harvester based on deep learning according to claim 5, characterized in that, The preprocessing includes: Grayscale the training images in the training samples to obtain a grayscale image corresponding to each training image; Perform smoothing filtering on each grayscale image to obtain a first image corresponding to each grayscale image; Perform histogram equalization on each first image to obtain a second image corresponding to each first image.

7. A clogging detection system for the cleaning mechanism of a corn harvester based on deep learning according to claim 5, characterized in that, The construction process of the Fast R-CNN model is as follows: According to the blockage detection requirements, adjust the convolutional layer structure of the Fast R-CNN basic model and set the filter size of each convolutional layer; Add a pooling layer, a region proposal network, and a classification layer to the Fast R-CNN base model to generate the Fast R-CNN model.

8. The clogging detection system for the cleaning mechanism of a corn harvester based on deep learning according to claim 5, characterized in that, When the target model meets the preset requirements, the process of outputting the target model as the pre-trained model is specifically as follows: Calculate the detection accuracy of the target model, and when the detection accuracy reaches the preset accuracy, determine that the target model meets the preset requirements.

9. An electronic device, characterized in that, The electronic device includes a processor, the processor is coupled to a memory, and at least one computer program is stored in the memory. The at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, At least one computer program is stored in the computer-readable storage medium. The at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images