A corn kernel breakage rate real-time detection method fusing an attention mechanism

Abstract

This invention discloses a method for detecting corn kernel breakage rate by incorporating an attention mechanism. This method further improves the lightweight one-stage object detection network YOLOv4-tiny by introducing a coordinate attention mechanism into the feature pyramid network to enhance the model's ability to detect dense corn kernels. Furthermore, an online mosaic data augmentation method is used during the model training phase to improve the model's ability to detect smaller-scale broken kernels. The method disclosed in this invention is easier to deploy on embedded platforms due to its low parameter size, and it has a high response speed while maintaining accuracy. It can efficiently detect the degree of corn kernel breakage and thus obtain the corn kernel breakage rate in real time.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically a real-time detection method for corn kernel breakage rate based on an improved YOLOv4-tiny. Background Technology

[0002] my country is a major corn-producing country, ranking second in the world in corn planting area and accounting for approximately 25% of global corn production. However, it is also a major corn importer, importing millions of tons of corn annually from Ukraine and the United States. This is largely due to the lower harvest quality during mechanized corn harvesting in my country, which easily leads to broken kernels and reduces the utilization rate of corn during storage and comprehensive use. Real-time feedback on the corn kernel breakage rate during harvesting allows for timely adjustments to parameters such as the harvester's threshing drum speed and concave plate gap, improving harvest quality and efficiency. Therefore, real-time detection methods for corn kernel breakage rate are of significant research importance as a key technology for improving harvest quality.

[0003] Currently, there are two main solutions for detecting corn kernel breakage rate: traditional image processing methods and deep learning methods. The former primarily uses image enhancement methods to preprocess the acquired images, then extracts features of interest using color and morphological information, and classifies the degree of corn kernel breakage using machine learning and clustering algorithms. The latter generally requires acquiring a large amount of image data of the target object, preprocessing and labeling it to build a dataset, then setting model hyperparameters, training the network model using the training set, and selecting the best model as the final corn kernel breakage rate detection model.

[0004] Traditional image processing solutions are highly sensitive to lighting conditions and require separate feature extraction methods for different types of corn kernels and those grown in different locations. Deep learning-based solutions, on the other hand, require a large amount of labeled data and suffer from high false positive and false negative rates when detecting small, densely packed corn kernels. Furthermore, the algorithms must be deployed on embedded platforms with limited computing resources, requiring high detection speed and accuracy while maintaining constraints on computational load and parameter count. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art, solve the problems of high false detection rate and serious missed detection caused by the small target size and dense distribution of corn kernels, and the poor real-time performance of complex networks when deployed on embedded platforms, and provide a real-time detection method for corn kernel breakage rate that integrates attention mechanism.

[0006] The technical solution adopted by the present invention to achieve the above objectives is: a real-time detection method for corn kernel breakage rate based on an attention mechanism, the method comprising the following steps:

[0007] Modeling steps: Build a real-time detection network for corn kernel breakage rate based on YOLOv4-tiny; preprocess data and construct a dataset; train the model using data from the dataset, and adjust and optimize the model parameters;

[0008] Breakage rate identification steps: Use an optimized network model to identify the image to be detected; calculate the kernel breakage rate using the corn kernel breakage detection results.

[0009] The steps for building a real-time corn kernel breakage rate detection network based on YOLOv4-tiny include: 1) Introducing a coordinate attention mechanism into the feature pyramid network and constructing a coordinate attention mechanism module, including the following steps:

[0010] A coordinate attention mechanism module is introduced into the feature pyramid network. Two spatial pooling kernels are used to perform the encoding operations shown in equations (1)-(2) along the horizontal and vertical coordinate directions respectively to obtain a C×H×1 feature map. and the feature map of C×1×W ;

[0011] (1)

[0012] (2)

[0013] in, This represents the feature of the c-th channel at height h after encoding. This represents the feature of the c-th channel at a width of w after encoding. Let h be the input feature of the c-th channel at height h and width i. Let h be the input feature of the c-th channel at height j and width w, where h is the height of the input feature map and w is the width of the input feature map.

[0014] The two generated feature maps are processed as shown in Equation (3) to form a C×1×(H+W) feature map. A 1×1 convolution is then used to compress its channel dimension from C to C / r dimensions with a shrinkage rate r. A nonlinear function is then applied for activation to obtain the intermediate attention map f. The obtained result f is then decomposed along the spatial dimension into a C / r×H×1 horizontal attention tensor f. h The vertical attention tensor f of C / r×1×W w ;

[0015] (3)

[0016] Where [.,.] represents the splicing operation along the spatial dimension, F1 is the dimensionality reduction operation along the channel dimension using 1×1 convolution, δ is the nonlinear activation function, f is the intermediate attention graph containing horizontal and vertical attention information, and z h This is a horizontal attention diagram before splicing, z w This is a vertical attention diagram before splicing.

[0017] For the two generated spatial tensors f h and f w Perform the operations shown in equations (4)-(5), using two sets of 1×1 convolutions to increase the channel dimension from C / r to C, and using the Sigmoid function for non-linear activation to obtain the coordinate attention module g that embeds the feature information. h and g w ;

[0018] (4)

[0019] (5)

[0020] Among them, f h For the horizontal attention tensor, F h For the dimensionality increase operation of the horizontal attention graph, f w For the vertical attention tensor, F w This is an upscaling operation for the vertical attention map, where σ is the Sigmoid activation function.

[0021] The attention module used, g h and g w By performing weighting operations on the input feature map and applying coordinate attention, the structure of the real-time apple detection network model is obtained.

[0022] The data preprocessing and dataset construction includes the following steps:

[0023] Construct a corn kernel dataset; data augmentation; prior box extraction.

[0024] The data augmentation operations include mirroring, Gaussian blurring, rotation, and adjustments to image brightness, scale, and contrast.

[0025] The prior box extraction includes: using the K-means algorithm to cluster the size of the bounding boxes in the label data of the enhanced corn kernel dataset described in step (2-2), and using the clustering results as the model prior boxes;

[0026] The process of training the model, adjusting and optimizing the model parameters, and employing an online mosaic data augmentation method includes the following steps:

[0027] In each training round, a batch of data is extracted from the image-enhanced corn kernel dataset. Four images are randomly extracted from this batch of data, cropped and stitched at random positions, and a new image is synthesized.

[0028] Repeat the above steps until a new number of images equal to the batch size is generated. Then, feed all the generated new images into the network as the data for this batch for training.

[0029] During training, hyperparameters are set and network weights are updated iteratively; the model network weights and hyperparameters with the lowest loss on the validation set are used as the final optimized network.

[0030] The process of using an optimized network model to identify the image to be detected includes the following steps:

[0031] a. Read the image to be detected; the image to be detected is a real-time image from the site or a test set image;

[0032] b. Input the image to be detected into the trained real-time corn kernel breakage rate detection network for identification and localization;

[0033] Repeat steps a through b until the images in the test set have been detected.

[0034] The calculation of the kernel breakage rate using the corn kernel breakage test results includes: the number N of intact kernels in the cumulative test results. w and the number of broken grains N b The grain breakage rate P is calculated according to formula (7). c .

[0035] (7)

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) The lightweight one-stage detection algorithm YOLOv4-tiny is used as the basic framework, resulting in a small model and low computational cost.

[0038] (2) Introduce the coordinate attention mechanism into the feature pyramid network to improve the network’s recognition performance for dense corn kernel targets;

[0039] (3) A mosaic data online augmentation strategy is adopted during training to effectively improve the model's ability to detect small targets and improve the model's detection accuracy;

[0040] (4) It is easy to deploy on an embedded platform and has a high detection speed while ensuring accuracy, which can meet the needs of real-time detection of corn kernels.

[0041] (5) This invention proposes a real-time detection method for corn kernel breakage rate that integrates attention mechanism. This method further improves the YOLOv4-tiny algorithm. The idea is to use attention mechanism to enhance the model's ability to detect dense kernel targets. In the model training stage, an online mosaic data enhancement strategy is used to improve the model's ability to detect small targets. This enables the model to efficiently detect the corn kernel breakage state in real time when deployed on an embedded platform, thereby obtaining an accurate corn kernel breakage rate. Attached Figure Description

[0042] Figure 1 This is a flowchart of the present invention;

[0043] Figure 2 This is a schematic diagram of the YOLOv4-tiny network structure;

[0044] Figure 3 A schematic diagram of the Basicblock structure, the basic unit that makes up the YOLOv4-tiny network;

[0045] Figure 4 This is a schematic diagram of the Resblock structure, which constitutes the YOLOv4-tiny network;

[0046] Figure 5 A schematic diagram of the network structure of the improved model after applying a coordinate attention module to the YOLOv4-tiny feature pyramid network;

[0047] Figure 6 This is a schematic diagram of the coordinate attention module structure;

[0048] Figure 7 This is a diagram showing the model's detection results for intact corn kernels.

[0049] Figure 8 The image shows the model's performance in detecting broken corn kernels.

[0050] In this context, Input represents the input, Basicblock is the basic block, Resblock is the residual block, concat means concatenation, conv represents the convolutional layer, Upsampling represents the upsampling operation, batchNormal represents the regularization layer, Leaky_ReLU represents an activation function, split means splitting along the channel dimension and retaining the latter half, ca represents the spatial attention mechanism module, maxpooling represents the max pooling operation, and YOLO head represents the model prediction layer. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0053] This invention uses corn kernels as the detection object. The training platform configuration is as follows: CPU: Intel(R) Core(TM) i7-8700 3.20GHz, RAM: 16GB; GPU: NVIDIA TITAN V, VRAM: 12GB; operating environment: Windows 10; Python version: 3.6. The method of this invention is implemented on the deep learning framework PyTorch version 1.2.0, CUDA version: 10.0, and cuDNN version: 7.4.1.

[0054] like Figure 1 The method for real-time detection of corn kernel breakage rate based on an attention mechanism, as shown, comprises five parts:

[0055] (1) Build a real-time detection network for corn kernel breakage rate based on YOLOv4-tiny;

[0056] (2) Data preprocessing and dataset construction;

[0057] (3) Use the data in the dataset to train the model, and adjust and optimize the model parameters;

[0058] (4) Use an optimized network model to identify the image to be detected;

[0059] (5) Calculate the grain breakage rate using the test results.

[0060] The first part involves building a real-time corn kernel breakage detection network based on YOLOv4-tiny, specifically including the following steps:

[0061] The corn kernel breakage rate detection task involved in this invention has high real-time requirements and generally needs to be deployed on an embedded platform with limited computing resources. Therefore, a platform such as... Figure 2-4This paper presents an improvement upon the lightweight one-stage object detection network YOLOv4-tiny.

[0062] Figure 2 This is a schematic diagram of the YOLOv4-tiny network structure. The overall network structure has 38 layers. The feature extraction network contains two basic units and three residual units. The feature pyramid network is used to fuse the extracted effective features, and two feature layers of different scales are used to classify and locate the target. Figure 3 This is a schematic diagram of the Basicblock structure, the basic unit that makes up the YOLOv4-tiny network; it includes a convolutional layer (conv), a batchnorm regularization layer, and a Leaky_ReLU activation function. Figure 4 The diagram shows the Resblock structure of the residual unit that makes up the YOLOv4-tiny network. It consists of two parts. The first part is directly constructed as a residual edge. The second part uses the feature layers retained after the split operation to construct the residual structure, and then it is concatenated with the first part. After max pooling, the output result is obtained. This structure can reduce the computational complexity while ensuring the accuracy of the network.

[0063] To compensate for the accuracy loss caused by the poor feature extraction capability of lightweight networks, this invention introduces a coordinate attention module into the feature pyramid network. This module is a lightweight attention mechanism module for computer vision proposed by Hou et al. in 2021. It can be easily inserted between two convolutional layers to strengthen target attention and improve the ability to detect dense targets.

[0064] like Figure 5 As shown, a coordinate attention mechanism module (CA) is introduced into the feature pyramid network. The schematic diagram of the CA module structure is shown below. Figure 6 As shown, two spatial pooling modules are used to perform the encoding operations shown in equations (1)-(2) along the horizontal and vertical coordinate directions respectively to obtain a C×H×1 feature map. and the feature map of C×1×W ;

[0065] (1)

[0066] (2)

[0067] in, This represents the feature of the c-th channel at height h after encoding. This represents the feature of the c-th channel at a width of w after encoding. Let h be the input feature of the c-th channel at height h and width i. Let h be the input feature of the c-th channel at height j and width w, where h is the height of the input feature map and w is the width of the input feature map.

[0068] The two generated feature maps are processed as shown in Equation (3) to form a C×1×(H+W) feature map. A 1×1 convolution is then used to compress its channel dimension from C to C / r dimensions with a shrinkage rate r. ReLU is then used for non-linear activation to obtain the intermediate attention map f. The obtained result f is then decomposed along the spatial dimension into a C / r×H×1 horizontal attention tensor f. h The vertical attention tensor f of C / r×1×W w ;

[0069] (3)

[0070] Where [.,.] represents the splicing operation along the spatial dimension, F1 is the dimensionality reduction operation along the channel dimension using 1×1 convolution, δ is the nonlinear activation function, f is the intermediate attention graph containing horizontal and vertical attention information, and z h This is a horizontal attention diagram before splicing, z w This is a vertical attention diagram before splicing.

[0071] For the two generated spatial tensors f h and f w Perform the operations shown in equations (4)-(5), using two sets of 1×1 convolutions to increase the channel dimension from C / r to C, and using the Sigmoid function for non-linear activation to obtain the coordinate attention module g that embeds the feature information. h and g w ;

[0072] (4)

[0073] (5)

[0074] Among them, f h For the horizontal attention tensor, F h For the dimensionality increase operation of the horizontal attention graph, f w For the vertical attention tensor, F w This is an upscaling operation for the vertical attention map, where σ is the Sigmoid activation function.

[0075] Using the attention module g h and g w Weighting operations are performed on the input feature map to apply coordinate attention.

[0076] The second part, data preparation, includes the following steps:

[0077] (2-1) Dataset Acquisition

[0078] The dataset used in this experiment is a self-built corn kernel dataset. Corn kernels of different degrees of breakage were randomly selected and laid out on a conveyor belt with a black background. The whole set was arranged in a sealed box with a constant light source. Images of this state were captured by a CCD camera and used as the corn kernel dataset. The dataset was divided into training set, validation set and test set in a ratio of 8:1:1.

[0079] (2-2) Data Augmentation

[0080] To improve the diversity of samples in the dataset, data augmentation was performed on the corn kernel dataset. Data augmentation operations included flipping, Gaussian blurring, rotation, and adjustments to image brightness, scale, and contrast. These operations expanded the corn kernel dataset to 10 times its original size.

[0081] The adjustment parameters need to be determined according to the specific task and data. Taking this invention as an example, the flipping includes horizontal flipping and vertical flipping; the Gaussian blur is randomly adjusted by 0.0-3.0; the rotation is randomly adjusted by -45° to 45°; the brightness is randomly adjusted by 0.8-4.0; the scale is randomly adjusted by 0.15-2.0; and the contrast is randomly adjusted by 0.75-1.5.

[0082] (2-3) Prior Box Extraction

[0083] The dataset was manually labeled using the LabelImg image annotation tool, resulting in PASCAL VOC format XML files as label files. During the annotation process, samples with less than 30% breakage were considered whole kernels and labeled as such; samples with more than 30% breakage were considered broken kernels and labeled as such. The resolution of the images in the corn kernel dataset was adjusted to 416×416, which is required for network input. The K-means algorithm was used to cluster the bounding boxes in the labels, with a cluster center of 6. The clustering results were used as the network's prior boxes, namely (19,30), (21,38), (25,34), (27,25), (28,40), and (32,45).

[0084] The third part, model training, includes the following steps:

[0085] (3-1) Loss Function

[0086] The loss function of the model in this invention, as shown in equation (6), consists of three parts: confidence loss, category loss, and bounding box loss.

[0087]

[0088]

[0089]

[0090] (6)

[0092] Among them, K 2 M represents the number of grid cells in the current feature map, and M represents the number of anchor boxes located on each grid cell. This indicates whether an object exists in the j-th anchor frame of the i-th grid; a value of 1 indicates existence, and a value of 0 indicates non-existence. , Let be the confidence score between the predicted bounding box and the ground truth bounding box on the i-th grid. This is a confidence error weighting factor to address the imbalanced sample problem. This indicates whether an object exists in the j-th anchor frame of the i-th grid; a value of 1 indicates no object, and a value of 0 indicates presence. , Let be the predicted probability and the true probability of the target class c on the i-th grid, IOU be the intersection-union ratio between the predicted and ground truth bounding boxes, d be the Euclidean distance between the center points of the predicted and ground truth bounding boxes, s be the diagonal length of the minimum closure rectangle between the predicted and ground truth bounding boxes, and w be the predicted probability and the true probability of the target class c on the i-th grid. gt h gt ...

[0093] (3-2) Training Strategy

[0094] This invention employs an online mosaic data augmentation strategy during training to improve the model's detection accuracy for small targets. In each training round, a batch of data is extracted from the corn kernel dataset described in step (2-1). Four images are randomly extracted from this batch, and they are cropped and stitched together at random positions to synthesize new images. This process is repeated until a new number of images equal to the batch size is generated. All the generated new images are then fed into the network for training as part of this batch of data.

[0095] (3-3) Training fine-tuning

[0096] The training process employs fine-tuning techniques. First, the weights of the model's feature extraction network are frozen, and hyperparameters are set for initial training. Then, the weights of the feature extraction network are unfrozen, allowing for normal updates and iterations, and hyperparameters are set to complete the training. Tensorboard is used to record data during training. The training set loss is recorded for each iteration, and the validation set loss is recorded for each training epoch, while the model weights are saved. The model with the lowest validation set loss during the subsequent unfrozen training is taken as the training result.

[0097] Hyperparameters need to be set according to the specific task and hardware configuration. Taking this invention as an example, the Adam optimizer is used during training with default parameters and a learning rate decay of 0.9. During the frozen training phase, the batch size is 64, and the initial learning rate is 1×10⁻⁶. -3 The training rounds are 50; during the unfreezing training phase, the batch size is 16, and the initial learning rate is 1×10. -4 The training rounds are 50.

[0098] Part Four: Recognition of the Image to be Detected

[0099] The images to be detected from the corn kernel dataset test set are sequentially input into the real-time corn kernel detection network trained in the third part for recognition and localization. For features extracted by the lightweight feature extraction network, a feature pyramid network with coordinate attention mechanism is used to combine feature information at different scales to enhance the detection of dense targets. After filtering with a confidence threshold of 0.5, the detection results are obtained. The model is tested on the corn kernel dataset test set mentioned in this invention, and the detection results are as follows: Figure 7-8 As shown in Table 1, the detection performance is as follows. The results show that the present invention can efficiently and in real time detect the crushing state of corn kernels.

[0100] Table 1 Model Detection Performance

[0101] Model Weight size / MB Desktop platform detection speed / (f·s⁻¹) Embedded platform detection speed / (f·s⁻¹) This invention 22.7 91.81 31.94

[0102] Part 5: Calculating Grain Breakage Rate Using Test Results

[0103] The number of intact grains N in the fourth part of the test results w and the number of broken grains N b Statistical analysis was conducted, and the grain breakage rate P was calculated according to formula (7). c .

[0104] (7)

[0105] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time detection of corn kernel breakage rate integrating an attention mechanism, characterized in that, The method includes the following steps: Modeling steps: Construct a real-time corn kernel breakage rate detection network based on YOLOv4-tiny; preprocess data and construct a dataset; train the model using data from the dataset, and adjust and optimize the model parameters; the construction of the real-time corn kernel breakage rate detection network based on YOLOv4-tiny includes the following steps: A coordinate attention mechanism module is introduced into the feature pyramid network. Two spatial pooling kernels are used to encode each channel along the horizontal and vertical coordinate directions to obtain feature maps in two directions. The two feature maps are concatenated, their dimensionality reduced, and activated using a nonlinear function to obtain an intermediate attention map f. Then, f is decomposed along the spatial dimension into a horizontal attention tensor f. h and vertical attention tensor f w ; For f h and f w The dimensionality is increased, and non-linear activation is performed using the Sigmoid function to obtain the coordinate attention module g that embeds feature information. h and g w ; Using the attention module g h and g w Weighting operations are performed on the input feature maps to apply coordinate attention, resulting in the structure of the real-time detection network model for corn kernel breakage rate. The process of training the model, adjusting and optimizing the model parameters, includes employing an online mosaic data augmentation method during training, comprising the following steps: In each training round, a batch of data is extracted from the corn kernel dataset. Four images are randomly extracted from this batch of data, cropped and stitched at random positions, and a new image is synthesized. Repeat the above steps until a new batch of images is generated, with the same number of images as the current batch. Then, feed all the generated images into the network as the current batch of data for training. During training, hyperparameters are set and network weights are updated iteratively; the model network weights and hyperparameters with the lowest loss on the validation set are used as the final optimized network. Breakage rate identification steps: Use an optimized network model to identify the image to be detected; calculate the kernel breakage rate using the corn kernel breakage detection results.

2. The method for real-time detection of corn kernel breakage rate based on an attention mechanism according to claim 1, characterized in that: The data preprocessing and dataset construction includes the following steps: constructing a corn kernel dataset; data augmentation; prior box extraction.

3. The method for real-time detection of corn kernel breakage rate based on an attention mechanism according to claim 2, characterized in that: Data augmentation operations include mirroring, Gaussian blurring, rotation, and adjustments to image brightness, scale, and contrast.

4. The method for real-time detection of corn kernel breakage rate based on an attention mechanism according to claim 2, characterized in that: In the prior box extraction step, the K-means algorithm is used to cluster the size of the bounding boxes in the label data of the image-enhanced corn kernel dataset, and the clustering results are used as the model prior boxes.

5. The method for real-time detection of corn kernel breakage rate based on an attention mechanism according to claim 1, characterized in that: The process of using an optimized network model to identify the image to be detected includes the following steps: a. Read the image to be detected; the image to be detected is a real-time image from the site or a test set image; b. Input the image to be detected into the trained real-time corn kernel breakage rate detection network for identification and localization; Repeat steps a to b until the images in the test set are detected, and identify whether the current image is a broken corn kernel image.

6. The method for real-time detection of corn kernel breakage rate based on an attention mechanism according to claim 1, characterized in that: The calculation of the kernel breakage rate using the corn kernel breakage test results includes: The cumulative number of intact grains N in the test results w and the number of broken grains N b Calculate the grain breakage rate according to the following formula. .

Images