Target tracking algorithm-based hole counting method for seeds sown by sower

Abstract

The invention discloses a target tracking algorithm-based sowing seed hole-falling counting method of a sowing machine. The method comprises the following steps of A, acquiring a sowing seed hole-falling video in a sowing process of the sowing machine in real time; b, constructing a sowing grain hole-falling data set based on the hole-falling video; c, constructing a sowing seed detection model for detecting and positioning sowing seeds; d, training to obtain a trained sowing grain detection model; and E, performing real-time target detection on the seeding grain hole-falling video to be subjected to hole-falling counting from a first frame by using the trained seeding grain detection model, and calculating the inter-frame grain distance of the same grain among different frame numbers by combining a target tracking algorithm from a second frame, when the inter-frame grain distance meets a set threshold condition, continuous tracking is carried out, and hole falling event judgment and counting are completed. According to the invention, the sowing grain hole falling condition in the sowing process can be accurately detected in real time, the non-sowing position and the sowing quantity are fed back, and the sowing precision and efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method for counting the number of seeds falling into a seed hole in a seeder based on a target tracking algorithm. Background Technology

[0002] In modern agricultural production, accurate seed counting during seed planting is crucial for ensuring even crop distribution and maximizing yield. Traditional corn planters rely on mechanical counters or manual counting, methods that are time-consuming, labor-intensive, and inefficient. With the development of computer vision technology, deep learning-based image recognition technology has provided new possibilities for automated seed counting. Summary of the Invention

[0003] The purpose of this invention is to provide a seed-grain-counting method based on a target tracking algorithm for seeders, which can detect the seed-grain-grain-counting situation in real time during the seeding process and provide feedback on unsown positions and seed quantities, thereby improving seeding accuracy and efficiency.

[0004] The present invention adopts the following technical solution:

[0005] A method for counting the number of seeds falling into a seed hole in a seeder based on a target tracking algorithm includes the following steps:

[0006] A: Real-time acquisition of video of seeds falling into holes during the seeding process of the seeder;

[0007] B: Obtain image samples from the seed-planting and seed-planting videos and construct a seed-planting and seed-planting dataset;

[0008] C: Construct a seed detection model for detecting and locating sown seeds; the seed detection model includes a backbone network, a neck network, and a detection head; the backbone network is used to extract the shape features, depth spatial features, and information association features of the sown seeds and output fused features; the neck network outputs a multi-scale enhanced feature map based on the fused features; the detection head predicts the bounding box, object confidence, and class probability of each candidate seed target location based on the multi-scale enhanced feature map;

[0009] D: Using the seed-dropping dataset, the constructed seed detection model is trained, validated, and tested to obtain the trained seed detection model.

[0010] E: Using the trained seed detection model, real-time target detection is performed on the seed landing video to be counted, starting from the first frame. Starting from the second frame, the inter-frame seed distance of the same seed between different frame numbers is calculated using a target tracking algorithm. When the inter-frame seed distance meets the set threshold condition, the tracking continues to complete the determination and counting of the landing event.

[0011] The backbone network includes a shape feature extraction module, a depth spatial feature extraction module, a seed information association module, and a feature fusion module. The shape feature extraction module is used to extract the shape features of the sown seeds. The depth spatial feature extraction module is used to extract the orientation angle of the sown seeds, as well as the spatial features of the distance and overlapping areas between the seeds. The seed information association module, based on the shape features and depth spatial features of the sown seeds, fuses the spatial and semantic information of the local environment where the seeds are located to obtain the information association features of the sown seeds. The feature fusion module is used to output the fused features by fusing the shape features, depth spatial features, and information association features of the sown seeds.

[0012] The neck network adopts a bidirectional feature pyramid structure, which upsamples the fused features output by the backbone network and then fuses them again with the shallow high-resolution features. Subsequently, the re-fused features are downsampled and passed up the layers to enhance the spatial localization accuracy of deep features. Finally, a multi-scale enhanced feature map is output.

[0013] Based on the feature information output by the neck network, the detection head predicts the bounding box, object confidence, and class probability of each candidate seed target location through parallel bounding box regression branch, object confidence branch, and class probability branch.

[0014] The loss function of the seed detection model is replaced by the Focal-ShapeIoU loss function, which is composed of Focal loss and Shape-IoU, to replace the original YOLOv11s model loss function. This combines the dynamic adjustment characteristics of Focal Loss with the shape and scale concern characteristics of ShapeIoU.

[0015] The steps for determining and counting the pitfall event in step E are as follows:

[0016] E1: For the first frame of the video, obtain the predicted seed positions, class probabilities and object confidence of all predicted seeds in the frame output by the seed detection model. Based on the set class threshold and confidence threshold, classify all predicted seed targets appearing in the first frame of the current seed-planting video. Predicted seed targets that meet the class determination are taken as valid seed targets and assigned target identifier numbers.

[0017] E2: Starting from the second frame, the target tracking algorithm continuously tracks the motion trajectory of the current effective seed target in subsequent video frames, predicts the center position coordinates of each effective seed target in the current frame in subsequent video frames, and updates the candidate seed target position information in real time.

[0018] E3: Based on the predicted center coordinates of the effective grain targets in the current frame, and combined with the center coordinates of the effective grain targets in the previous M frames, a continuity determination is made.

[0019] E4: Record the temporary position of the continuous valid seed target in the current frame and mark the continuous valid seed target as "tracking status"; mark the non-continuous valid seed target as "tracking doubt status";

[0020] E5: Process each frame according to the above steps and record the spatiotemporal trajectory of each continuous effective grain target from the first detection to the end of tracking.

[0021] E6: Recognize the image of the last frame in which the seed appears, and determine whether the seed appears in the acupoint; if so, it is determined that the seed placement is successful; otherwise, it is determined that the seed placement is unsuccessful, the seed placement failure counter is incremented by 1, and the current seed position is recorded.

[0022] In step E1, for the first frame image of the current seed seed landing video, the category is first determined based on the category probability of each predicted seed in the first frame image output by the seed seed detection model and the set category threshold.

[0023] If the current predicted category probability of the grain target is greater than or equal to the set category threshold, then a further confidence determination is made based on the object confidence of the predicted grain target and the set confidence threshold.

[0024] If the confidence level of the current predicted grain target is greater than or equal to the set confidence level threshold, then the current predicted grain target is determined to conform to the corresponding sowing category and is a valid grain target; then proceed to step E2;

[0025] If the confidence level of the currently predicted seed target is less than the set confidence threshold, the currently predicted seed target is determined to be an untrusted seed target, and the seed landing event is considered to have failed. No further processing will be performed on the currently predicted seed target; at the same time, the identification failure counter is incremented by 1.

[0026] If the probability of the current predicted grain target category is less than the set category threshold, the current candidate grain target is determined to be inconsistent with the corresponding sowing category, regarded as background interference or false detection target, and will not be further numbered, tracked and counted.

[0027] Subsequently, for all valid seed targets that meet the category and confidence criteria, a unique target identifier is assigned based on the spatial arrangement order of the valid seed targets in the image; then proceed to step E2.

[0028] In step E3, if the Euclidean distance between the center coordinates of the effective seed target in any two adjacent frames is less than or equal to the set distance threshold, the effective seed target is determined to be a continuous effective seed target; otherwise, it is determined to be a non-continuous effective seed target, the seed trajectory has an abnormal interruption, and the candidate seed is recorded as an abnormal trajectory, and the tracking failure counter of the candidate seed number is incremented by 1.

[0029] In step E4, for a valid seed target that is in a "tracking doubt state" for J consecutive frames, if the valid seed target reappears in subsequent frames and meets the determination condition of step E3, i.e., the valid seed target is a continuous valid seed target, then the valid seed target is restored to the "tracking state"; if the number of frames in which the valid seed target fails to track exceeds the preset number of frames G, then it is determined that the valid seed target has landed in the hole or left the field of view, the tracking of the candidate seed is terminated and the feature point set is removed.

[0030] In step E2, the center point of the predicted bounding box of each valid seed target retained after thresholding in the first frame image is first used as the initial feature point set. Local image gradient information around the target is extracted, and a feature point-target mapping relationship is established with each target identifier. Then, starting from the second frame, for all valid seed targets in the initial feature point set in the current and previous frames, optical flow is used to calculate the effective seed targets with the same target identifier number between the two frames from the points in the previous frame. Point in the current frame The velocity vector change is used to obtain the motion direction and velocity estimate of each effective seed target in the feature point set in the pixel space, and then the displacement vector of the seed in the next frame is calculated to obtain the predicted position of the next frame. This predicted position is the continuation point of the seed's motion trajectory in the current frame.

[0031] This invention can detect the seed placement in the planting hole in real time and accurately, and provide feedback on unsown locations and seed quantity, thereby improving planting accuracy and efficiency.

[0032] In this invention, the seed detection model effectively enhances its ability to express the features of seeds sown in holes through a three-branch feature extraction and depth enhancement mechanism. The three-branch structure and... The use of fusion mechanisms significantly enhances the hierarchical expressive power of features.

[0033] In this invention, a loss function is specifically designed. Based on the prediction results and real labels output by the detection head, the bounding box regression loss, object confidence loss and category probability loss are calculated respectively. The total loss function is obtained by weighting and summing them according to preset weights. The model parameters are optimized through backpropagation to improve the positioning accuracy and classification accuracy of corn kernels.

[0034] In this invention, an optical flow target tracking algorithm can be specially constructed by fusing optical flow and target tracking technology based on the basic principle of optical flow method. Under typical imaging conditions of sowing operation, it can achieve highly robust tracking of falling effective grain targets and realize continuous tracking, real-time prediction and updating of the position information of effective grain targets. Attached Figure Description

[0035] Figure 1This is a schematic diagram of the process of the present invention. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0037] In this invention, taking a corn planter as an example, a method for counting the number of seeds falling into a planting hole based on a target tracking algorithm is specifically described.

[0038] like Figure 1 As shown, the seed-grain-counting method for a seeder based on an optical flow target tracking algorithm, as described in this invention, includes the following steps:

[0039] A: Real-time acquisition of video of seeds falling into holes during the seeding process of the seeder;

[0040] In this invention, a video acquisition device can be installed on the seeder, and the video acquisition device can be used to continuously acquire real-time video of the seed-dropping process. The video acquisition device can be a high-definition camera installed on the seeding device in the seeder, and the real-time video of the seed-dropping process with a set resolution can be obtained by adjusting the angle of the high-definition camera.

[0041] In this embodiment, taking the counting of corn kernels falling into the hole as an example, a 180-frame high-definition camera can be fixed in the planting device of the corn planter using a mold, and connected to a computer using a USB interface. The computer can then collect real-time video of the corn kernel falling into the hole process captured by the high-definition camera.

[0042] B: Obtain image samples from the seed sowing and seed dropping video as positive and negative samples, and construct the seed sowing and seed dropping dataset;

[0043] In this invention, image samples are obtained from the seed-planting hole video and positive and negative samples are selected; then, the Maize-SD diffusion model is trained using some positive and negative samples, and the trained model is used to generate multi-pose seed-planting samples for data augmentation, and finally the seed-planting hole dataset is constructed.

[0044] Traditional data augmentation methods rely on geometric transformations, noise, and grayscale. This invention employs the Maize-SD diffusion model as an image generation model for small target detection tasks involving seed kernels (such as corn kernels), to augment the seed kernel dataset. The Maize-SD diffusion model optimizes the seed kernel dataset through multiple poses, removing irrelevant information and diversifying the training data distribution. This exposes the model to more data variations, preventing over-reliance on specific patterns in the original data.

[0045] In this embodiment, from the real-time video of the corn kernels falling into the hole captured by a high-definition camera, frames are extracted to obtain image samples, and appropriate positive and negative samples are selected according to a set standard. Then, a portion of the samples are used to train the Maize-SD diffusion model, and the trained model is used to generate multi-pose corn kernel samples for data augmentation to construct a corn kernel falling into the hole dataset.

[0046] Finally, the pixel values ​​of the image samples in the seed-planting hole dataset can be normalized from [0, 255] to [0, 1], and then the normalized pixel information can be used as the input information for the subsequent seed-planting detection model.

[0047] C: Construct a seed detection model for detecting and locating sown seeds;

[0048] In this invention, the seed detection model is constructed based on the improved YOLOv11s model, which can effectively improve the detection speed and feature extraction effect of seeds sown in the hole state; the seed detection model includes a backbone network, a neck network, and a detection head;

[0049] The backbone network is used to extract the shape features, depth spatial features, and information association features of the sown grains through three parallel branches, and output the fused features through feature fusion.

[0050] The neck network employs a bidirectional feature pyramid structure, which upsamples the fused features output from the backbone network and further fuses them with shallow high-resolution features (fine edge and position information features of the grains). The output features are then downsampled and passed up level by level to enhance the accuracy of candidate grain target localization in the deep features, and output multi-scale enhanced feature maps. In the neck network, the fusion of deep feature upsampling and shallow high-resolution features is a conventional technique in this field and will not be elaborated here.

[0051] The detection head is used to enhance the feature map based on the output of the neck network at multiple scales. It predicts the bounding box, object confidence, and class probability of each candidate seed target location through parallel bounding box regression branch, object confidence branch, and class probability branch.

[0052] In this embodiment, the backbone network of the seed detection model adopts an Efficient Multi-Branch architecture, utilizing three parallel branches to extract the shape features, depth spatial features, and information association features of the seed. The input pixel information of the seed detection model is... The outputs are feature tensors respectively. ;in, This represents the pixel information after the dataset has been normalized. The tensor representing the shape characteristics of the sown seeds. Represents the spatial feature tensor of seed depth. Represents the grain information association tensor;

[0053] In this invention, the backbone network of the seed detection model includes a shape feature extraction module, a depth spatial feature extraction module, a seed information association module, and a feature fusion module;

[0054] The shape feature extraction module is used to extract the shape features of the sown kernels (such as the elongated oval shape of corn kernels); the shape feature extraction module can employ... Convolution is used to compress and enhance the distinctive shape of grains; the expression for the shape feature extraction module is:

[0055] ;

[0056] in, Represents the shape feature tensor. express convolution, This represents the pixel information of the input data;

[0057] Since the seeds are in motion during sowing, the depth spatial feature extraction module is used to extract the orientation angle (horizontal or lateral) of the sown seeds, as well as the spatial features of the distance and overlapping areas between the seeds. The depth spatial feature extraction module employs grouped convolutions... and pointwise convolution The constructed depthwise separable convolution DSConv; depthwise separable convolution captures features spatially using grouped convolutions and performs channel blending through pointwise convolutions. The computational complexity of depthwise separable convolution is significantly lower than that of standard convolution, but it has better sensitivity to feature representations of local defects. The expression for the depth spatial feature extraction module is:

[0058]

[0059] in, Represents the spatial feature tensor of seed depth. This represents pointwise convolution. Indicates grouped convolution;

[0060] The seed information association module, based on accurate identification of seed shape and depth spatial features, integrates spatial and semantic information of the local environment where the seed is located (such as the structure of the sowing hole, neighboring seeds, background material, and falling state) to obtain the sown seed information, thereby achieving high-precision and robust intelligent judgment of sowing quality (falling state, quantity, and posture). The global seed information association module, built on the CSP framework and supporting dynamically adjustable convolutional kernels, inputs pixel information... After background segmentation, the seed feature tensor is processed by the ConvCSP deep augmentation unit to capture more complex seed sowing context connectivity information. Based on the GELAN concept, the deep augmentation unit is designed to support dynamic adjustment of the convolution kernel size. This enables the seed detection model to extract features at different scales, expanding the receptive field and particularly helping to capture more complex spatial features of the seeding state. The expression for the multi-scale feature extraction module for seed sowing is:

[0061] ;

[0062] in, This represents the grain information association tensor before depth enhancement. This indicates that background segmentation is performed on the first dimension (dim=1). express The grain feature tensor and background information tensor formed after background segmentation This represents the grain information association tensor after depth enhancement. It is a depth enhancement unit composed of three similar sub-modules connected in series, where, The combination of convolution kernel sizes used within these submodules is specified; in this invention, n = 1 or 2;

[0063] The feature fusion module is used to integrate the extracted seed shape features, depth spatial features, and information association features into a single sheet-like spliced ​​feature by concatenating them along the channel dimension. This is then processed by a... Convolution maps concatenated features to fused features in the output. The expression for the feature assembly module is:

[0064]

[0065] Existing YOLOv11 models typically perform target detection on feature maps at multiple scales. However, the feature maps used in the original architecture for detecting small targets such as seeds have relatively low resolution. For example, the P3 / 8 feature layer in the original YOLOv11 model has a relatively low resolution for targets with dimensions of [missing information]. arrive For tiny targets around one pixel in size, the receptive field of the feature map is still too large, easily losing the position and detailed information of small targets. This makes it difficult for the existing YOLOv11s model to accurately detect and locate small targets like seeds. Therefore, in this invention, a seed detection model is constructed based on the YOLOv11s model. It can utilize the Efficient Multi-Branch architecture, which extracts different features from the seeds using three parallel branches, thereby effectively improving the detection speed and feature extraction effect of seeds sown in the planting hole state.

[0066] In this invention, the seed detection model effectively enhances its ability to express the features of seeds sown in holes through a three-branch feature extraction and depth enhancement mechanism. The three-branch structure and... The use of fusion mechanisms significantly enhances the hierarchical expressive power of features.

[0067] In this invention, the YOLOv11s neck network is used to process the fused features output by the feature fusion module. The system processes and transmits strong semantic information (so that the model "knows that this is a corn kernel") and precise location information (so that the model "knows where the kernel boundary is").

[0068] The neck network employs a bidirectional feature pyramid structure, first combining the deep, high-semantic features (fusion features) output by the backbone network. After upsampling, the features are fused again with shallow high-resolution features (fine edge and position information features of kernels) to enhance the classification ability of small-scale corn kernels. Then, the fused features are downsampled and passed up the scale to enhance the spatial positioning accuracy of deep features. Finally, a multi-scale enhanced feature map is output to enable the detection head to achieve high recall and high positioning accuracy of kernels.

[0069] Based on the feature information output by the neck network, the detection head predicts the bounding box (predicting the normalized center coordinates and width and height of the corn kernel) of each candidate kernel target location through parallel bounding box regression branch, object confidence branch, and class probability branch. The object confidence (the probability value of the existence of a real corn kernel target within the bounding box) and class probability (such as the probability value of each candidate kernel belonging to the "corn kernel" category in the single-class detection mode (identifying only one target, corn kernel) of the seed kernel detection model) are calculated.

[0070] In this invention, a loss function is specifically designed. Based on the prediction results and real labels output by the detection head, the bounding box regression loss, object confidence loss and category probability loss are calculated respectively. The total loss function is obtained by weighting and summing them according to preset weights. The model parameters are optimized through backpropagation to improve the positioning accuracy and classification accuracy of corn kernels.

[0071] The YOLOv11s model employs the CIoU loss function, which suffers from significant localization errors and insufficient balance for small targets like seeds (objects with a pixel area less than 32×32). The CIoU loss function also exhibits a relatively large positional deviation between the predicted seed size bounding box and the actual seed shape bounding box. It insufficiently penalizes predicted boxes with little or no intersection with the actual bounding boxes (i.e., small target samples like seeds, such as corn kernels), making it difficult to fully learn from seed samples during training. Particularly for small targets with significant aspect ratio differences, such as when seeds are elongated or deformed under high-speed camera conditions, localization errors may occur, failing to accurately reflect the matching degree of the seed's bounding box.

[0072] In view of the above situation, in this invention, the loss function of the seed detection model is Focal-ShapeIoU to improve the positioning accuracy of corn kernels.

[0073] The Focal-ShapeIoU loss function consists of Focal loss and Shape-IoU. Focal-ShapeIoU combines the dynamic adjustment characteristics of Focal Loss with the shape and scale-focused characteristics of ShapeIoU, forming a more effective loss function. Focal loss is a loss function based on binary cross-entropy (CE). It is a dynamically scaled cross-entropy loss. Through a dynamic scaling factor, the weights of easily distinguishable corn kernel samples can be dynamically reduced during training, thereby quickly focusing the attention on those difficult-to-distinguish corn kernel samples.

[0074] Focal loss is defined as:

[0075]

[0076] in, This represents the positive class probability of the corn kernel predicted by the model, that is, the probability that the pixel box of the kernel belongs to the target class; This represents the balance factor, used to control the weights between positive and negative samples of grains, thus resolving class imbalance. This represents the focus parameter, used to control the degree of "reduction" in weighting easily classified grains, allowing the model to focus more on difficult-to-classify grains;

[0077] ShapeIoU, based on the IoU of the YOLOv11s model, incorporates a method for measuring the shape and scale of the seed bounding box. It not only considers the overlap area between the predicted and ground truth boxes but also measures the shape difference between them by introducing a seed shape distance and a penalty term. The ShapeIoU loss is defined as:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] in, The distance used to measure the geometric difference between the predicted bounding box and the true bounding box. This represents the sum after nonlinear compression of the width and height errors. Represents the grain prediction box. Represents the actual frame of the grain. This represents the overlap area between the predicted grain bounding box and the actual grain bounding box. This represents the total area of ​​the predicted grain bounding box and the actual grain bounding box. This represents the height weighting coefficient associated with the grain prediction frame. This represents the width weighting coefficient associated with the grain prediction box; and These represent the x and y coordinates of the center of the prediction box, respectively. and These represent the x and y coordinates of the center of the ground truth bounding box, respectively; the subscript 'c' indicates the predicted bounding box, and the subscript 'gt' indicates the ground truth bounding box. and These represent the height and width of the input image after normalization, which are usually scaled (e.g., divided by the image size or using a logarithmic transformation). This represents the relative error after normalization of the predicted box; This indicates summing over two dimensions, h and w. This indicates that a nonlinear mapping is used, resulting in a small contribution from small errors and a significant increase from large errors. →0, this term approaches 0; As the value increases, this item rises rapidly; It is a hyperparameter representing the steepness of the curve; These represent the relative errors after normalization of the predicted bounding box, incorporating both scale and shape information. This represents the absolute difference between the predicted grain bounding box width and the actual grain bounding box width. This represents the absolute difference between the predicted grain bounding box height and the actual grain bounding box height. It is the normalized denominator of the ratio of the relative scale error between the predicted grain frame width and the actual grain frame width. It is the normalized numerator of the ratio of the relative scale error between the predicted grain frame height and the actual grain frame height, thus avoiding the dominance of large scale error.

[0087] D: Using the seed-dropping dataset, the constructed seed detection model is trained, validated, and tested to obtain the trained seed detection model.

[0088] E: Using the trained seed detection model, real-time target detection is performed on the seed-falling video of the seed to be counted, starting from the first frame. Starting from the second frame, the inter-frame seed distance of the same seed between different frame numbers is calculated by combining the target tracking algorithm. When the inter-frame seed distance meets the set threshold condition, the tracking is continuously performed. Seeds that move smoothly, have continuous trajectories and fall into the hole in the entire visible time period are judged as successfully falling into the hole, thus completing the judgment and counting of the falling event.

[0089] In step E, real-time target detection is performed on the images starting from the first frame of the seed-planting video, based on the seed detection model. The specific process of real-time target detection is as follows:

[0090] The video stream to be processed is decoded frame by frame in chronological order to obtain the current frame image; subsequently, the preprocessed image pixel information is... The input is fed into the trained seed detection model, which extracts the shape feature tensor of the seed through three parallel branches of the backbone network. Depth space feature tensor and information-related tensors Computational fusion features fusion features After being fused again by the neck network, the detection head outputs predicted bounding boxes, their center coordinates, object confidence scores, and class probabilities in parallel across multiple feature scales, ultimately obtaining the location information of all corn kernels in the current frame. The obtained predicted kernel bounding box center coordinates, object confidence scores, and class probabilities are used as the initial observation input for the target tracking algorithm, subsequently used for cross-frame trajectory association and corn pit counting.

[0091] In this invention, the target tracking algorithm is used to determine and count the occurrence of seeds falling into holes based on the detection results of each frame of the image output by the seed detection model. The specific steps are as follows:

[0092] E1: For the first frame of the video, obtain the predicted seed positions (i.e., the center coordinates of the predicted bounding box corresponding to the predicted seed), class probabilities, and object confidence scores of all predicted seed positions (i.e., the center coordinates of the predicted bounding box corresponding to the predicted seed) in the first frame of the video output by the seed detection model. Based on the set class threshold and confidence threshold, classify all predicted seed targets appearing in the first frame of the current seed-planting video. Predicted seed targets that meet the class determination are taken as valid seed targets and assigned target identifier numbers.

[0093] In this embodiment, for the first frame image of the current seed seed landing video, the category is first determined based on the category probability of each predicted seed in the first frame image output by the seed seed detection model and the set category threshold (which can be set to 0.5 in this embodiment).

[0094] If the current predicted category probability of the grain target is greater than or equal to the set category threshold, then further confidence determination is performed based on the object confidence of the predicted grain target and the set confidence threshold (which can be set to 0.7 in this embodiment).

[0095] If the confidence level of the current predicted grain target is greater than or equal to the set confidence level threshold, then the current predicted grain target is determined to conform to the corresponding sowing category and is a valid grain target; then proceed to step E2;

[0096] If the confidence level of the current predicted grain target is less than the set confidence level threshold, the current predicted grain target is determined to be an untrusted grain target, and the grain landing event is considered to have failed. No further processing, such as subsequent detection result association and target tracking, will be performed on the current predicted grain target. At the same time, the identification failure counter will be incremented by 1 for real-time statistics of sowing quality anomalies.

[0097] If the probability of the current predicted grain target category is less than the set category threshold, the current candidate grain target is determined to be inconsistent with the corresponding sowing category, regarded as background interference or false detection target, and will not be further numbered or participated in subsequent tracking and counting.

[0098] Subsequently, for all valid seed targets that meet the category and confidence criteria, a unique target identifier is assigned according to the spatial arrangement order of the valid seed targets in the image, and is sequentially denoted as No. 1, No. 2... No. NUM, where NUM is the number of all valid seed targets in the first frame image; the assigned target identifier remains unchanged in its subsequent frames, that is, the same valid seed target maintains the same number in different frames of the image to ensure consistency; then proceed to step E2; in this embodiment, the valid seed targets are sorted in ascending order according to the ordinate of the center point of the prediction box corresponding to the image;

[0099] In this embodiment, the target identifier adopts a global continuous numbering strategy, that is, the target number of the first valid grain target in the current seeding hole video is the next sequence number of the number assigned at the end of the previous seeding hole video, ensuring that the numbering between different seeding operation video sequences is not repeated or skipped.

[0100] E2: Starting from the second frame, the target tracking algorithm continuously tracks the motion trajectory of the current effective seed target in subsequent video frames, predicts the center position coordinates of each effective seed target in the current frame in subsequent video frames, and updates the candidate seed target position information in real time.

[0101] The continuous tracking of the effective grain target trajectory can be achieved using existing target tracking algorithms, which will not be elaborated here.

[0102] E3: Based on the predicted center coordinates of the effective grain targets in the current frame, and combined with the center coordinates of the effective grain targets in the previous M frames, a continuity determination is made.

[0103] If the Euclidean distance between the center coordinates of the effective seed target in any two adjacent frames is less than or equal to the set distance threshold, then the effective seed target is determined to have smooth motion and continuous trajectory throughout the entire visible time period, and belongs to the continuous effective seed target.

[0104] Otherwise, it is determined to be a non-continuous valid grain target, the grain trajectory has an abnormal interruption, and the candidate grain is recorded as an abnormal trajectory. The tracking failure counter of the candidate grain number is incremented by 1.

[0105] In this embodiment, M can be set to 3; at this time, the current frame and the three frames before the current frame together form a four-frame image judgment combination;

[0106] E4: Record the temporary position of the continuous valid seed target in the current frame and mark the continuous valid seed target as "tracking status"; mark the non-continuous valid seed target as "tracking doubt status";

[0107] In this embodiment, for a valid seed target that is in a "tracking doubt state" for J consecutive frames (J=3), if the valid seed target reappears in subsequent frames and meets the determination condition of step E3, i.e., the valid seed target is a continuous valid seed target, then the valid seed target is restored to the "tracking state"; if the number of frames in which the valid seed target fails to track (i.e. is in a tracking doubt state) exceeds the preset number of frames G (G=9), then it is determined that the valid seed target has landed in the hole or left the field of view, the tracking of the candidate seed is terminated and the feature point set is removed;

[0108] E5: Process each frame according to the above steps and record the spatiotemporal trajectory of each continuous effective grain target from the first detection to the end of tracking.

[0109] E6: Recognize the image of the last frame in which the seed appears, and determine whether the seed appears in the acupoint; if so, it is determined that the seed placement is successful; otherwise, it is determined that the seed placement is unsuccessful, the seed placement failure counter is incremented by 1, and the current seed position is recorded.

[0110] In this invention, based on the fundamental principles of optical flow, a special optical flow target tracking algorithm is constructed by fusing optical flow with target tracking technology. Under typical imaging conditions in sowing operations, this algorithm achieves highly robust tracking of falling effective grain targets, enabling continuous tracking, real-time prediction, and updating of the effective grain target's position information. The specific implementation steps are as follows:

[0111] Optical flow is an existing computer vision algorithm used to estimate the motion vectors of pixels or feature points in an image sequence. Its basic principle is based on the assumption of constant brightness and spatial smoothness constraints. By solving the motion field implied by the gray-level changes between adjacent frames, it achieves continuous modeling of the target motion trajectory.

[0112] Based on existing optical flow algorithms, this invention specifically designs an optical flow target tracking method for small-scale agricultural targets, addressing the characteristics of small target size, high movement speed, and strong environmental interference in corn kernel landing scenarios. This method uses the center points of the effective grain bounding boxes output by the seed detection model as the initial feature point set. By optimizing optical flow estimation and trajectory management strategies, it achieves highly robust, low-latency continuous tracking of a single corn kernel throughout its entire fall from the air to its stable landing in the hole.

[0113] The specific principle of the optical flow method in this invention is as follows:

[0114] In this embodiment, it is assumed that a position in a certain frame of the video image is pixels in the time dimension The light intensity below is The pixel moved The time taken to reach the next consecutive frame after the distance is The optical flow method assumes that the light intensity of a pixel remains constant before and after movement, i.e.:

[0115]

[0116] Expanding the right-hand side of equation (1) using a first-order Taylor expansion, we get:

[0117]

[0118] in, Indicates the image at position ,time Brightness / intensity It represents minute displacements in space, specifically the distance moved horizontally and vertically, measured in pixels. Indicates the image in Spatial gradient in direction, i.e., the rate of change of pixel values ​​in the horizontal direction (fundamentals of edge detection). Indicates the image in Spatial gradient in direction, i.e., the rate of change of pixel values ​​in the vertical direction. It represents the rate of change of an image in the time direction, that is, the change in brightness at the same location over time (such as due to the movement of an object). Represent a higher-order infinitesimal; then eliminate both sides of equation (2) simultaneously. and divide by We can obtain:

[0119]

[0120] set up and Optical flow along shaft and The velocity vector of the axis, i.e.:

[0121]

[0122] make These represent the image grayscale along... The partial derivative in the direction, i.e.:

[0123]

[0124] In summary, the optical flow constraint formula can be obtained as follows:

[0125]

[0126] in, All of these can be obtained from video image data, and This is the desired optical flow vector. Since there are two unknowns in equation (6), it cannot be solved using a single pixel. Therefore, it can be used... Multiple equations are established for each pixel, where n represents n rows and m represents m columns:

[0127]

[0128] in, Indicates at point along Gray-scale change rate in the axial direction (horizontal), Indicates at point along Gray-scale change rate in the axial direction (perpendicular), For optical flow along Velocity vector of axis (unit: pixels / frame); For optical flow along The velocity vector of the axis, At point The brightness of the location changes over time;

[0129] By solving the above equations, the optical flow path can be obtained. shaft and Change in velocity vector of the shaft :

[0130]

[0131] in, and These represent the gradient energy in the horizontal and vertical directions, respectively, reflecting the local texture intensity of the grain. The gradient covariance represents the consistency of the movement direction of different grains. This indicates the correlation between the horizontal gradient and its time variation. This indicates the correlation between the vertical gradient and its time variation.

[0132] Specifically, in step E2 of this invention:

[0133] First, the center point of the predicted bounding box of each valid seed target retained after threshold screening in the first frame image is used as the initial feature point set. The gradient information of the surrounding local image is extracted, and a feature point-target mapping relationship corresponding one-to-one with the target identifier is established.

[0134] Then, starting from the second frame, for all valid seed targets in the initial feature point set of the current frame and the previous frame image, the optical flow method is used to calculate the effective seed targets with the same target identifier number between the two frames from the points in the previous frame. Point in the current frame velocity vector change The algorithm obtains the motion direction and velocity estimate of each effective seed target in the feature point set in the pixel space, and then calculates the displacement vector of the seed in the next frame to obtain the predicted position of the next frame. This predicted position is the continuation point of the seed's motion trajectory in the current frame.

[0135] ,

[0136] in, and Optical flow along shaft and The velocity vector of the axis, and These represent the gradient energy in the horizontal and vertical directions, respectively, reflecting the local texture intensity of the grain. The gradient covariance represents the consistency of the movement direction of different grains. This indicates the correlation between the horizontal gradient and its time variation. This indicates the correlation between the vertical gradient and its time variation.

[0137] In this invention, the trained seed detection model can be deeply integrated with the optical flow target tracking algorithm to construct a generalizable and transferable intelligent counting system for corn seed placement. When the intelligent counting system for corn seed placement is first applied to a new seeder, the following standardized deployment process is executed:

[0138] (1) Collect video data of new machine operation: Under various field operation conditions (including different forward speeds, seeding frequency and lighting conditions), the industrial camera installed directly above the seeding area continuously collects high-definition video sequences of no less than one complete seeding cycle at a frame rate of no less than 60 frames / second, ensuring that the video contains typical working conditions such as normal seeding, missed seeding and seed offset.

[0139] (2) Construct a new model adaptation sample set: Sample key frames uniformly over time from the collected video and extract image regions as original samples; Train the Maize-SD diffusion model using some of the original samples, and use this model to perform semantically guided generative enhancement on the original samples to synthesize positive samples of corn kernels with real texture and morphological distribution, and combine background replacement and occlusion simulation to generate negative samples, forming a domain-adaptive seeding and hole-dropping fine-tuning dataset for the new model.

[0140] (3) Perform lightweight model transfer learning: Using the original seed detection model (based on YOLOv11 architecture) as the initial weights, a high-precision three-branch backbone network and a bidirectional feature pyramid structure neck network are used to perform sample transfer training on the fine-tuned dataset; combined with the improved loss function Focal-ShapeIoU, which is suitable for identifying small target seeds, the localization accuracy and classification robustness of small target seeds, tilted and partially occluded seeds are improved.

[0141] (4) Integrated detection, tracking and counting: When the system is running, the real-time video stream is input frame by frame into the fine-tuned seed detection model. Starting from the first frame, the predicted box, category probability and object confidence of all candidate seeds are output. Then, the optical flow target tracking algorithm is used to perform data association on the detection results of each seed. If it is within the set threshold range, the trajectory of each corn seed is generated and the success of the seed landing is detected.

[0142] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for counting the number of seeds falling into a seed hole in a seeder based on a target tracking algorithm, characterized in that, Includes the following steps: A: Real-time acquisition of video of seeds falling into holes during the seeding process of the seeder; B: Obtain image samples from the seed-planting and seed-planting videos and construct a seed-planting and seed-planting dataset; C: Construct a seed detection model for detecting and locating sown seeds; the seed detection model includes a backbone network, a neck network, and a detection head; the backbone network is used to extract the shape features, depth spatial features, and information association features of the sown seeds and output fused features; the neck network outputs a multi-scale enhanced feature map based on the fused features; the detection head predicts the bounding box, object confidence, and class probability of each candidate seed target location based on the multi-scale enhanced feature map; D: Using the seed-dropping dataset, the constructed seed detection model is trained, validated, and tested to obtain the trained seed detection model. E: Using the trained seed detection model, real-time target detection is performed on the seed landing video to be counted, starting from the first frame. Starting from the second frame, the inter-frame seed distance of the same seed between different frame numbers is calculated using a target tracking algorithm. When the inter-frame seed distance meets the set threshold condition, the tracking continues to complete the determination and counting of the landing event.

2. The method for counting holes according to claim 1, characterized in that: The backbone network includes a shape feature extraction module, a deep spatial feature extraction module, a grain information association module, and a feature fusion module; The shape feature extraction module is used to extract the shape features of the sown seeds; the depth spatial feature extraction module is used to extract the orientation angle of the sown seeds, as well as the spatial features of the distance between seeds and the overlapping areas. The seed information association module, based on the shape and depth spatial features of the sown seeds, integrates the spatial and semantic information of the local environment where the seeds are located to obtain the information association features of the sown seeds. feature The fusion module is used to output fused features by fusing the shape features, depth spatial features, and information association features of the sown grains.

3. The method for counting holes according to claim 1, characterized in that: The neck network adopts a bidirectional feature pyramid structure, which upsamples the fused features output by the backbone network and then fuses them again with the shallow high-resolution features. The re-fused features are then downsampled and passed up the hierarchy to enhance the spatial localization accuracy of deep features; finally, a multi-scale enhanced feature map is output.

4. The method for counting holes according to claim 1, characterized in that: Based on the feature information output by the neck network, the detection head predicts the bounding box, object confidence, and class probability of each candidate seed target location through parallel bounding box regression branch, object confidence branch, and class probability branch.

5. The method for counting holes according to claim 1, characterized in that: The loss function of the seed detection model is the Focal-ShapeIoU loss function, which is composed of Focal loss and Shape-IoU. This combines the dynamic adjustment characteristics of Focal Loss with the shape and scale-related characteristics of ShapeIoU.

6. The method for counting holes according to claim 1, characterized in that: The steps for determining and counting the pitfall event in step E are as follows: E1: For the first frame of the video, obtain the predicted seed positions, class probabilities and object confidence of all predicted seeds in the frame output by the seed detection model. Based on the set class threshold and confidence threshold, classify all predicted seed targets appearing in the first frame of the current seed-planting video. Predicted seed targets that meet the class determination are taken as valid seed targets and assigned target identifier numbers. E2: Starting from the second frame, the target tracking algorithm continuously tracks the motion trajectory of the current effective seed target in subsequent video frames, predicts the center position coordinates of each effective seed target in the current frame in subsequent video frames, and updates the candidate seed target position information in real time. E3: Based on the predicted center coordinates of the effective grain targets in the current frame, and combined with the center coordinates of the effective grain targets in the previous M frames, a continuity determination is made. E4: Record the temporary position of the continuous valid seed target in the current frame and mark the continuous valid seed target as "tracking state"; mark the non-continuous valid seed target as "tracking doubtful state"; E5: Process each frame according to the above steps and record the spatiotemporal trajectory of each continuous effective grain target from the first detection to the end of tracking. E6: Recognize the image of the last frame in which the seed appears, and determine whether the seed appears in the acupoint; if so, it is determined that the seed placement is successful; otherwise, it is determined that the seed placement is unsuccessful, the seed placement failure counter is incremented by 1, and the current seed position is recorded.

7. The method for counting holes according to claim 6, characterized in that: In step E1, for the first frame image of the current seed seed landing video, the category is first determined based on the category probability of each predicted seed in the first frame image output by the seed seed detection model and the set category threshold. If the current predicted category probability of the grain target is greater than or equal to the set category threshold, then a further confidence determination is made based on the object confidence of the predicted grain target and the set confidence threshold. If the confidence level of the current predicted grain target is greater than or equal to the set confidence level threshold, then the current predicted grain target is determined to conform to the corresponding sowing category and is a valid grain target; then proceed to step E2; If the confidence level of the currently predicted seed target is less than the set confidence threshold, the currently predicted seed target is determined to be an untrusted seed target, and the seed landing event is considered to have failed. No further processing will be performed on the currently predicted seed target; at the same time, the identification failure counter is incremented by 1. If the probability of the current predicted grain target category is less than the set category threshold, the current candidate grain target is determined to be inconsistent with the corresponding sowing category, regarded as background interference or false detection target, and will not be further numbered, tracked and counted. Subsequently, for all valid seed targets that meet the category and confidence criteria, a unique target identifier is assigned based on the spatial arrangement order of the valid seed targets in the image. Then proceed to step E2.

8. The method for counting holes according to claim 6, characterized in that: In step E3, if the Euclidean distance between the center coordinates of the effective seed target in any two adjacent frames is less than or equal to the set distance threshold, the effective seed target is determined to be a continuous effective seed target; otherwise, it is determined to be a non-continuous effective seed target, the seed trajectory has an abnormal interruption, and the candidate seed is recorded as an abnormal trajectory, and the tracking failure counter of the candidate seed number is incremented by 1.

9. The method for counting holes according to claim 6, characterized in that: In step E4, for a valid seed target that is in a "tracking doubt state" for J consecutive frames, if the valid seed target reappears in subsequent frames and meets the determination condition of step E3, i.e., the valid seed target is a continuous valid seed target, then the valid seed target is restored to the "tracking state"; if the number of frames in which the valid seed target fails to track exceeds the preset number of frames G, then it is determined that the valid seed target has landed in the hole or left the field of view, the tracking of the candidate seed is terminated and the feature point set is removed.

10. The hole counting method according to claim 6, characterized in that: In step E2, the center point of the predicted bounding box of each valid seed target retained after thresholding in the first frame image is first used as the initial feature point set. Local image gradient information around the target is extracted, and a feature point-target mapping relationship is established with each target identifier. Then, starting from the second frame, for all valid seed targets in the initial feature point set in the current and previous frames, optical flow is used to calculate the effective seed targets with the same target identifier number between the two frames from the points in the previous frame. Point in the current frame The velocity vector change is used to obtain the motion direction and velocity estimate of each effective seed target in the feature point set in the pixel space, and then the displacement vector of the seed in the next frame is calculated to obtain the predicted position of the next frame. This predicted position is the continuation point of the seed's motion trajectory in the current frame.

Images