Cotton seedling counting method based on multi-target tracking technology

Abstract

The invention discloses a cotton seedling counting method based on multi-target tracking, and the method comprises the steps: collecting video data in the growth process of cotton seedlings, carrying out the frame extraction, generating a detection image, labeling the cotton seedlings and weeds in the image, obtaining a labeled image, and dividing the labeled image into a training set and a test set; a cotton seedling detection model is constructed based on a Center Net target detection algorithm, training is performed by using the training set input model, and identification results of cotton seedlings and weeds are output; a multi-target tracker is used for tracking a detection result, tracking targets are classified through a trust mechanism, and trusted targets are processed preferentially; and matching the tracking target and the detection target through a cascade matching strategy, and calculating the matching cost to obtain a final seedling counting result. The cotton seedling counting method effectively improves the accuracy and real-time performance of cotton seedling counting, is suitable for precise management of large-scale cotton planting, and has important practical application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a cotton seedling counting method based on multi-target tracking. BACKGROUND

[0002] Cotton is one of the most important economic crops in the world, and its planting process directly affects yield, agricultural production efficiency and rational use of resources. In China, as the world's largest cotton producer, how to improve the accuracy and efficiency of cotton planting through intelligent means has become a research hotspot. In the process of cotton planting, accurate counting of seedling quantity and distribution is one of the core links to realize precision agricultural management.

[0003] Traditional cotton seedling counting methods usually rely on manual visual inspection or simple image processing techniques. Although these methods have certain applicability in certain specific scenarios, they generally have low efficiency, and the accuracy is greatly affected by environmental interference, which cannot meet the needs of large-scale planting scenarios. Especially the manual visual inspection method consumes a lot of manpower and time, and cannot meet the requirements of efficient and real-time management of modern agriculture. The counting method based on traditional image processing is limited by the complexity and dynamic changes of the background, and cannot guarantee stability and accuracy in dynamic environments.

[0004] In recent years, the rapid development of deep learning technology, especially the progress of convolutional neural networks (CNN) and target detection algorithms, has greatly promoted the research progress of agricultural intelligence. Deep learning-driven target detection technology can realize automatic detection and counting of cotton seedlings through large-scale data set training, especially in camera motion or complex environments, and has shown superior detection performance. However, existing target detection models still face many challenges in practical applications, such as target occlusion, dynamic changes in the number of targets, and ID switching problems in the tracking process, which will affect the accuracy and stability of the final counting.

[0005] Although existing target detection methods have achieved good results in static images, in dynamic environments, especially in the growth process of cotton seedlings, due to target occlusion, high similarity and background interference, the detection results are unstable. In the application of multi-target tracking, problems such as target loss, false detection and target switching cannot be completely avoided. Traditional methods often rely on single target detection or tracking technology, which is difficult to cope with these complex environmental changes and target dynamics.

[0006] Therefore, how to count cotton seedlings in real time and accurately based on more accurate multi-target tracking technology combined with deep learning algorithm has become a difficulty and research focus in current agricultural technology research. In particular, how to solve the problems of tracking failure, target loss and mismatching in traditional methods through advanced target detection and tracking technology has important significance for improving the accuracy and stability of cotton seedling counting. SUMMARY

[0007] In view of the deficiencies of the prior art, the purpose of the present application is to provide a cotton seedling counting method based on multi-target tracking, which is used to improve the accuracy and real-time performance of cotton seedling counting under environmental changes and target dynamics.

[0008] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a cotton seedling counting method based on multi-target tracking, comprising: Step S1, collecting video data of a cotton seedling collection period, frame extracting the video data to obtain a plurality of detection images, labeling the positioning frame of cotton seedlings and weeds and the identification of cotton seedlings in each detection image to obtain a plurality of labeled images, and dividing each labeled image into a training set and a test set according to a predetermined proportion; Step S2, constructing a cotton seedling target detection model based on a CenterNet target detection algorithm, inputting each labeled image in the training set into the cotton seedling target detection model, and the cotton seedling target detection model is used to output the identification result of the cotton seedling and weed object in each labeled image as a detection target; Step S3, the tracker generates corresponding tracking targets according to each detection target and performs tracking, and in the tracking process, each tracking target is divided into trusted tracking targets and untrusted tracking targets based on a trust mechanism, and the tracker is assigned to track unmatched detection targets, including newly appeared detection targets, misdetected detection targets and data association failed detection targets; Step S4, using a cascade matching strategy to match the tracking targets and the detection targets, first matching the trusted tracking targets that are not lost, then sequentially matching the tracking targets with the unmatched detection targets according to the length of the tracking target loss time, and preferentially matching the tracking targets with short loss time; Step S5, matching the trusted tracking targets and the untrusted tracking targets that fail to match together with the unmatched detection targets, preferentially matching the trusted tracking targets, and calculating the matching cost between each tracking target and the detection target; Step S6, counting all trusted tracking targets, including the total number of failed and still working trackers, to obtain the final cotton seedling counting result.

[0009] Further, the step S1 comprises: Step S11: Collect the video data during the seedling collection period after a preset time period following cotton sowing; Step S12: Divide the video data into a detection set and a counting set in a ratio of 8:2, and perform frame extraction on the data in the detection set at a frequency of 3 frames / second to obtain multiple detection images; Step S13: Use an annotation tool to annotate the bounding boxes of cotton seedlings and weeds and the identifiers of cotton seedlings in each of the detected images to obtain each annotated image, so as to form an image dataset, and divide the image dataset into a training set and a test set in a 9:1 ratio; Step S14: Perform data augmentation processing on the labeled images in the training set sequentially by translating, flipping, rotating, scaling and cropping.

[0010] Further, in step S2, the target detection process of the cotton seedling target detection model includes: Step S21: A stacked hourglass network is used as the backbone network of the cotton seedling target detection model, and an attention mechanism SE module is integrated into all residual blocks in the backbone network. The input labeled images are downsampled by 4 times through the backbone network to obtain feature maps. Step S22: Input the feature map into the target detection branch and the identity embedding branch to predict the target bounding box and extract the target appearance features, respectively. The target detection branch contains three parallel detection heads, which are used to predict the target center point and category, regress the center point offset and the bounding box size, respectively. The identity embedding branch contains one detection head, which is used to extract appearance features to distinguish different detected targets. Step S23: The detection head used to predict the target center point and category generates a key point heatmap with C channels, where the number of channels C is equal to the number of target categories, and C=2, corresponding to the categories of cotton seedlings and weeds respectively. The target center point probability, the regression center point offset and the positioning box size are calculated based on the key point heatmap. Step S24: The detection head used to extract appearance features outputs the appearance features, and then uses cosine distance to calculate the similarity between the appearance features of the two detection targets, cotton seedlings and weeds.

[0011] Furthermore, during the feature extraction process of each labeled image by the backbone network, the image is first downsampled through multiple downsampling layers and then upsampled through multiple upsampling layers. The output of each downsampling layer is summed point by point with the output of the corresponding upsampling layer through a short-circuit connection.

[0012] Further, step S3 includes: Step S31: After an untrusted tracking target successfully matches a detected target in multiple consecutive frames, it is promoted to a trusted tracking target and participates in the final count. Step S32: If the trusted tracking target fails to match the detected target during the data association process, a certain number of frames are still retained. If the target successfully matches in subsequent frames, tracking is resumed; otherwise, it fails. Step S33: For a detected target that fails to match, a new tracker is assigned to track it and it is marked as an untrusted tracking target. If it matches successfully consecutively, it is promoted to a trusted tracking target; otherwise, it is discarded.

[0013] Furthermore, in step S3, the Kalman filter algorithm predicts the prior trajectory state matrix and covariance matrix in the next frame based on the trajectory state and covariance matrix of the tracked target in the current image frame. Then, it associates the prior trajectory state with the detected target in the next frame. After successful association, it dynamically updates the trajectory state and covariance matrix of the tracked target in the next frame using the observed values.

[0014] Furthermore, the prediction process for the prior trajectory state matrix and covariance matrix in the next frame is configured as follows:

[0015]

[0016] in, and Let K and K represent the trajectory state and covariance matrix of the tracked target k in the i-th frame, respectively. Represents the state transition matrix. Represents the covariance matrix in process noise. and Let represent the prior trajectory state matrix and covariance matrix in the (i+1)th frame of the image; The trajectory state and covariance matrix update process for the tracked target in the next frame is configured as follows:

[0017] in, This represents the observed value. This represents the transformation matrix from the state space to the observation space. This represents the transpose of the transformation matrix. Indicates Kalman gain, The covariance matrix represents the observation error. This represents the prior observed value.

[0018] Further, step S4 includes: Step S41: In the cascaded matching strategy, trusted tracking targets that have not been lost are matched first, and the track status and the target appearance features are updated for successfully matched tracking targets. Step S42: When the number of consecutive tracking frames of an untrusted tracking target exceeds a preset threshold, the untrusted tracking target is promoted to a trusted tracking target. Step S43: Among the tracking targets that failed to match, untrusted tracking targets are directly discarded. When the loss time of a trusted tracking target exceeds a preset maximum time threshold, the corresponding tracker fails; otherwise, it is retained. In step S44, for the detected targets that fail to match, a new tracker is assigned to track them and they are marked as untrusted tracking targets.

[0019] Further, in step S5, the Mahalanobis distance between the target localization boxes of the trusted tracking target and the detected target is calculated using a similarity metric, and the cosine distance between the target appearance features of the trusted tracking target and the detected target is calculated, and then the matching cost is obtained by weighting the Mahalanobis distance and the cosine distance. When the Mahalanobis distance or the cosine distance is greater than a preset distance threshold, the trusted tracking target is considered to be mismatched with the detected target, and the KM algorithm is used to perform optimal matching on the trusted tracking target.

[0020] Furthermore, the calculation process for the matching cost is configured as follows:

[0021]

[0022]

[0023] in, This represents the Mahalanobis distance. Represents the cosine distance, Indicates the target of detection The state vector, Indicates the target being tracked The state vector, Indicates the target being tracked The covariance matrix, Indicates a trusted tracking target The saved number One appearance feature vector, This represents the matching cost. The last time a trusted tracking target is saved. One appearance feature vector, Indicates the target of detection The appearance feature vector, This represents the preset weight hyperparameters. This represents the threshold value of the Mahalanobis distance. The threshold value represents the cosine distance.

[0024] The beneficial effects of this invention are: This invention, based on multi-target tracking technology and combined with deep learning-based target detection algorithms, provides an efficient and accurate method for counting cotton seedlings, offering the following advantages: (1) Improve counting accuracy: This invention uses a deep learning model based on the CenterNet target detection algorithm to accurately identify cotton seedlings and weeds, and effectively reduce the impact of environmental interference on the counting results, thus ensuring the accuracy of the counting.

[0025] (2) Strong adaptability to dynamic environment: The present invention adopts a multi-target tracking strategy, which can effectively track the changes of seedlings in dynamic environment, solves the counting problem of traditional methods when the target is occluded and the number changes, and ensures the stability of the counting process.

[0026] (3) Reduced human intervention: Through automated detection and tracking, the need for human intervention and visual inspection is reduced, which significantly improves work efficiency, reduces labor costs, and meets the needs of modern agriculture for efficient and real-time management.

[0027] (4) Applicable to large-scale applications: The counting method of the present invention can be applied to large-scale cotton planting scenarios, and can quickly and accurately count seedlings, providing data support for precision agriculture and improving the level of intelligent agricultural production management.

[0028] (5) Reliable tracking and matching mechanism: The system adopts a trust mechanism and cascading matching strategy, prioritizes matching trusted tracking targets, and optimizes tracking accuracy through matching cost calculation, effectively solving problems such as target loss and false detection, and improving the robustness of the overall system. Attached Figure Description

[0029] Figure 1 This is a flowchart of the steps in the cotton seedling counting method based on multi-target tracking in this invention; Figure 2 This is a flowchart of step S1 in this invention; Figure 3 This is a flowchart of step S2 in this invention; Figure 4 This is a flowchart of step S3 in this invention; Figure 5 This is a flowchart of step S4 in this invention; Figure 6This is a visualization of the detection results of the test samples in the self-built dataset of this invention; Figure 7 This is a diagram illustrating the detection process of tracking the target from frame 138 to frame 142 in this invention; Figure 8 This is a diagram illustrating the detection process of tracking the target from frame 60 to frame 63 in this invention. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Identical components are denoted by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "upper," and "lower" used in the following description refer to directions in the accompanying drawings, and the terms "bottom surface," "top surface," "inner," and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0031] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a cotton seedling counting method based on multi-target tracking, which can improve the accuracy and real-time performance of cotton seedling counting under environmental changes and target dynamic conditions, including: Step S1: Collect video data of cotton seedling collection period, extract frames from video data to obtain several detection images, and label each detection image with the positioning boxes of cotton seedlings and weeds and the identification of cotton seedlings to obtain multiple labeled images. Divide each labeled image into training set and test set according to a preset ratio. Step S2: Construct a cotton seedling target detection model based on the CenterNet target detection algorithm. Input each labeled image in the training set into the cotton seedling target detection model. The cotton seedling target detection model is used to output the recognition results of cotton seedlings and weeds in each labeled image as detection targets. Step S3: The tracker generates corresponding tracking targets based on each detection target and tracks them. During the tracking process, the tracking targets are divided into trusted tracking targets and untrusted tracking targets based on a trust mechanism. The tracker is assigned to track unmatched detection targets. Unmatched detection targets include newly appearing detection targets, falsely detected detection targets, and detection targets with failed data association. Step S4: Use a cascaded matching strategy to match the tracking target and the detection target. First, match trusted tracking targets that have not been lost. Then, match the tracking targets with unmatched detection targets according to the length of time the tracking targets have been lost, and prioritize matching tracking targets with shorter loss times. Step S5: Match the unmatched trusted tracking targets and untrusted tracking targets together with the unmatched detection targets, prioritizing the matching of trusted tracking targets, and calculate the matching cost between each tracking target and the detection target. Step S6: Count all trusted tracking targets, including both failed and still working trackers, to obtain the final cotton seedling count result.

[0032] Working principle of Example 1: Approximately three weeks after cotton sowing, the seedling collection period began. Researchers used handheld cameras to walk at a constant speed along the cotton planting rows, capturing raw video data. The video was captured at a frame rate of 3 frames per second. The collected raw video data was then processed by frame extraction to obtain multiple detection images. Cotton seedlings and weeds in each detection image were manually labeled, generating a labeled image dataset. This dataset was divided into training and testing sets in a 9:1 ratio for training and evaluating the object detection model.

[0033] A CenterNet-based object detection model was used to identify cotton seedlings. Stacked Hourglass Networks were employed as the backbone for feature extraction. Each input frame was downsampled by a factor of 4 through the backbone to obtain a feature map. This feature map was then fed into the object detection branch and the identity embedding branch, which predicted the target's bounding box and extracted its appearance features, respectively. The feature map passed through multiple downsampling and upsampling layers, ultimately outputting the desired detection result. To improve performance, an SE module was incorporated into all residual blocks to enhance the focus on key information.

[0034] After the target in each frame of the image is processed by the detection model, the bounding boxes and appearance features of cotton seedlings and weeds are output. To track the targets, this invention combines Kalman filtering with a multi-target tracking algorithm. In each frame, the Kalman filter predicts the position information of the target in the current frame based on the target state of the previous frame (i.e., predicts the prior state of the target). For detected cotton seedling targets, the predicted bounding box of the target in the next frame is obtained through Kalman filtering, and multi-target tracking is then performed.

[0035] During the tracking process, this invention classifies targets into "trusted tracking targets" and "untrusted tracking targets" through a trust mechanism. Trusted targets are preferentially matched with the detected target. For targets that fail to match, an IoU matching strategy is adopted, prioritizing matching with trusted tracking targets. During the IoU matching process, if the target's matching degree is lower than a set threshold, the match is considered a failure, and a new tracker is reassigned for tracking.

[0036] The Kalman filter algorithm predicts the target's state (including position, velocity, etc.) in the next frame using the target's state and covariance matrix from the previous frame. For successfully matched targets, the Kalman filter updates the target's state based on the actual observations and performs state corrections. For unmatched targets, if their loss time exceeds the maximum time threshold, the corresponding tracker is disabled; otherwise, it is retained, and the loss time counter is incremented by 1.

[0037] During the matching process, a cascading matching strategy is employed: first, trusted tracking targets that have not been lost are matched preferentially, followed by targets with shorter loss times. For detected targets that cannot be matched, a new tracker is assigned to track them, and they are marked as untrusted tracking targets. If an untrusted target successfully matches for several consecutive frames, it is promoted to a trusted tracking target. If an untrusted target fails to match within a set number of frames, it is discarded.

[0038] To ensure target matching accuracy, this invention utilizes appearance features for secondary verification. The similarity between the detected and tracked targets is evaluated by calculating the Mahalanobis distance and cosine distance. If the similarity exceeds a set threshold, a match is considered successful. Otherwise, the matching cost is set to infinity, and the matching result is further optimized using an optimal matching algorithm (such as the KM algorithm). The appearance features of each target are continuously updated in each frame, ensuring that the appearance features dynamically adjust as the target's position changes.

[0039] After all video frames have been processed, all trusted tracking targets will be counted, including those that have failed and those that are still active. The final result is the count of cotton seedlings.

[0040] Reference Figure 6 The detection results of the cotton seedling target detection model in this invention were visualized on the test samples of the self-built dataset. In the figure, the red box represents weeds and the green box represents cotton seedlings. It can be seen from the figure that the cotton seedling target detection model in this invention can accurately detect cotton seedlings and distinguish weeds well, which significantly improves the recognition accuracy.

[0041] Reference Figure 7 In the figure, the solid box represents the posterior localization box of the trusted tracking target. The trusted tracking target 103 in frame 138 was missed by the target detection model from frame 139 to frame 141. At this time, the trusted tracking target was not discarded, but retained and continued to participate in the data association process of subsequent frames. In frame 142, the target was detected again and successfully matched with the lost trusted tracking target 103, thereby resuming the tracking of the target and avoiding ID switching errors.

[0042] Reference Figure 8In the figure, the solid box represents the posterior localization box of the trusted tracking target, and the dashed box represents the posterior localization box of the untrusted tracking target. The trusted tracking target 25 moved to the edge of the video screen in the 62nd frame, and only part of it was visible. Although the appearance model failed, it could still match the correct detection target through IoU matching, thus maintaining correct tracking of the target and avoiding ID switching errors.

[0043] This invention employs a combination of Kalman filtering and target detection models to accurately track cotton seedlings in dynamic environments. Through trust mechanisms, cascaded matching strategies, and appearance feature matching, it avoids ID switching errors and target loss. Experimental results demonstrate that this invention exhibits high accuracy and stability in variable environments, meeting the precise counting requirements of large-scale cotton cultivation.

[0044] Preferred, refer to Figure 2 Step S1 includes: Step S11: Collect video data during the seedling collection period one preset time period after cotton sowing; Step S12: Divide the video data into a detection set and a counting set in an 8:2 ratio, and perform frame extraction on the data in the detection set at a frequency of 3 frames / second to obtain multiple detection images. Step S13: Use the annotation tool to annotate the bounding boxes of cotton seedlings and weeds and the identifiers of cotton seedlings in each detection image to obtain an annotated image dataset, and divide the image dataset into training set and test set in a 9:1 ratio. Step S14: Perform data augmentation processing on the labeled images in the training set by sequentially translating, flipping, rotating, scaling and cropping.

[0045] Specifically, in this embodiment, step S11 is used to collect video data: During a predetermined time period after cotton sowing (e.g., approximately three weeks after sowing), the seedling collection period begins. At this time, the cotton seedlings have largely emerged, making them easy to observe and track. Researchers use a handheld camera to photograph the area where the cotton rows are planted, ensuring that each frame contains clear images of cotton seedlings and weeds. The frame rate of the video data is set to 3 frames per second, and the video recording time is set according to actual needs to cover the key stages of cotton seedling growth.

[0046] Step S12 involves data partitioning and frame extraction: the acquired raw video data is divided into a detection set and a count set in an 8:2 ratio. The detection set is used to train the object detection model, and the count set is used for the final cotton seedling count. To improve processing efficiency, three frames are extracted from each second of the video data, resulting in multiple detection images. This ensures that each detection image covers a continuous time window while maintaining sufficient data diversity to enhance the robustness of the object detection model.

[0047] Step S13: Image Annotation. For each detected image obtained from frame extraction, manual annotation is performed using a specialized annotation tool. During the annotation process, bounding boxes are marked for cotton seedlings and weeds in each image to ensure that the targets are correctly identified in the image. Each cotton seedling also needs to be uniquely identified to facilitate subsequent target tracking. In this way, an annotated image dataset is obtained.

[0048] The labeled image dataset will be divided into training and testing sets in a 9:1 ratio. The training set will be used for model training, and the testing set will be used for model evaluation and validation. The accuracy of the annotation directly affects the performance of object detection and tracking; therefore, it is crucial to ensure that each object (cotton seedlings and weeds) is accurately identified during the annotation process.

[0049] Step S14 involves data augmentation: To enhance the generalization ability of the object detection model and avoid overfitting, the labeled images in the training set will be processed using data augmentation techniques. Specific augmentation operations include: Translation: Translate the image horizontally and vertically.

[0050] Flip: Flip the image horizontally or vertically.

[0051] Rotation: Randomly rotating an image within a certain angle range.

[0052] Scaling: Increases the diversity of data by changing the size of the image.

[0053] Cropping: Randomly crop a portion of the image to simulate different shooting angles and distances.

[0054] These data augmentation methods increase the number of images in the training set and improve the robustness of the model in different scenarios, enabling the object detection model to better adapt to various changes in real-world applications.

[0055] After the above steps, an enhanced dataset was finally obtained, including training and testing sets. This dataset not only contains bounding boxes and labeling information for cotton seedlings and weeds, but also includes rich image variations, enabling the object detection model to be effectively trained and detected in various complex environments.

[0056] Preferred, refer to Figure 3 In step S2, the target detection process of the cotton seedling target detection model includes: Step S21: A stacked hourglass network is used as the backbone network of the cotton seedling target detection model, and an attention mechanism SE module is integrated into all residual blocks in the backbone network. The input labeled images are downsampled by 4 times through the feature extraction backbone network to obtain feature maps. Step S22: Input the feature map into the target detection branch and the identity embedding branch to predict the target bounding box and extract the target appearance features, respectively. The target detection branch contains three parallel detection heads, which are used to predict the target center point and category, regress the center point offset and the bounding box size, respectively. The identity embedding branch contains one detection head, which is used to extract appearance features to distinguish different detected targets. Step S23: The detection head used to predict the target center point and category generates a key point heatmap with C channels. The number of channels C is equal to the number of target categories, where C=2, corresponding to the categories of cotton seedlings and weeds respectively. The target center point probability, regression center point offset and positioning box size are calculated based on the key point heatmap. Step S24: The detection head used to extract appearance features outputs appearance features, and then the cosine distance is used to calculate the similarity between the appearance features of the two detection targets, cotton seedlings and weeds.

[0057] This invention provides a deep learning-based method for cotton seedling target detection, employing a stacked hourglass network as the backbone network and combining it with an attention mechanism (SE module) for accurate detection and tracking of cotton seedlings. The specific implementation process is as follows: Step S2: Target detection process of the cotton seedling target detection model; Step S21: Backbone Network and Feature Extraction: In this embodiment, Stacked Hourglass Networks are used as the backbone network of the cotton seedling target detection model. This network gradually extracts spatial features of the image through multiple downsampling and upsampling processes. In the residual blocks of each layer, an attention mechanism (SE module, Squeeze-and-Excitation) is incorporated to improve the network's focus on key information. The SE module processes the input feature map U... Each channel undergoes global average pooling to obtain a feature vector of length C. This feature vector is then input into a fully connected layer, and the output vector has a length of C. After processing with the ReLU activation function, the input is fed into a fully connected layer, restoring the feature vector length to C. Then, the Sigmoid activation function is used, so the values ​​in the output vector are all between 0 and 1, representing the weights of the corresponding channels.

[0058] The input labeled image is processed through a stacked hourglass network for feature extraction, and the feature map is obtained by downsampling by four times. During the downsampling process, the spatial resolution of the image gradually decreases, but the number of channels increases, allowing the network to capture more contextual information. In this way, the network can effectively extract the spatial features of cotton seedlings.

[0059] Step S22: Object Detection Branch and Identity Embedding Branch: The extracted feature maps are then fed into two parallel branches: Object detection branch: This branch is used to predict the target bounding box and category information of cotton seedlings. The object detection branch contains three parallel detection heads, each consisting of a 3×3 convolutional block with 256 channels and a 1×1 convolutional layer. The feature map is processed by the detection heads to output a result with the desired number of channels and unchanged size.

[0060] The first detection head is used to predict the target center point and its category, the second detection head is used to regress the offset of the target center point, and the third detection head is used to regress the target's bounding box size.

[0061] Identity Embedding Branch: This branch extracts the appearance features of the target to distinguish different targets, especially cotton seedlings from weeds. The feature vector output by the identity embedding branch is used for subsequent target matching and tracking.

[0062] Step S23: Keypoint Heatmap and Target Localization: The first detection head in the target detection branch generates a keypoint heatmap, with the number of channels C equal to the number of target categories (in this embodiment, C=2, corresponding to cotton seedlings and weeds respectively). Each heatmap channel represents the probability of the center point of the target category. Based on this heatmap, the center point position of each target, the regressed center point offset, and the size of the localization box can be obtained. By calculating the confidence level of each point in the keypoint heatmap, the specific locations of all cotton seedlings and weeds in the image can be identified, and the corresponding localization box sizes can be regressed.

[0063] Step S24: Appearance Feature Extraction and Similarity Calculation: The appearance feature vectors output from the identity embedding branch are used to distinguish different targets, particularly cotton seedlings and weeds. In this process, cosine distance is used to measure the similarity between the appearance features of cotton seedlings and weeds.

[0064] Specifically, the formula for calculating the cosine distance is:

[0065] Represents the cosine distance. The last time a trusted tracking target is saved. One appearance feature vector, This represents the appearance feature vector of the detected target. Indicates a trusted tracking target The saved number A set of appearance feature vectors. By calculating similarity, cotton seedlings and weeds can be effectively distinguished, providing a reliable basis for subsequent tracking and counting.

[0066] Preferably, during the feature extraction process of each labeled image by the backbone network, the image is first downsampled through multiple downsampling layers and then upsampled through multiple upsampling layers. The output of each downsampling layer is summed point by point with the output of the corresponding upsampling layer through a short-circuit connection.

[0067] Specifically, in this embodiment, this approach helps the network better preserve image details while enhancing its expressive power, enabling accurate localization and classification during target detection of cotton seedlings and weeds. After downsampling at each layer, the feature map size decreases, but the number of channels increases, allowing global information to be captured at a lower resolution. During upsampling, low-resolution features are restored to a higher resolution through point-by-point summation, thereby enhancing the model's ability to capture details. This structure enables the network to effectively identify and distinguish different targets, especially the subtle differences between cotton seedlings and weeds.

[0068] Preferred, refer to Figure 4 Step S3 includes: Step S31: After an untrusted tracking target successfully matches a detected target in multiple consecutive frames, it is promoted to a trusted tracking target and participates in the final count. Step S32: If the trusted tracking target fails to match the detected target during the data association process, a certain number of frames are still retained. If the target successfully matches in subsequent frames, tracking is resumed; otherwise, it fails. Step S33: For a detected target that fails to match, assign a new tracker to track it and mark it as an untrusted tracking target. If it matches successfully consecutively, it will be promoted to a trusted tracking target; otherwise, it will be discarded.

[0069] Preferably, in step S3, the Kalman filter algorithm predicts the prior trajectory state matrix and covariance matrix in the next frame based on the trajectory state and covariance matrix of the tracked target in the current image frame, and then associates the prior trajectory state with the detected target in the next frame. After successful association, the trajectory state and covariance matrix of the tracked target in the next frame are dynamically updated through the observation values.

[0070] In video data, each tracked target corresponds to a trajectory state. It contains the target's location and velocity information:

[0071] in, The coordinates of the center point of the target location box. The aspect ratio of the positioning frame. The height of the positioning frame, and For their respective speeds.

[0072] Preferably, the prediction process for the prior trajectory state matrix and covariance matrix in the next frame is configured as follows:

[0073]

[0074] in, and Let K and K represent the trajectory state and covariance matrix of the tracked target k in the i-th frame, respectively. Represents the state transition matrix. Represents the covariance matrix in process noise. and Let represent the prior trajectory state matrix and covariance matrix in the (i+1)th frame of the image; After obtaining the prior trajectory state and covariance matrix of the tracked target in frame i+1, the trajectory state is correlated with the detected target in frame i+1. For successfully correlated targets, the observed values ​​are used to... The trajectory state is updated to obtain the trajectory state and covariance matrix of the target in frame i+1. The update process is as follows: The trajectory state and covariance matrix update process for the tracked target in the next frame is configured as follows:

[0075] in, Represents the observed value. This represents the transformation matrix from the state space to the observation space. This represents the transpose of the transformation matrix. Indicates Kalman gain, The covariance matrix represents the observation error. This represents the prior observation.

[0076] Preferred, refer to Figure 5 Step S4 includes: Step S41: In the cascaded matching strategy, trusted tracking targets that have not been lost are matched first, and the tracking targets that are successfully matched are updated with trajectory status and target appearance features. Step S42: When the number of consecutive tracking frames of an untrusted tracking target exceeds a preset threshold, the untrusted tracking target is promoted to a trusted tracking target. Step S43: Among the tracking targets that failed to match, untrusted tracking targets are directly discarded. When the loss time of a trusted tracking target exceeds the preset maximum time threshold max_age, the corresponding tracker is invalidated; otherwise, it is retained. In step S44, for the detected targets that fail to match, a new tracker is assigned to track them and they are marked as untrusted tracking targets.

[0077] Preferably, in step S5, the Mahalanobis distance between the target localization boxes of the trusted tracking target and the detected target is calculated using a similarity metric, and the cosine distance between the target appearance features of the trusted tracking target and the detected target is calculated, and then the matching cost is obtained by weighting the Mahalanobis distance and the cosine distance. When the Mahalanobis distance or cosine distance is greater than the preset distance threshold, the trusted tracking target is considered to be mismatched with the detected target, and the KM algorithm is used to perform optimal matching on the trusted tracking target.

[0078] Preferably, the matching cost calculation process is configured as follows:

[0079]

[0080] in, This represents the Mahalanobis distance. Indicates the target of detection The state vector, Indicates the target being tracked The state vector, Indicates the target being tracked The covariance matrix, Indicates a trusted tracking target The saved number One appearance feature vector, This represents the matching cost. The last time a trusted tracking target is saved. One appearance feature vector, Indicates the target of detection The appearance feature vector, This represents the preset weight hyperparameters. This represents the threshold value of the Mahalanobis distance. The threshold value represents the cosine distance.

[0081] Specifically, in this embodiment, when the Mahalanobis distance or cosine distance is greater than a set threshold, the trusted tracking target is considered to be mismatched with the detected target, and therefore the matching cost is set to infinity. The KM algorithm is used for optimal matching. If the missing time (missing_age) of a trusted tracking target exceeds the maximum time (max_age), the corresponding tracker is invalidated; otherwise, it is retained, and missing_age = missing_age + 1. For detected targets that fail to match, a new tracker is assigned to track them, and they are marked as untrusted tracking targets.

[0082] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for counting cotton seedlings based on multi-target tracking, characterized in that, include: Step S1: Collect video data of cotton seedling collection period, extract frames from the video data to obtain several detection images, and label each detection image with the positioning boxes of cotton seedlings and weeds and the identifier of cotton seedlings to obtain multiple labeled images. Divide each labeled image into training set and test set according to a preset ratio. Step S2: Construct a cotton seedling target detection model based on the CenterNet target detection algorithm. Input each of the labeled images in the training set into the cotton seedling target detection model. The cotton seedling target detection model is used to output the recognition results of cotton seedlings and weeds in each of the labeled images as detection targets. Step S3: The tracker generates corresponding tracking targets based on each detection target and tracks them. During the tracking process, the tracking targets are divided into trusted tracking targets and untrusted tracking targets based on a trust mechanism. The tracker is assigned to track unmatched detection targets. Unmatched detection targets include newly appearing detection targets, falsely detected detection targets, and detection targets with failed data association. Step S4: A cascaded matching strategy is used to match the tracking target and the detection target. First, trusted tracking targets that have not been lost are matched. Then, according to the length of time the tracking target has been lost, it is matched with unmatched detection targets in turn, and the tracking targets with shorter loss times are matched first. Step S5: Match the unmatched trusted tracking targets and untrusted tracking targets together with the unmatched detection targets, prioritizing the matching of trusted tracking targets, and calculate the matching cost between each tracking target and the detection target. Step S6: Count all trusted tracking targets, including both failed and still working trackers, to obtain the final cotton seedling count result.

2. The cotton seedling counting method based on multi-target tracking according to claim 1, characterized in that: Step S1 includes: Step S11: Collect the video data during the seedling collection period after a preset time period following cotton sowing; Step S12: Divide the video data into a detection set and a counting set in a ratio of 8:2, and perform frame extraction on the data in the detection set at a frequency of 3 frames / second to obtain multiple detection images; Step S13: Use an annotation tool to annotate the bounding boxes of cotton seedlings and weeds and the identifiers of cotton seedlings in each of the detected images to obtain each annotated image, so as to form an image dataset, and divide the image dataset into a training set and a test set in a 9:1 ratio; Step S14: Perform data augmentation processing on the labeled images in the training set sequentially by translating, flipping, rotating, scaling and cropping.

3. The cotton seedling counting method based on multi-target tracking according to claim 1, characterized in that: In step S2, the target detection process of the cotton seedling target detection model includes: Step S21: A stacked hourglass network is used as the backbone network of the cotton seedling target detection model, and an attention mechanism SE module is integrated into all residual blocks in the backbone network. The input labeled images are downsampled by 4 times through the backbone network to obtain feature maps. Step S22: Input the feature map into the target detection branch and the identity embedding branch to predict the target bounding box and extract the target appearance features, respectively. The target detection branch contains three parallel detection heads, which are used to predict the target center point and category, regress the center point offset and the bounding box size, respectively. The identity embedding branch contains one detection head, which is used to extract appearance features to distinguish different detected targets. Step S23: The detection head used to predict the target center point and category generates a key point heatmap with C channels, where the number of channels C is equal to the number of target categories, and C=2, corresponding to the categories of cotton seedlings and weeds respectively. The target center point probability, the regression center point offset and the positioning box size are calculated based on the key point heatmap. Step S24: The detection head used to extract appearance features outputs the appearance features, and then uses cosine distance to calculate the similarity between the appearance features of the two detection targets, cotton seedlings and weeds.

4. The cotton seedling counting method based on multi-target tracking according to claim 3, characterized in that: During the feature extraction process of the labeled images by the backbone network, the images are first downsampled through multiple downsampling layers and then upsampled through multiple upsampling layers. The outputs of each downsampling layer are summed point by point with the outputs of the corresponding upsampling layers through short-circuit connections.

5. The cotton seedling counting method based on multi-target tracking according to claim 1, characterized in that: Step S3 includes: Step S31: After an untrusted tracking target successfully matches a detected target in multiple consecutive frames, it is promoted to a trusted tracking target and participates in the final count. Step S32: If the trusted tracking target fails to match the detected target during the data association process, a certain number of frames are still retained. If the target successfully matches in subsequent frames, tracking is resumed; otherwise, it fails. Step S33: For a detected target that fails to match, a new tracker is assigned to track it and it is marked as an untrusted tracking target. If it matches successfully consecutively, it is promoted to a trusted tracking target; otherwise, it is discarded.

6. The cotton seedling counting method based on multi-target tracking according to claim 2, characterized in that: In step S3, the Kalman filter algorithm predicts the prior trajectory state matrix and covariance matrix in the next frame based on the trajectory state and covariance matrix of the tracked target in the current image frame. Then, it associates the prior trajectory state with the detected target in the next frame. After successful association, it dynamically updates the trajectory state and covariance matrix of the tracked target in the next frame using the observed values.

7. The cotton seedling counting method based on multi-target tracking according to claim 6, characterized in that: The prediction process for the prior trajectory state matrix and covariance matrix in the next frame is configured as follows: in, and Let K and K represent the trajectory state and covariance matrix of the tracked target k in the i-th frame, respectively. Represents the state transition matrix. Represents the covariance matrix in process noise. and Let represent the prior trajectory state matrix and covariance matrix in the (i+1)th frame of the image; The trajectory state and covariance matrix update process for the tracked target in the next frame is configured as follows: in, This represents the observed value. This represents the transformation matrix from the state space to the observation space. This represents the transpose of the transformation matrix. Indicates Kalman gain, The covariance matrix represents the observation error. This represents the prior observed value.

8. The cotton seedling counting method based on multi-target tracking according to claim 7, characterized in that: Step S4 includes: Step S41: In the cascaded matching strategy, trusted tracking targets that have not been lost are matched first, and the track status and the target appearance features are updated for successfully matched tracking targets. Step S42: When the number of consecutive tracking frames of an untrusted tracking target exceeds a preset threshold, the untrusted tracking target is promoted to a trusted tracking target. Step S43: Among the tracking targets that failed to match, untrusted tracking targets are directly discarded. When the loss time of a trusted tracking target exceeds a preset maximum time threshold, the corresponding tracker fails; otherwise, it is retained. In step S44, for the detected targets that fail to match, a new tracker is assigned to track them and they are marked as untrusted tracking targets.

9. The cotton seedling counting method based on multi-target tracking according to claim 2, characterized in that: In step S5, the Mahalanobis distance between the target localization boxes of the trusted tracking target and the detected target is calculated using a similarity metric, and the cosine distance between the target appearance features of the trusted tracking target and the detected target is calculated. Then, the matching cost is obtained by weighting the Mahalanobis distance and the cosine distance. When the Mahalanobis distance or the cosine distance is greater than a preset distance threshold, the trusted tracking target is considered to be mismatched with the detected target, and the KM algorithm is used to perform optimal matching on the trusted tracking target.

10. The cotton seedling counting method based on multi-target tracking according to claim 9, characterized in that: The process for calculating the matching cost is configured as follows: in, This represents the Mahalanobis distance. Represents the cosine distance, Indicates the target of detection The state vector, Indicates the target being tracked The state vector, Indicates the target being tracked The covariance matrix, Indicates a trusted tracking target The saved number One appearance feature vector, This represents the matching cost. The last time a trusted tracking target is saved. One appearance feature vector, Indicates the target of detection The appearance feature vector, This represents the preset weight hyperparameters. This represents the threshold value of the Mahalanobis distance. The threshold value represents the cosine distance.

Images