Apple counting method based on AD-YOLO model and MR-SORT method
By using the AD-YOLO model and MR-SORT method in fruit counting, combined with the attention mechanism and the YOLOv8 model, the missed and missed detection of apple counting in complex orchard environments is solved, and the accuracy and efficiency of counting are improved.
Patent Information
- Application Number
- CN202410202994.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-02-23
AI Technical Summary
In complex orchard environments, existing fruit counting methods are prone to problems such as missed, missed and inaccurate apple counting time, especially due to the degradation of detection performance caused by the occlusion of fruits and changes in light.
Using the Apple counting method based on the AD-YOLO model and the MR-SORT method, combined with the attention mechanism and the YOLOv8 model, the detection model is enhanced through the global attention mechanism, full-dimensional dynamic convolution and spatial pyramid pooling layer, and SIFT feature description is used. Sub- and VLAD image retrieval algorithms perform appearance feature matching to reduce error detection and missed detection.
In complex orchard environments, it significantly reduces missed detection and missed detection, improves the accuracy of apple counting, and reduces the time required for counting.
Smart Images

Figure CN118097658B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of deep learning, and in particular relates to an apple counting method based on an AD-YOLO model and an MR-SORT method. Background Art
[0002] With the rapid development of intelligent agriculture, the importance of fruit counting in agricultural production management has become increasingly prominent. At present, fruit counting methods are widely used in orchard yield prediction tasks. In addition to estimating orchard yields, fruit counting can also be used to calculate the maturity rate of fruits, thereby helping fruit farmers determine the best time to pick fruits and increase the sales price and profit of fruits. In video streams, fruit counting can be regarded as a multi-target tracking problem based on tracking by detection (TBD). However, the fruit counting method using the TBD paradigm is easily affected by occlusion and detection performance, especially for fruits that are close in color to the background and grow densely. This effect is more obvious.
[0003] Although a variety of mature multi-object tracking methods have been proposed and widely used for fruit counting, challenges still exist in complex agricultural scenes. Many TBD-based fruit counting methods associate bounding boxes with tracks frame by frame based on IoU distance, depth information, or optical flow. While this achieves good overall performance in terms of tracking precision and accuracy, the number of returned false counts of apples is relatively high. This phenomenon is caused by the fact that fruits are easily occluded in agricultural environments and the light changes significantly in complex orchard environments. Some tracking methods introduce appearance feature matching to reduce ID switching. However, many TBD methods with appearance feature matching may not be directly applicable to the field of fruit counting. For example, DeepSORT has shown excellent performance in the field of pedestrian tracking. However, in addition to the dataset for training the object detector, DeepSORT also requires a CNN trained on a large-scale re-identification (Re-ID) dataset, which leads to high data collection and annotation costs. Unlike tasks in the security field such as pedestrian tracking, there are few public datasets for fruit Re-ID. To address this problem, the local aggregated descriptor vector image retrieval method is used to replace CNN to calculate the appearance similarity of two objects. However, this method is still limited by the performance of the detector and the basic tracker. Summary of the invention
[0004] In view of the technical problems of missed detection, false detection, long counting time and inaccurate apple counting in the above-mentioned prior art in a complex orchard environment, the present invention provides an apple counting method based on the AD-YOLO model and the MR-SORT method, which reduces the false detection and missed detection in a complex orchard environment by combining advanced structures such as the attention mechanism and the YOLOv8 model.
[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0006] The apple counting method based on the AD-YOLO model and the MR-SORT method includes the following steps:
[0007] S1. Obtain image data for training and video data for counting test, preprocess the image data to obtain preprocessed image data, and construct a data set;
[0008] S2. Build the apple detection model AD-YOLO based on YOLOv8, GAM, ODConv and SimCSPSPPF;
[0009] S3. Use the training set to train on the constructed AD-YOLO network, select the best model as the final AD-YOLO model, and compare the results with other SOTA models;
[0010] S4. Based on BoT-SORT, a multi-target tracking method MR-SORT is proposed to build an apple counting channel;
[0011] S5. Input the test video data into the apple counting channel, obtain the results, and establish evaluation indicators for the performance evaluation of the counting method, and compare the results with other SOTA models.
[0012] The data set in S1 includes a detection data set and a tracking data set;
[0013] The tracking data set is as follows: The tracking data comes from the Apple tracking public database. This data consists of 7 videos with a resolution of 1920×1080 and a frame rate of 30fps. Each video is 1 minute. This data set has corresponding annotation information in the format of [frame, ID, x, y, w, h, Cf, C, v], where: frame is the frame number; ID is the unique identifier of the apple in the entire video sequence; x, y, w, h are the horizontal and vertical coordinates, width, and height of the left vertex of the bounding box respectively; Cf is the confidence; C is the class identifier; v is the visibility. Since there is only one class, the class identifiers in the database are all 1, and the confidence and visibility are set to 1;
[0014] The detection dataset is: the detection dataset comes from two aspects: on the one hand, three public databases, and on the other hand, part of the data in the above-mentioned tracking dataset. The three public databases containing apple RGB images are the Minnesota Apple dataset, the Washington State University dataset, and the Fuji dataset. The three public databases have corresponding annotation information. Part of the data in the tracking dataset is 960 images captured from 7 tracking data videos at a certain frequency.
[0015] The preprocessing method in S1 is: annotation information format conversion, data enhancement and data partitioning, and the data enhancement includes horizontal rotation, vertical rotation, brightness change, contrast change, saturation change, Gaussian noise, salt and pepper noise, size change and motion blur effect.
[0016] The specific construction method of the apple detection model AD-YOLO in S2 is:
[0017] Replace the ordinary convolution in the CBS unit in the backbone network of the YOLOv8 model with the full-dimensional dynamic convolution ODConv; add the global attention mechanism GAM before the SPPF unit in the backbone network of the YOLOv8 model; replace the SPPF unit in the backbone network of the YOLOv8 model with the more advanced spatial pyramid pooling layer SimCSPSPPF.
[0018] The training method in S3 is as follows: initialize the trainable parameters in the network and input the training set into the network in batches; construct a loss function and calculate the loss based on the predicted value and the true label, and use the optimization algorithm to back-propagate and update the network parameters until the loss no longer decreases within a certain range, at which time the network parameters are saved as the final model; then use the same data set and the same training strategy to train other SOTA models, including Faster R-CNN, YOLOv5s, YOLOv7-tiny, and YOLOv8s, and compare the parameter results of these models.
[0019] The multi-target tracking method MR-SORT in S4 is as follows: the ResNeSt50 network used in BoT-SORT to extract appearance features is replaced by SIFT feature descriptor extraction, and then the VLAD image retrieval algorithm is used for appearance feature matching, and the output of the target detection module is used as the input of the multi-target tracking module, and the two are combined into an apple counting channel.
[0020] The evaluation indicators in S5 include the number of ID switching, MOTA multi-target tracking accuracy, and MAE average accuracy error. The formula is as follows:
[0021]
[0022]
[0023] In the above formula, for the tth frame, m t is the number of missed detections, i.e. FN, false negative; fp t is the number of false positives, i.e. FP; mme trepresents the number of ID switches, gt is the true value; MAE is the average precision error of all test videos, m is the number of videos, y is the predicted number of apples in the video, and gt is the actual number of apples in the video.
[0024] Other SOTA models in S5 include: YOLOv8s and DeepSORT, YOLOv8s and Cascade-SORT, YOLOv8s and BoT-SORT.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] The present invention reduces false detection and missed detection in complex orchard environments by combining advanced structures such as the attention mechanism with the YOLOv8 model; and improves the accuracy of apple counting by combining a non-deep learning image appearance feature matching method with the BoT-SORT method, while also reducing the time required for counting. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0028] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.
[0029] Figure 1 The figure is a flow chart of the apple counting method based on the AD-YOLO model and the MR-SORT method of the present invention.
[0030] Figure 2 This is a schematic diagram of the AD-YOLO network model constructed by the present invention.
[0031] Figure 3 This is a structural diagram of the MR-SORT tracking method constructed by the present invention.
[0032] Figure 4 These are the counting results of 7 videos based on the AD-YOLO and MR-SORT counting methods of the present invention. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0034] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0035] like Figure 1-4 As shown, this embodiment is implemented under the pytorch deep learning framework. This embodiment proposes an apple counting method based on the AD-YOLO model and the MR-SORT method. By combining advanced structures such as the attention mechanism and the YOLOv8 model, apple detection in a complex orchard environment is achieved; by combining a non-deep learning image appearance feature matching method and the BoT-SORT method, apple counting in orchard videos is achieved.
[0036] The overall process of this embodiment is as follows Figure 1 As shown, the specific implementation steps are as follows:
[0037] Step 1: Data preparation
[0038] The data sets in this embodiment include a detection data set and a tracking data set, which specifically include:
[0039] Tracking data set: The tracking data comes from the Apple Tracking Public Database (shot in 2021). This data consists of 7 videos with a resolution of 1920×1080, a frame rate of 30fps, and each video is 1 minute. This dataset has corresponding annotation information in the format: [frame, ID, x, y, w, h, Cf, C, v], where: frame is the frame number; ID is the unique identifier of the apple in the entire video sequence; x, y, w, h are the horizontal and vertical coordinates, width, and height of the left vertex of the bounding box, respectively; Cf is the confidence; C is the class identifier; v is the visibility. Because there is only one class, the class identifiers in the database are all 1. In addition, considering that the data represents the true value, the confidence and visibility are set to 1.
[0040] Detection dataset: The detection dataset comes from two aspects: on the one hand, three public databases, and on the other hand, part of the data in the above tracking dataset. The three public databases containing apple RGB images are the Minnesota Apple dataset, the Washington State University dataset, and the Fuji dataset. The three public databases have corresponding annotation information. Part of the data in the tracking dataset is 960 images cut out from 7 tracking data videos at a certain frequency.
[0041] The above datasets were preprocessed to convert the different annotation information of the detection datasets into the format data required for training. Then the data were rotated horizontally, vertically, and subjected to brightness, contrast, saturation, Gaussian noise, salt and pepper noise, size change, and motion blur effects to enhance the robustness of the data. The datasets were then divided into training, validation, and test sets in a ratio of 8:1:1 for subsequent model training.
[0042] Step 2: Building an Apple Detection Model
[0043] The constructed AD-YOLO model is based on the convolutional neural network YOLOv8s. The specific network model structure is as follows Figure 2 As shown. This embodiment uses lightweight YOLOv8s. YOLOv8s is a lightweight parameter structure derived from the YOLOv8 algorithm. It includes a backbone network, a neck network, and a prediction output head. The backbone network uses convolution operations to extract features of different scales from RGB (red, green, and blue) color images. At the same time, the role of the neck network is to fuse the features extracted by the backbone network. The feature pyramid structure (feature pyramid network, FPN) is usually used to aggregate low-level features into high-level representations. The head layer is responsible for predicting the target category and using three groups of detection detectors of different sizes to select and detect image content.
[0044] In order to enhance the feature extraction capability of the original YOLOv8 model, this embodiment replaces the ordinary convolution in the CBS unit in the YOLOv8 model backbone network with full-dimensional dynamic convolution ODConv. ODConv introduces a multi-dimensional attention mechanism through a parallel strategy to learn more flexible attention to the four dimensions of the convolution kernel space. The ODConv (full-dimensional dynamic convolution) network has three more dimensions than the traditional convolution network, namely the input channel, output channel, and spatial domain dimension.
[0045] In the backbone network of the YOLOv8 model, a global attention mechanism GAM is added before the SPPF unit. The GAM attention mechanism is an attention mechanism similar to CBAM and SAM. It also uses the channel attention mechanism and the spatial attention mechanism.
[0046] The SPPF unit in the backbone network of the YOLOv8 model is replaced with a more advanced spatial pyramid pooling layer SimCSPSPPF. The SimCSPSPPF module is a combination of SPPCSPC and SPPF. Compared with the SPPF module, it improves accuracy while keeping the speed almost unchanged.
[0047] Step 3: Detection model training and performance evaluation
[0048] Initialize the trainable parameters in the network and input the training set into the network in batches; construct the loss function and calculate the loss based on the predicted value and the true label, and use the optimization algorithm to back-propagate and update the network parameters until the loss no longer decreases within a certain range. At this time, save the network parameters as the final model.
[0049] In order to verify the optimization effect of the three newly added modules on the model, this embodiment conducted an ablation experiment, and the results are shown in Table 1.
[0050] Table 1 Detection model ablation experiment results
[0051]
[0052] As can be seen from Table 1, after the introduction of the global attention mechanism GAM, the mAP value of the YOLOv8s model increased by 1.8%, the precision and recall rates increased by 1.0%, and the number of parameters increased by only 6.67M; in Model 3, after adding the ODConv method in the feature extraction step, the accuracy rate only increased by 0.1%, while the recall rate increased by 0.8%, thereby increasing the mAP value by 0.4%; after adding GAM, ODConv and SimCSPSPPF to the trunk network of Model 4, the mAP increased to 95.8% and the accuracy increased to 96.9%. In summary, after adding GAM or SimCSPSPPF to YOLOv8, the detection performance is significantly improved; when only ODConv is added, the mAP value does not change much after the convergence stage. This embodiment names Model 4 as AD-YOLO, which has significantly improved network detection performance compared with the YOLOv8s model, laying a better foundation for realizing apple tracking and counting behaviors.
[0053] Then, we use the same dataset and the same training strategy to train other SOTA models (Faster R-CNN, YOLOv5s, YOLOv7-tiny, and YOLOv8s), and compare the parameter results of these models. The results are shown in Table 2.
[0054] Table 2 Performance comparison of AD-YOLO model with other SOTA models
[0055]
[0056] It can be seen from Table 2 that the AD-YOLO model proposed in this embodiment enhances the ability to correctly detect apples in a complex orchard environment while retaining a certain degree of real-time performance, laying a foundation for apple tracking and counting.
[0057] Step 4: Build an apple counting channel
[0058] The basic tracking algorithm used in this embodiment is BoT-SORT. The BoT-SORT algorithm is a multi-target tracking algorithm that combines motion and appearance information, camera motion compensation, and a more accurate Kalman filter state vector. These improvements are integrated into ByteTrack to enhance the robustness of target tracking and are more suitable for scenes with camera motion.
[0059] In pedestrian tracking, CNN is used to extract the appearance features of objects and re-identify the objects, which requires large-scale classification data sets for training. Unlike pedestrian tracking, for the same fruit, the frame-by-frame changes in its appearance features are negligible. Traditional image retrieval methods can also replace CNN to achieve higher re-identification accuracy. This embodiment regards object re-identification as an image retrieval problem and uses the local aggregate descriptor vector (VLAD) method to measure the similarity between different objects.
[0060] VLAD is a simple and efficient image retrieval algorithm used before the rise of deep learning. First, VLAD calculates the descriptors of the SIFT feature points in two objects. The SIFT feature descriptor is a 128-dimensional vector. Then the cluster center of the two image descriptors is calculated through the K-means clustering algorithm, and then the similarity of the two images is calculated. The closer the similarity is to 1, the more similar the two images are.
[0061] In this embodiment, the ResNeSt50 network used in BoT-SORT to extract appearance features is replaced by the SIFT feature descriptor, and then the VLAD image retrieval algorithm is used to match the appearance features. In the first and second matching of BoT-SORT, the setting of the matching threshold is very important. If the threshold is set too high, it will lead to an increase in FP and a decrease in FN; if the threshold is set too low, it will lead to the opposite effect. In summary, this embodiment sets the threshold relatively low, and then proposes a method of filling in the gaps (re-matching) to match the unmatched tracks and detection targets. The MR-SORT method proposed in this embodiment is as follows. Figure 3 shown.
[0062] The output of the target detection module is used as the input of the multi-target tracking module, and the two are combined into an apple counting channel, whose structure is as follows: Figure 1 shown.
[0063] Step 5: Counting method performance evaluation
[0064] The counting results and the evaluation index of the tracking algorithm are used to evaluate the performance of the counting method. This embodiment conducts an ablation experiment to verify the performance of the detector and tracker proposed in this embodiment. The results are shown in Table 3.
[0065] Table 3 Counting channel ablation experiment results
[0066] frame Detector Tracker True Value Predicted value IDS↓ MOTA↑ MAE↓ 1 YOLOv8s BoT-SORT 8447 9771 835 84.40% 0.17 2 AD-YOLO BoT-SORT 9204 742 85.10% 0.09 3 YOLOv8s MR-SORT 9488 603 84.90% 0.14 4 AD-YOLO MR-SORT 8984 538 85.60% 0.07
[0067] Comparing the results of framework 1 and framework 2, it can be seen that when the AD-YOLO model is used as the detector, MOTA is improved by 0.7%, and IDS and MAE are reduced. Comparing the results of framework 1 and framework 3, it can be seen that when the MR-SORT method is used as the tracker, MOTA is improved by 0.5%, and IDS and MAE are reduced. It can be seen from the results in the table that the detector and tracker proposed in this embodiment have good effects.
[0068] In order to verify the effectiveness of the multi-target tracking method proposed in this embodiment on the orchard video, other trackers with superior performance are tested using the same test video. The results are shown in Table 4.
[0069] Table 4 Performance comparison results of the method in this embodiment and other SOTA models
[0070]
[0071] The table shows the superiority of the method proposed in this embodiment. In terms of IDS, MOTA and MAE, the method proposed in this embodiment is superior to other SOTA methods. The counting results of the 7 videos based on the AD-YOLO and MR-SORT methods proposed in this embodiment are shown in Figure 2. Figure 4 As shown, it can be seen that the predicted value is basically consistent with the true value.
[0072] The apple counting method based on the AD-YOLO model and the MR-SORT method proposed in this embodiment reduces the false detection and missed detection in complex orchard environments by combining advanced structures such as the attention mechanism and the YOLOv8 model; by combining a non-deep learning image appearance feature matching method and the BoT-SORT method, the accuracy of apple counting is improved, and the time required for counting is also reduced. From the results, it can be seen that compared with other SOTA models, the counting method proposed in this embodiment has a higher accuracy rate, laying a solid foundation for orchard yield estimation.
[0073] Only the preferred embodiments of this embodiment are described in detail above, but this embodiment is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of this embodiment, and various changes should be included in the protection scope of this embodiment.
Claims
1. Apple counting method based on AD-YOLO model and MR-SORT method, Features: The following steps are involved: S1. Obtain image data for training and video data for counting test, preprocess the image data to obtain preprocessed image data, and construct a data set; S2. Build the apple detection model AD-YOLO based on YOLOv8, GAM, ODConv and SimCSPSPPF; replace the ordinary convolution in the CBS unit in the backbone network of the YOLOv8 model with the full-dimensional dynamic convolution ODConv; add the global attention mechanism GAM before the SPPF unit in the backbone network of the YOLOv8 model; replace the SPPF unit in the backbone network of the YOLOv8 model with the more advanced spatial pyramid pooling layer SimCSPSPPF; S3. Use the training set to train on the constructed AD-YOLO network, select the best model as the final AD-YOLO model, and compare the results with other SOTA models; S4. Based on BoT-SORT, a multi-target tracking method MR-SORT is proposed to construct an apple counting channel. The appearance feature extraction using the ResNeSt50 network in BoT-SORT is changed to the SIFT feature descriptor extraction, and then the VLAD image retrieval algorithm is used for appearance feature matching. The output of the target detection module is used as the input of the multi-target tracking module, and the two are combined into an apple counting channel. S5. Input the test video data into the apple counting channel, obtain the results, and establish evaluation indicators for the performance evaluation of the counting method, and compare the results with other SOTA models.
2. The apple counting method based on the AD-YOLO model and the MR-SORT method according to claim 1, Features: The data set in S1 includes a detection data set and a tracking data set; The tracking data set is as follows: The tracking data comes from the Apple tracking public database. This data consists of 7 videos with a resolution of 1920×1080 and a frame rate of 30fps. Each video is 1 minute. This data set has corresponding annotation information in the format of [frame, ID, x, y, w, h, Cf, C, v], where: frame is the frame number; ID is the unique identifier of the apple in the entire video sequence; x, y, w, h are the horizontal and vertical coordinates, width, and height of the left vertex of the bounding box respectively; Cf is the confidence; C is the class identifier; v is the visibility. Since there is only one class, the class identifiers in the database are all 1, and the confidence and visibility are set to 1; The detection dataset is: the detection dataset comes from two aspects: on the one hand, three public databases, and on the other hand, part of the data in the above-mentioned tracking dataset. The three public databases containing apple RGB images are the Minnesota Apple dataset, the Washington State University dataset, and the Fuji dataset. The three public databases have corresponding annotation information. Part of the data in the tracking dataset is 960 images captured from 7 tracking data videos at a certain frequency.
3. The apple counting method based on the AD-YOLO model and the MR-SORT method according to claim 1, Features: The preprocessing method in S1 is: annotation information format conversion, data enhancement and data partitioning, and the data enhancement includes horizontal rotation, vertical rotation, brightness change, contrast change, saturation change, Gaussian noise, salt and pepper noise, size change and motion blur effect.
4. The apple counting method based on the AD-YOLO model and the MR-SORT method according to claim 1, Features: The training method in S3 is: initializing the trainable parameters in the network and inputting the training set into the network in batches; According to the predicted value and the true label, a loss function is constructed and the loss is calculated. The optimization algorithm is used for back propagation to update the network parameters until the loss no longer decreases within a certain range. At this time, the network parameters are saved as the final model. Then, other SOTA models are trained using the same dataset and the same training strategy. Other SOTA models include Faster R-CNN, YOLOv5s, YOLOv7-tiny, and YOLOv8s, and the parameter results of these models are compared.
5. The apple counting method based on the AD-YOLO model and the MR-SORT method according to claim 1, Features: The evaluation indicators in S5 include the number of ID switching, MOTA multi-target tracking accuracy, and MAE average accuracy error. The formula is as follows: In the above formula, for the tth frame, m t is the number of missed detections, i.e. FN, false negative; fp t is the number of false positives, i.e. FP; mme t Indicates the number of ID switches, gt is a true value; MAE is the average precision error of all test videos, m is the number of videos, y is the predicted number of apples in the video, and gt is the actual number of apples in the video.
6. The apple counting method based on the AD-YOLO model and the MR-SORT method according to claim 1, Features: Other SOTA models in S5 include: YOLOv8s and DeepSORT, YOLOv8s and Cascade-SORT, YOLOv8s and BoT-SORT.
