Apple fruit detection method and system based on deep learning
By improving the YOLOv5s model and incorporating the SE attention mechanism and BiFPN structure, the problems of gloss and occlusion effects in apple fruit detection were solved, enabling efficient fruit detection on mobile devices and meeting the real-time needs of smart agriculture.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-05
- Publication Date
- 2026-04-10
AI Technical Summary
Existing technologies for apple fruit detection suffer from several problems: high gloss on the fruit surface leads to strong reflections, and shading by branches and leaves severely affects detection accuracy. Furthermore, deep learning models require a large amount of computation, making them difficult to apply in real time on mobile devices.
The YOLOv5s model was trained and improved by adding SE attention mechanism and BiFPN structure, combined with CIoULoss to optimize the detection performance of the model in complex environments, and the model was made lightweight to adapt to mobile devices. The NCNN toolchain was used for porting.
Maintaining high detection accuracy and recall in complex environments, it achieves real-time stable operation on mobile devices, improves detection efficiency, and meets the real-time visual perception needs of smart agriculture.
Smart Images
Figure CN121838121A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of agricultural artificial intelligence and computer vision, and particularly relates to an apple fruit detection method and system based on deep learning. BACKGROUND
[0002] Apple planting has a wide economic and industrial foundation in China, but manual picking has been the mainstream for a long time, which is time-consuming and labor-intensive, and the efficiency is limited. In recent years, automatic picking technology based on image recognition has gradually become a research hotspot. However, there are still several outstanding problems in apple fruit detection: the high glossiness of the fruit surface easily produces strong light reflection, resulting in blurred recognition boundary; the color of branches and leaves is close to that of the fruit, and the detection accuracy obviously decreases when the shielding is serious; although some deep learning models can improve the accuracy, the calculation amount is large, and it is difficult to apply in real time on mobile terminals. Therefore, there is an urgent need for a solution that can maintain high detection accuracy in complex environments while taking into account lightweight and real-time performance. SUMMARY
[0003] The purpose of the present application is to provide an apple fruit detection method and system based on deep learning that can improve the efficiency of apple fruit detection.
[0004] Technical scheme: The apple fruit detection method based on deep learning provided by the present application comprises:
[0005] (1) Collecting apple fruit images and making image data sets;
[0006] (2) Enhancing the image data, and inputting the enhanced image data into a YOLOv5s model for training;
[0007] (3) Improving the trained YOLOv5s model to obtain an improved detection model;
[0008] (4) Using the improved detection model to detect the apple to be detected.
[0009] Further, the step (1) comprises:
[0010] Collecting apple fruit images from random orchards; the shooting conditions include sunny, cloudy, backlight and night fill light, and the shooting angles include horizontal, upward and downward; using LabelImg tool to manually label the collected images to generate data sets.
[0011] Further, the step (2) enhances the image data by randomly flipping, color disturbance, scale scaling and Mosaic splicing.
[0012] Further, step (2) involves inputting the enhanced image data into the YOLOv5s model for training. Specifically, YOLOv5s is used as the basic detection model for training. During the training process, the input image size is set to 640×640, the batch size is 32, the initial learning rate is 0.01, and a cosine annealing strategy is used for dynamic adjustment until the preset total number of training rounds is reached.
[0013] Further, step (3) includes:
[0014] SE attention mechanism is added to the backbone of the trained YOLOv5s model to enhance the expressive power of the feature channels; BiFPN structure is introduced in the Neck part to achieve multi-scale feature fusion; and CIoULoss is introduced into the loss function to improve the accuracy of bounding box regression.
[0015] Furthermore, it also includes: exporting the improved detection model to ONNX format and converting it into .param and .bin files using the NCNN toolchain; then integrating it into the Android Studio development environment to complete the mobile APP development.
[0016] Based on the same inventive concept, the present invention also provides an apple fruit detection system based on deep learning, comprising:
[0017] The acquisition module is used to acquire images of apple fruits and create image datasets;
[0018] The training module is used to augment image data, and the augmented image data is input into the YOLOv5s model for training;
[0019] The improvement module is used to improve the trained YOLOv5s model to obtain an improved detection model;
[0020] The detection module is used to detect the apples to be tested using an improved detection model.
[0021] Based on the same inventive concept, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the deep learning-based apple fruit detection method according to any of the preceding claims.
[0022] Based on the same inventive concept, the present invention also provides a computing device, comprising: one or more processors, one or more memories, and one or more programs, the programs being stored in the memory and configured to be executed by the processor, the programs being loaded onto the processor to implement the steps of the deep learning-based apple fruit detection method according to any of the preceding claims.
[0023] Based on the same inventive concept, the present invention also provides a storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the steps of the deep learning-based apple fruit detection method according to any one of the preceding claims.
[0024] Beneficial effects: Compared with existing technologies, this invention significantly enhances the model's ability to identify apple targets in complex orchard environments by introducing an SE attention mechanism and a multi-scale feature fusion structure, achieving simultaneous improvement in detection accuracy and robustness. While maintaining the model's lightweight design, the improved YOLOv5 model was successfully ported to the Android platform, enabling it to run stably on mobile devices at a rate exceeding 25 frames per second. This invention exhibits high detection accuracy and recall under various lighting and occlusion conditions, not only improving the overall efficiency of apple fruit detection but also providing a reliable and practical technical solution for real-time visual perception in smart agriculture scenarios. Attached Figure Description
[0025] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation
[0026] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
[0027] As attached Figure 1 As shown, the deep learning-based apple fruit detection method in this embodiment includes:
[0028] (1) Collect images of apple fruits and create an image dataset;
[0029] (2) Enhance the image data and input the enhanced image data into the YOLOv5s model for training;
[0030] (3) Improve the trained YOLOv5s model to obtain the improved detection model;
[0031] (4) Use the improved detection model to detect the apples to be tested.
[0032] Specifically, in step (1), approximately 500 images of apples were collected in an orchard environment, with an image resolution of 1920×1080. The shooting conditions covered various scenarios, including sunny days, cloudy days, backlighting, and nighttime lighting, and the shooting angles included horizontal, upward, and downward views. All collected images were manually labeled using the LabelImg tool, with the label category being "apple," and finally, an XML format labeled file was generated. This dataset was used for subsequent model training and validation.
[0033] In step (2), YOLOv5s was used as the basic detection model and trained in a high-performance computing environment. The hardware environment included dual NVIDIA RTX 3090 GPUs and 128GB of memory, the operating system was Ubuntu 20.04, and the training framework was PyTorch 1.12. During training, the input image size was set to 640×640, the batch size was 32, the initial learning rate was 0.01, and a cosine annealing strategy was used for dynamic adjustment. The total number of training epochs was 100. To enhance the generalization ability of the model, various data augmentation methods were used, such as random flipping, color perturbation, scale scaling, and Mosaic stitching.
[0034] In step (3), improvements are made to the original YOLOv5 structure: a Squeeze-and-Excitation (SE) attention mechanism is added to the Backbone part to enhance the expressive power of the feature channels; a BiFPN structure is introduced to the Neck part to achieve multi-scale feature fusion, thereby improving the model's detection performance for small and occluded targets; and CIoU Loss is introduced into the loss function to improve the accuracy of bounding box regression. Experimental results show that the improved model achieves an average detection accuracy (mAP) of approximately 5% higher than the original YOLOv5s in the apple detection task.
[0035] In step (4), the improved detection model is used to detect the apple to be detected.
[0036] The performance of different models was compared on the test set, and the results are shown in Table 1:
[0037]
[0038] As can be seen from the above comparison results, the improved model proposed in this invention is superior to the original model in terms of detection accuracy, precision, and recall, while still maintaining good real-time performance on mobile devices.
[0039] This embodiment also includes: after training, exporting the model to ONNX format and converting it into .param and .bin files using the NCNN toolchain. Then, it is integrated into the Android Studio development environment to complete the mobile app development. This app can achieve real-time camera capture, object detection inference, and result visualization. The deployed application installation package is approximately 65MB in size and runs stably.
[0040] In orchard harvesting experiments, the model of this invention achieved a detection rate of over 92% for apples under unobstructed conditions; even when the fruit was more than 50% obscured by branches and leaves, the recall rate remained above 84%. Testing on an Android device equipped with a Snapdragon 888 processor and 8GB of RAM showed a detection frame rate between 27 and 30 FPS, meeting the real-time requirements of intelligent harvesting scenarios.
[0041] Based on the same inventive concept, this embodiment also provides an apple fruit detection system based on deep learning, including:
[0042] The acquisition module is used to acquire images of apple fruits and create image datasets;
[0043] The training module is used to augment image data, and the augmented image data is input into the YOLOv5s model for training;
[0044] The improvement module is used to improve the trained YOLOv5s model to obtain an improved detection model;
[0045] The detection module is used to detect the apples to be tested using an improved detection model.
[0046] Based on the same inventive concept, this embodiment also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the deep learning-based apple fruit detection method according to any of the above claims.
[0047] Based on the same inventive concept, this embodiment also provides a computing device, including: one or more processors, one or more memories, and one or more programs, the programs being stored in the memory and configured to be executed by the processor, the programs being loaded onto the processor to implement the steps of the deep learning-based apple fruit detection method according to any of the preceding claims.
[0048] Based on the same inventive concept, this embodiment also provides a storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the steps of the deep learning-based apple fruit detection method according to any of the preceding claims.
Claims
1. A deep learning-based apple fruit detection method, characterized in that, include: (1) Collect images of apple fruits and create an image dataset; (2) Enhance the image data and input the enhanced image data into the YOLOv5s model for training; (3) Improve the trained YOLOv5s model to obtain the improved detection model; (4) Use the improved detection model to detect the apples to be tested.
2. The apple fruit detection method based on deep learning according to claim 1, characterized in that, Step (1) includes: Images of apples were collected from random orchards under various conditions, including sunny, cloudy, backlit, and nighttime lighting, and from horizontal, upward, and downward angles. The collected images were manually labeled using the LabelImg tool to generate a dataset.
3. The apple fruit detection method based on deep learning according to claim 1, characterized in that, The image data enhancement described in step (2) is specifically achieved by randomly flipping, color perturbation, scaling, and Mosaic stitching to enhance the image data.
4. The apple fruit detection method based on deep learning according to claim 1, characterized in that, Step (2) involves inputting the enhanced image data into the YOLOv5s model for training. Specifically, YOLOv5s is used as the basic detection model for training. During the training process, the input image size is set to 640×640, the batch size is 32, the initial learning rate is 0.01, and the cosine annealing strategy is used for dynamic adjustment until the preset total number of training rounds is reached.
5. The apple fruit detection method based on deep learning according to claim 1, characterized in that, Step (3) includes: SE attention mechanism is added to the backbone of the trained YOLOv5s model to enhance the expressive power of the feature channels; BiFPN structure is introduced in the Neck part to achieve multi-scale feature fusion; and CIoULoss is introduced into the loss function to improve the accuracy of bounding box regression.
6. The apple fruit detection method based on deep learning according to claim 1, characterized in that, Also includes: The improved detection model was exported to ONNX format and then converted into .param and .bin files using the NCNN toolchain; Then, it was integrated into the Android Studio development environment to complete the mobile app development.
7. An apple fruit detection system based on deep learning, characterized in that, include: The acquisition module is used to acquire images of apple fruits and create image datasets; The training module is used to augment image data, and the augmented image data is input into the YOLOv5s model for training; The improvement module is used to improve the trained YOLOv5s model to obtain an improved detection model; The detection module is used to detect the apples to be tested using an improved detection model.
8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the deep learning-based apple fruit detection method according to any one of claims 1 to 6.
9. A computing device, characterized in that, include: One or more processors, one or more memories, and one or more programs, said programs being stored in the memory and configured to be executed by the processor, said programs, when loaded onto the processor, implementing the steps of the deep learning-based apple fruit detection method according to any one of claims 1 to 6.
10. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the steps of the deep learning-based apple fruit detection method according to any one of claims 1 to 6.