A method for detecting cherry maturity in a dense orchard based on a lightweight PAM-DETR algorithm

Abstract

The present application provides a kind of dense orchard cherry ripeness detection method based on lightweight PAM-DETR algorithm, it is related to intelligent agriculture and computer vision technology field.The method comprises: S1. construct and carry out enhancement pretreatment including multiple complex scenes cherry image dataset;S2. construct the PAM-DETR detection model based on RT-DETR-R18 framework, the model integrates PREBlock feature extraction module, AIFI-ASSA attention module, MEFPN feature fusion network and PIoU loss function;S3. the PAM-DETR model is trained using the pretreated dataset, and the model parameters are optimized to improve the detection performance;S4. the image to be detected orchard is input into the trained model, and the maturity class and positioning boundary box of cherry are output.The present application can effectively improve the detection performance in dense shading, light fluctuation and fruit overlapping and other complex orchard environment, and provides an effective technical scheme for the intelligent unmanned harvesting of cherry.

Description

Technical Field

[0001] This invention relates to the field of intelligent agriculture technology, and more specifically, to a computer vision target detection technology based on the lightweight PAM-DETR algorithm, which is particularly suitable for automated detection of cherry ripeness in dense orchard environments. Background Technology

[0002] Cherries are nutritious but have a short shelf life and are easily perishable. In natural orchards, fruit ripening is uneven, and manual harvesting and sorting are costly. Accurate ripeness detection is key to digital orchard management and unmanned harvesting. However, dense orchards present complex challenges such as leaf shading, fluctuating light, and overlapping fruits, posing difficulties for automated detection.

[0003] Traditional machine learning methods rely on manual feature extraction, which has poor adaptability. In the complex and dynamic environment of orchards, the detection accuracy and robustness are difficult to meet the requirements, and they cannot cope with the interference caused by variables such as light and shading.

[0004] While deep learning methods can automatically extract features, CNN-type models lack global information modeling capabilities and are prone to missing detections when dealing with dense small targets. Transformer-type models (such as RT-DETR) have global modeling capabilities, but they suffer from high computational complexity and information dilution during feature fusion, making it difficult to balance accuracy, speed, and resource consumption.

[0005] Therefore, there is an urgent need for a new method for cherry ripeness detection that can adapt to the complex scenarios of dense orchards, balance detection accuracy and deployment efficiency, and improve real-time performance while ensuring lightweight design, so as to break through the existing technical bottlenecks. Summary of the Invention

[0006] The purpose of this invention is to provide a method for detecting cherry ripeness in dense orchards based on the lightweight PAM-DETR algorithm, addressing the problems of insufficient robustness and difficulty in balancing accuracy and lightweight design in existing technologies for complex orchard environments. This method achieves high-precision, lightweight, and real-time detection of cherry ripeness in dense orchards by designing an innovative feature extraction module, optimizing the attention mechanism, constructing an efficient feature fusion network, and improving the loss function, providing reliable technical support for intelligent harvesting robots.

[0007] The method includes the following steps: S1. Construct a multi-scenario cherry ripeness dataset and perform preprocessing; S2. Construct a lightweight PAM-DETR detection model. The model is based on the RT-DETR-R18 framework and integrates the PREBlock feature extraction module, AIFI-ASSA attention module, MEFPN feature fusion network, and PIOU loss function. S3. Train the PAM-DETR model using the preprocessed dataset and optimize the model parameters to improve detection performance; S4. Input the orchard image to be detected into the trained model, and output the cherry ripeness classification result and target location information.

[0008] Preferably, in step S1, constructing a multi-scene cherry maturity dataset and performing preprocessing involves: acquiring cherry images with different lighting conditions, shooting angles, fruit overlap, and leaf occlusion by combining on-site shooting with collection from public datasets; classifying cherries into three categories based on maturity: unripe, semi-ripe, and ripe; using the LabelImg tool to annotate bounding boxes; and dividing the dataset into training, validation, and test sets in a 7:1:2 ratio; and performing data augmentation operations such as brightness / contrast adjustment, random rotation, and random cropping on the training set to expand data diversity and improve model generalization ability.

[0009] Preferably, in step S2, the PREBlock feature extraction module is the core of the model backbone network. It constructs the PREConv component by integrating partial convolution (PConv) and reparameterized convolution (RepConv), and then combines it with the residual BasicBlock. PREConv selects a portion of the input feature map for calculation, while the remaining channels are directly retained as identity channels. During the training phase, a multi-branch structure is used to extract features, and during the inference phase, reparameterization technology is used to merge them into a single 3×3 convolution kernel, which reduces the amount of computation and memory usage while ensuring feature extraction capability.

[0010] Furthermore, in step S2, the AIFI-ASSA attention module is an improvement on the AIFI module based on RT-DETR, and adopts a dual-branch self-attention structure: the first branch filters low-similarity noise interactions through sparse self-attention (SSA) of squared ReLU, and the second branch retains standard softmax dense self-attention (DSA) to avoid information loss; the two branches are fused through learnable adaptive weights, which effectively improves the feature representation quality in complex scenarios.

[0011] Furthermore, in step S2, the MEFPN feature fusion network adopts a dual-path structure (bottom-up and top-down), integrates four scale feature maps (including a high-resolution P2 layer), assigns adaptive weights to features of different scales through the fast normalization fusion (FNFusion) module, and optimizes feature representation by combining the RepC3 module and the gConvC3 module with a gating mechanism, thereby achieving efficient fusion of deep and shallow semantic information and spatial information, and enhancing the detection capability of multi-scale targets and occluded targets.

[0012] Furthermore, in step S2, the PIoU loss function is used to replace the original GIoU loss function. The absolute distance deviation between the predicted box and the ground truth box is normalized to a relative proportional deviation by using an adaptive penalty factor based on the target size. A gradient adjustment function is designed based on the anchor box quality to differentiate samples of different difficulty, thereby accelerating model convergence and improving localization accuracy.

[0013] Furthermore, in step S3, the model training is based on the Ubuntu 22.04 system, configured with an Intel Xeon E5-2698v4 CPU and an NVIDIA A100 GPU, and uses the Python 3.8 and PyTorch 2.2.0 framework. The training parameters are set as follows: 150 iterations, batch size 4, input image size 640×640 pixels, and the AdamW optimizer has an initial learning rate of 0.0001 and momentum of 0.9. The parameters are updated iteratively through the training set, and the performance is monitored on the validation set to avoid overfitting.

[0014] Furthermore, in step S4, the orchard image to be detected is resized and then input into the trained PAM-DETR model. The model extracts features through the backbone network, enhances semantic information through the AIFI-ASSA module, and fuses multi-scale features through the MEFPN network. Finally, it outputs detection results including maturity category, bounding box coordinates, and confidence level, supporting detection scenarios with image, video, and real-time camera input.

[0015] The beneficial effects of this invention are as follows:

[0016] This invention discloses a method for detecting cherry maturity in dense orchards based on the lightweight PAM-DETR algorithm. The PREBlock module achieves model lightweighting, reducing the number of parameters and computational cost by 26.6% and 27.0% respectively compared to the baseline model, making it suitable for edge device deployment. The AIFI-ASSA module and MEFPN network work together to improve robustness in complex scenarios, effectively addressing issues such as leaf shading, light fluctuations, and dense fruit overlap. The PIOU loss function optimizes localization accuracy, achieving a model mAP@0.5 of 79.5%, a 3.7% improvement over the original RT-DETR. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall architecture of the PAM-DETR model in the dense orchard cherry ripeness detection method based on the lightweight PAM-DETR algorithm described in this invention.

[0018] Figure 2 This is a schematic diagram of the PREConv module in the dense orchard cherry ripeness detection method based on the lightweight PAM-DETR algorithm described in this invention.

[0019] Figure 3 This is a schematic diagram comparing the structure of the PREConv module in the training and inference stages of a dense orchard cherry ripeness detection method based on the lightweight PAM-DETR algorithm described in this invention.

[0020] Figure 4 This is a schematic diagram of the ASSA module in the dense orchard cherry ripeness detection method based on the lightweight PAM-DETR algorithm described in this invention.

[0021] Figure 5 This is a schematic diagram of the MEFPN feature fusion network in the dense orchard cherry ripeness detection method based on the lightweight PAM-DETR algorithm described in this invention.

[0022] Figure 6 This is a schematic diagram of the training accuracy and verification loss curves in the ablation experiment training process of the dense orchard cherry ripening detection method based on the lightweight PAM-DETR algorithm described in this invention.

[0023] Figure 7 This is a schematic diagram comparing the detection performance of the intensive orchard cherry ripening detection method based on the lightweight PAM-DETR algorithm described in this invention with that of the original RT-DETR in different scenarios. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0025] The present invention provides a method for detecting cherry maturity in dense orchards based on the lightweight PAM-DETR algorithm. This method achieves efficient detection in complex environments through multi-module collaborative optimization, and is implemented according to the following core steps: S1. Construct a multi-scenario cherry ripeness dataset and perform preprocessing; S2. Construct a lightweight PAM-DETR detection model. The model is based on the RT-DETR-R18 framework and integrates the PREBlock feature extraction module, AIFI-ASSA attention module, MEFPN feature fusion network and PIOU loss function. S3. Train the PAM-DETR model using the preprocessed dataset and optimize the model parameters to improve detection performance; S4. Input the orchard image to be detected into the trained model, and output the cherry ripeness classification result and target location information.

[0026] In one specific embodiment, step S1, acquiring cherry image data and performing preprocessing, specifically involves: the first batch of images being captured using a Xiaomi 14 smartphone at a distance of 10-50cm, covering various shooting angles such as low angle and eye-level, as well as complex environments such as strong light, backlight, overlapping fruits, and leaf occlusion; the second batch of images originates from the Kaggle public cherry recognition dataset, with a single image resolution of 4000×3000 pixels. The acquired images are labeled using LabelImg to create a dataset, where mature fruits are labeled as (ripe_cherry), semi-ripe fruits as (half_cherry), and unripe fruits as (unripe_cherry).

[0027] In this embodiment, the dataset is randomly divided into a training set, a validation set, and a test set in a 7:1:2 ratio, with an initial training set of 1925 images, a validation set of 275 images, and a test set of 550 images. Multi-dimensional enhancement operations are then performed on the training set to expand data diversity, including brightness / contrast adjustments, random rotation, and random cropping. The final enhanced training set contains 5275 images. The resolution is uniformly set to 640×640 pixels before inputting the images into the model to maintain input consistency.

[0028] In one specific embodiment, in step S2, the PAM-DETR model is as follows: Figure 1 As shown, it mainly includes a backbone network (PREBlock), an attention feature extraction module (AIFI-ASSA), a multi-scale feature fusion module (MEFPN), a decoder, and a PIOU loss function. These components work together to achieve feature extraction, enhancement, fusion, and accurate localization and classification.

[0029] In this embodiment, the PREBlock module in PAM-DETR is as follows: Figure 2 As shown, the core component of the model's backbone network is the PREConv component, which is constructed by integrating partial convolution (PConv) and reparameterized convolution (RepConv), and then combined with the residual BasicBlock structure. The PREConv component selects a continuous subset of channels from the input feature map as computation channels, and the remaining channels are directly retained as identity channels to avoid loss of feature information. The proportion of computation channels is set to 1 / 4 to balance computational efficiency and feature representation. Through the stacking of multiple PREBlock layers, the backbone network extracts cherry image features layer by layer, outputting four feature maps of different dimensions, namely P2, P3, P4, and P5 from high resolution to low resolution, respectively adapting to the detection requirements of cherry targets at different scales.

[0030] In this embodiment, the structure diagram of the PREBlock module in PAM-DETR during the training / inference phase is as follows: Figure 3 As shown. During the training phase, RepConv in PREConv adopts a multi-branch structure, including Conv3×3, Conv1×1, and an identity branch. Each branch is paired with a batch normalization (BN) layer to extract spatial features, supplement channel information, and retain original input features, respectively. Multi-path feature learning enhances the model's expressive power. During the inference phase, structural reparameterization technology is used to equivalently merge the parameters of the multi-branch and BN layers into a single 3×3 convolutional kernel, eliminating branch redundancy and improving inference efficiency. The computational and memory access costs of REConv are calculated using the following formulas: in , The height and width of the input / output feature maps. The kernel size is [size]. This refers to the number of channels involved in the calculation. When the percentage of calculated channels... At that time, its computational cost is only 1 / 16 of that of standard convolution, and its memory access cost is reduced to 1 / 4 of that of standard convolution.

[0031] In this embodiment, the AIFI-ASSA attention feature extraction module in PAM-DETR is as follows: Figure 4 As shown, it is an improved AIFI module based on the RT-DETR model, specifically designed for semantic enhancement of the high-dimensional feature map P5 output by the backbone network.

[0032] Specifically, the AIFI-ASSA module receives the low-resolution, high-dimensional feature map P5 output from the backbone network. After layer normalization, it is divided into non-overlapping local windows and flattened. A query (Q), key (K), and value (V) matrix is ​​generated through linear projection. The module employs a two-branch self-attention structure: the SSA branch filters low-similarity interactions to remove background noise, while the DSA branch retains all feature interactions to avoid losing key information. The two branches are fused using learnable adaptive weights to balance feature sparsity and completeness. The processed P5 feature map, denoted as F5, has significantly enhanced semantic expressive power and can accurately capture the core features of cherry ripeness. It is then combined with features from P2, P3, and P4. Figure 1 Multi-scale fusion is performed with the input MEFPN module.

[0033] In this embodiment, the MEFPN module in PAM-DETR is as follows: Figure 5 As shown, the core module for multi-scale feature fusion is used to receive the P2, P3, and P4 feature maps output by the backbone network and the F5 feature map processed by the AIFI-ASSA module. Through deep fusion, it achieves complementary advantages of deep and shallow features, adapting to complex scenarios such as scale differences and overlapping of cherry targets in dense orchards.

[0034] Specifically, the MEFPN module adopts a dual-path structure of bottom-up and top-down, integrating the FNFusion module, the RepC3 module, and the gConvC3 module with a gating mechanism. The FNFusion module assigns learnable adaptive weights to features at different scales, ensuring weight rationality through a non-negative activation function to achieve dynamic differentiation of feature contributions. Its fusion formula is as follows: in The middle part represents the input features. To correspond to non-negative learnable weights, It is a very small constant to avoid numerical instability problems caused by a denominator of zero.

[0035] In this embodiment, the gConvC3 module in MEFPN replaces the RepC3 module in the deep layers of the original model. It consists of a gating filtering branch and a lightweight feature extraction branch. The gating filtering branch first performs batch normalization on the input features to stabilize their distribution, then transforms the dimensions through pointwise convolution (PWConv1) and generates a gating signal using the Sigmoid function. This signal accurately filters effective cherry target features and suppresses redundant information such as leaf occlusion and background interference. The lightweight feature extraction branch uses a depthwise separable convolution structure, adjusting the channel dimensions through pointwise convolution (PWConv2) and combining it with depthwise convolution (DWConv) to capture local spatial features such as fruit texture and edges, reducing computational complexity while ensuring feature extraction capabilities. The outputs of the two branches are integrated through element-wise multiplication, then unified through pointwise convolution (PWConv3), and finally superimposed with the residual connections of the original input to avoid feature degradation.

[0036] The decoder layer uses the existing decoder layer in the RT-DETR model, which will not be elaborated here.

[0037] In this embodiment, the PAM-DETR model uses the PioU loss function instead of the original GIoU loss function in RT-DETR to optimize the localization accuracy of the cherry target bounding box. Its penalty factor P is based on the width and height of the cherry's true bounding box, normalizing the absolute distance between the predicted box and the corresponding edges of the true bounding box to a relative proportional deviation strongly correlated with the size of the true bounding box. This eliminates the metric differences in distance error for cherry targets of different scales, ensuring the penalty strength matches the actual size of the fruit. Simultaneously, PioU is supplemented by a gradient adjustment function f(x), which quantifies the matching quality between the anchor box and the cherry's true bounding box through the penalty factor P. Differential gradient update strategies are implemented for detection samples of varying difficulty: accelerating the regression convergence of medium-quality anchor boxes, suppressing harmful gradients of low-quality anchor boxes, stabilizing the optimization process of high-quality anchor boxes, and avoiding overfitting. The loss function formula is as follows: in, , , , These are the absolute distances between the corresponding edges of the predicted bounding box and the ground truth bounding box, respectively. , represents the width and height of the ground truth bounding box, and IoU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box, used to measure the degree of matching of the overlapping areas of the two.

[0038] In one specific embodiment, step S3 involves training and evaluating the constructed PAM-DETR model using the preprocessed dataset. Model training was conducted on an Ubuntu 22.04 operating system, with an Intel Xeon E5-2698 v4 CPU and an NVIDIA A100 GPU. The software environment used Python 3.8.20, PyTorch 2.2.0 framework, and CUDA 12.1. The experimental hyperparameter settings are shown in Table 1 below.

[0039] Table 1 Hyperparameter Settings

[0040] The model weights are iteratively optimized using the training set data, and the mean accuracy (mAP@0.5) and loss function value are monitored using the validation set to avoid overfitting.

[0041] To demonstrate the superiority of each improved module, this invention designed five sets of ablation experiments, as shown in Table 2 below. Table 2 Ablation experiments of PAM-DETR

[0042] Experimental results show that the performance of the PAM-DETR model is gradually improved by introducing innovative modules. Compared with the original RT-DETR, PAM-DETR improves mAP@0.5 by 3.7%, reduces the number of parameters and computation by 26.6% and 27.0% respectively, while maintaining a real-time processing speed of 70 FPS, achieving a balance between accuracy and efficiency.

[0043] To further explore the impact of each improved module on the model training dynamics, a comparison curve of training accuracy and validation loss in the ablation experiment was plotted. For example... Figure 6As shown, the improvement in training accuracy of the model exhibits a phased pattern: a rapid increase in the first 50 epochs, a slow increase from epochs 50 to 120, and then a gradual convergence. Starting from the baseline model (black curve), each addition of an improved module significantly improves accuracy. Furthermore, the improved PAM-DETR model (red curve) exhibits the fastest convergence and smoothest curve in its validation loss, demonstrating a good synergistic effect between the modules and the PIOU loss function.

[0044] In one specific embodiment, in step S4, the trained PAM-DETR model is deployed in the intelligent cherry harvesting system to detect and locate the maturity of the collected orchard images. Specifically, the system acquires original images of the orchard to be detected through a camera, adjusts the resolution to 640×640 pixels, and then inputs them into the trained PAM-DETR model. Finally, the decoder outputs the bounding box coordinates, maturity category (immature, semi-mature, mature), and corresponding confidence score for each detected target. The system supports input of single images, video streams, and real-time camera footage. The detection results can be directly visualized and overlaid on the original image or converted into structured data for the robot actuator to perform precise harvesting.

[0045] Figure 7 This paper illustrates the visual comparison results of the proposed method under complex factors such as fruit occlusion, similar backgrounds, and uneven lighting. These influencing factors often interact and overlap, further increasing the detection difficulty. Specifically, the first row shows the original image, the second row shows the result of the original method, and the third row shows the result of the proposed method. Bright yellow arrows are used to highlight the differences. It can be seen that in the densely occluded scene shown in group (a), the original method missed some overlapping fruits; in group (b), under the interference of similar color backgrounds, the original method misdetected leaves as fruits; in the backlit scene in group (c), the original method generated a false detection box with low confidence and incorrect positioning in the high-contrast area at the junction of light and shadow in the upper right corner. In contrast, the proposed method did not produce the above-mentioned false detections and missed detections, demonstrating more stable detection performance.

[0046] This invention provides a method for cherry ripeness detection in dense orchards based on a lightweight PAM-DETR algorithm. By integrating a lightweight PREBlock feature extraction module, a dual-branch adaptive AIFI-ASSA attention module, an efficient MEFPN feature fusion network, and a PIOU loss function, a high-precision, low-complexity PAM-DETR detection model is constructed. This method effectively solves the problem of insufficient robustness in dense orchard environments caused by leaf shading, light fluctuations, fruit overlap, and background similarity. While significantly reducing the number of model parameters and computational load, it improves the detection accuracy (mAP@0.5) to 79.5% and maintains a real-time processing capability of 70 FPS, providing reliable technical support for intelligent unmanned cherry harvesting.

[0047] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for detecting cherry ripeness in dense orchards based on a lightweight PAM-DETR algorithm, characterized in that, Includes the following steps: S1. Construct a multi-scenario cherry ripeness dataset and perform preprocessing; S2. Construct a lightweight PAM-DETR detection model. The model is based on the RT-DETR-R18 framework and integrates the PREBlock feature extraction module, AIFI-ASSA attention module, MEFPN feature fusion network and PIOU loss function. S3. Train the PAM-DETR model using the preprocessed dataset and optimize the model parameters to improve detection performance; S4. Input the orchard image to be detected into the trained PAM-DETR model, and output the cherry ripeness classification result and target location information.

2. The method according to claim 1, characterized in that, In step S1, a multi-scene cherry maturity dataset is constructed and preprocessed. Specifically, cherry images containing different lighting conditions, shooting angles, fruit overlap, and leaf occlusion scenes are collected. Cherries are divided into three categories according to maturity: immature, semi-ripe, and ripe, and bounding boxes are labeled. The dataset is divided into training set, validation set, and test set, and data augmentation operations are performed on the training set.

3. The method according to claim 1, characterized in that, In step S2, the PREBlock feature extraction module constructs the PREConv component by integrating partial convolution and reparameterized convolution, and then combines it with the residual block BasicBlock to form a single 3x3 convolution kernel. PREConv adopts a multi-branch structure during the training phase and is merged into a single 3x3 convolution kernel during the inference phase through reparameterization technology.

4. The method according to claim 1, characterized in that, In step S2, the AIFI-ASSA attention module adopts a dual-branch self-attention structure: the first branch filters low-similarity noise interactions through sparse self-attention with squared ReLU, and the second branch retains standard softmax dense self-attention. The two branches are fused using learnable adaptive weights.

5. The method according to claim 1, characterized in that, In step S2, the MEFPN feature fusion network adopts a dual-path structure of bottom-up and top-down, integrates four scale feature maps, assigns adaptive weights to features of different scales through a fast normalization fusion module, and includes a gConvC3 module with a gating mechanism.

6. The method according to claim 1, characterized in that, In step S2, the PIOU loss function normalizes the absolute distance deviation between the predicted box and the ground truth box into a relative proportional deviation through an adaptive penalty factor based on the target size, and designs a gradient adjustment function based on the anchor box quality to differentiate samples of different difficulty.

7. The method according to claim 1, characterized in that, In step S3, when training the PAM-DETR model using the preprocessed dataset, multi-scale features of cherry images are extracted through the backbone network, and then the multi-scale features are enhanced and fused through the neck network. The decoder then generates prediction results of cherry ripeness category and bounding box. The difference between the prediction results and the real annotations is continuously optimized according to the loss function, so as to train the model to obtain weight parameters that are closer to the real distribution.

8. The method according to claim 1, characterized in that, In step S4, orchard images from single images, video streams, or real-time camera footage are uniformly adjusted to a predetermined size and then input into the PAM-DETR model, outputting the bounding box coordinates, maturity category, and corresponding confidence level for each cherry target.

Images