Unmanned aerial vehicle aerial photography bergamot pear fruit dropping detection method, system and equipment and medium

Abstract

The invention discloses an unmanned aerial vehicle aerial photography bergamot pear fruit drop detection method, system and device and a medium, and relates to the technical field of fruit drop detection, and the method comprises the steps: collecting a bergamot pear fruit drop image aerial photography by an unmanned aerial vehicle, inputting the bergamot pear fruit drop image into a pre-trained improved YOLOv8 model, obtaining a feature map of a corresponding scale through a backbone network formed by CAS-ViT, and carrying out the detection of the fruit drop of the bergamot pear fruit drop image; image pixel relation capturing is carried out on the feature map of the corresponding scale through an EMA module, a final feature map is output, key area positioning is carried out on the final feature map through a detection head, and a bergamot pear fruit drop detection result is generated. The improved model provided by the invention is superior to an original model in key indexes such as accuracy, and the detection effect of the dropped fruits is improved.

Description

Technical Field

[0001] This invention relates to the field of fruit drop detection technology, and in particular to a method, system, equipment and medium for detecting fruit drop in fragrant pears using drone aerial photography. Background Technology

[0002] In recent years, the continuous expansion of the planting area of ​​fragrant pears and the frequent occurrence of natural disasters (such as hail and windstorms) have significantly impacted the yield and quality of fragrant pears. After a disaster, how to quickly and accurately obtain information on fallen fruit in fragrant pear orchards is a key need for agricultural insurance companies and fruit farmers in post-disaster claims and loss assessment.

[0003] Currently, orchard damage assessment mainly relies on manual experience for visual inspection and yield estimation. This method is time-consuming and labor-intensive, heavily influenced by subjective factors, and its accuracy and timeliness cannot meet the needs of large-scale disaster assessment, making it difficult to achieve objective and automated fruit detection and counting. Therefore, there is an urgent need for an efficient and automated technology for detecting and counting fallen pear fruit to support disaster loss assessment and insurance claims.

[0004] In recent years, unmanned aerial vehicle (UAV) remote sensing technology has been widely used in agricultural monitoring due to its high spatial resolution, operational flexibility, and low cost. Compared with satellite remote sensing, UAV remote sensing can achieve high-precision image acquisition at a smaller scale, making it more suitable for orchard-level crop monitoring and yield assessment. However, for automatic fruit detection in UAV images, current research mainly employs deep learning-based target detection algorithms, such as Faster R-CNN, YOLOv3, YOLOv5, and YOLOv8. Related research has achieved some results on crops such as apples, citrus, watermelons, and jujubes, but most studies focus on the identification and counting of normally growing fruits.

[0005] Most existing fruit detection models are designed for normal growing environments and lack robust designs for post-disaster environments in orchards (such as hail damage, leaf cover, uneven lighting, etc.). From the perspective of drones, the target size of pears is small and the density is high, and existing models are not accurate enough in detecting small targets. When the background is complex and the fruit is similar in color to the ground weeds or tree trunks, traditional models are prone to false detection or false detection. At present, there is a lack of dedicated models and datasets for detecting and counting fallen pears after disasters, which makes it difficult to support practical application scenarios such as agricultural insurance.

[0006] Therefore, existing technologies cannot effectively solve the problems of difficulty in identifying small targets, low detection accuracy under shading conditions, and large counting errors in the detection of fallen pear fruit after disasters. This results in low efficiency and insufficient accuracy in disaster assessment, making it difficult to meet the needs of agricultural insurance loss assessment and rapid post-disaster evaluation. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the prior art by providing a method, system, equipment, and medium for detecting fallen pears using drone aerial photography, thereby solving the problems in the prior art.

[0008] The present invention specifically provides the following technical solution: A method for detecting fallen pear fruit using drone aerial photography, comprising the following steps: Collect aerial images of fallen pears taken by drone; The images of fallen pears are input into a pre-trained and improved YOLOv8 model. The improved YOLOv8 model is based on the original YOLOv8 model, with the backbone network CSPDarkNet replaced by CAS-ViT and the detection head embedded multi-scale attention mechanism EMA module. The CAS-ViT contains multiple cascaded coding units, and multiple sequentially connected coding blocks are stacked in each coding unit. Features of fallen pear images are extracted and input into coding units. In each coding unit, the feature map output by the previous coding unit is received, and local features are extracted block by block in sequence through multiple coding blocks of this coding unit, as well as the association model with other regions. The final feature map is downsampled through the coding layer to obtain the feature map at the corresponding scale of the coding unit, which is used as the input of the next coding unit. The feature map output by the final encoding unit is divided into multiple sub-features in the cross-channel dimension direction by the EMA module, and attention weights of different branches are extracted for each sub-feature in parallel. Cross-space learning is performed on the outputs of different branches to encode global spatial information, and matrix dot product calculation is performed on multiple attention weights to obtain multiple spatial attention maps. Multiple spatial attention maps are fused, and the input feature map is weighted position by position using the fusion result to obtain an output feature map that encodes global spatial information. The detection head is then used to locate key regions in the output feature map to generate pear fruit drop detection results.

[0009] Preferably, the step of performing cross-space learning on the outputs of different branches to encode global spatial information specifically involves: Set up two 1×1 branches and one 3×3 branch; In the two 1×1 branches, a one-dimensional global average pooling operation is used to encode the two spatial direction channels, and after factoring the output into two vectors, two non-linear Sigmoid functions are used to fit the two-dimensional bidirectional distribution of the linear convolution. In the 3×3 branch, only one 3×3 convolution kernel is stacked to capture local channel interactions; Global spatial information is encoded in the outputs of 1×1 and 3×3 branches using a cross-spatial learning method.

[0010] Preferably, the method of encoding global spatial information in the outputs of the 1×1 and 3×3 branches using a cross-spatial learning approach specifically involves: In the 1×1 branch, the channel dimensions are compressed and redistributed through pointwise convolution; In the 3×3 branch, spatial modeling is performed on the local neighborhood to preserve fine-grained structural and texture information; Flatten the outputs of the two branches in the spatial dimension, and construct the correlation matrix between any two spatial locations through matrix dot product to obtain the attention weights across space; The attention weights are used to perform position-wise weighting and fusion of the features of the 1×1 and 3×3 branches to obtain an output feature map that simultaneously contains global spatial information and local detail representation.

[0011] Preferably, before inputting the image of the fallen pear into the pre-trained and improved YOLOv8 model, the method further includes: The improved YOLOv8 model is optimized and trained using the shape intersection-union ratio (SIoU) loss function to obtain the pre-trained improved YOLOv8 model. The SIoU loss function includes: angle loss Λ, distance loss ∆, shape loss Ω, and intersection-union ratio loss U; its specific expression is as follows: .

[0012] Preferably, the angle loss Λ specifically refers to: The angle loss is obtained by comparing the angle output by the improved YOLOv8 model with the true angle; the specific expression is: ; ; ; ; in, The distance between the center points of the ground truth bounding box and the predicted bounding box. c h The height difference between the centers of the ground truth bounding box and the predicted bounding box. for b and b gt The angle between the line connecting two points and the horizontal line, sin( () is the opposite side of a right triangle compared to its hypotenuse. , ) represents the center coordinates of the true bounding box. , () represents the coordinates of the center point of the prediction box.

[0013] Preferably, the distance loss ∆ is specifically: The distance loss ∆ is obtained by measuring the distance between the center points of the predicted bounding box and the ground truth bounding box using the improved YOLOv8 model. The specific expression is as follows: ; ; Where ∆ represents the distance loss. c wb , c hb They are respectively b Dot and b gt The difference between the x-coordinate and y-coordinate of a point. For the "similarity score" of the x-direction offset, This is a "similarity score" for the y-direction offset. , () represents the coordinates of the center point of the prediction box. The magnification factor for angle control, ( , () represents the relative offset of the center point in the x and y directions.

[0014] Preferably, the shape loss Ω specifically refers to: The shape loss Ω is obtained by comparing the shape similarity between the target region output by the model and the real target. The specific expression is as follows: ; ; ; in, w , h To predict the width and height of the bounding box, w gt , h gt The width and height of the actual bounding box. Controlling the degree of attention paid to shape loss, , These represent the relative differences between the predicted bounding box width and height and the actual bounding box width and height, respectively.

[0015] This invention provides a drone-based aerial photography system for detecting fallen pear fruit, comprising: The image acquisition module is used to acquire aerial images of fallen pears taken by drones. The model input module is used to input the image of the fallen pear into the pre-trained improved YOLOv8 model; wherein the improved YOLOv8 model is based on the original YOLOv8 model, the backbone network CSPDarkNet is replaced with CAS-ViT, and the detection head embedding multi-scale attention mechanism EMA module is used. The CAS-ViT contains multiple cascaded coding units, and multiple sequentially connected coding blocks are stacked in each coding unit. The encoding module is used to extract features from the image of fallen pears and input them into the encoding unit. In each encoding unit, it receives the feature map output by the previous encoding unit and performs local feature extraction block by block in sequence through multiple encoding blocks of this encoding unit, as well as association modeling with other regions. The final feature map is downsampled through the encoding layer to obtain the feature map at the corresponding scale of the encoding unit, which is used as the input of the next encoding unit. The attention module is used to divide the feature map output by the final encoding unit into multiple sub-features in the cross-channel dimension direction through the EMA module, extract attention weights for different branches of each sub-feature in parallel, learn across space from the outputs of different branches, encode global spatial information, and calculate matrix dot product of multiple attention weights to obtain multiple spatial attention maps. The detection module is used to fuse multiple spatial attention maps and weight the input feature map position by position using the fusion result to obtain an output feature map that encodes global spatial information. The detection head is used to locate key regions in the output feature map to generate the pear fruit drop detection result.

[0016] The present invention provides a computer device, including a memory and a processor. The memory stores a program, and when the program is executed by the processor, the processor performs the steps of the above-described method for detecting fallen pears by drone aerial photography.

[0017] The present invention provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method for detecting fallen pears by drone aerial photography.

[0018] Compared with the prior art, the present invention has the following significant advantages: This invention proposes a CES-YOLOv8 model suitable for detecting hail-damaged pear fruit drop in unstructured environments. By inputting drone-captured images into an improved YOLOv8 model, each coding unit in the backbone network receives the feature map from the previous coding unit. Local feature extraction is then performed block-by-block sequentially through multiple coding blocks within the unit, along with correlation modeling with other regions, to obtain the feature map at the corresponding scale of that coding unit. This map serves as the input for the next coding unit. This approach overcomes the limitation of traditional convolution focusing only on local receptive fields, improving the model's ability to perceive global information. In feature extraction, the model integrates global information to correlate local features with other regions in the image. The model captures image feature information more precisely and comprehensively. It performs cross-channel feature map partitioning through the EMA module and parallel cross-spatial learning to obtain multiple spatial attention maps. Through EMA processing, the performance of the output detection head is improved, solving the problem of crowded and overlapping targets in the image and enhancing the capture of small target features. Finally, by fusing multiple spatial attention maps, an output feature map encoding global spatial information is obtained. Finally, region localization is performed to obtain the detection result of fallen pear fruit. The improved model is used to process the original image accordingly, reducing the difficulty of detection due to occlusion of branches and leaves of fruit-bearing trees to a controllable range, and ultimately achieving a high recognition accuracy. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of fruit drop after a disaster in an embodiment of the present invention; Figure 2 This is an overall technical flow diagram in an embodiment of the present invention; Figure 3 This is an example diagram of data augmentation samples in an embodiment of the present invention; wherein, Figure 3 (a) is the original image. Figure 3 (b) is an image rotated 45°. Figure 3 (c) is a horizontally flipped image. Figure 3 (d) is a Gaussian blurred image. Figure 3 (e) is the HSV transformed image. Figure 3 (f) is the salt and pepper noise image; Figure 4 This is a diagram of the CES-YOLOv8 network architecture in an embodiment of the present invention; Figure 5 This is a diagram of the convolutional additive self-attention structure in an embodiment of the present invention; Figure 6 This is the overall architecture of the Cas-ViT network in this embodiment of the invention, wherein the framed part is the architecture diagram of the Ni stacked block of the coding layer of the Cas-ViT backbone network; Figure 7 This is a structural diagram of the EMA module in an embodiment of the present invention; Figure 8 This is a schematic diagram of SIoU calculation parameters in an embodiment of the present invention; Figure 9 This is a schematic diagram of the detection and counting results in an embodiment of the present invention; wherein, Figure 9 of (a), Figure 9 (b) Figure 9 (c) Figure 9 (d) Figure 9 (e) and Figure 9 (f) represents the images under different detection and counting conditions; Figure 10 This is a visualization of the heatmaps for detecting fragrant pear fruit using YOLOv8 and CES-YOLOv8 in this embodiment of the invention; wherein, Figure 10 (a1) Figure 10 (a2) Figure 10 (a3) and Figure 10 (a4) represents different original aerial images. Figure 10 (b1) Figure 10 (b2) Figure 10 (b3) and Figure 10 (b4) represents the images after YOLOv8 processing of different original images. Figure 10 (c1) Figure 10 (c2) Figure 10 (c3) and Figure 10 (c4) represents the images after CES-YOLOv8 processing of different original images; Figure 11 A flowchart illustrating a method for detecting fallen pear fruit using drone aerial photography, provided as an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] A review of existing technologies reveals the following problems: Insufficient small target detection capability: Pear fruits in UAV imagery are small in size and high in density. General models such as YOLOv8n are insufficient in multi-scale feature fusion, resulting in a significant decrease in the recognition rate of small targets.

[0022] Reason: The original model has too many downsampling layers, resulting in the loss of shallow feature information, making it difficult for the model to capture the detailed features of the falling fruit.

[0023] Background interference leads to false detections: After hail disasters, the ground is often covered with a large amount of weeds, broken leaves, fruit peels, and shadows, the colors of which are similar to fallen fruit. Existing detection models do not impose feature constraints on complex post-disaster backgrounds, making it easy to misidentify non-target areas as fruit.

[0024] Reason: The general model does not introduce background suppression or channel attention mechanisms, resulting in insufficient feature discrimination.

[0025] Insufficient model generalization and robustness: Existing solutions are mainly trained based on manual photography or conventional orchard datasets, without optimizing the network structure or data augmentation strategies for post-disaster environments (uneven lighting, severe shading, etc.), resulting in unstable model performance in complex scenarios.

[0026] Reason: The training samples are too limited, and the model is less adaptable to extreme samples.

[0027] Detection efficiency is limited: Although some methods can improve accuracy, the large number of model parameters and high computational complexity make it difficult to achieve real-time inference on drones or mobile devices.

[0028] Reason: The improved model did not balance the need for lightweight design with the need for detection accuracy.

[0029] Therefore, although existing technical solutions have made some progress in fruit detection tasks, they still have technical defects such as low detection accuracy, insufficient robustness, and low computing efficiency when dealing with the problem of automatic detection and counting of fallen pears in complex post-disaster environments, making it difficult to meet the application needs of rapid loss assessment in agricultural insurance after disasters.

[0030] Currently, visual recognition research has been conducted on various fruit detection methods. However, for pears, current research only focuses on the simple identification of fruits in the canopy of normally growing trees. Research on target detection and counting of pears after hail damage is limited. Furthermore, existing results still have room for improvement in accuracy and positioning precision when dealing with green fruits and complex background environments, such as those obscured by ground weeds and tree trunks. Addressing the challenges of large pear orchard areas, small and dense fruit targets from a drone's perspective, and obstruction by hail-damaged leaves, this invention proposes a pear fruit drop detection and counting method based on drone imagery and CES-YOLOv8, enabling rapid detection and counting of pear fruit drop caused by hail damage. This invention provides a pear fruit drop detection and counting method based on drone imagery and an improved YOLOv8 model (CES-YOLOv8). By improving the feature fusion structure and lightweight detection head design, the accuracy and robustness of pear fruit drop detection are enhanced, enabling rapid identification and quantity counting of fallen fruit after disasters.

[0031] like Figure 4 and Figure 11 As shown in the figure, this embodiment of a method for detecting fallen pear fruit using drone aerial photography includes the following steps: Step S1: Collect aerial images of fallen pears taken by drone.

[0032] The drone used in this embodiment of the invention is a DJI Mavic 3M drone (DJI Technology Co., Ltd., Shenzhen, China), equipped with a Hasselblad 4 / 3 CMOS RGB aerial camera. This camera has a field of view (FOV) of 84°, a focal length of 24 mm, an aperture range of f / 2.8 to f / 11, an imaging focal length range of 1 m to infinity, and an image resolution of 5268×3956 pixels (JPEG format).

[0033] The drone data collection period was from July 20th to July 22nd, 2024, and the filming location was a hail-stricken area in a pear orchard in Xinjiang Uygur Autonomous Region. During the filming period, the weather was mostly cloudy with good lighting conditions and a solar altitude angle greater than 50° (from 11:00 to 15:00) to ensure uniform image brightness and minimize shadows. Flight parameters were set as follows: flight altitude: 15m; flight speed: 1.5 m / s; lens attitude: perpendicular to the ground; forward overlap: 85%; lateral overlap: 75%.

[0034] Approximately 650 image data points were obtained after acquisition. The original images were then orthophoto stitched using Pix4D Mapper software (version 4.4.12, Pix4DSA, Switzerland) to obtain an orthophoto image with a resolution of 35755 px × 43860 px (see [link to Pix4D Mapper software]). Figure 1 ).

[0035] Image preprocessing steps include: (1) Generate dense point cloud data using Pix4D's built-in algorithm; (2) Construct a regional digital elevation model (DEM); (3) Generate high-resolution orthophotos for subsequent detection and labeling of fallen pear fruit.

[0036] This step ensures the geometric accuracy and radiometric consistency of the images, providing high-quality raw data input for subsequent model training.

[0037] II. Construction of a dataset for pear fruit drop caused by hail disaster: To train the pear fruit drop detection and counting model of this invention, a pear fruit drop sample dataset was constructed based on hail disasters. The production process includes four stages: data labeling, data cropping, data partitioning, and data augmentation.

[0038] Data annotation: Before model training, the fallen pears in the orthophotos need to be manually labeled. This invention uses the Labelme annotation software (https: / / github.com / wkentaro / labelme) to manually select the targets, labeling them as "pear". Each label box is strictly limited to a single fallen pear to ensure annotation accuracy and sample consistency.

[0039] Data cropping: Due to the large size of UAV aerial images, which are unsuitable for direct input into the detection network, the labeled images were uniformly cropped into 512×512 pixel image blocks, generating a total of 5852 sample images. This processing effectively reduces the difference in model input size and improves training convergence speed.

[0040] Data partitioning: The sample data was divided into training, validation, and test sets in a 7:1.5:1.5 ratio. This partitioning strategy maintains data balance while ensuring the independence of model validation.

[0041] Data Augmentation: Considering the complexities that UAV images may encounter under different shooting conditions, such as uneven lighting, noise interference, and blurring, this invention performs diverse augmentations on the training and validation sets. The augmentation methods employed include: image rotation (45°); horizontal flipping; Gaussian blur; salt-and-pepper noise reduction; and HSV color space transformation, specifically as follows: Figure 3 As shown.

[0042] The data augmentation process described above significantly improved the model's generalization ability and robustness in complex natural scenes, enabling the model to adapt to fruit drop recognition tasks under different lighting, background, and occlusion conditions. The distribution statistics of the sample sets before and after augmentation are shown in Table 1.

[0043] Table 1. Sample size distribution of images of fallen pears Through the above-described dataset construction process, this invention has created a representative dataset of post-disaster pear fruit drop images with abundant complex background samples, providing reliable training support for subsequent detection and counting based on the CES-YOLOv8 model.

[0044] The training and testing of the model were both conducted on the Ubuntu 20.04 focal operating system. The deep learning framework used was PyTorch 1.13.1, CUDA version 12.2, Python version 3.8.0, the CPU was an Intel(R) Xeon(R)Gold 5318Y@3.4GHz, and the GPU was an NVIDIA A40. All comparison algorithms were run under the same environment. The network training parameters were set according to the experimental conditions, as shown in Table 2.

[0045] Table 2 Training Parameter Settings To accurately and objectively evaluate the model's performance, this study uses precision (P), recall (R), mean average precision (mAP), and detection speed to assess the model's accuracy and real-time detection performance. mAP@0.5 represents the mAP at 50% IoU threshold. mAP@0.5:0.95 represents the mAP at 50%–95% IoU threshold. The specific formulas for each metric are as follows.

[0046] ; ; ; ; ; Where TP represents the number of correctly classified positive samples, FP represents the number of samples that were actually negative but were incorrectly classified as positive samples, FN represents the number of incorrectly classified positive samples, and N is the number of detected categories.

[0047] Step S2: Input the image of fallen pears into the pre-trained improved YOLOv8 model; the improved YOLOv8 model is based on the original YOLOv8 model, the backbone network CSPDarkNet is replaced with CAS-ViT, and the multi-scale attention mechanism EMA module is embedded in the detection head. CAS-ViT contains multiple cascaded coding units, and multiple sequentially connected coding blocks are stacked in each coding unit.

[0048] To improve the accuracy of pear fruit drop detection and counting in hail disasters, an improved pear fruit drop target detection and counting model, CES-YOLOv8, is proposed by improving the backbone and head parts of YOLOv8 and introducing a CAS-ViT network structure, an EAM module, and a SIoU loss function. First, the backbone network CSPDarkNet is replaced with CAS-ViT, breaking the limitation of traditional convolution focusing only on local receptive fields and improving the model's ability to perceive global information. Second, to improve the performance of the output detection head and address the problem of crowded and overlapping targets in the image, an efficient multi-scale attention mechanism (EMA) is embedded before the detection head to enhance the capture of small target features. Finally, to alleviate false detections and localization errors caused by overlapping pears, SIoU is used instead of the original CIoU as the bounding box loss function, allowing the network to pay attention to the difference between the orientation of the predicted box and the ground truth box during model training, thus fusing orientation information to improve the model's convergence speed and efficiency. The CES-YOLOv8 network structure is as follows. Figure 1 As shown.

[0049] Step S3: Extract the features of the fallen pear image and input them into the coding unit. In each coding unit, the network receives the feature map output by the previous coding unit and inputs it into multiple coding blocks stacked in this coding unit in sequence. Local feature extraction and association modeling with other regions are performed through multiple coding blocks in this coding unit in sequence (i.e., the output of the previous block is used as the input of the next block). The final feature map obtained by the last coding block is downsampled through the coding layer to obtain the feature map of the corresponding scale of the coding unit, which is used as the input of the next coding unit.

[0050] The ViT (Vision Transformer) model, proposed by Google in 2020, applies the Transformer to image classification. Vision Transformers (ViTs) utilize a self-attention mechanism to extend the model dependencies of images, automatically learning the correlations between different regions in an image. The ViT model transforms image classification into a sequence processing task by segmenting the input into a series of equally sized image patches, each representing a local image region. An embedding layer encodes each patch, thus transforming the image classification task into a sequence processing task. This approach not only enables the model to capture spatial relationships within images but also allows the Transformer model to work directly on image data without relying on the inductive bias of convolutional neural networks. While the ViT model performs well in image classification, it also has limitations, such as high computational complexity and the difficulty of simultaneously achieving accuracy, efficiency, and ease of deployment in mobile devices and real-time applications.

[0051] To address the aforementioned issues, Zhang et al. proposed the Convolutional Additive Label Mixer (CATM) to reduce computational overhead and developed a lightweight network, the Convolutional Additive Self-attention Visual Transformer (CAS-ViT), such as... Figure 5 As shown, CAS-ViT combines the advantages of Convolutional Neural Networks (CNN) and Vision Transformers (ViT) by introducing a convolutional additive self-attention mechanism. Its structure diagram is shown below. Figure 2 As shown, in feature extraction, the global information integration capability is used to associate local features with other regions in the image, enabling the model to capture image feature information more precisely and comprehensively. Secondly, compared with the traditional self-attention mechanism, the convolutional additive self-attention mechanism reduces redundant data computation by rationally allocating the computational resources of convolution and self-attention, and performs preliminary feature extraction on local features, while effectively constraining and guiding the scope of subsequent self-attention mechanisms.

[0052] The Cas-ViT backbone network consists of a coding layer with four coding units, and its network architecture is as follows: Figure 6 As shown above, each encoding layer uses Patch Emdedding for two downsampling operations to obtain a feature map of size [size missing]. , , , The feature map, where C iLet i ∈ {1, 2, 3, 4} represent the channels of the feature map. Each coding unit contains N channels. i Stacked blocks, such as Figure 6 As shown below, it consists of an integration subnetwork, CATM, and MLP. The integration subnetwork is composed of three deep convolutional layers activated by ReLU, which fuses the inputs of the convolutional layers and the self-attention layer to alleviate the gradient vanishing problem.

[0053] Step S4: Divide the feature map output by the final encoding unit into multiple sub-features across the channel dimension using the EMA module, extract attention weights for different branches of each sub-feature in parallel, perform cross-space learning on the outputs of different branches, encode global spatial information, and perform matrix dot product calculation on multiple attention weights to obtain multiple spatial attention maps.

[0054] In step S4, to improve the performance of the output detection head and address the issues of small pear targets in drone-captured images, resulting in crowding and occlusion, an efficient multi-scale attention mechanism (EMA) is embedded before the detection head to enhance the model's attention to and capture of pear target features. When detecting and locating targets, the detection head may lose some detailed information, leading to missed detections and false detections. The embedded EMA uses parallel subnetworks to evenly distribute spatial semantic features in each feature group. To achieve rich feature aggregation, a cross-spatial learning method is used to aggregate multi-scale spatial structure information, avoiding performance degradation caused by more complex sequential processing and deep convolutions. This effectively captures pixel-level attention feature values ​​of the pear and establishes dimensional dependencies. The learned attention weights are used to enhance key regions in sub-features, generating accurate pear target localization information.

[0055] EMA structure as Figure 7As shown, for the input feature map, EMA divides it into G sub-features across the channel dimension, learning different semantic features. Each sub-feature has a shape of C / / G×H×W, where C is the number of input channels, and H and W represent the spatial height and width of the input feature, respectively. Subsequently, to aggregate multi-scale spatial structure information, 1×1 and 3×3 branches are placed in parallel to extract attention weights for the grouped feature maps. In the two 1×1 branches, one-dimensional global average pooling is used to encode the channels in both spatial directions, where X Avg Pool is the average pooling along the horizontal direction, and Y Avg Pool is the average pooling along the vertical direction. After factoring the output into two vectors, two non-linear sigmoid functions are used to fit the two-dimensional bidirectional distribution of the linear convolution. In the 3×3 branch, only one 3×3 convolution kernel is stacked to capture local channel interactions and expand the feature space. Then, a cross-spatial learning method is used to encode global spatial information in the outputs of the 1×1 and 3×3 branches. For efficient computation, the sigmoid function is used again to fit the linear transformation, and the two generated spatial attention weights are multiplied by a matrix to obtain two spatial attention maps. The output feature map within each group is calculated as a set of the two generated spatial attention weights, capturing the relationship between each pixel in the image and its surrounding pixels, highlighting the semantic and structural information of the entire image. The feature maps of the outputs from multiple branches are mapped into a set of spatial attention maps to characterize the association between each spatial location and the global context.

[0056] Global spatial information is encoded in the outputs of the 1×1 and 3×3 branches using a cross-spatial learning method, specifically as follows: The EMA module first feeds the input features into 1×1 and 3×3 convolutional branches respectively. The 1×1 branch compresses and redistributes the channel dimension through pointwise convolution, highlighting the global response of different semantic channels; the 3×3 branch focuses on spatial modeling of local neighborhoods, preserving fine-grained structural and texture information. Then, the outputs of the two branches are flattened in the spatial dimension, and a correlation matrix between any two spatial locations is constructed through matrix dot products to obtain cross-spatial attention weights, thus explicitly encoding the global spatial relationships of the entire feature map. Finally, these attention weights are used to perform positionally weighted and fused features from the 1×1 and 3×3 branches (e.g., weighted summation or concatenation followed by 1×1 convolution) to obtain an output feature map that simultaneously contains global spatial information and local detail representation.

[0057] Step S5: Fuse multiple spatial attention maps, and use the fusion result to weight the input feature map position by position to obtain an output feature map that encodes global spatial information. Use the detection head to locate key regions in the output feature map and generate the pear fruit drop detection result.

[0058] Before inputting the images of fallen pears into the pre-trained and improved YOLOv8 model, the following steps are also included: In the YOLOv8 algorithm, the bounding box regression loss uses CIoU loss. CIoU mainly focuses on bounding box regression metrics such as distance, overlap area, and aspect ratio between the predicted and ground truth boxes, but it does not consider the mismatch in direction between the predicted and ground truth boxes. This deficiency leads to slow model convergence and low efficiency because the predicted boxes may "wander" during training, resulting in a worse model. To address these issues, this study replaces the original loss function with the shape intersection over union (SIoU) loss function. The improved YOLOv8 model is optimized and trained using the SIoU loss function to obtain a pre-trained improved YOLOv8 model; the SIoU loss function includes: angle loss Λ, distance loss ∆, shape loss Ω, and intersection over union loss U.

[0059] 10) The angle loss Λ is the difference between the angle output by the model and the actual angle. The specific calculation formula is as follows.

[0060] ; ; ; ; In the formula, The distance between the center points of the ground truth bounding box and the predicted bounding box, such as Figure 8 As shown, ch is the height difference between the centers of the ground truth bounding box and the predicted bounding box. Let sin(bgt) be the angle between the line connecting points b and bgt and the horizontal line. ( ) is the ratio of the opposite side to the hypotenuse of a right triangle. , () represents the coordinates of the center of the true bounding box. , () represents the coordinates of the center point of the prediction box.

[0061] The distance loss ∆ is the distance between the center point of the model's predicted bounding box and the center point of the ground truth bounding box. Its penalty cost is positively correlated with the angle loss. The formula for calculating the distance loss ∆ is as follows.

[0062] ; ; In the formula, ∆ represents the distance loss, and cwb and chb are the differences between the x and y coordinates of point b and point bgt, respectively. For the "similarity score" of the x-direction offset, This is a "similarity score" for the y-direction offset. , () represents the coordinates of the center point of the prediction box. The magnification factor for angle control, ( , () represents the relative offset of the center point in the x and y directions.

[0063] The shape loss Ω represents the shape similarity between the target region output by the model and the real target. The specific calculation formula is as follows.

[0064] ; ; ; In the formula, w and h are the width and height of the prediction box. w gt , h gt The width and height of the actual bounding box. Controlling the degree of attention paid to shape loss, The parameter range is between 2 and 6. , These represent the relative differences between the predicted bounding box width and height and the actual bounding box width and height, respectively.

[0065] The Intersection over Union (IoU) is the ratio between the intersection and union of the model's output bounding box and the ground truth bounding box. The specific calculation formula is shown below.

[0066] ; In summary, the Lbox definition of the SIoU bounding box loss function is as follows.

[0067] .

[0068] Experimental results: To address the challenges of accurate detection and counting of fallen pears after hailstorms in complex natural environments, given their small size and susceptibility to obstruction by branches and leaves, this study proposes a CES-YOLOv8 model for detecting fallen pears after hailstorms in unstructured environments. This model, based on YOLOv8, enables accurate detection and counting of fallen pears after hailstorms in natural settings. The counting images are shown below. Figure 9 As shown.

[0069] CES-YOLOv8 achieved a detection accuracy of 89.9% and a recall rate of 86.4% for pears, an F1 score of 88.1%, and a mAP@0.5 of 93.9%. Comparative experiments showed that the CES-YOLOv8 network improved the accuracy, recall, and mean precision by 2.5%, 1.1%, and 0.5% respectively compared to YOLOv8. Furthermore, CES-YOLOv8 achieved high recognition accuracy even under complex conditions such as dense fruit distribution, weed obstruction, and overlapping fruit.

[0070] To further verify the superiority of CES-YOLOv8 on the pear fruit drop dataset, this study employs the Grad-CAM (Gradient-weighted class activation mapping) heatmap visualization method to visualize key regions of the images from both the YOLOv8 and CES-YOLOv8 models. Examples of heatmap visualizations for both models before and after the improvement are shown below. Figure 10 As shown, the areas of greater interest are highlighted in the Grad-CAM map.

[0071] This invention proposes a drone aerial photography system for detecting fallen pear fruit, comprising: an image acquisition module, a model input module, an encoding module, an attention module, a detection module, and a detection module.

[0072] The system comprises several modules: an image acquisition module for acquiring aerial images of fallen pears taken by a drone; a model input module for inputting these images into a pre-trained, improved YOLOv8 model; and an improved YOLOv8 model based on the original YOLOv8 model, replacing the backbone network CSPDarkNet with CAS-ViT and embedding a multi-scale attention mechanism (EMA) module before the detector head. CAS-ViT contains multiple cascaded coding units, each with stacked sequentially connected coding blocks. An encoding module extracts features from the fallen pear images and inputs them into the coding units. Within each coding unit, it receives the feature map output from the previous coding unit and performs local feature extraction block by block sequentially through multiple coding blocks within the unit, as well as constructing associations with other regions. The module performs downsampling on the final feature map through the encoding layer to obtain the feature map at the corresponding scale of the encoding unit, which serves as the input to the next encoding unit. The attention module divides the feature map output by the final encoding unit into multiple sub-features across the channel dimension through the EMA module, and extracts attention weights for different branches of each sub-feature in parallel. It performs cross-space learning on the outputs of different branches to encode global spatial information, and performs matrix dot product calculation on multiple attention weights to obtain multiple spatial attention maps. The detection module fuses multiple spatial attention maps and uses the fusion result to perform position-wise weighting on the input feature map to obtain an output feature map that encodes global spatial information. The detection head then uses the output feature map to locate key regions and generate the pear fruit drop detection result.

[0073] The present invention also provides a computer device, including a memory and a processor. The memory stores a program, and when the program is executed by the processor, the processor performs the steps of a method for detecting fallen pears by drone aerial photography.

[0074] According to the disclosed embodiments, the computer device can communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth communication, etc.) or with any device that enables the computing device to communicate with one or more other computing devices (e.g., router, demodulator, etc.).

[0075] The present invention also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of a method for detecting fallen pears by drone aerial photography.

[0076] According to the disclosed embodiments, the storage medium can be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, the storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0077] The above description, in conjunction with specific preferred embodiments, provides a more detailed explanation of the present invention. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for detecting fruit drop of a pear by using a UAV, characterized in that, The method comprises the steps of: collecting a UAV aerial shot pear fruit drop image; inputting the pear fruit drop image into an improved YOLOv8 model after pre-training; wherein the improved YOLOv8 model is based on the original YOLOv8 model, the main network CSPDarkNet is replaced by CAS-ViT, and a multi-scale attention mechanism EMA module is embedded in front of the detection head, the CAS-ViT includes a plurality of cascaded encoding units, and a plurality of encoding blocks connected in sequence are stacked in each encoding unit; extracting the pear fruit drop image features and inputting them into the encoding unit, in each encoding unit, receiving the feature map output by the previous encoding unit, and sequentially extracting local features and modeling the association with other regions through a plurality of encoding blocks of the encoding unit, and the final feature map is down-sampled through the encoding layer to obtain the feature map corresponding to the scale of the encoding unit as the input of the next encoding unit; dividing the feature map output by the final encoding unit into a plurality of sub-features in the cross-channel dimension direction through the EMA module, extracting attention weights of different branches in parallel for each sub-feature, learning across space for the outputs of different branches, encoding global spatial information, and performing matrix dot product calculation on a plurality of attention weights to obtain a plurality of spatial attention maps; fuse the plurality of spatial attention maps, and weight the input feature map by position through the fusion result to obtain an output feature map encoding global spatial information, and use the detection head to locate the key area of the output feature map to generate a pear fruit drop detection result. 2.The method of claim 1, wherein, The learning across space for the outputs of different branches, encoding global spatial information, specifically comprises: setting two 1×1 branches and one 3×3 branch; in the two 1×1 branches, a one-dimensional global average pooling operation is used to encode the two spatial direction channels, and after factorizing two vectors from the output, two nonlinear Sigmoid functions are used to fit the two-dimensional two-way distribution of linear convolution; in the 3×3 branch, only one 3×3 convolution kernel is stacked to capture local channel interaction; use a cross-space learning method to encode global spatial information in the outputs of the 1×1 and 3×3 branches. 3.The method of claim 2, wherein, The use of a cross-space learning method to encode global spatial information in the outputs of the 1×1 and 3×3 branches, specifically comprises: in the 1×1 branch, compress and redistribute the channel dimension through point-by-point convolution; in the 3×3 branch, spatial modeling is performed on the local neighborhood to retain fine-grained structure and texture information; flatten the outputs of the two branches in the spatial dimension, construct a correlation matrix between any two spatial positions through matrix dot product, and obtain the attention weight across space; use the attention weight to weight and fuse the features of the 1×1 and 3×3 branches by position, and obtain an output feature map that simultaneously expresses global spatial information and local details. 4.The method of claim 1, wherein, Before inputting the pear fruit drop image into the improved YOLOv8 model after pre-training, it further comprises the steps of: The improved YOLOv8 model is optimized and trained by using a shape intersection-over-union SIoU loss function, to obtain a pre-trained improved YOLOv8 model; wherein the SIoU loss function comprises: an angle loss Λ, a distance loss ∆, a shape loss Ω and an intersection-over-union loss U; and the specific expression is: 。 5. The method of claim 4, wherein the method is used for detecting the fallen fruit of the pear by the UAV. The angle loss Λ is specifically: ​ The angle loss is obtained by the difference between the angle output by the improved YOLOv8 model and the real angle; and the specific expression is: ； ； ； ； wherein, is the distance between the real box and the predicted box center point, c h is the height difference between the real box and the predicted box center, is b and b gt is the angle between the two points and the horizontal line, sin( ) is the opposite side to the hypotenuse of the right triangle, , ) is the real box center coordinate, , ) is the predicted box center point coordinate.

6. The method of claim 4, wherein the method is characterized by, The distance loss ∆ is specifically: The distance loss ∆ is obtained by the distance between the center point of the bounding box predicted by the improved YOLOv8 model and the real bounding box; and the specific expression is: ； ； wherein, is the distance loss, c wb , c hb are the horizontal and vertical coordinates of the b point and b gt difference between the horizontal and vertical coordinates of the "similarity score" for the x-direction offset, "similarity score" for the y-direction offset. , ) are the predicted bounding box center point coordinates. is the angle control magnification factor, , are the relative offsets of the center point in the x and y directions.

7. The method of claim 4, wherein the method is used for detecting the fallen fruit of the pear by the UAV. The shape loss Ω is specifically: ​ The shape loss Ω is obtained by the shape similarity between the target region output by the model and the real target; and the specific expression is: ； ； ； wherein, w , h are the width and height of the predicted box, w gt , h gt are the width and height of the real box, controls the degree of attention to the shape loss, , are the relative differences of the predicted box width and height with respect to the real box width and height, respectively.

8. A UAV aerial shooting pear fruit drop detection system, characterized in that, comprising: An image acquisition module is configured to acquire a pear fruit drop image taken by a UAV; A model input module is configured to input the pear fruit drop image into the pre-trained improved YOLOv8 model; wherein the improved YOLOv8 model is based on an original YOLOv8 model, replaces a backbone network CSPDarkNet with a CAS-ViT, and embeds a multi-scale attention mechanism EMA module in front of a detection head; the CAS-ViT comprises a plurality of cascaded encoding units, and each encoding unit stacks a plurality of encoding blocks connected in sequence; An encoding module is configured to extract features of the pear fruit drop image and input the features into an encoding unit; in each encoding unit, a feature map output by a previous encoding unit is received, local feature extraction is performed on the feature map by a plurality of encoding blocks of the encoding unit in sequence, association modeling with other regions is performed, the final feature map is down-sampled by an encoding layer to obtain a feature map of a corresponding scale of the encoding unit, and the feature map is input into a next encoding unit. An attention module is configured to divide the feature map output by the final encoding unit into a plurality of sub-features in a cross-channel dimension direction by the EMA module, extract attention weights of different branches in parallel for each sub-feature, perform cross-space learning on outputs of different branches to encode global spatial information, and perform matrix point multiplication calculation on the plurality of attention weights to obtain a plurality of spatial attention maps. A detection module is configured to fuse the plurality of spatial attention maps, perform position-by-position weighting on the input feature map based on a fusion result, obtain an output feature map encoding global spatial information, and perform key region positioning on the output feature map by the detection head to generate a pear fruit drop detection result.

9. A computer device, comprising: The computer program is executed by the processor to implement the steps of the pear fruit drop detection method of the UAV taken by the UAV according to any one of claims 1 to 7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the pear fruit drop detection method of the UAV taken by the UAV according to any one of claims 1 to 7.

Images