Method and system for lightweight estimation of yield of horticultural crop fruits
By combining the improved YOLO11 model with an RGB-D camera and depth images, the problems of accuracy and cost in fruit yield estimation were solved, enabling accurate fruit identification and efficient yield detection.
Patent Information
- Application Number
- CN202510130774.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-06
- Publication Date
- 2025-10-21
- Estimated Expiration
- 2045-02-06
AI Technical Summary
Existing fruit yield estimation methods find it difficult to strike a balance between accuracy, efficiency, and cost. The RGB image method has occlusion problems, and the three-dimensional point cloud method is computationally complex and has high equipment costs, making it difficult to promote in large-scale planting scenarios.
A lightweight fruit detection model is adopted, which combines an RGB-D camera and depth images. The YOLO11 model is improved to a weighted loss of DFL distributed focal loss and GIoU regression loss. Combined with region tracking and counting and area-weight regression models, real-time monitoring of fruit quantity and weight is achieved.
It enables accurate identification, classification, and segmentation of fruits in complex environments, improving the accuracy and efficiency of yield detection while reducing equipment and computing costs.
Smart Images
Figure CN120071333B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of precision agriculture technology, and in particular to a lightweight estimation method and system for the fruit yield of fruit-bearing horticultural crops. Background Art
[0002] Fruit yield estimation for fruiting horticultural crops (such as fruits, apples, and oranges) is a crucial research topic in modern agriculture, crucial for optimizing crop management, improving agricultural production efficiency, and achieving precision agriculture. Currently, fruit yield estimation is primarily performed through manual visual inspection. While this method is simple and intuitive, it is inefficient and subject to significant human influence, making it difficult to apply in large-scale crop production scenarios. With the advancement of computer vision and image processing technologies, researchers have proposed fruit yield estimation methods based on RGB images. These methods use cameras to capture fruit images and employ machine learning or deep learning algorithms to detect and count the fruit. However, RGB images only provide two-dimensional information about the fruit and cannot accurately reflect its three-dimensional spatial distribution and occlusion relationships, limiting estimation accuracy. To overcome the limitations of two-dimensional images, some studies have used 3D point cloud data for fruit modeling and estimation. While this method provides more accurate spatial information, it typically requires expensive LiDAR equipment or complex multi-view image stitching techniques, resulting in high cost and computational complexity, making it unsuitable for scenarios requiring high real-time performance.
[0003] Current fruit yield estimation methods struggle to strike a balance between accuracy, efficiency, and cost. RGB image methods suffer from occlusion issues, while 3D point cloud methods are computationally complex and require high equipment costs. Furthermore, existing technologies still have significant room for improvement in terms of equipment lightweighting, algorithm simplification, and cost-effective implementation. Summary of the Invention
[0004] In response to the problems existing in the existing technology, the present invention proposes a lightweight estimation method and system for the fruit yield of fruit-bearing horticultural crops, which can realize accurate identification, classification and segmentation of fruits in complex environments and achieve precise yield detection.
[0005] The above purpose is achieved through the following technical solutions:
[0006] The present invention first provides a lightweight estimation system for fruit yield of fruit-bearing horticultural crops, comprising the following steps:
[0007] S1. Build a lightweight fruit detection model: Using the YOLO11 model as the framework, we introduce the FasterNet module into the residual feature learning C3k2 unit of the YOLO11 model. We also change the Conv function of the YOLO11 model to DSCOnv, and set the regression loss of the YOLO11 model to a weighted combination of the DFL distribution focus loss and the GIoU regression loss. We train the lightweight fruit detection model on a dataset labeled with three different maturity levels.
[0008] S2. Acquire multiple consecutive RGB images and depth images using an RGB-D camera. The multiple consecutive RGB images and depth images are acquired by an image acquisition device by capturing images of fruit of a fruiting horticultural crop in the current yield estimation area, where the fruit is the object to be detected.
[0009] S3. Align the RGB image and depth image obtained in step S2;
[0010] S4. The RGB image obtained in step S3 is sequentially input into the lightweight fruit detection model constructed in step S1 to obtain the fruit target box information and semantic segmentation information output by the lightweight fruit detection model;
[0011] S5. Filter out the fruit outside the set camera-fruit distance threshold based on the fruit target frame information obtained in step S4 and the camera-fruit distance information in the depth image obtained in step S3, thereby obtaining the fruit within the set distance in each frame of the RGB image;
[0012] S6 by the regional tracking counting method in step S5 obtained in each frame RGB image within a set distance tracking fruit counting, obtain the total number of consecutive frames RGB image fruit;
[0013] S7. Obtain the fruit area according to the fruit semantic segmentation information obtained in step S4 and the camera-fruit distance information in the depth image, and obtain the fruit weight according to the established fruit area-weight regression model;
[0014] S8. Display the statistical information of the number of fruits obtained in step S6, the fruit area and the fruit weight obtained in step S7 in a visual interface to monitor the yield in real time.
[0015] Furthermore, the lightweight fruit detection model includes an image input module, a feature extraction module, a feature fusion module and a recognition module; the feature extraction module includes multiple cascaded first neural network units, the feature fusion module includes multiple cascaded second neural network units, and the recognition module includes a weighted loss function of VFL zoom loss function, DFL distributed focus loss function and GIoU regression loss function.
[0016] Furthermore, the area tracking and counting method described in step S6 specifically includes the following sub-steps:
[0017] S6-1. Select multiple consecutive frames of RGB images and depth images symmetrical to the center line as random areas;
[0018] S6-2. Calculate the number of fruits in the random area obtained in step S6-1: The size of the random area is used as the input of the particle swarm optimization algorithm and brought into the Bytetrack tracking algorithm. First, the video is detected: the fruit in the video is detected using a lightweight fruit detection model to obtain the target detection box and the corresponding features; then the target is predicted: the Kalman filter algorithm is used to predict the position and state of the target in the next frame of the video; finally, the target is matched: the Hungarian algorithm is used to optimally match several targets between the two frames before and after the video to obtain the trajectory of the target in the video. The target trajectory that fails to match will be temporarily saved and continue to participate in the subsequent frame prediction matching until the target fails to match for multiple consecutive frames and is considered to have disappeared, and the trajectory is then deleted; the Bytetrack tracking algorithm outputs the number of fruit ids in the random area, that is, the number of fruits in the random area;
[0019] S6-3. The root mean square error between the number of fruits in the random region obtained in step S6-2 and the true value is used as the fitness value of the particle swarm optimization algorithm. The size of the random region corresponding to the minimum convergence fitness value is the optimal region size.
[0020] S6-4. The optimal region size obtained in step S6-3 is used as the region size for yield estimation and is introduced into the Bytetrack tracking algorithm to obtain the number of fruits in the optimal region.
[0021] Furthermore, the fruit area-weight regression model described in step S7 measures the area and weight of 35 large, medium and small tomatoes in three weight ranges of more than 200g, 100-200g, and 0-100g, and establishes a weight-fruit area regression relationship according to the position of the tomato in the image (top / bottom or middle area).
[0022] Furthermore, the camera-fruit distance threshold in step S5 is set according to the crop planting row spacing of the fruit.
[0023] Furthermore, the dataset annotated with three different maturity labels in step S1 is obtained by the following method:
[0024] Get the original data set containing fruits and leaves;
[0025] Determining seed pixels in the original data set, and segmenting a fruit image from the original data set according to the seed pixels;
[0026] Performing image synthesis using the fruit image as an object and the original data set as a background;
[0027] The background and fruit in the image are then segmented using a region-information-based segmentation method, and the segmented fruit image is expanded: seed pixels in the original dataset are manually selected, and pixels near the seed pixels are retrieved. Similar regions are aggregated, and all pixels in the image are traversed using the region-information segmentation method to complete the image segmentation. The segmented fruit image is then used as the "object" and the rest of the original dataset as the "background" for image synthesis. This increases the number of fruit samples in the image, resulting in a dataset with a preset number of fruits.
[0028] After obtaining the dataset, LabelImg annotation software was used to add marking boxes to the fruits of different maturity in the dataset to obtain the annotated dataset. The annotated dataset was divided into training set, test set and validation set according to the preset ratio.
[0029] The present invention also provides a lightweight estimation system for the fruit yield of fruit-bearing horticultural crops, which includes a fruit recognition and positioning processor and a program or instruction that can be run on the fruit recognition and positioning processor. When the program or instruction is executed by the fruit recognition and positioning processor, it implements the lightweight estimation method for the fruit yield of fruit-bearing horticultural crops as described above.
[0030] Furthermore, the system also includes an image acquisition device and a visualization device; the image acquisition device and the visualization device are respectively connected to the fruit recognition and positioning processor;
[0031] The image acquisition device is used to collect a video stream of fruits planted in the area to be estimated in real time, and send the video stream to the fruit recognition and positioning processor so that the fruit recognition and positioning processor can obtain multiple frames of continuous images based on the video stream;
[0032] The visualization device is used to perform real-time display based on the information output by the fruit identification and positioning processor.
[0033] The present invention also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for lightweight estimation of fruit yield of fruit-bearing horticultural crops as described above is implemented.
[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for lightweight estimation of fruit yield of fruit-bearing horticultural crops as described above is implemented.
[0035] The beneficial effects of the present invention compared to the prior art are:
[0036] The present invention obtains the number of fruits based on fruit target frame information and RGB-D information; obtains the weight of the fruit based on fruit segmentation information, RGB-D information and a regression model; and displays the fruit yield (quantity and weight) in a visual interface for real-time monitoring. The lightweight fruit detection model is trained based on a data set labeled with three different maturity labels. The lightweight fruit detection model is based on the YOLO11 model, introduces a FasterNet module into the residual feature learning C3k2 unit of the YOLO11 model, changes the Conv of the YOLO11 model to DSCOnv, and sets the regression loss of the YOLO11 model to a weighted loss of the DFL distribution focus loss and the GIoU regression loss. Therefore, the present invention can realize accurate recognition, classification and segmentation of fruits in complex environments, and realize accurate yield detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a flow chart of a lightweight method for estimating the fruit yield of fruit-bearing horticultural crops provided by the present invention;
[0038] Figure 2 Schematic diagram of the structure of the lightweight fruit detection model provided by the present invention;
[0039] Figure 3 Schematic diagram of the structure of the C3k2-F unit provided by the present invention;
[0040] Figure 4 This is a schematic structural diagram of a lightweight system for estimating the fruit yield of fruit-bearing horticultural crops provided by the present invention;
[0041] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0042] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.
[0043] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Figure 1This is one of the flow charts of a lightweight estimation method for fruit yield of fruit-bearing horticultural crops provided by the present invention. The execution subject of each step in the method can be a fruit yield detection device, which can be implemented by software and / or hardware. The device can be integrated into an electronic device, which can be a terminal device (such as a smart phone, a personal computer, a learning machine, etc.), a server (such as a local server or a cloud server, or a server cluster, etc.), a processor, or a chip, etc. Figure 1 As shown, the method may include the following steps:
[0044] S1. Constructing a lightweight fruit detection model: Using the YOLO11 model as the framework, we introduced the FasterNet module into the residual feature learning C3k2 unit of the YOLO11 model. We also changed the Conv of the YOLO11 model to DSCOnv, and set the regression loss of the YOLO11 model to a weighted loss of the DFL distribution focus loss and the GIoU regression loss. We trained a dataset labeled with three different maturity levels to obtain a lightweight fruit detection model. The details are as follows:
[0045] The YOLO (You Only Look Once) model is an object recognition and positioning algorithm based on deep neural networks. YOLO has the characteristics of fast running speed and suitability for real-time computing.
[0046] FasterNet (Faster Network) is an efficient network architecture that aims to improve inference speed while maintaining excellent performance by simplifying model design and optimizing calculations. Unlike traditional large-scale models, FasterNet adopts a layered design and efficient operator layout, which reduces parameters and computation while ensuring the model's expressiveness and generalization performance. Its core feature is that it greatly improves the efficiency of the model through structural innovations, such as lightweight convolution modules and improved data flow paths. In addition, FasterNet introduces a task-specific adaptive adjustment mechanism that can dynamically optimize computational allocation, allowing the model to run efficiently on various hardware platforms. This design gives FasterNet significant advantages in both inference efficiency and resource utilization, making it suitable for scenarios with high speed and performance requirements.
[0047] VFL (VariFocal Loss) is a dynamically scaled binary cross-entropy loss. It uses an asymmetric weighting of training examples. VFL downweights negative examples that address class imbalance during training, while increasing the weight of generating high-quality positive examples. This focuses training on high-quality positive examples, better addressing class imbalance.
[0048] DFL (Distribution Focal Loss) adjusts sample weights by introducing a focal factor and a distribution parameter. The focal factor is primarily used to alleviate class imbalance by assigning higher weights to rare classes, allowing the model to focus more on their classification accuracy. The distribution parameter, on the other hand, controls the shape of the sample distribution, providing additional weighting when processing difficult samples.
[0049] GIoU (Generalized Intersection over Union) is an improved object detection loss function designed to address the shortcomings of IoU when bounding boxes have no or partial overlap. GIoU builds on IoU by introducing the concept of closure region. By calculating the minimum closure region between the predicted and target boxes, GIoU enhances the loss function's ability to distinguish non-completely overlapping boxes. Compared to IoU, GIoU can more effectively optimize bounding box regression, improving the performance of object detectors in complex scenarios.
[0050] Specifically, the present invention uses the existing YOLO11 model as its foundational framework. First, the FasterNet lightweight module is added to its residual feature learning C3k2 unit, effectively reducing model parameters without compromising model accuracy. Second, the VFL classification loss function better addresses the imbalance in the number of fruit samples, enhancing the reliability of the network model. Finally, the GIoU regression loss combined with the DFL distributed focus loss significantly addresses the difficulty in identifying and locating fruits due to dense fruit growth and mutual occlusion.
[0051] In this example, model training and testing were performed on a workstation equipped with an 11th Gen Intel i9 CPU and an NVIDIA GTX3080Ti GPU, using PyTorch version 1.10.2 for network model construction. Parameter settings included preloading a weight file trained on the Coco dataset; the input fruit dataset had an image resolution of 640×640, training for 200 epochs, and a batch size of 8. Model weights were updated after each iteration, and model performance was tested using the test set data. The model weight with the highest recognition accuracy was saved as the weight of the trained lightweight fruit detection model.
[0052] The dataset annotated with three different maturity labels in this embodiment is obtained in the following way:
[0053] Get the original data set containing fruits and leaves;
[0054] Manually determining seed pixels in the original data set, and segmenting a fruit image from the original data set according to the seed pixels;
[0055] The fruit image is used as an object and the original data set is used as a background for image synthesis to obtain a data set in which the number of fruits in the image meets a preset number requirement.
[0056] It should be noted that since the sample data of fruits of different maturity in the original data set collected by the image acquisition device may be unbalanced, the present embodiment can use a region information segmentation method to segment the background and fruit in the image, and expand the segmented fruit image to ensure the balance of the sample data of fruits of different maturity.
[0057] Specifically, a pixel in the image is randomly selected as a seed pixel. Nearby pixels are retrieved, similar regions are clustered, and the region information segmentation method is used to traverse all pixels in the image, completing the image segmentation. The segmented fruit image is then used as the "object" and the rest of the original dataset as the "background" for image synthesis. This increases the number of fruit samples in the image, resulting in a dataset with the desired number of fruits.
[0058] After obtaining the dataset, we can use LabelImg annotation software to add label boxes to the fruits of different maturity levels in the dataset to obtain the labeled dataset. According to the preset ratio, the labeled dataset is divided into training set, test set, and validation set for training, and then a trained lightweight fruit detection model is obtained.
[0059] The embodiment of the present invention solves the problem of imbalance in sample proportions of fruits of different maturity by performing pixel-level data expansion on the original data set, thereby improving the robustness of the lightweight fruit detection model and expanding the scope of application of the lightweight fruit detection model.
[0060] In some embodiments, the lightweight fruit detection model includes an image input module, a feature extraction module, a feature fusion module, and a recognition module, wherein the feature extraction module includes a plurality of cascaded first neural network units, the feature fusion module includes a plurality of cascaded second neural network units, and the recognition module includes a weighted loss function of a VFL zoom loss function, a DFL distribution focus loss function, and a GIoU regression loss function;
[0061] The first neural network unit includes any one of a convolutional DSConv unit, a residual feature learning C3k2-F unit based on a FasterNet lightweight module, a feature fusion SPPF unit, and a C2PSA unit based on a PSA self-attention mechanism module;
[0062] The second neural network unit includes any one of a convolution DSConv unit, a residual feature learning C3k2-F unit based on a FasterNet lightweight module, an image splicing Concat unit, and an image upsampling Upsample unit.
[0063] In this embodiment, reference Figure 2 The lightweight fruit detection model includes an input image input module 201, a Backbone feature extraction module 202, a Neck feature fusion module 203 and a Prediction recognition module 204.
[0064] The input image input module 201 is used to input an RGB image with a resolution of 640×640. The Backbone feature extraction module 202 includes multiple cascaded first neural network units; the Neck feature fusion module 203 is constructed based on the PANet network, including multiple cascaded second neural network units, and the input of the current neural network unit is the output of the previous neural network unit, or the output of the previous neural network unit and the previous N neural network units, where N is a positive integer greater than 1; the Prediction recognition module 204 includes a weighted loss function of the VFL zoom loss function, the DFL distributed focus loss function and the GIoU regression loss function.
[0065] The residual feature learning C3k2-F unit is used to learn the residual features; the convolution DSConv unit is used to perform convolution, normalization and activation function calculation on the input image; the feature fusion SPPF unit is used to fuse the features of input images of different sizes, enriching the expression ability of the output image, which is conducive to the detection of targets with large size differences in different growth cycles; the feature fusion C2PSA unit is used to fuse the input images through comprehensive processing of channel and spatial features, improving the model's ability to focus on the target area and global feature expression ability, which is conducive to detecting targets or segmenting fine structures in complex scenes; the image upsampling Upsample unit is used to upsample the input image, and the image splicing Concat unit is used to calculate the Concat function on the input image.
[0066] In this embodiment, in order to improve the recognition accuracy of the lightweight fruit detection model for the target, reduce the model parameters to a certain extent, and improve the real-time detection speed of the model, a residual feature learning C3k2-F unit based on the FasterNet module is proposed, which increases the feature extraction capability of the backbone network and provides a reliable front-end information source for the feature fusion stage. The residual feature learning C3k2-F unit based on the FasterNet module is constructed based on the FasterNet lightweight module. The FasterNet lightweight module is used to reduce the parameters of the model. Specifically, Figure 3 The C3k2-F unit is composed of a DSConvBNSiLU network layer 301, a Split network layer 302, n FasterNet Block network layers 303, a Concat network layer 304, a ConvBNSiLU network layer 305, and a C3k-F network layer 306. In this embodiment, the number of FasterNet Block network layers 303 and C3k-F network layers 306 can be flexibly adjusted according to actual conditions and is not limited thereto.
[0067] Optionally, in this embodiment, the feature extraction module 202 can be composed of at least three DSConv units, at least three C3k2-F units, at least one SPFF unit and at least one C2PSA unit in cascade, and any two C3k2-F units are not adjacent, and the C2PSA unit is the last unit in the feature extraction module 202.
[0068] It should be noted that, in the embodiment of the present invention, the number and cascade mode of the DSConv units, C3k2-F units, SPFF units and C2PSA units in the Backbone feature extraction module 202 can be determined based on prior knowledge. In the embodiment of the present invention, the number and cascade mode of the DSConv units, C3k2-F units, SPFF units and C2PSA units in the Backbone feature extraction module 202 are not specifically limited.
[0069] Optionally, the Neck feature fusion module 203 may be composed of at least three DSConv units, at least three C3k2-F units, multiple Concat units and Upsample units in cascade, and any two C3k2-F units are not adjacent.
[0070] In the embodiment of the present invention, the number and cascade mode of the DSConv unit, C3k2-F unit, concat unit and Upsample unit in the Neck feature fusion module 203 can be determined based on prior knowledge. In the embodiment of the present invention, the number and cascade mode of the DSConv unit, C3k2-F unit, Concat unit and Upsample unit in the Neck feature fusion module 203 are not specifically limited.
[0071] In order to facilitate the understanding of the lightweight fruit detection model in the embodiment of the present invention, the lightweight fruit detection model in the embodiment of the present invention is described below through an example. The connection relationship of each neural network unit in the lightweight fruit detection model is shown in Table 1 and Figure 2 The model parameters of the lightweight fruit detection model are shown in Table 1.
[0072] Among them, the remark "n=1" of the C3k2-F unit 210 in Table 1 indicates that the number of FasterNetBlock modules or C3k-F modules in the internal structure of the C3k2-F unit is 1; the remark "n=2" of the C3k2-F unit 214 in Table 1 indicates that the number of FasterNet Block modules or C3k-F modules in the internal structure of the C3k2-F unit is 2.
[0073] The remark “210, 217” of the Concat unit 218 in Table 1 indicates that the input of the Concat unit 218 is the output of the C3k2-F unit 210 and the Upsample unit 217; the remark “213, 220” of the Concat unit 221 in Table 1 indicates that the input of the Concat unit 221 is the output of the C3k2-F unit 213 and the Upsample unit 220.
[0074] Table 1 Structural relationship and model parameters of lightweight fruit detection model
[0075]
[0076]
[0077] Specifically, in this embodiment, the image of each frame is sequentially input into the lightweight fruit detection model, and the coordinates of the fruit marking box output by the lightweight fruit detection model are obtained, including:
[0078] Inputting the image into the feature extraction module through the image input module, obtaining a first feature map output by a first C3k2-F unit, a second feature map output by a second C3k2-F unit, and a third feature map output by a C2PSA unit in the feature extraction module, where the second C3k2-F unit is a subordinate unit of the first C3k2-F unit;
[0079] Inputting the first feature map, the second feature map, and the third feature map into the first Concat unit, the second Concat unit, and the Upsample unit in the feature fusion module in sequence, obtaining a first feature matrix output by the first C3k2-F unit, a second feature matrix output by the second C3k2-F unit, and a third feature matrix output by the third C3k2-F unit in the feature fusion module, wherein the second Concat unit is a subordinate unit of the first Concat unit, the third C3k2-F unit is a subordinate unit of the second C3k2-F unit, and the second C3k2-F unit is a subordinate unit of the first C3k2-F unit;
[0080] The first feature matrix, the second feature matrix and the third feature matrix are output to the recognition module, the fruit is identified by the weighted loss function of the VFL zoom loss function, the DFL distribution focus loss function and the GIoU regression loss function in the recognition module, and the pixel size is segmented in the fruit by a labeling box containing spatial coordinate information.
[0081] In this embodiment, after the input image input module 201 inputs the current frame image into the Backbone feature extraction module 202, the Backbone feature extraction module 202 can perform fruit feature extraction on the current frame image, and the C3k2-F unit 210 (i.e., the first C3k2-F unit) in the Backbone feature extraction module 202 can output the first feature map to the Concat unit 218 in the feature fusion module 203; the C3k2-F unit 212 (i.e., the second C3k2-F unit) in the feature extraction module 202 can output the second feature map to the Concat unit 221 in the feature fusion module 203; and the C2PSA unit 216 in the feature extraction module 202 can output the third feature map to the Upsample unit 222 and the Concat unit 227 in the feature fusion module 203.
[0082] The C3k2-F unit 217 (i.e., the first C3k2-F unit) in the Neck feature fusion module 203 can output a first feature matrix to the Loss function module 205 in the Prediction recognition module 204, the C3k2-F unit 225 (i.e., the second C3k2-F unit) in the Neck feature fusion module 203 can output a second feature matrix to the Loss function module 205, and the C3k2-F unit 228 (i.e., the third C3k2-F unit) in the Neck feature fusion module 203 can output a third feature matrix to the Loss function module 205. The size of the first feature matrix is 80×80×45, the size of the second feature matrix is 40×40×45, and the size of the third feature matrix is 20×20×45.
[0083] The loss function module 205 can identify the target fruit based on the first feature matrix, the second feature matrix and the third feature matrix using the GIoU+DFL regression loss function 229 and the VFL classification loss function 230, and can mark the identified fruit in the current frame image through a marking box, and then output the spatial coordinate information of the marked fruit.
[0084] S2. Acquire continuous multiple-frame RGB images and depth images through an RGB-D camera. In this embodiment, continuous multiple-frame images are acquired in the form of video streams, which can avoid the problem of fruit information loss due to missed shots. Specifically, the image acquisition device can move in the area to be detected according to a preset route, and is used to collect video streams of fruits planted in the area to be detected in real time, thereby obtaining each frame of an image containing fruits from the video stream. The area to be detected is determined according to actual conditions, for example, it can be a field area for open-air planting, or it can be a greenhouse area. This is not specifically limited in the embodiments of the present invention. The preset route is a route for identifying fruits planted in the area to be detected, and the preset route can be determined based on prior knowledge and / or actual conditions. The preset route is not specifically limited in the embodiments of the present invention.
[0085] S3. Align the RGB image and depth image obtained in step S2;
[0086] S4. The RGB image obtained in step S3 is sequentially input into the lightweight fruit detection model constructed in step S1 to obtain the fruit target box information and semantic segmentation information output by the lightweight fruit detection model;
[0087] S5. Filter out the fruit outside the set camera-fruit distance threshold based on the fruit target frame information obtained in step S4 and the camera-fruit distance information in the depth image obtained in step S3, thereby obtaining the fruit within the set distance in each frame of the RGB image;
[0088] S6. The fruit is tracked and counted within a set distance in each frame of the RGB image obtained in step S5 by the area tracking and counting method to obtain the total number of fruits in multiple consecutive RGB images; in this embodiment, the method specifically comprises the following sub-steps:
[0089] S6-1. Select multiple consecutive frames of RGB images and depth images symmetrical to the center line as random areas;
[0090] S6-2. Calculate the number of fruits in the random area obtained in step S6-1: The size of the random area is used as the input of the particle swarm optimization algorithm and brought into the Bytetrack tracking algorithm. First, the video is detected: the fruit in the video is detected using a lightweight fruit detection model to obtain the target detection box and the corresponding features; then the target is predicted: the Kalman filter algorithm is used to predict the position and state of the target in the next frame of the video; finally, the target is matched: the Hungarian algorithm is used to optimally match several targets between the two frames before and after the video to obtain the trajectory of the target in the video. The target trajectory that fails to match will be temporarily saved and continue to participate in the subsequent frame prediction matching until the target fails to match for multiple consecutive frames and is considered to have disappeared, and the trajectory is then deleted; the Bytetrack tracking algorithm outputs the number of fruit ids in the random area, that is, the number of fruits in the random area;
[0091] S6-3. The root mean square error between the number of fruits in the random region obtained in step S6-2 and the true value is used as the fitness value of the particle swarm optimization algorithm. The size of the random region corresponding to the minimum convergence fitness value is the optimal region size.
[0092] S6-4. The optimal region size obtained in step S6-3 is used as the region size for yield estimation and is introduced into the Bytetrack tracking algorithm to obtain the number of fruits in the optimal region.
[0093] S7. Obtain the fruit area according to the fruit semantic segmentation information obtained in step S4 and the camera-fruit distance information in the depth image, and obtain the fruit weight according to the established fruit area-weight regression model; specifically, obtain the weight of the fruit according to the fruit segmentation information, RGB-D information and regression model, traverse the pixels in the target frame, and obtain the RGB values of these pixels. Because the segmentation mask RGB values of the three fruits with different maturity are different, the RGB values of each pixel in each target frame are judged to determine the maturity of the segmented fruit. Then, these pixels are accumulated and counted to finally obtain the pixel area of fruits with different maturity. The actual size of the fruit is obtained by converting the target fruit pixel area segmented by the improved YOLO11 algorithm and the measured distance from the camera to the fruit through a formula. The fruit area-weight regression model divides the fruit into three weight grades: large, medium and small according to weight, and divides each fruit into three regions: upper, middle and lower. A linear regression relationship is established between the areas of the upper, middle and lower regions of the fruit and the three weight grades of large, medium and small, respectively.
[0094] S8. The fruit count, fruit area, and fruit weight statistics obtained in step S6, as well as those obtained in step S7, are displayed in a visual interface for real-time yield monitoring. This visual interface is divided into three main areas: a function selection area allows the user to start and pause camera detection; a detection result display area provides a visual interface for the algorithm used in this study; and a yield information display area displays the number of fruits at three different maturity levels, the number of fruits at three different weight ranges, and the total number of fruits. Specifically, clicking the Start Detection button in the GUI invokes the improved YOLOv8 algorithm for object detection and semantic segmentation according to predefined button controls. Upon completion of detection, the algorithm outputs the target type, distance, and segmented pixel area. The actual fruit size is calculated using camera internal parameters, and the relationship between size and weight is established to determine the weight of the target fruit. The camera captures the target detection and semantic segmentation results, with the type and confidence level displayed above the target and four values displayed below the target: the coordinates (x, y) of the target fruit's center pixel, the measured distance in cm, and the weight in g. The detection result display area displays the statistically calculated number of fruits at each maturity level, the number of fruits in each weight range, and the total number of fruits. The system designs an area counting function within the camera image. Draw a line within the middle three-fifths of the camera image and set up motion tracking counting. Once a target is detected within this area, the target type is determined and a cumulative count is performed. A weight range is also set to determine the target weight and count it. The results of each cumulative count are ultimately displayed in real time behind the corresponding text on the interface. Clicking the Stop Detection button in the GUI pauses the camera from capturing the image. Clicking Start Detection restarts the camera from capturing the image and counting.
[0095] In addition, the present invention also provides a lightweight estimation system for fruit yield of fruit-bearing horticultural crops, such as Figure 4 As shown, the system includes a fruit identification and positioning processor 701 and a program or instruction that can be run on the fruit identification and positioning processor.
[0096] It should be noted that the specific process of the program or instruction running on the fruit recognition and positioning processor executing the above-mentioned fruit recognition and positioning method can be found in the contents of the above-mentioned embodiments, and will not be repeated in the embodiments of the present invention.
[0097] Optionally, the fruit recognition and positioning processor may be an NVIDIA Jeston Xavier NX development board with a power of 15 W. The fruit recognition and positioning processor may execute the fruit recognition and positioning method based on a Pytorch 1.10.2 framework.
[0098] Based on the contents of the above embodiments, the lightweight estimation system for fruit yield of fruit-bearing horticultural crops of the present invention further includes: a power supply 702 , an image acquisition device 703 and a display device 704 .
[0099] The power supply 702 is connected to the fruit recognition and positioning processor 701 and the display device 704 , thereby providing power to the fruit recognition and positioning processor 701 and the display device 704 .
[0100] The image acquisition device 703 is used to collect a video stream of the fruits planted in the area to be detected in real time, and transmit the video stream to the fruit recognition and positioning processor so that the fruit recognition and positioning processor can obtain the current frame image based on the video stream;
[0101] The display device 704 is used to receive and display the quantity and weight information of the fruits sent by the fruit identification and positioning processor 701.
[0102] The yield detection system of the embodiment of the present invention improves the robustness of the identification of fruits of different maturity levels, can improve the accuracy of the identification of fruits of different maturity levels, and can achieve accurate yield detection.
[0103] Figure 5 The electronic device provided by the present invention is a schematic diagram of the structure. As shown in the figure, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. The processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the lightweight fruit yield estimation method based on RGB-D information. The method includes:
[0104] Acquire a continuous multi-frame image, wherein the continuous multi-frame image is obtained by capturing images of a fruit currently to be inspected according to an image acquisition device, wherein the fruit is the object currently to be inspected;
[0105] Inputting the image of each frame into the lightweight fruit detection model in sequence, and obtaining the target frame information and segmentation information of the fruit output by the lightweight fruit detection model;
[0106] Obtaining the number of fruits according to the fruit target frame information and RGB-D information;
[0107] Obtaining the weight of the fruit based on the fruit segmentation information, the RGB-D information, and the regression model;
[0108] The yield (quantity and weight) of the fruit is displayed in a visual interface for real-time monitoring;
[0109] Among them, the lightweight fruit detection model is trained based on a data set labeled with three different maturity labels. The lightweight fruit detection model is based on the YOLO11 model. The FasterNet module is introduced into the residual feature learning C3k2 unit of the YOLO11 model, the Conv of the YOLO11 model is changed to DSCOnv, and the regression loss of the YOLO11 model is set to the weighted loss of DFL distribution focus loss and GIoU regression loss.
[0110] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0111] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a lightweight method for estimating fruit yield of fruit-bearing horticultural crops based on an RGB-D camera, as provided in the above methods. The method comprises:
[0112] Acquire a continuous multi-frame image, wherein the continuous multi-frame image is obtained by capturing images of a fruit currently to be inspected according to an image acquisition device, wherein the fruit is the object currently to be inspected;
[0113] Inputting the image of each frame into the lightweight fruit detection model in sequence, and obtaining the target frame information and segmentation information of the fruit output by the lightweight fruit detection model;
[0114] Obtaining the number of fruits according to the fruit target frame information and RGB-D information;
[0115] Obtaining the weight of the fruit based on the fruit segmentation information, the RGB-D information, and the regression model;
[0116] The yield (quantity and weight) of the fruit is displayed in a visual interface for real-time monitoring;
[0117] Among them, the lightweight fruit detection model is trained based on a data set labeled with three different maturity labels. The lightweight fruit detection model is based on the YOLO11 model. The FasterNet module is introduced into the residual feature learning C3k2 unit of the YOLO11 model, the Conv of the YOLO11 model is changed to DSCOnv, and the regression loss of the YOLO11 model is set to the weighted loss of DFL distribution focus loss and GIoU regression loss.
[0118] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.
[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims
1. A lightweight method for estimating the fruit yield of fruit-bearing horticultural crops, characterized in that: The method comprises the following steps: S1. Build a lightweight fruit detection model: Use the YOLO11 model as the framework and introduce the FasterNet module into the residual feature learning C3k2 unit of the YOLO11 model. Change the Conv of the YOLO11 model to DSCOnv, and set the regression loss of the YOLO11 model to a weighted loss of the DFL distributed focus loss and the GIoU regression loss. A lightweight fruit detection model is trained on a dataset labeled with three different maturity levels. S2. Acquire multiple consecutive RGB images and depth images using an RGB-D camera. The multiple consecutive RGB images and depth images are acquired by an image acquisition device by capturing images of fruit of a fruiting horticultural crop in the current yield estimation area, where the fruit is the object to be detected. S3. Align the RGB image and depth image obtained in step S2; S4. The RGB image obtained in step S3 is sequentially input into the lightweight fruit detection model constructed in step S1 to obtain the fruit target box information and semantic segmentation information output by the lightweight fruit detection model; S5. Filter out the fruit outside the set camera-fruit distance threshold based on the fruit target frame information obtained in step S4 and the camera-fruit distance information in the depth image obtained in step S3, thereby obtaining the fruit within the set distance in each frame of the RGB image; S6. The fruit is tracked and counted within a set distance in each frame of the RGB image obtained in step S5 by the area tracking and counting method to obtain the total number of fruits in multiple consecutive RGB images, specifically comprising the following sub-steps: S6-1. Select multiple consecutive frames of RGB images and depth images symmetrical to the center line as random areas; S6-2. Calculate the number of fruits in the random area obtained in step S6-1: The size of the random area is used as the input of the particle swarm optimization algorithm and brought into the Bytetrack tracking algorithm. First, the video is detected: the fruit in the video is detected using a lightweight fruit detection model to obtain the target detection box and the corresponding features; then the target is predicted: the Kalman filter algorithm is used to predict the position and state of the target in the next frame of the video; finally, the target is matched: the Hungarian algorithm is used to optimally match several targets between the two frames before and after the video to obtain the trajectory of the target in the video. The target trajectory that fails to match will be temporarily saved and continue to participate in the subsequent frame prediction matching until the target fails to match for multiple consecutive frames and is considered to have disappeared, and the trajectory is then deleted; the Bytetrack tracking algorithm outputs the number of fruit ids in the random area, that is, the number of fruits in the random area; S6-3. The root mean square error between the number of fruits in the random region obtained in step S6-2 and the true value is used as the fitness value of the particle swarm optimization algorithm. The size of the random region corresponding to the minimum convergence fitness value is the optimal region size. S6-4. The optimal region size obtained in step S6-3 is used as the region size for yield estimation, and the number of fruits in the optimal region is obtained by the Bytetrack tracking algorithm; S7. Obtaining the fruit area based on the fruit semantic segmentation information and the camera-fruit distance information in the depth image, and obtaining the fruit weight based on the established fruit area-weight regression model; S8. Display the statistical information of the number of fruits obtained in step S6, the fruit area and the fruit weight obtained in step S7 in a visual interface to monitor the yield in real time.
2. A lightweight method for estimating fruit yield of fruit-bearing horticultural crops according to claim 1, characterized in that: The lightweight fruit detection model includes an image input module, a feature extraction module, a feature fusion module and a recognition module; the feature extraction module includes multiple cascaded first neural network units, the feature fusion module includes multiple cascaded second neural network units, and the recognition module includes a weighted loss function of the DFL distribution focus loss function and the GIoU regression loss function, and a VFL zoom loss function.
3. The lightweight method for estimating fruit yield of fruit-bearing horticultural crops according to claim 1, characterized in that: The fruit area-weight regression model in step S7 measures the area and weight of multiple fruits of different weight sizes, wherein the area measurement is completed by collecting RGB images of the fruit with an RGB-D camera, and then a weight-fruit area regression relationship is established according to the position of the fruit in the image, wherein the position is the upper, lower or middle area.
4. The lightweight method for estimating fruit yield of fruit-bearing horticultural crops according to claim 1, characterized in that: The camera-fruit distance threshold described in step S5 is set according to the crop planting row spacing of the fruit.
5. The lightweight method for estimating fruit yield of fruit-bearing horticultural crops according to claim 1, characterized in that: The dataset annotated with three different maturity labels in step S1 is obtained by the following method: Get the original data set containing fruits and leaves; Manually determining seed pixels in the original data set, and segmenting a fruit image from the original data set according to the seed pixels; Performing image synthesis using the fruit image as an object and the original data set as a background; The background and fruit in the image are then segmented using a region-information-based segmentation method, and the segmented fruit image is expanded. A pixel in the image is randomly selected as a seed pixel, and pixels near the seed pixel are retrieved. Similar regions are clustered, and the region-information-based segmentation method is used to traverse all pixels in the image to complete the image segmentation. The segmented fruit image is then used as the "object" and the other original datasets as the "background" for image synthesis. This increases the number of fruit samples in the image and results in a dataset with a preset number of fruits. After obtaining the dataset, LabelImg annotation software was used to add marking boxes to the fruits of different maturity in the dataset to obtain the annotated dataset. The annotated dataset was divided into training set, test set and validation set according to the preset ratio.
6. A lightweight estimation system for fruit yield of fruit-bearing horticultural crops, characterized by: The system includes a fruit identification and positioning processor and a program or instruction that can be run on the fruit identification and positioning processor. When the program or instruction is executed by the fruit identification and positioning processor, it implements the lightweight estimation method for fruit yield of fruit-bearing horticultural crops as described in one of claims 1-5.
7. A lightweight fruit yield estimation system for fruit-bearing horticultural crops according to claim 6, characterized in that: The system also includes an image acquisition device and a visualization device; the image acquisition device and the visualization device are respectively connected to the fruit recognition and positioning processor; The image acquisition device is used to collect a video stream of fruits planted in the area to be estimated in real time, and send the video stream to the fruit recognition and positioning processor so that the fruit recognition and positioning processor can obtain multiple frames of continuous images based on the video stream; The visualization device is used to perform real-time display based on the information output by the fruit identification and positioning processor.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for lightweight estimation of fruit yield of fruit-bearing horticultural crops as claimed in any one of claims 1 to 5 is implemented.
9. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements a lightweight method for estimating the fruit yield of fruit-bearing horticultural crops as described in any one of claims 1 to 5.