Strawberry picking maturity detection method and system
By combining high-definition images, thermal radiation images and three-dimensional point cloud data, a strawberry ripening detection model is constructed, which solves the problem of single strawberry ripening detection indicators in the existing technology, and improves the accuracy of strawberry ripening and picking judgment Accuracy enables more comprehensive feature representation and automatic picking capabilities.
Patent Information
- Application Number
- CN202411056028.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-08
AI Technical Summary
The existing strawberry ripening method has a single index, and it is impossible to comprehensively evaluate the pickability of strawberries from multiple dimensions, resulting in a low recognition accuracy.
Using a variety of image feature fusion methods, including high-definition images, thermal radiation images and three-dimensional point cloud data, a strawberry ripening detection model is constructed, and a residual neural network, attention module, feature fusion unit and weight calculation unit are used to capture different images. The features collected by the device are fused to determine whether the strawberry reaches the preset maturity threshold.
It improves the accuracy of strawberry ripening and the accuracy of pickable judgment, provides a more comprehensive feature representation, and enhances the ability of the automatic picking robot to identify and locate strawberries.
Smart Images

Figure CN118918580A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and more specifically, to a strawberry picking maturity detection method and system. Background Art
[0002] Smart picking uses integrated intelligent technology to enable the picking system to imitate human intelligence, with the ability to think, perceive, learn, reason and judge, and solve certain problems in picking by itself. The future development direction of smart picking includes four characteristics: intelligence, integration and hierarchy, flexibility and socialization. Automatic picking robots will develop towards higher intelligence and autonomy. Through the continuous progress of deep learning and machine vision technology, the accuracy of fruit recognition and positioning will be improved, and more complex picking actions will be achieved. Among them, how to detect the maturity of fruits in complex scenes and accurately select the fruits that can be picked is an important part of realizing the smart picking function.
[0003] The prior art discloses a strawberry maturity detection method and system based on improved YOLOv8, comprising: collecting strawberry images to be detected; performing maturity recognition on the strawberry images to be detected based on an improved YOLOv8 model to obtain strawberry maturity detection results; wherein the improved YOLOv8 model is trained through a training set, and the training set data is a set of strawberry images with maturity annotated. However, although this method can improve the accuracy of strawberry recognition and positioning and picking of target detection models in open scenes, the maturity detection index is single and it is impossible to comprehensively evaluate the pickability of strawberries from multiple dimensions. Summary of the invention
[0004] In order to solve the problem that the current strawberry maturity detection index is single and the recognition accuracy is low, the present invention proposes a strawberry picking maturity detection method and system, which combines multiple image features to judge whether the strawberry maturity reaches the strawberry maturity threshold, thereby improving the accuracy of maturity detection.
[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:
[0006] A strawberry picking maturity detection method, comprising:
[0007] S1. Collect image data samples of strawberries in the greenhouse, including high-definition image data, thermal radiation image data and three-dimensional point cloud data of strawberries;
[0008] S2. Preprocess and annotate the collected image data samples, construct a data set, and divide the data set into a training set, a validation set, and a test set;
[0009] S3. construct a strawberry maturity detection model, train the strawberry maturity detection model using a training set, evaluate the strawberry maturity detection model using a validation set, test the effectiveness of the strawberry maturity detection model using a test set, and obtain a trained strawberry maturity detection model;
[0010] S4. Collect image data of the strawberries to be detected, input the image data into a trained strawberry maturity detection model, and output the strawberry maturity detection result and the picking judgment result.
[0011] This technical solution constructs a strawberry maturity detection model and fuses the features collected by different cameras to obtain a more comprehensive feature representation, further improving the accuracy of judging whether the strawberries are ready to be picked.
[0012] Preferably, the preprocessing process includes scaling, cropping, and normalization.
[0013] Preferably, in step S2, high-definition image data of strawberries are obtained using a high-definition shooting device; the high-definition image data are annotated using a CVAT annotation tool to obtain strawberry positioning marks in the high-definition image data, generate annotation data, and export the annotation data and store it in a data set.
[0014] Here, the CVAT annotation tool is used to identify, locate and classify strawberries in the image, and annotation information such as bounding boxes, segmentation masks or key points are created for each target object to achieve automatic recognition and classification of objects such as strawberries.
[0015] Preferably, in step S2, a thermal radiation image of the strawberry is obtained by using an infrared imaging camera; the thermal radiation image is annotated by using a LabelImg annotation tool to obtain an infrared maturity label of the strawberry, and generate annotation data;
[0016] Analyze the thermal radiation image of strawberries using spectral analysis technology to obtain the release rate of chemical substances on the strawberry surface and generate annotation data;
[0017] Use OpenCV image processing tools to analyze the color of strawberries in infrared images, calculate the red coverage of strawberry colors, and generate annotation data;
[0018] The above annotation data are exported and stored in the dataset.
[0019] Here, since ripe strawberries are usually slightly hotter than unripe strawberries and objects at different temperatures emit different amounts of infrared radiation, the degree of ripeness of the strawberries can be reflected by the heat distribution of the strawberry thermal radiation image and the difference in the concentration of chemicals released from the outer skin of the strawberries at different stages.
[0020] Preferably, in step S2, three-dimensional point cloud data of the strawberry is obtained using a 3D TOF camera; the three-dimensional point cloud data is annotated using a LabelFusion annotation tool to obtain three-dimensional information of the strawberry, including the spatial position, shape and size information of the strawberry, generate annotation data, and export the annotation data and store it in a data set.
[0021] Here, labeling tools such as LabelFusion are combined to achieve accurate measurement of the three-dimensional spatial position and shape of strawberries, thereby providing the necessary information for the automated picking robot to achieve the best planning path and accurate picking; the labeling process includes creating bounding boxes, semantic labels, and instance segmentation for the strawberries and their surroundings to facilitate subsequent processing and analysis.
[0022] Preferably, based on the YOLOv10-x target detection algorithm, a strawberry maturity detection model is constructed; the strawberry maturity detection model comprises a residual neural network, an attention module, a feature fusion unit, a key point detection unit and a weight calculation unit; the residual neural network is used to extract deep features of input image data, and the deep features include color features of high-definition image data, heat features of thermal radiation image data and depth features of three-dimensional point cloud data; the attention module enhances attention to important features by introducing an attention mechanism; the feature fusion unit is used to merge the deep features of the image; the key point detection network is used to judge whether the input image data is aligned according to the target detection and feature point positioning results output by the feature fusion unit; the weight calculation unit performs weight calculation on the fused image features of the aligned image data to judge whether the strawberry maturity reaches a preset strawberry maturity threshold.
[0023] Here, a strawberry maturity detection model is constructed based on the YOLOv10-x object detection algorithm, and the preprocessed image data is input into the strawberry maturity detection model to obtain a more comprehensive feature representation.
[0024] Preferably, the weight calculation is performed based on the loss function of the model, and during the training process, the model parameters are adjusted by a back propagation algorithm to minimize the difference between the predicted value and the actual value.
[0025] Preferably, the activation function of the yolov10-x target detection algorithm is the LeakyRelu function, and the calculation formula of the LeakyRelu function is:
[0026]
[0027] Among them, the value of α is 0.01.
[0028] Preferably, a strawberry maturity threshold is preset, and the strawberry red coverage, surface chemical substance release rate and three-dimensional information are combined to determine whether the strawberry maturity reaches the strawberry maturity threshold. If so, the strawberries reach the picking standard, otherwise, the strawberries do not reach the picking standard.
[0029] A strawberry picking maturity detection system, comprising:
[0030] A data acquisition module is used to collect image data samples of strawberries in the greenhouse, wherein the image data samples include high-definition image data, thermal radiation image data and three-dimensional point cloud data of the strawberries;
[0031] The image processing module is used to annotate the collected image data samples after preprocessing, construct a data set, and divide the data set into a training set, a validation set, and a test set;
[0032] A model training module is used to construct a strawberry maturity detection model, train the strawberry maturity detection model using a training set, evaluate the strawberry maturity detection model using a validation set, and test the effectiveness of the strawberry maturity detection model using a test set to obtain a trained strawberry maturity detection model;
[0033] The output module is used to input the image data of the strawberries to be detected into the trained strawberry maturity detection model, and output the strawberry maturity detection results and the picking judgment results.
[0034] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0035] The invention provides a strawberry picking maturity detection method and system. Firstly, image data samples of strawberries in a greenhouse are collected, including high-definition image data, thermal radiation image data and three-dimensional point cloud data of the strawberries. Then, the collected image data samples are pre-processed and annotated to construct a data set. The data set is used to train a strawberry maturity detection model to obtain a trained strawberry maturity detection model. Finally, image data of strawberries to be detected are collected, the image data are input into the trained strawberry maturity detection model, and a strawberry maturity detection result and a picking judgment result are output. The features collected by different camera devices are fused through the strawberry maturity detection model to obtain a more comprehensive feature representation, so as to further improve the accuracy of the strawberry picking judgment. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 A schematic diagram showing the flow of the strawberry picking maturity detection method proposed in Example 1 of the present invention;
[0037] Figure 2 A rendering showing a strawberry image recognized by a high-definition camera in Example 1 of the present invention;
[0038] Figure 3 The effect diagram of strawberry volatiles photographed by the infrared imaging camera in Example 1 of the present invention is shown;
[0039] Figure 4 It shows the effect picture of taking strawberries by a 3D TOF camera in Embodiment 1 of the present invention;
[0040] Figure 5 The figure shows the structure diagram of the strawberry picking maturity detection system proposed in Example 3 of the present invention. DETAILED DESCRIPTION
[0041] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0042] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;
[0043] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.
[0044] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0045] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent;
[0046] Example 1
[0047] like Figure 1 As shown, this embodiment proposes a strawberry picking maturity detection method, comprising:
[0048] S1. Collect image data samples of strawberries in the shed. In this embodiment, high-definition image data of strawberries is obtained by high-definition shooting equipment to obtain clear image data of strawberries; thermal radiation images of strawberries are obtained by infrared imaging cameras, using infrared thermal imaging technology to capture the heat distribution of strawberries; 3DTOF (Time-of-Flight) cameras are used to scan strawberries to obtain three-dimensional point cloud data of strawberries, including spatial position, shape and size information of strawberries;
[0049] S2. After preprocessing the collected image data samples, they are annotated and constructed into a data set, which is divided into a training set, a validation set and a test set; the preprocessing process includes scaling, cropping, normalization, etc., and the specific process of the marking includes:
[0050] S21. Figure 2As shown, the high-definition image data is annotated using the CVAT (Computer Vision Annotation Tool) annotation tool to obtain the strawberry positioning mark in the high-definition image data, and the annotation data is exported and stored in a data set; wherein, the CVAT is a tool for image and video annotation, which can help users to annotate and annotate the collected strawberry images, and the implementation principle is mainly based on computer vision technology, by identifying, locating and classifying the strawberries in the image, creating a bounding box, a segmentation mask or key points and other annotation information for each target object, so as to realize automatic recognition and classification of objects such as strawberries.
[0051] During the CVAT annotation process, users first upload the high-definition image data to be annotated, and then annotate the strawberries in the image through the user interface provided by CVAT. During the annotation process, users can manually draw bounding boxes, segmentation masks or key points, or use the automatic annotation functions provided by CVAT, such as automatic detection and segmentation based on deep learning models, to assist the annotation process.
[0052] After labeling is completed, CVAT will generate the labeling results and store them in the dataset in the form of JSON, XML or other formats.
[0053] S22. Figure 3 As shown, the thermal radiation image is annotated using the LabelImg annotation tool to obtain the infrared maturity mark of the strawberry; the LabelImg annotation tool is usually used for target detection tasks, draws bounding boxes and assigns labels to them, and uses imaging software to analyze these images to generate a heat distribution map;
[0054] Spectral analysis techniques, such as near infrared spectroscopy (NIR) or Raman spectroscopy, are used to analyze the thermal radiation images of strawberries to obtain the release rate of chemical substances on the surface of the strawberries and determine whether the maturity threshold has been reached. Professional image processing software and chemical analysis software can be used to process and analyze the data to evaluate the maturity and quality of the strawberries.
[0055] Use OpenCV image processing tools to analyze the color of strawberries in infrared images and calculate the red coverage of strawberries;
[0056] The above annotation data are exported and stored in the dataset.
[0057] S23. Figure 4As shown in the figure, the three-dimensional point cloud data is annotated using the LabelFusion annotation tool to obtain the three-dimensional information of the strawberry, and the annotation data is exported and stored in the data set; the collected point cloud data is annotated using the LabelFusion annotation tool to achieve accurate measurement of the three-dimensional spatial position and shape of the strawberry; the annotation process includes creating bounding boxes, semantic labels, and instance segmentation for the strawberry and its surroundings to facilitate subsequent processing and analysis. Based on the annotated three-dimensional information, the automated picking system can calculate the best picking path, which usually involves path planning algorithms such as RRT (Rapidly Exploring Random Trees) or A* algorithms to help the picking robot avoid obstacles and find the shortest path from the starting point to the strawberry.
[0058] S3. construct a strawberry maturity detection model, train the strawberry maturity detection model using a training set, evaluate the strawberry maturity detection model using a validation set, test the effectiveness of the strawberry maturity detection model using a test set, and obtain a trained strawberry maturity detection model;
[0059] S4. Collect image data of the strawberries to be detected, input the image data into a trained strawberry maturity detection model, and output the strawberry maturity detection result and the picking judgment result.
[0060] In addition, in the actual operation process, it is also necessary to integrate the data information (red coverage, chemical release rate, three-dimensional information, etc.) from different sources into a unified processing framework. In the present embodiment, computer vision libraries such as OpenCV are used to process and analyze these data, so as to facilitate subsequent model input. Further, the OpenCV also provides a kind of supervision mechanism, which can monitor the image processing process of strawberry in real time, and ensure the consistency of the quality and processing of data; the supervision mechanism can include data verification, anomaly detection, model performance monitoring, etc.
[0061] Example 2
[0062] This embodiment is based on Embodiment 1 and further describes the strawberry maturity detection model.
[0063] Based on the yolov10-x target detection algorithm, a strawberry maturity detection model is constructed; the strawberry maturity detection model includes a residual neural network (Residual Neural Network, ResNet), an attention module, a feature fusion unit, a key point detection unit and a weight calculation unit.
[0064] The residual neural network is used to extract deep features of input image data. By introducing skip connections, gradients are allowed to flow directly through some layers, thereby solving the gradient vanishing problem in deep network training. In this embodiment, the deep features include color features of high-definition image data, heat features of thermal radiation image data, and depth features of three-dimensional point cloud data.
[0065] The attention module enhances the focus on important features and improves the accuracy of feature extraction by introducing attention mechanisms such as SENet, Transformer, etc. In this embodiment, the attention module adopts Self-Attention in Transformer.
[0066] The feature fusion unit is used to merge the deep features of the image to obtain a more comprehensive feature representation. Feature fusion can be achieved in many ways, such as splicing, element addition, weight merging, etc.
[0067] The key point detection network is used to determine whether the input image data is aligned based on the target detection and feature point positioning results output by the feature fusion unit. For the alignment of feature points, the model needs to be able to accurately identify and locate the key points in the image, so a key point detection network (such as Hourglass, CPM, etc.) or special training based on yolov10-x is required; if the detected feature point position is consistent with the expected position, the image data can be considered aligned; if there is a deviation, the image data may need to be further adjusted or re-collected.
[0068] The weight calculation unit performs weight calculation on the fused image features of the aligned image data to determine whether the strawberry maturity reaches a preset strawberry maturity threshold. In the present embodiment, the weight calculation is performed based on the loss function of the model, and during the training process, the model parameters are adjusted by the back propagation algorithm to minimize the difference between the predicted value and the actual value. Specifically, the strawberry maturity threshold is preset, and the strawberry red coverage, surface chemical release rate and three-dimensional information are combined to determine whether the strawberry maturity reaches the strawberry maturity threshold. If so, the strawberry reaches the standard for picking, otherwise, the strawberry does not reach the standard for picking.
[0069] In this embodiment, the activation function of the yolov10-x target detection algorithm is the LeakyRelu function, and the calculation formula of the LeakyRelu function is:
[0070]
[0071] Among them, the value of α is 0.01.
[0072] Example 3
[0073] like Figure 5 As shown, this embodiment provides a strawberry picking maturity detection system, including:
[0074] The data acquisition module 101 is used to collect image data samples of strawberries in the greenhouse, and the image data samples include high-definition image data, thermal radiation image data and three-dimensional point cloud data of strawberries. In this embodiment, high-definition image data of strawberries is obtained by using high-definition shooting equipment to obtain clear strawberry image data; thermal radiation images of strawberries are obtained by using infrared imaging cameras, and infrared thermal imaging technology is used to capture the heat distribution of strawberries; 3D TOF (Time-of-Flight) cameras are used to scan strawberries to obtain three-dimensional point cloud data of strawberries; further, the data acquisition module 101 is installed on a small mechanical arm of a picking robot.
[0075] The image processing module 102 is used to annotate the collected image data samples after preprocessing, construct a data set, and divide the data set into a training set, a validation set, and a test set;
[0076] The model training module 103 is used to construct a strawberry maturity detection model, train the strawberry maturity detection model using a training set, evaluate the strawberry maturity detection model using a validation set, and test the effectiveness of the strawberry maturity detection model using a test set to obtain a trained strawberry maturity detection model;
[0077] The output module 104 is used to input the image data of the strawberries to be detected into the trained strawberry maturity detection model, and output the strawberry maturity detection result and the picking judgment result.
[0078] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A strawberry picking maturity detection method, characterized in that: include: S1. Collect image data samples of strawberries in the greenhouse, including high-definition image data, thermal radiation image data and three-dimensional point cloud data of strawberries; S2. Preprocess and annotate the collected image data samples, construct a data set, and divide the data set into a training set, a validation set, and a test set; S3. construct a strawberry maturity detection model, train the strawberry maturity detection model using a training set, evaluate the strawberry maturity detection model using a validation set, test the effectiveness of the strawberry maturity detection model using a test set, and obtain a trained strawberry maturity detection model; S4. Collect image data of the strawberries to be detected, input the image data into a trained strawberry maturity detection model, and output the strawberry maturity detection result and the picking judgment result.
2. The strawberry picking maturity detection method according to claim 1, characterized in that: In step S2, the preprocessing process includes scaling, cropping, and normalization.
3. The strawberry picking maturity detection method according to claim 1, characterized in that: In step S2, high-definition image data of strawberries is obtained by using a high-definition shooting device; the high-definition image data is annotated using a CVAT annotation tool to obtain strawberry positioning marks in the high-definition image data, generate annotation data, and export the annotation data and store it in a data set.
4. The strawberry picking maturity detection method according to claim 1, characterized in that: In step S2, a thermal radiation image of the strawberry is obtained by using an infrared imaging camera; the thermal radiation image is annotated by using a LabelImg annotation tool to obtain an infrared maturity label of the strawberry and generate annotation data; Analyze the thermal radiation image of strawberries using spectral analysis technology to obtain the release rate of chemical substances on the strawberry surface and generate annotation data; Use OpenCV image processing tools to analyze the color of strawberries in infrared images, calculate the red coverage of strawberry colors, and generate annotation data; The above annotation data are exported and stored in the dataset.
5. The strawberry picking maturity detection method according to claim 1, characterized in that: In step S2, the three-dimensional point cloud data of the strawberry is obtained by using a 3DTOF camera; the three-dimensional point cloud data is annotated by using a LabelFusion annotation tool to obtain the three-dimensional information of the strawberry, generate annotation data, and export the annotation data and store it in a data set.
6. The strawberry picking maturity detection method according to any one of claims 1 to 5, characterized in that: Based on the YOLOv10-x target detection algorithm, a strawberry maturity detection model is constructed; the strawberry maturity detection model comprises a residual neural network, an attention module, a feature fusion unit, a key point detection unit and a weight calculation unit; the residual neural network is used to extract deep features of input image data, and the deep features include color features of high-definition image data, heat features of thermal radiation image data and depth features of three-dimensional point cloud data; the attention module enhances attention to important features by introducing an attention mechanism; the feature fusion unit is used to merge the deep features of the image; the key point detection network is used to determine whether the input image data is aligned according to the target detection and feature point positioning results output by the feature fusion unit; the weight calculation unit performs weight calculation on the fused image features of the aligned image data to determine whether the strawberry maturity reaches a preset strawberry maturity threshold.
7. The strawberry picking maturity detection method according to claim 6, characterized in that: The weight calculation is performed based on the loss function of the model. During the training process, the model parameters are adjusted through the back propagation algorithm to minimize the difference between the predicted value and the actual value.
8. The strawberry picking maturity detection method according to claim 6, characterized in that: The activation function of the yolov10-x target detection algorithm is the LeakyRelu function, and the calculation formula of the LeakyRelu function is: Among them, the value of α is 0.
01.
9. The strawberry picking maturity detection method according to claim 6, characterized in that: A strawberry maturity threshold is preset, and the strawberry red coverage, surface chemical release rate and three-dimensional information are combined to determine whether the strawberry maturity has reached the strawberry maturity threshold. If so, the strawberry has reached the standard for picking, otherwise, the strawberry has not reached the standard for picking.
10. A strawberry picking maturity detection system, characterized in that: include: A data acquisition module is used to collect image data samples of strawberries in the greenhouse, wherein the image data samples include high-definition image data, thermal radiation image data and three-dimensional point cloud data of the strawberries; The image processing module is used to annotate the collected image data samples after preprocessing, construct a data set, and divide the data set into a training set, a validation set, and a test set; A model training module is used to construct a strawberry maturity detection model, train the strawberry maturity detection model using a training set, evaluate the strawberry maturity detection model using a validation set, and test the effectiveness of the strawberry maturity detection model using a test set to obtain a trained strawberry maturity detection model; The output module is used to input the image data of the strawberries to be detected into the trained strawberry maturity detection model, and output the strawberry maturity detection results and the picking judgment results.
Citation Information
Cited By
Fruit ripening detection method and electronic equipment
CN121027436A
