A method, system, electronic device and medium for determining the weight of strawberry fruits

Abstract

The present invention discloses a method, system, electronic device and medium for determining the weight of strawberry fruits, relating to the technical field of digital agriculture. The method includes: obtaining the overall images of a plurality of target strawberries; applying a single-stage object detection model to segment each target strawberry in the overall images to obtain multiple images of single target strawberries; and inputting the multiple images of single target strawberries into a strawberry mass estimation model to obtain the fresh weight of each single target strawberry. The present invention can improve the accuracy of estimating the weight of strawberry fruits.

Description

Technical Field

[0001] The present invention relates to the technical field of digital agriculture, and particularly to a method, a system, an electronic device and a medium for determining the weight of strawberry fruits. Background Art

[0002] In recent years, the planting area of strawberries in China has been continuously expanding, and the output has ranked first in the world. Strawberry fruits contain various nutrients and are deeply loved by consumers. However, the planting and picking of strawberries mainly rely on manual labor, and due to their short harvesting period, time-consuming picking, and poor storage tolerance, etc., the progress of their commercial promotion has been restricted. With the popularization of the high-rise cultivation mode of strawberries, it is of great significance to implement mechanized strawberry picking. Due to the differences between individual fruits, the maturity and weight of fruits are not the same at the same time. Therefore, it is necessary to classify and grade during mechanized picking. The fresh weight of strawberries is directly related to strawberry yield and classification. High-precision non-destructive estimation of strawberry fresh weight can provide a technical basis for optimizing the strawberry growth process. At the same time, realizing accurate strawberry classification and grading helps to achieve the digitization, automation and intelligence of strawberry cultivation and picking, which is of great significance in agricultural production and research.

[0003] With the development of software and hardware technologies related to computer vision, machine vision and image processing technologies based on visible light have become research hotspots for non-destructive estimation of crop phenotypic parameters. However, for non-destructive estimation and grading research of strawberry quality based on machine vision, feature selection is required, with high labor costs, and the quality of feature selection will greatly affect the accuracy of estimation. In addition, the expression ability of shallow machine learning models is limited, and the fitting effect for complex problems is poor. Compared with shallow machine learning, deep learning can achieve spontaneous feature learning, contains more hidden layers and can achieve end-to-end model output, and has a better fitting effect for complex problems. Therefore, it is particularly important to develop a method for strawberry quality estimation and fruit grading based on deep learning. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, a system, an electronic device and a medium for determining the weight of strawberry fruits, which can improve the accuracy of strawberry fruit weight estimation.

[0005] To achieve the above purpose, the present invention provides the following solutions:

[0006] A method for determining the weight of strawberry fruits, the determination method comprising:

[0007] Obtain overall images of a plurality of target strawberries;

[0008] Apply a single-stage object detection model to segment each target strawberry in the overall image to obtain multiple images of single target strawberries;

[0009] Input the images of multiple single target strawberries into the strawberry quality estimation model to obtain the fresh weight of each single target strawberry; wherein, the strawberry quality estimation model is obtained by training a deep learning model with a training set; the deep learning model includes a feature extraction network and a regression network connected in sequence; the training set includes images of multiple single strawberry samples and the fresh weight of each single strawberry sample.

[0010] Optionally, the single-stage object detection model is the YOLOv8 model.

[0011] Optionally, the feature extraction network is a CNN network.

[0012] Optionally, the regression network includes a fully connected layer and an activation function.

[0013] Optionally, the training process of the deep learning model includes:

[0014] Use the image of the single strawberry sample as the input and the corresponding fresh weight of the single strawberry sample as the output to train the deep learning model.

[0015] When the number of training times is greater than the preset number of training times, obtain the trained deep learning model and use the trained deep learning model as the strawberry quality estimation model.

[0016] A strawberry fruit weight determination system applies the above strawberry fruit weight determination method. The determination system includes:

[0017] An acquisition module for acquiring the overall images of multiple target strawberries.

[0018] A segmentation module for applying a single-stage object detection model to segment each target strawberry in the overall image to obtain multiple images of single target strawberries.

[0019] A prediction module for inputting the multiple images of single target strawberries into the strawberry quality estimation model to obtain the fresh weight of each single target strawberry; wherein, the strawberry quality estimation model is obtained by training a deep learning model with a training set; the deep learning model includes a feature extraction network and a regression network connected in sequence; the training set includes images of multiple single strawberry samples and the fresh weight of each single strawberry sample.

[0020] An electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above strawberry fruit weight determination method.

[0021] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above strawberry fruit weight determination method.

[0022] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0023] A method, system, electronic device and medium for determining the weight of strawberry fruits provided by the present invention obtain the overall images of multiple target strawberries, apply a single-stage object detection model to segment each target strawberry in the overall image to obtain multiple images of single target strawberries, and then input the multiple images of single target strawberries into a strawberry quality estimation model to obtain the fresh weight of each single target strawberry; wherein, the strawberry quality estimation model is obtained by training a deep learning model with a training set; the training set includes multiple images of single strawberry samples and the fresh weight of each single strawberry sample. The present invention estimates the weight of single strawberry fruits by applying a deep learning model, thereby improving the accuracy of estimating the weight of single strawberry fruits. Description of the Drawings

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 It is a flowchart of the method for determining the weight of strawberry fruits of the present invention;

[0026] Figure 2 It is a block diagram of model construction and training in the actual application of the present invention;

[0027] Figure 3 It is a flowchart of applying the method for determining the weight of strawberry fruits in the actual application of the present invention for strawberry fruit grading;

[0028] Figure 4 It is a diagram for obtaining the strawberry dataset of the present invention;

[0029] Figure 5 It is a diagram of the YOLOv8 strawberry recognition result of the present invention;

[0030] Figure 6 It is a structural diagram of the strawberry quality estimation model of the present invention.

[0031] Explanation of the reference numerals in the drawings of the specification:

[0032] 1. Image of a single target strawberry; 2. First convolutional layer; 3. First average pooling layer; 4. Second convolutional layer; 5. Second average pooling layer; 6. Third convolutional layer; 7. Third average pooling layer; 8. Fourth convolutional layer; 9. Fourth average pooling layer; 10. Fifth convolutional layer; 11. Fifth average pooling layer; 12. Sixth convolutional layer; 13. Fully connected layer. Detailed implementation manner

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] The object of the present invention is to provide a method, system, electronic device and medium for determining the weight of strawberry fruits, which can improve the accuracy of estimating the weight of strawberry fruits.

[0035] In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0036] Embodiment 1

[0037] As Figure 1 and Figure 3 shown, the present invention provides a method for determining the weight of strawberry fruits, and the determination method includes:

[0038] Step S1: Obtain the overall images of multiple target strawberries. Specifically, the overall images are RGB images.

[0039] Step S2: Apply a single-stage object detection model to segment each target strawberry in the overall image to obtain multiple images of single target strawberries; specifically, the single-stage object detection model is the YOLOv8 model.

[0040] Step S3: Input the multiple images of single target strawberries into the strawberry quality estimation model to obtain the fresh weights of each single target strawberry; wherein, the strawberry quality estimation model is obtained by training a deep learning model with a training set; the deep learning model includes a feature extraction network and a regression network connected in sequence; the training set includes multiple images of single strawberry samples and the fresh weights of each single strawberry sample.

[0041] Specifically, the feature extraction network is a CNN network.

[0042] Further, the regression network includes a fully connected layer 13 and an activation function.

[0043] In addition, the fresh weight results of strawberries obtained in step S3 are graded into special-grade fruits, first-grade fruits, second-grade fruits, and third-grade fruits: among them, special-grade fruits are those with a single strawberry fruit weight ≥ 20 grams; first-grade fruits are those with 15 grams ≤ single strawberry fruit weight < 20 grams; second-grade fruits are those with 8 grams ≤ single strawberry fruit weight < 15 grams, and third-grade fruits are those with a single strawberry fruit weight < 8 grams.

[0044] In practical applications, as Figure 2 shown, the RGB image of a single strawberry sample is collected, and the fresh weight of the single strawberry sample is measured manually to form an image dataset. Specifically, the RGB camera is installed from a top-down perspective of the strawberry, hanging about 0.70 meters directly above the strawberry. The image format collected is PNG, and the original resolution of the image is 1920×1080 pixels. The computer drives the RGB camera on the image acquisition platform to capture the top-down RGB images of strawberries throughout the entire development and growth stage from small fruits to large fruits. The RGB camera outputs RGB images, and the phenotypic parameters of each strawberry are measured manually to form an image dataset. Further, the RGB image includes fruit size, color, and shape features. The phenotypic parameters of the strawberry sample include fresh weight, and the fresh weight of the strawberry sample is measured by an electronic balance. Among them, the image acquisition platform is a fixed platform used to place the strawberries to be photographed; capturing the top-down RGB images of strawberries throughout the entire development and growth stage is to ensure that the strawberry dataset has data for the entire development and growth stage of strawberries, so as to obtain the fresh weight corresponding to strawberries of different sizes.

[0045] As a specific implementation method, as Figure 4 and Figure 5 shown, based on the top-down RGB image of the strawberry, the YOLOv8 model is used to accurately identify the strawberry image. The strawberry image is cropped using the method of specified central cropping. Specifically, the center point of the strawberry and the four vertices of the minimum circumscribed matrix are identified through the YOLOv8 model, and the strawberry image is cropped in a central cropping manner based on the center point coordinates.

[0046] Specifically, first, the collected strawberry image data is annotated through labelme to obtain the corresponding json file. The dataset is divided into training, validation, and test sets according to 6:2:2. Then, the json file is converted into a txt file that can be read by the YOLO series standard, and the corresponding yaml file is established for model training, validation, and testing. Then, the strawberry image is cropped through the center of the detection frame identified by YOLOv8.

[0047] As a specific implementation, a deep learning model for strawberry quality estimation is constructed, and an error function of the deep learning model for strawberry quality estimation is set. The output of the deep learning model for strawberry quality estimation is a vector with a length of 1, representing the fresh weight. The feature extraction part is based on the independently constructed CNN284 network. The number 284 indicates that the input image is 284×284. The network consists of 6 convolutional layers and 5 average pooling layers, which are used to extract effective strawberry features from RGB images. The regression network part is composed of an FC layer and the activation function ReLU. The activation function ReLU is specifically:

[0048] f(x) = max(0, x);

[0049] where x is the output value of the upper-layer neuron, and f(x) is the input value of the lower-layer neuron

[0050] It can be seen from the activation function ReLU that when the input value is greater than 0, the gradient is always 1, and there is no problem of gradient disappearance, and the convergence speed is fast. When the input value is less than 0, the sparsity of the network is increased. The greater the sparsity, the more representative the features extracted by the network, and the stronger the generalization ability.

[0051] The loss function used for model training in the present invention is:

[0052]

[0053] where S i is the true value of the weight of the i-th single strawberry sample, E i is the predicted value of the weight of the i-th single strawberry sample, and m is the total number of strawberry samples.

[0054] Such as Figure 6As shown, the CNN284 network includes a first convolutional layer 2, a first average pooling layer 3, a second convolutional layer 4, a second average pooling layer 5, a third convolutional layer 6, a third average pooling layer 7, a fourth convolutional layer 8, a fourth average pooling layer 9, a fifth convolutional layer 10, a fifth average pooling layer 11, a sixth convolutional layer 12, and a fully connected layer 13 connected in sequence. The input image is an image 1 of a single target strawberry with a size of 284×284×3, which is input to the first convolutional layer 2, and the output is a feature map of 280×280×16. The feature map of 280×280×16 is input to the first average pooling layer 3, and the output is a feature map of 140×140×16. The feature map of 140×140×16 is input to the second convolutional layer 4, and the output is a feature map of 136×136×32. The feature map of 136×136×32 is input to the second average pooling layer 5, and the output is a feature map of 68×68×32. The feature map of 68×68×32 is input to the third convolutional layer 6, and the output is a feature map of 64×64×64. The feature map of 64×64×64 is input to the third average pooling layer 7, and the output is a feature map of 32×32×64. The feature map of 32×32×64 is input to the fourth convolutional layer 8, and the output is a feature map of 28×28×128. The feature map of 28×28×128 is input to the fourth average pooling layer 9, and the output is a feature map of 14×14×128. The feature map of 14×14×128 is input to the fifth convolutional layer 10, and the output is a feature map of 10×10×256. The feature map of 10×10×256 is input to the fifth average pooling layer 11, and the output is a feature map of 5×5×256. The feature map of 5×5×256 is input to the sixth convolutional layer 12, and the output is a feature map of 1×1×512.

[0055] In the regression network part, the feature map of 1×1×512 is input to the fully connected layer 13. After passing through the activation function ReLU, the output is a feature map of 1×1×1, and the feature map of 1×1×1 is the predicted value of the fresh weight of a single target strawberry.

[0056] The first convolutional layer 2, the second convolutional layer 4, the third convolutional layer 6, the fourth convolutional layer 8, the fifth convolutional layer 10, and the sixth convolutional layer 12 all include a convolutional layer, a BN layer, and an activation function.

[0057] The BN operation is directly connected after the convolution operation. After the training is completed, the parameters of the convolution sum and the parameters of the BN operation will be fixed. BN is a linear transformation after normalizing each pixel point of the input feature map, and the transformation parameters are the same. During the network inference process, BN is fused into the convolution layer. Specifically, the parameters of BN are used to change the parameters at each position in the convolution kernel, which can accelerate the convergence speed during model training, make the model training process more stable, and avoid gradient explosion or gradient disappearance. The activation function follows the BN operation immediately, that is, the convolution operation + BN operation + ReLU operation together constitute one layer.

[0058] As a specific implementation manner, the training process of the deep learning model includes:

[0059] Step 1: Use the image of the single strawberry sample as the input and the fresh weight of the corresponding single strawberry sample as the output to train the deep learning model.

[0060] Step 2: When the number of training times is greater than the preset number of training times, obtain the trained deep learning model and use the trained deep learning model as the strawberry quality estimation model.

[0061] In practical applications, when training the deep learning model, the Dropout method is adopted in the FC layer of the model to randomly discard neurons, and the discard rate is set to 0.5. The Adam optimizer is used, and the number of training epochs, batch size, and initial learning rate of the CNN284 network part are set. Specifically, the initial learning rate of the model is set to 0.0001, and the learning rate drops to 10% of the original learning rate after every 20 training times. The mini-batch size is set to 128, and the maximum number of training times is set to 300. By performing correlation and error analysis on the estimated fresh weight of the strawberry samples, the model parameters are adjusted and optimized in reverse.

[0062] The beneficial effects of the present invention are: The present invention adopts deep learning technology, accurately identifies and segments strawberries through the YOLOv8 model, and based on the visible light RGB image features, uses the deep learning model for strawberry quality estimation to accurately estimate the strawberry quality and classify the group of fruits.

[0063] Embodiment 2

[0064] On the basis of Embodiment 1, Embodiment 2 of the present application provides a result verification method for the estimation and grading method in Embodiment 1:

[0065] To verify the effectiveness of the method in Example 1, in this example, a dataset was collected from the intelligent greenhouse of Lushan Botanical Garden, Chinese Academy of Sciences, Nanchang, Jiangxi Province. The test materials were strawberries of 4 varieties, including Ningyu, Tianxianzui, Hongyan, and Heizhenzhu strawberries, covering all stages from small fruits to large fruits of strawberry fruits. There were a total of 392 strawberry sample data. The dataset was collected according to the method described in Stage 1 and the strawberries were accurately identified and segmented based on the YOLOv8 model. The identified and segmented strawberries were divided into training, validation, and test sets, and then brought into the deep learning model for strawberry quality estimation to estimate the fresh weight, and were graded according to the fresh weight.

[0066] As shown in the prediction results in Table 1 and Table 2, it can be seen that the strawberry quality estimation and grading method shows excellent performance in estimating the fresh weight of strawberries, and the determination coefficient R 2 index reaches 0.975, indicating that the estimation result of the method in Example 1 has a good correlation with the true value; on the one hand, the normalized root mean square error NRMSE reaches 0.080, and the mean absolute percentage error MAPE reaches 0.077, indicating that the proposed strawberry quality estimation and grading method has potential application value.

[0067] Table 1 Test results table of the deep learning model for strawberry quality estimation

[0068] Strawberry performance parameters <![CDATA[R 2 > NRMSE MAPE0.975 Fresh weight 0.975 0.080 0.077

[0069] Table 2 Strawberry grading result table

[0070]

[0071]

[0072]

[0073] It can be seen from the experimental results that the present invention can successfully identify and segment strawberries, accurately estimate the weight for grading and classification, with excellent performance, which can provide a technical basis for optimizing the strawberry growth process. At the same time, it can achieve accurate strawberry grading, contribute to the digitization, automation, and intelligence of strawberry cultivation and picking, and has important significance and application value in agricultural production and research.

[0074] Example 3

[0075] To execute the method corresponding to Example 1 above to achieve the corresponding functions and technical effects, the following provides a strawberry fruit weight determination system, and the determination system includes:

[0076] An acquisition module, configured to acquire overall images of multiple target strawberries.

[0077] A segmentation module, configured to apply a single-stage object detection model to segment each target strawberry in the overall image to obtain multiple images of single target strawberries.

[0078] A prediction module, configured to input images of multiple said single target strawberries into a strawberry quality estimation model to obtain the fresh weight of each single target strawberry; wherein, the strawberry quality estimation model is obtained by training a deep learning model with a training set; the deep learning model includes a feature extraction network and a regression network connected in sequence; the training set includes images of multiple single strawberry samples and the fresh weight of each said single strawberry sample.

[0079] Embodiment 4

[0080] An embodiment of the present invention provides an electronic device, including a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the strawberry fruit weight determination method of Embodiment 1.

[0081] Optionally, the above electronic device may be a server.

[0082] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the strawberry fruit weight determination method of Embodiment 1.

[0083] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0084] Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for determining the weight of strawberry fruits, characterized in that, The determination method includes: Obtain the overall images of multiple target strawberries; the overall images are RGB images; Apply a single-stage object detection model to segment each target strawberry in the overall image to obtain multiple images of single target strawberries; Input the multiple images of single target strawberries into a strawberry quality estimation model to obtain the fresh weights of each single target strawberry; wherein, the strawberry quality estimation model is obtained by training a deep learning model with a training set; the deep learning model includes a feature extraction network and a regression network connected in sequence; the training set includes RGB images of multiple single strawberry samples at the entire developmental growth stage and the fresh weights of each single strawberry sample; the RGB images of multiple single strawberry samples at the entire developmental growth stage include fruit size, color, and shape features; The feature extraction network includes a convolutional layer and a BN layer; fuse the BN layer into the convolutional layer and use the parameters of the BN layer to change the parameters of each position in the convolutional kernel; The FC layer of the deep learning model uses the Dropout method to randomly discard neurons.

2. The strawberry fruit weight determination method according to claim 1, characterized in that, The single-stage object detection model is the YOLOv8 model.

3. The strawberry fruit weight determination method according to claim 1, characterized in that, The feature extraction network is a CNN network.

4. The strawberry fruit weight determination method according to claim 1, characterized in that, The regression network includes a fully connected layer and an activation function.

5. The strawberry fruit weight determination method according to claim 1, characterized in that The training process of the deep learning model includes: Use the images of single strawberry samples as input and the corresponding fresh weights of single strawberry samples as output to train the deep learning model; When the number of training times is greater than the preset number of training times, obtain the trained deep learning model and use the trained deep learning model as the strawberry quality estimation model.

6. A strawberry fruit weight determination system, characterized in that, The determination system includes: An acquisition module for obtaining the overall images of multiple target strawberries; the overall images are RGB images; A segmentation module for applying a single-stage object detection model to segment each target strawberry in the overall image to obtain multiple images of single target strawberries; A prediction module for inputting the multiple images of single target strawberries into a strawberry quality estimation model to obtain the fresh weights of each single target strawberry; wherein, the strawberry quality estimation model is obtained by training a deep learning model with a training set; the deep learning model includes a feature extraction network and a regression network connected in sequence; the training set includes RGB images of multiple single strawberry samples at the entire developmental growth stage and the fresh weights of each single strawberry sample; the RGB images of multiple single strawberry samples at the entire developmental growth stage include fruit size, color, and shape features; The feature extraction network includes a convolutional layer, a BN layer, and an activation function; fuse the BN layer into the convolutional layer and use the parameters of the BN layer to change the parameters of each position in the convolutional kernel; The FC layer of the deep learning model uses the Dropout method to randomly discard neurons.

7. An electronic device, characterized in that, It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the strawberry fruit weight determination method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program which, when executed by a processor, implements the strawberry fruit weight determination method according to any one of claims 1 to 5.

Images