Non-destructive calculation method and system for number of open-air grapes

Abstract

The invention belongs to the technical field of grape management, and particularly relates to an open-air grape grain number non-destructive calculation method, which comprises the following steps: S1, photographing grapes in a fruit setting period or a harvesting period; s2, performing target detection on the grape berries based on a deep learning model to obtain the range of grape fruits; s3, based on the result of S2, obtaining a plurality of parameters of the grapes, including morphological characteristic parameters, large and small grains and abnormal grains; and S4, establishing a relationship between the parameters obtained in the S3 and an actual grape number, and applying a result to other grape bunch image counting. The invention further provides a system for non-destructive calculation of the number of open-air grapes. According to the method, multi-feature fusion counting is realized, the relationship between a plurality of morphological features and the number of grape fruits is adopted, and the method is stable and reliable.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of grape planting, and particularly relates to a non-destructive counting method and system for open-air grape grains. BACKGROUND

[0002] Grape is an important economic crop, which has important significance for agricultural production and the income of fruit farmers. The fruit grain number statistics of grape clusters in the fruit setting period (the stage after pollination and fertilization, when young fruits begin to develop stably) and the harvest period (the stage when fruits are mature and can be harvested) are crucial for grape planting management. For example, fruit grain counting can be used for early yield prediction of grapes, developing thinning schemes, evaluating the effects of cultivation measures, and monitoring diseases, pests and adversity. However, the traditional fruit grain counting method usually relies on manual completion, which is time-consuming and labor-intensive, especially in large-scale vineyards, where the cost of manual counting is high. At the same time, due to the limitation of sampling range, the traditional method is prone to inaccurate evaluation and is easily affected by subjective judgment. Insufficient sampling may lead to inaccurate prediction and is easily affected by subjective judgment, and there are problems such as non-uniform counting standards and poor repeatability.

[0003] With the development of computer technology, researchers tend to use computer vision and image processing technology to solve the problem of grape fruit grain counting. This method mainly uses high-definition cameras or unmanned aerial vehicles to capture grape cluster images, and combines deep learning models to realize automatic identification and counting of fruit grains. Although computer vision methods have obvious advantages in efficiency and objectivity compared with traditional methods, there are still many challenges in their practical application.

[0004] 1) Fruit grain overlap and branch and leaf shading lead to missed detection or misjudgment. In the actual environment, images can only obtain two-dimensional features of the fruit cluster, while the fruit cluster itself has a three-dimensional structure, and the fruit grains on the same fruit cluster may be mutually occluded. At the same time, the branches and leaves of the grape may also occlude the fruit cluster, causing the occluded part to not be presented in the captured image, thereby leading to a low counting result.

[0005] 2) Illumination conditions affect recognition accuracy. Under strong light conditions, the image captured by the camera may be overexposed, resulting in high brightness and difficulty in distinguishing fruit grain features; while the shadow produced by the branches and leaves will reduce the local brightness of the image, and the contrast between the fruit grains and the background will be weakened, which also affects the counting accuracy.

[0006] 3) High requirements for device resolution and computing power. In the early fruit setting period, the size of grape fruit grains is usually 2-4 mm, and the camera needs to have high spatial resolution to clearly distinguish the fruit grain morphology; at the same time, the processing of high-resolution images also puts high requirements on the computing power of the computing device.

[0007] 4) Different grape varieties model generalization ability is insufficient. Due to the large difference between grape varieties, for example, the tight and loose cluster has significant difference in spatial distribution characteristics, the existing model often needs to be retrained for different varieties, and it is difficult to directly migrate and apply. SUMMARY

[0008] In view of the problems in the background art, the present application provides a grape nondestructive counting method based on a deep learning method, comprising: S1, taking a photo of the grape at the fruit setting period or the harvest period; S2, target detection of the grape berry based on a deep learning model to obtain the range of the grape berry; S3, based on the result of S2, obtaining a plurality of parameters of the grape, including: morphological feature parameters, large and small particles and abnormal particles; S4, establishing the relationship between the parameters obtained in S3 and the actual number of grapes, and applying the results to other grape cluster image counting.

[0009] The present application also provides a non-destructive calculation system for open-air grape particle number, comprising a computer executable program, which, when executed, completes the method as described above.

[0010] The technical effects of the present application include: the present application establishes the prediction relationship between a plurality of morphological feature parameters of the grape cluster and the total number of grape berries, completes the low computational power counting of grape berries, realizes the nondestructive counting of grapes at the fruit setting period and the harvest period, and reduces the cost.

[0011] The present application realizes the nondestructive counting of grape berries, provides necessary input data for farmland intelligent management, and provides data basis and technical support for grape yield estimation, and helps orchard management decision and fine cultivation.

[0012] In one embodiment, the present application provides a reference calibration board. A calibration board with a fixed size is set during the shooting process, which provides a reliable reference benchmark for feature calculation (such as scale conversion and area measurement) when counting grape berries, significantly improving the accuracy and consistency of counting.

[0013] In one embodiment, the present application realizes multi-feature fusion counting. The present application uses the relationship between a plurality of morphological features and the number of grape berries, which is stable and reliable. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to make the present application easier to understand, the present application will be described in more detail by referring to the specific embodiments shown in the accompanying drawings. These drawings only depict typical embodiments of the present application and should not be considered as limiting the scope of protection of the present application.

[0015] Figure 1 The flowchart of the method of the present application.

[0016] Figure 2Flow chart of one embodiment of the method of the present application.

[0017] Figure 3 A structural diagram of the calibration board of the present application is shown.

[0018] Figure 4 A schematic diagram of the photographing of the method of the present application is shown.

[0019] Figure 5 An experimental environment diagram for verifying the method of the present application is shown.

[0020] Figure 6 An experimental environment diagram for verifying the method of the present application is shown.

[0021] Figure 7 An effect diagram of one example of the method of the present application is shown. DETAILED DESCRIPTION

[0022] Embodiments of the present application are described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present application and implement it, but the listed examples are not limiting of the present application, and the examples and technical features in the examples described below can be combined with each other without conflict, wherein the same components are denoted by the same reference signs.

[0023] As shown in Figures 1-2 , the method of the present application comprises: S1: field grape photo data collection.

[0024] S1.1. In the grape fruit setting period, randomly select grape clusters with representative (including: grape clusters with uniform growth and grape clusters with uneven growth), and ensure the diversity of the samples. Place the peduncle of the selected cluster in the radial notch of the PVC background plate, so that the entire grape cluster is placed in front of the PVC background plate, keep the grape main stem parallel to the background plate and the calibration plate, and avoid imaging asymmetry or partial berries being blocked due to the inclination or rotation of the grape cluster.

[0025] The PVC background plate is preferably square, black background, about 2mm thick, with a radial single-sided notch, the notch length is about 8cm, for fixing the grape peduncle; the background plate has a black and white checkerboard pattern calibration plate on it, the side length of a single square is 1cm, for image scale calibration. Figure 3 A sample of a PVC background plate is shown.

[0026] S1.2. Take a photo of the grape, keep the camera lens optical axis perpendicular to the plane of the background plate when taking the photo, and take a front image to avoid pitch angle deviation caused by camera inclination and reduce perspective distortion; during the photo taking process, it should be ensured that the background plate is full of the picture and the calibration plate is completely visible; avoid taking pictures in the early morning with fog or in the late afternoon to ensure the imaging quality.

[0027] Preferably, a smartphone is used for shooting, and the smartphone is fixed on a support for shooting to ensure imaging stability and measurement accuracy. Figure 4 The correct angle for shooting is shown.

[0028] S1.3. Put all the photographed grape clusters into a frozen bag and record the number, and take back to the laboratory for subsequent counting method training and result verification.

[0029] S2: Target detection of grapes based on deep learning algorithm to obtain the range of each grape berry.

[0030] S2.1. Data segmentation, the obtained RGB image sample set of grape clusters is divided into training data set (30%) and test data set (70%) according to the preset proportion. The training data set is used for model training and parameter optimization, and the test data set is used for evaluating the generalization performance of the model.

[0031] S2.2, label the training data set. LabelMe or LabelX tools are used to manually label the position and area of grape berries on the training data set images.

[0032] S2.3. Take the labeled data set as the training input of the model, output the boundary box of single grape berry, locate the label, and record the boundary box position and confidence of each grape berry.

[0033] Preferably, a convolutional neural network is used as a training model, and the labeled data is input into the convolutional neural network model, and the convolutional neural network model outputs the boundary box of single grape berry. The reason for using a convolutional neural network (CNN) deep learning model is that CNN automatically extracts high-level semantic features of images through multiple layers of convolution, pooling and nonlinear activation function, and establishes a classification decision boundary, without the need for manual design of feature extraction algorithm, which significantly reduces the influence of subjective intervention on the detection result. The process of CNN is more efficient, and it has good adaptability to light changes, occlusion and complex background.

[0034] The confidence score is generated by the target detection model for each bounding box, which is the degree of certainty of the model that the target exists in the box.

[0035] S3: Filtering the results of S2 to obtain grape parameters, including morphological characteristics, size particles, and abnormal particles.

[0036] S3.1, get the accurate grape mask by greenness index segmentation.

[0037] 1) According to the set confidence threshold, the effective bounding box is screened, and the area of all effective bounding boxes is set to True on the mask to obtain the preliminary grape area mask.

[0038] 2) The preliminary mask is subjected to convex hull processing to obtain the minimum convex polygon region containing all grapes.

[0039] 3) Based on the greenness index, the grape is separated from the background, and the Excess Green (ExG) index is preferably used for segmentation. ExG is a color space-based vegetation index that enhances the contrast between vegetation (green area) and background to achieve segmentation. The ExG value of vegetation is significantly higher than that of non-vegetation area, and its calculation formula is: where R, G, and B are the normalized pixel values of the single band.

[0040] 4) Combine the convex hull mask and ExG threshold, and further refine the grape area based on the morphological processing method. Perform erosion and dilation operations on the mask to remove small noise points and obtain a cleaner grape mask.

[0041] S3.2, according to the confidence score, the number of visible grape berries is screened: based on the results of step S2.3, according to the set confidence threshold, the effective bounding box is screened, and the number of effective bounding boxes is the number of visible grape berries.

[0042] S3.3, according to the number of pixels of the grape mask and the image spatial resolution, the grape surface area is calculated.

[0043] 1) Obtain the image spatial resolution, detect the calibration board on the picture, and calculate the spatial resolution (cm / pixel) of the image, which is obtained by the ratio of the actual length of the chessboard to the corresponding length on the picture, and the calculation formula is: , where n_pixel is the number of pixels corresponding to a 1 cm side length of the chessboard in the image; 2) Obtain the visible surface area of a single grape (i.e., the visible grape surface area in the image), and the calculation formula is: , where is the image coordinates of the upper left corner and the lower right corner of the bounding box, is the pixel area of a single grape; 3) Obtain the real visible surface area of the grape, and the formula is: , where n is the number of visible grape berries in S3.2.

[0044] S3.4, calculate the average surface area of grape berries, i.e. the ratio of the visual surface area of the grapes to the number of grape berries; , S3.5, calculate the shape index, which is used to describe the overall contour shape of the grape cluster. The eccentricity is selected as the evaluation parameter. The calculation method is: the ratio of the major axis to the minor axis of the grape mask (the major axis is the longitudinal distance from the end of the stem to the cluster tip, and the minor axis is the maximum transverse distance perpendicular to the major axis). Among them, the value closer to 1 indicates that the grape is closer to spherical, and the larger the value, the more elongated the berry shape.

[0045] S3.6, calculate the tightness index. The tightness index reflects the tightness of the fruit arrangement.

[0046] The two-dimensional image method is used to calculate the projection area ratio, i.e. the ratio of the sum of the projection areas of all berries to the total area of the grape cluster contour. The calculation formula of this index is: , where is the area (number of pixels) of the grape mask, is the area (number of pixels) of the grape convex hull mask, and the higher the ratio, the tighter it is.

[0047] S4. Establish the relationship between the parameters obtained in S3 and the actual number of grapes , and apply the results to other grape cluster image counting.

[0048] 1) Filter the results of S2.4 to remove grape single-particle bounding boxes below the confidence threshold; 2) Based on the area calculation results of single grape berries in S3.3 (using pixel area or actual area cm 2 ), statistics such as maximum value, minimum value, mean value, standard deviation, etc. are calculated; 3) Perform abnormal particle detection and mark small particles according to the set threshold; 4) Perform Gaussian fitting to generate a visual histogram of the distribution of grape berry sizes.

[0049] S5: Use a photobox to obtain the number of berries of a grape cluster, which is used to establish a model and verify the accuracy of the model prediction.

[0050] Preferably, a photobox with a light source is designed, as shown in Figure 5 and Figure 6 , which adopts a detachable design for easy installation and field movement. The photobox has a black background plate inside, and a PVC bracket is provided at the upper part for fixing a smartphone, which ensures image acquisition under controlled light conditions and reduces the influence of external environmental changes on imaging quality.

[0051] S5.1, place a tray with a black bottom on the black background at the bottom of the image collection box, place the whole grape bunch on the tray, and place the mobile phone on the top PVC plate to take a photo, and record the label of the grape bunch at the same time; S5.2, separate the fruit grains and stems of the grape bunch in S5.1 in the tray, only keep the fruit grains, and disperse the fruit grains to avoid adhesion of the fruit grains, and take a photo in the same way as S5.1 to count; the relationship between the total number of real grapes and the grape characteristic parameters obtained in S3-S4 is established, and the XGBoost model is preferably selected; the pictures are grouped in a ratio of 3:7, cross-validation is performed, and the model is constructed. The relationship constructed is: , x represents the characteristic parameters of S3-S4, including the number of visible grape grains, the visible surface area of grapes, the average surface area of grape grains, the shape index eccentricity, the compactness index, and the size of grains, y represents the real number of grapes obtained by the image collection box.

[0052] S5.3, verify the model results with the images in the test data set.

[0053] S6: repeat steps S1-S5 during the harvest period to obtain a grape counting model during the harvest period.

[0054] The present application has been tested on 3 varieties of grapes in a certain country vineyard, and the counting results have been verified by the number of grapes when the grapes are dispersed in the image collection box, such as Figure 7 The results show that the consistency of the present application and the artificial counting results is very high (correlation coefficient R 2 =0.99), and has a wide application prospect.

[0055] The above-described embodiments are only the preferred specific embodiments of the present application, and the phrases "in one embodiment", "in another embodiment", "in yet another embodiment" or "in other embodiments" are all used to refer to one or more of the same or different embodiments according to the present disclosure. The usual changes and replacements made by those skilled in the art within the scope of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A non-destructive method for calculating the number of grapes in open-field cultivation, characterized in that, include: S1, Take photos of grapes during the fruit setting or harvesting period; S2, based on a deep learning model, performs target detection on grape berries to obtain the range of grape berries; S3, based on the results of S2, obtains multiple parameters of grapes, including: morphological characteristic parameters, grape size, and abnormal grapes; S4 establishes the relationship between the parameters obtained in S3 and the actual number of grapes, and applies the results to the counting of other grape bunch images.

2. The non-destructive calculation method for the number of grapes in open-field cultivation according to claim 1, characterized in that, Step S2 includes: S2.1, Perform data segmentation: Divide the acquired RGB image sample set of grape bunches into training dataset and test dataset according to a preset ratio; S2.2, Annotate the training dataset: Manually annotate the location and region of grape berries on the training dataset images; S2.3 uses the labeled dataset as the training input for the model, outputs the bounding box of a single grape, locates and labels the bounding box position and confidence level of each grape.

3. The non-destructive calculation method for the number of grapes in open-field cultivation according to claim 1, characterized in that, Step S3 includes: S3.1, the grape mask is obtained by segmentation based on the greenness index; S3.2, Filter the number of visible grape berries based on confidence scores; S3.3, Calculate the surface area of ​​the grapes based on the number of pixels in the grape mask and the spatial resolution of the image; S3.4 Calculate the average surface area of ​​grape berries, which is the ratio of the visible surface area of ​​grapes to the number of grape berries. S3.5, calculate the shape index; S3.6, Calculate the compactness index.

4. The non-destructive calculation method for the number of grapes in open-field cultivation according to claim 1, characterized in that, The steps to find S3.1 include: 1) Filter valid bounding boxes according to the set confidence threshold, and set the regions of all valid bounding boxes to true on the mask to obtain the initial region mask of the grape. 2) Perform convex hull processing on the initial mask to obtain the smallest convex polygon region containing all the grapes; 3) Calculate the ExG exponent for segmentation. The calculation formula is as follows: , where R, G, and B are the normalized pixel values; 4) Combining convex hull mask and ExG threshold, based on morphological processing methods, the mask is eroded and expanded to remove small noise.

5. The non-destructive calculation method for the number of grapes in open-field cultivation according to claim 1, characterized in that, Step S3.3 includes: 1) Obtain the image spatial resolution using the following formula: , Where n_pixel is the number of pixels corresponding to a chessboard side length of 1cm in the image; 2) Obtain the visible surface area of ​​a single grape using the following formula: , in These are the coordinates of the top-left and bottom-right corners of the bounding box. The pixel area of ​​a single grape; 3) The formula for the actual visible surface area of ​​grapes is: 。 6. The non-destructive calculation method for the number of grapes in open-field cultivation according to claim 1, characterized in that, Also includes: S5: Obtain the number of grapes in the grape bunch to verify the accuracy of the model's prediction; S6: Repeat steps S1-S5 during the harvest period to obtain the grape counting model during the harvest period.

7. The non-destructive calculation method for the number of open-field grapes according to claim 5, characterized in that, Step S5 includes: S5.1, Take a picture of the whole bunch of grapes and record the label of the grape bunch at the same time; S5.2 Separate the grapes from the stems in S5.1, keeping only the grapes. Separate the grapes to prevent them from sticking together. Take a picture and count them. Establish a relationship between the total number and the morphological characteristics, size, etc. of the grapes. S5.3 Validate the model results using images from the test dataset.

8. The non-destructive calculation method for the number of grapes in open-field cultivation according to claim 1, characterized in that, In step S5.2, the relationship constructed is as follows: x represents the number of visible grape berries, the visible surface area of ​​grapes, the average surface area of ​​grape berries, the shape index eccentricity, the firmness index, and the size of the berries, and y represents the actual number of grapes in S5.

1.

9. A non-destructive calculation system for the number of grapes in open-field cultivation, characterized in that, include: A computer-executable program, which, when run, performs the method as described in any one of claims 1-9.

10. The non-destructive calculation system for the number of grapes in open fields according to claim 9, characterized in that, include: The background board has a black background and a radial single-sided cut for fixing the grape bunch axis. The background board has a black and white checkerboard patterned calibration plate.

Images