Vegetable and fruit period grape growth state inspection method based on machine vision
By segmenting and extracting features from images of grape bunches using machine vision technology, the shortcomings of manual data collection methods are overcome, enabling rapid and accurate detection of grape growth status and supporting intelligent decision-making.
Patent Information
- Application Number
- CN202410516272.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-26
- Publication Date
- 2026-01-09
AI Technical Summary
In current grape research during the fruit and vegetable season, data collection is done manually, resulting in a large workload, inconsistent standards, and hindering the intelligentization of grape research. Furthermore, the problem of fruit and leaf shading remains unresolved.
A machine vision-based approach was adopted to segment and extract features from grape bunch images by training a target detection model, calculate the central branch curve of the grape bunch and grape information, screen out real grapes, and calculate growth status indicators such as bunch type, number of fruits and compactness.
It enables rapid and accurate detection of grape growth status, supports intelligent decision-making, reduces manual operation, and improves the standardization and efficiency of detection.
Smart Images
Figure CN121305486A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of machine vision and image processing, and particularly relates to a method for inspecting the growth state of vegetable and fruit period grapes based on machine vision. BACKGROUND
[0002] China is one of the largest grape producing countries in the world, and in recent years the yield and planting area of grapes have been increasing. However, in the research on vegetable and fruit period grapes, data collection is generally carried out manually. However, due to the similar color of the fruit of the vegetable and fruit period grape and the grape leaf, the small size of the fruit, the easy blocking of the fruit by the branch and the grape leaf, and the possible blocking and adhesion between the grape clusters, these factors greatly affect the decision on the vegetable and fruit period grape. In addition, this collection method is labor-intensive, and the standards are not uniform, which seriously affects the intelligent process of the research on vegetable and fruit period grapes.
[0003] Machine vision technology is a technology that extracts features by collecting images of target objects, and can qualitatively or quantitatively analyze the target. It has the advantages of fast speed, non-destructiveness and low cost. The application of machine vision recognition algorithm to the research on vegetable and fruit period grapes can realize real-time monitoring of a large area and achieve the purpose of classifying single grape fruits, which plays a significant supporting role in the decision on vegetable and fruit period grapes. Therefore, the present application provides a method for inspecting the growth state of vegetable and fruit period grapes based on machine vision, which can comprehensively, conveniently, quickly and standardly detect vegetable and fruit period grapes. SUMMARY
[0004] The present application mainly provides a method for inspecting the growth state of vegetable and fruit period grapes based on machine vision, which can comprehensively, conveniently, quickly and standardly detect vegetable and fruit period grapes.
[0005] To achieve the above-mentioned purpose, the present application provides the following solutions:
[0006] The present application relates to a method for inspecting the growth state of vegetable and fruit period grapes based on machine vision, which comprises the following steps:
[0007] The grape cluster image is input into the first target detection model trained, to obtain a grape head image and a grape cluster image, and a first grape cluster center branch curve is calculated according to the grape head image and the grape cluster image;
[0008] The grape cluster image is input into the second target detection model trained, to obtain grape fruit information, and the grape fruit information includes the boundary coordinates, the confidence and the category of the grape fruit;
[0009] The grape fruit information is filtered to obtain grape fruit, including setting a confidence threshold, if the confidence of the grape fruit is greater than the confidence threshold, the grape fruit is grape fruit, otherwise it is not grape fruit.
[0010] calculating the growth state of the grape cluster, the growth state comprising: a grape cluster cluster shape, a number of grape fruits, a grape fruit compactness, and grape weak fruit information, the grape cluster cluster shape comprising a maximum envelope curve of a grape cluster and a second grape cluster center branch curve, the second grape cluster center branch curve being a curve obtained by correcting a first grape cluster center branch curve according to the maximum envelope curve.
[0011] Preferably, before inputting the grape cluster image into the first target detection model trained, the following steps are included:
[0012] Collecting grape cluster images at different angles and marking them as a training set, the grape cluster image containing a grape cluster and a grape cluster, the grape cluster containing grape fruits;
[0013] K-means clustering is performed on the aspect ratio of the grape cluster in the training set to obtain k1 clusters, and the coordinate proportion value of the cluster tip relative to the center point and the aspect ratio expectation value of each cluster are calculated in turn;
[0014] K-means clustering is performed on the aspect ratio of the grape fruit in the training set to obtain k2 clusters, and the length expectation value and the width expectation value of each cluster are calculated in turn;
[0015] K-means clustering is performed on the center point of the grape fruit in the training set to obtain k3 clusters, and the center point coordinates of each cluster are calculated in turn.
[0016] Preferably, the second target detection model is a feature mapping branch added to the feature extraction network, the feature mapping branch being constructed according to the training set and mapping the candidate box to the feature extraction network.
[0017] Preferably, the candidate box is constructed according to the training set as follows: k3 center points are constructed, the coordinates of the center points being the center point coordinates of the k3 clusters, and k2 candidate boxes are constructed for each center point, the length and width of the candidate box being the length expectation value and the width expectation value of the k2 clusters.
[0018] Preferably, the candidate box is mapped to the feature extraction network as follows: the center point coordinates, length and width of the candidate box are mapped to the feature map extracted by the feature extraction network, and the mapping ratio is the ratio of the grape cluster image to the feature map extracted by the feature extraction network.
[0019] Preferably, the calculation of the first grape cluster center branch curve according to the grape cluster image and the grape cluster image comprises:
[0020] The boundary coordinates of the grape cluster in the grape cluster image are obtained by using an edge detection algorithm, and a straight line fitting operation is performed to obtain a grape cluster fitting curve;
[0021] The aspect ratio of the grape cluster image is calculated, the class cluster where the grape cluster image is located is determined, the coordinate of the grape cluster tip point is calculated according to the coordinate proportion value of the tip point relative to the center point in the class cluster where the grape cluster image is located, the coordinate of the grape cluster tip point is fitted with the grape cluster base curve to obtain a first grape cluster center stem curve.
[0022] Preferably, the calculation method of the maximum envelope curve is that boundary coordinates of all grape fruits are constituted into a data set, maximum values and minimum values of the column coordinates under the condition that the row coordinates of the grape fruits are the same are obtained as the column boundary coordinates of the maximum envelope curve under the row coordinates, or maximum values and minimum values of the row coordinates under the condition that the column coordinates of the grape fruits are the same are obtained as the row boundary coordinates of the maximum envelope curve under the column coordinates.
[0023] The modification of the first grape cluster center stem curve according to the maximum envelope curve is that the center point coordinate of the maximum envelope curve is calculated, the center point coordinate is fitted with the first grape cluster center stem curve to obtain a second grape cluster center stem curve.
[0024] Preferably, the number of grape fruits is the number of grape fruits to be determined whose confidence is greater than a confidence threshold.
[0025] The compactness of the grape fruit is that the center points of the grape fruits are clustered according to the distance to obtain the number of grape fruits in each category, the area surrounded by the envelope curve of each category of grape fruits is calculated, and the ratio of the area surrounded by the envelope curve of each category of grape fruits to the number of grape fruits in the category is taken as the compactness of the grape fruits in the category.
[0026] The grape weak fruit information is that the grape weak fruits are screened according to the grape fruit categories, and the grape weak fruit information is the boundary coordinates of the grape weak fruits.
[0027] Preferably, the grape cluster image acquisition method is that the inspection robot moves along a preset path to collect grape cluster images of target objects; when the inspection robot moves along the preset path, the current environment information is collected, it is determined whether there is an obstacle in front, and if there is an obstacle, the moving path of the inspection robot is adjusted.
[0028] Preferably, the grape cluster image of the target object includes: inputting the image collected by the inspection robot into a grape cluster detection model to determine whether there is a grape cluster, and if there is a grape cluster, collecting grape cluster images at different angles.
[0029] The application relates to a machine vision-based inspection method for the growth state of grape in the vegetable and fruit period. The first target detection model and the second target detection model are used to obtain the growth state of grape bunches, including grape cluster type, grape fruit quantity, grape fruit compactness and grape weak fruit information, which plays an important decision support role in the processing mode of grape in the vegetable and fruit period. The second target detection model is added with a feature mapping branch constructed according to the predicted possible position of grape fruit and the potential length-width ratio of the grape fruit on the basis of the first target detection model in which the grape cluster has been segmented, so that the convergence speed of the model can be accelerated while the accuracy is ensured. BRIEF DESCRIPTION OF DRAWINGS
[0030] Figure 1 The application relates to a machine vision-based inspection method for the growth state of grape in the vegetable and fruit period. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be specifically described below in combination with examples and drawings, but the protection scope of the application is not limited thereto.
[0032] The terms "first", "second" and the like can be used herein to describe various concepts, but these concepts are not limited by these terms unless specifically stated. These terms are only used to distinguish one concept from another. For example, the first target detection model can be referred to as the second target detection model without departing from the scope of the application, and similarly, the second target detection model can be referred to as the first target detection model.
[0033] Referring to the accompanying Figure 1 The application relates to a machine vision-based inspection method for the growth state of grape in the vegetable and fruit period.
[0034] S101. The grape cluster image is input to the trained first target detection model to obtain grape cluster images and grape cluster images, and the first grape cluster center branch curve is calculated according to the grape cluster images and grape cluster images;
[0035] The machine learning model learns patterns and rules through samples in the training set so as to be able to make predictions for unseen data. The training set usually contains a large number of samples and covers various situations that the model may encounter in the application stage. Therefore, before the grape cluster image is input to the trained first target detection model, a training set needs to be constructed for model training.
[0036] In the application, grape cluster images of different angles are collected and labeled as a training set. The grape cluster images contain grape clusters and grape clusters, and the grape clusters contain grape fruits. Labeling refers to adding labels, attributes and other information to sample data, which can be divided into manual annotation and automatic annotation, and is not limited herein.
[0037] The aspect ratio of the grape cluster in the training set is clustered by K-means clustering to obtain k1 clusters, and the coordinate proportion value of the cluster tip relative to the center point and the aspect ratio expectation value of each cluster are calculated in turn. The specific method is: the coordinate proportion value of the cluster tip relative to the center point of each sample in the cluster is calculated in turn, and then the average value is taken as the coordinate proportion value of the cluster, wherein the coordinate proportion value includes the horizontal coordinate proportion and the vertical coordinate proportion. Similarly, the aspect ratio of each sample in the cluster is calculated in turn, and then the average value is taken as the aspect ratio expectation value of the cluster.
[0038] The aspect ratio of the grape cluster in the training set is clustered by K-means clustering to obtain k1 clusters, and the coordinate proportion value of the cluster tip relative to the center point and the aspect ratio expectation value of each cluster are calculated in turn. The specific method is: the coordinate proportion value of the cluster tip relative to the center point of each sample in the cluster is calculated in turn, and then the average value is taken as the coordinate proportion value of the cluster, wherein the coordinate proportion value includes the horizontal coordinate proportion and the vertical coordinate proportion. Similarly, the aspect ratio of each sample in the cluster is calculated in turn, and then the average value is taken as the aspect ratio expectation value of the cluster.
[0039] The center point of the grape cluster in the training set is clustered by K-means clustering to obtain k3 clusters, and the center point coordinates of each cluster are calculated in turn. The specific method is: the center point coordinates of each sample in the cluster are calculated in turn, and then the average value is taken as the center point coordinates of the cluster.
[0040] In an embodiment, the grape cluster image is a grape cluster image collected manually. In another embodiment, it is obtained by intelligent collection of a patrol robot. The patrol robot moves along a preset path to collect grape cluster images of target objects; and when the patrol robot moves along the preset path, it collects current environmental information to determine whether there is an obstacle in front, and if there is an obstacle, it adjusts the moving path of the patrol robot.
[0041] In an embodiment, collecting grape cluster images of target objects includes: inputting the image collected by the patrol robot into a grape cluster detection model to determine whether there is a grape cluster, and if there is a grape cluster, collecting grape cluster images at different angles. It should be noted that whether there is a grape cluster can be determined by the grape cluster detection model, which can be determined by calculating the aspect ratio of the image using a traditional image algorithm; or it can be determined by training a target detection network, which is not limited here.
[0042] In an embodiment, one or more sensors are installed on the inspection robot to collect current environmental information, such as an infrared sensor installed to detect obstacle information through the principle of triangulation, an ultrasonic sensor installed to detect obstacle information by measuring the time of sound wave reflection, or a laser radar installed to provide more accurate distance and position information to detect obstacle information; the inspection robot updates the target trajectory in real time to bypass the obstacle due to the input of the sensor sensing the presence of the obstacle. Since the obstacle avoidance algorithm is a mature solution, further description is not provided here.
[0043] In an embodiment, a gimbal system is provided on the inspection robot, and the collection device, such as a camera, is installed on the gimbal. The camera can capture images at different angles by adjusting the angle of the gimbal.
[0044] The target detection model combines image classification and positioning, uses a target detection deep learning model, and can identify one or more objects in an image and predict their positions. In one embodiment, the first target detection model of the present application can be a target detection model such as YOLO V5, YOLO Vx, Cascade, Faster RCNN, etc., which is not limited here. According to the characteristics of grape cluster images and grape cluster images, the first target detection model can accurately separate the two parts from the grape cluster image.
[0045] S102. Input the grape cluster image into the trained second target detection model to obtain the pending grape fruit information, the pending grape fruit information including the boundary coordinates, confidence and category of the pending grape fruit; wherein the category of the grape fruit is divided into strong fruit, weak fruit, two fruit and three fruit according to actual experience;
[0046] In one embodiment, the second target detection model adds a feature mapping branch to the feature extraction network. The feature mapping branch constructs candidate boxes according to the training set and maps the candidate boxes to the feature extraction network. The feature extraction network can be VGG, Googlenet, ResNet50, MobileNetV2, etc., which is not limited here.
[0047] In an embodiment, constructing candidate boxes according to the training set is: constructing k3 center points, the coordinates of the center points being the center point coordinates of k3 class clusters, constructing k2 candidate boxes for each center point, the length and width of the candidate boxes being the length expectation value and width expectation of k2 class clusters.
[0048] It should be noted that the center point coordinates are k3 center point coordinates multiplied by the center point coordinates of the grape cluster image, and the coordinates of the k2 candidate boxes are the length expectation value and width expectation of k2 class clusters multiplied by the length and width of the grape cluster image.
[0049] It can be understood that the near-far situation may occur in the process of collecting images, so the size and center point coordinates of the grape fruit may be different due to the difference between the near and far images. In the application, the proportional method can effectively avoid the above problems.
[0050] In an embodiment, the mapping of the candidate box to the feature extraction network is: mapping the center point coordinates, length and width of the candidate box to the feature map extracted by the feature extraction network, and the mapping ratio is the ratio of the grape cluster image to the feature map extracted by the feature extraction network. Based on the prior knowledge of the training set, the possible position of the grape fruit and its potential length-width ratio can be predicted. Mapping the predicted grape fruit position and length-width to the original feature extraction network can accelerate the convergence speed of the model while ensuring the accuracy.
[0051] It should be noted that after the candidate box is mapped to the feature map, bilinear interpolation processing is required to align the image size for subsequent classification processing.
[0052] S103. Screening the grape fruit according to the to-be-determined grape fruit information, including: pre-setting a confidence threshold, if the confidence of the to-be-determined grape fruit is greater than the confidence threshold, the to-be-determined grape fruit is a grape fruit, otherwise it is not a grape fruit;
[0053] It can be understood that not all detection boxes contain real grape fruits, so it is necessary to screen them out to exclude non-real grape fruits and screen out real grape fruits for subsequent judgment of the growth state of the grape cluster.
[0054] S104. Calculating the growth state of the grape cluster, the growth state including: grape cluster cluster type, number of grape fruits, grape fruit compactness, and grape weak fruit information, the grape cluster cluster type including a maximum envelope curve of the grape cluster and a second grape cluster center branch curve, the second grape cluster center branch curve being a curve obtained by correcting the first grape cluster center branch curve according to the maximum envelope curve.
[0055] In an embodiment, the method for obtaining the second grape cluster center branch curve can include the following steps:
[0056] S1041. Calculating the first grape cluster center branch curve according to the grape cluster image and the grape cluster image, and the specific calculation method is:
[0057] The edge detection algorithm is used to obtain the boundary coordinates of the grape cluster in the grape cluster image, and a straight line fitting operation is performed to obtain a grape cluster fitting curve. It should be noted that the edge detection algorithm can be Sobel, Canny, Laplacian, etc., and the straight line fitting can use Hough transform, least squares method, etc., which are not limited here.
[0058] The aspect ratio of the grape cluster image is calculated, the class cluster where the grape cluster image is located is determined, the coordinate of the grape cluster tip point is calculated according to the coordinate proportion value of the tip point relative to the center point in the class cluster, the coordinate of the grape cluster tip point is fitted with the grape cluster base fitting curve to obtain a first grape cluster center stem curve, and the specific method for determining the class cluster where the grape cluster image is located is as follows: the aspect ratio of the grape cluster image is calculated, the absolute value of the difference between the aspect ratio and the aspect ratio of each class cluster of k1 class clusters is calculated in sequence, and the class cluster with the minimum absolute value of the difference is taken as the class cluster to which the grape cluster image belongs.
[0059] S1042. Calculating the maximum envelope curve: constructing a data set with the boundary coordinates of all grape fruits, obtaining the maximum value and the minimum value of the column coordinate under the condition that the row coordinates of the grape fruits are the same as the maximum envelope curve, or obtaining the maximum value and the minimum value of the row coordinate under the condition that the column coordinates of the grape fruits are the same as the maximum envelope curve; both methods can be used, and no limitation is made herein.
[0060] S1043. According to the maximum envelope curve, the first grape cluster center stem curve is modified to obtain a second grape cluster center stem curve, and the modification method is as follows: the center point coordinate of the maximum envelope curve is calculated, and the center point coordinate is fitted with the first grape cluster center stem curve to obtain the second grape cluster center stem curve.
[0061] In one embodiment, the number of grape fruits is the number of grape fruits with a confidence degree greater than a confidence degree threshold; one calculation method is as follows: the grape fruits are screened according to the confidence degree, the number of each type of grape fruit is determined, for example, the number of healthy fruits is a, the number of weak fruits is b, the number of two-fruit clusters is c, and the number of three-fruit clusters is d, and the number of grape fruits is a+b+2c+3d.
[0062] In one embodiment, the compactness of the grape fruit is as follows: the center points of the grape fruits are clustered according to the distance to obtain the number of grape fruits in each category, the area surrounded by the envelope curve of each category of grape fruit is calculated, and the ratio of the area surrounded by the envelope curve of each category of grape fruit to the number of grape fruits in the category is taken as the compactness of the grape fruit in the category.
[0063] In one embodiment, the grape weak fruit information is as follows: the grape fruits are screened according to the confidence degree, the grape weak fruits are screened according to the grape fruit category, and the grape weak fruit information is the boundary coordinates of the grape weak fruits.
[0064] The final output of the target detection model is the grape cluster shape, the number of grape fruits, the compactness of the grape fruits, and the grape weak fruit information, which is fed back to the farm to enable the farm to timely understand the current grape growth information and facilitate the decision-making of subsequent picking, pruning and other operations.
[0065] The above descriptions are only the preferred embodiments of the present application, and are not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A machine vision-based method for inspecting the growth state of grapevine in the fruiting stage, characterized in that, The application relates to a grape cluster image processing method and device. The grape cluster image is input into a first target detection model, grape head images and grape cluster images are obtained, and a first grape cluster center branch curve is calculated according to the grape head images and the grape cluster images; The grape cluster image is input into a first target detection model, grape head images and grape cluster images are obtained, and a first grape cluster center branch curve is calculated according to the grape head images and the grape cluster images; The grape cluster image is input into a first target detection model, grape head images and grape cluster images are obtained, and a first grape cluster center branch curve is calculated according to the grape head images and the grape cluster images; 2. The method of claim 1, wherein the method further comprises: Before the grape cluster image is input into the first target detection model, the following steps are included: Different-angle grape cluster images are collected and marked as a training set, the grape cluster images contain grape heads and grape clusters, and the grape clusters contain grape fruits; The aspect ratio of the grape clusters in the training set is clustered by using K-means clustering, k1 class clusters are obtained, and the coordinate proportion value of the cluster tip relative to the center point and the aspect ratio expectation value of each class cluster are calculated in sequence; The aspect ratio of the grape clusters in the training set is clustered by using K-means clustering, k1 class clusters are obtained, and the coordinate proportion value of the cluster tip relative to the center point and the aspect ratio expectation value of each class cluster are calculated in sequence; The aspect ratio of the grape clusters in the training set is clustered by using K-means clustering, k1 class clusters are obtained, and the coordinate proportion value of the cluster tip relative to the center point and the aspect ratio expectation value of each class cluster are calculated in sequence; 3. The method of claim 2, wherein the method further comprises: The second target detection model is a feature mapping branch added in a feature extraction network, the feature mapping branch is used for constructing a candidate box according to the training set and mapping the candidate box to the feature extraction network.
4. The method of claim 3, wherein the method further comprises: The candidate box is constructed by constructing k3 center points, the coordinates of the center points are the center point coordinates of the k3 class clusters, and k2 candidate boxes are constructed for each center point, the length and width of the candidate boxes are respectively the length expectation value and the width expectation value of the k2 class clusters.
5. The method of claim 4, wherein the method further comprises: The candidate box is mapped to the feature extraction network by mapping the center point coordinates, the length and the width of the candidate box to a feature map extracted by the feature extraction network, and the mapping proportion is the proportion of the grape cluster image and the feature map extracted by the feature extraction network.
6. The method of claim 2, wherein the method further comprises: The first grape cluster center branch curve is calculated according to the grape head image and the grape cluster image, including the following steps: The aspect ratio of the grape cluster image is calculated, the class cluster where the grape cluster image is located is judged, the coordinate of the grape cluster tip point is calculated according to the coordinate proportion value of the cluster tip relative to the center point of the class cluster where the grape cluster image is located, the coordinate of the grape cluster tip point is fitted with the grape head fitting curve, and the first grape cluster center branch curve is obtained.
7. The method of claim 6, wherein the method further comprises: The maximum envelope curve is calculated by constructing a data set of boundary coordinates of all grape fruits, and taking the maximum and minimum of the column coordinates under the same row coordinate as the column boundary coordinates of the maximum envelope curve under the row coordinate, or taking the maximum and minimum of the row coordinates under the same column coordinate as the row boundary coordinates of the maximum envelope curve under the column coordinate. The first grape cluster center branch curve is corrected according to the maximum envelope curve by calculating the center point coordinates of the maximum envelope curve, fitting the center point coordinates and the first grape cluster center branch curve, and obtaining the second grape cluster center branch curve.
8. The method for inspecting the growth status of grapes during the fruit and vegetable season based on machine vision according to claim 1, characterized in that, The number of grape fruits is the number of grape fruits with a confidence greater than a confidence threshold. The compactness of grape fruits is obtained by clustering the center points of grape fruits according to distances, calculating the number of grape fruits in each class, calculating the area surrounded by the envelope curve of each class of grape fruits, and calculating the ratio of the area surrounded by the envelope curve of each class of grape fruits to the number of grape fruits in the class as the compactness of the grape fruits in the class. The grape weak fruit information is obtained by screening grape weak fruits according to grape fruit classes, and the grape weak fruit information is the boundary coordinates of the grape weak fruits.
9. The method for inspecting the growth status of grapes during the fruit and vegetable season based on machine vision according to claim 1, characterized in that, The grape cluster image is obtained by moving the inspection robot along a preset path, collecting the grape cluster image of the target object, collecting the current environment information when the inspection robot moves along the preset path, and determining whether there is an obstacle in front. If there is an obstacle, the moving path of the inspection robot is adjusted.
10. The method of claim 9, wherein the method further comprises: The grape cluster image of the target object is collected by inputting the image collected by the inspection robot into a grape cluster detection model, determining whether there is a grape cluster, and collecting grape cluster images at different angles if there is a grape cluster.