Method for detecting maturity of tomatoes and tomato picking system

Abstract

The invention discloses a tomato maturity detection method and a tomato picking system, and particularly relates to the technical field of tomato maturity detection and picking, the maturity of a to-be-detected tomato is judged according to the central point of a target image of the to-be-detected tomato and the middle points from the central point to four corner line segments, and RGB values of the five points are obtained. The method comprises the following steps: acquiring a maturity cardinal number of tomatoes to be detected, acquiring the complete maturity of the tomatoes to be detected according to the maturity cardinal number and environmental color variables, and after maturity detection is completed, controlling a picking vehicle through a PID controller module in a tomato picking system to pick mature tomatoes; according to the method, the tomato maturity detection accuracy is improved, accurate judgment on the tomato maturity is realized, and the tomato picking accuracy and the working efficiency are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of tomato maturity detection and picking, and in particular to a tomato maturity detection method and a tomato picking system. Background Art

[0002] Tomato picking currently still relies on a lot of traditional manual labor, with a low degree of mechanization. Tomatoes are plants that mature in stages, and different tomatoes mature at different times. Traditional manual collection is time-consuming and labor-intensive, with a low collection rate, and the fruit is easily damaged, consuming a lot of human resources. Automated picking robots can effectively solve the problem of labor shortage and reduce labor loss, but tomato picking carts are still not popularized in the market. Domestic and foreign carts are still in the research and development experimental stage. Many designed tomato picking robots are not suitable for outdoor and large farms. At present, the cost and price of tomato picking robots are too high, and the accuracy and efficiency of tomato maturity analysis are also low, resulting in low market benefits. Currently, they can only be used during the tomato maturity period.

[0003] In the automatic tomato picking and maturity recognition system, the recognition and positioning of tomatoes is one of the key technologies, which is of great significance to the realization of automatic intelligent picking. The maturity recognition of tomatoes is mainly based on traditional image processing, traditional machine learning and deep learning. The traditional algorithm has poor scene adaptability and poor detection effect in situations with obvious lighting changes, severe background interference and target occlusion, which leads to low maturity recognition efficiency and accuracy, affecting the efficiency of tomato picking. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is to propose a method for detecting tomato maturity and a tomato picking system, so as to solve the problems of poor tomato maturity detection effect and low picking efficiency in the prior art.

[0005] The technical solution adopted by the present invention to solve the technical problem is a method for detecting the maturity of tomatoes, which is characterized by comprising the steps of:

[0006] S1, obtaining a target image containing a tomato to be detected;

[0007] S2. Obtain the center point of the target image of the tomato to be detected and the midpoints of the four corner segments from the center point, and obtain the RGB values ​​of the five points to determine the maturity of the tomato to be detected. When RG>70, the tomato to be detected is ripe; when 70>RG>-40, the tomato to be detected is half-ripe; when RG<-40, the tomato to be detected is immature; when B>140, an error is reported;

[0008] S3. Obtain the maturity cardinality through the color difference between RGB, and obtain the complete maturity of the tomato to be tested according to the environmental color variables, the maturity cardinality and the maturity value at the time of testing. The complete maturity formula is as follows:

[0009]

[0010] Wherein, the environmental color variable is the ratio of the value of the weather to be detected to the value of the sunny weather.

[0011] Furthermore, in the step S1, the target image of the tomato to be detected is segmented by OpenCV to be divided into a training set, a validation set and a test set.

[0012] Furthermore, the validation set is used to perform a fit test on the target image of the tomato to be detected that cannot be identified.

[0013] Furthermore, a confidence score is output for the target image of the tomato to be detected through a YOLOv8 network.

[0014] Furthermore, the recognition capability of the YOLOv8 network is improved by changing the threshold of the confidence score.

[0015] Furthermore, in the rendering process in step S4, a color difference operation is performed using the spherical harmonic function.

[0016] Furthermore, the YOLOv8 network model of the tomato to be detected is visualized by normalizing the confusion matrix.

[0017] Furthermore, when the total value of the RGB value is lower than 150, it exceeds the target image range of the tomato to be detected.

[0018] A second aspect of the present invention provides a tomato picking system, so that the system can perform a method for detecting tomato maturity.

[0019] Furthermore, the system comprises:

[0020] An acquisition module, used for acquiring a target image containing a tomato to be detected;

[0021] A judgment module is used to judge the RGB value of the target image of the tomato to be detected;

[0022] A grasping module, used for grasping the tomato to be detected as a ripe tomato;

[0023] A sensor module to detect the position of the tomato picking vehicle;

[0024] PID controller module, used to adjust the speed and direction of the tomato picking vehicle;

[0025] The execution module is used to execute the speed and direction of the tomato picking vehicle.

[0026] Furthermore, the tomato picking vehicle is motion controlled by the PID controller module, and the control quantity formula controlled by the PID controller module is as follows:

[0027]

[0028] Where e(t) is the difference between the current position and the target position of the tomato picking vehicle, K p is the proportional gain, K i is the integral gain, K d is the differential gain.

[0029] Furthermore, the proportional gain is set to respond to the position error, the integral gain is used to correct the long-term deviation of the picking vehicle, and the differential gain is adjusted to reduce the oscillation coefficient of the tomato maturity detection system.

[0030] Compared with the prior art, the present invention has at least the following beneficial effects:

[0031] (1) The method for detecting the maturity of tomatoes described in the present invention uses a five-point method to detect the maturity of tomatoes, thereby avoiding interference from leaves or other factors outside the tomatoes when taking points, and avoiding environmental interference based on the maturity base and environmental color variables, thereby improving the accuracy of maturity detection;

[0032] (2) Compared with the original YOLOv5 model, the YoloV8 network reduces the number of parameters, reduces the amount of calculation, increases the detection speed, and maintains good detection accuracy;

[0033] (3) The tomato picking system described in the present invention can return the coordinates of the tomatoes after identifying the maturity of the tomatoes, and control the robotic arm through the PID control module to move along with the position of the tomatoes to accurately capture the tomatoes, thereby improving the picking efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of a method for detecting tomato maturity;

[0035] Figure 2 Schematic diagram of image classification dataset;

[0036] Figure 3 This is a schematic diagram of the precision and recall information of the training results;

[0037] Figure 4 This is a schematic diagram of the normalized confusion matrix;

[0038] Figure 5This is a schematic diagram of the fish fin-style flexible three-claw. DETAILED DESCRIPTION

[0039] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0040] Embodiment 1

[0041] To improve the detection accuracy of tomato maturity, Figure 1 As shown, the present invention provides a method for detecting tomato maturity, characterized in that it comprises the steps of:

[0042] S1, obtaining a target image containing a tomato to be detected;

[0043] S2. Obtain the center point of the target image of the tomato to be detected and the midpoints of the four corner segments from the center point, and obtain the RGB values ​​of the five points to determine the maturity of the tomato to be detected. When RG>70, the tomato to be detected is ripe; when 70>RG>-40, the tomato to be detected is half-ripe; when RG<-40, the tomato to be detected is immature; when B>140, an error is reported;

[0044] S3. Obtain the maturity cardinality through the color difference between RGB, and obtain the complete maturity of the tomato to be tested according to the environmental color variables, the maturity cardinality and the maturity value at the time of testing. The complete maturity formula is as follows:

[0045]

[0046] Wherein, the environmental color variable is the ratio of the value of the weather to be detected to the value of the sunny weather.

[0047] Before using a computer to process images, the real image must be photographed and then converted into a digital image that can be processed by the computer for computer analysis and calculation. When collecting images in the greenhouse, the ambient lighting should be appropriate. The camera is connected to the computer and mobile phone through the EZVIZ Cloud, and the captured images are saved locally at any time for subsequent processing. The captured image format is JPG. The target image is cut and processed through opencv to filter the effective part and avoid the image size being too large to cause the graphics card load to be too high during training. Before testing the maturity of tomatoes, the maturity of tomatoes is divided into three categories: mature, semi-mature and immature. Easydata is used for annotation. When annotating, small and inconspicuous tomatoes are ignored to prevent interference with the detection ability of the image, and the exported XML annotation file is converted into a txt file that can be used by yolo through an algorithm.

[0048] Furthermore, the annotated images are divided into three categories of data through further screening and operated through the YoloV8 model, such as Figure 2 The three types of data shown are training set, validation set and test set. The training set contains 80% of the pictures in the complete data set, and the validation set selects 10% of the relatively unclear and difficult-to-recognize pictures as a test of the fit to prevent overfitting or underfitting from making the model unable or difficult to recognize new data. The test set selects the last 10% of relatively clear pictures. These tomatoes with clear details can improve the accuracy and recall rate of model recognition.

[0049] Furthermore, although the model trained by the YoloV8 network can accurately identify the outline and number of tomatoes, the YOLOv8 network needs to output the confidence score for the target image of the tomato to be detected, and the recognition ability of the YOLOv8 network can be improved by changing the threshold of the confidence score, such as Figure 3 As shown in the figure, after the evaluation of the confidence score, the measured MAP value is higher, MAP = P*R, where P is the recognition accuracy, indicating whether the object recognized in a picture is the object trained by the model. The higher the value, the higher the probability that the object is correct when it is recognized. R is the ratio of the number of correctly recognized objects to the number of all correct objects in the picture. The higher the value, the more objects the model recognizes in the picture and the higher the coverage.

[0050] Furthermore, if Figure 4 As shown in the figure, the normalized confusion matrix visualizes the YOLOv8 network model for detecting tomatoes. Dividing the value of each cell by the actual number of samples in the category can intuitively distinguish the accuracy of the model. The darker the blue area, the more prominent the corresponding ability. The figure is darkest in the x_tomato and y_tomato directions, indicating that the recognition of tomatoes is high.

[0051] Furthermore, the center point of the target image of the tomato to be detected and the midpoints of the four corner segments from the center point are obtained, and the RGB values ​​of the five points are obtained to determine the maturity of the tomato to be detected. When RG>70, the tomato to be detected is ripe; when 70>RG>-40, the tomato to be detected is half-ripe; when RG<-40, the tomato to be detected is unripe. However, due to the reflective highlight area of ​​the tomato itself, when B>140, an error is reported, and when the blue value of the RGB of a point is greater than the red and green values, it indicates that the tomato is immature. The point is not on the tomato and is invalid. When the sum of the RGB values ​​of a point does not exceed 150, it means that the point is no longer on the tomato and the point is skipped directly. In addition, when the leaves are close to the tomato, the stem may be scanned, but the five-point method solves this problem. It will identify five points to avoid the leaves. When a red point is scanned, it means that the whole tomato is ripe. The maturity of the point is directly calculated and all interfering points are eliminated. When a green tomato leaf is scanned, it will be impossible to determine whether it is a tomato leaf, but because the color is consistent, the impact on maturity can be ignored.

[0052] Furthermore, considering the environmental factors, when EZVIZ Cloud calls the camera to take pictures, it will take pictures of tomatoes under different ambient light. At this time, there will be various problems in judging the maturity (such as cloudy days, sunny days, and rainy days), but it can be judged and checked through the color space and saturation of computer vision. The color space value when the picture is taken is calculated through the OpenCV algorithm. The color space value formula is as follows:

[0053]

[0054]

[0055]

[0056]

[0057] In the formula, θ is the angle value of the hue. In the HSV color space, the hue is represented by an angle of 0 to 360 degrees, where 0 degrees is red, 120 degrees is green, and 240 degrees is blue. H is the hue, S is the saturation, and I is the intensity. The environment under the right light of a sunny day is used as the base value of the environmental color variable. The maturity base is obtained by the color difference between RGB, and the complete maturity of the tomato to be tested is obtained based on the environmental color variable, the maturity base and the maturity value at the time of testing. The complete maturity formula is as follows:

[0058]

[0059] In the formula, the environmental color variable is the ratio of the weather value to be tested to the sunny weather value. In this way, the degree of maturity can be adjusted in real time under a changing environment, reducing the error caused by weather to the environmental color.

[0060] Embodiment 2

[0061] A second aspect of the present invention provides a tomato picking system, which enables the system to execute a method for detecting tomato maturity, and the system performs corresponding tomato picking operations according to the modules.

[0062] The system includes:

[0063] An acquisition module, used for acquiring a target image containing a tomato to be detected;

[0064] A judgment module is used to judge the RGB value of the target image of the tomato to be detected;

[0065] A grasping module, used for grasping the tomato to be detected as a ripe tomato;

[0066] A sensor module to detect the position of the tomato picking vehicle;

[0067] PID controller module, used to adjust the speed and direction of the tomato picking vehicle;

[0068] The execution module is used to execute the speed and direction of the tomato picking vehicle.

[0069] Among them, Figure 5 As shown, the grasping module adopts a practical fish-fin-shaped flexible three-claw. The special deformation degree when grasping objects can effectively protect the outer surface of the grasped fruit from being damaged. When the fish-fin shape contacts the supported object, the top and bottom of the fish-fin will bend toward the object, so that the claws cover the surface of the fruit to achieve the clamping effect. With a 25kg servo, it can grasp the fruit easily and quickly, reduce the grasping time, and improve the picking accuracy. It is also equipped with a JGB37-528R131-08 motor. The rated voltage of the motor is 8V. The actual speed is about 80RPM at a voltage of 7.7V, and the rated full speed of the motor is 90RPM.

[0070] Furthermore, after the tomato maturity detection is completed, the tomato picking system performs the picking work, obtains image information through the camera, and then performs image processing, that is, binarizes the image. In order to reduce interference and make the image smoother, the image is eroded and expanded. Then, the target maximum area contour and the minimum circumscribed circle are obtained to obtain the object tracking area. Then, the movement angle of the robot arm is solved according to inverse kinematics, and the servo is controlled to track the object. In order to reduce interference and make the image smoother, the image needs to be eroded and expanded. When tracking the object, the X and Y coordinates of the center point of the image are set as the set values, and the X and Y coordinates of the current identified target are used as the input values. Inverse kinematics calculations are performed to obtain the target position.

[0071] Furthermore, the PID controller module is used in the tomato picking system to adjust the speed and direction of the tomato picking vehicle; the sensor module uses a positioning sensor to detect the position of the tomato picking vehicle and provide real-time feedback; the execution module uses a motor and a steering system to execute the speed and direction of the tomato picking vehicle according to the controller's instructions. The PID control module achieves precise regulation of the system by calculating the proportion, integration, and differentiation of the error. The control quantity formula controlled by the PID controller module is as follows:

[0072]

[0073] Where e(t) is the difference between the current position and the target position of the tomato picking vehicle, K p is the proportional gain, K i is the integral gain, K d is the differential gain. According to the current error, a control amount proportional to the error is generated, with a fast response speed, but steady-state errors are prone to occur; the steady-state error is eliminated through the integral gain, and the control amount is adjusted by gradually accumulating the error, thereby eliminating long-term deviations; the differential gain is used to predict the error change rate, reduce oscillation, and improve the stability of the system. According to different field environments and the width of crop rows, the PID parameters need to be adjusted to obtain the best control effect, so that the tomato picking system can accurately drive on the field path during use, effectively avoid obstacles and smoothly reach the picking position, maintain a relatively stable driving speed during the picking process, effectively avoid the problem of damaging the fruit due to excessive speed, and can significantly improve the movement accuracy of the car, especially in complex terrain or turns, which can effectively reduce errors.

[0074] Embodiment 3

[0075] Since the insect detection system is crucial to the growth of tomatoes, insects are captured in the field and the YOLOv8 algorithm is used for insect recognition. The Chilo suppressalis and Rice leaf folder are identified, and an alarm is triggered when the model identifies the insect.

[0076] In summary, a method for detecting tomato maturity and a tomato picking system of the present invention use the center point of the target image of the tomato to be detected and the midpoint of the four angular segments from the center point to obtain the RGB values ​​of the five points to judge the maturity of the tomato to be detected, and avoid environmental interference according to the maturity cardinality and environmental color variables, thereby improving the accuracy of maturity detection. The YoloV8 network is used to build a recognition system with a high success rate compared to the original YOLOv5 model. The image processing process designed includes image acquisition, data set construction, model training and optimization, thereby achieving accurate judgment of tomato maturity. Its high MAP value, precision and recall rate indicate that the model has excellent recognition performance, and can return tomato coordinates after tomato maturity recognition. The mechanical arm is controlled by the PID control module to move with the position of the tomato, accurately capture, and improve picking efficiency. In terms of hardware selection, a fish fin-type flexible three-claw is selected, which can easily and quickly capture the fruit, effectively improving the accuracy and work efficiency of capture.

[0077] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0078] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0079] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0080] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A method for detecting tomato maturity, characterized in that: Includes steps: S1, obtaining a target image containing a tomato to be detected; S2. Obtain the center point of the target image of the tomato to be detected and the midpoints of the four corner segments from the center point, and obtain the RGB values ​​of the five points to determine the maturity of the tomato to be detected. When RG>70, the tomato to be detected is ripe; when 70>RG>-40, the tomato to be detected is half-ripe; when RG<-40, the tomato to be detected is immature; when B>140, an error is reported; S3. Obtain the maturity cardinality through the color difference between RGB, and obtain the complete maturity of the tomato to be tested according to the environmental color variables, the maturity cardinality and the maturity value at the time of testing. The complete maturity formula is as follows: Wherein, the environmental color variable is the ratio of the value of the weather to be detected to the value of the sunny weather.

2. A method for detecting tomato maturity according to claim 1, characterized in that: In the step S1, the target image of the tomato to be detected is segmented by OpenCV to be divided into a training set, a validation set and a test set.

3. A method for detecting tomato maturity according to claim 2, characterized in that: The validation set is used to perform a fit test on the target image of the tomato to be detected that cannot be identified.

4. A method for detecting tomato maturity according to claim 1, characterized in that: The confidence score of the target image of the tomato to be detected is output through the YOLOv8 network.

5. A method for detecting tomato maturity according to claim 4, characterized in that: The recognition ability of the YOLOv8 network is improved by changing the threshold of the confidence score.

6. A method for detecting tomato maturity according to claim 4, characterized in that: The YOLOv8 network model of the tomato to be detected is visualized through the normalized confusion matrix.

7. A method for detecting tomato maturity according to claim 1, characterized in that: When the total value of the RGB value is less than 150, it exceeds the target image range of the tomato to be detected.

8. A tomato picking system, characterized in that: include: An acquisition module, used for acquiring a target image containing a tomato to be detected; A judgment module is used to judge the RGB value of the target image of the tomato to be detected; A grasping module, used for grasping the tomato to be detected as a ripe tomato; A sensor module to detect the position of the tomato picking vehicle; PID controller module, used to adjust the speed and direction of the tomato picking vehicle; The execution module is used to execute the speed and direction of the tomato picking vehicle.

9. A tomato picking system according to claim 8, characterized in that: The tomato picking vehicle is motion controlled by the PID controller module, and the control quantity formula controlled by the PID controller module is as follows: Where e(t) is the difference between the current position and the target position of the tomato picking vehicle, K p is the proportional gain, K i is the integral gain, K d is the differential gain.

10. A tomato picking system according to claim 9, characterized in that: The proportional gain is set to respond to the position error, the integral gain is used to correct the long-term deviation of the picking vehicle, and the differential gain is adjusted to reduce the oscillation coefficient of the tomato maturity detection system.

Images