Tomato stamen positioning and orientation identification method in facility production

Abstract

The invention relates to a tomato stamen positioning and orientation identification method in facility production, which comprises the following steps: S1, taking a tomato stamen picture through a depth camera to obtain a tomato stamen depth image and coordinate information and depth information of each pixel point in the tomato stamen depth image; s2, performing noise reduction on the tomato stamen depth image, and performing processing to obtain an HSV format tomato stamen image; s3, segmenting the HSV format tomato stamen image to obtain an HSV format tomato stamen segmented image; step S6, determining a pitch angle and a yaw angle of the tomato stamen by using the orientation of the tomato stamen so as to obtain orientation angle information of the tomato stamen in the three-dimensional space; s7, obtaining the actual position information of the tomato stamen by using the coordinate information and depth information of each pixel in the depth image of the tomato stamen; and step S8, the pollination robot performs pollination operation according to the actual position information and orientation angle information of the tomato stamens. According to the invention, accurate position and angle information of tomato stamens can be provided.

Description

Technical Field

[0001] The invention relates to the field of plant posture recognition, and in particular to a method for positioning and identifying the orientation of tomato pistils in facility production. Background Art

[0002] Tomato is one of the most widely planted vegetables in my country, but the success rate of pollination by insects and other media is low in the natural environment. Therefore, tomato pollination has gradually shifted to facility cultivation at this stage. However, relying on artificial pollination has problems such as uneven pollination and high labor intensity. Using pollination robots instead of manual work can significantly improve the quality and efficiency of pollination. Due to the different orientations of tomato stamens, it is difficult for pollination robots to accurately locate the pistil, which is not conducive to subsequent pollination operations. Summary of the invention

[0003] In view of the defects of the prior art, the present invention provides a method for positioning and identifying the orientation of tomato pistils in facility production, so as to solve the problem that the position and orientation of the pistil cannot be accurately positioned during machine pollination, resulting in a low pollination rate.

[0004] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for locating and identifying the orientation of tomato pistils in facility production, comprising the following steps: Step S1, installing a depth camera on the pollination robot arm of the pollination robot, taking a photo of a tomato pistil, and obtaining a tomato pistil depth image and coordinate information and depth information of each pixel point in the tomato pistil depth image; Step S2, using Gaussian filtering to reduce noise on the tomato pistil depth image, and using the OpenCV library to process the tomato pistil depth image after noise reduction to obtain a tomato pistil image in HSV format; Step S3, using an image threshold segmentation technique to segment the tomato pistil in the HSV format tomato pistil image from the background to obtain an HSV format tomato pistil segmentation image; Step S4, segmenting the tomato pistil from the tomato pistil depth image to obtain a tomato pistil image; Step S5, extracting features from the tomato pistil image to obtain the orientation of the tomato pistil in three-dimensional space; Step S6, using the orientation of the tomato pistil, determining the pitch angle θp of the tomato pistil relative to the vertical direction and the yaw angle θy relative to the horizontal plane, thereby obtaining the orientation angle information of the tomato pistil in three-dimensional space; Step S7, using the coordinate information and depth information of each pixel in the tomato pistil depth image, combined with the depth camera parameters, according to the three-dimensional coordinate formula conversion to obtain the actual position information of the tomato pistil; Step S8, using the actual position information and orientation angle information of the tomato pistil obtained in the above steps, the pollination robot controls the pollination mechanical arm to perform precise operations according to the information.

[0005] As a preferred solution, in step S2, the tomato pistil depth image is normalized to a range of 0 to 255 using the normalize function in the OpenCV library; the normalized image is converted into an RGB pseudo-color image using the applyColorMap function; and the RGB pseudo-color image is converted into a tomato pistil image in HSV format using the cvtColor function.

[0006] As a preferred solution, in step S3, the hue of the tomato pistil H is set between 26 and 34, the saturation S is set between 43 and 255, and the brightness V is set between 46 and 255; the lowest value of the HSV threshold range is set to [26, 43, 46], and the highest value is set to [34, 255, 255], and the tomato pistil within the HSV threshold range is separated from the background, thereby obtaining a tomato pistil segmentation image in HSV format.

[0007] As a preferred solution, in step S4, the HSV threshold range set in step S3 is used to generate a binary mask image by segmenting the tomato pistil image area in the HSV format, wherein the tomato pistil area within the HSV threshold range is set to white (pixel value is 1), and the background outside the HSV threshold range is set to black (pixel value is 0); the Canny edge detection algorithm is used to extract the edge in the binary mask image to find the precise boundary of the tomato pistil area; based on the edge detection result, the contour detection algorithm findContours function is used to extract the contour of the pistil area; finally, the area within the contour is used as the tomato pistil, and a segmentation mask is generated, which can be used to separate the tomato pistil area in the original tomato pistil depth image to obtain a tomato pistil image.

[0008] As a preferred solution, in step S5, the coordinate information of each pixel point of the tomato pistil image is obtained by using a depth camera, and feature extraction is performed on the segmented tomato pistil image.

[0009] As a preferred solution, step S5 includes the following sub-steps: Step S51: Calculate the mean μ of the segmented tomato pistil image point cloud data, and the calculation formula is as follows: Where μ is the mean vector of all point coordinates, P i =(x i ,y i ,z i ) is the coordinate of the i-th point, N is the total number of coordinate points; Step S52: Calculate the covariance matrix C of the point cloud, and the calculation formula is as follows: Step S53: Perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalue λ and eigenvector v. The calculation formula is as follows: Cv i =λ i v i In the formula, λ i represents the variance of the data in the direction of the eigenvector; v i is the corresponding eigenvector, indicating the principal axis in that direction; Step S53: The eigenvector v corresponding to the obtained maximum eigenvalue max This is the direction of the tomato pistil, and the calculation formula is as follows:

[0010] As a preferred solution, in step S6, the pitch angle θp refers to the main axis vector v max The angle between the projection on the vertical direction (z-axis) and the horizontal plane is used to determine the upward or downward angle of the stamen relative to the horizontal plane. The calculation formula is as follows: In the formula, is a unit vector pointing in the vertical direction; The yaw angle θy refers to the main axis vector v max The angle between the projection on the horizontal plane (xy plane) and a reference direction is used to determine the rotation direction of the stamen in the horizontal plane. The calculation formula is as follows: In the formula, and are unit vectors pointing along the x and y axes; If θp=0°, θy=0°, the pistil is horizontal and points in the positive direction of the x-axis; If θp=45°, θy=90°, the pistil is tilted 45° upward and points to the positive direction of the y-axis; If θp=-30°, θy=-45°, the pistil is tilted downward by 30° and points to the fourth quadrant between the x-axis and the y-axis.

[0011] The beneficial effects of the present application are as follows: the present invention can obtain a tomato pistil depth image based on the depth camera and the coordinate information and depth information of each pixel in the tomato pistil depth image. The tomato pistil depth image is subjected to image denoising and image segmentation in turn, so as to better eliminate background information and segment the tomato pistil. In feature extraction, the orientation angle of the tomato pistil can be obtained based on the coordinate information, and the coordinate information and depth information of each pixel in the tomato pistil depth image can be used in combination with the depth camera parameters to obtain the actual position information of the tomato pistil according to the three-dimensional coordinate formula conversion, thereby providing accurate position and angle information for the pollination robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the overall flow chart of the present invention;

[0014] Figure 2 A tomato stamen orientation flow chart is obtained for the present invention. DETAILED DESCRIPTION

[0015] The following is combined with Figure 1-2 The present invention is further described in detail with reference to the accompanying drawings and specific embodiments.

[0016] The present application embodiment provides a method for locating and identifying the orientation of tomato pistils in facility production, comprising the following steps: Step S1, installing a depth camera on the pollination robot arm of the pollination robot, taking a photo of a tomato pistil, and obtaining a tomato pistil depth image and coordinate information and depth information of each pixel point in the tomato pistil depth image; Step S2, using Gaussian filtering to reduce noise on the tomato pistil depth image, and using the OpenCV library to process the tomato pistil depth image after noise reduction to obtain a tomato pistil image in HSV format; Step S3, using an image threshold segmentation technique to segment the tomato pistil in the HSV format tomato pistil image from the background to obtain an HSV format tomato pistil segmentation image; Step S4, segmenting the tomato pistil from the tomato pistil depth image to obtain a tomato pistil image; Step S5, extracting features from the tomato pistil image to obtain the orientation of the tomato pistil in three-dimensional space; Step S6, using the orientation of the tomato pistil, determining the pitch angle θp of the tomato pistil relative to the vertical direction and the yaw angle θy relative to the horizontal plane, thereby obtaining the orientation angle information of the tomato pistil in three-dimensional space; Step S7, using the coordinate information and depth information of each pixel in the tomato pistil depth image, combined with the depth camera parameters, according to the three-dimensional coordinate formula to obtain the actual position information of the tomato pistil; Step S8, using the actual position information and orientation angle information of the tomato pistil obtained in the above steps, the pollination robot controls the pollination mechanical arm to perform precise operations according to the information.

[0017] In step S2, the tomato pistil depth image is normalized to a range of 0 to 255 using the normalize function in the OpenCV library; the normalized image is converted into an RGB pseudo-color image using the applyColorMap function; and the RGB pseudo-color image is converted into an HSV format tomato pistil image using the cvtColor function. It should be noted that the parts not described in detail in this application are all prior art.

[0018] In step S3, the hue of the tomato pistil H is set between 26 and 34, the saturation S is set between 43 and 255, and the brightness V is set between 46 and 255; the lowest value of the HSV threshold range is set to [26, 43, 46], and the highest value is set to [34, 255, 255], so that the tomato pistil within the HSV threshold range is separated from the background, thereby obtaining a tomato pistil segmentation image in HSV format.

[0019] In step S4, the HSV threshold range set in step S3 is used to generate a binary mask image for the HSV format tomato pistil segmentation image area, wherein the tomato pistil area within the HSV threshold range is set to white (pixel value is 1), and the background outside the HSV threshold range is set to black (pixel value is 0); the Canny edge detection algorithm is used to extract the edge in the binary mask image to find the precise boundary of the tomato pistil area; based on the edge detection result, the contour detection algorithm findContours function is used to extract the contour of the pistil area; finally, the area within the contour is used as the tomato pistil, and a segmentation mask is generated, which can be used to separate the tomato pistil area in the original tomato pistil depth image to obtain a tomato pistil image.

[0020] In step S5, the coordinate information of each pixel point of the tomato pistil image is obtained by using the depth camera, and feature extraction is performed on the segmented tomato pistil image.

[0021] Step S5 includes the following sub-steps: Step S51: Calculate the mean μ of the segmented tomato pistil image point cloud data, and the calculation formula is as follows: Where μ is the mean vector of all point coordinates, P i =(x i ,y i ,z i ) is the coordinate of the i-th point, N is the total number of coordinate points; Step S52: Calculate the covariance matrix C of the point cloud, and the calculation formula is as follows: Step S53: Perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalue λ and eigenvector v. The calculation formula is as follows: Cv i =λ i v i In the formula, λ i represents the variance of the data in the direction of the eigenvector; v i is the corresponding eigenvector, indicating the principal axis in that direction; Step S53: The eigenvector v corresponding to the obtained maximum eigenvalue max This is the direction of the tomato pistil, and the calculation formula is as follows:

[0022] In step S6, the pitch angle θp refers to the main axis vector v max The angle between the projection on the vertical direction (z-axis) and the horizontal plane is used to determine the upward or downward angle of the stamen relative to the horizontal plane. The calculation formula is as follows: In the formula, is a unit vector pointing in the vertical direction; The yaw angle θy refers to the main axis vector v max The angle between the projection on the horizontal plane (xy plane) and a reference direction is used to determine the rotation direction of the stamen in the horizontal plane. The calculation formula is as follows: In the formula, and are unit vectors pointing along the x and y axes; If θp=0°, θy=0°, the pistil is horizontal and points in the positive direction of the x-axis; If θp=45°, θy=90°, the pistil is tilted 45° upward and points to the positive direction of the y-axis; If θp=-30°, θy=-45°, the pistil is tilted downward by 30° and points to the fourth quadrant between the x-axis and the y-axis.

[0023] In step S7, the calculation formula is as follows:

[0024] Where: X', Y', Z' are the actual coordinates in three-dimensional space; (u', v') are the two-dimensional coordinates of the pixel point in the image; (c' x ,c' y ) is the center point of the image; D' is the depth value corresponding to the pixel; f'x and f' yis the focal length of the camera.

[0025] The present invention can obtain a tomato pistil depth image based on the depth camera and the coordinate information and depth information of each pixel in the tomato pistil depth image. The tomato pistil depth image is subjected to image denoising and image segmentation in turn, so as to better eliminate background information and segment the tomato pistil. In feature extraction, the orientation angle of the tomato pistil can be obtained according to the coordinate information, and the coordinate information and depth information of each pixel in the tomato pistil depth image are combined with the depth camera parameters and converted according to the three-dimensional coordinate formula to obtain the actual position information of the tomato pistil, thereby providing accurate position and angle information for the pollination robot.

[0026] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for locating and identifying the orientation of tomato pistils in facility production, characterized in that: The steps include: Step S1, installing a depth camera on the pollination robot arm of the pollination robot, taking a photo of a tomato pistil, and obtaining a tomato pistil depth image and coordinate information and depth information of each pixel point in the tomato pistil depth image; Step S2, using Gaussian filtering to reduce noise on the tomato pistil depth image, and using the OpenCV library to process the tomato pistil depth image after noise reduction to obtain a tomato pistil image in HSV format; Step S3, using an image threshold segmentation technique to segment the tomato pistil in the HSV format tomato pistil image from the background to obtain an HSV format tomato pistil segmentation image; Step S4, segmenting the tomato pistil from the tomato pistil depth image to obtain a tomato pistil image; Step S5, extracting features from the tomato pistil image to obtain the orientation of the tomato pistil in three-dimensional space; Step S6, using the orientation of the tomato pistil, determining the pitch angle θp of the tomato pistil relative to the vertical direction and the yaw angle θy relative to the horizontal plane, thereby obtaining orientation angle information of the tomato pistil in three-dimensional space; Step S7, using the coordinate information and depth information of each pixel in the tomato pistil depth image, combined with the depth camera parameters, the actual position information of the tomato pistil is converted according to the three-dimensional coordinate formula. Step S8, using the actual position information and orientation angle information of the tomato pistil obtained in the above steps, the pollination robot controls the pollination mechanical arm to perform precise operations according to the information.

2. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S2, the normalize function in the OpenCV library is used to normalize the tomato pistil depth image to a range of 0 to 255; the applyColorMap function is used to convert the normalized image into an RGB pseudo-color image; and the cvtColor function is used to convert the RGB pseudo-color image into a tomato pistil image in HSV format.

3. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S3, the hue of the tomato pistil H is set between 26 and 34, the saturation S is set between 43 and 255, and the brightness V is set between 46 and 255; the lowest value of the HSV threshold range is set to [26, 43, 46], and the highest value is set to [34, 255, 255], so that the tomato pistil within the HSV threshold range is separated from the background, thereby obtaining a tomato pistil segmentation image in HSV format.

4. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S4, the HSV threshold range set in step S3 is used to generate a binary mask image for the HSV format tomato pistil segmentation image area, wherein the tomato pistil area within the HSV threshold range is set to white (pixel value is 1), and the background outside the HSV threshold range is set to black (pixel value is 0); the Canny edge detection algorithm is used to extract the edge in the binary mask image to find the precise boundary of the tomato pistil area; based on the edge detection result, the contour detection algorithm findContours function is used to extract the contour of the pistil area; finally, the area within the contour is used as the tomato pistil, and a segmentation mask is generated, which can be used to separate the tomato pistil area in the original tomato pistil depth image to obtain a tomato pistil image.

5. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S5, the coordinate information of each pixel point of the tomato pistil image is obtained by using the depth camera, and feature extraction is performed on the segmented tomato pistil image.

6. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 5, characterized in that: Step S5 includes the following sub-steps: Step S51: Calculate the mean μ of the segmented tomato pistil image point cloud data, and the calculation formula is as follows: Where μ is the mean vector of all point coordinates, P i =(x i ,y i ,z i ) is the coordinate of the i-th point, N is the total number of coordinate points; Step S52: Calculate the covariance matrix C of the point cloud, and the calculation formula is as follows: Step S53: Perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalue λ and eigenvector v. The calculation formula is as follows: Cv i =λ i v i In the formula, λ i represents the variance of the data in the direction of the eigenvector; v i is the corresponding eigenvector, indicating the principal axis in that direction; Step S53: The eigenvector v corresponding to the obtained maximum eigenvalue max This is the direction of the tomato pistil, and the calculation formula is as follows:

7. A method for locating and identifying the orientation of tomato pistils in facility production according to claim 6, characterized in that: In step S6, the pitch angle θp refers to the main axis vector v max The angle between the projection on the vertical direction (z-axis) and the horizontal plane is used to determine the upward or downward angle of the stamen relative to the horizontal plane. The calculation formula is as follows: In the formula, is a unit vector pointing in the vertical direction; The yaw angle θy refers to the main axis vector v max The angle between the projection on the horizontal plane (xy plane) and a reference direction is used to determine the rotation direction of the stamen in the horizontal plane. The calculation formula is as follows: In the formula, and are unit vectors pointing along the x and y axes; If θp=0°, θy=0°, the pistil is horizontal and points in the positive direction of the x-axis; If θp=45°, θy=90°, the pistil is tilted 45° upward and points to the positive direction of the y-axis; If θp=-30°, θy=-45°, the pistil is tilted downward by 30° and points to the fourth quadrant between the x-axis and the y-axis.

Images