A method for locating and identifying the orientation of tomato pistils in facility production

Abstract

This application provides a method for locating and identifying the orientation of tomato pistils in facility production, comprising the following steps: Step S1, photographing a tomato pistil using a depth camera to obtain a tomato pistil depth image and the coordinate and depth information of each pixel in the tomato pistil depth image; Step S2, denoising the tomato pistil depth image and processing it to obtain an HSV format tomato pistil image; Step S3, segmenting the HSV format tomato pistil image to obtain an HSV format tomato pistil segmented image; Step S6, using the tomato pistil orientation to determine the pitch and yaw angles of the tomato pistil, thereby obtaining the orientation angle information of the tomato pistil in three-dimensional space; Step S7, using the coordinate and depth information of each pixel in the tomato pistil depth image to obtain the actual position information of the tomato pistil; Step S8, a pollination robot performs pollination operations based on the actual position information and orientation angle information of the tomato pistil. This application can provide precise position and angle information of tomato pistils.

Description

Technical Field

[0001] The present invention relates to the field of plant posture recognition, and in particular to a method for positioning and identifying the orientation of tomato pistils in facility production. Background Art

[0002] Tomatoes are one of the most widely cultivated vegetables in my country, but pollination by insects and other pollinators has a low success rate in the wild. Consequently, tomato pollination has gradually shifted to greenhouse cultivation. However, relying on manual pollination presents challenges such as uneven pollination and high labor intensity. Using pollination robots instead of manual pollination can significantly improve pollination quality and efficiency. However, due to the varying orientations of tomato stamens, precise positioning of the pistil by pollination robots is difficult, hindering subsequent pollination operations. Summary of the Invention

[0003] In response to the defects of the existing technology, the present invention provides a method for locating and identifying the orientation of tomato pistils in facility production to solve the problem that the current machine pollination cannot accurately locate the position and orientation of the pistil, resulting in a low pollination rate.

[0004] To achieve the above-mentioned object, the technical solution adopted by the present invention is: a method for locating and identifying the orientation of tomato pistils in facility production, comprising the following steps:

[0005] Step S1, installing a depth camera on the pollination robot arm of the pollination robot to take a photo of the tomato pistil to obtain a depth image of the tomato pistil and coordinate information and depth information of each pixel point in the depth image of the tomato pistil;

[0006] Step S2, using Gaussian filtering to reduce noise on the tomato pistil depth image, and processing the reduced noise tomato pistil depth image using the OpenCV library to obtain a tomato pistil image in HSV format;

[0007] Step S3, using an image threshold segmentation technique to segment the tomato pistil from the background in the tomato pistil image in the HSV format to obtain a tomato pistil segmentation image in the HSV format;

[0008] Step S4, segmenting the tomato pistil from the tomato pistil depth image to obtain a tomato pistil image;

[0009] Step S5: Extract features from the tomato pistil image to determine the orientation of the tomato pistil in three-dimensional space. Step S6: Use the tomato pistil orientation to determine the pitch angle θp of the tomato pistil relative to the vertical direction and the yaw angle θy of the tomato pistil relative to the horizontal plane, thereby obtaining the orientation angle information of the tomato pistil in three-dimensional space. Step S7: Use the coordinate information and depth information of each pixel in the tomato pistil depth image, combined with the depth camera parameters, to convert according to the three-dimensional coordinate formula to obtain the actual position information of the tomato pistil.

[0010] In step S8, the pollination robot uses the actual position information and orientation angle information of the tomato pistil obtained in the above steps to control the pollination robot arm to perform precise operations according to the information.

[0011] As a preferred solution, in step S2, the tomato pistil depth image is normalized to a range of 0 to 255 using the normalize function in the OpenCV library; the normalized image is converted into an RGB pseudo-color image using the applyColorMap function; and the RGB pseudo-color image is converted into an HSV format tomato pistil image using the cvtColor function.

[0012] As a preferred solution, in step S3, the hue H of the tomato pistil is set between 26 and 34, the saturation S is set between 43 and 255, and the brightness V is set between 46 and 255; the lowest value of the HSV threshold range is set to [26, 43, 46], and the highest value is set to [34, 255, 255], so that the tomato pistil within the HSV threshold range is separated from the background, thereby obtaining an HSV format tomato pistil segmentation image.

[0013] As a preferred solution, in step S4, the HSV threshold range set in step S3 is used to segment the tomato pistil image region in HSV format to generate a binary mask image, wherein the tomato pistil region within the HSV threshold range is set to white (pixel value 1), and the background outside the HSV threshold range is set to black (pixel value 0); the Canny edge detection algorithm is used to extract edges in the binary mask image to find the precise boundary of the tomato pistil region; based on the edge detection results, the contour detection algorithm findContours function is used to extract the contour of the pistil region; finally, the area within the contour is used as the tomato pistil, and a segmentation mask is generated. The mask can be used to separate the tomato pistil region from the original tomato pistil depth image to obtain a tomato pistil image.

[0014] As a preferred solution, in step S5, the coordinate information of each pixel point of the tomato pistil image is obtained using a depth camera, and feature extraction is performed on the segmented tomato pistil image.

[0015] As a preferred solution, step S5 includes the following sub-steps:

[0016] Step S51: Calculate the mean μ of the segmented tomato pistil image point cloud data using the following formula:

[0017]

[0018] Where μ is the mean vector of all point coordinates, P i =(x i ,y i ,zi ) is the coordinate of the i-th point, N is the total number of coordinate points;

[0019] Step S52: Calculate the covariance matrix C of the point cloud. The calculation formula is as follows:

[0020]

[0021] Step S53: Perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalue λ and eigenvector v. The calculation formula is as follows:

[0022] Cv i =λ i v i

[0023] Where λ i Represents the variance of the data in the direction of the eigenvector; v i is the corresponding eigenvector, indicating the principal axis in that direction;

[0024] Step S53: The eigenvector v corresponding to the obtained maximum eigenvalue max This is the direction of the tomato pistil, calculated as follows:

[0025]

[0026] As a preferred solution, in step S6, the pitch angle θp refers to the main axis vector v max The angle between the projection on the vertical direction (z-axis) and the horizontal plane is used to determine the upward or downward angle of the stamen relative to the horizontal plane. The calculation formula is as follows:

[0027]

[0028] Where, is a unit vector pointing in the vertical direction;

[0029] The yaw angle θy refers to the main axis vector v max The angle between the projection on the horizontal plane (xy plane) and a reference direction is used to determine the rotation direction of the pistil in the horizontal plane. The calculation formula is as follows:

[0030]

[0031] Where, and are unit vectors pointing to the x and y axes;

[0032] If θp = 0°, θy = 0°, the pistil is horizontal and points in the positive direction of the x-axis;

[0033] If θp = 45°, θy = 90°, the pistil is tilted 45° upward and points in the positive direction of the y-axis;

[0034] If θp = -30°, θy = -45°, the pistil is tilted 30° downward and points to the fourth quadrant between the x-axis and the y-axis.

[0035] The beneficial effects of the present application are as follows: based on the tomato pistil depth image obtained by the depth camera and the coordinate information and depth information of each pixel point in the tomato pistil depth image, the tomato pistil depth image can be subjected to image denoising and image segmentation in sequence, thereby better eliminating background information and segmenting the tomato pistil. In feature extraction, the orientation angle of the tomato pistil can be obtained based on the coordinate information, and the coordinate information and depth information of each pixel point in the tomato pistil depth image can be used in combination with the depth camera parameters to obtain the actual position information of the tomato pistil according to the three-dimensional coordinate formula, thereby providing accurate position and angle information for the pollination robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the overall flow chart of the present invention;

[0037] Figure 2 A tomato stamen orientation flow chart is obtained for the present invention. DETAILED DESCRIPTION

[0038] The following is combined with Figure 1-2 The present invention is further described in detail with reference to the accompanying drawings and specific embodiments.

[0039] The present application provides a method for locating and identifying the orientation of tomato pistils in facility production, including the following steps:

[0040] Step S1, installing a depth camera on the pollination robot arm of the pollination robot to take a photo of the tomato pistil to obtain a depth image of the tomato pistil and coordinate information and depth information of each pixel point in the depth image of the tomato pistil;

[0041] Step S2, using Gaussian filtering to reduce noise on the tomato pistil depth image, and processing the reduced noise tomato pistil depth image using the OpenCV library to obtain a tomato pistil image in HSV format;

[0042] Step S3, using an image threshold segmentation technique to segment the tomato pistil from the background in the tomato pistil image in the HSV format to obtain a tomato pistil segmentation image in the HSV format;

[0043] Step S4, segmenting the tomato pistil from the tomato pistil depth image to obtain a tomato pistil image;

[0044] Step S5, extracting features from the tomato pistil image to determine the orientation of the tomato pistil in three-dimensional space;

[0045] Step S6: Using the orientation of the tomato pistil, determine the pitch angle θp of the tomato pistil relative to the vertical direction and the yaw angle θy of the tomato pistil relative to the horizontal plane, thereby obtaining the orientation angle information of the tomato pistil in three-dimensional space. Step S7: Using the coordinate information and depth information of each pixel in the tomato pistil depth image, combined with the depth camera parameters, convert according to the three-dimensional coordinate formula to obtain the actual position information of the tomato pistil.

[0046] In step S8, the pollination robot uses the actual position information and orientation angle information of the tomato pistil obtained in the above steps to control the pollination robot arm to perform precise operations according to the information.

[0047] In step S2, the tomato pistil depth image is normalized to a range of 0 to 255 using the normalize function in the OpenCV library; the normalized image is converted to an RGB pseudo-color image using the applyColorMap function; and the RGB pseudo-color image is converted to an HSV format tomato pistil image using the cvtColor function. It should be noted that any portion not described in detail in this application is prior art.

[0048] In step S3, the hue H of the tomato pistil is set between 26 and 34, the saturation S is set between 43 and 255, and the brightness V is set between 46 and 255. The lowest value of the HSV threshold range is set to [26, 43, 46], and the highest value is set to [34, 255, 255]. The tomato pistil within the HSV threshold range is separated from the background, thereby obtaining a tomato pistil segmentation image in HSV format.

[0049] In step S4, the HSV threshold range set in step S3 is used to generate a binary mask image for the HSV format tomato pistil segmentation image region, wherein the tomato pistil region within the HSV threshold range is set to white (pixel value 1), and the background outside the HSV threshold range is set to black (pixel value 0); the Canny edge detection algorithm is used to extract the edges in the binary mask image to find the precise boundary of the tomato pistil region; based on the edge detection results, the contour detection algorithm findContours function is used to extract the contour of the pistil region; finally, the area within the contour is used as the tomato pistil, and a segmentation mask is generated. This mask can be used to separate the tomato pistil region from the original tomato pistil depth image to obtain a tomato pistil image.

[0050] In step S5, the coordinate information of each pixel point of the tomato pistil image is obtained using a depth camera, and feature extraction is performed on the segmented tomato pistil image.

[0051] Step S5 includes the following sub-steps:

[0052] Step S51: Calculate the mean μ of the segmented tomato pistil image point cloud data using the following formula:

[0053]

[0054] Where μ is the mean vector of all point coordinates, P i =(x i ,y i ,z i ) is the coordinate of the i-th point, N is the total number of coordinate points;

[0055] Step S52: Calculate the covariance matrix C of the point cloud. The calculation formula is as follows:

[0056]

[0057] Step S53: Perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalue λ and eigenvector v. The calculation formula is as follows:

[0058] Cv i =λ i v i

[0059] Where λ i Represents the variance of the data in the direction of the eigenvector; v i is the corresponding eigenvector, indicating the principal axis in that direction;

[0060] Step S53: The eigenvector v corresponding to the obtained maximum eigenvalue max This is the direction of the tomato pistil, calculated as follows:

[0061]

[0062] In step S6, the pitch angle θp refers to the main axis vector v max The angle between the projection on the vertical direction (z-axis) and the horizontal plane is used to determine the upward or downward angle of the stamen relative to the horizontal plane. The calculation formula is as follows:

[0063]

[0064] Where, is a unit vector pointing in the vertical direction;

[0065] The yaw angle θy refers to the main axis vector v max The angle between the projection on the horizontal plane (xy plane) and a reference direction is used to determine the rotation direction of the pistil in the horizontal plane. The calculation formula is as follows:

[0066]

[0067] Where, and are unit vectors pointing to the x and y axes;

[0068] If θp = 0°, θy = 0°, the pistil is horizontal and points in the positive direction of the x-axis;

[0069] If θp = 45°, θy = 90°, the pistil is tilted 45° upward and points in the positive direction of the y-axis;

[0070] If θp = -30°, θy = -45°, the pistil is tilted 30° downward and points to the fourth quadrant between the x-axis and the y-axis.

[0071] In step S7, the calculation formula is as follows:

[0072]

[0073] Where: X', Y', Z' are the actual coordinates in three-dimensional space; (u', v') are the two-dimensional coordinates of the pixel point in the image; (c' x ,c' y ) is the center point of the image; D' is the depth value corresponding to the pixel; f'x and f' y is the focal length of the camera.

[0074] The present invention can obtain a tomato pistil depth image from a depth camera and the coordinate information and depth information of each pixel in the tomato pistil depth image. The tomato pistil depth image is subjected to image denoising and image segmentation in sequence, thereby better removing background information and segmenting the tomato pistil. In feature extraction, the orientation angle of the tomato pistil can be obtained based on the coordinate information. The coordinate information and depth information of each pixel in the tomato pistil depth image are combined with the depth camera parameters and converted according to a three-dimensional coordinate formula to obtain the actual position information of the tomato pistil, thereby providing accurate position and angle information for the pollination robot.

[0075] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for locating and identifying the orientation of tomato pistils in facility production, characterized in that: The steps include: Step S1, installing a depth camera on the pollination robot arm of the pollination robot to take a photo of the tomato pistil to obtain a depth image of the tomato pistil and coordinate information and depth information of each pixel point in the depth image of the tomato pistil; Step S2, using Gaussian filtering to reduce noise on the tomato pistil depth image, and processing the reduced noise tomato pistil depth image using the OpenCV library to obtain a tomato pistil image in HSV format; Step S3, using an image threshold segmentation technique to segment the tomato pistil from the background in the tomato pistil image in the HSV format to obtain a tomato pistil segmentation image in the HSV format; Step S4, segmenting the tomato pistil from the tomato pistil depth image to obtain a tomato pistil image; Step S5, extracting features from the tomato pistil image to determine the orientation of the tomato pistil in three-dimensional space; Step S5 includes the following sub-steps: Step S51: Calculate the mean μ of the segmented tomato pistil image point cloud data using the following formula: Where μ is the mean vector of all point coordinates, P i =(x i ,y i ,z i ) is the coordinate of the i-th point, N is the total number of coordinate points; Step S52: Calculate the covariance matrix C of the point cloud. The calculation formula is as follows: Step S53: Perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalue λ and eigenvector v. The calculation formula is as follows: Cv i =λ i v i Where λ i Represents the variance of the data in the direction of the eigenvector; v i is the corresponding eigenvector, indicating the principal axis in that direction; Step S54: The eigenvector v corresponding to the obtained maximum eigenvalue max This is the direction of the tomato pistil, calculated as follows: Step S6, using the orientation of the tomato pistil, determining the pitch angle θp of the tomato pistil relative to the vertical direction and the yaw angle θy of the tomato pistil relative to the horizontal plane, thereby obtaining the orientation angle information of the tomato pistil in three-dimensional space; Step S7, using the coordinate information and depth information of each pixel in the tomato pistil depth image, combined with the depth camera parameters, and converting according to the three-dimensional coordinate formula to obtain the actual position information of the tomato pistil; In step S8, the pollination robot uses the actual position information and orientation angle information of the tomato pistil obtained in the above steps to control the pollination robot arm to perform precise operations according to the information.

2. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S2, the tomato pistil depth image is normalized to the range of 0 to 255 using the normalize function in the OpenCV library; the normalized image is converted into an RGB pseudo-color image using the applyColorMap function; and the RGB pseudo-color image is converted into an HSV format tomato pistil image using the cvtColor function.

3. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S3, the hue H of the tomato pistil is set between 26 and 34, the saturation S is set between 43 and 255, and the brightness V is set between 46 and 255. The lowest value of the HSV threshold range is set to [26, 43, 46], and the highest value is set to [34, 255, 255]. The tomato pistil within the HSV threshold range is separated from the background, thereby obtaining a tomato pistil segmentation image in HSV format.

4. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S4, the HSV threshold range set in step S3 is used to segment the tomato pistil image region in HSV format to generate a binary mask image, wherein the tomato pistil region within the HSV threshold range is set to white with a pixel value of 1, and the background outside the HSV threshold range is set to black with a pixel value of 0. The Canny edge detection algorithm is used to extract the edges in the binary mask image to find the precise boundary of the tomato pistil region. Based on the edge detection results, the contour detection algorithm findContours function is used to extract the contour of the pistil region. Finally, the area within the contour is used as the tomato pistil, and a segmentation mask is generated. The mask can be used to separate the tomato pistil region from the original tomato pistil depth image to obtain a tomato pistil image.

5. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S5, the coordinate information of each pixel point of the tomato pistil image is obtained using a depth camera, and feature extraction is performed on the segmented tomato pistil image.

6. The method for locating and identifying the orientation of tomato pistils in facility production according to claim 1, characterized in that: In step S6, the pitch angle θp refers to the main axis vector v max The angle between the projection on the vertical z-axis and the horizontal plane is used to determine the upward or downward angle of the pistil relative to the horizontal plane. The calculation formula is as follows: Where, is a unit vector pointing in the vertical direction; The yaw angle θy refers to the main axis vector v max The angle between the projection on the horizontal xy plane and a reference direction is used to determine the rotation direction of the pistil in the horizontal plane. The calculation formula is as follows: Where, and are unit vectors pointing to the x and y axes; If θp = 0°, θy = 0°, the pistil is horizontal and points in the positive direction of the x-axis; If θp = 45°, θy = 90°, the pistil is tilted 45° upward and points in the positive direction of the y-axis; If θp = -30°, θy = -45°, the pistil is tilted 30° downward and points to the fourth quadrant between the x-axis and the y-axis.

Images