Tomato stem recognition and three-dimensional reconstruction method based on deep learning
By using deep learning and Mask RCNN models to identify tomato stems, and combining color and depth cameras, accurate 3D reconstruction of tomato stems was achieved, solving the problem of stem identification and reconstruction in existing technologies and improving the harvesting efficiency and accuracy of agricultural robots.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2022-09-01
- Publication Date
- 2026-06-02
AI Technical Summary
Existing 3D reconstruction methods struggle to effectively identify and reconstruct tomato stems, especially when leaves and fruits obscure the view, affecting the accuracy and efficiency of the harvesting process.
A deep learning-based approach was adopted, using Mask RCNN convolutional neural network combined with color and depth cameras to acquire, calibrate, and identify images of tomato stems. The Mask RCNN model was used to generate the minimum bounding rectangle pixel vertex coordinates of the stem segment, and the radius and height of the stem were calculated by matching the depth and color images. The results were then converted into the robot's base coordinate system to complete the reconstruction of the stem obstacle.
It achieves accurate and rapid identification and 3D reconstruction of tomato stems, improving the obstacle avoidance ability of agricultural robots during harvesting, and has high accuracy, stability and anti-interference ability.
Smart Images
Figure CN115439746B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of agricultural robots, and in particular to a method for tomato stem recognition and 3D reconstruction based on deep learning. Background Technology
[0002] In addition to fruit recognition and localization, the vision system of a tomato harvesting robot also plays a crucial role in the recognition and reconstruction of stems. Tomato stems and leaves can obscure the fruit; while the impact of leaves is relatively minor, the stems, being generally stiff, significantly affect harvesting. During harvesting, agricultural robots must avoid the stems to harvest effectively. Therefore, research on the recognition and 3D reconstruction of fruit and vegetable stems is of great significance.
[0003] Currently, common 3D reconstruction methods mainly include: voxel-based, point cloud-based, and mesh-based descriptions of 3D objects. Point cloud-based methods describe 3D objects at high resolution, and the reconstructed 3D objects contain not only shape information but also color information. Point cloud processing generates irregular data formats, reducing memory consumption but preventing the use of convolutional operations; furthermore, the number of points in a point cloud is finite. For the reconstruction of tomato stems, the obstacles generated by point clouds contain a large amount of leaf surface noise, significantly impacting obstacle avoidance planning; therefore, point clouds are not a suitable processing method. When reconstructing stems, the key is the spatial pose information of the stem; the requirement for high detail in stem features is not critical. When using convolutional neural networks (CNNs) to recognize objects, classification and location information of the target object can be obtained. Compared to traditional methods, CNNs are faster and provide more complete recognition information. In conclusion, using deep learning CNNs as a stem recognition method, and reconstructing stem obstacles based on the 3D information of the recognition box, is highly necessary. Summary of the Invention
[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a deep learning-based method for tomato stem recognition and 3D reconstruction, which can accurately and quickly identify and reconstruct tomato stems, facilitating obstacle avoidance during robot harvesting.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] A deep learning-based method for tomato stem recognition and 3D reconstruction includes the following steps:
[0007] S1. Use a camera to collect tomato stem images from different locations and angles in the tomato greenhouse, taking into account different light conditions, leaf and fruit shading conditions, and establish a tomato stem image dataset.
[0008] S2. Perform parameter calibration for the color camera and depth camera;
[0009] S3. Using Mask RCNN convolutional neural network training, a better recognition effect is obtained, and the trained Mask RCNN model is used to recognize tomato color images to obtain the pixel vertex coordinates of the minimum bounding rectangle of the stem segment.
[0010] S4. Using depth and color image matching, obtain the vertex coordinates and center 3D coordinates based on the camera coordinate system, and calculate the radius r and height h of the stem;
[0011] S5. Convert the stem coordinates based on camera coordinates to a representation based on the robot base coordinate system, and use the vertex coordinates to solve the spatial pose RPY angle of the stem;
[0012] S6. Convert the spatial attitude angle into ROS four elements, and create the tomato stem obstacle based on the radius r, height h, and center three-dimensional coordinates.
[0013] Furthermore, the specific method for establishing the tomato stem image dataset in step S1 is as follows:
[0014] S1-1. The stems of tomatoes are irregular in length and shape, and are more likely to be blocked by leaves and fruits. When collecting data, it is necessary to consider not only the light conditions, but also various interferences.
[0015] S1-2. The grid discretization marking method is adopted, and the stem is marked with regular quadrilaterals to ensure that the area of the pixels contained in the quadrilateral is as consistent as possible with the area of the pixels contained in the stem. The long side of the quadrilateral is along the growth direction of the stem, and the short side is parallel to the cross section of the stem. The grid is kept as a quadrilateral during the marking process.
[0016] Furthermore, the parameter calibration of the color camera and depth camera described in step S2 is specifically performed as follows:
[0017] S2-1. Take pictures of the calibrated chessboard in different poses using a color camera and a depth camera, and use the calibration function findChessboard Corners to obtain the pixel coordinates of the corresponding corner points;
[0018] S2-2. By inputting the coordinates of the interior corner points through the calibration function calibrateCamera, the intrinsic parameter matrix of the depth camera is obtained;
[0019] S2-3. Register the camera to obtain the color and depth information of the object. Utilize the transformation relationship between the depth image and the color image and the world coordinate system to align the depth image and the color image through coordinate transformation.
[0020] Further, step S3 utilizes the trained Mask RCNN model to recognize the tomato color image and obtain the pixel vertex coordinates of the minimum bounding rectangle of the stem segment; the specific method is as follows:
[0021] S3-1. Using Mask RCNN, ResNet and feature pyramid structures, as well as the prediction branch of Mask, are used for contour localization of the target object; the stem image is input into the backbone network to extract image features and generate a two-dimensional matrix;
[0022] S3-2. Use RPN to generate a certain number of regions of interest on a two-dimensional matrix, perform binary classification on the candidate regions and correct the positions of the candidate boxes, and discard some candidate boxes through non-maximum suppression;
[0023] S3-3. The remaining candidate regions are fed into ROIAlign, and the two-dimensional matrices of all candidate regions are transformed into feature vectors of a specified size.
[0024] S3-4. Perform classification prediction, regression prediction, and mask prediction on candidate regions.
[0025] Further, in step S4, the vertex pixel coordinates of the stem segment's contour moments are converted into camera-based spatial three-dimensional coordinates, and the center three-dimensional coordinates, stem radius r, and height h are calculated. The specific implementation method is as follows:
[0026] S4-1. The four corner points of the profile moment in a two-dimensional plane are P i (x i ,y i (i = 1, 2, 3 or 4), the side length L of the rectangle is obtained from the coordinates of the adjacent corner points;
[0027] S4-2. After obtaining the long and short sides of the rectangle, calculate the stem radius r and height h;
[0028] S4-3. Find the three-dimensional coordinates of the center point by taking the midpoint of the shorter side.
[0029] Further, in step S5, the stem coordinates based on camera coordinates are converted to a representation based on the robot base coordinate system, and the spatial pose RPY angle of the stem is solved using the vertex coordinates. The specific method is as follows:
[0030] S5-1. By transforming the camera coordinate system and the world coordinate system, the coordinates of the tomato stem center based on the camera are converted to those based on the world coordinate system.
[0031] S5-2. The orientation of the profile moment is obtained by solving the rotation angles of the profile moment about each axis. Take the midpoints A and B of the short side of the profile moment, and project the two points onto the XY plane to obtain A'(x1,y1,z1) and B'(x2,y2,z2). Calculate the rotation angle α of the profile moment about the Z axis; similarly, obtain the rotation angle β about the Y axis and the rotation angle γ about the X axis.
[0032] Furthermore, in step S6, the spatial attitude angle is converted into a ROS quaternion, and the tomato stem obstacle is created based on the radius r, height h, and center three-dimensional coordinates. The specific method is as follows: ROS uses quaternions to describe the attitude of an object. A quaternion consists of a scalar and a three-dimensional vector. The spatial attitude angle RPY is converted into a ROS quaternion using the properties of quaternions. Combined with the radius r, height h, and center coordinates x, y, z, the stem obstacle is reconstructed.
[0033] Compared with the prior art, the beneficial effects of the present invention are:
[0034] 1) This invention can use Mask RCNN convolutional neural network to detect and locate tomato stems, with high accuracy, good stability, strong anti-interference ability and high versatility.
[0035] 2) This invention combines the Mask R-CNN model and the ROS system, using the target boxes generated by the Mask R-CNN model to convert the coordinates and other information of the obstacles reconstructed by ROS, thus completing the reconstruction of the stalk obstacle and achieving a good reconstruction effect. Attached Figure Description
[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. The technical solutions of the present invention will be further elaborated in detail below with reference to the accompanying drawings and embodiments.
[0037] Figure 1 This is a flowchart of the deep learning-based tomato stem recognition and 3D reconstruction method of the present invention.
[0038] Figure 2 This is a captioned image of a tomato stem from the present invention.
[0039] Figure 3 This is a structural diagram of the Mask RCNN model for tomato stem recognition in this invention.
[0040] Figure 4 This image shows the recognition effect of the tomato stem using a trained Mask RCNN model, as tested in this invention.
[0041] Figure 5This is a diagram illustrating the pose of a three-dimensional object in ROS according to the present invention.
[0042] Figure 6 This is a schematic diagram illustrating the method for solving the rotation angle of the profile moment about the Z-axis according to the present invention.
[0043] Figure 7 This is an illustration of the identification effect of tomato stems according to the present invention.
[0044] Figure 8 This is a reconstruction effect diagram of tomato stem obstacles under ROS according to the present invention. Detailed Implementation
[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0046] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0047] like Figure 1 As shown, the present invention provides a deep learning-based method for tomato stem recognition and 3D reconstruction, with the following specific steps:
[0048] S1. Using a camera, tomato stem images were collected from different locations and angles in the tomato greenhouse, taking into account various lighting conditions, leaf and fruit shading, etc., to establish a tomato stem image dataset. The dataset was then labeled using the Labelme annotation tool. Specifically:
[0049] S1-1. The growth of tomato stems is random, and the length and shape of the stems are irregular. They are more likely to be blocked by leaves and fruits. When collecting data, it is necessary to consider not only the light conditions, but also various interferences.
[0050] S1-2. Labeling tomato stems as a whole weakens the trunk shape features extracted by the convolutional neural network and requires more data collection. A grid discretization labeling method is adopted, using regularly shaped quadrilaterals to label the stems. This ensures that the area of the pixels contained in the quadrilateral is as consistent as possible with the area of the pixels contained in the stem. The long side of the quadrilateral follows the growth direction of the stem, and the short side is parallel to the stem cross-section. The grid remains quadrilateral during the labeling process. The labeling result is as follows: Figure 2 As shown.
[0051] S2. Perform parameter calibration for the color camera and depth camera;
[0052] S2-1. Using Zhang Zhengyou's calibration method, a checkerboard pattern is used for camera intrinsic parameter calibration and image registration. During calibration, the camera position remains constant while the position and orientation of the checkerboard pattern are changed. The correspondence between each feature point on the calibration board and its corresponding planar image point is established. A corner point localization algorithm is used to extract the coordinates of the interior corner points (excluding checkerboard boundary corner points) in each calibration board image in the pixel coordinate system. During calibration, the program sequentially reads the calibration board image and extracts the corner point information. Using this corner point information as initial values, the corner point information of the calibration board image at the sub-pixel level is further determined. The pixel coordinates of the corresponding interior corner points are obtained. The calibration board images under different orientations are converted to grayscale images. Based on the color characteristics of the black and white blocks on the calibration board, the OpenCV calibration function `findChessboard Corners` is used to obtain the corresponding corner point pixel coordinates.
[0053] S2-2. Obtain the intrinsic parameter matrix D of the depth camera by inputting the coordinates of the interior corner points using the OpenCV calibration function calibrateCamera. i ;
[0054]
[0055] S2-3. After obtaining the camera's intrinsic parameter matrix, register the camera to acquire the object's color and depth information, avoiding pixel mismatches. Let the transformation matrix from depth image to color image be [R]. d2r ,T d2r If ], then a relationship exists:
[0056]
[0057] Among them, X d Y d Z d X represents the three-dimensional coordinates of a point on the depth image. r Y r Z r R represents the three-dimensional coordinates of a point on the converted color image. At the same moment, for the same point in three-dimensional space, R... d T d R represents the rotation matrix and translation vector of the depth image relative to the world coordinate system. r T r Representing the rotation matrix and translation vector of the color image relative to the world coordinate system, we can obtain:
[0058] R d2r =R r ·R d -1
[0059] T d2r =T r -Rd2r ·T d
[0060] After coordinate transformation, the depth image and color image are aligned.
[0061] S3. Using Mask R-CNN convolutional neural network training, a good recognition effect was obtained. The Mask R-CNN model structure diagram is shown below. Figure 3 As shown. The trained Mask R-CNN model is then used to recognize tomato color images, obtaining the pixel vertex coordinates of the minimum bounding rectangle of the stem segment. The specific method is as follows:
[0062] S3-1. The ResNet, feature pyramid structure, and prediction branch of the Mask R-CNN network are used to locate the contour of the target object. The stem image is input into the backbone network to extract image features and generate a two-dimensional matrix.
[0063] S3-2. Use RPN to generate a certain number of regions of interest on a two-dimensional matrix, perform binary classification on the candidate regions and correct the candidate box positions, and discard some candidate boxes through non-maximum suppression.
[0064] S3-3: The remaining candidate regions are fed into ROIAlign, and the two-dimensional matrices of all candidate regions are transformed into feature vectors of a specified size.
[0065] S3-4. Perform classification prediction, regression prediction, and mask prediction on the candidate regions to generate the required labeled image.
[0066] Test color images using a trained Mask R-CNN model, such as Figure 4 The results shown are partial detection results from 100 tomato stem images. 563 stem segments were detected correctly, 21 stem segments were detected incorrectly, and 116 stem segments were not detected, with a detection accuracy of 96.4%.
[0067] S4. Using depth and color image matching, obtain the vertex coordinates and center 3D coordinates based on the camera coordinate system, and calculate the radius r and height h of the stem; Figure 5 The image shows the description of the pose of a 3D object in ROS according to this invention. ROS reconstructs objects of different shapes. After the tomato stem is segmented, each segment's model approximates a cylinder. The generation of a cylinder requires its height, base radius, center point coordinates, and pose. Common spatial pose angles are represented using RPY. ROS uses four elements x, y, z, and w to represent the pose, and the general mathematical expression is:
[0068] q = (x, y, z, w) = xi + yj + zk + w
[0069] Where i, j, and k represent the basis of the spatial vectors, and satisfy i 2 +j 2 +k 2 =ijk=-1, and |q|=1 is a unit quaternion.
[0070] The radius r and height h of the tomato stem do not change during coordinate transformation. This is calculated first, and the principle is as follows:
[0071] S4-1. The three-dimensional information of the stem is obtained based on the spatial coordinates of the corner points. Let P be the four corner points of the profile moment in the two-dimensional plane. i (x i ,y i (i = 1, 2, 3, or 4), the side length of the rectangle can be obtained from the coordinates of adjacent corner points:
[0072]
[0073] Among them, (x j ,y j The coordinates of the adjacent corner points are used, and the longer one, L, is obtained through calculation. l The shorter one is the shorter side L. s .
[0074] S4-2. After obtaining the long and short sides of the rectangle, the radius r and height h of the stem can be calculated:
[0075] r = L s / 2,h=L l
[0076] S4-3. The depth information of a point can be obtained from the pixel coordinates in the two-dimensional plane. Since the four corner points of the contour moments cannot be guaranteed to fall accurately on the stem, the midpoint of the shorter side is used to solve for the three-dimensional coordinates of the center point. Assuming the midpoint depths of P1P2 and P3P4 are a and b respectively, the three-dimensional coordinates of the center P0 are:
[0077] P0 = (x1 + r, y1 - h / 2, (a + b) / 2)
[0078] S5. Convert the stem coordinates based on camera coordinates to a representation based on the robot's base coordinate system, and use the vertex coordinates to solve for the spatial pose (RPY) angle of the stem; the specific method is as follows:
[0079] S5-1. Using the transformation relationship between the camera coordinate system and the world coordinate system, transform the coordinates of the tomato stem center based on the camera coordinate system to those based on the world coordinate system. w -x w y w z w With camera coordinate system oc -x c y c z c The transformation equation is as follows:
[0080]
[0081] Where R is a 3×3 rotation matrix and T is a three-dimensional translation vector.
[0082] S5-2. The orientation of the profile moment is obtained by solving for the rotation angles of the profile moment about each axis. The midpoints A and B of the shorter sides of the profile moment are taken, and these two points are projected onto the XY plane to obtain A'(x1,y1,z1) and B'(x2,y2,z2). For example... Figure 6 The diagram illustrates the method for calculating the rotation angle of the profile moment about the Z-axis according to this invention. RPY represents the object's rotation about the x, y, and z axes, used to represent the object's spatial pose. ROS uses quaternions x, y, z, and w to represent the spatial pose, requiring the acquisition of the object's rotation angle based on the base coordinate system. Finally, the conversion between the quaternion and the RPY angle is completed. The rotation angle α of the profile moment about the Z-axis is:
[0083]
[0084] Similarly, the rotation angle β around the Y-axis and the rotation angle γ around the X-axis are:
[0085]
[0086]
[0087] S6. Convert the spatial attitude angle into ROS four elements, and create the tomato stem obstacle based on the radius r, height h, and center three-dimensional coordinates of the tomato stem. The specific method is as follows:
[0088] ROS uses quaternions to describe the pose of an object. A quaternion consists of a scalar and a three-dimensional vector, and is an extension of the complex number domain. Let q be a quaternion, and its mathematical expression is:
[0089] q = (x, y, z, w) = xi + yj + zk + w
[0090] Among them, i 2 +j 2 +k 2 =ijk=-1, and |q|=1 is a unit quaternion.
[0091] By utilizing the properties of quaternions, the spatial attitude angle RPY can be converted into a ROS quaternion. Combined with the radius r, height h, and center coordinates x, y, z, the stalk obstacle can be reconstructed. Figure 7 The image shown is an illustration of the recognition effect of tomato stems. Figure 8The image shown is a reconstruction result of tomato stem obstacles in ROS.
[0092] The above embodiment describes a deep learning-based method for tomato stem recognition and 3D reconstruction. It utilizes cameras to collect tomato stem images from different locations and angles within a tomato greenhouse, establishing a tomato stem image dataset. Parameter calibration of the color and depth cameras is performed. A Mask R-CNN convolutional neural network is trained to achieve good recognition results. The trained Mask R-CNN model is then used to recognize tomato color images, obtaining the pixel vertex coordinates of the minimum bounding rectangle of the stem segment. Depth and color images are matched to obtain vertex coordinates and center 3D coordinates based on the camera coordinate system, and the stem radius *r* and height *h* are calculated. The stem coordinates based on the camera coordinate system are converted to a robot-based coordinate system representation, and the spatial pose (RPY) angle of the stem is solved using the vertex coordinates. The spatial pose angle is converted to ROS four-element representation, and tomato stem obstacles are created based on the tomato stem radius *r*, height *h*, and center 3D coordinates.
[0093] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made according to the purpose of the invention. Any changes, modifications, substitutions, combinations or simplifications made based on the spirit and principle of the technical solution of the present invention shall be equivalent substitutions. As long as they meet the purpose of the invention and do not deviate from the technical principle and inventive concept of the present invention, they shall fall within the protection scope of the present invention.
Claims
1. A method for tomato stem recognition and 3D reconstruction based on deep learning, characterized in that, Includes the following steps: S1. Use a camera to collect tomato stem images from different locations and angles in the tomato greenhouse, taking into account different light conditions, leaf and fruit shading conditions, and establish a tomato stem image dataset. S2. Perform parameter calibration for the color camera and depth camera; S3. Using Mask RCNN convolutional neural network training, a better recognition effect is obtained, and the trained Mask RCNN model is used to recognize tomato color images to obtain the pixel vertex coordinates of the minimum bounding rectangle of the stem segment. S4. Using depth and color image matching, obtain the vertex coordinates and center 3D coordinates based on the camera coordinate system, and calculate the radius of the stem. r and height h ; S5. Convert the stem coordinates based on camera coordinates to a representation based on the robot base coordinate system, and use the vertex coordinates to solve the spatial pose RPY angle of the stem; S6. Convert the spatial attitude angle into a ROS quaternion, and then convert it according to the radius. r and height h The three-dimensional coordinates of the center complete the creation of the tomato stem obstacle.
2. The method for tomato stem recognition and 3D reconstruction based on deep learning according to claim 1, characterized in that, The specific method for establishing the tomato stem image dataset in step S1 is as follows: S1-1. The stems of tomatoes are irregular in length and shape, and are more likely to be blocked by leaves and fruits. When collecting data, it is necessary to consider not only the light conditions, but also various interferences. S1-2. The grid discretization marking method is adopted, and the stem is marked with regular quadrilaterals to ensure that the area of the pixels contained in the quadrilateral is as consistent as possible with the area of the pixels contained in the stem. The long side of the quadrilateral is along the growth direction of the stem, and the short side is parallel to the cross section of the stem. The grid is kept as a quadrilateral during the marking process.
3. The method for tomato stem recognition and 3D reconstruction based on deep learning according to claim 1, characterized in that, The parameter calibration of the color camera and depth camera mentioned in step S2 is specifically performed as follows: S2-1. Take pictures of the calibrated chessboard in different poses using a color camera and a depth camera, and use the calibration function findChessboard Corners to obtain the pixel coordinates of the corresponding corner points; S2-2. By inputting the coordinates of the interior corner points through the calibration function calibrateCamera, the intrinsic parameter matrix of the depth camera is obtained; S2-3. Register the camera to obtain the color and depth information of the object. Utilize the transformation relationship between the depth image and the color image and the world coordinate system to align the depth image and the color image through coordinate transformation.
4. The deep learning-based tomato stem recognition and 3D reconstruction method according to claim 1, characterized in that, Step S3 uses a trained Mask RCNN model to recognize tomato color images and obtain the pixel vertex coordinates of the minimum bounding rectangle of the stem segment; the specific method is as follows: S3-1. Using Mask RCNN, ResNet and feature pyramid structures, as well as the prediction branch of Mask, are used for contour localization of the target object; the stem image is input into the backbone network to extract image features and generate a two-dimensional matrix; S3-2. Use RPN to generate a certain number of regions of interest on a two-dimensional matrix, perform binary classification on the candidate regions and correct the positions of the candidate boxes, and discard some candidate boxes through non-maximum suppression; S3-3. The remaining candidate regions are fed into ROI Align, and the two-dimensional matrices of all candidate regions are transformed into feature vectors of a specified size. S3-4. Perform classification prediction, regression prediction, and mask prediction on candidate regions.
5. The method for tomato stem recognition and 3D reconstruction based on deep learning according to claim 1, characterized in that, In step S4, the vertex pixel coordinates of the stem segment's contour moments are converted into camera-based spatial 3D coordinates, and the center 3D coordinates and stem radius are calculated. r and height h The specific implementation method is as follows: S4-1. The four corner points of the profile moment in a two-dimensional plane are P i ( x i , y i ), i =1, 2, 3, or 4, and obtain the side length of the rectangle based on the coordinates of adjacent corner points. L ; S4-2. After obtaining the long and short sides of the rectangle, calculate the stem radius. r and height h ; S4-3. Find the three-dimensional coordinates of the center point by taking the midpoint of the shorter side.
6. The method for tomato stem recognition and 3D reconstruction based on deep learning according to claim 1, characterized in that, In step S5, the stem coordinates based on camera coordinates are converted to a representation based on the robot's base coordinate system, and the spatial pose RPY angle of the stem is solved using the vertex coordinates. The specific method is as follows: S5-1. By transforming the camera coordinate system and the world coordinate system, the coordinates of the tomato stem center based on the camera are converted to those based on the world coordinate system. S5-2. The orientation of the profile moment is obtained by solving for the rotation angles of the profile moment about each axis, and the midpoint of the shorter side of the profile moment is taken. A , B Projecting the two points onto XY Plane, to obtain A’ ( x 1, y 1, z 1) and B’ ( x 2, y 2, z 2) The calculated profile moment around Z Angle of rotation of the shaft α Similarly, we obtain the result of the loop. Y Angle of rotation of the shaft β and around X Angle of rotation of the shaft γ .
7. The method for tomato stem recognition and 3D reconstruction based on deep learning according to claim 1, characterized in that, In step S6, the spatial attitude angle is converted into a ROS quaternion, and then based on the radius... r and height h The tomato stem obstacle is created using the central three-dimensional coordinates. The specific method is as follows: ROS uses quaternions to describe the object's pose; a quaternion consists of a scalar and a three-dimensional vector. The spatial pose angle RPY is converted to a ROS quaternion using the properties of quaternions, combined with the radius... r and height h and center coordinates x , y , z Complete the reconstruction of the stem obstacle.