A method for pose estimation and fruit peduncle cutting point positioning of pitaya based on a three-dimensional key point detection model

Abstract

The application provides a pitaya pose estimation and fruit stem cutting point positioning method based on a three-dimensional key point detection model, acquires pitaya RGB-D image data, constructs a pitaya fruit and branch point cloud set and performs attitude fitting, obtains spatial three-dimensional key point coordinates, and performs segmentation and labeling on the pitaya fruit and branch of a color image, constructs a three-dimensional key point detection model based on Mask R-CNN, and finally trains the three-dimensional key point detection model by using segmentation and labeling data and three-dimensional key point coordinate data, predicts the attitude vector when the pitaya is grabbed and the spatial three-dimensional coordinates of the fruit stem cutting point when the pitaya is cut, and realizes the pose estimation and three-dimensional fruit stem cutting point positioning when the pitaya is grabbed and cut.

Description

Technical Field

[0001] The present invention relates to the fields of automatic fruit picking robots and computer vision technology, and in particular to a method for estimating the posture of pitaya and locating the fruit stem shearing point based on a three-dimensional key point detection model. Background Art

[0002] With rising consumption levels and a growing awareness of healthy eating in my country, dragon fruit, a tropical fruit, has gained popularity among consumers for its unique appearance, vibrant skin, and sweet taste, leading to growing market demand. Traditional agriculture relies heavily on manual labor, but with the continued expansion of cultivated areas, harvesting the fruit faces a potential labor shortage.

[0003] Fruit-harvesting robots have developed rapidly in recent years, playing a crucial role in addressing labor shortages, improving harvesting efficiency, and reducing harvesting costs. Harvesting pitaya requires specialized scissors to cut the fruit from both ends of the branch, retaining the stem to preserve its freshness during storage and transportation. However, most existing pitaya-harvesting robots use a direct grasping method, which can damage the fruit's peel and lead to rot during storage or transportation.

[0004] Therefore, in the visual task of the pitaya picking robot, in addition to accurately identifying the pitaya, determining the growth posture vector of the fruit, and providing accurate grasping information for the picking robot, it is also necessary to locate the shearing point of the pitaya branch and provide the shearing actuator at the end with accurate three-dimensional coordinates of the shearing point, which is of great significance for the lossless picking of pitaya. Summary of the Invention

[0005] In view of this, the present invention uses RGB-D image data of pitaya in an unstructured pitaya orchard environment to obtain the real three-dimensional coordinates of key points during the pitaya picking process. A method for estimating the posture of pitaya and locating the shearing points of the fruit stem based on a three-dimensional key point detection model is proposed. This method achieves accurate estimation of the posture of the pitaya fruit and accurate positioning of the shearing points of the branches, thereby solving the problem of precise grasping and shearing required during automated pitaya picking operations. To achieve the above objectives, the present invention provides the following technical solutions:

[0006] Step (1): Build a dragon fruit RGB-D image data acquisition system to collect the dragon fruit RGB-D image dataset in an unstructured dragon fruit orchard environment;

[0007] Step (2): construct a three-dimensional point cloud image using the dragon fruit RGB-D image to obtain the true three-dimensional coordinates of the dragon fruit pose points and the shearing points of the corresponding branches in the camera coordinate system;

[0008] Step (3): Use the dragon fruit color image to perform instance segmentation and annotation of the dragon fruit fruit and its corresponding branches, and construct a dragon fruit image segmentation dataset;

[0009] Step (4): Construct a 3D key point detection model based on Mask R-CNN to predict the 3D coordinates of the dragon fruit pose points and the corresponding branch cutting points;

[0010] Step (5): Determine the fruit posture vector as a three-dimensional vector from the navel point to the fruit stem point, and the shear point coordinates are the spatial three-dimensional coordinates of the two end points of the branch in the predicted camera coordinate system.

[0011] The step (1) comprises:

[0012] A dragon fruit RGB-D image data acquisition system was constructed. The dragon fruit in the public dragon fruit dataset was labeled using Labelme annotation software. The dragon fruit was then trained on YOLOv8 to recognize dragon fruit and obtain a pre-trained YOLOv8 model for recognizing dragon fruit.

[0013] The graphical user interface (GUI) was used to initialize and layout the user interface components of the pitaya RGB-D image acquisition system. This included displaying color images, depth images, target recognition result images, collecting image counts, saving images, and exiting the system.

[0014] Using an Intel RealSense D435i camera, we enable depth and color image capture by calling the rs.pipeline(), rs.config(), and enable_stream() methods. We set the resolution of both depth and color images to 640×480. By aligning the depth and color data, we unify their coordinate references, enabling the device to output multimodal image data in a unified depth alignment mode. We also use the loaded YOLOv8 model to perform object detection on the color image, generate prediction results, draw annotated images, and update the corresponding display components in the GUI.

[0015] When you choose to save the current frame (by clicking the "SAVE" component on the system interface or the "Space" button on the keyboard), the save_frame() method executes. A storage directory is created based on the predefined file path structure, and the color image and depth image corresponding to the current moment, as well as the image output and annotated by YOLOv8 inference, are stored. The image counter component increments by one, and the position information of the dragon fruit target detection frame (including the 2D bounding box coordinates and size), the depth information of the corresponding pixel position, and the spatial coordinates obtained by back-projection of the camera intrinsic parameters are written to a CSV file for record and storage.

[0016] When you choose to exit the system (click the "EXIT" component on the system interface or press the "esc" button on the keyboard), execute the exit_app() method to stop the depth camera data pipeline, close the window, release related resources, and ensure that the system terminates in an orderly manner.

[0017] During the acquisition of dragon fruit RGB-D images in an unstructured orchard environment, the camera was controlled parallel to the fruit tree, and the imaging distance was approximately 0.2-0.7m.

[0018] The step (2) comprises:

[0019] The collected dragon fruit color image and depth image data are used to generate a corresponding point cloud set. The point cloud set is imported into the CloudCompare point cloud annotation tool to perform ellipsoid fitting on the dragon fruit point cloud and obtain the three-dimensional coordinates of the ellipsoid center.

[0020] The pitaya fruit pose points include three key points, including the center of mass point P0, the fruit stem point P1, and the fruit navel point P2. The center point of the fitted point cloud ellipsoid is determined as the center of mass point P0 of the pitaya fruit. The coordinates of the two point clouds farthest from the ellipsoid center point on the ellipsoid point cloud are determined as the fruit stem point and the fruit navel point of the pitaya fruit. The specific calculation formula is as follows:

[0021] Assume that the point set of the ellipsoid point cloud is P = {p1,p2,…,p n}, the center of the ellipsoid is c,

[0022] The Euclidean distance from each point to the center is:

[0023] d i =||p i -c||,i=1,2…,n

[0024] Where ||·|| represents the Euclidean norm of the vector. By calculating the distance of all points, the point with the largest distance to the center point and its symmetrical points about the center point are the fruit stem point P1 and the fruit navel point P2.

[0025] A small-scale quadratic surface fitting is performed on the front end point cloud of the corresponding branch of the pitaya fruit, and the maximum dimension direction of the quadratic surface oriented bounding box is determined, and the two endpoints are taken as the shearing points C1 and C2 in two directions of the branch.

[0026] The three-dimensional spatial coordinates of the centroid point P0, fruit stem point P1, fruit navel point P2 and two shear points C1 and C2 obtained by fitting in the camera coordinate system constitute the real data set for three-dimensional key point detection.

[0027] The step (3) comprises:

[0028] The pitaya color image is segmented by instance on the pitaya fruit and its corresponding branches using the Via annotation tool to construct an instance segmentation dataset. The ellipsoid and cylinder fitted in step (2) are back-projected onto the color image to obtain a complete pitaya fruit and branch mask. The mask is then processed to obtain the fitted segmentation data for the fruit and branch. The original segmentation annotation data and the fitted segmentation annotation data are applied simultaneously to the instance segmentation network to train the model to segment and predict the posture of the occluded fruit, thereby improving the robustness of the model.

[0029] The step (4) comprises:

[0030] The 3D key point detection model is based on the network architecture of Mask R-CNN. The basic Mask R-CNN includes: the residual network ResNet as the backbone network; the feature pyramid network FPN for feature extraction; the region proposal network RPN generates candidate regions ROI on the feature map; for the ROI provided by RPN, the ROIAlign operation is performed on the region of interest to accurately extract the feature information of the corresponding region from the feature map, and the extracted features flow into the classification and bounding box regression head; the same ROI features are input into the mask prediction head Mask Head, and the convolutional layer stacking is used to predict the pixel-level segmentation mask to obtain the binary mask of the dragon fruit and branches, realizing instance segmentation.

[0031] After the mask segmentation task is completed, the generated mask is fused with the feature map after ROI alignment. This fusion uses an attention mechanism to highlight the key information required for keypoint prediction in the feature space. The fused features are then input into the extended 3D keypoint prediction head.

[0032] The 3D keypoint detection head adds a depth prediction branch to the standard keypoint detection head, integrating spatial information from multi-layer convolutional feature maps. This updated 3D keypoint detection head generates a keypoint heatmap and outputs the predicted depth value for each keypoint. Using the 2D coordinates and depth information along with the camera intrinsics, the keypoint positions are back-projected into the camera coordinate system, yielding the 3D coordinates of the five keypoints.

[0033] Specifically, the three-dimensional key point coordinates (X k ,Y k ,Z k ) is calculated as follows:

[0034]

[0035] Among them, (u k ,v k ) is the pixel coordinate of the predicted k-th key point in the image coordinate system, is the depth value of the kth key point obtained by the prediction branch, f x ,f y is the horizontal and vertical focal length of the color camera. (c x ,c y ) are the principal point coordinates, i.e., the two-dimensional coordinates formed by projecting the optical center onto the image plane along the optical axis.

[0036] The step (5) comprises:

[0037] The fruit posture is defined as the three-dimensional vector v pointing from the navel point P2 to the stalk point P1.

[0038] The predicted pose vector v is calculated using the key points predicted in step (4) p , and perform cosine similarity calculation and optimization constraints on the pose. Update the loss calculation to a linear combination of key point heat map loss, key point depth prediction loss, and pose estimation cosine loss, where the key point heat map loss uses a cross entropy-based loss to predict the heat map. Compared with the real heat map For point-by-point comparison, the formula is as follows:

[0039]

[0040] where CE(·) denotes the cross entropy loss and i,k denotes the kth keypoint in the i-th instance.

[0041] Keypoint depth prediction loss measures the predicted depth value With the real depth value The difference between them is measured using L1 loss (absolute value error), and the calculation formula is as follows:

[0042]

[0043] The pose estimation loss is calculated by With real posture The cosine loss of the direction is measured. First, the cosine similarity is calculated. The formula is as follows:

[0044]

[0045] Converting similarity into loss, the cosine loss can be defined as:

[0046]

[0047] The total loss function can be obtained by linearly combining the above loss terms:

[0048] L total =L H +αL d+βL cos

[0049] Among them, α and β are weight hyperparameters used to balance the contribution of each loss term in the overall optimization process.

[0050] By optimizing the total loss and performing backpropagation and parameter updates on the model, the accuracy of keypoint heatmap predictions, the accuracy of depth estimation, and the consistency of fruit pose and orientation distribution can be simultaneously improved during training, thereby enhancing the model's robustness and performance in multiple dimensions. This simultaneous optimization of keypoint accuracy and pose consistency further improves the model's robustness and accuracy.

[0051] The beneficial effects of the present invention are:

[0052] (1) The dragon fruit RGB-D image data acquisition system designed by the present invention can efficiently acquire RGB-D image data of dragon fruit in an unstructured orchard environment, providing accurate image data for subsequent research on automated dragon fruit picking; it can also be applied to the acquisition of RGB-D image data of various types of fruits and vegetables. It has universal applicability;

[0053] (2) The present invention proposes to use five key points, including the navel, stalk, centroid, and shearing points in two directions, to describe the comprehensive information of pitaya during grasping and shearing. The growth posture of pitaya is simulated by fitting the pitaya with an ellipsoid and fitting the front end of the sheared branch with a quadratic surface. The precise three-dimensional spatial coordinate data of the five key points of pitaya are further obtained, providing accurate data for subsequent model training.

[0054] (3) The three-dimensional key point detection head proposed in the present invention adds a depth detection branch on the basis of the two-dimensional key point heat map detection, and performs attention fusion on the generated mask and the feature map after the candidate region is aligned, so as to highlight the key information required for key point prediction in the feature space; in addition, the total loss is updated to a linear combination of the key point heat map loss, the key point depth prediction loss and the cosine loss of the pose estimation, which can improve the robustness of the model in multiple dimensions and achieve the simultaneous optimization of the key point accuracy and pose consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of a process for estimating the posture of a pitaya and locating the cutting point of a fruit stem based on a three-dimensional key point detection head in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the results of fitting a pitaya fruit with an ellipsoid and fitting a branch with a quadratic surface in an embodiment of the present invention;

[0057] Figure 3Schematic diagram of the network structure of the pitaya pose estimation and fruit stem shearing point positioning model based on three-dimensional key point detection in an embodiment of the present invention; DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] like Figure 1 FIG. 1 is a flow chart of dragon fruit pose estimation and fruit stem shear point positioning based on a three-dimensional key point detection head in an embodiment of the present invention, and the steps include S1 to S5:

[0060] Step S1: construct a dragon fruit RGB-D image data acquisition system to collect dragon fruit RGB-D image data in an unstructured dragon fruit orchard environment.

[0061] In this example of the present invention, a dragon fruit RGB-D image data acquisition system is constructed. The user interface includes displaying the color image RGB_IMAGE, displaying the pseudo-color mapping image DEPTH_IMAGE of the depth image, displaying the YOLOv8 target recognition result image YOLOv8_dragon in real time, displaying the real-time acquisition image count IMAGE_COUNTING, saving the image control SAVE and exiting the system control EXIT.

[0062] When acquiring RGB-D images of pitaya, the Intel RealSense D435i camera was used. By calling methods such as rs.pipeline(), rs.config(), and enable_stream(), depth image and color image capture were started and displayed under the system's "RGB_IMAGE" and "DEPTH_IMAGE". The resolution of both the depth image and color image was set to 640×480 to facilitate real-time observation of whether the camera was being called normally.

[0063] At the same time, the loaded YOLOv8 model is used to perform target detection on the current color image, which is displayed in real time in "YOLOv8_dragon". When the detection frame is displayed in the middle of the image, the collector clicks "SAVE" and executes the save_frame() method to save the color image, depth image, and the result image output and labeled by YOLOv8 reasoning at the current moment. At the same time, the image counting component "IMAGE_COUNTING" displays a number plus one, and writes the position information of the dragon fruit target detection frame (including the two-dimensional bounding box coordinates and size), the depth information of the corresponding pixel position, and the spatial coordinates obtained by back-projection of the camera intrinsic reference into a CSV file for record preservation.

[0064] When the number of collected images reaches the collection requirement, the collector clicks "EXIT" and executes the exit_app() method to stop the depth camera data pipeline, close the window, release related resources, and ensure that the system terminates in an orderly manner.

[0065] In step S2, a point cloud set is generated using the pitaya RGB-D data. In the CloudCompare point cloud tool, an ellipsoid is fitted to the pitaya fruit and a small-scale quadratic surface is fitted to the front end of the pitaya branch. The three-dimensional spatial coordinates of the center of mass, stalk point, navel point, and shear points at both ends of the branch that represent the fruit posture are calculated.

[0066] In this example of the present invention, the 3D coordinates are calculated using the collected RGB-D images and camera parameters to create a point cloud dataset, and an ellipsoid is fitted to the point cloud of the pitaya fruit in CloudCompare, and a quadratic surface is fitted to the front end cloud of the corresponding branch to ensure that the fitted ellipsoid is consistent with the real pitaya size and growth direction, and the fitted quadratic surface is consistent with the growth direction of the branch, so that the center of mass of the ellipsoid is aligned with the center of mass of the pitaya, and the endpoints of the major axis are aligned with the fruit stalk point and the fruit navel point of the pitaya fruit, and the two endpoints of the maximum dimension direction of the quadratic surface oriented bounding box are aligned with the real shear point.

[0067] like Figure 2 As shown in the figure, it is a schematic diagram of the results of ellipsoid fitting of pitaya fruit and quadratic surface fitting of branch tip in an embodiment of the present invention. The fitting results can restore the growth posture of pitaya and branches. By calculating the three-dimensional coordinates of the fitted geometric point cloud data, the spatial three-dimensional coordinates of the fruit centroid, fruit stem point, fruit navel point and shear points in two directions of the branch in the camera coordinate system are obtained, and the real data of the key points are used to train the subsequent three-dimensional key point detection model.

[0068] The three-dimensional spatial coordinates of the centroid of the point cloud and the two points farthest from the centroid are calculated for the fitted ellipsoid point cloud, and the three-dimensional spatial coordinates are back-projected onto the color image. At the same time, the two-dimensional and three-dimensional angles ensure that the fitted ellipsoid conforms to the growth posture of the dragon fruit, and the navel point obtained by the fitting calculation corresponds to the visible part of the color image.

[0069] The spatial three-dimensional coordinates of the two endpoints of the maximum dimension direction of the bounding box of the quadratic surface of the fitted branch tip are obtained, and the coordinates are back-projected onto the color image. At the same time, the two-dimensional and three-dimensional angles ensure that the fitted quadratic surface conforms to the growth posture of the dragon fruit branch. The positions of the two shearing points obtained by fitting calculation correspond to the entry points during manual shearing in the color image.

[0070] In step S3, the color image is segmented and annotated using the Via annotation tool to obtain a true segmentation dataset of fruits and branches. The fitted ellipsoid and quadratic surface are reprojected onto the color image to obtain a complete segmentation dataset of fruits and branches.

[0071] The pitaya color image and its corresponding branches are manually segmented using the Via annotation tool to construct an original instance segmentation dataset. The ellipsoid and cylinder fitted in step S2 are then back-projected onto the color image to obtain a complete pitaya mask of the fruit and branches. The mask is then processed to obtain a segmentation dataset after fitting the fruit and branches.

[0072] Applying the original segmentation annotation data and the segmentation annotation data after fitting projection to the instance segmentation network at the same time can train the model to segment and predict the posture of occluded fruits, thereby improving the robustness of the model.

[0073] In step S4, a 3D keypoint detection model based on Mask R-CNN is constructed, a 3D keypoint detection head is designed, a depth prediction branch is added to the traditional keypoint detection head, and the total loss is designed as a linear combination of the keypoint heatmap loss, the keypoint depth prediction loss, and the cosine loss of the pose estimation vector.

[0074] like Figure 3As shown, it is a schematic diagram of the network structure for pitaya pose estimation and fruit stem cutting point positioning based on three-dimensional key point detection. In this example of the present invention, the RGB image containing pitaya and branches is input into the ResNet backbone network for feature extraction. ResNet extracts multi-layer feature maps through layer-by-layer convolution and residual structure. Then FPN fuses these feature maps from top to bottom to obtain a set of feature maps with multi-scale characteristics and a balance of high resolution and high semantic information. RPN uses the feature map of FPN to generate a series of candidate boxes (ROI) at different scales. These candidate boxes roughly determine the area of ​​the pitaya or branch in the image. The region of interest alignment ROIAlign accurately aligns the candidate boxes given by RPN on the FPN feature map and extracts the feature blocks of the corresponding area.

[0075] The features processed by ROIAlign are fed into the classification and bounding box regression head to accurately determine the object's category and bounding box position. Simultaneously, the same ROI features are fed into the mask head to perform pixel-level segmentation on the object, generating a precise binary mask of the dragon fruit and its branches.

[0076] The generated target mask is fused with the ROI feature map, and an attention mechanism is introduced to enable the network to highlight local and detailed features related to keypoint prediction. At this stage, the mask information guides the feature map to focus on specific parts of the object, thereby facilitating subsequent keypoint localization and depth estimation.

[0077] The fused features are fed into the extended 3D keypoint detection head. This head adds a depth prediction branch to the standard keypoint detection head. In addition to outputting traditional 2D keypoint heatmaps, the network also outputs corresponding depth information.

[0078] The obtained 2D key point coordinates and depth information are used, and the coordinate back-projection is performed in combination with the intrinsic parameters of the camera to map the 2D key points back to the 3D world coordinate system, and finally the 3D coordinates of the 5 key points in the camera coordinate system are obtained.

[0079] Step S5, utilizes the segmentation annotation data set and the three-dimensional key point data set to train the three-dimensional key point detection model, and determines the posture estimation vector when the dragon fruit is grasped and the spatial three-dimensional coordinates of the fruit stem shearing point when shearing.

[0080] In the example of the present invention, the posture vector is obtained by calculating the predicted key points, and the cosine similarity is calculated for the vector. The loss calculation is updated to a linear combination of the key point heat map loss, the key point depth prediction loss and the cosine loss of the pose estimation. The total loss is used to backpropagate the model and update the parameters, thereby improving the accuracy of the key point heat map prediction, the accuracy of the depth estimation and the consistency of the fruit posture direction distribution, achieving simultaneous optimization of the key point accuracy and posture consistency, and further improving the robustness and accuracy of the model.

[0081] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for estimating the posture of pitaya and locating the fruit stem shearing point based on a three-dimensional key point detection model, characterized in that: include: Step (1): Build a dragon fruit RGB-D image data acquisition system to collect the dragon fruit RGB-D image dataset in an unstructured dragon fruit orchard environment; Step (2): Use the dragon fruit RGB-D image to construct a three-dimensional point cloud image, import the point cloud set into the CloudCompare point cloud annotation tool, perform ellipsoid fitting on the dragon fruit point cloud, perform quadratic surface fitting on the front end point cloud of the fruit corresponding to the branch, calculate and obtain the true three-dimensional coordinates of the center of mass point, fruit stem point and fruit navel point of the dragon fruit fruit and the shear points at both ends of the corresponding branch in the camera coordinate system, and construct a three-dimensional key point dataset of the dragon fruit; Step (3): Use the dragon fruit color image to perform instance segmentation and annotation on the dragon fruit fruit and its corresponding branches, and construct a dragon fruit instance segmentation dataset; Step (4): Construct a 3D key point detection model based on Mask R-CNN. After the mask segmentation task, the model performs attention fusion on the generated mask and the feature map aligned with the candidate region and inputs the result into a 3D key point detection head. The 3D key point detection head adds a depth prediction branch to the standard key point detection feature extraction path. While generating the key point heat map, the depth value of the key point is predicted. Finally, the spatial coordinates of the 3D key point in the camera coordinate system are calculated by back-projecting the predicted 2D coordinate value and depth value using the camera intrinsic reference, thereby realizing the prediction of the spatial 3D coordinates of the dragon fruit posture point and the shearing point of its corresponding branch. Step (5): The three-dimensional key point detection model is trained using a pitaya instance segmentation dataset and a pitaya three-dimensional key point dataset, and the fruit pose vector is defined as a three-dimensional vector from the fruit navel point to the fruit stem point. The predicted key points are calculated to obtain a predicted pose vector, and the cosine similarity between the two is calculated. The corresponding cosine loss is calculated from the cosine similarity, and the total loss is defined as a linear combination of the depth value L1 loss, the cosine loss of the pose vector, and the key point heat map loss. The total loss is used to perform back propagation and parameter update on the model, and finally the pose vector of the fruit is determined. The shear point coordinates are the three-dimensional coordinates of the two end points of the branch in the predicted camera coordinate system.

2. a kind of pitaya posture estimation and fruit stem shearing point positioning method based on three-dimensional key point detection model according to claim 1, is characterized in that, The step (1) constructs a real-time target detection and data acquisition system based on a graphical user interface (GUI) and a depth perception device. The dragon fruit RGB-D image collected by the system is taken by an Intel RealSense D435i camera. During the acquisition process, the camera is controlled parallel to the fruit tree, and the pixel resolution of the color image and the depth image is fixed at 640×480.

3. a kind of pitaya posture estimation and fruit stem shearing point positioning method based on three-dimensional key point detection model according to claim 1, is characterized in that, The dragon fruit RGB-D image data acquisition system includes the following steps: initializing and configuring a device for a multimodal image stream to capture color and depth images at the same time; performing target detection on the color image using a YOLOv8 model and displaying the target detection result image in real time; providing an interactive user interface that supports triggering the saving of images and metadata through buttons or shortcut keys; automatically creating a storage directory and saving the color image, depth image, and target detection result image; analyzing the detection results to write target position information and corresponding depth values ​​into a data record file; and providing an exit operation after completing data capture to stop the device data pipeline and release resources.

4. a kind of pitaya posture estimation and fruit stem shearing point positioning method based on three-dimensional key point detection model according to claim 1, is characterized in that, The step (2) uses the collected dragon fruit color image and depth image data to generate a corresponding point cloud set; the point cloud set is imported into the CloudCompare point cloud annotation tool, and an ellipsoid is fitted on the dragon fruit point cloud, and the three-dimensional coordinates of the ellipsoid center are obtained at the same time.

5. a kind of pitaya posture estimation and fruit stem shearing point positioning method based on three-dimensional key point detection model according to claim 1, is characterized in that, The three-dimensional coordinates of the center of mass of the pitaya fruit in step (2) are determined by the center point of the fitted point cloud ellipsoid, the three-dimensional coordinates of the fruit stem point and the fruit navel point are determined by the coordinates of the two point clouds farthest from the center point of the ellipsoid on the ellipsoid point cloud, and the two endpoints of the maximum dimension direction of the quadratic surface oriented bounding box fitted to the branch point cloud are taken as the three-dimensional coordinates of the shearing points in two directions of the branches.

6. A method for locating pitaya posture and fruit stem shearing points based on a three-dimensional key point detection model according to claim 1, characterized in that: In the step (3), the pitaya color image is segmented and annotated on the pitaya fruit and its corresponding branches using the Via annotation tool to construct a pitaya instance segmentation dataset.

Images