A solanaceous fruit harvesting robot and its fruit cluster observation pose estimation method and device

Abstract

A solanaceous fruit-harvesting robot and its fruit cluster observation pose estimation method and apparatus are disclosed. The solanaceous fruit-harvesting robot includes a fruit cluster observation pose estimation device, which implements a fruit cluster observation pose estimation method. The method includes the following steps: acquiring RGB image data and depth data of solanaceous plants; normalizing the RGB image data and depth data respectively, and inputting them into a trained bimodal fusion semantic segmentation model to predict the masks corresponding to the two categories of fruit and stem in the RGB image; extracting the depth information corresponding to the connected regions of the mask and performing local mask refinement using density clustering; clustering to extract point clouds of single fruit clusters, calculating the center information of the point cloud of each fruit cluster, and extracting the point clouds of the stems adjacent to the fruit clusters for straight line fitting; and determining the observation pose of the end-effector camera based on the relative position of the center of the fruit cluster point cloud and the perpendicular point of the corresponding stem fitted line. This invention effectively improves the recognition and positioning accuracy of the fruit stem harvesting point by the end-effector camera of the harvesting robot.

Description

Technical Field

[0001] This invention relates to agricultural harvesting machinery, and in particular to a solanaceous fruit harvesting robot based on dual-modal fusion, and its method and device for estimating the pose of fruit cluster observation. Background Technology

[0002] Solanaceous crops, rich in nutrients and diverse in varieties, are widely popular and cultivated globally. However, harvesting them is a labor-intensive task, and rising labor costs are exacerbated by the aging agricultural workforce. Solanaceous harvesting robots can autonomously perform harvesting tasks, effectively reducing human intervention, lowering labor costs, and improving production efficiency. Stem-cutting harvesting avoids damage to the fruit and stems caused by pulling, making it a superior harvesting method. Stem-cutting harvesting requires the robot to accurately identify and locate the harvesting point, such as the stem of a tomato, placing high demands on the accuracy of the identification algorithm and depth sensor. The depth camera used must be able to capture images of the tomato stem and reliably obtain depth values ​​of the target object. Detecting stems and other targets helps the harvesting robot perceive the surrounding environment, allowing for appropriate obstacle avoidance and grasping. Semantic segmentation methods can effectively detect pixel regions corresponding to tomato stems and fruits in images and are often used for tasks such as detecting fruits, stems, and stalks. However, most existing technologies only use RGB images as input for semantic segmentation prediction, lacking effective utilization of depth information. During the harvesting of solanaceous fruits in greenhouse environments, the stems and fruits easily obstruct the harvesting point at the stem, preventing the robot from recognizing and harvesting. Optimal camera observation pose can avoid occlusion during stem recognition and localization, while maintaining high measurement accuracy for the depth camera. However, most current harvesting robots observe fruits from random angles, lacking estimation of optimal observation poses, resulting in inaccurate harvesting point localization. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies of the prior art by providing a solanaceous fruit harvesting robot and a method and apparatus for estimating the pose of fruit clusters.

[0004] To achieve the above objectives, the present invention provides a method for estimating the pose of fruit cluster observations, comprising the following steps:

[0005] S100: Collect RGB image data and depth data of solanaceous plants;

[0006] S200: After normalizing the RGB image data and depth data respectively, input them into the trained dual-modal fusion semantic segmentation model to predict the masks corresponding to the two categories of fruit and stem in the RGB image.

[0007] S300. Extract the depth information corresponding to the connected regions of the mask, and refine the local mask using density clustering method.

[0008] S400: Clustering to extract point clouds of individual fruit clusters, calculating the center information of the point cloud of each cluster, and extracting the point clouds of nearby stems for linear fitting; and

[0009] S500: Determine the observation pose of the end camera based on the relative position of the fruit cluster point cloud center and the vertical point of the corresponding stem fitting straight line.

[0010] In the above-mentioned method for estimating the pose of fruit cluster observations, the dual-modal fusion semantic segmentation model is obtained using the following training method:

[0011] S001. Construct a bimodal semantic segmentation model based on a deep convolutional neural network. The network structure of the model includes two inputs: a three-channel RGB data input and a single-channel depth data input.

[0012] S002. A feature fusion and correction module is added to the bimodal semantic segmentation model to perform channel correction and spatial correction on RGB features and depth features respectively; after feature fusion in combination with attention mechanism, features are extracted through backbone network and the mask corresponding to the specified category in RGB image is output.

[0013] S003. A monocular depth estimation algorithm is used to generate depth maps for labeled dataset images that lack corresponding depth data. A dual-modal fusion semantic segmentation model is trained using color images and the generated depth images as a pre-trained model.

[0014] S004. Fine-tune the pre-trained model using the actual collected fruit cluster color image and depth map data to obtain the final trained dual-modal fusion semantic segmentation model.

[0015] The above-mentioned method for estimating the pose of fruit cluster observation requires normalization of the three-channel RGB data and the single-channel depth data before input, and the construction of a dual-modal fruit cluster fruit-stem semantic segmentation dataset.

[0016] In the above-mentioned fruit cluster observation pose estimation method, in step S300, connected region detection is performed on the mask of each category predicted by the semantic segmentation model, and the local depth value data corresponding to each connected region is extracted from the corresponding depth data.

[0017] The above-mentioned method for estimating the pose of fruit cluster observations further includes, in step S300:

[0018] S310. Extract connected components from the single-class mask map output by the model. If the area of ​​the connected component is less than a set area threshold T... area If so, then ignore the connected component;

[0019] S320. Extract the depth value from the depth map corresponding to each remaining connected component, and cluster the depth values; exclude clusters with an area smaller than a set threshold from the mask to retain regions with significant depth features; and

[0020] S330. Extract all regions with significant depth features to form updated mask regions, and integrate the mask regions of individual categories into a single depth map.

[0021] The above-mentioned method for estimating the pose of fruit cluster observations, wherein step S400, which involves acquiring data using an RGB-D camera, further includes:

[0022] S410. Cluster the depth value data of each extracted mask area, calculate the pixel area corresponding to each class in the clustering result, split the connected but different clusters, and remove noise points and depth values ​​with areas smaller than the set value to filter noise and redundant information.

[0023] S420. Based on the camera intrinsic parameters of the RGB-D camera, convert the extracted depth values ​​of each local region into a local point cloud. For the local point cloud categorized as fruit, calculate the center coordinates and the radius of the spatial envelope sphere as the position information of each cluster of fruit; and

[0024] S430. Extract the point cloud of the stem category within a specified radius around the center of each fruit cluster, extract the point cloud of the stem near the fruit segment through cluster analysis, and perform linear fitting.

[0025] The above-mentioned method for estimating the pose of fruit cluster observations further includes, in step S500:

[0026] S510. Find the perpendicular point from the center of the fruit cluster to the fitted straight line segment of the stem, and initially generate the initial pose from the center of the fruit cluster toward the perpendicular point.

[0027] S520. Based on the initial pose, rotate and then translate to obtain the final end-camera observation pose.

[0028] The above-mentioned method for estimating the pose of fruit cluster observations further includes, in step S510:

[0029] S511. Determine the coordinates of the perpendicular point D on the corresponding straight line segment from the center point C of a single fruit cluster point cloud:

[0030] Where, the foot of the perpendicular D is the intersection of the perpendiculars from the center point C of the fruit cluster to line segment AB, and let the starting point and ending point of the line segment be P and P, respectively. A and P B ,but:

[0031]

[0032] vector Let t represent the vector from the starting point A of the line segment to the center point C of the fruit string point cloud; t is a scalar representing the vector of line segment AB. A proportion used to determine the position of the perpendicular foot;

[0033] S512. Determine the camera's orientation vector from its position P. C Point to target location P D The vector is denoted by , where the z-axis is the direction the camera is facing, the y-axis is perpendicular to the camera base and points upwards from the camera, and the x-axis follows the right-hand rule and is perpendicular to both the z-axis and the y-axis.

[0034] Normalization is performed:

[0035]

[0036] S513, Calculation The angle between the vector's projection onto the XOZ plane and the Z-axis is determined. If this angle is greater than a set angle threshold α, then according to P... D With P A Relative position adjustment on the X-axis

[0037]

[0038] S514, Calculate the left vector via the above direction With direction vector The cross product is obtained and normalized to:

[0039]

[0040] It is the upward direction in the world coordinate system, and the default value is (0,-1,0);

[0041] S515, Recalculate the upper vector via direction vector With the right vector The cross product is obtained and normalized to:

[0042] as well as

[0043] S516. Construct the rotation matrix R rot The initial pose is:

[0044]

[0045] To better achieve the above objectives, the present invention also provides a fruit cluster observation pose estimation device, which is installed on a solanaceous fruit picking robot. The device includes a data acquisition unit and a data processor installed at the end of the robotic arm of the picking robot. The data acquisition unit is used to acquire RGB image data and depth data within the field of view and transmit them to the data processor. The data processor is used to implement the above-mentioned fruit cluster observation pose estimation method to improve the identification and positioning accuracy of the fruit stem picking point by the solanaceous fruit picking robot.

[0046] To better achieve the above objectives, the present invention also provides a solanaceous fruit harvesting robot, wherein the above-mentioned fruit cluster observation pose estimation method is used to obtain the observation pose of the end camera, so as to improve the accuracy of the end camera in identifying and locating the fruit stem harvesting point.

[0047] The technical advantages of this invention are as follows:

[0048] This invention presents a method for estimating the observation pose of fruit clusters based on dual-modal fusion. This method effectively utilizes data from the end-effector sensors of a harvesting robot to improve the recognition performance of targets such as fruits and stems. By estimating the optimal observation pose of the end-effector camera, the accuracy of the harvesting robot's end-effector camera in identifying and locating fruit stem picking points is improved. It fully integrates RGB and depth data to detect tomato fruits and stems grown in a greenhouse environment, enhancing the harvesting robot's effective use of multimodal sensor data and improving the visual algorithm's ability to identify important information about tomato plants. A monocular depth estimation model is used to generate a depth map for pre-training, compensating for the lack of corresponding depth values ​​in the semantic segmentation dataset. Based on the spatial relationship between tomato fruits and surrounding stems, an optimal cluster observation pose estimation method is determined for the end-effector camera, enabling the detection of occlusion of the stem and fruit by the stem itself at the picking point, thus improving the picking point detection rate.

[0049] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the present invention. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the working principle of an embodiment of the present invention. Detailed Implementation

[0051] The structural and working principles of the present invention will be described in detail below with reference to the accompanying drawings:

[0052] This invention relates to a solanaceous fruit-harvesting robot that employs a fruit cluster observation pose estimation method to obtain the observation pose of the end-effector camera, thereby improving the accuracy of the end-effector camera in identifying and locating the fruit stem picking point. The solanaceous fruit-harvesting robot includes a fruit cluster observation pose estimation device, which is mounted on the robot and comprises a data acquisition unit and a data processor installed at the end of the robot's robotic arm. The data acquisition unit acquires RGB image data and depth data within the observation field of view and transmits them to the data processor. An RGB-D camera can be used, preferably an Intel Realsense L515 depth camera. The data processor implements the fruit cluster observation pose estimation method to improve the accuracy of the solanaceous fruit-harvesting robot in identifying and locating the fruit stem picking point. Since the composition, structure, relative positions and connections between the components, and functions of the data acquisition unit and data processor of the solanaceous fruit-harvesting robot and the fruit cluster observation pose estimation device are all mature existing technologies, they will not be elaborated upon here. The following only describes the fruit cluster observation pose estimation method of this invention in detail.

[0053] See Figure 1 , Figure 1 This is a schematic diagram illustrating the working principle of an embodiment of the present invention. The present invention employs monocular depth estimation technology to generate corresponding depth maps for color image samples lacking depth information, thereby enriching the bimodal training dataset; pre-training is performed using this data to enhance the model's ability to recognize tomato fruit clusters and stems; local masks are refined by combining connected component extraction and density clustering methods; fruit cluster location information is extracted based on density clustering, and straight line fitting is performed on the near-fruit stem; optimal observation pose estimation is achieved based on the spatial relationship between the fruit cluster center and the near-fruit stem. The fruit cluster observation pose estimation method of the present invention includes the following steps:

[0054] Step S100: Collect RGB image data and depth data of solanaceous plants;

[0055] Step S200: After normalizing the RGB image data and depth data respectively, input them into the trained dual-modal fusion semantic segmentation model to predict the masks corresponding to the two categories of fruit and stem in the RGB image;

[0056] Step S300: Extract the depth information corresponding to the connected regions of the mask, and refine the local mask using density clustering method;

[0057] Step S400: Cluster and extract point clouds of individual fruit clusters, calculate the center information of the point cloud of each cluster, and extract the point clouds of the stems adjacent to the fruit clusters for linear fitting; and

[0058] Step S500: Determine the observation pose of the end camera based on the relative position of the fruit cluster point cloud center and the vertical point of the corresponding stem fitting straight line.

[0059] The dual-modal fusion semantic segmentation model is obtained using the following training method:

[0060] Step S001: Construct a bimodal semantic segmentation model based on a deep convolutional neural network. The network structure of the model includes two inputs: three-channel RGB data input and single-channel depth data input. Before input, the three-channel RGB data and the single-channel depth data need to be normalized respectively, and a bimodal fruit cluster fruit-stem semantic segmentation dataset is constructed.

[0061] Step S002: Add a feature fusion and correction module to the bimodal semantic segmentation model to perform channel correction and spatial correction on RGB features and depth features respectively; after feature fusion in combination with attention mechanism, extract features through backbone network and output mask corresponding to specified category in RGB image;

[0062] Step S003: Using a monocular depth estimation algorithm, depth maps are generated for labeled dataset images lacking corresponding depth data. A dual-modal fusion semantic segmentation model is trained using color images and the generated depth images as a pre-trained model.

[0063] Step S004: Use the actual collected fruit cluster color image and depth map data to fine-tune the pre-trained model to obtain the final trained dual-modal fusion semantic segmentation model.

[0064] In step S300, connected component detection is performed on the mask for each category predicted by the semantic segmentation model, and local depth value data corresponding to each connected component is extracted from the corresponding depth data. This further includes:

[0065] Step S310: Extract connected components from the single-class mask image output by the model. If the area of ​​the connected component is less than a set area threshold T... area If so, then ignore the connected component;

[0066] Step S320: Extract the depth value from the depth map corresponding to each remaining connected component, and cluster the depth values; exclude clusters with an area smaller than a set threshold from the mask to retain regions with significant depth features; and

[0067] Step S330: Extract all regions with significant depth features to form updated mask regions, and integrate the mask regions of individual categories into a single depth map.

[0068] In step S400, data is acquired using an RGB-D camera, which further includes:

[0069] Step S410: Cluster the depth value data of each extracted mask area, calculate the pixel area corresponding to each class in the clustering result, split the connected but different clusters, and remove noise points and depth values ​​with areas smaller than the set value to filter noise and redundant information.

[0070] Step S420: Based on the camera intrinsic parameters of the RGB-D camera, convert the extracted depth values ​​of each local region into a local point cloud. For the local point cloud categorized as fruit, calculate the center coordinates and the radius of the spatial envelope sphere as the position information of each cluster of fruit; and

[0071] Step S430: Extract the point cloud of the stem category within a specified radius around the center of each fruit cluster, extract the stem point cloud near the fruit segment through cluster analysis, and perform linear fitting.

[0072] Step S500 further includes:

[0073] Step S510: Find the perpendicular point from the center of the fruit cluster to the fitted straight line segment of the stem, and initially generate the initial pose from the center of the fruit cluster toward the perpendicular point.

[0074] Step S520: Rotate and translate based on the initial pose to obtain the final end-camera observation pose.

[0075] Step S510 further includes:

[0076] Step S511: Determine the coordinates of the perpendicular point D on the corresponding straight line segment from the center point C of a single fruit cluster point cloud:

[0077] Where the foot of the perpendicular D is the intersection of the perpendiculars from the center point C of the fruit string point cloud to the line segment AB. The foot of the perpendicular is calculated by projecting Pc as Pcenter. Let the start and end points of the line segment be Pc and Pc, respectively. A and P B ,but:

[0078]

[0079] vector Let t represent the vector from the starting point A of the line segment to the center point C of the fruit string point cloud; t is a scalar representing the vector of line segment AB. A proportion used to determine the position of the perpendicular foot;

[0080] Step S512: Determine the camera's orientation vector from camera position P. C Point to target location P D The vector is denoted by , where the z-axis is the direction the camera is facing, the y-axis is perpendicular to the camera base and points upwards from the camera, and the x-axis follows the right-hand rule and is perpendicular to both the z-axis and the y-axis.

[0081] Normalization is performed:

[0082]

[0083] Step S513, Calculation The angle between the vector's projection onto the XOZ plane and the Z-axis is determined. If this angle is greater than a set angle threshold α, then according to P... D With P A Relative position adjustment on the X-axis

[0084]

[0085] Step S514: Calculate the left vector via the above direction With direction vector The cross product is obtained and normalized to:

[0086]

[0087] It is the upward direction in the world coordinate system, and the default value is (0,-1,0);

[0088] Step S515: Recalculate the upper vector via direction vector With the right vector The cross product is obtained and normalized to:

[0089] as well as

[0090] Step S516: Construct the rotation matrix R rot The initial pose is:

[0091]

[0092] This invention identifies solanaceous fruit clusters and stems based on a dual-modal fusion semantic segmentation method. It utilizes a monocular depth estimation method to generate depth maps, compensating for the lack of depth data in conventional semantic segmentation datasets. Furthermore, it integrates color image information with depth information to improve the detection accuracy of solanaceous fruits and stems. Spatial point cloud technology is used to extract and deeply analyze solanaceous stems and fruit clusters, thereby estimating the optimal observation pose. This helps optimize the observation angle of solanaceous harvesting robots, improving the quality and efficiency of data acquisition.

[0093] The following uses tomatoes as an example to illustrate the specific application of the fruit cluster observation pose estimation method of the present invention.

[0094] First, we build and train a bimodal fusion semantic segmentation model.

[0095] The visual sensors mounted on the end effector of solanaceous fruit harvesting robots can generally acquire RGB image data and depth data within the field of view. However, most current visual detection models for harvesting robots primarily use RGB image data as input for inference and prediction, lacking effective utilization of information from other sensors. Furthermore, depth sensors typically exhibit measurement noise, which can easily increase model prediction errors. Correcting and fusing data from different modalities is an effective way to improve the accuracy of visual detection in harvesting robots. This invention presents a semantic segmentation model capable of feature correction and fusion of RGB and depth modal data. The model's network structure includes two inputs: a three-channel RGB image and single-channel depth data. Both inputs are initialized and then fed into the feature correction and fusion module. The feature correction and fusion module comprises three parts: channel correction, spatial correction, and fusion. After correction and fusion processing, the fused features are input into the semantic segmentation encoder structure for feature encoding, and finally, the decoder outputs a mask corresponding to each category.

[0096] To address the issues of insufficient bimodal image data and annotations, the Depth Anything monocular depth estimation algorithm is first employed to generate depth maps from labeled tomato fruit-stem semantic segmentation dataset images lacking corresponding depth information. A bimodal fusion semantic segmentation model is then trained by combining color images and the generated depth images, and this model serves as the pre-training foundation. Subsequently, this pre-trained model is applied to a real-world dataset for further training and optimization.

[0097] During the data acquisition phase, an Intel RealSense L515 depth camera was used to collect RGB image data and depth data of tomato plants in the tomato orchard. By utilizing the alignment function provided by the camera's SDK, accurate alignment between the acquired RGB images and depth maps was ensured. The acquired RGB images were saved in JPEG format, while the depth data was stored in NPY format for subsequent processing and analysis.

[0098] Furthermore, Labelme software was used to perform fine contour annotation on target categories in the images, such as tomato fruits and stems. After annotation, the sample data was divided into training, validation, and test sets in an 8:1:1 ratio to ensure the comprehensiveness of model training and the accuracy of evaluation. During the model training phase, the network was built with pre-trained weights and trained using aligned and normalized data, ultimately yielding a semantic segmentation model for tomato fruits and stems.

[0099] Secondly, local mask refinement is performed.

[0100] Because segmentation models may have biases in edge prediction, and depth cameras may introduce errors during measurement, these issues can lead to inaccurate mapping of the mask obtained from the segmentation model onto the depth map, incorrectly incorporating background depth information. To address these problems, this embodiment first uses OpenCV's `connectedComponentsWithStats` function to extract connected components from the single-class mask map output by the model. An area threshold T is then set... area If set to 500 pixels, if the area of ​​any connected component is less than the threshold T area If so, then ignore the component.

[0101] For each remaining connected component, the depth value in the corresponding depth map is extracted, and the DBSCAN method is used to cluster the depth values ​​within each connected component. The coordinates (u, v) of each pixel are combined with its corresponding depth value to form a feature vector, which is used as the input to the DBSCAN algorithm. In this embodiment, the parameters of the DBSCAN algorithm are set to eps = 0.5 and min_samples = 30. Based on the clustering results of the DBSCAN algorithm, if the area of ​​a cluster is less than a set threshold, the depth values ​​of all pixels within that cluster are set to 0, i.e., these regions are excluded from the mask. This step helps to further refine the mask, ensuring that only regions with significant depth features are preserved.

[0102] Finally, all regions with non-zero depth values ​​are extracted to form the updated mask region. This process yields a more accurate mask, and then these individual mask regions are integrated into a single depth map. Based on camera intrinsics, the spatial point cloud corresponding to each mask category is calculated using the following formula:

[0103] Z = M(u, v) · d · s;

[0104]

[0105] Where X, Y, and Z are the three-dimensional spatial coordinates; d is the depth image value at pixel position (u, v); s is the depth scaling factor; u and v are pixel coordinates, where u is the column coordinate and v is the row coordinate; c x c y f represents the coordinates of the principal point in the camera's intrinsic parameters. x , f y is the camera focal length; M(u, v) is the mask function, used to select only pixels within the mask area for calculation.

[0106] Then, fruit cluster extraction is performed.

[0107] To merge spatially close but unconnected tomato fruit point clouds into complete tomato clusters, this embodiment uses the DBSCAN algorithm to cluster the spatial point clouds of tomato categories. The DBSCAN algorithm can identify and aggregate neighboring point clouds into clusters based on spatial density, thus achieving a complete representation of the tomato cluster. After clustering, the coordinates of the center point of each tomato cluster need to be determined. This can be achieved by calculating the centroid (i.e., the average coordinates) of all points within each cluster. The calculation formula is as follows:

[0108]

[0109] Among them, P C P represents the center coordinates of the point cloud; N represents the number of points in the point cloud; P represents the center coordinates of the point cloud. i Let be the three-dimensional coordinates of the i-th point.

[0110] The bounding box of the fruit cluster point cloud is represented as follows:

[0111]

[0112] The radius of the fruit cluster envelope sphere is:

[0113]

[0114] Next, a straight line fitting was performed on the stem near the fruit section.

[0115] For each extracted cluster of tomato points, calculate the distance from the stem point cloud to the center of the tomato cluster. Extract a distance from the center of the bounding box plus an extra radius from the stem point cloud, and set a distance threshold.

[0116] T dis =r+a;

[0117] Calculate the Euclidean distance from each point in the point cloud to the center point:

[0118] distances = ||PP C ||2;

[0119] Use the calculated distances to filter out points that meet the criteria:

[0120] indeices={i|distance[i]≤t dis};

[0121] Use the selected set of indices to extract corresponding points from the original point cloud, forming a subset:

[0122] {P n} = P[indices];

[0123] PCA (Principal Component Analysis) was used to analyze the selected point cloud P. nPerform a line fitting operation to obtain a straight line. This is not limited to using the PCA method for line fitting.

[0124] Data decentralization: computation {P} n The mean μ of} will decentralize the data X={P n}-μ.

[0125] Calculate the covariance matrix:

[0126]

[0127] Determine the eigenvalues ​​λ and eigenvectors V, and select the eigenvector V with the largest eigenvalue. max As the main direction of point cloud.

[0128] Take the mean point obtained from PCA as the reference point on the straight line.

[0129] Parameterization of linear equations: Where t is a parameter, which can take any real value.

[0130] Finally, the observed pose is estimated.

[0131] First, determine the coordinates of the perpendicular point from the center of a single fruit cluster's point cloud to its corresponding straight line segment.

[0132] Let the starting point and ending point of the straight line segment be P and P', respectively. A and P B Line segment direction vector:

[0133]

[0134] vector Let A be the vector from the starting point A of the line segment to the center point C of the fruit string point cloud. The calculation formula is:

[0135]

[0136] Calculate the value of t at the perpendicular point D: t is a scalar representing a vector. A proportion is used to determine the position of the foot of the perpendicular, D. The foot of the perpendicular, D, is the intersection of the perpendiculars from point C to line segment AB. The formula for calculating t is as follows:

[0137]

[0138] The coordinates of the foot of the perpendicular are:

[0139]

[0140] After determining the coordinates of the perpendicular point, the observation attitude is determined. In determining the observation attitude, the viewing angle of the camera mounted on the end effector is defined as a local tool coordinate system. In this coordinate system, the z-axis is defined as the camera's facing direction, i.e., a straight line pointing in front of the camera; the y-axis is perpendicular to the camera base and points upwards, representing the top of the camera; while the x-axis follows the right-hand rule, is perpendicular to both the z-axis and y-axis, and points to the left of the camera.

[0141] Determine the camera's orientation vector, which is derived from the camera's position P. C Point to target location P D The vector is then normalized:

[0142]

[0143] Considering factors such as stem interference preventing the harvesting end from reaching the observation pose, the vector was adjusted. To address issues such as obstruction and inaccessibility by the robotic arm. Through calculation... The vector's projection onto the XOZ plane forms an angle with the Z-axis. If this angle is greater than a given angle threshold α, then according to P... D With P A Relative position adjustment on the X-axis

[0144]

[0145] Assumption This is the upward direction in the world coordinate system, defaulting to (0, -1, 0). Next, calculate the left vector. via the above direction With direction vector The cross product is obtained and normalized:

[0146]

[0147] Recalculate the upper vector via direction vector With the right vector The cross product is obtained to ensure that it is consistent with... and Orthogonalization and normalization:

[0148]

[0149] Construct a rotation matrix R rot Expressing attitude:

[0150]

[0151] Finally, attitude and position adjustments are made to ensure the end-point camera has a good observation distance and angle. The camera is rotated 30° counterclockwise around its observation axis (pitch axis). This adjustment gives the camera a tilted, downward-looking perspective, which helps to better capture details of the target area. After attitude adjustment, the camera is moved a distance D in the negative direction of its current orientation (i.e., the opposite direction of the current camera orientation). This step ensures the camera remains within a suitable viewing distance of the target area. Distance D is set based on the camera's field of view and the optimal distance for depth imaging. The final camera observation pose is determined, at a distance D diagonally above the target center point, observing from a downward angle.

[0152] In this embodiment, a bimodal semantic segmentation model based on a deep convolutional neural network is first constructed. This model's network structure includes two inputs: three-channel RGB data and single-channel depth data, both of which are normalized before input. A feature fusion and correction module is added to the model to perform channel and spatial correction on the RGB and depth features respectively. Then, an attention mechanism is used for feature fusion. Features are extracted through a backbone network, and the mask corresponding to the specified category in the RGB image is inferred and output. Next, the bimodal fusion model is pre-trained. A monocular depth estimation algorithm is used to generate depth maps from labeled datasets lacking corresponding depth data. The bimodal fusion semantic segmentation model is trained using color images and the generated depth images as a pre-trained model. Finally, fine-tuning training is performed using color images and depth map data actually acquired by an RGB-D camera to obtain the final semantic segmentation model for tomato stems and pedicels.

[0153] During the process, an RGB-D camera was used as the data acquisition device to collect RGB images and depth data of tomato plants. After normalizing the RGB and depth data, they were input into a trained bimodal semantic segmentation model to predict the masks corresponding to the two categories of tomato fruit and stem in the image. Connected region detection was performed on the masks predicted by the semantic segmentation model for each category, and local depth values ​​corresponding to each connected region were extracted from the corresponding depth data. The extracted depth values ​​of each mask region were clustered, and the pixel area corresponding to each cluster was calculated. Connected but different clusters were separated, and noise points and depth values ​​with excessively small areas were removed to filter out noise and redundant information. Based on the camera intrinsic parameters, the extracted local region depth values ​​were converted into local point clouds. For the local point cloud of the tomato fruit category, its center coordinates and spatial envelope radius were calculated as the position information of each fruit cluster. Point clouds of the stem category within a specified radius around the center of each fruit cluster were extracted, and the stem point cloud near the fruit segment was extracted through cluster analysis and then fitted with a straight line. Finally, the observation pose is estimated: First, the perpendicular point from the center of the fruit cluster to the fitted straight line segment of the stem is calculated, and an initial pose is generated from the center of the fruit cluster toward the perpendicular point. Based on this initial pose, the viewpoint is rotated and then translated to obtain the final observation pose.

[0154] This invention constructs a dual-modal semantic segmentation model using RGB images and depth data. It generates depth maps using monocular depth estimation to supplement the missing depth information in traditional semantic segmentation datasets. Furthermore, by fusing color images and depth data, it improves the detection accuracy of tomato fruits and stems. By accurately extracting local spatial point clouds of tomato stems and fruit clusters and conducting in-depth analysis of their spatial relationships, a better observation pose is estimated. This significantly improves the accuracy of identifying and locating tomato stem picking points from the perspective of the end-effector camera of a solanaceous fruit harvesting robot.

[0155] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A fruit cluster observation pose estimation method, characterized by, The dual-modal fusion semantic segmentation model is used to identify the fruit clusters and stems of solanaceous plants, and the color image information and the depth information generated by the monocular depth estimation algorithm are fused to improve the detection accuracy of solanaceous fruit and stem; the spatial point cloud is used to extract and analyze the solanaceous stem and fruit cluster, and the best observation pose is estimated to optimize the observation angle of the solanaceous picking robot, and the quality and efficiency of data acquisition are improved, including the following steps: S100, collecting RGB image data and depth data of solanaceous plants; S200, after normalizing the RGB image data and depth data, inputting the trained dual-modal fusion semantic segmentation model to predict the mask corresponding to the fruit and stem in the RGB image; S300, extracting the depth information corresponding to the connected region of the mask, and combining the density clustering method to refine the local mask; S400, clustering and extracting single fruit string point cloud, calculating the center information of each cluster of fruit string point cloud, and extracting the stem point cloud near the fruit string for straight line fitting; and S500, determining the end camera observation pose according to the relative position of the fruit string point cloud center and the vertical point of the corresponding stem fitting straight line; The dual-modal fusion semantic segmentation model is obtained by the following training method: S001, constructing a dual-modal semantic segmentation model based on a deep convolutional neural network, the network structure of the model including two inputs, three-channel RGB data input and single-channel depth data input; S002, adding a feature fusion correction module to the dual-modal semantic segmentation model, performing channel correction and spatial correction on the RGB feature and depth feature respectively; after feature fusion combined with the attention mechanism, extracting the feature through the backbone network, and outputting the mask corresponding to the specified class in the RGB image; S003, using a monocular depth estimation algorithm to generate a depth map for the labeled data set image lacking corresponding depth data, and using the color image and the generated depth image to train the dual-modal fusion semantic segmentation model as a pre-training model; and S004, fine-tuning the pre-training model using the actual collected fruit cluster color and depth data to obtain the final trained dual-modal fusion semantic segmentation model; In step S500, further comprising: S510, finding the vertical point of the stem straight line segment from the fruit cluster center, and preliminarily generating the initial pose from the fruit cluster center to the vertical point; and S520, rotating and translating based on the initial pose to obtain the final end camera observation pose; Step S510 further comprises: S511, determine the single fruit string point cloud center point C to the corresponding straight line segment on the foot D point coordinates: ; Wherein, the foot D is the perpendicular line intersection point from the fruit string point cloud center point C to the line segment AB, and the straight line segment starting point and the ending point are respectively and Then: ； ； ； vector Let t represent the vector from the starting point A of the line segment to the center point C of the fruit string point cloud; t is a scalar representing the vector of line segment AB. A proportion used to determine the position of the perpendicular foot; S512, determine the orientation vector of the camera is from the camera position a vector pointing to the target position , the z-axis is the orientation direction of the camera, the y-axis is perpendicular to the camera base and points to the upper side of the camera, and the x-axis follows the right-hand rule, perpendicular to the z-axis and the y-axis; Normalization processing: ； S513、compute The vector is projected on the XOZ plane and the angle with the Z axis is calculated If the angle is greater than a set angle threshold Then the relative position of the X axis is adjusted According to the following equation : ； S514, compute left vector , by the up direction and the cross product of the direction vector , and normalized to: ； is the up direction in the world coordinate system, by default (0,-1,0); S515, Recompute the up vector , by the cross product of the direction vector and the right vector , and normalize to: ; and S516, construct a rotation matrix for the initial pose: 。 2. The cluster of fruit pose estimation method of claim 1, wherein, Before input, the three-channel RGB data and the single-channel depth data are normalized respectively, and a dual-modal fruit cluster fruit-stem semantic segmentation data set is constructed.

3. The cluster of fruit pose estimation method of claim 1, wherein, In step S300, the mask of each class predicted by the semantic segmentation model is detected for connected region, and the local depth value data corresponding to each connected region is extracted from the corresponding depth data.

4. The cluster of fruit pose estimation method of claim 3, wherein, In step S300, further comprising: S310, extract connected components in the single-class mask map output by the model, if the area of a connected component is less than a set area threshold then ignore the connected component; S320, extracting the depth value in the depth map corresponding to each remaining connected component, and clustering the depth value; excluding the cluster with an area less than a set threshold from the mask to reserve the area with significant depth feature; and S330, extracting all the areas with significant depth feature to form the updated mask area, and integrating the mask area of a single class into a depth map.

5. The cluster of fruit pose estimation method of claim 1, wherein, In step S400, the RGB-D camera is used to collect data, which further includes: S410, clustering the depth value data of each extracted mask area, calculating the pixel area corresponding to each class in the clustering result, splitting the areas connected but in different clusters, and eliminating the noise points and the depth value with an area less than a set value to filter the noise and redundant information; S420, converting the extracted depth value of each local area into a local point cloud according to the camera intrinsic parameter of the RGB-D camera, and calculating the center coordinates and the space envelope sphere radius of the local point cloud of the fruit as the position information of each cluster of fruits; and S430, extracting the point cloud of the stem in a specified radius around the center of each cluster of fruits, extracting the near-fruit stem point cloud through clustering analysis, and performing linear fitting.

6. A fruit cluster observation pose estimation device provided on a solanaceous fruit picking robot, comprising a data collector mounted at the end of a mechanical arm of the picking robot and a data processor, characterized in that, The data collector is used to acquire the RGB image data and the depth data in the observation field of view and transmit to the data processor, and the data processor is used to realize the fruit cluster observation pose estimation method of any one of claims 1-5 to improve the recognition and positioning accuracy of the fruit stem picking point of the solanaceous fruit picking robot.

7. A robot for picking eggplants, characterized in that, The fruit cluster observation pose estimation method of any one of claims 1-5 is used to acquire the observation pose of the end camera to improve the recognition and positioning accuracy of the fruit stem picking point of the end camera. The fruit cluster observation pose estimation method of any one of claims 1-5 is used to acquire the observation pose of the end camera to improve the recognition and positioning accuracy of the fruit stem picking point of the end camera.

Images