Solanaceous fruit picking robot and fruit cluster observation pose estimation method and device thereof

Abstract

The invention relates to a solanaceous fruit picking robot and a fruit cluster observation pose estimation method and device.The solanaceous fruit picking robot comprises the fruit cluster observation pose estimation device, the device is used for achieving the fruit cluster observation pose estimation method, and the method comprises the following steps that RGB image data and depth data of solanaceous fruit plants are collected; respectively normalizing the RGB image data and the depth data, inputting the RGB image data and the depth data into the trained bimodal fusion semantic segmentation model, and predicting masks corresponding to two types of fruits and stalks in the RGB image; extracting depth information corresponding to a mask communication region, and carrying out local mask refinement in combination with a density clustering method; single fruit bunch point clouds are extracted in a clustering mode, point cloud center information of each cluster of fruit bunches is calculated, and stalk point clouds close to the fruit bunches are extracted for linear fitting; and determining the observation pose of the end camera according to the relative position of the point cloud center of the fruit bunch and the vertical point of the corresponding stem fitting straight line. The identification and positioning precision of the picking mechanical arm tail end camera on the fruit stem picking point is effectively improved.

Description

Technical Field

[0001] The invention relates to agricultural harvesting machinery, in particular to a solanaceous fruit picking robot based on dual-modal fusion and a method and device for estimating the position and posture of a fruit cluster thereof. Background Art

[0002] Solanaceae crops are rich in nutrition and rich in variety. They are widely popular and widely planted around the world. However, their harvesting is a labor-intensive task. As the aging of the agricultural labor force continues to intensify, labor costs continue to rise. Solanaceae picking robots can replace manual labor to perform autonomous picking operations, effectively reducing manual participation in the production process, reducing labor costs and improving production efficiency. Stem-cutting picking can avoid damage to fruits and stems caused by pulling, and is a better way to pick fruits. Stem-cutting picking requires the picking robot to accurately identify and locate the picking points of solanaceae such as tomato stems, which requires high accuracy of the recognition algorithm and depth sensor. The depth camera used must be able to capture the stems of the tomatoes to be picked and be able to obtain the depth value of the target object more stably. Detecting targets such as stems can help the picking robot perceive the environmental information around the fruit, so as to avoid obstacles and grasp it reasonably. Semantic segmentation methods can effectively detect the pixel areas corresponding to tomato stems and fruits in images, and are often used for detection tasks such as fruits, stalks, and stems. However, most existing technologies only use RGB images as input for semantic segmentation prediction, and lack effective use of depth information. During the eggplant picking operation in a greenhouse environment, the stalks and fruits of the eggplants easily block the picking points on the stems of the fruits, making it impossible for the robot to identify and pick them. A better camera observation posture can avoid occlusion during the identification and positioning of the fruit stems, while maintaining a high measurement accuracy of the depth camera. However, most current picking robots have random observation angles for the fruits and lack estimation of better observation postures, resulting in inaccurate positioning of the picking points. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a solanaceous fruit picking robot and a method and device for estimating the position and posture of its fruit cluster observation in view of the above-mentioned defects of the prior art.

[0004] In order to achieve the above object, the present invention provides a method for estimating the position and posture of a fruit cluster, which comprises the following steps:

[0005] S100, collecting RGB image data and depth data of Solanaceae plants;

[0006] S200, after normalizing the RGB image data and the depth data respectively, inputting the trained bimodal fusion semantic segmentation model to predict masks corresponding to the two categories of fruit and stem in the RGB image;

[0007] S300, extracting depth information corresponding to the connected area of ​​the mask, and performing local mask refinement in combination with a density clustering method;

[0008] S400, clustering and extracting a single fruit bunch point cloud, calculating the center information of each fruit bunch point cloud, and extracting the stem point cloud adjacent to the fruit bunch for linear fitting; and

[0009] S500, determining the observation posture of the end camera according to the relative position of the center of the fruit bunch point cloud and the vertical point of the corresponding stem fitting straight line.

[0010] In the above-mentioned fruit cluster observation pose estimation method, the dual-modal fusion semantic segmentation model is obtained by the following training method:

[0011] S001. Construct a dual-modal semantic segmentation model based on a deep convolutional neural network. The network structure of the model includes two inputs, namely three-channel RGB data input and single-channel depth data input;

[0012] S002. Add a feature fusion correction module to the bimodal semantic segmentation model to perform channel correction and spatial correction on the RGB features and the depth features respectively; after feature fusion in combination with the attention mechanism, extract features through the backbone network, and output a mask corresponding to the specified category in the RGB image;

[0013] S003. Use a monocular depth estimation algorithm to generate a depth map for the labeled dataset images that lack corresponding depth data, and use the color images and the generated depth images to train a dual-modal fusion semantic segmentation model as a pre-trained model;

[0014] S004. Use the actually collected fruit cluster color image and depth image data to fine-tune the pre-trained model to obtain a finally trained dual-modal fusion semantic segmentation model.

[0015] In the above-mentioned fruit cluster observation pose estimation method, the three-channel RGB data and the single-channel depth data need to be normalized before input, and a bimodal fruit cluster fruit-stem semantic segmentation dataset is constructed.

[0016] In the above-mentioned method for estimating the observed pose of a fruit cluster, in step S300, a connected region detection is performed on the mask of each category predicted by the semantic segmentation model, and the corresponding depth data is extracted to obtain the local depth value data corresponding to each connected region.

[0017] The above-mentioned method for estimating the observed pose of a fruit cluster, wherein step S300 further comprises:

[0018] S310, extracting connected components from the single-category mask image output by the model, if the area of ​​the connected component is less than the set area threshold T area , then ignore the connected component;

[0019] S320, extracting the depth value in the depth map corresponding to each remaining connected component, and clustering the depth values; excluding clusters whose areas are smaller than a set threshold from the mask to retain areas with significant depth features; and

[0020] S330, extracting all regions with significant depth features to form updated mask regions, and integrating mask regions of a single category into a depth map.

[0021] The above-mentioned method for estimating the pose of a fruit cluster observation, wherein in step S400, an RGB-D camera is used to collect data, further comprising:

[0022] S410, clustering the extracted depth value data of each mask area, and calculating the pixel area corresponding to each category in the clustering result, splitting the areas that are connected but different clusters, and removing noise points and depth values ​​with an area less than a set value to filter noise and redundant information;

[0023] S420, converting the extracted depth value of each local area into a local point cloud according to the camera internal parameters of the RGB-D camera, and obtaining the center coordinates and the radius of the spatial envelope sphere for the local point cloud of the fruit category as the location information of each cluster of fruit; and

[0024] S430, extracting the point cloud of the stem category within a specified radius around the center of each cluster of fruit, extracting the point cloud of the stem near the fruit segment through cluster analysis, and performing straight line fitting.

[0025] The above-mentioned method for estimating the observed pose of a fruit cluster, wherein step S500 further comprises:

[0026] S510, finding the vertical point from the center of the fruit cluster to the fitted stem straight line segment, and preliminarily generating an initial position from the center of the fruit cluster toward the vertical point of view;

[0027] S520: Rotate and translate based on the initial posture to obtain a final end camera observation posture.

[0028] In the above-mentioned method for estimating the observed pose of a fruit cluster, step S510 further comprises:

[0029] S511, determine the coordinates of the point D on the perpendicular line segment from the center point C of the single fruit bunch point cloud to the corresponding straight line segment:

[0030] The foot of the perpendicular D is the intersection of the perpendicular line from the center point C of the fruit cluster to the line segment AB. Let the starting point and the end point of the straight line segment be P and A and P B ,but:

[0031]

[0032] vector represents the vector from the starting point A of the line segment to the center point C of the fruit string point cloud; t is a scalar, representing the vector of the line segment AB A ratio on the vertical axis to determine the position of the foot of the vertical axis;

[0033] S512, determine the camera's orientation vector from the camera position P C Point to the target position P D The z-axis is the direction of the camera, the y-axis is perpendicular to the camera base and points upwards, and the x-axis follows the right-hand rule and is perpendicular to the z-axis and y-axis.

[0034] Perform normalization:

[0035]

[0036] S513, Calculation The angle between the projection of the vector on the XOZ plane and the Z axis, if the angle is greater than the set angle threshold α, then according to P D With P A Relative position adjustment on the X axis

[0037]

[0038] S514, calculate the left vector Through the upward direction With direction vector The cross product is obtained and normalized to:

[0039]

[0040] is the upward direction in the world coordinate system, the default is (0,-1,0);

[0041] S515, recalculate the upper vector By direction vector With the right vector The cross product is obtained and normalized to:

[0042] as well as

[0043] S516, construct the rotation matrix R rot is the initial pose:

[0044]

[0045] In order to better achieve the above-mentioned objectives, the present invention also provides a fruit cluster observation pose estimation device, which is arranged on a Solanaceae fruit picking robot, and includes a data collector and a data processor installed at the end of the mechanical arm of the picking robot, wherein the data collector is used to obtain RGB image data and depth data within the observation field of view, and transmit them to the data processor, and the data processor is used to implement the above-mentioned fruit cluster observation pose estimation method to improve the recognition and positioning accuracy of the Solanaceae fruit picking robot for the fruit stalk picking point.

[0046] In order to better achieve the above objectives, the present invention also provides a solanaceous fruit picking robot, in which the above-mentioned fruit cluster observation pose estimation method is used to obtain the observation pose of the terminal camera to improve the recognition and positioning accuracy of the terminal camera for the fruit stalk picking point.

[0047] The technical effects of the present invention are:

[0048] The present invention is based on a fruit cluster observation pose estimation method based on dual-modal fusion, which can effectively utilize the terminal sensor data of the picking robot to improve the recognition performance of targets such as fruits and fruit stems; by estimating the optimal observation pose of the terminal camera of the picking robot, the recognition and positioning accuracy of the fruit stem picking point by the terminal camera of the picking robot arm is improved. The RGB and depth data are fully integrated to detect the tomato fruits and stems grown in the greenhouse environment of the facility, which improves the effective use of multimodal sensor data by the picking robot and enhances the recognition performance of the visual algorithm for important information of tomato plants; the monocular depth estimation model is used to generate a depth map for pre-training, which makes up for the lack of corresponding depth values ​​in the semantic segmentation data set; according to the spatial position relationship between the tomato fruit and the surrounding stems, the optimal cluster observation pose estimation of the terminal camera of the robot arm is determined, which can check the occlusion of the stem picking point by the stem and the fruit itself, and improve the detection rate of the picking point.

[0049] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 FIG. 1 is a diagram showing the working principle of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The structural principle and working principle of the present invention are described in detail below in conjunction with the accompanying drawings:

[0052] The eggplant and fruit picking robot of the present invention adopts a fruit cluster observation posture estimation method to obtain the observation posture of the terminal camera, so as to improve the recognition and positioning accuracy of the terminal camera for the fruit stalk picking point. The eggplant and fruit picking robot includes a fruit cluster observation posture estimation device, which is arranged on the eggplant and fruit picking robot, including a data collector and a data processor installed at the end of the mechanical arm of the eggplant and fruit picking robot. The data collector is used to obtain RGB image data and depth data within the observation field of view, and transmit them to the data processor. An RGB-D camera can be used, preferably an Intel Realsense L515 depth camera; the data processor is used to implement the fruit cluster observation posture estimation method, so as to improve the recognition and positioning accuracy of the eggplant and fruit picking robot for the fruit stalk picking point. Because the composition, structure, mutual position relationship, connection relationship and function of the data collector and data processor of the eggplant and fruit picking robot and the fruit cluster observation posture estimation device are all relatively mature existing technologies, they are not described here in detail. The following is only a detailed description of the fruit cluster observation posture estimation method of the present invention.

[0053] See also Figure 1 , Figure 1 This is a working principle diagram of an embodiment of the present invention. The present invention uses monocular depth estimation technology to generate corresponding depth maps for color image samples that lack depth information to enrich the dual-modal training data set; uses these data for pre-training to enhance the model's recognition ability of tomato clusters and stems; combines connected region extraction and density clustering methods to refine local masks; extracts fruit cluster position information based on density clustering and linear fitting of stems near the fruit segment; and estimates the optimal observation pose based on the spatial position relationship between the fruit cluster center and the stems near the fruit segment. The fruit cluster observation pose estimation method of the present invention includes the following steps:

[0054] Step S100, collecting RGB image data and depth data of Solanaceae plants;

[0055] Step S200: After normalizing the RGB image data and the depth data respectively, the trained dual-modal fusion semantic segmentation model is input to predict masks corresponding to the two categories of fruit and stem in the RGB image;

[0056] Step S300, extracting the depth information corresponding to the connected area of ​​the mask, and performing local mask refinement in combination with the density clustering method;

[0057] Step S400, clustering and extracting point clouds of single fruit bunches, calculating the center information of the point clouds of each cluster of fruit bunches, and extracting point clouds of stems adjacent to the fruit bunches for linear fitting; and

[0058] Step S500, determining the observation posture of the end camera according to the relative position of the center of the fruit bunch point cloud and the vertical point of the corresponding stem fitting straight line.

[0059] The bimodal fusion semantic segmentation model is obtained by the following training method:

[0060] Step S001, constructing a dual-modal semantic segmentation model based on a deep convolutional neural network, wherein the network structure of the model includes two inputs, namely, three-channel RGB data input and single-channel depth data input; the three-channel RGB data and the single-channel depth data need to be normalized before input, and a dual-modal fruit cluster fruit-stem semantic segmentation dataset is constructed;

[0061] Step S002: adding a feature fusion correction module to the bimodal semantic segmentation model to perform channel correction and spatial correction on the RGB features and the depth features respectively; after feature fusion in combination with the attention mechanism, extracting features through the backbone network, and outputting a mask corresponding to the specified category in the RGB image;

[0062] Step S003: using a monocular depth estimation algorithm to generate a depth map for the labeled dataset image that lacks corresponding depth data, and using the color image and the generated depth image to train a dual-modal fusion semantic segmentation model as a pre-trained model;

[0063] Step S004: fine-tune the pre-trained model using the actually collected fruit cluster color image and depth image data to obtain a finally trained dual-modal fusion semantic segmentation model.

[0064] In step S300, connected area detection is performed on the mask of each category predicted by the semantic segmentation model, and the local depth value data corresponding to each connected area is extracted from the corresponding depth data.

[0065] Step S310: extract connected components from the single-category mask map output by the model. If the area of ​​the connected component is less than the set area threshold T area , then ignore the connected component;

[0066] Step S320, extracting the depth value in the depth map corresponding to each remaining connected component, and clustering the depth values; excluding clusters with an area smaller than a set threshold from the mask to retain areas with significant depth features; and

[0067] Step S330: extract all regions with significant depth features to form updated mask regions, and integrate the mask regions of a single category into a depth map.

[0068] In step S400, using an RGB-D camera to collect data further includes:

[0069] Step S410: cluster the extracted depth value data of each mask area, calculate the pixel area corresponding to each category in the clustering result, split the connected but different cluster areas, and remove noise points and depth values ​​with an area less than a set value to filter noise and redundant information;

[0070] Step S420: converting the extracted depth value of each local area into a local point cloud according to the camera internal parameters of the RGB-D camera, and obtaining the center coordinates and the radius of the spatial envelope sphere of the local point cloud of the fruit category as the position information of each cluster of fruit; and

[0071] Step S430, extracting the point cloud of the stem category within a specified radius around the center of each cluster of fruit, extracting the point cloud of the stem near the fruit segment through cluster analysis, and performing straight line fitting.

[0072] Step S500 further includes:

[0073] Step S510, finding the vertical point from the center of the fruit cluster to the fitted stem straight line segment, and preliminarily generating an initial posture from the center of the fruit cluster toward the vertical point of view;

[0074] Step S520: rotating and translating based on the initial posture to obtain the final end camera observation posture.

[0075] Step S510 further includes:

[0076] Step S511, determine the coordinates of the point D on the perpendicular line segment from the center point C of the single fruit bunch point cloud to the corresponding straight line segment:

[0077] The foot of the perpendicular D is the intersection of the perpendicular line from the center point C of the fruit string point cloud to the line segment AB. Pc is used as Pcenter to project the foot of the perpendicular. The starting point and the end point of the straight line segment are P A and P B ,but:

[0078]

[0079] vector represents the vector from the starting point A of the line segment to the center point C of the fruit string point cloud; t is a scalar, representing the vector of the line segment AB A ratio on the vertical axis to determine the position of the foot of the vertical axis;

[0080] Step S512: Determine the camera's orientation vector from the camera position P C Point to the target position P D The z-axis is the direction of the camera, the y-axis is perpendicular to the camera base and points upwards, and the x-axis follows the right-hand rule and is perpendicular to the z-axis and y-axis.

[0081] Perform normalization:

[0082]

[0083] Step S513: Calculate The angle between the projection of the vector on the XOZ plane and the Z axis, if the angle is greater than the set angle threshold α, then according to P D With P A Relative position adjustment on the X axis

[0084]

[0085] Step S514, calculate the left vector Through the upward direction With direction vector The cross product is obtained and normalized to:

[0086]

[0087] is the upward direction in the world coordinate system, the default is (0,-1,0);

[0088] Step S515: Recalculate the upper vector By direction vector With the right vector The cross product is obtained and normalized to:

[0089] as well as

[0090] Step S516: construct the rotation matrix R rot is the initial pose:

[0091]

[0092] The present invention identifies the fruit clusters and stems of eggplants based on a dual-modal fusion semantic segmentation method, generates a depth map with the help of a monocular depth estimation method to make up for the lack of depth data in conventional semantic segmentation data sets, and integrates color image information and depth information to improve the detection accuracy of eggplant fruits and stems. The eggplant stems and fruit clusters are extracted and analyzed in depth through spatial point cloud technology to estimate the optimal observation posture. This helps to optimize the observation angle of eggplant picking robots and improve the quality and efficiency of data collection.

[0093] The specific application of the fruit cluster observation pose estimation method of the present invention is described in detail below using tomatoes as an example.

[0094] First, the bimodal fusion semantic segmentation model is built and trained.

[0095] The visual sensor carried at the end of the eggplant and fruit picking robot can generally obtain RGB image data and depth data within the observation field of view, but most of the current picking robot visual detection models basically only use RGB image data as input for reasoning and prediction, lacking effective use of other sensor information. At the same time, since the depth sensor generally has measurement noise, it is easy to increase the model prediction error. Correcting and fusing data of different modalities is an effective way to improve the visual detection accuracy of the picking robot. The present invention can perform feature correction and fusion of RGB and depth modal data on the semantic segmentation model, and the network structure of the model includes two inputs, namely three-channel RGB images and single-channel depth data. The two input data are respectively input into the feature correction and fusion module after the initialization layer. In the feature correction and fusion module, three parts are included: channel correction, spatial correction and fusion. After the correction and fusion process, the fused features are input into the semantic segmentation encoder structure for feature encoding, and finally the mask corresponding to each category is output through the decoder.

[0096] To address the problem of insufficient bimodal image data and annotations, the Depth Anything monocular depth estimation algorithm is first used to generate depth maps for the annotated tomato fruit-stem semantic segmentation dataset images that lack corresponding depth information. A bimodal fusion semantic segmentation model is trained by combining color images and generated depth images, and this model is used as a pre-training basis. Subsequently, the pre-trained model is applied to the real collected dataset for further training and optimization.

[0097] During the data collection phase, the Intel Realsense L515 depth camera was used to collect RGB image data and depth data of tomato plants in the tomato garden. The alignment function provided by the camera SDK was used to ensure accurate alignment between the collected RGB images and depth maps. The collected RGB images were saved in JPEG format, while the depth data were stored in NPY format files for subsequent processing and analysis.

[0098] Furthermore, Labelme software was used to perform fine contour annotation of target categories in the image, such as tomato fruits and stems. After the annotation was completed, the sample data was divided into training set, validation set and test set in a ratio of 8:1:1 to ensure the comprehensiveness of model training and the accuracy of evaluation. In the model training stage, the network was constructed to load the pre-trained weights, and the aligned and normalized data was loaded for training, and finally the semantic segmentation model of tomato fruits and stems was obtained.

[0099] Secondly, local mask refinement is performed.

[0100] Since the segmentation model may have deviations in edge prediction and the depth camera may have errors in measurement, these problems may cause the mask obtained from the segmentation model to be mapped inaccurately in the depth map, and the background depth information may be incorrectly included. To solve these problems, this embodiment first uses the connectedComponentsWithStats function of OpenCV to extract connected components from the single-category mask map output by the model. By setting an area threshold T area , if it is set to 500 pixels, if the area of ​​any connected component is smaller than the threshold T area , the component is ignored.

[0101] For each remaining connected component, extract the depth value in the depth map corresponding to each component, and use the DBSCAN method to cluster the depth values ​​within each connected component. Combine the coordinates (u, v) of each pixel with its corresponding depth value to form a feature vector, which is used as the input of the DBSCAN algorithm. In this embodiment, the parameters of the DBSCAN algorithm are set to eps=0.5 and min_samples=30. According to the clustering results of the DBSCAN algorithm, if the area of ​​a cluster is less than the set threshold, the depth values ​​of all pixels in the cluster are set to 0, that is, these areas are excluded from the mask. This step helps to further refine the mask to ensure that only those areas with significant depth features are retained.

[0102] Finally, all regions with non-zero depth values ​​are extracted to form updated mask regions. Through this process, a more accurate mask is obtained, and then these single-category mask regions are integrated into a depth map. According to the camera intrinsic parameters, the spatial point cloud corresponding to each category mask is calculated separately, and the calculation formula is:

[0103] Z = M (u, v) · d · s;

[0104]

[0105] Where X, Y, and Z are three-dimensional space coordinates; d is the depth image value at the pixel position (u, v); s is the depth scale factor; u and v are pixel coordinates, where u is the column coordinate and v is the row coordinate; c x , c y is the principal point coordinate in the camera intrinsic parameter; f x , f y is the focal length of the camera; M(u, v) is the mask function, which is used to select only the pixels within the mask area for calculation.

[0106] Then, the fruit cluster extraction is performed.

[0107] In order to merge the spatially close but unconnected tomato fruit point clouds into a complete tomato bunch, the DBSCAN algorithm is used in this embodiment to cluster the spatial point clouds of the tomato category. The DBSCAN algorithm can identify and aggregate adjacent point clouds into clusters based on spatial density, thereby achieving a complete representation of the tomato bunch. After clustering is completed, it is necessary to determine the coordinates of the center point of each tomato bunch point cloud cluster, which can be achieved by calculating the center of mass (i.e., average coordinates) of all points in each cluster. The calculation formula is as follows:

[0108]

[0109] Among them, P C is the center coordinate of the point cloud; N is the number of points in the point cloud; P i is the three-dimensional coordinate of the i-th point.

[0110] The bounding box of the fruit cluster point cloud is expressed as follows:

[0111]

[0112] The radius of the fruit cluster envelope is:

[0113]

[0114] Next, a straight line fitting of the stem near the fruit segment is performed.

[0115] For each extracted tomato bunch point cloud, calculate the distance from the stem point cloud to the center of the tomato bunch. Extract the distance from the center of the object box in the radius plus the extra range from the stem point cloud and set the distance threshold:

[0116] T dis =r+a;

[0117] Calculate the Euclidean distance from each point in the point cloud to the center point:

[0118] distances=||PP C || 2 ;

[0119] Use the calculated distance to filter out points that meet the conditions:

[0120] indeices={i|distance[i]≤t dis};

[0121] Use the filtered index set to extract the corresponding points from the original point cloud to form a subset:

[0122] {P n} = P[indices];

[0123] Use PCA (principal component analysis) to filter out the point cloud Pn Perform straight line fitting to obtain a straight line. This is not limited to using the PCA method to perform straight line fitting.

[0124] Data Decentralization: Computing {P n}, decentralize the data X={P n}-μ.

[0125] Compute the covariance matrix:

[0126]

[0127] Determine the eigenvalue λ and eigenvector V, and select the eigenvector V with the largest eigenvalue max as the main direction of the point cloud.

[0128] Take the mean point obtained by PCA as the reference point on the straight line,

[0129] Parameterization of the equation of a line: Here, t is a parameter and can take any real value.

[0130] Finally, the observation pose is estimated.

[0131] First, the coordinates of the foot point of the perpendicular line segment from the center of the point cloud of a single fruit bunch to its corresponding straight line segment are obtained.

[0132] Assume that the starting point and end point of the straight line segment are P A and P B , the line segment direction vector:

[0133]

[0134] vector Represents the vector from the starting point A of the line segment to the center point C of the fruit string point cloud. The calculation formula is:

[0135]

[0136] Calculate the t value of the vertical point D: t is a scalar, indicating the vector A ratio on the line segment AB is used to determine the position of the foot of the perpendicular D. The foot of the perpendicular D is the intersection of the perpendicular line from point C to line segment AB. The calculation formula for t is as follows:

[0137]

[0138] The coordinates of the foot point are:

[0139]

[0140] After determining the coordinates of the perpendicular point, the observation posture is determined. In the process of determining the observation posture, the viewing angle of the camera carried by the end effector is defined as a local tool coordinate system. In this coordinate system, the z-axis is defined as the direction of the camera, that is, the straight line pointing in front of the camera; the y-axis is perpendicular to the camera base and points upward, indicating the top of the camera; and the x-axis follows the right-hand rule, is perpendicular to the z-axis and the y-axis, and points to the left side of the camera.

[0141] Determine the camera's heading vector, which is the vector from the camera position P C Point to the target position P D , and normalize it:

[0142]

[0143] Taking into account the situation that the picking end cannot reach the observation position due to stem interference, the vector To solve the problems of occlusion and inaccessibility of the robot arm. The angle between the projection of the vector on the XOZ plane and the Z axis. If the angle is greater than the given angle threshold α, then according to P D With P A Relative position adjustment on the X axis

[0144]

[0145] Assumptions is the upward direction in the world coordinate system, and the default value is (0,-1,0). Next, calculate the left vector Through the upward direction With direction vector The cross product is obtained and normalized:

[0146]

[0147] Recalculate the up vector By direction vector With the right vector The cross product of and Orthogonal and normalized:

[0148]

[0149] Construct a rotation matrix R rot , express attitude:

[0150]

[0151] Finally, the attitude and position adjustments are performed to give the end camera a better observation distance and angle. The camera will be rotated 30° counterclockwise around its viewing direction axis (pitch axis). This adjustment will allow the camera to obtain an oblique downward perspective, which will help to better capture the details of the target area. After the attitude adjustment is completed, the camera will be moved a distance D in the negative direction (that is, the opposite direction of the current camera direction). This step will ensure that the camera remains within the appropriate viewing distance of the target area. The distance D is set according to the camera's field of view and the optimal distance for depth imaging. Finally, the observation posture of the camera is determined, and observation is performed at a downward angle at a distance D above the center of the target.

[0152] In this embodiment, a dual-modal semantic segmentation model based on a deep convolutional neural network is first constructed. The network structure of the model contains two inputs, namely, three-channel RGB data input and single-channel depth data input, which need to be normalized before input. A feature fusion correction module is added to the model, and channel correction and spatial correction are performed on the RGB features and depth features respectively, and then feature fusion is performed in combination with the attention mechanism, and then the features are extracted through the backbone network, and the mask corresponding to the specified category in the output RGB image is inferred; then the dual-modal fusion model is pre-trained. A monocular depth estimation algorithm is used to generate a depth map for the annotated data set image that lacks corresponding depth data, and the color image and the generated depth image are used to train the dual-modal fusion semantic segmentation model as a pre-training model. The color image and depth map data actually collected by the RGB-D camera are then used for fine-tuning training to obtain the final semantic segmentation model of tomato stems and stalks.

[0153] During use, an RGB-D camera is used as a data acquisition device to collect RGB images and depth data of tomato plants. After normalizing the RGB data and depth data, the trained dual-modal semantic segmentation model is input to predict the masks corresponding to the two categories of tomato fruit and stem in the image; the connected area detection is performed on the mask of each category predicted by the semantic segmentation model, and the local depth value data corresponding to each connected area is extracted from the corresponding depth data; the depth value data of each extracted mask area is clustered, and the corresponding pixel area of ​​each category in the clustering result is calculated, the connected but different cluster areas are split, and the noise points and the depth values ​​with too small area are eliminated to filter the noise and redundant information; according to the camera internal parameters, the depth value of each local area extracted is converted into a local point cloud. For the local point cloud of tomato fruit, its center coordinates and the radius of the spatial envelope sphere are obtained as the location information of each cluster of fruit; the point cloud of the stem category within the specified radius around the center of each cluster of fruit clusters is extracted, and the point cloud of the stem near the fruit segment is extracted through cluster analysis, and a straight line fitting is performed. Finally, the observation pose is estimated: First, the vertical point from the center of the fruit cluster to the fitted straight line segment of the stem is calculated, and an initial pose from the center of the fruit cluster toward the vertical point is initially generated. Based on this initial pose, the final observation pose is obtained by rotation and translation.

[0154] The present invention constructs a dual-modal semantic segmentation model of RGB images and depth data, and uses a monocular depth estimation method to generate a depth map to supplement the missing depth information in the traditional semantic segmentation data set, and then improves the detection accuracy of tomato fruits and stems by fusing color images and depth data. By accurately extracting the local spatial point cloud of tomato stems and fruit clusters and conducting an in-depth analysis of their spatial position relationship, a better observation posture is estimated. The accuracy of identifying and locating the picking point of the tomato stem under the perspective of the terminal camera of the eggplant picking robot is significantly improved.

[0155] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for estimating the pose of a fruit cluster, characterized in that: The steps include: S100, collecting RGB image data and depth data of Solanaceae plants; S200, after normalizing the RGB image data and the depth data respectively, inputting the trained bimodal fusion semantic segmentation model to predict masks corresponding to the two categories of fruit and stem in the RGB image; S300, extracting depth information corresponding to the connected area of ​​the mask, and performing local mask refinement in combination with a density clustering method; S400, clustering and extracting a single fruit bunch point cloud, calculating the center information of each fruit bunch point cloud, and extracting the stem point cloud adjacent to the fruit bunch for linear fitting; and S500, determining the observation posture of the end camera according to the relative position of the center of the fruit bunch point cloud and the vertical point of the corresponding stem fitting straight line.

2. The method for estimating the position and posture of a fruit cluster according to claim 1, wherein: The bimodal fusion semantic segmentation model is obtained by the following training method: S001. Construct a dual-modal semantic segmentation model based on a deep convolutional neural network. The network structure of the model includes two inputs, namely three-channel RGB data input and single-channel depth data input; S002. Adding a feature fusion correction module to the bimodal semantic segmentation model to perform channel correction and spatial correction on the RGB features and the depth features respectively; After feature fusion combined with the attention mechanism, features are extracted through the backbone network, and the mask corresponding to the specified category in the RGB image is output; S003. Use a monocular depth estimation algorithm to generate a depth map for the labeled dataset images that lack corresponding depth data, and use the color images and the generated depth images to train a dual-modal fusion semantic segmentation model as a pre-trained model; as well as S004. Use the actually collected fruit cluster color image and depth image data to fine-tune the pre-trained model to obtain a finally trained dual-modal fusion semantic segmentation model.

3. The method for estimating the position and posture of a fruit cluster according to claim 2, wherein: The three-channel RGB data and single-channel depth data need to be normalized before input, and a bimodal fruit cluster fruit-stem semantic segmentation dataset is constructed.

4. The method for estimating the position and posture of a fruit cluster according to claim 1, wherein: In step S300, connected region detection is performed on the mask of each category predicted by the semantic segmentation model, and local depth value data corresponding to each connected region is extracted from the corresponding depth data.

5. The method for estimating the position and posture of a fruit cluster according to claim 4, wherein: Step S300 further includes: S310: extracting connected components from the single-category mask image output by the model. If the area of ​​the connected component is less than the set area threshold T area , then ignore the connected component; S320, extracting the depth value in the depth map corresponding to each remaining connected component, and clustering the depth values; excluding clusters whose areas are smaller than a set threshold from the mask to retain areas with significant depth features; and S330, extracting all regions with significant depth features to form updated mask regions, and integrating mask regions of a single category into a depth map.

6. The method for estimating the position and posture of a fruit cluster according to claim 1, wherein: In step S400, using an RGB-D camera to collect data further includes: S410, clustering the extracted depth value data of each mask area, and calculating the pixel area corresponding to each category in the clustering result, splitting the areas that are connected but different clusters, and removing noise points and depth values ​​with an area less than a set value to filter noise and redundant information; S420, converting the extracted depth value of each local area into a local point cloud according to the camera internal parameters of the RGB-D camera, and obtaining the center coordinates and the radius of the spatial envelope sphere for the local point cloud of the fruit category as the location information of each cluster of fruit; and S430, extracting the point cloud of the stem category within a specified radius around the center of each cluster of fruit, extracting the point cloud of the stem near the fruit segment through cluster analysis, and performing straight line fitting.

7. The method for estimating the position and posture of a fruit cluster according to claim 1, wherein: Step S500 further includes: S510, finding the vertical point from the center of the fruit cluster to the fitted stem straight line segment, and preliminarily generating an initial position from the center of the fruit cluster toward the vertical point of view; and S520: Rotate and translate based on the initial posture to obtain a final end camera observation posture.

8. The method for estimating the position and posture of a fruit cluster according to claim 7, wherein: Step S510 further includes: S511, determine the coordinates of the point D on the perpendicular line segment from the center point C of the single fruit bunch point cloud to the corresponding straight line segment: The foot of the perpendicular D is the intersection of the perpendicular line from the center point C of the fruit cluster point cloud to the line segment AB. Let the starting point and the end point of the straight line segment be P and A and P B ,but: vector represents the vector from the starting point A of the line segment to the center point C of the fruit string point cloud; t is a scalar, representing the vector of the line segment AB A ratio on the vertical axis to determine the position of the foot of the vertical axis; S512, determine the camera's orientation vector from the camera position P C Point to the target position P D The z-axis is the direction of the camera, the y-axis is perpendicular to the camera base and points upwards, and the x-axis follows the right-hand rule and is perpendicular to the z-axis and y-axis. Perform normalization: S513, calculation The angle between the projection of the vector on the XOZ plane and the Z axis, if the angle is greater than the set angle threshold α, then according to P D With P A Relative position adjustment on the X axis S514, calculate the left vector Through the upward direction With direction vector The cross product is obtained and normalized to: is the upward direction in the world coordinate system, the default is (0,-1,0); S515, recalculate the upper vector By direction vector With the right vector The cross product is obtained and normalized to: as well as S516, construct the rotation matrix R rot is the initial pose:

9. A fruit cluster observation and posture estimation device, arranged on a solanaceous fruit picking robot, comprising a data collector and a data processor installed at the end of the mechanical arm of the picking robot, characterized in that: The data collector is used to obtain RGB image data and depth data within the observation field of view and transmit them to the data processor. The data processor is used to implement the fruit cluster observation pose estimation method described in any one of claims 1-8 to improve the recognition and positioning accuracy of the eggplant picking robot for the fruit stalk picking points.

10. A solanaceae fruit picking robot, characterized in that: The fruit cluster observation pose estimation method described in any one of claims 1 to 8 is used to obtain the observation pose of the terminal camera to improve the recognition and positioning accuracy of the fruit stalk picking point by the terminal camera.

Images