Three-dimensional position recognition method and system for fruits on a tree based on a single two-dimensional image

Abstract

The application provides a three-dimensional position and posture recognition method for fruits on a tree based on a single two-dimensional image, so as to improve the safety and accuracy of automatic picking. The method mainly comprises the following steps: the position and posture of the fruit in the three-dimensional space are represented by a fruit eye point and a normal direction of a plane on which the fruit eye point is located, a fruit posture labeling tool based on a two-dimensional image is developed, and a fruit posture data set is constructed; a deep convolutional neural network for recognizing the three-dimensional position and posture of the fruit based on the two-dimensional image is constructed, and model training is performed by using the labeled data; a fruit canopy image is shot, target detection is firstly performed on the fruit in the image, and then the trained model is used to recognize the position and posture of the single fruit, so that the position and posture of the fruit are determined. In the actual picking process, the application can help to realize high-precision and low-damage picking operation of the fruit.

Description

Technical Field

[0001] This invention relates to the field of agricultural intelligent robots, and in particular to a method for recognizing the three-dimensional pose of fruit on a tree based on a single two-dimensional image. Background Technology

[0002] In existing technologies, the identification of the three-dimensional pose of fruits often adopts methods based on color two-dimensional images and depth images RGB-D: traditional image processing algorithms or deep learning algorithms; or methods based on color two-dimensional images RGB: indirect pose identification or direct pose identification based on perspective-n-point projection (PnP) algorithms.

[0003] Specifically, traditional image processing algorithms based on RGB-D images use methods such as nonlinear least squares and Hough transform to process RGB-D data of fruits and calculate their pose. Deep learning algorithms based on RGB-D images leverage the PointNet neural network architecture, capable of end-to-end processing of point cloud data, to achieve fruit pose recognition entirely based on deep learning. Indirect pose recognition using the PnP algorithm based on RGB images predicts the coordinates of important corner points, such as the object's bounding box, projected onto a 2D image, and then solves the PnP algorithm based on multiple 3D-2D corresponding points to obtain the object's pose. Direct pose recognition based on RGB images uses expressions such as quadruples and Lie algebras to directly solve for the rotation parameters of the object's pose in 3D space through deep learning algorithms.

[0004] However, the aforementioned existing technology has the following technical defects:

[0005] 1. RGB-D image-based methods: Pose recognition relies on depth data captured by depth cameras or other devices, but the depth information collected in outdoor orchards is unstable and can easily have a negative impact on the accuracy of fruit pose recognition.

[0006] 2. RGB image-based methods: These methods only study objects in controlled environments and do not address the problem of fruit pose recognition in uncontrolled orchard environments. Uncontrolled orchards refer to outdoor orchards where the environment is complex and variable, with factors such as background environment, lighting, occlusion, and pose being unrestricted. Summary of the Invention

[0007] This invention proposes a simple, accurate, and robust method for recognizing the three-dimensional pose of fruits based on two-dimensional images, which solves the problem of recognizing the grasping posture of automated picking robots in uncontrolled orchard environments.

[0008] This invention uses the fruit's navel point and the normal direction of the plane containing the navel point to represent the fruit's position and posture in three-dimensional space. It provides a tool for labeling fruit position and three-dimensional posture based on two-dimensional images, offering reliable annotation values ​​for the training and testing of deep learning models. Recognizing the fruit's posture in three-dimensional space based on two-dimensional images eliminates the need for point cloud data, avoiding negative interference from unstable depth information acquired outdoors. Utilizing deep learning algorithms for end-to-end fruit pose recognition from two-dimensional images is not limited by fruit shape, type, or image acquisition environment, resulting in higher accuracy and better robustness and generalization. Furthermore, it can recognize fruits in arbitrary postures in uncontrolled orchard environments, demonstrating good robustness and scalability.

[0009] Specifically, this invention provides a method for recognizing the three-dimensional pose of fruit on a tree based on a single two-dimensional image, including:

[0010] Step 1: Obtain multiple fruit tree images with fruit. Based on the fruit navel point and the normal of the plane containing the fruit navel point in the fruit tree image, label the fruit pose in the fruit tree image. Train a fruit pose recognition model based on a deep convolutional network using the fruit tree images.

[0011] Step 2: The harvesting robot takes a picture of the target fruit tree and performs target detection on the picture with the fruit as the target. The detected fruit image is input into the fruit pose recognition model to obtain the target fruit pose. The end effector of the harvesting robot is then controlled to perform the task operation according to the target fruit pose and the rotation direction of the target fruit.

[0012] The method for recognizing the 3D pose of fruit on a tree based on a single 2D image, specifically includes the following in step 1: labeling the pose.

[0013] Label the fruit navel point in the fruit tree image. Model an ellipsoid based on the shape of the fruit. One end of the minor axis of the ellipsoid represents the connection point between the fruit and the pedicel, and the other end represents the fruit navel point. By adjusting the size and rotation direction of the ellipsoid, make its projection match the fruit in the fruit tree image. At this time, take the minor axis of the ellipsoid as the rotation direction of the fruit, pointing from the fruit navel point to the connection point between the fruit and the pedicel.

[0014] The method for recognizing the 3D pose of fruit on a tree based on a single 2D image, the training process in step 1 includes:

[0015] A multi-layer deep convolutional neural network was constructed using residual modules as the feature extraction layer for this fruit pose recognition model.

[0016] A deconvolutional layer is used to restore the low-resolution feature map output by the feature extraction layer to a high-resolution feature map, and a heatmap is generated using this high-resolution feature map to predict the fruit navel point P in the map. iThe location of the fruit and the loss function Loss1 are shown below, where n represents the number of fruits in the dataset. H(·) represents the heatmap generated by the network, where H(·) represents the heatmap corresponding to the labeled fruit navel point.

[0017] Using this fruit pose recognition model, multi-scale feature fusion and a fully connected classification layer are applied to obtain the unit normal vector of the plane containing the fruit navel point. The loss function Loss2 uses the cosine distance loss shown in the following formula. represents the unit normal vector predicted by the network, and v represents the labeled fruit pose;

[0018]

[0019]

[0020] Loss M2 = w1 Loss1 + w2 Loss2

[0021] Using the final loss function Loss M2 Train the fruit pose recognition model.

[0022] The method for recognizing the three-dimensional pose of fruit on a tree based on a single two-dimensional image, step 2 includes:

[0023] During the harvesting process, the harvesting robot uses a depth camera to determine the two-dimensional coordinates (px) of the fruit's navel point in the pixel coordinate system using the following formula. P py P Convert ) to 3D coordinates in the camera coordinate system (px) C py C pz C ), pz C This represents the value of the fruit navel point in the Z direction of the camera coordinate system, where K represents the camera intrinsic parameter, and f x f y Let (nx, ny, nz) represent the focal lengths of the camera on the x and y axes, respectively. Let (u0, v0) represent the coordinates of the origin of the image coordinate system in the pixel coordinate system. This controls the end effector to move towards (px) with a feed angle of (nx, ny, nz). C py C pz C Move forward to complete the fruit-grabbing operation;

[0024]

[0025] This invention also proposes a three-dimensional pose recognition system for fruits on trees based on a single two-dimensional image, including:

[0026] The training module is used to acquire multiple fruit tree images with fruits, and to label the fruit pose in the fruit tree image based on the fruit navel point and the normal of the plane containing the fruit navel point; and to train a fruit pose recognition model based on a deep convolutional network using the fruit tree images.

[0027] The recognition module is used by the picking robot to take on-site images of the target fruit tree, and to perform target detection on the on-site images with the fruit as the target. The detected fruit images are input into the fruit pose recognition model to obtain the target fruit pose, so as to control the end effector of the picking robot to perform task operations according to the target fruit pose and the rotation direction of the target fruit.

[0028] The aforementioned three-dimensional pose recognition system for tree fruits based on a single two-dimensional image, wherein the pose labeling specifically includes:

[0029] Label the fruit navel point in the fruit tree image. Model an ellipsoid based on the shape of the fruit. One end of the minor axis of the ellipsoid represents the connection point between the fruit and the pedicel, and the other end represents the fruit navel point. By adjusting the size and rotation direction of the ellipsoid, make its projection match the fruit in the fruit tree image. At this time, take the minor axis of the ellipsoid as the rotation direction of the fruit, pointing from the fruit navel point to the connection point between the fruit and the pedicel.

[0030] The aforementioned three-dimensional pose recognition system for fruit on a tree based on a single two-dimensional image includes the following training process:

[0031] A multi-layer deep convolutional neural network was constructed using residual modules as the feature extraction layer for this fruit pose recognition model.

[0032] A deconvolutional layer is used to restore the low-resolution feature map output by the feature extraction layer to a high-resolution feature map, and a heatmap is generated using this high-resolution feature map to predict the fruit navel point P in the map. i The location of the fruit and the loss function Loss1 are shown below, where n represents the number of fruits in the dataset. H(·) represents the heatmap generated by the network, where H(·) represents the heatmap corresponding to the labeled fruit navel point.

[0033] Using this fruit pose recognition model, multi-scale feature fusion and a fully connected classification layer are applied to obtain the unit normal vector of the plane containing the fruit navel point. The loss function Loss2 uses the cosine distance loss shown in the following formula. represents the unit normal vector predicted by the network, and v represents the labeled fruit pose;

[0034]

[0035]

[0036] Loss M2 = w1 Loss1 + w2 Loss2

[0037] Using the final loss function Loss M2 Train the fruit pose recognition model.

[0038] The aforementioned three-dimensional pose recognition system for fruit on a tree based on a single two-dimensional image, wherein the recognition module is used for:

[0039] During the harvesting process, the harvesting robot uses a depth camera to determine the two-dimensional coordinates (px) of the fruit's navel point in the pixel coordinate system using the following formula. P py P Convert ) to 3D coordinates in the camera coordinate system (px) C py C pz C ), pz C The value of the fruit navel point in the Z direction of the camera coordinate system is represented by K, where f represents the camera intrinsic parameter. x f y Let (nx, ny, nz) represent the focal lengths of the camera on the x and y axes, respectively. Let (u0, v0) represent the coordinates of the origin of the image coordinate system in the pixel coordinate system. This controls the end effector to move towards (px) with a feed angle of (nx, ny, nz). C py C pz C Move forward to complete the fruit-grabbing operation;

[0040]

[0041] The present invention also proposes a storage medium for storing a program that executes any of the tree fruit three-dimensional pose recognition methods based on a single two-dimensional image.

[0042] The present invention also proposes a client for any of the aforementioned three-dimensional pose recognition systems for fruits on trees based on a single two-dimensional image.

[0043] As can be seen from the above solutions, the advantages of the present invention are:

[0044] This invention is a visual algorithm solution to improve the safety and success rate of fruit picking. It can better adapt to different types of end effectors and avoid collisions between the end effector and the fruit stem during the fruit picking process, which would cause the fruit to be picked to shift and reduce damage to the fruit, fruit tree and even robotic arm.

[0045] The proposed pose annotation tool can annotate the three-dimensional pose of fruits on two-dimensional images, providing reliable data labels for the training and application of deep convolutional neural networks.

[0046] In the fruit posture recognition process, no depth information is required, which can better resist the negative impact of unstable depth information collected outdoors on model performance. The use of deep convolutional neural networks improves the model's inference speed, robustness and scalability, and enhances the performance of posture recognition, so that the present invention can be better applied to the vision system of harvesting robots. Attached Figure Description

[0047] Figure 1 This is a flowchart of the overall technical solution of the present invention;

[0048] Figure 2 This section provides data and labeled sample images of fruit in the canopy of fruit trees.

[0049] Figure 3 A schematic diagram of the fruit pose annotation tool interface;

[0050] Figure 4 This is a schematic diagram of a fruit pose recognition model. Detailed Implementation

[0051] To mitigate the negative impact of unstable point cloud data collected in complex orchard environments on fruit pose recognition, a three-dimensional pose recognition method for tree fruits based on a single two-dimensional image is proposed to improve the safety and accuracy of automated harvesting. The main steps include: 1) Representing the fruit's position and pose in three-dimensional space using the fruit navel point and the normal direction of the plane containing the navel point, developing a fruit pose annotation tool based on two-dimensional images, and constructing a fruit pose dataset. 2) Constructing a deep convolutional neural network for fruit pose recognition based on two-dimensional images, and training the model using training data. 3) During testing, firstly, target detection is performed on the fruits in the image, and then the trained model is used to recognize the pose of individual fruits, determining their position and pose. In actual harvesting, this invention can help achieve high-precision, low-damage fruit harvesting operations.

[0052] To achieve the above-mentioned objectives, this invention designs a method for three-dimensional pose recognition of fruit on a tree based on two-dimensional images and deep convolutional neural networks, comprising the following steps:

[0053] 1) Training the fruit detection model. First, a large number of images of fruit tree canopies with ripe fruit were collected in a real orchard environment. Then, the collected 2D images were manually labeled to construct a fruit detection dataset. As an object detection task, a deep convolutional neural network model for detecting fruit was designed and implemented, and the fruit object detection model was trained end-to-end using the fruit detection dataset.

[0054] 2) Training of the Fruit Pose Recognition Model. Based on the fruit detection box, the position and pose of the fruit in 3D space are represented by the fruit navel point and the normal direction of the plane containing the fruit navel point. An annotation tool is developed to manually annotate the pose of individual fruit images, constructing a fruit pose recognition dataset. A deep convolutional neural model for recognizing fruit poses is designed and implemented, and the fruit pose recognition model is trained end-to-end using the fruit pose recognition dataset. The plane containing the fruit navel point is the nearest neighbor region on the fruit centered on the fruit navel point, such as a 2*2 pixel region, which can be fitted into a plane (three non-collinear points can define a plane), i.e., the plane containing the fruit navel point.

[0055] 3) Fruit Pose Recognition in Situ. First, the trained target detection model is used to identify fruit targets in 2D images captured by the harvesting robot. Then, the fruit pose recognition model is used to predict the rotation direction (3D pose) and target position of each fruit. Finally, based on the pose information predicted by the model, combined with depth sensing equipment and a robotic arm, the harvesting robot moves towards the fruit (blossom point) at an appropriate angle (fruit rotation direction) to complete high-precision, low-damage tasks. These tasks require adjusting the robotic arm's movements according to the fruit's pose to improve automation performance, such as harvesting target fruits, bagging target fruits, and packaging harvested fruits.

[0056] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0057] This invention provides a method for fruit pose recognition based on two-dimensional images, the method flow is as follows: Figure 1 The following will introduce the implementation process in three steps.

[0058] 1. Training of the fruit detection model

[0059] 1) Construction of the Fruit Detection Dataset. In a real orchard setting, image acquisition equipment was used to photograph the canopy of ripe fruit trees. The camera was positioned 0.3m to 1m away from the canopy to ensure that multiple clear fruit targets were included in the captured images. Images of the fruit trees were recorded under various angles, distances, lighting conditions, and occlusion to enhance data diversity. Using general annotation software, the acquired images were manually annotated to construct the fruit detection dataset. For each fruit target, its bounding box was a closed rectangular area composed of four coordinate points (e.g., ...). Figure 2 The coordinates of the top left and bottom right corners of the rectangular area, {(bx1,by1),(bx2,by2)}, are recorded as the annotation results.

[0060] 2) Fruit Detection Model Training. A target detection model (such as the FaceBoxes architecture) is used to implement the fruit target detection task. The fruit detection model M is trained end-to-end using the fruit detection dataset. detection Input an image of the fruit tree canopy and the corresponding rectangular region annotation information of the fruit, and train M. detection Detect all fruit regions in the output image.

[0061] 3) Other possible implementations. In this step, the image acquisition of the canopy of mature fruit trees may take place in an outdoor orchard or other similar setting; the image acquisition device may be a mobile phone, a high-definition camera, or other image acquisition devices of various specifications capable of acquiring RGB two-dimensional images; for the detection of fruit targets in the canopy image, the detection model may also be implemented according to other architectures, such as Faster R-CNN, YOLO, CenterNet, etc.

[0062] 2. Training of the Fruit Pose Recognition Model

[0063] 1) Construction of the Fruit Pose Recognition Dataset. Based on the fruit target detection bounding boxes {(bx1,by1),(bx2,by2)}, further manual annotation is performed on the acquired images to construct a fruit pose recognition dataset. Specifically, the position and pose of the fruit in three-dimensional space are represented by the fruit navel point and the normal direction of the plane containing the fruit navel point. An annotation tool (e.g., ...) capable of annotating the fruit position and three-dimensional pose on two-dimensional images is developed. Figure 3 To ensure the reliability of the labels in the dataset, the annotation tool consists of two interfaces: fruit navel annotation and fruit posture annotation. In the fruit navel annotation interface, the fruit and its corresponding navel are represented by the same color. In the fruit posture annotation interface, based on its shape features, the fruit is modeled as an ellipsoid, with one end of its minor axis being the connection point between the fruit and the stem (yellow dot on the ellipsoid), and the other end being the fruit navel (blue dot on the ellipsoid). By adjusting the size and rotation direction of the ellipsoid, its projection is made to match the fruit image. At this point, the straight line formed by the fruit navel and the connection point between the fruit and the stem, passing through the center of the fruit, is the line indicating the direction of the fruit's rotation, pointing from the fruit navel to the connection point between the fruit and the stem.

[0064] 2) Fruit pose recognition model training. The pose of a fruit is determined by the fruit navel point and the unit normal vector passing through the center of the fruit and perpendicular to the plane containing the navel point. Therefore, this invention transforms the fruit pose recognition problem into two tasks: navel point detection and plane normal vector prediction. Based on a multi-task learning framework, the model design and training are completed (e.g., ...). Figure 4Using a hard parameter sharing approach, a 50-layer deep convolutional neural network is first constructed using residual modules as a shared feature extraction layer. Then, different network structures and loss functions are adopted for different tasks. For the fruit navel detection task, deconvolution operations are used to recover high-resolution feature maps, and then the fruit navel point P is predicted based on the heatmap. i The location of the fruit and the loss function are shown in Formula 1, where n represents the number of fruits in the dataset. The heatmap represents the network prediction, and H(·) represents the heatmap obtained from the ground truth. For the planar normal vector prediction task, multi-scale feature fusion plus a fully connected classification layer is used to obtain the unit normal vector of the plane containing the navel point. The loss function adopts cosine distance loss (as shown in Equation 2). Let v represent the unit normal vector predicted by the network, and v represent the labeled fruit pose. For the collaborative optimization of the model loss, a weighted fusion approach is used to balance the convergence speed of each task (as shown in Equation 3). For the fruit pose recognition model M based on a multi-task framework... estimation Input a 2D image of a fruit scaled to a fixed size, train to output the coordinates (px) of the fruit's navel point in the 2D image. P py P ) and its pose in three-dimensional space (nx, ny, nz).

[0065]

[0066]

[0067] Loss M2 = w1 Loss1 + w2 Loss2 (3)

[0068] 3) Other possible implementations. In this step, the data annotation software can adjust the fitted shape, such as a cone or other shapes, according to the type of fruit being labeled; for the representation of fruit posture, Euler angles, quadruples, or other representations may be used; the parameter sharing method in the multi-task learning framework may also be soft sharing, hierarchical sharing, or other sharing methods; when extracting two-dimensional image features, the feature extraction network may be composed of different numbers of residual modules, or it may be implemented using VGG, Inception, or other self-designed feature extraction networks; for the detection of the fruit navel point, convolution plus sampling may be used to recover the high-resolution feature map, or HRNet or other keypoint detection network designs may be used; for the prediction of the unit normal vector representing the fruit posture, different multi-scale fusion methods or other self-designed deep convolutional network structures may be used; for multi-task collaborative optimization, loss function weighting or other dynamic adjustment strategies may be used.

[0069] 3. Fruit pose recognition

[0070] 1) The camera is 0.3m to 1m away from the fruit tree to capture images of the tree canopy with fruit. The images are then analyzed using the target detection model M. detection Detect the fruit target in the 2D image to obtain the rectangular region of the fruit.

[0071] 2) Using fruit pose recognition model M estimation According to M detection Pose recognition was performed on the cropped 2D image of the fruit to obtain the coordinates (px) of the fruit navel point in the image. P py P ), confidence score s point The unit normal vector (nx, ny, nz) of the plane containing the fruit navel is also considered. For fruits with a navel confidence level of less than 0.2, the navel is considered invisible from this viewpoint, and harvesting is not possible.

[0072] 3) During the actual grasping process, the picking robot uses a depth camera to determine the two-dimensional coordinates (px) of the fruit navel point in the pixel coordinate system. P py P Convert ) to 3D coordinates in the camera coordinate system (px) C py C pz C ), such as formula 4, pz C The value of the fruit navel point in the Z direction of the camera coordinate system is represented by K, where f represents the camera intrinsic parameter. x f y Let (x, y) represent the focal lengths of the camera on the x and y axes, respectively, and (u0, v0) represent the coordinates of the origin of the image coordinate system in the pixel coordinate system. Then, the harvesting robot controls the end effector to move towards (px) with a feed angle of (nx, ny, nz). C py C pz C Move forward to complete the fruit-grabbing operation.

[0073]

[0074] 4) Other possible implementations. In the actual grasping process, the picking robot may use a binocular camera or other methods to obtain the two-dimensional coordinates (px) of the fruit navel point in the pixel coordinate system. P py P Convert ) to 3D coordinates in the camera coordinate system (px) C py C pz CWhen determining whether the fruit navel is visible, the confidence threshold can be set to other values ​​according to the actual situation. When dealing with fruits where the fruit navel is not visible, other judgment conditions can be set to determine whether the fruit can be picked from this viewpoint. Alternatively, the position of the fruit navel can be preset based on the unit normal vector representing the rotation direction, or other possible methods can be used to complete the picking operation of fruits where the fruit navel is not visible from this viewpoint.

[0075] 4. Recognition performance:

[0076] In this embodiment, a citrus pose recognition dataset containing 505 two-dimensional images was constructed, representing 1925 citrus fruits. 1577 fruits had visible navels, and 348 fruits did not. 80% of these were used as the training dataset, and 20% as the test dataset to evaluate network performance. During the recognition process, the angle between the predicted direction and the annotation method was used to measure the model's pose recognition error. For fruits with visible navels, over 80% of the fruits had a pose direction error of no more than 11.25°, and up to 97% of the fruits had a pose direction error of less than 30°. However, for citrus fruits with invisible navels, the pose deviation was larger, with an average error reaching 20°. In a simple single-tree citrus grasping simulation experiment, the grasping operation based on this invention achieved a fruit picking success rate exceeding 90%.

[0077] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0078] This invention also proposes a three-dimensional pose recognition system for fruits on trees based on a single two-dimensional image, including:

[0079] The training module is used to acquire multiple fruit tree images with fruits, and to label the fruit pose in the fruit tree image based on the fruit navel point and the normal of the plane containing the fruit navel point; and to train a fruit pose recognition model based on a deep convolutional network using the fruit tree images.

[0080] The recognition module is used by the picking robot to take on-site images of the target fruit tree, and to perform target detection on the on-site images with the fruit as the target. The detected fruit images are input into the fruit pose recognition model to obtain the target fruit pose, so as to control the end effector of the picking robot to perform task operations according to the target fruit pose and the rotation direction of the target fruit.

[0081] The aforementioned three-dimensional pose recognition system for tree fruits based on a single two-dimensional image, wherein the pose labeling specifically includes:

[0082] Label the fruit navel point in the fruit tree image. Model an ellipsoid based on the shape of the fruit. One end of the minor axis of the ellipsoid represents the connection point between the fruit and the pedicel, and the other end represents the fruit navel point. By adjusting the size and rotation direction of the ellipsoid, make its projection match the fruit in the fruit tree image. At this time, take the minor axis of the ellipsoid as the rotation direction of the fruit, pointing from the fruit navel point to the connection point between the fruit and the pedicel.

[0083] The aforementioned three-dimensional pose recognition system for fruit on a tree based on a single two-dimensional image includes the following training process:

[0084] A multi-layer deep convolutional neural network was constructed using residual modules as the feature extraction layer for this fruit pose recognition model.

[0085] A deconvolutional layer is used to restore the low-resolution feature map output by the feature extraction layer to a high-resolution feature map. This high-resolution feature map is then used to generate a heatmap to predict the location of the fruit navel point Pi in the heatmap. The loss function Loss1 is shown below, where n represents the number of fruits in the dataset. H(·) represents the heatmap generated by the network, where H(·) represents the heatmap corresponding to the labeled fruit navel point.

[0086] Using this fruit pose recognition model, multi-scale feature fusion and a fully connected classification layer are applied to obtain the unit normal vector of the plane containing the fruit navel point. The loss function Loss2 uses the cosine distance loss shown in the following formula. represents the unit normal vector predicted by the network, and v represents the labeled fruit pose;

[0087]

[0088]

[0089] Loss M2 = w1Loss1 + w2Loss2

[0090] Using the final loss function Loss M2 Train the fruit pose recognition model.

[0091] The aforementioned three-dimensional pose recognition system for fruit on a tree based on a single two-dimensional image, wherein the recognition module is used for:

[0092] During the harvesting process, the harvesting robot uses a depth camera to determine the two-dimensional coordinates (px) of the fruit's navel point in the pixel coordinate system using the following formula. P py P Convert ) to 3D coordinates in the camera coordinate system (px) C py C pz C ), pz CThis represents the value of the fruit navel point in the Z direction of the camera coordinate system, where K represents the camera intrinsic parameter, and f x f y Let (nx, ny, nz) represent the focal lengths of the camera on the x and y axes, respectively. Let (u0, v0) represent the coordinates of the origin of the image coordinate system in the pixel coordinate system. This controls the end effector to move towards (px) with a feed angle of (nx, ny, nz). C py C pz C Move forward to complete the fruit-grabbing operation;

[0093]

[0094] The present invention also proposes a storage medium for storing a program that executes any of the tree fruit three-dimensional pose recognition methods based on a single two-dimensional image.

[0095] The present invention also proposes a client for any of the aforementioned three-dimensional pose recognition systems for fruits on trees based on a single two-dimensional image.

Claims

1. A method for recognizing the three-dimensional pose of fruit on a tree based on a single two-dimensional image, characterized in that, include: Step 1: Obtain multiple fruit tree images with fruit in them. Based on the fruit navel point and the normal of the plane containing the fruit navel point in the fruit tree image, label the pose of the fruit in the fruit tree image. The fruit tree image was used to train a fruit pose recognition model based on a deep convolutional network; The specific annotation of the pose label includes: annotating the fruit navel point of the fruit in the fruit tree image, modeling an ellipsoid with the shape of the fruit, one end of the minor axis of the ellipsoid representing the connection point between the fruit and the fruit stalk, and the other end representing the fruit navel point. By adjusting the size and rotation direction of the ellipsoid, its projection matches the fruit in the fruit tree image. At this time, the minor axis of the ellipsoid is used as the rotation direction of the fruit, pointing from the fruit navel point to the connection point between the fruit and the fruit stalk. Step 2: The harvesting robot takes a picture of the target fruit tree and performs target detection on the picture with the fruit as the target. The detected fruit image is input into the fruit pose recognition model to obtain the target fruit pose. The end effector of the harvesting robot is then controlled to perform the task operation according to the target fruit pose and the rotation direction of the target fruit. The training process in step 1 includes: A multi-layer deep convolutional neural network was constructed using residual modules as the feature extraction layer for this fruit pose recognition model. A deconvolutional layer is used to restore the low-resolution feature map output by the feature extraction layer to a high-resolution feature map, and a heatmap is generated using this high-resolution feature map to predict the fruit navel point in the map. Location, loss function As shown below, This indicates the number of fruits in the dataset. This indicates that the network generates a heatmap. This represents the heat map corresponding to the marked fruit navel point; Using this fruit pose recognition model, multi-scale feature fusion and a fully connected classification layer are used to obtain the unit normal vector of the plane containing the fruit navel point. The loss function is... The cosine distance loss is expressed by the following formula. This represents the unit normal vector of the network prediction. Indicates the posture of the labeled fruit; Using the final loss function Train the fruit pose recognition model.

2. The method for recognizing the three-dimensional pose of fruit on a tree based on a single two-dimensional image as described in claim 1, characterized in that, Step 2 includes: During the harvesting process, the harvesting robot uses a depth camera to determine the two-dimensional coordinates of the fruit's navel point in the pixel coordinate system using the following formula. Transform into 3D coordinates in the camera coordinate system , Indicates the fruit navel point in the camera coordinate system The value of direction, This represents the camera's intrinsic parameters, among which , These represent the camera at... shaft and On-axis focal length, This indicates the coordinates of the image coordinate system origin in the pixel coordinate system, controlling the end effector to... The feed angle, towards Move forward and complete the fruit-grabbing operation; = .

3. A three-dimensional pose recognition system for fruit on a tree based on a single two-dimensional image, characterized in that, include: The training module is used to acquire multiple fruit tree images with fruits, and to label the pose of the fruits in the fruit tree images based on the fruit navel point and the normal of the plane containing the fruit navel point. The fruit tree image was used to train a fruit pose recognition model based on a deep convolutional network; The specific annotation of the pose label includes: annotating the fruit navel point of the fruit in the fruit tree image, modeling an ellipsoid with the shape of the fruit, one end of the minor axis of the ellipsoid representing the connection point between the fruit and the fruit stalk, and the other end representing the fruit navel point. By adjusting the size and rotation direction of the ellipsoid, its projection matches the fruit in the fruit tree image. At this time, the minor axis of the ellipsoid is used as the rotation direction of the fruit, pointing from the fruit navel point to the connection point between the fruit and the fruit stalk. The recognition module is used to capture on-site images of the target fruit tree by the picking robot, and to perform target detection on the on-site image with the fruit as the target. The detected fruit image is input into the fruit pose recognition model to obtain the target fruit pose, so as to control the end effector of the picking robot to perform task operations according to the target fruit pose and the rotation direction of the target fruit. This training module is used for: A multi-layer deep convolutional neural network was constructed using residual modules as the feature extraction layer for this fruit pose recognition model. A deconvolutional layer is used to restore the low-resolution feature map output by the feature extraction layer to a high-resolution feature map, and a heatmap is generated using this high-resolution feature map to predict the fruit navel point in the map. Location, loss function As shown below, This indicates the number of fruits in the dataset. This indicates that the network generates a heatmap. This represents the heat map corresponding to the marked fruit navel point; Using this fruit pose recognition model, multi-scale feature fusion and a fully connected classification layer are used to obtain the unit normal vector of the plane containing the fruit navel point. The loss function is... The cosine distance loss is expressed by the following formula. This represents the unit normal vector of the network prediction. Indicates the posture of the labeled fruit; Using the final loss function Train the fruit pose recognition model.

4. The three-dimensional pose recognition system for tree fruits based on a single two-dimensional image as described in claim 3, characterized in that, This identification module is used for: During the harvesting process, the harvesting robot uses a depth camera to determine the two-dimensional coordinates of the fruit's navel point in the pixel coordinate system using the following formula. Transform into 3D coordinates in the camera coordinate system , Indicates the fruit navel point in the camera coordinate system The value of direction, This represents the camera's intrinsic parameters, among which , These represent the camera at... shaft and On-axis focal length, This indicates the coordinates of the image coordinate system origin in the pixel coordinate system, controlling the end effector to... The feed angle, towards Move forward and complete the fruit-grabbing operation; = .

5. A storage medium for storing a program that executes the method for recognizing the three-dimensional pose of fruit on a tree based on a single two-dimensional image as described in claim 1 or 2.

6. A client for implementing the three-dimensional pose recognition system for fruit on a tree based on a single two-dimensional image as described in claim 3 or 4.

Images