A method and model for reasonable pose decision of mechanical arm gripper for banana picking robot vision

Abstract

The application discloses a kind of for banana picking robot vision mechanical arm gripper reasonable pose decision method and model, including definition three-dimensional space coordinate system, mechanical arm gripper reasonable pose decision;Its mechanical arm gripper reasonable pose decision process, by three-dimensional point cloud based on coordinate system CL and fruit sequence axis pose detection, mechanical arm gripper pose and fruit sequence axis pose pre-registration, based on coordinate system JB mechanical arm gripper pose calculation, decision model objective function calculation and mechanical arm gripper reasonable pose decision 5 sub-processes constitute;The application relates to the field of agricultural automation.The mechanical arm gripper reasonable pose decision method and model for banana picking robot vision in simulation test platform and field test platform run self-researched vision model, execute RGB+D integrated feature image segmentation, fruit sequence axis three-dimensional point cloud calculation, fruit sequence axis pose detection, mechanical arm gripper reasonable pose decision process, satisfy the real-time requirement of banana picking robot operation.

Description

Technical Field

[0001] This invention relates to the field of agricultural automation technology, specifically to a method and model for making reasonable pose decisions for the gripper of a robotic arm used in banana harvesting robots. Background Technology

[0002] The banana harvesting robot under development consists of a driving chassis, a six-degree-of-freedom robotic arm (including a fruit cluster gripper and a cutter at the end of the robotic arm), a fruit cluster hanging and storage bin, and a control system. It employs a terrain-adaptive balancing driving mode for harvesting banana clusters. The control system comprises a microprocessor, a terrain-adaptive balancing system, a natural light monitoring system, a wind monitoring system, a fruit cluster search and tracking system, and a binocular depth camera vision system. Before performing the banana cluster harvesting process, to minimize waiting time and ensure timely and smooth execution of gripping, cutting the fruit cluster axis, and hanging and transferring, the robot's vision system needs to perform a pre-registration task between the robotic arm gripper pose and the fruit cluster axis pose. Before performing pre-registration, a reasonable pose decision for the robotic arm gripper needs to be made, and before making that decision, the pose of the fruit cluster axis must first be determined. To address the challenges of identifying the inflorescence axis during the optimal banana harvesting period, where the inflorescence axis, other targets, and background are all similarly green, resulting in difficulty in identifying the inflorescence axis; the small size of the inflorescence axis image leading to sparse data and decreased perception accuracy; and the impact of complex, dynamic, and unstructured harvesting environments on image quality, existing vision technologies for banana harvesting suffer from insufficient perception accuracy, poor robustness, and low efficiency.

[0003] Therefore, mimicking the visual behavior of banana farmers harvesting bananas in commercial banana cultivation, this paper proposes a method and model for determining the reasonable pose of a robotic arm gripper based on 3D point cloud fruit sequence axis pose detection. Simultaneously, it explores an RGB+D integrated feature image segmentation algorithm to develop the vision system of the banana harvesting robot and improve its perception accuracy, robustness, and real-time performance. Reasonable pose refers to the range of poses that the robotic arm gripper's movement must meet temporal, spatial, and mechanical requirements during the sequential processes of fruit sequence axis detection, pre-registration between the robotic arm gripper and the fruit sequence axis, clamping and cutting, and hanging and transporting. Summary of the Invention

[0004] This invention is achieved through the following technical solution: a method for determining the reasonable pose of a robotic arm gripper for a banana-picking robot, comprising the following steps:

[0005] Step A1: Define a three-dimensional spatial coordinate system;

[0006] Step A2: Decision on the appropriate pose of the robotic arm gripper;

[0007] The process of making a reasonable pose decision for the robotic arm gripper consists of five sub-processes: 3D point cloud and fruit sequence axis pose detection based on coordinate system CL, pre-registration of robotic arm gripper pose and fruit sequence axis pose, robotic arm gripper pose calculation based on coordinate system JB, calculation of objective function of decision model, and decision of reasonable pose for robotic arm gripper. It is used to describe the decision method of the PCA+NLP robotic arm gripper reasonable pose decision model operation process.

[0008] Specifically, the 3D point cloud and fruit inflorescence axis pose detection based on coordinate system CL employs the RGB+D integrated feature image segmentation method to extract the fruit inflorescence axis, processing image types and scenes; the 3D point cloud of the fruit inflorescence axis based on coordinate system CL is calculated based on the fruit inflorescence axis depth data and camera parameters, and the result is denoted as... ;

[0009] The pre-registration of the robotic arm gripper pose and the fruit sequence axis pose uses the origin of coordinate system JZ. exist The origin of coordinate system GZ is pre-aligned on the plane. A smaller distance The strategy achieves pre-registration;

[0010] The JB-based robotic arm gripper pose calculation adopts a vision-before-action working mode, and the camera and robotic arm are designed to belong to two independent mechanical and control systems.

[0011] The objective function of the decision model is selected as the occlusion rate of fruit branches and leaves as the objective function for the reasonable pose decision.

[0012] The optimal pose decision for the robotic arm gripper is determined by the origin of the robotic arm gripper coordinate system JZ. Pre-approximation to the origin of the fruit order axis coordinate system GZ A preset distance Waiting for pose adjustment.

[0013] Preferably, the pose detection of the 3D point cloud and fruit order axis based on coordinate system CL specifically includes:

[0014] Principal component analysis (PCA) was used to determine the eigenvalue matrix of the three-dimensional point cloud of the fruit axis. and eigenvector matrix subscript Characterizes the transformation from the original coordinate system CL to the target coordinate system GZ, or observes the coordinate system GZ based on the coordinate system CL;

[0015] eigenvalue matrix The elements satisfy the relation These correspond to the three coordinate axes of coordinate system GZ. and eigenvector matrix Each column;

[0016] Three-dimensional point cloud of fruit sequence axis Calculate coordinate system transformation Translation vector and rotation matrix They are used to characterize the fruit order axis pose based on the coordinate system CL;

[0017] coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction;

[0018] coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction;

[0019] coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction.

[0020] Preferably, the pre-registration of the robotic arm gripper pose and the fruit sequence axis pose specifically involves:

[0021] Using the origin of coordinate system JZ exist The origin of coordinate system GZ is pre-aligned on the plane. A smaller distance Strategies;

[0022] Therefore, the coordinate axes of the JZ coordinate system of the robotic arm gripper are... Always parallel to the coordinate axes of the fruit order axis coordinate system GZ The translation vector from coordinate system GZ to coordinate system JZ is The rotation matrix is , The tail of the arrow is at the origin of coordinate system GZ. The vector tip is at the origin of coordinate system JZ. ;

[0023] Command Vector coordinate axes relative to coordinate system JZ The grippers are collinear and point in the same direction, ensuring that the opening of the robotic arm's gripper always faces the fruit sequence axis, and the center of the gripper opening is always aligned with the centroid of the fruit sequence axis to be gripped. The only change is in length expansion or contraction (vector mode). (variation) and around the coordinate axis rotation angle .

[0024] Preferably, the calculation of the robotic arm gripper pose based on coordinate system JB specifically involves:

[0025] The camera coordinate system CL is designed to have three degrees of freedom: based on the coordinate system JB and around the coordinate axes. Horizontal rotation angle , around coordinate axes pitch and rotation angle And the lifting height h, the coordinates of the origin of coordinate system CL based on coordinate system JB are: ;

[0026] Let coordinate system transformation The translation vector is The rotation matrix is Let coordinate system CL be about the coordinate axes The rotation matrix is , around the coordinate axis The rotation matrix is Then the following relationship exists:

[0027] , ;

[0028] Coordinate system transformation The total rotation matrix of the "ZYX combined rotation" for:

[0029] ;

[0030] Perform chain coordinate system transformation The pose of the robotic arm gripper based on the coordinate system JB is obtained, that is, the translation vector. and rotation matrix .

[0031] Preferably, the objective function of the decision model is specifically calculated using the following formula:

[0032] ;

[0033] Among them, the Specifically, it refers to the shading rate of fruit cluster branches and leaves. The area of ​​the incomplete fruit cluster target frame when the fruit cluster is obscured by branches and leaves is the visible part of the fruit cluster. These are the areas of the complete fruit cluster target boxes where the entire fruit cluster is visible when there are no branches or leaves obstructing the view; they are obtained through image detection.

[0034] The variable Let be the camera pose, where the camera azimuth angle is . Causal order tracking requires independent regulation; h is a non-regulated variable affecting the shading rate of branches and leaves; and camera pitch angle and camera height are also considered. These are all the controllable variables that affect the shading rate of fruiting branches and leaves;

[0035] The variable d represents the distance between the robot and the plant, which is naturally generated during banana picking and is a non-regulated variable that affects the shading rate of branches and leaves.

[0036] Preferably, the decision on the reasonable pose of the robotic arm gripper specifically involves:

[0037] distance The upper limit threshold is To avoid excessively long idle time during subsequent processes, the lower threshold is: To avoid damage caused by the gripper opening being too close to the fruit cluster axis, and considering the time, space, and mechanical requirements for gripping, cutting, hanging, and transporting the fruit cluster axis, the reasonable rotation angle range of the robotic arm gripper coordinate system JZ based on coordinate system JB is derived as follows: Let represent a sector representing the movement of the robotic arm gripper aligned with the fruit sequence axis above the horizontal plane. Transform the original coordinate system JB to the target coordinate system JZ, and rotate the coordinate system JZ around the coordinate axes. If the rotation angle is , then:

[0038] .

[0039] Therefore, a reasonable rotation angle range of the robotic arm gripper around the coordinate axis is derived, as shown in the above formula. In summary, the goal of the reasonable pose decision of the robotic arm gripper is twofold: firstly, to minimize the occlusion rate of fruit cluster branches and leaves; and secondly, to meet the spatial and mechanical requirements for gripping, cutting, hanging, and transporting. Therefore, principal component analysis (PCA) is used to detect the pose of the fruit cluster axis, with the occlusion rate of fruit cluster branches and leaves as the objective function. Nonlinear programming (NLP) is used to determine the camera pitch angle and camera height, and then the translation vector and rotation matrix that meet the reasonable pose requirements of the robotic arm gripper are calculated.

[0040] A reasonable pose decision model for the robotic arm gripper used in banana picking robot vision is proposed. The pose of the fruit cluster axis is determined by principal component analysis (PCA). Within the safe distance range for picking, the rotation angle of the variable vector around the coordinate axis of the fruit cluster axis GZ is adjusted. The coordinate axes of coordinate system JZ and coordinate system GZ are always parallel, the vector is always collinear with the coordinate axis of coordinate system JZ, and the vector is always perpendicular to the coordinate axis of coordinate system GZ. Nonlinear programming (NLP) is used until the occlusion rate of fruit cluster branches and leaves is minimized. At this point, the pose of the robotic arm gripper is within the reasonable pose range, and the pose of the robotic arm gripper and the pose of the fruit cluster axis are pre-registered.

[0041] The specific operational formula for the decision-making model is as follows:

[0042] ;

[0043] in, and The translation vector and rotation matrix represent the transformation from the original coordinate system JB to the target coordinate system JZ, respectively, characterizing the pose of the robotic arm gripper based on coordinate system JB. The subscripts are used to represent these transformations. Characterizes the transformation from coordinate system JB to coordinate system JZ;

[0044] and The pose of the binocular depth camera. and For the fruit sequence axis pose of the 3D point cloud, and For the position and orientation variables of the robotic arm gripper;

[0045] These are the decision variables in the decision-making model. It is the objective function of the decision model, where Transformation and These are the coordinate axes of coordinate system CL around coordinate system JB. and coordinate axes The rotation angle, h is the camera height;

[0046] ;

[0047] in, Here, represents the upper and lower limits of the height threshold, and d represents the distance between the robot and the banana plant. These are the upper and lower limits of the distance threshold. The coordinates of the origin of the camera coordinate system CL on coordinate system JB;

[0048] Let coordinate system GZ be about coordinate system CL. rotation angle, Let coordinate system JZ be about coordinate system GZ. The rotation angle;

[0049] The pre-registration distance is the distance from the origin of the fruit sequence axis coordinate system GZ to the origin of the robot arm gripper coordinate system JZ, or the translation vector of the GZ→JZ transformation. The model;

[0050] and These are the JB→CL transformation and the coordinate system CL around the coordinate axes, respectively. and rotation matrix;

[0051] Therefore, the decision-making problem of the reasonable pose of the robotic arm gripper can be summarized as "PCA+NLP robotic arm gripper reasonable pose decision-making model".

[0052] This invention provides a method and model for determining the appropriate pose of a robotic arm gripper in a banana-picking robot's vision system. It offers the following advantages:

[0053] This paper presents a method and model for determining the optimal pose of the robotic arm gripper in a banana-harvesting robot. A simulation platform was built using a 13th generation Intel® Core i9-13900K processor (3GHz, 24 physical cores and 32 threads), an NVIDIA GeForce RTX 4090 graphics card, and an Ubuntu 18.04 operating system. The platform was equipped with the CUDA 11.1.4 parallel computing platform and programming model, the OpenCV 4.8.0 cross-platform computer vision and machine learning software library, and the PyTorch 2.0.1 open-source deep learning Python framework. A field test platform was constructed using an NVIDIA OrinNX microprocessor (an AI-based supercomputing platform) and an Intel RealSense 455 RGB+D binocular depth camera. The self-developed vision model was run on both the simulation and field test platforms to perform processes such as RGB+D integrated feature image segmentation, 3D point cloud computing of the fruit cluster axis, fruit cluster axis pose detection, and optimal pose determination for the robotic arm gripper. Simulation tests showed a precision of 0.95, a recall of 0.98, a mean average precision (mAP) of 0.97, and an inference time of 20 ms. Field tests showed a precision of 0.95, a recall of 0.97, an mAP of 0.96, and an inference time of 150.3 ms, meeting the real-time requirements of banana harvesting robot operations. Attached Figure Description

[0054] Figure 1This is a schematic diagram of the depth image, pseudo-color depth image, and RGB image for visual detection of fruit axis pose according to the present invention.

[0055] Figure 2 This is a schematic diagram of the three-dimensional point cloud and the three-dimensional point cloud pose of the fruit order axis based on the coordinate system CL of the present invention;

[0056] Figure 3 This is a schematic diagram of the target frame of the banana inflorescence, fruit, and inflorescence axis of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Please see Figure 1-3 The present invention provides a technical solution:

[0059] A method for determining the appropriate pose of a robotic arm gripper for vision in a banana-picking robot includes the following steps:

[0060] Step A1: Define a three-dimensional spatial coordinate system;

[0061] Step A2: Decision on the appropriate pose of the robotic arm gripper;

[0062] The process of making a reasonable pose decision for the robotic arm gripper consists of five sub-processes: 3D point cloud and fruit sequence axis pose detection based on coordinate system CL, pre-registration of robotic arm gripper pose and fruit sequence axis pose, robotic arm gripper pose calculation based on coordinate system JB, calculation of objective function of decision model, and decision of reasonable pose for robotic arm gripper. It is used to describe the decision method of the PCA+NLP robotic arm gripper reasonable pose decision model operation process.

[0063] Specifically, the 3D point cloud and fruit inflorescence axis pose detection based on coordinate system CL employs the RGB+D integrated feature image segmentation method to extract the fruit inflorescence axis, processing image types and scenes; the 3D point cloud of the fruit inflorescence axis based on coordinate system CL is calculated based on the fruit inflorescence axis depth data and camera parameters, and the result is denoted as... ;

[0064] Principal component analysis (PCA) was used to determine the eigenvalue matrix of the three-dimensional point cloud of the fruit axis. and eigenvector matrix subscript Characterizes the transformation from the original coordinate system CL to the target coordinate system GZ, or observes the coordinate system GZ based on the coordinate system CL;

[0065] eigenvalue matrix The elements satisfy the relation These correspond to the three coordinate axes of coordinate system GZ. and eigenvector matrix Each column;

[0066] Three-dimensional point cloud of fruit sequence axis Calculate coordinate system transformation Translation vector and rotation matrix They are used to characterize the fruit order axis pose based on the coordinate system CL;

[0067] coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction;

[0068] coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction;

[0069] coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction.

[0070] The pre-registration of the robotic arm gripper pose and the fruit sequence axis pose uses the origin of coordinate system JZ. exist The origin of coordinate system GZ is pre-aligned on the plane. A smaller distance The strategy is to achieve pre-registration; the origin of coordinate system JZ is used. exist The origin of coordinate system GZ is pre-aligned on the plane. A smaller distance Strategies;

[0071] Therefore, the coordinate axes of the JZ coordinate system of the robotic arm gripper are... Always parallel to the coordinate axes of the fruit order axis coordinate system GZ The translation vector from coordinate system GZ to coordinate system JZ is The rotation matrix is , The tail of the arrow is at the origin of coordinate system GZ. The vector tip is at the origin of coordinate system JZ. ;

[0072] Command Vector coordinate axes relative to coordinate system JZ The grippers are collinear and point in the same direction, ensuring that the opening of the robotic arm's gripper always faces the fruit sequence axis, and the center of the gripper opening is always aligned with the centroid of the fruit sequence axis to be gripped. The only change is in length expansion or contraction (vector mode). (variation) and around the coordinate axis rotation angle .

[0073] The robotic arm gripper pose calculation based on coordinate system JB adopts a vision-first, action-first working mode. The camera and robotic arm are designed to belong to two independent mechanical and control systems. The camera coordinate system CL is designed to have three degrees of freedom, namely, based on coordinate system JB and around the coordinate axes. Horizontal rotation angle , around coordinate axes pitch and rotation angle And the lifting height h, the coordinates of the origin of coordinate system CL based on coordinate system JB are: ;

[0074] Let coordinate system transformation The translation vector is The rotation matrix is Let coordinate system CL be about the coordinate axes The rotation matrix is , around the coordinate axis The rotation matrix is Then the following relationship exists:

[0075] , ;

[0076] Coordinate system transformation The total rotation matrix of the "ZYX combined rotation" for:

[0077] ;

[0078] Perform chain coordinate system transformation The pose of the robotic arm gripper based on the coordinate system JB is obtained, that is, the translation vector. and rotation matrix .

[0079] The objective function of the decision model is selected as the occlusion rate of fruiting branches and leaves as the objective function for determining the reasonable pose; the specific formula for calculating the objective function of the decision model is as follows:

[0080] ;

[0081] Among them, the Specifically, it refers to the shading rate of fruit cluster branches and leaves. The area of ​​the incomplete fruit cluster target frame when the fruit cluster is obscured by branches and leaves is the visible part of the fruit cluster. These are the areas of the complete fruit cluster target boxes where the entire fruit cluster is visible when there are no branches or leaves obstructing the view; they are obtained through image detection.

[0082] The variable Let be the camera pose, where the camera azimuth angle is . Causal order tracking requires independent regulation; h is a non-regulated variable affecting the shading rate of branches and leaves; and camera pitch angle and camera height are also considered. These are all the controllable variables that affect the shading rate of fruiting branches and leaves;

[0083] The variable d represents the distance between the robot and the plant, which is naturally generated during banana picking and is a non-regulated variable that affects the shading rate of branches and leaves.

[0084] The optimal pose decision for the robotic arm gripper is determined by the origin of the robotic arm gripper coordinate system JZ. Pre-approximation to the origin of the fruit order axis coordinate system GZ A preset distance Waiting for pose adjustment;

[0085] distance The upper limit threshold is To avoid excessively long idle time during subsequent processes, the lower threshold is: To avoid damage caused by the gripper opening being too close to the fruit cluster axis, and considering the time, space, and mechanical requirements for gripping, cutting, hanging, and transporting the fruit cluster axis, the reasonable rotation angle range of the robotic arm gripper coordinate system JZ based on coordinate system JB is derived as follows: Let represent a sector representing the movement of the robotic arm gripper aligned with the fruit sequence axis above the horizontal plane. Transform the original coordinate system JB to the target coordinate system JZ, and rotate the coordinate system JZ around the coordinate axes. If the rotation angle is , then:

[0086] .

[0087] Therefore, a reasonable rotation angle range of the robotic arm gripper around the coordinate axis is derived, as shown in the above formula. In summary, the goal of the reasonable pose decision of the robotic arm gripper is twofold: firstly, to minimize the occlusion rate of fruit cluster branches and leaves; and secondly, to meet the spatial and mechanical requirements for gripping, cutting, hanging, and transporting. Therefore, principal component analysis (PCA) is used to detect the pose of the fruit cluster axis, with the occlusion rate of fruit cluster branches and leaves as the objective function. Nonlinear programming (NLP) is used to determine the camera pitch angle and camera height, and then the translation vector and rotation matrix that meet the reasonable pose requirements of the robotic arm gripper are calculated.

[0088] A reasonable pose decision model for the robotic arm gripper used in banana picking robot vision is proposed. The pose of the fruit cluster axis is determined by principal component analysis (PCA). Within the safe distance range for picking, the rotation angle of the variable vector around the coordinate axis of the fruit cluster axis GZ is adjusted. The coordinate axes of coordinate system JZ and coordinate system GZ are always parallel, the vector is always collinear with the coordinate axis of coordinate system JZ, and the vector is always perpendicular to the coordinate axis of coordinate system GZ. Nonlinear programming (NLP) is used until the occlusion rate of fruit cluster branches and leaves is minimized. At this point, the pose of the robotic arm gripper is within the reasonable pose range, and the pose of the robotic arm gripper and the pose of the fruit cluster axis are pre-registered.

[0089] The specific operational formula for the decision-making model is as follows:

[0090] ;

[0091] in, and The translation vector and rotation matrix represent the transformation from the original coordinate system JB to the target coordinate system JZ, respectively, characterizing the pose of the robotic arm gripper based on coordinate system JB. The subscripts are used to represent these transformations. Characterizes the transformation from coordinate system JB to coordinate system JZ;

[0092] and The pose of the binocular depth camera. and For the fruit sequence axis pose of the 3D point cloud, and For the position and orientation variables of the robotic arm gripper;

[0093] These are the decision variables in the decision-making model. It is the objective function of the decision model, where Transformation and These are the coordinate axes of coordinate system CL around coordinate system JB. and coordinate axes The rotation angle, h is the camera height;

[0094] ;

[0095] in, Here, represents the upper and lower limits of the height threshold, and d represents the distance between the robot and the banana plant. These are the upper and lower limits of the distance threshold. The coordinates of the origin of the camera coordinate system CL on coordinate system JB;

[0096] Let coordinate system GZ be about coordinate system CL. rotation angle, Let coordinate system JZ be about coordinate system GZ. The rotation angle;

[0097] The pre-registration distance is the distance from the origin of the fruit sequence axis coordinate system GZ to the origin of the robot arm gripper coordinate system JZ, or the translation vector of the GZ→JZ transformation. The model;

[0098] and These are the JB→CL transformation and the coordinate system CL around the coordinate axes, respectively. and rotation matrix;

[0099] Therefore, the decision-making problem of the reasonable pose of the robotic arm gripper can be summarized as "PCA+NLP robotic arm gripper reasonable pose decision-making model".

[0100] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method and model for determining the reasonable pose of a robotic arm gripper used in a banana-picking robot, characterized in that, Includes the following steps: Step A1: Define a three-dimensional spatial coordinate system; Step A2: Decision on the appropriate pose of the robotic arm gripper; The process of making a reasonable pose decision for the robotic arm gripper consists of five sub-processes: 3D point cloud and fruit sequence axis pose detection based on coordinate system CL, pre-registration of robotic arm gripper pose and fruit sequence axis pose, robotic arm gripper pose calculation based on coordinate system JB, calculation of objective function of decision model, and decision of reasonable pose for robotic arm gripper. Specifically, the 3D point cloud and fruit inflorescence axis pose detection based on coordinate system CL employs the RGB+D integrated feature image segmentation method to extract the fruit inflorescence axis, processing image types and scenes; the 3D point cloud of the fruit inflorescence axis based on coordinate system CL is calculated based on the fruit inflorescence axis depth data and camera parameters, and the result is denoted as... ; The pre-registration of the robotic arm gripper pose and the fruit sequence axis pose uses the origin of coordinate system JZ. exist The origin of coordinate system GZ is pre-aligned on the plane. A smaller distance The strategy achieves pre-registration; The JB-based robotic arm gripper pose calculation adopts a vision-before-action working mode, and the camera and robotic arm are designed to belong to two independent mechanical and control systems. The objective function of the decision model is selected as the occlusion rate of fruit branches and leaves as the objective function for the reasonable pose decision. The optimal pose decision for the robotic arm gripper is determined by the origin of the robotic arm gripper coordinate system JZ. Pre-approximation to the origin of the fruit order axis coordinate system GZ A preset distance Waiting for pose adjustment.

2. The method and model for determining the reasonable pose of a robotic arm gripper for vision in a banana-picking robot according to claim 1, characterized in that: The specific steps of the 3D point cloud and fruit order axis pose detection based on coordinate system CL are as follows: Principal component analysis (PCA) was used to determine the eigenvalue matrix of the three-dimensional point cloud of the fruit axis. and eigenvector matrix subscript Characterizes the transformation from the original coordinate system CL to the target coordinate system GZ, or observes the coordinate system GZ based on the coordinate system CL; eigenvalue matrix The elements satisfy the relation These correspond to the three coordinate axes of coordinate system GZ. and eigenvector matrix Each column; Three-dimensional point cloud of fruit sequence axis Calculate coordinate system transformation Translation vector and rotation matrix They are used to characterize the fruit order axis pose based on the coordinate system CL; coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction; coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction; coordinate axes The eigenvalues ​​and corresponding eigenvectors Based on coordinate system CL and originating from the centroid Characterizing coordinate axes The unit vector of direction.

3. The method and model for determining the reasonable pose of a robotic arm gripper for vision in a banana-picking robot according to claim 1, characterized in that: The pre-registration of the robotic arm gripper pose and the fruit sequence axis pose is specifically as follows: Using the origin of coordinate system JZ exist The origin of coordinate system GZ is pre-aligned on the plane. A smaller distance Strategies; Therefore, the coordinate axes of the JZ coordinate system of the robotic arm gripper are... Always parallel to the coordinate axes of the fruit order axis coordinate system GZ The translation vector from coordinate system GZ to coordinate system JZ is The rotation matrix is , The tail of the arrow is at the origin of coordinate system GZ. The vector tip is at the origin of coordinate system JZ. ; Command Vector coordinate axes relative to coordinate system JZ The grippers are collinear and point in the same direction, ensuring that the opening of the robotic arm's gripper always faces the fruit sequence axis, and the center of the gripper opening is always aligned with the centroid of the fruit sequence axis to be gripped. The only change is in length expansion or contraction (vector mode). (variation) and around the coordinate axis rotation angle .

4. The method and model for determining the reasonable pose of a robotic arm gripper for vision in a banana-picking robot according to claim 1, characterized in that: The specific calculation of the robotic arm gripper pose based on the JB coordinate system is as follows: The camera coordinate system CL is designed to have three degrees of freedom: based on the coordinate system JB and around the coordinate axes. Horizontal rotation angle , around coordinate axes pitch and rotation angle And the lifting height h, the coordinates of the origin of coordinate system CL based on coordinate system JB are: ; Let coordinate system transformation The translation vector is The rotation matrix is Let coordinate system CL be about the coordinate axes The rotation matrix is , around the coordinate axis The rotation matrix is Then the following relationship exists: ， ； Coordinate system transformation The total rotation matrix of the "ZYX combined rotation" for: ； Perform chain coordinate system transformation The pose of the robotic arm gripper based on the coordinate system JB is obtained, that is, the translation vector. and rotation matrix .

5. The method and model for determining the reasonable pose of a robotic arm gripper for vision in a banana-picking robot according to claim 1, characterized in that, The specific formula for calculating the objective function of the decision model is as follows: ； Among them, the Specifically, it refers to the shading rate of fruit cluster branches and leaves. The area of ​​the incomplete fruit cluster target frame when the fruit cluster is obscured by branches and leaves is the visible part of the fruit cluster. It is the area of ​​the complete fruit cluster target frame where the entire fruit cluster is visible when there are no branches or leaves obstructing the view; The variable Let be the camera pose, where the camera azimuth angle is . Causal order independent regulation, h is a non-regulated variable affecting the shading rate of branches and leaves, camera pitch angle and camera height These are all the adjustable variables that affect the shading rate of fruiting branches and leaves, where variable d is the distance between the robot and the plant.

6. The method and model for determining the reasonable pose of a robotic arm gripper for vision in a banana-picking robot according to claim 1, characterized in that, The specific method for determining the optimal pose of the robotic arm gripper is as follows: distance The upper limit threshold is To avoid excessively long idle time during subsequent processes, the lower threshold is: To avoid damage caused by the gripper opening being too close to the fruit cluster axis; Taking into account the time, space, and mechanical requirements for clamping, cutting, hanging, and transferring the fruit sequence axis, the reasonable rotation angle range of the robotic arm gripper coordinate system JZ based on coordinate system JB is derived as follows: , representing a sector of the robotic arm gripper moving above the horizontal plane in alignment with the fruit sequence axis; set up Transform the original coordinate system JB to the target coordinate system JZ, and rotate the coordinate system JZ around the coordinate axes. If the rotation angle is , then: 。 7. A decision model for the rational pose of a robotic arm gripper used in a banana-picking robot, characterized in that: Principal component analysis (PCA) is used to determine the pose of the fruit cluster axis. Within the safe distance for picking, the rotation angle of the variable vector model around the coordinate axis of the fruit cluster axis GZ is adjusted, and the coordinate axes of coordinate system JZ are always parallel to the coordinate axes of coordinate system GZ. The vector is always collinear with the coordinate axes of coordinate system JZ and always perpendicular to the coordinate axes of coordinate system GZ. Nonlinear programming (NLP) is used until the occlusion rate of fruit cluster branches and leaves is minimized. At this point, the pose of the robotic arm gripper is within a reasonable pose range, and the pose of the robotic arm gripper and the pose of the fruit cluster axis are pre-registered.

Images