Visual touch fusion safe motion positioning method and device in canopy of rigid-flexible hybrid mechanical arm

Abstract

The invention discloses a method and a device for positioning in-canopy visual touch fusion safe movement of a rigid-flexible hybrid mechanical arm. The method comprises the following steps of: dividing various heterogeneous objects in a fruit tree canopy into avoidance targets and compliance targets; for an avoidance target, the dynamic characteristics of the avoidance target are used as obstacles, and a dynamic path is planned, so that the rigid-flexible hybrid mechanical arm strictly avoids the target; for a compliance target, the rigid-flexible hybrid mechanical arm goes deep into a fruit tree canopy and makes contact with the compliance target, integrated joint tension / tension obtained by tension / tension sensors on the flexible arm and corresponding integrated joint theoretical tension / tension are used for making difference, error signals are generated, feedback adjustment is conducted on the error signals of the driving rope, and then the compliance target is obtained by controlling a tension / tension disc type motor. And the rope length of the driving rope is dynamically adjusted, the flexible arm is driven to generate a dynamic bending deformation angle, and the flexible and smooth movement of the rigid-flexible hybrid mechanical arm when the rigid-flexible hybrid mechanical arm makes contact with a compliant target is achieved. The method is widely suitable for autonomous continuous planning of the rigid-flexible hybrid mechanical arm in various fruit tree complex canopy scenes.

Description

Technical Field

[0001] The present invention relates to the field of agricultural robots, and in particular to a method and a device for safe motion positioning of a rigid-flexible hybrid robotic arm with visual-tactile fusion within a canopy. Background Art

[0002] As one of the world's largest fruit producers, my country ranks first in the world in the production of fruits such as apples and pears. Rigid-flexible hybrid robotic arms have the characteristics of high-speed displacement and flexible passage through narrow and confined areas due to their rigid structure and flexible continuum hybrid structure. However, when picking fruits in the canopy of fruit trees, rigid-flexible hybrid robotic arms often face complex environmental interference caused by various heterogeneous objects. A single rigid link robotic arm will cause collision and picking failure if it comes into contact with large hard branches. The continuous control method of the flexible arm is complex and the movement is relatively slow, resulting in low harvesting efficiency and significant fluctuations in movement accuracy.

[0003] Existing harvesting robot products and test prototypes, as well as rigid-flexible hybrid robotic arms, are all used in relatively standard and structured industrial scenes or interference-free indoor simulation links. However, they have shortcomings such as poor adaptability to scene structures in the canopy of fruit trees and easy interference from the inside of the canopy. When the rigid-flexible hybrid robotic arm goes deep into the canopy of fruit trees, it must avoid hard obstacles and needs to flexibly adapt to soft obstacles, which leads to complex harvesting strategies for multiple types of heterogeneous objects, dynamic re-planning of detour paths, and cumbersome precision compensation control. Summary of the invention

[0004] In view of the shortcomings in the prior art, the present invention provides a method and device for safe motion positioning of a rigid-flexible hybrid robotic arm through visual-tactile fusion within the canopy, which effectively solves the problem of difficult motion planning of the rigid-flexible hybrid robotic arm under multiple types of heterogeneous objects in the canopy of complex fruit trees, and realizes autonomous fruit picking of the rigid-flexible hybrid robotic arm in the canopy of complex fruit trees.

[0005] The present invention achieves the above technical objectives through the following technical means.

[0006] A method for safe motion positioning of a rigid-flexible hybrid manipulator using visual-tactile fusion within the canopy:

[0007] Obtain the canopy information of fruit trees, and divide the multiple heterogeneous objects in the canopy of fruit trees into avoidance targets and compliance targets according to the impact risk intensity and the probability of impact occurrence;

[0008] For the avoidance target, the rigid-flexible hybrid robotic arm avoids it in the following way: the fruit tree canopy information is input into the fruit tree canopy avoidance target semantic segmentation model, the avoidance target plane coordinates in the fruit tree canopy real-time image information are output, and the avoidance target space point set is obtained in combination with the depth value of the real-time image information; the avoidance target space point set is used as the root node, and the field of view space body area is recursively segmented successively until the space volume contained in each node reaches the preset minimum volume, and the avoidance target dynamic features based on the field of view space body area are constructed by traversing the avoidance target space point set; the avoidance target dynamic features are used as obstacles, and the flexible arm avoids the target with the deformation curvature τ c , avoid bending deformation angle θ c As posture constraints, dynamic path planning is performed with the initial position of the end effector as the path starting planning point and the fruit picking position as the path ending planning point;

[0009] For the compliant target, the rigid-flexible hybrid manipulator avoids the target and then contacts the compliant target. Each pull / tension sensor on the flexible arm obtains the corresponding integrated joint pull / tension T in real time. n , T n The corresponding integrated joint theoretical pull / tension T L The error is ΔT m , based on the ΔT m Adjust the pull / tension of the driving rope, and then dynamically adjust the rope length of the driving rope to form a dynamic bending deformation angle of the flexible arm after contacting the compliant target; after the rigid-flexible hybrid robotic arm contacts the compliant target, the dynamic bending deformation angle of the flexible arm is compensated, and the flexible arm runs at the compensated angle, driving the end effector to grasp the fruit target.

[0010] Furthermore, the avoidance target and compliance target are determined by:

[0011] The trunk, main branches and secondary main branches with high impact risk intensity of τ1 and impact probability of δ1 in the fruit tree canopy picking area are set as vision-based avoidance targets, and the fruiting mother branches, fruiting branches and nutrient branches with impact risk intensity of τ2 and impact probability of δ2 in the fruit tree canopy picking area are set as tactile-based compliance targets, and: τ1<τ2, δ2<δ1.

[0012] Furthermore, the avoidance deformation curvature τ c and avoidance bending deformation angle θ c Respectively meet:

[0013]

[0014] θ c =τ c *L soft

[0015] Among them, Wst W is the width of the minimum space volume when the space volume contained by each node reaches the preset minimum volume. s is the safe distance between the flexible arm and the avoidance target, L soft is the length of the flexible arm.

[0016] Furthermore, the dynamic bending deformation angle of the compensation flexible arm is specifically: a force-position model C of the rigid-flexible hybrid manipulator is established by a linear elastic regression model k , calculate the real-time position deformation Δx of the end effector when the rigid-flexible hybrid manipulator contacts the compliance target, obtain the end position compensation of the rigid-flexible hybrid manipulator, and determine the compensation deflection θ of the flexible arm integrated joint k , the flexible arm integrated joint compensation deflection θ k As the control instruction of the rigid-flexible hybrid robot arm, it compensates the dynamic bending deformation angle of the flexible arm.

[0017] Furthermore, the force-position model C of the rigid-flexible hybrid manipulator k For: C k :x actual =f(θ L )+f(θ0)+Δx, the flexible arm integrated joint compensates the deflection θ k Satisfies: Δx=f(θ k )=K -1 *F real ; Among them, f(θ L ) represents the position of the connection between the rigid arm and the flexible arm, x actual represents the actual position of the end effector, f(θ0) is the theoretical position of the end effector of the flexible arm, and f(θ k ) is the end position compensation of the rigid-flexible hybrid manipulator, K represents the stiffness matrix of the flexible arm, and F real represents the external stress acting on the flexible arm due to the compliance target.

[0018] Furthermore, by f(θ k )=K -1 *F real Derived θ k , as the kinematic input parameters of the rigid-flexible hybrid robot arm, generate the control instructions of the rigid-flexible hybrid robot arm, drive the flexible arm posture and the rope length change of the pulling / tensioning driving rope, and compensate the dynamic bending deformation angle of the flexible arm.

[0019] Furthermore, the vertex of the dynamic bending deformation angle of the flexible arm is the collision orientation, and the collision orientation is obtained by: the tension sensor on the flexible arm detects the tension change ΔT of the tension driving rope in real time i , combined with the rope direction vector of the pull / tension drive rope Calculate the touch force Through the force balance and the geometric relationship of the flexible arm, the change of the center of mass position of the flexible arm ΔC is calculated soft , establish the positional relationship between the center of mass of the flexible arm and the collision point, that is, the collision orientation.

[0020] Furthermore, the touch external force satisfy: The change in the center of mass position of the flexible arm ΔC soft satisfy: The position relationship between the center of mass of the flexible arm and the collision point is: c (x c ,y c )=P0(x0,y0)+ΔC soft , where n is the total number of integrated joints of the flexible arm, P0(x0,y0) is the initial point of the center of mass of the flexible arm, and k i It is the adjustment factor for pulling / tensioning the drive rope.

[0021] Furthermore, dynamic path planning is performed to obtain the desired position of the end effector, and the error between the actual position and the desired position of the end effector is dynamically adjusted based on the PID controller to ensure the motion accuracy of the flexible arm.

[0022] A canopy visual-tactile fusion safe motion positioning device for a rigid-flexible hybrid manipulator comprises a "hand" deep vision module, a pull / tension disc motor, a pull / tension driving rope, a pull / tension sensor and a visual servo controller; the "hand" deep vision module is installed on the upper part of the end effector of the flexible arm; there are k pull / tension disc motors in total, where k is an even number, and the pull / tension disc motors are divided into two groups, which are respectively installed on one side of the end of the flexible arm; each pull / tension disc motor output terminal is fixed to the head end of a pull / tension driving rope, and the end of the pull / tension driving rope passes through a plurality of flexible arm integrated joint body deformable shells in sequence and is fixed to the tail of the end effector; a pull / tension sensor is installed at the output terminal of the pull / tension disc motor and at each connection between the pull / tension driving rope and the flexible arm integrated joint body deformable shell; the "hand" deep vision module, the pull / tension disc motor and the pull / tension sensor all communicate with the visual servo controller.

[0023] The beneficial effects of the present invention are as follows: the present invention solves two key technical difficulties: the complex visual-touch fusion control and insufficient motion precision of the rigid-flexible hybrid manipulator after the rigid-flexible hybrid manipulator penetrates deep into the canopy of the fruit tree. Specifically: the fruit tree canopy classification obstacle avoidance safety strategy divides the complex and multi-category heterogeneous objects in the fruit tree canopy into avoidance targets and compliance targets. The anti-collision avoidance rigid-flexible hybrid motion sub-method realizes the high-precision control of the rigid-flexible hybrid manipulator to bypass the avoidance targets in the canopy of the fruit tree. The collision compliance rigid-flexible hybrid control sub-method greatly simplifies the control process of the rigid-flexible hybrid manipulator when the manipulator contacts the compliance target after the rigid-flexible hybrid manipulator penetrates deep into the canopy of the fruit tree, thereby improving the motion stability of the rigid-flexible hybrid manipulator. The collision compliance rigid-flexible hybrid precision compensation sub-method can effectively improve the operating precision of the rigid-flexible hybrid manipulator. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a structural diagram of the rigid-flexible hybrid robotic arm of the present invention;

[0025] FIG2( a ) is a schematic diagram of the structure of the flexible multi-joint continuum according to the present invention;

[0026] FIG2( b ) is a cross-sectional view of the flexible multi-joint continuum of the present invention;

[0027] Figure 3 Constructing a flow chart for the dynamic characteristics of the target avoidance of the present invention;

[0028] Figure 4 This is a schematic diagram of the semantic segmentation model of the avoidance target described in the present invention;

[0029] Figure 5 A schematic diagram of the anti-collision target avoidance bending deformation angle according to the present invention;

[0030] Figure 6 This is a schematic diagram of collision orientation judgment based on mass center change according to the present invention;

[0031] Figure 7 This is a flow chart of safe motion positioning of the rigid-flexible hybrid robotic arm within the canopy for visual-tactile fusion;

[0032] In the figure, 1. rigid arm, 2. pulling / tensioning disc motor, 3. pulling / tensioning drive rope, 4. compression spring, 5. "on-hand" depth vision module, 6. end effector, 7. flexible arm, 8. U-shaped connector, 9. pulling / tensioning disc motor output terminal, 10. flexible arm integrated joint body deformable shell, 11. integrated joint body connecting bolt, 12. pulling / tension sensor. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0034] A canopy visual-tactile fusion safe motion positioning device for a rigid-flexible hybrid robotic arm, comprising a "hand" deep vision module 5, a pull / tension disc motor 2, a pull / tension drive rope 3, a pull / tension sensor 12 and a visual servo controller, see Figure 1 , Figure 2(b); The "on-hand" depth vision module 5 is installed on the upper part of the end effector 6 of the flexible arm 7; there are a total of k pulling / tensioning disc motors 2 (k is an even number), and the value of k is related to the number of control parameters of the flexible arm 7; wherein, k / 2 pulling / tensioning disc motors 2 are arranged on one side of the end of the flexible arm 7, and another k / 2 pulling / tensioning disc motors 2 are arranged on the other side of the end of the flexible arm 7, and each pulling / tensioning disc motor output terminal 9 is connected and fixed to the head end of a pulling / tensioning drive rope 3, and the end of the pulling / tensioning drive rope 3 passes through a plurality of flexible arm integrated joint body deformation shells 10 in turn and is fixed to the tail of the end effector 6 of the flexible arm 7; a plurality of flexible arm integrated joint body deformation shells 10 are formed into a whole through an integrated joint body connecting bolt 11, as the shell of the flexible arm 7, and each A compression spring 4 is provided between the flexible arm integrated joint body deformable shell 10 and the flexible arm 7, and the flexible arm 7 and its shell and the compression spring 4 together constitute a flexible multi-joint continuum; there are multiple pull / tension sensors 12, wherein a pull / tension sensor 12 is installed at the pull / tension disc motor output terminal 9 and at each connection between the pull / tension drive rope 3 and the flexible arm integrated joint body deformable shell 10; the visual servo controller receives the depth color image information of the "hand" depth vision module 5, performs environmental feature recognition, extraction and positioning, and plans the detour and path of the collision target; the visual servo controller also receives the pull / tension sensor 12 to sense the pull / tension changes of the pull / tension drive rope 3, and controls the driving rigid-flexible hybrid robotic arm to complete the visual servo control of "detouring hard obstacles and complying with soft obstacles".

[0035] The flexible arm 7 and the pulling / tensioning disc motor 2 are both mounted on a U-shaped connecting member 8, and the U-shaped connecting member 8 is also connected to the rigid arm 1, see FIG. 2(a).

[0036] like Figure 7 As shown, a method for safe motion positioning of a rigid-flexible hybrid manipulator in a canopy by visual-touch fusion, including a fruit tree canopy classification obstacle avoidance safety strategy, a collision avoidance rigid-flexible hybrid motion sub-method, a collision orientation judgment sub-method, a collision compliance rigid-flexible hybrid control sub-method, and a collision compliance rigid-flexible hybrid precision compensation sub-method; the "hand" deep vision module 5 obtains the fruit tree canopy information, and divides the environmental targets in the fruit tree canopy into avoidance targets and compliance targets based on the fruit tree canopy classification obstacle avoidance safety strategy; based on the collision avoidance rigid-flexible hybrid motion sub-method, the dynamic characteristics of the avoidance target are used as obstacles, and the flexible arm avoids the deformation curvature τ c , avoid bending deformation angle θ cAs the posture constraint, dynamic path planning is performed with the initial position of the end effector as the path starting point and the fruit picking position as the path ending point. After the rigid-flexible hybrid robot arm contacts the compliant target, the flexible arm uses the corresponding integrated joint tension T obtained in real time by each tension sensor 12 on the flexible arm based on the collision compliance rigid-flexible hybrid control sub-method. n Theoretical pull / tension T with integrated joint L The error signal generated by the difference is combined with proportional control to adjust the tension of the pull / tension drive rope 3 output by the pull / tension disk motor 2, and dynamically adjust the rope length of the drive rope 3 to form the dynamic bending deformation angle of the flexible arm after contacting the compliance target; based on the collision compliance rigid-flexible hybrid precision compensation sub-method, a rigid-flexible hybrid manipulator force-position model C is established. k , calculate the real-time position deformation Δx of the end effector when the rigid-flexible hybrid manipulator contacts the compliance target, and then determine the compensation deflection θ of the integrated joint k , which is used as the control instruction of the rigid-flexible hybrid robot arm to compensate the dynamic bending deformation angle of the flexible arm and achieve precision compensation after conforming to the target.

[0037] The fruit tree canopy classification obstacle avoidance safety strategy includes the impact risk intensity-impact occurrence probability combination classification of multiple types of heterogeneous objects in the fruit tree canopy, the classification-based obstacle avoidance safety criterion, and the classification-based obstacle avoidance safety method.

[0038] Among them, the impact risk intensity-impact probability combination classification is as follows: the trunk, main branches and secondary main branches with high impact risk intensity of τ1 and impact probability of δ1 in the fruit tree canopy picking area are set as visual avoidance targets, the fruiting mother branches, fruiting branches and nutrient branches with impact risk intensity of τ2 and impact probability of δ2 in the fruit tree canopy picking area are set as tactile compliance targets, and the leaves with impact risk intensity of τ3 and impact probability of δ3 are not considered as obstacle avoidance objects. Among them, the relationship between impact risk intensity and impact probability is as follows:

[0039] τ1<τ2<τ3

[0040] δ3<δ2<δ1

[0041] Among them, the classification-based obstacle avoidance safety criterion is: the rigid-flexible hybrid robot arm needs to maintain a safe distance d from the avoidance target s , strictly avoid collision; the rigid-flexible hybrid manipulator can contact with the compliant target, and the pull / tension drive rope 3 is driven by the pull / tension disc motor 2 to adjust the torque, and the rigid-flexible hybrid manipulator undergoes configuration deformation, that is, the length of one or two sets of pull / tension drive ropes 3 is changed to achieve bending deformation of the rigid-flexible hybrid manipulator. Among them, the safety distance d s The calculation method is:

[0042] ds =k*d r

[0043] Where k is the safety factor of the rigid-flexible hybrid robot arm, d r is the radius of the rigid-flexible hybrid robot.

[0044] Among them, the classification-based obstacle avoidance safety method is: the fruit tree canopy information obtained by the "hand" deep vision module 5 obtains the avoidance target features in the fruit tree canopy, and sets them as local obstacles in the process of the rigid-flexible hybrid robot arm moving, and the rigid-flexible hybrid robot arm is driven to strictly avoid such targets to ensure no contact throughout the process; after the rigid-flexible hybrid robot arm completes the bypass avoidance target, it goes deep into the fruit tree canopy and contacts the compliance target, and the integrated joint tension / tension T obtained by each tension / tension sensor 12 on the flexible arm n The theoretical pulling / tension T of the corresponding integrated joint (a section of the flexible arm corresponding to each flexible arm integrated joint body deformation shell 10) L The error signal of the pulling / tensioning driving rope 3 is feedback-adjusted based on proportional control, and then the length of the pulling / tensioning driving rope 3 is dynamically adjusted by controlling the pulling / tensioning disc motor 2, so as to drive the flexible arm 7 in the rigid-flexible hybrid manipulator to generate a dynamic bending deformation angle, thereby realizing the smooth and steady movement of the rigid-flexible hybrid manipulator when contacting and complying with the target.

[0045] The fruit tree canopy classification obstacle avoidance safety strategy is summarized in the following table:

[0046]

[0047] The anti-collision avoidance rigid-flexible hybrid motion sub-method includes avoidance target identification and positioning, construction of avoidance target dynamic characteristics, and curved avoidance path planning.

[0048] The avoidance target recognition and positioning is specifically as follows: the deep vision module 5 on the hand collects the fruit tree canopy image information and transmits it to the computer, annotates the avoidance target features in the image, establishes a fruit tree canopy avoidance target image dataset, and uses the fruit tree canopy avoidance target image dataset as input information, imports it into the semantic segmentation network for iterative training, and generates a fruit tree canopy avoidance target semantic segmentation model M. d (See Figure 4 ) ; The real-time image information of the fruit tree canopy is used as input information and imported into the fruit tree canopy avoidance target semantic segmentation model M d The avoidance target plane coordinates in the real-time image information of the fruit tree canopy are output, and the avoidance target space point set P is obtained by combining the depth value of the real-time image information. ob ;

[0049] Among them, the construction of the dynamic features of the avoidance target is specifically as follows: the field of view space area is established with the field of view range of the deep vision module 5 on the "hand", and the real-time avoidance target space point set P ob As the root node, it recursively divides the field of view space area until the space volume contained in each node reaches the preset minimum volume V i , traverse the real-time avoidance target space point set P ob , and then construct the dynamic characteristics of the avoidance target based on the field of view space volume area; see Figure 3 ;

[0050] The specific planning of curved avoidance path is as follows: using the real-time avoidance target space point set P ob The flexible arm 7 bends and avoids according to the dynamic characteristics of the real-time avoidance target, and the avoidance deformation curvature τ of the flexible arm 7 c , avoid bending deformation angle θ c (See Figure 5 ) is calculated as follows:

[0051]

[0052] θ c =τ c *L soft

[0053] Among them, W st is the width of the minimum space body, W s is the safe distance between the flexible arm 7 and the avoidance target, L soft is the length of the flexible arm 7;

[0054] The flexible arm 7 is used to avoid the deformation curvature τ c Control the joint bending to generate the avoidance bending deformation angle θ c , the initial position of the end effector 6 is used as the path starting point and the fruit picking position is used as the path ending point for dynamic path planning, the desired position of the end effector 6 is obtained, and the error between the actual position and the desired position of the end effector 6 is dynamically adjusted based on the PID controller to ensure the motion accuracy of the flexible arm 7; the error formula is as follows:

[0055]

[0056] Among them, K p is the proportional gain coefficient of the PID controller, K i is the integral gain coefficient of the PID controller, K d is the differential gain coefficient of the PID controller, e(t) is the error between the actual position and the expected position of the end effector 6, and μ(t) is the precision adjustment amount of the flexible arm 7.

[0057] The collision orientation judgment sub-method is specifically as follows: the rigid-flexible hybrid robotic arm penetrates into the canopy of the fruit tree, contacts and collides with a compliant target with an impact probability of δ2, and the tension sensor 12 on the flexible arm detects the tension change ΔT of the tension driving rope 3 in real time. i , combined with the rope direction vector of the pull / tension drive rope 3 Calculate the touch force Through the force balance and the geometric relationship of the flexible arm, the change in the center of mass position of the flexible arm 7 ΔC is calculated. soft , establish the positional relationship between the center of mass of the flexible arm and the collision point, that is, the collision orientation, such as Figure 6 As shown; among them, the external force Change in mass center position ΔC soft , the position relationship between the center of mass of the flexible arm and the collision point P c (x c ,y c )as follows:

[0058]

[0059] P c (x c ,y c )=P0(x0,y0)+ΔC soft

[0060] Among them, P0(x0,y0) is the initial point of the center of mass of the flexible arm.

[0061] The collision compliance rigid-flexible hybrid control sub-method is specifically as follows: the rigid-flexible hybrid manipulator moves and contacts the compliance target, and each pull / tension sensor 12 on the flexible arm obtains the corresponding integrated joint pull / tension T in real time. n , which is consistent with the integrated joint theoretical pull / tension T L The error is ΔT m , based on proportional control, the pull / tension disc motor 2 is controlled to adjust the pull / tension of the pull / tension driving rope 3, and then the rope length of the driving rope 3 is dynamically adjusted to form a dynamic bending deformation angle of the flexible arm after contact compliance target; the proportional control model ΔL of the pull / tension rope disc motor i , the dynamic bending deformation angle is calculated as follows:

[0062] ΔL i =k i ·K p ΔT m

[0063]

[0064] Where n is the total number of integrated joints of the flexible arm, k i K is the adjustment factor for pulling / tensioning the drive rope.p is the proportional control gain matrix of the pull / tension disk motor 2, ΔT m is the pull / tension error signal of the integrated joint, θ n is the dynamic bending deformation angle of the flexible arm (its vertex is the collision orientation), K θ is the stiffness coefficient of the flexible arm.

[0065] The collision compliance rigid-flexible hybrid precision compensation sub-method is specifically as follows: after the rigid-flexible hybrid manipulator contacts the compliant target, its force-position model C is established using a linear elastic regression model. k , calculate the real-time position deformation Δx of the end effector when the rigid-flexible hybrid manipulator contacts the compliance target, obtain the end position compensation of the rigid-flexible hybrid manipulator, and determine the compensation deflection θ of the integrated joint k , the flexible arm integrated joint compensation deflection θ k As the control instruction of the rigid-flexible hybrid manipulator, the dynamic bending deformation angle of the flexible arm is compensated; among them, the force-position model C of the rigid-flexible hybrid manipulator k for:

[0066] C k :x actual =f(θ L )+f(θ0)+Δx

[0067] Δx=f(θ k )=K -1 *F real

[0068] Among them, f(θ L ) represents the position of the connection between the rigid arm and the flexible arm, x actual represents the actual position of the end effector, f(θ0) is the theoretical position of the end effector of the flexible arm, and f(θ k ) is the end position compensation of the rigid-flexible hybrid manipulator, K represents the stiffness matrix of the flexible arm, and F real represents the external stress acting on the flexible arm in accordance with the target; f(θ k )=K -1 *F real Derived θ k , as the kinematic input parameter of the rigid-flexible hybrid manipulator, generates the control command of the rigid-flexible hybrid manipulator, drives the flexible arm posture and the rope length change of the pulling / tensioning driving rope, and compensates the dynamic bending deformation angle θ of the flexible arm n .

[0069] A method for safe motion positioning of a rigid-flexible hybrid robot arm by visual-tactile fusion in a canopy, comprising the following steps:

[0070] Step 1: The rigid-flexible hybrid robot arm and the canopy visual-touch fusion safe motion positioning device of the rigid-flexible hybrid robot arm are powered on and started, and the "hand-held" deep vision module 5 automatically starts image acquisition to obtain real-time image information of the fruit tree canopy;

[0071] Step 2: According to the impact risk intensity and the probability of impact occurrence, the multiple heterogeneous objects in the fruit tree canopy are divided into avoidance targets and compliance targets;

[0072] Step 3: For the avoidance target, the rigid-flexible hybrid robot arm strictly avoids it in the following ways:

[0073] The “hand” deep vision module 5 takes the captured fruit tree canopy information as input information and inputs it into the fruit tree canopy avoidance target semantic segmentation model M d , obtain the avoidance target space point set P ob ; Based on the "hand" depth vision module 5, the visual field space area is established, and the avoidance target space point set P ob As the root node, it recursively divides the field of view space area until the space volume contained in each node reaches the preset minimum volume V i , by traversing the avoidance target space point set P ob , construct the dynamic characteristics of the avoidance target based on the field of view space volume area; take the dynamic characteristics of the avoidance target as the obstacle, and the flexible arm avoids the deformation curvature τ c , avoid bending deformation angle θ c As posture constraints, dynamic path planning is performed with the initial position of the end effector as the path starting point and the fruit picking position as the path ending point. The error between the actual position and the expected position of the end effector is dynamically adjusted based on the PID controller.

[0074] Step 4: For the compliant target, the rigid-flexible hybrid manipulator contacts the compliant target, and the pull / tension drive rope 3 is driven by the pull / tension disc motor 2 to adjust the torque.

[0075] The rigid-flexible hybrid robot arm moves in contact with the target, and the flexible arm pull / tension sensor 12 obtains the corresponding integrated joint tension T in real time. n , which is consistent with the integrated joint theoretical pull / tension T L The error is ΔT m , based on ΔT m , adjusting the pulling / tension of the driving rope 3, and then dynamically adjusting the rope length of the driving rope 3, forming a dynamic bending deformation angle of the flexible arm after contacting the compliance target; wherein the dynamic bending deformation angle is the collision orientation;

[0076] After the rigid-flexible hybrid manipulator contacts the compliant target, its force-position model C is established using the linear elastic regression model. k, the real-time deformation Δx of the end effector of the rigid-flexible hybrid manipulator is calculated when it complies with the contact compliance target, the end compensation of the rigid-flexible hybrid manipulator is obtained, and the compensation deflection θ of the flexible arm integrated joint is determined k , the flexible arm integrated joint compensation deflection θ k As a control instruction for the rigid-flexible hybrid robot arm, it compensates for the dynamic bending angle of the flexible arm; the flexible arm runs at the compensated angle, driving the end effector to grasp the fruit target.

[0077] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A method for safe motion positioning of a rigid-flexible hybrid robot arm by visual-tactile fusion within the canopy, characterized by: Obtain the canopy information of fruit trees, and divide the multiple heterogeneous objects in the canopy of fruit trees into avoidance targets and compliance targets according to the impact risk intensity and the probability of impact occurrence; For the avoidance target, the rigid-flexible hybrid manipulator avoids it in the following ways: input the fruit tree canopy information into the fruit tree canopy avoidance target semantic segmentation model, output the avoidance target plane coordinates in the fruit tree canopy real-time image information, and combine the depth value of the real-time image information to obtain the avoidance target space point set; The avoidance target space point set is used as a root node, and the field of view space body area is recursively divided one by one until the space volume contained in each node reaches a preset minimum volume, and the avoidance target dynamic features based on the field of view space body area are constructed by traversing the avoidance target space point set; Taking the dynamic characteristics of the avoidance target as an obstacle, the flexible arm avoids the deformation curvature τ c , avoid bending deformation angle θ c As posture constraints, dynamic path planning is performed with the initial position of the end effector as the path starting planning point and the fruit picking position as the path ending planning point; For the compliant target, the rigid-flexible hybrid manipulator avoids the target and then contacts the compliant target. Each pull / tension sensor on the flexible arm obtains the corresponding integrated joint pull / tension T in real time. n , T n The corresponding integrated joint theoretical pull / tension T L The error is ΔT m , based on the ΔT m Adjust the pull / tension of the driving rope, and then dynamically adjust the rope length of the driving rope to form a dynamic bending deformation angle of the flexible arm after contacting the compliant target; after the rigid-flexible hybrid robotic arm contacts the compliant target, the dynamic bending deformation angle of the flexible arm is compensated, and the flexible arm runs at the compensated angle, driving the end effector to grasp the fruit target.

2. The intracanopy visual-tactile fusion safe motion positioning method according to claim 1, characterized in that: The avoidance target and compliance target are determined by: The trunk, main branches and secondary main branches with high impact risk intensity of τ1 and impact probability of δ1 in the fruit tree canopy picking area are set as vision-based avoidance targets, and the fruiting mother branches, fruiting branches and nutrient branches with impact risk intensity of τ2 and impact probability of δ2 in the fruit tree canopy picking area are set as tactile-based compliance targets, and: τ1<τ2, δ2<δ1.

3. The intracanopy visual-tactile fusion safe motion positioning method according to claim 2, characterized in that: The avoidance deformation curvature τ c and avoidance bending deformation angle θ c Respectively meet: i c =t c *L soft Among them, W st W is the width of the minimum space volume when the space volume contained by each node reaches the preset minimum volume. s is the safe distance between the flexible arm and the avoidance target, L soft is the length of the flexible arm.

4. The intracanopy visual-tactile fusion safe motion positioning method according to claim 1, characterized in that: The dynamic bending deformation angle of the compensation flexible arm is specifically: the force-position model C of the rigid-flexible hybrid manipulator is established by a linear elastic regression model k , calculate the real-time position deformation Δx of the end effector when the rigid-flexible hybrid manipulator contacts the compliance target, obtain the end position compensation of the rigid-flexible hybrid manipulator, and determine the compensation deflection θ of the flexible arm integrated joint k , the flexible arm integrated joint compensation deflection θ k As the control instruction of the rigid-flexible hybrid robot arm, it compensates the dynamic bending deformation angle of the flexible arm.

5. The intracanopy visual-tactile fusion safe motion positioning method according to claim 4, characterized in that: The force-position model C of the rigid-flexible hybrid manipulator k For: C k :x actual =f(θ L )+f(θ0)+Δx, the flexible arm integrated joint compensates the deflection θ k Satisfies: Δx=f(θ k )=K -1 *F real ; Among them, f(θ L ) represents the position of the connection between the rigid arm and the flexible arm, x actual represents the actual position of the end effector, f(θ0) is the theoretical position of the end effector of the flexible arm, and f(θ k ) is the end position compensation of the rigid-flexible hybrid manipulator, K represents the stiffness matrix of the flexible arm, and F real represents the external stress acting on the flexible arm due to the compliance target.

6. The intracanopy visual-tactile fusion safe motion positioning method according to claim 5, characterized in that: By f(θ k )=K -1 *F real Derived θ k , as the kinematic input parameters of the rigid-flexible hybrid robot arm, generate the control instructions of the rigid-flexible hybrid robot arm, drive the flexible arm posture and the rope length change of the pulling / tensioning driving rope, and compensate the dynamic bending deformation angle of the flexible arm.

7. The intracanopy visual-tactile fusion safe motion positioning method according to claim 1, characterized in that: The vertex of the dynamic bending deformation angle of the flexible arm is the collision orientation, and the collision orientation is obtained by the following method: the tension sensor on the flexible arm detects the tension change ΔT of the tension driving rope in real time. i , combined with the rope direction vector of the pull / tension drive rope Calculate the touch force Through the force balance and the geometric relationship of the flexible arm, the change of the center of mass position of the flexible arm ΔC is calculated soft , establish the positional relationship between the center of mass of the flexible arm and the collision point, that is, the collision orientation.

8. The intracanopy visual-tactile fusion safe motion positioning method according to claim 7, characterized in that: The touching external force satisfy: The change in the center of mass position of the flexible arm ΔC soft satisfy: The position relationship between the center of mass of the flexible arm and the collision point is: c (x c ,y c )=P0(x0,y0)+ΔC soft , where n is the total number of integrated joints of the flexible arm, P0(x0,y0) is the initial point of the center of mass of the flexible arm, and k i It is the adjustment factor for pulling / tensioning the drive rope.

9. The intracanopy visual-tactile fusion safe motion positioning method according to claim 1, characterized in that: Perform dynamic path planning to obtain the desired position of the end effector, and dynamically adjust the error between the actual position and the desired position of the end effector based on the PID controller to ensure the motion accuracy of the flexible arm.

10. A device for implementing the intra-canopy visual-tactile fusion safe motion positioning method as described in any one of claims 1 to 9, characterized in that: The invention comprises a "hand" deep vision module (5), a pulling / tensioning disk motor (2), a pulling / tensioning driving rope (3), a pulling / tensioning sensor (12) and a visual servo controller; the "hand" deep vision module (5) is installed on the upper part of the end effector (6) of the flexible arm (7); there are k pulling / tensioning disk motors (2), wherein k is an even number, and the pulling / tensioning disk motors (2) are divided into two groups, which are respectively installed on one side of the end of the flexible arm (7); each pulling / tensioning disk motor output terminal (9) is connected to a pulling / tensioning driving rope (3 ), the end of the pulling / tension driving rope (3) passes through a plurality of flexible arm integrated joint body deformable shells (10) in sequence and is then fixed to the tail of the end effector (6); a pulling / tension sensor (12) is installed at the pulling / tension disc motor output terminal (9) and at each connection between the pulling / tension driving rope (3) and the flexible arm integrated joint body deformable shell (10); the "hand" depth vision module (5), the pulling / tension disc motor (2) and the pulling / tension sensor (12) all communicate with the visual servo controller.

Images