An in-canopy visual touch fusion safety motion positioning method and device for a rigid-flexible hybrid robot arm

Abstract

The application discloses a kind of crown layer inside view touch fusion safety movement positioning method and device of rigid-flexible hybrid manipulator, and multiple kinds of heterogeneous objects in fruit tree crown layer are divided into avoidance target and compliance target;For avoidance target, the dynamic characteristics of avoidance target are used as obstacles, and dynamic paths are planned, so that rigid-flexible hybrid manipulator strictly avoids such targets;For compliance target, rigid-flexible hybrid manipulator penetrates into fruit tree crown layer and contacts compliance target, and the integrated joint tension of each tension sensor on flexible arm is subtracted from the corresponding integrated joint theoretical tension, to generate error signal, and the error signal of driving rope is feedback adjusted, and then the rope length of driving rope is dynamically adjusted by controlling tension disc type motor, to drive flexible arm to generate dynamic bending deformation angle, to realize compliant and smooth movement when rigid-flexible hybrid manipulator contacts compliance target.The application is widely applicable to autonomous continuous planning of rigid-flexible hybrid manipulator in various fruit tree complex crown layer scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of agricultural robots, in particular to a crown layer in-sight contact fusion safety motion positioning method and device of a rigid-flexible hybrid manipulator. BACKGROUND

[0002] China is one of the largest fruit producing countries in the world, with apple and pear production ranking first in the world. The rigid-flexible hybrid manipulator has the characteristics of high-speed displacement motion and flexible passing through narrow and limited areas due to its rigid structure and flexible continuum hybrid structure. However, the rigid-flexible hybrid manipulator often faces complex environmental disturbances caused by multiple heterogeneous objects during fruit picking in the fruit tree crown layer. Single rigid link manipulator contact with large hard branches will cause collision and picking failure, and the flexible arm continuous control method is complex and the motion is relatively slow, resulting in low picking efficiency and obvious fluctuation of motion precision.

[0003] Existing picking robot products and test machines based on rigid-flexible hybrid manipulators are applied to relatively standard and structured industrial scenes or indoor simulation links without interference, but they have poor scene structure adaptability and are prone to interference on the inside of the tree crown. The rigid-flexible hybrid manipulator must avoid hard obstacles and flexibly adapt to soft obstacles when it is deeply inserted into the fruit tree crown layer, resulting in complex picking strategies for multiple heterogeneous objects, dynamic re-planning of detour paths, and tedious precision compensation control. SUMMARY

[0004] To solve the problems in the prior art, the present application provides a crown layer in-sight contact fusion safety motion positioning method and device of a rigid-flexible hybrid manipulator, which effectively solves the problem of motion planning difficulty of the rigid-flexible hybrid manipulator in a complex fruit tree crown layer with multiple heterogeneous objects, and realizes autonomous fruit picking of the rigid-flexible hybrid manipulator in the complex fruit tree crown layer.

[0005] The present application achieves the above technical purpose through the following technical means.

[0006] A crown layer in-sight contact fusion safety motion positioning method of a rigid-flexible hybrid manipulator comprises the following steps:

[0007] Obtain the information of the fruit tree crown layer, and divide the multiple heterogeneous objects in the fruit tree crown layer into avoidance targets and compliant targets according to the impact risk intensity and the impact occurrence probability.

[0008] For the avoidance target, the rigid-flexible hybrid manipulator avoids by the following way: inputting the fruit tree canopy information into the fruit tree canopy avoidance target semantic segmentation model, outputting the avoidance target plane coordinates in the real-time image information of the fruit tree canopy, combining the depth value of the real-time image information, obtaining the avoidance target space point set; taking the avoidance target space point set as a root node, recursively segmenting the field of view space body region, until the space volume contained by each node reaches the preset minimum volume, constructing the avoidance target dynamic characteristics based on the field of view space body region by traversing the avoidance target space point set; taking the avoidance target dynamic characteristics as an obstacle, the flexible arm avoids the obstacle with the avoidance deformation curvature τ c , the avoidance bending deformation angle θ c as the posture constraint, and planning a dynamic path with the initial position of the end effector as the path starting planning point and the fruit picking position as the path terminating planning point;

[0009] For the compliant target, the rigid-flexible hybrid manipulator contacts the compliant target after bypassing the avoidance target, and each tensile force sensor on the flexible arm obtains the corresponding integrated joint tensile force T n , the error between the corresponding integrated joint theoretical tensile force T n and the corresponding integrated joint theoretical tensile force T L is ΔT m , and the driving rope tensile force is adjusted based on the ΔT m , and the rope length of the driving rope is dynamically adjusted, forming the dynamic bending deformation angle of the flexible arm after contacting the compliant target; after the rigid-flexible hybrid manipulator contacts the compliant target, the dynamic bending deformation angle of the flexible arm is compensated, and the flexible arm runs at the compensated angle, driving the end effector to grasp the fruit target.

[0010] Further, the avoidance target and the compliant target are determined by the following way:

[0011] The main stem, main branch and secondary main branch in the fruit tree canopy picking area with high impact risk intensity τ1 and impact probability δ1 are set as the vision-based avoidance target, and the result branch, result branch and nutrient branch in the fruit tree canopy picking area with impact risk intensity τ2 and impact probability δ2 are set as the tactile-based compliant target, and: τ1 < τ2, δ2 < δ1.

[0012] Further, the avoidance deformation curvature τ c and the avoidance bending deformation angle θ c respectively satisfy:

[0013]

[0014] θ c = τ c *L soft

[0015] wherein, Wst W is the width of the minimum space volume required for each node to reach a preset minimum volume. s To ensure a safe distance between the flexible arm and the target to be avoided, L soft The length of the flexible arm.

[0016] Furthermore, the dynamic bending deformation angle of the compensated flexible arm is specifically determined by establishing a force-position model C of the rigid-flexible hybrid manipulator using a linear elastic regression model. k The real-time position deformation Δx of the end effector when the rigid-flexible hybrid robotic arm contacts and conforms to the target is calculated. The end position compensation of the rigid-flexible hybrid robotic arm is obtained, and the compensation deflection θ of the integrated joint of the flexible arm is determined. k The flexible arm integrates a joint to compensate for the deflection θ. k As control commands for a rigid-flexible hybrid robotic arm, they compensate for the dynamic bending deformation angle of the flexible arm.

[0017] Furthermore, the force-position model C of the rigid-flexible hybrid robotic arm k For: C k :x actual =f(θ) L The flexible arm integrated joint compensation deflection θ is given by f(θ0) + Δx. k Satisfy: Δx=f(θ) k ) = K -1 *F real ; where f(θ) L ) indicates the location of the connection between the rigid arm and the flexible arm, x actual f(θ0) represents the actual position of the end effector, and f(θ0) represents the theoretical position of the end effector of the flexible arm. k ) represents the end-effector position compensation amount of the rigid-flexible hybrid robotic arm, K represents the stiffness matrix of the flexible arm, and F represents the end-effector position compensation amount. real This indicates that the external stress acting on the flexible arm conforms to the target.

[0018] Furthermore, by f(θ) k ) = K -1 *F real Derive θ k As the kinematic input parameters of the rigid-flexible hybrid robotic arm, it generates control commands for the rigid-flexible hybrid robotic arm, drives the pose of the flexible arm and the change in the length of the tension / tension drive rope, and compensates for the dynamic bending deformation angle of the flexible arm.

[0019] Furthermore, the vertex of the dynamic bending deformation angle of the flexible arm is the collision orientation, which is obtained by the following method: the tension / tension sensor on the flexible arm detects the tension / tension change ΔT of the tension / tension drive rope in real time. i Combined with the rope direction vector of the tension / tension drive rope Calculate the contact force The change in the position of the center of mass of the flexible arm, ΔC, is calculated by considering the force balance and the geometric relationship of the flexible arm. soft Establish the positional relationship between the center of mass of the flexible arm and the collision point, i.e., the collision orientation.

[0020] Furthermore, the external force of contact satisfy: The change in the center of mass position of the flexible arm, ΔC soft satisfy: The positional relationship between the center of mass of the flexible arm and the collision point is: P c (x c ,y c )=P0(x0,y0)+ΔC soft Where n is the total number of integrated joints in the flexible arm, P0(x0,y0) is the initial position of the centroid of the flexible arm, and k i This is the adjustment factor for the tension / tension drive rope.

[0021] Furthermore, dynamic path planning is performed to obtain the desired position of the end effector. Based on the PID controller, the error between the actual position and the desired position of the end effector is dynamically adjusted to ensure the motion accuracy of the flexible arm.

[0022] A canopy-based visual-touch fusion safety motion positioning device for a rigid-flexible hybrid robotic arm includes a depth vision module on the "hand," a tension / tension disc motor, a tension / tension drive rope, a tension sensor, and a visual servo controller. The depth vision module on the "hand" is mounted on the upper part of the end effector of the flexible arm. There are k tension / tension disc motors in total, where k is an even number. The tension / tension disc motors are divided into two groups and installed on one side of the end of the flexible arm. The output terminal of each tension / tension disc motor is fixed to the head end of a tension / tension drive rope. The end of the tension / tension drive rope passes through multiple flexible arm integrated joint deformable shells and is fixed to the tail of the end effector. Tension / tension sensors are installed at the output terminals of the tension / tension disc motors and at each connection point between the tension / tension drive rope and the flexible arm integrated joint deformable shell. The depth vision module on the "hand," the tension / tension disc motor, and the tension / tension sensor all communicate with the visual servo controller.

[0023] The application has the advantages that the application solves two key technical difficulties of complex control and insufficient action precision of visual and tactile fusion after the rigid-flexible hybrid manipulator deeply enters the fruit tree canopy, specifically, the fruit tree canopy classification obstacle avoidance safety strategy divides the complex multi-class heterogeneous objects in the fruit tree canopy into avoidance targets and compliant targets, the anti-collision avoidance rigid-flexible hybrid motion sub-method realizes high-precision control of the rigid-flexible hybrid manipulator deeply entering the fruit tree canopy and avoiding the target, the collision compliant rigid-flexible hybrid control sub-method greatly simplifies the control process of the rigid-flexible hybrid manipulator when the manipulator contacts the compliant target after deeply entering the fruit tree canopy, improves the motion stability of the rigid-flexible hybrid manipulator, and the collision compliant rigid-flexible hybrid precision compensation sub-method can effectively improve the operation precision of the rigid-flexible hybrid manipulator. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The rigid-flexible hybrid manipulator structure is described in the application.

[0025] Fig. 2(a) is a schematic diagram of the flexible multi-joint continuum structure described in the application.

[0026] Fig. 2(b) is a cross-sectional view of the flexible multi-joint continuum described in the application.

[0027] Figure 3 The avoidance target dynamic feature construction flowchart is described in the application.

[0028] Figure 4 The avoidance target semantic segmentation model schematic diagram is described in the application.

[0029] Figure 5 The anti-collision target avoidance bending deformation angle schematic diagram is described in the application.

[0030] Figure 6 The collision direction judgment schematic diagram based on the centroid change is described in the application.

[0031] Figure 7 The canopy-in visual and tactile fusion safety motion positioning flowchart of the rigid-flexible hybrid manipulator is described in the application.

[0032] In the figure, 1. rigid arm, 2. tension disc motor, 3. tension driving rope, 4. compression spring, 5. "hand on" depth vision module, 6. end effector, 7. flexible arm, 8. U-shaped connecting piece, 9. tension disc motor output terminal, 10. flexible arm integrated joint body deformation shell, 11. integrated joint body connecting bolt, 12. tension sensor. DETAILED DESCRIPTION

[0033] The application will be further described below in combination with the drawings and specific embodiments, but the protection scope of the application is not limited thereto.

[0034] A kind of rigid-flexible hybrid manipulator's crown layer in-situ vision touch fusion safety motion positioning device, including "hand on" depth vision module 5, pull / tension disc motor 2, pull / tension driving rope 3, tension sensor 12 and vision servo controller, see Figure 1 Figure 2 (b);The "hand on" depth vision module 5 is installed on the upper portion of the flexible arm 7 end effector 6;Pull / tension disc motor 2 is k (k is even), the value of k is related to the number of flexible arm 7 control parameters;Wherein, k / 2 pull / tension disc motor 2 is arranged on one side of the flexible arm 7 end, and the other k / 2 pull / tension disc motor 2 is arranged on the other side of the flexible arm 7 end, and each pull / tension disc motor output terminal 9 is connected with the head end of 1 pull / tension driving rope 3;The tail end of pull / tension driving rope 3 is fixed to the tail of flexible arm 7 end effector 6 after passing through a plurality of flexible arm integrated joint body deformation shell 10 in turn;A plurality of flexible arm integrated joint body deformation shell 10 is formed as a whole by integrated joint body connecting bolt 11, as the shell of flexible arm 7, and a compression spring 4 is provided between each flexible arm integrated joint body deformation shell 10 and flexible arm 7, and the flexible arm 7 and its shell, compression spring 4 together constitute flexible multi-joint continuum;The tension sensor 12 has a plurality of, wherein pull / tension disc motor output terminal 9, pull / tension driving rope 3 and flexible arm integrated joint body deformation shell 10 are provided with tension sensor 12 at each connection;The vision servo controller receives the depth color image information of "hand on" depth vision module 5, carries out environment feature recognition extraction and positioning, plans the detour and path of collision target;Vision servo controller also receives the tension change of pull / tension driving rope 3 sensed by tension sensor 12, controls the driving of rigid-flexible hybrid manipulator to complete the vision servo control of "hard obstacle detour, soft obstacle compliance".

[0035] The flexible arm 7 and pull / tension disc motor 2 are installed on the U-shaped connecting piece 8, and the U-shaped connecting piece 8 is also connected with the rigid arm 1, see Figure 2 (a).

[0036] As shown in Figure 7 A rigid-flexible hybrid manipulator's crown layer in-situ vision touch fusion safety motion positioning method, including fruit tree crown layer classification obstacle avoidance safety strategy, anti-collision avoidance rigid-flexible hybrid motion sub-method, collision direction judgment sub-method, collision compliant rigid-flexible hybrid control sub-method, collision compliant rigid-flexible hybrid precision compensation sub-method;"Hand on" depth vision module 5 obtains fruit tree crown layer information, and the fruit tree crown layer classification obstacle avoidance safety strategy divides the environment target in fruit tree crown layer into avoidance target and compliant target;Based on the anti-collision avoidance rigid-flexible hybrid motion sub-method, the dynamic characteristics of avoidance target are used as obstacles, and the flexible arm is deformed with avoidance curvature τ c , avoidance bending deformation angle θ cAs a posture constraint, dynamic path planning is performed with the end effector initial position as the path starting planning point and the fruit picking position as the path ending planning point; after the rigid-flexible hybrid manipulator contacts the compliant target, the flexible arm adjusts the tensile force of the tensile drive rope 3 output by the tensile-disk motor 2 based on the error signal generated by the difference between the real-time integrated joint tensile force T n and the theoretical tensile force T L , dynamically adjusts the rope length of the drive rope 3, and forms the dynamic bending deformation angle of the flexible arm after contacting the compliant target; based on the collision-compliant rigid-flexible hybrid precision compensation sub-method, a force-position model C k of the rigid-flexible hybrid manipulator is established to calculate the real-time position deformation amount Δx of the end effector when the rigid-flexible hybrid manipulator contacts the compliant target, and then determine the compensation deflection amount θ k of the integrated joint, which is used as the control instruction of the rigid-flexible hybrid manipulator to compensate the dynamic bending deformation angle of the flexible arm and realize precision compensation after the compliant target.

[0037] The fruit tree canopy classification obstacle avoidance safety strategy includes impact risk intensity-impact occurrence probability combined classification of multi-class heterogeneous objects in the fruit tree canopy, classification-based obstacle avoidance safety criteria, and classification-based obstacle avoidance safety method.

[0038] The impact risk intensity-impact occurrence probability combined classification is as follows: the main stem, main branch and secondary main branch in the fruit tree canopy picking area with high impact risk intensity τ1 and impact occurrence probability δ1 are set as visual-based avoidance targets, the result branch, result branch and nutrient branch in the fruit tree canopy picking area with impact risk intensity τ2 and impact occurrence probability δ2 are set as tactile-based compliant targets, and the leaf with impact risk intensity τ3 and impact occurrence probability δ3 is not considered as an obstacle avoidance object. The size relationship of the impact risk intensity and the impact occurrence probability is as follows:

[0039] τ1 < τ2 < τ3

[0040] δ3 < δ2 < δ1

[0041] The classification-based obstacle avoidance safety criteria are as follows: the rigid-flexible hybrid manipulator needs to maintain a safe distance d s from the avoidance target to strictly avoid collision; the rigid-flexible hybrid manipulator can contact the compliant target, the tensile-disk motor 2 drives the tensile force drive rope 3 to adjust the torque, the rigid-flexible hybrid manipulator deforms in configuration, i.e. the length of one or two groups of tensile force drive ropes 3 changes, realizing the bending deformation of the rigid-flexible hybrid manipulator. The calculation method of the safe distance d s is as follows:

[0042] ds = k * d r

[0043] In the formula, k is the safety factor of the rigid-flexible hybrid robot arm, d r is the radius of the rigid-flexible hybrid robot arm.

[0044] Wherein, the classification-based obstacle avoidance safety method is: obtaining the fruit tree canopy information from the "hand" depth vision module 5, obtaining the avoidance target features in the fruit tree canopy, and setting the rigid-flexible hybrid robot arm as a local obstacle during the movement process, planning the local obstacle avoidance scene path of the rigid-flexible hybrid robot arm, driving the rigid-flexible hybrid robot arm to strictly avoid such targets, and ensuring no contact throughout the process; after the rigid-flexible hybrid robot arm completes the avoidance of the target, it deeply penetrates into the fruit tree canopy and contacts the compliant target, and the integrated joint tension T n of each flexible arm on the flexible arm is obtained by the tension sensor 12, and the corresponding integrated joint theoretical tension T L is obtained by the tension sensor 12, the difference between the two is subtracted to generate an error signal, the error signal of the tension driving rope 3 is feedback adjusted based on proportional control, and then the rope length of the tension driving rope 3 is dynamically adjusted through the control of the tension disc motor 2, the flexible arm 7 in the rigid-flexible hybrid robot arm generates a dynamic bending deformation angle, and the compliant and smooth motion of the rigid-flexible hybrid robot arm when contacting the compliant target is realized.

[0045] The fruit tree canopy classification obstacle avoidance safety strategy is summarized in the following table:

[0046]

[0047] The anti-collision avoidance rigid-flexible hybrid motion sub-method includes avoidance target identification and positioning, construction of avoidance target dynamic characteristics, and bending avoidance path planning.

[0048] Wherein, the avoidance target identification and positioning is specifically: the fruit tree canopy image information collected by the "hand" depth vision module 5 is transmitted to the computer, the avoidance target features in the image are labeled, the fruit tree canopy avoidance target image dataset is established, the fruit tree canopy avoidance target image dataset is input into the semantic segmentation network for iterative training, and the fruit tree canopy avoidance target semantic segmentation model M d (see Figure 4 ) is generated; the real-time image information of the fruit tree canopy is input into the fruit tree canopy avoidance target semantic segmentation model M d , and the plane coordinates of the avoidance target in the real-time image information of the fruit tree canopy are output, combined with the depth value of the real-time image information, and then the avoidance target space point set P ob is obtained.

[0049] The dynamic characteristics of the avoidance target are built by establishing a field of view space region in the field of view range of the "hand" depth vision module 5, and a real-time avoidance target space point set P ob The field of view space region is recursively partitioned as a root node until each node contains a space volume reaching a preset minimum volume V i The real-time avoidance target space point set P ob is traversed to build the dynamic characteristics of the avoidance target based on the field of view space region; see Figure 3 ;

[0050] The curved avoidance path planning is specifically as follows: the real-time avoidance target space point set P ob is used to bend and avoid by the flexible arm 7 according to the real-time dynamic characteristics of the avoidance target, and the avoidance deformation curvature τ c and the avoidance bending deformation angle θ c (see Figure 5 ) of the flexible arm 7 are calculated as follows:

[0051]

[0052] θ c = τ c *L soft

[0053] Where W st is the width of the minimum space volume, W s is the safety distance between the flexible arm 7 and the avoidance target, and L soft is the length of the flexible arm 7.

[0054] The flexible arm 7 controls the joint bending with the avoidance deformation curvature τ c , and generates the avoidance bending deformation angle θ c . The initial position of the end effector 6 is taken as the path starting planning point, and the fruit picking position is taken as the path ending planning point to perform dynamic path planning, so as to obtain the expected position of the end effector 6. The error between the actual position and the expected position of the end effector 6 is dynamically adjusted based on the PID controller to ensure the motion accuracy of the flexible arm 7. The error formula is as follows:

[0055]

[0056] Where K p is the proportional gain coefficient of the PID controller, K i is the integral gain coefficient of the PID controller, K d is the differential gain coefficient of the PID controller, e(t) is the error between the actual position and the expected position of the end effector 6, and μ(t) is the accuracy adjustment amount of the flexible arm 7.

[0057] The collision orientation judgment sub-method is specifically that the rigid-flexible hybrid robot arm is deeply inserted into the crown layer of the fruit tree, contacts and collides with the compliant target with a collision probability of δ2, and the tensile force sensor 12 on the flexible arm detects the tensile force change ΔT of the tensile driving rope 3 in real time i , and the rope direction vector of the tensile driving rope 3 is combined to calculate the touch external force The mass center position change amount ΔC of the flexible arm 7 is calculated through force balance and geometric relationship of the flexible arm soft , and the position relationship between the mass center position of the flexible arm and the collision point, i.e. the collision orientation, is established, as shown in Figure 6 ; wherein the touch external force The mass center position change amount ΔC soft , and the position relationship P c (x c ,y c ) between the mass center position of the flexible arm and the collision point are as follows:

[0058]

[0059] P c (x c ,y c )=P0(x0,y0)+ΔC soft

[0060] Wherein P0(x0,y0) is the initial point of the mass center position of the flexible arm.

[0061] The collision compliant rigid-flexible hybrid control sub-method is specifically that the rigid-flexible hybrid robot arm moves and contacts the compliant target, each tensile force sensor 12 on the flexible arm obtains the corresponding integrated joint tensile force T n in real time, the error of which with the integrated joint theoretical tensile force T L is ΔT m , the tensile disc motor 2 is controlled based on proportional control, the tensile force of the tensile driving rope 3 is adjusted, and then the rope length of the driving rope 3 is dynamically adjusted to form the dynamic bending deformation angle of the flexible arm after the contact compliant target; the proportional control model ΔL i of the tensile rope disc motor and the dynamic bending deformation angle calculation are as follows:

[0062] ΔL i =k i ·K p ·ΔT m

[0063]

[0064] Wherein n is the total number of integrated joints of the flexible arm, k i is the adjustment coefficient of the tensile driving rope, and Kp is a proportional control gain matrix of the tensioned disk motor 2, ΔT m is a tension error signal of the integrated joint, θ n is a dynamic bending deformation angle of the flexible arm (the vertex is the collision orientation), K θ is a stiffness coefficient of the flexible arm.

[0065] The collision compliant rigid-flexible hybrid accuracy compensation sub-method specifically comprises the following steps: k , a force-position model C k of the rigid-flexible hybrid manipulator is established by using a linear elastic regression model, a real-time position deformation amount Δx of the end effector of the rigid-flexible hybrid manipulator when the rigid-flexible hybrid manipulator contacts and complies with the target is calculated, an end position compensation amount of the rigid-flexible hybrid manipulator is obtained, and a compensation deflection amount θ k of the integrated joint of the flexible arm is determined. k

[0066] C k : x actual =f(θ L )+f(θ0)+Δx

[0067] Δx=f(θ k )=K -1 *F real

[0068] Wherein, f(θ L ) represents the position of the connection between the rigid arm and the flexible arm, x actual represents the actual position of the end effector, f(θ0) is the theoretical position of the end effector of the flexible arm, f(θ k ) is the end position compensation amount of the rigid-flexible hybrid manipulator, K represents the stiffness matrix of the flexible arm, and F real represents the external stress of the compliant target acting on the flexible arm; θ k is derived from f(θ -1 )=K real *F k , as a kinematic input parameter of the rigid-flexible hybrid manipulator, generates a rigid-flexible hybrid manipulator control instruction, drives the change of the rope length of the flexible arm pose and the tensioned driving rope, and compensates the dynamic bending deformation angle θ n of the flexible arm.

[0069] A canopy visual fusion safety motion positioning method of a rigid-flexible hybrid manipulator, specifically comprising the following steps:

[0070] ​Step one, the rigid-flexible hybrid manipulator is powered on, and the "hand" depth vision module 5 automatically starts image acquisition to obtain real-time image information of the fruit tree canopy;

[0071] Step two, according to the impact risk intensity and impact probability, the multiple heterogeneous objects in the fruit tree canopy are divided into avoidance targets and compliant targets;

[0072] Step three, for avoidance targets, the rigid-flexible hybrid manipulator strictly avoids by the following way

[0073] The "hand" depth vision module 5 inputs the captured fruit tree canopy information as input information into the fruit tree canopy avoidance target semantic segmentation model M d , to obtain the avoidance target spatial point set P ob ; based on the "hand" depth vision module 5, the field space volume region is established, and the avoidance target spatial point set P ob is recursively segmented as the root node until the space volume of each node reaches the preset minimum volume V i ; by traversing the avoidance target spatial point set P ob , the avoidance target dynamic characteristics based on the field space volume region are constructed; taking the avoidance target dynamic characteristics as obstacles, the flexible arm takes the avoidance deformation curvature τ c and the avoidance bending deformation angle θ c as pose constraints, and plans a dynamic path with the initial position of the end effector as the path starting point and the fruit picking position as the path termination point, and dynamically adjusts the error between the actual position and the expected position of the end effector based on the PID controller;

[0074] Step four, for compliant targets, the rigid-flexible hybrid manipulator contacts the compliant target, and the pull / tension disc motor 2 drives the pull / tension driving rope 3 to adjust the torque

[0075] The rigid-flexible hybrid manipulator moves to contact the compliant target, and the pull / tension sensor 12 on the flexible arm real-time obtains the corresponding integrated joint pull / tension T n , and the error between the integrated joint theoretical pull / tension T L is ΔT m , based on ΔT m , the pull / tension of the driving rope 3 is adjusted, and the rope length of the driving rope 3 is dynamically adjusted to form the dynamic bending deformation angle of the flexible arm after contacting the compliant target; wherein, the dynamic bending deformation angle is the collision direction;

[0076] After the rigid-flexible hybrid manipulator contacts the compliant target, a linear elastic regression model is established to establish its force-position model C k, the real-time deformation amount Δx of the end effector of the rigid-flexible hybrid manipulator when the contact compliance target is complied with is calculated, the end compensation amount of the rigid-flexible hybrid manipulator is obtained, and the compensation deflection amount θ of the flexible arm integrated joint is determined k the compensation deflection amount θ of the flexible arm integrated joint is obtained k As the control instruction of the rigid-flexible hybrid manipulator, the dynamic bending angle of the flexible arm is compensated; the flexible arm runs at the compensated angle to drive the end effector to grasp the fruit target.

[0077] The embodiments are preferred embodiments of the present application, but the present application is not limited to the above embodiments, and any obvious improvements, replacements or modifications made by those skilled in the art without departing from the essential content of the present application shall fall within the protection scope of the present application.

Claims

1. A method for safe motion positioning of a rigid-flexible hybrid robotic arm with intracoronascopic vision-tactile fusion, characterized in that, The rigid-flexible hybrid manipulator comprises a rigid arm and a rope-driven flexible arm arranged at the end of the rigid arm, acquires information of a fruit tree canopy through a "hand-mounted" depth vision module, and divides multiple heterogeneous objects in the fruit tree canopy into avoidance targets and compliant targets according to impact risk intensity and impact occurrence probability. For the avoidance targets, the rigid-flexible hybrid manipulator avoids by: inputting the information of the fruit tree canopy into a fruit tree canopy avoidance target semantic segmentation model to output plane coordinates of the avoidance targets in real-time image information of the fruit tree canopy, combining depth values of the real-time image information to obtain a spatial point set of the avoidance targets; The spatial point set of the avoidance targets is taken as a root node, and a field of view spatial body region is recursively segmented, until each node contains a preset minimum volume, and a dynamic feature of the avoidance targets under the field of view spatial body region is constructed by traversing the spatial point set of the avoidance targets. Taking the dynamic characteristics of the target as an obstacle, the flexible arm avoids deformation curvature τ c , avoids bending deformation angle θ c As a posture constraint, the initial position of the end effector is taken as the path starting planning point, and the fruit picking position is taken as the path terminating planning point to plan a dynamic path. For the compliance target, the rigid-flexible hybrid manipulator contacts the compliance target after avoiding the target, and each tension sensor on the flexible arm obtains the corresponding integrated joint tension T n of the flexible arm in real time n The error between T L and the corresponding integrated joint theoretical tension T m of the flexible arm is ΔT m , and the tension of the driving rope is adjusted based on the ΔT m , so as to dynamically adjust the rope length of the driving rope, and form the dynamic bending deformation angle of the flexible arm after contacting the compliance target; after the rigid-flexible hybrid manipulator contacts the compliance target, the dynamic bending deformation angle of the flexible arm is compensated, the flexible arm runs at the compensated angle, and drives the end effector to grasp the fruit target.

2. The intra-canopy visual touch fusion safety motion positioning method of claim 1, wherein, The avoidance targets and the compliant targets are determined by: The main stem, the main branch and the vice main branch in the fruit tree canopy picking area with an impact risk intensity of τ1 and an impact occurrence probability of δ1 are set as vision-based avoidance targets, and the result branch, the result shoot and the nutrient branch in the fruit tree canopy picking area with an impact risk intensity of τ2 and an impact occurrence probability of δ2 are set as tactile-based compliant targets, and τ1 < τ2 and δ2 < δ1.

3. The intra-crown visual touch fusing safety motion positioning method of claim 2, wherein, the avoidance deformation curvature τ c and the avoidance bending deformation angle θ c respectively satisfy: θ c = τ c * L soft wherein W st is the minimum width of the space volume for each node to contain a space volume reaching a preset minimum volume, W s is the safety distance between the flexible arm and the avoidance target, L soft is the length of the flexible arm.

4. The intra-canopy visual touch fusion safety motion positioning method of claim 1, wherein, The dynamic bending deformation angle of the compensation flexible arm is specifically: a linear elastic regression model is used to establish a force-position model C of the rigid-flexible hybrid mechanical arm k The real-time position deformation amount Δx of the end effector of the rigid-flexible hybrid mechanical arm when the rigid-flexible hybrid mechanical arm contacts the compliance target is calculated, the end position compensation amount of the rigid-flexible hybrid mechanical arm is obtained, and the compensation deflection amount θ of the flexible arm integrated joint is determined k The compensation deflection amount θ of the flexible arm integrated joint is sent to the joint controller of the flexible arm integrated joint k The dynamic bending deformation angle of the compensation flexible arm is used as a control instruction of the rigid-flexible hybrid mechanical arm.

5. The intra-canopy visual touch fusion safety motion positioning method of claim 4, wherein, The force-position model C of the rigid-flexible hybrid manipulator k is: k : x actual = f(θ L ) + f(θ0) + Δx, the flexible arm integrated joint compensates the deflection amount θ k , which satisfies: Δx = f(θ k ) = K -1 * F real ; wherein f(θ L ) represents the position of the connection between the rigid arm and the flexible arm, x actual represents the actual position of the end effector, f(θ0) is the theoretical position of the end effector of the flexible arm, f(θ k ) is the end position compensation amount of the rigid-flexible hybrid manipulator, K represents the stiffness matrix of the flexible arm, and F real represents the external stress of the compliant target acting on the flexible arm.

6. The intra-canopy visual touch fusion safety motion positioning method of claim 5, wherein, f(θ k ) = K -1 * F real θ k is derived as a rigid-flexible hybrid robot kinematics input parameter, generates rigid-flexible hybrid robot control instructions, drives the flexible arm pose and the rope length change of the pull / tension driving rope, and compensates for the dynamic bending deformation angle of the flexible arm.

7. The intra-canopy visual touch fusion safety motion positioning method of claim 1, wherein, The vertex of the dynamic bending deformation angle of the flexible arm is the collision orientation, which is obtained by the following way: the tensile / tension force sensor on the flexible arm detects the tensile / tension force change ΔT of the tensile / tension driving rope in real time i , combined with the rope direction vector of the tensile / tension driving rope Calculate the touch external force Through force balance and geometric relationship of the flexible arm, calculate the centroid position change amount ΔC of the flexible arm soft , establish the position relationship between the centroid position of the flexible arm and the collision point, that is, the collision orientation.

8. The intra-crown visual touch fusing safety motion positioning method of claim 7, wherein, The touch external force Satisfies: The centroid position change amount ΔC of the flexible arm soft Satisfies: The position relationship between the centroid position of the flexible arm and the collision point is P c (x c ,y c )=P0(x0,y0)+ΔC soft , wherein n is the total number of integrated joints of the flexible arm, P0(x0,y0) is the initial point of the centroid position of the flexible arm, k i is the adjustment coefficient of the tension driving rope.

9. The intra-canopy visual touch fusion safety sports positioning method according to claim 1, wherein, Dynamic path planning is performed to obtain a desired position of an end effector, and a PID controller is used to dynamically adjust an error between an actual position and the desired position of the end effector to ensure the motion accuracy of the flexible arm.

10. A device for implementing the method of claim 1-9, wherein, The "hand-mounted" depth vision module (5), the tensile / drawing disc motor (2), the tensile / drawing driving rope (3), the tensile force sensor (12) and the visual servo controller are included; the "hand-mounted" depth vision module (5) is installed on the upper part of the end effector (6) of the flexible arm (7); the tensile / drawing disc motor (2) is k in total, where k is an even number, the tensile / drawing disc motor (2) is divided into two groups, and is installed on one side of the end of the flexible arm (7); each tensile / drawing disc motor output terminal (9) is fixed with the head end of one tensile / drawing driving rope (3), the tail end of the tensile / drawing driving rope (3) is fixed to the tail of the end effector (6) after sequentially passing through a plurality of flexible arm integrated joint body deformation housings (10), and the tensile force sensor (12) is installed at the tensile / drawing disc motor output terminal (9), the connection between the tensile / drawing driving rope (3) and the flexible arm integrated joint body deformation housing (10); the "hand-mounted" depth vision module (5), the tensile / drawing disc motor (2) and the tensile force sensor (12) all communicate with the visual servo controller.

Images