A user-friendly real-time control method for a gesture-based redundant soft robot

Abstract

The present invention relates to the field of soft robot control methods, and in particular to a user-friendly real-time control method for redundant soft robots based on gestures. The method includes the following steps: S1: Generate an interaction space within a certain range through an optical hand tracking module, and in real-time track the hand state within the interaction space to obtain gesture information; S2: When the gesture information changes instantaneously, use the gesture attitude sequence in the form of smoothed quaternions as the input signal, and obtain a drive command according to the input signal; S3: Based on the drive command, calculate the specified attitude of the robot tip, and according to the specified attitude, linearly drive the tip of the soft robot to make a turning action corresponding to the hand movement. The present invention can not only provide an intuitive and natural interaction experience, but also ensure the stability and accuracy of the robot movement.

Description

Technical Field

[0001] The present invention relates to the field of soft robot control methods, and in particular to a user-friendly real-time control method for redundant soft robots based on gestures. Background Art

[0002] Soft robots are usually composed of materials with soft touch, and this structural characteristic is very ideal in many practical applications, such as scenarios of robot-assisted minimally invasive surgery, human-computer interaction, and object manipulation. However, different from traditional robotic arms with rotary joints, generally, externally driven soft robots are driven by cables or tendons. If we want to fully control the tip of a soft robot, an inverse kinematics (IK) mapping from the actuator space to the task space is required, while considering the geometrically associated configuration space. However, for a soft robot with two articulated segments and only the tip position given, the result often has multiple inverse solutions, and only by strictly defining the required tip direction (i.e., tip pose) can this problem be solved. Despite these difficulties, controlling with a joint input device with multiple dimensions is relatively simple. For example, traditional soft robots use different devices, providing a simple method for omnidirectional movement of the robot tip. However, it also has defects. Once the robot has more degrees of freedom than the input device, controlling the robot with a single input device will become complex and unfriendly to users. And if additional constraints are needed during the operation, it will exceed the capabilities of these devices. Although the Leap Controller (LMC) (an optical hand tracking module) can provide complex gesture inputs for rigid robots and theoretically can achieve simultaneous control of multiple segments of redundant soft robots, but there is no relevant research at present. Therefore, it is necessary to focus on researching a new scheme for steering redundant multi-segment soft robots based on a new controller, laying a theoretical and methodological foundation for the control system of user-friendly redundant soft robots. Summary of the Invention

[0003] Aiming at the problems of poor intuitiveness, high learning cost, complex and inapplicable input devices in the steering control of existing multi-segment redundant soft robots, a teleoperation control method based on the LMC optical hand tracking module is proposed. The method includes the following steps:

[0004] S1: Generate a certain range of interaction space through the optical hand tracking module, and in real-time track the hand state within the interaction space to obtain gesture information;

[0005] S2: When the gesture information changes instantaneously, use the gesture pose sequence in the form of a smoothed quaternion as the input signal, and obtain a drive command according to the input signal;

[0006] S3: Based on the driving instruction, calculate the specified posture of the robot's tip. According to the specified posture, the tip of the cable-driven soft robot makes a turning motion corresponding to the hand movement;

[0007] Among them, the positional relationship between the palm coordinate system {P} and the coordinate system {L} of the optical hand tracking module is expressed as and the positional relationship between the tip coordinate system {T} of the robot and the world coordinate system {W} is expressed as forming a certain mapping relationship formula:

[0008] When all fingers bend towards the palm, the mapping relationship formula is:

[0009] When at least one finger is in the straight state, the mapping relationship formula is:

[0010] Among them, represents the homogeneous transformation matrix of {A} relative to {B}, Γ is the position scaling matrix, and G sign is the control parameter including various gesture patterns.

[0011] Preferably, the method for obtaining the gesture information includes:

[0012] Using the Kalman filter as a recursive estimator to update the current estimated state x from the prior estimated state combined with the current noise measurement t :

[0013]

[0014] Among them, K t represents the Kalman gain matrix, z t represents the measurement vector, and H represents the observation matrix;

[0015] According to the current estimated state x t , obtain the gesture information.

[0016] Preferably, the calculation method of the Kalman gain matrix K t is:

[0017]

[0018] Among them, represents the prior error covariance matrix, F t represents the state transition matrix applied to the state vector at the previous time step.

[0019] Preferably, the method for obtaining the driving instruction includes:

[0020] Define the hand pose Return an n×n rotation matrix for representing the normal direction of the palm, and use a low-pass filter with spherical linear interpolation to create a sequence of poses q′ in quaternion form:

[0021] q':=SLERP(q 0 ,q 1 ,h LPF )

[0022] where SLERP(·) represents the spherical linear interpolation function, q0 and q1 are the endpoints of the preset interpolation range, and h LPF represents the interpolation parameter in [0,1], ‖q 0 ,q 1 ‖ D represents the angular distance between the quaternions q 0 and q 1 in radians, h RL ∈[0,1] represents the range limit, and b low is the low-pass parameter;

[0023] Use the pose sequence q′ as the input signal to obtain a drive command according to the input signal.

[0024] Preferably, the calculation method for the specified pose of the robot tip includes:

[0025] Define the drive command q [t] at the instantaneous moment t as:

[0026] q [t] =[q 0 ,q c T

[0027] where q c represents the line drive command;

[0028] Based on the drive command q [t] , obtain the homogeneous transformation matrix of the robot tip coordinate system {T} relative to the world coordinate system {W}:

[0029]

[0030] where the soft robot is vertically mounted on the linear slider, represents the position change of the linear slider, and:

[0031]

[0032] ​

[0033] Based on the linear tip velocity in the world coordinate system {W} Calculate the corresponding driving speed of the robot tip as

[0034]

[0035] where J ∈ R 3×7 is the wide Jacobian matrix of the redundant robot, represents the right pseudo-inverse operator, represents the null space of the projection J;

[0036] Use the damped least squares method to represent the optimization relationship between the change in driving speed Δq over a certain period of time and the change in tip position Δx over a certain period of time: s.t. A·△q ≤ b and q

[0037]

[0038] min ≤ q ≤ q max ,

[0039] where, represents the desired instantaneous change in the tip attitude, Ω i (Δq c ) represents the change in the optimized tip attitude, λ represents a non-zero damping constant, A = diag[10 - 3 , 1, 1, 1, 1, 1, 1], b = 10 - 2 ·ones(7, 1),

[0040] By solving the optimization relationship, update to obtain the specified attitude of the robot tip.

[0041] Preferably, the calculation method of the position scaling matrix Γ is:

[0042]

[0043] where I 3×3 represents the 3×3 identity matrix, 0 1×3 represents the 1×3 zero matrix, represents the z-axis parameter in the position data of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, c x , c y and c z respectively represent the adjustment parameters on the x-axis, y-axis and z-axis;

[0044] The position relationship of the tip coordinate system {T} of the robot relative to the world coordinate system {W} is expressed

[0045]

[0046] wherein, represents the rotation matrix of the tip coordinate system {T} relative to the world coordinate system {W}, represents the position of the tip coordinate system {T} relative to the world coordinate system {W}, and <·> represents the normal direction operator of the rotation matrix;

[0047] The position relationship of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module is expressed

[0048]

[0049] wherein, represents the rotation matrix of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, <·> represents the normal direction operator of the rotation matrix, represents the position of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the x-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the y-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the z-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, and 01×3 represents a 1×3 zero matrix.

[0050] Preferably, the gesture patterns include a palm pattern, a V-shaped pattern, a thumbs-up pattern, and a fist pattern; the method for digitally representing the gesture patterns using a Boolean vector includes: defining the straight state of each finger as a Boolean value 1, and the curled state of the finger towards the palm as a Boolean value 0, then the vector representation of the palm pattern is: G 1 =[1,1,1,1,1] T , the vector representation of the V-shaped pattern is: G 2 =[0,1,1,0,0] T , the vector representation of the thumbs-up pattern is: G 3 =[1,0,0,0,0] T , the vector representation of the fist pattern is: G 4 =[0,0,0,0,0] T .

[0051] Preferably, the control parameter G for multiple gesture patterns sign has a matrix representation as follows:

[0052] In the palm mode, the control parameter G for multiple gesture patterns sign is:

[0053]

[0054] In the V-shaped mode, the control parameter G for multiple gesture patterns sign is:

[0055]

[0056] In the thumb-up mode, the control parameter G for multiple gesture patterns sign is:

[0057]

[0058] where, I 3×3 represents a 3×3 identity matrix, 0 1×3 represents a 1×3 zero matrix, represents the rotation matrix of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, and <·> represents the normal direction operator of the rotation matrix.

[0059] Preferably, the kinematic model of the wire-driven soft robot is constructed based on compressible curvature, where the configuration of the (n - 1)-th decoupled segment is composed of the bending angle bending direction and curvature as follows:

[0060]

[0061]

[0062]

[0063] where, s n-1 represents the compression length of the (n - 1)-th segment, ζ is the installation angle between adjacent segments, r represents the distance between the nerve axis and the cable axis, f k,i represents the force exerted by the i-th cable on the k-th segment, represents the angle in the bending direction of the (n - 1)-th segment, represents the angle in the bending direction of the n-th segment.

[0064] The present invention also provides a user-friendly real-time control system for a redundant soft robot based on gestures, which implements the user-friendly real-time control method for a redundant soft robot based on gestures, and includes the following modules:

[0065] A gesture information acquisition module, which is used to generate an interaction space of a certain range through an optical hand tracking module, and in real-time track the hand state within the interaction space to obtain gesture information;

[0066] A driving instruction calculation module, which is used to, when the gesture information changes instantaneously, use the gesture posture sequence in the form of a smoothly processed quaternion as an input signal, and obtain a driving instruction according to the input signal;

[0067] A robot tip driving module, which is used to, based on the driving instruction, calculate the specified posture of the robot tip, and according to the specified posture, linearly drive the tip of the soft robot to make a turning action corresponding to the hand movement;

[0068] Among them, the positional relationship between the palm coordinate system {P} and the coordinate system {L} of the optical hand tracking module is expressed as and the positional relationship between the tip coordinate system {T} of the robot and the world coordinate system {W} is expressed as to form a certain corresponding relational expression:

[0069] When all fingers are bent towards the palm, the relational expression is:

[0070] When at least one finger is in a straight state, the relational expression is:

[0071] Among them, represents the homogeneous transformation matrix of {A} relative to {B}, Γ is the position scaling matrix, and G sign is a control parameter including various gesture modes.

[0072] The above technical solution of the present invention has the following advantages compared with the prior art:

[0073] 1) Based on the master-slave control system of LMC, the present invention controls the redundant soft robot through gestures, uses the inverse kinematics fast solution scheme from the task space to the configuration space, and through real-time optimization, timely compiles the gesture and motion information into executable commands of the robot system, and then controls the turning action of the line-driven redundant soft robot.

[0074] 2) Based on the multi-modal gesture-robot mapping mechanism, the robot can execute complex postures according to the semantics of single-handed gestures and wrist movements, and can complete complex constrained movements, such as position changes on a specific axis or locking the proximal segment through real-time optimized gesture changes, solving the problems of poor intuitiveness and inability to control complex movements when current traditional input devices are used for redundant soft robots.

[0075] 3) An innovative filtering algorithm (Kalman filter and low-pass filter with spherical linear interpolation) is proposed. Based on the proposed filtering algorithm, the present invention allows continuous three-dimensional gesture input to smooth the seamless switching of robot movements and modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to make the content of the present invention easier to be clearly understood, the following further details the present invention according to specific embodiments of the present invention in conjunction with the drawings, where

[0077] Figure 1 is a schematic diagram of a user controlling a soft robot using gestures;

[0078] Figure 2 is a flowchart of the implementation of a user-friendly real-time control method for a redundant soft robot based on gestures provided in an embodiment of the present invention;

[0079] Figure 3 in (a) represents a schematic diagram of the structural dimensions and assembly process of a two-segment wire-driven soft robot, and (b) represents a segmented constant curvature model of the soft robot considering accidental axial compression and cascaded compression of each segment caused by cable drive;

[0080] Figure 4 is a schematic diagram of a three-dimensional hand bone model reconstructed in the interaction space generated by an optical hand tracking module;

[0081] Figure 5 is a schematic diagram of four gesture modes for controlling the tip of a wire-driven soft robot;

[0082] Figure 6 is the experimental verification result of the tracking accuracy of the optical hand tracking module on the hand pose after filtering;

[0083] Figure 7When (a) in the figure represents the palm gesture mode and the direction points forward and downward, it shows the control simulation result diagram of the cable-driven soft robot; when (b) represents the right side of the user view, it shows the control simulation result diagram of the cable-driven soft robot; when (c) represents the V-shaped gesture mode and the direction tilts to the left, it shows the control simulation result diagram of the cable-driven soft robot; when (d) represents the gesture with the thumb up forcing the robot to maintain a straight configuration and only the Z-axis direction is movable, it shows the control simulation result diagram of the cable-driven soft robot; when (e) and (f) both represent the condition of freezing the proximal segment and using the position of the fist to control the position of the distal tip, it shows the control simulation result diagram of the cable-driven soft robot;

[0084] Figure 8 In the figure, (a) represents the error result generated by using the palm mode to manipulate the robot tip to draw an "∞" pattern in the air, and (b) represents the error result generated by using the palm mode to control the tip to rotate around the fixed point [-15, -20, 140]mm in the world coordinate system {W};

[0085] Figure 9 In the figure, (a) represents the transition transformation result of manipulating the robot tip by switching between the palm mode and the fist mode, (b) represents the transition transformation result of manipulating the robot tip by switching between the palm mode and the thumb-up mode, (c) represents the transition transformation result of manipulating the robot tip by switching between the palm mode and the V-shaped mode, and (d) represents the transition transformation result of continuously switching the single mode to manipulate the robot tip;

[0086] Figure 10 In the figure, (a) represents the schematic diagram of controlling the soft robot to perform random steering motion for about 130s, and (b) represents the update frequency of the inverse kinematics and motor drive data for each cyclic motion;

[0087] Figure 11 In the figure, (a) and (b) are respectively the experimental results of manipulating the robot end to reach any position in the working space and hover and remain stationary and change the attitude, and (c) and (d) are both the experimental results of controlling the end to move in the three-dimensional space;

[0088] Figure 12 It is the evaluation result of the motion accuracy of the robot in free space through key point fitting. Detailed implementation mode

[0089] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments given are not intended to limit the present invention.

[0090] Embodiment 1

[0091] Reference Figures 1 - 2 As shown, aiming at the problem that it is difficult to fully control the turning of a multi-segment redundant soft robot, a method for controlling the turning of a two-segment soft robot using gestures based on an optical hand tracking module (LMC) is proposed. This method includes the following steps:

[0092] S1: Generate an interaction space within a certain range through the optical hand tracking module, and continuously track the hand state within the interaction space to obtain gesture information;

[0093] S2: When the gesture information changes instantaneously, use the gesture pose sequence in the form of a smoothed quaternion as the input signal, and obtain a driving instruction according to the input signal;

[0094] S3: Based on the driving instruction, calculate the specified pose of the robot tip, and according to the specified pose, linearly drive the tip of the soft robot to make a turning motion corresponding to the hand movement.

[0095] In this embodiment, a two-segment linearly driven soft robot (CDSR) is used. Its structural dimensions and assembly process are as shown in Figure 3 (a). The total weight of this soft robot is 5 grams, and other physical properties are shown in Table 1, where E is the Young's modulus; K a is the axial stiffness; K T is the bending stiffness; L k is the original length of the k-th segment. Each segment of this soft robot is driven by 3 cables. Since the driving cables of the distal segment need to pass through the proximal segment, the proximal segment has 6 cable channels, but only 3 of them are used for its own driving. The design of cable coupling is beneficial to the compactness and miniaturization of the robot structure, but real-time drive decoupling is required for effective control. In addition, a main hollow channel is reserved in the center of each segment.

[0096] Table 1

[0097]

[0098] The soft robot is modeled based on the compressible curvature hypothesis, as shown in Figure 3 (b). Considering the accidental axial compression and cascading compression of each segment caused by cable driving, an improved piecewise constant curvature (PCC) model is provided. In the non-expandable PCC model, the k-th segment can be geometrically described in the configuration space, including the bending angle θ k , the bending direction φ k and the curvature κ k , but this method ignores the case of a cable-driven soft robot with significant strain caused by the driving mechanism itself and segment coupling. In the compressible curvature model, the decoupled configuration of the (n - 1)-th segment is determined by the bending angle Bending direction and curvature Composition:

[0099]

[0100]

[0101]

[0102] where s n-1 represents the compression length of the (n - 1)-th segment, ζ is the installation angle between adjacent segments, r represents the distance between the nerve axis and the cable axis, f k,i represents the force exerted by the i-th cable on the k-th segment, represents the angle in the bending direction of the (n - 1)-th segment, represents the angle in the bending direction of the n-th segment. Since the non-extensible cable always maintains tension during operation, the tension of the cable is similar to the motor drive based on Hooke's law, i.e., Δf k,i ∝Δq k,i . The decoupling rule for the most distal segment is given by this equation: where ψ n =[θ n φ n κ n T .

[0103] Specifically in S1, the LMC uses two monochromatic infrared cameras and three infrared LEDs to monitor the surrounding approximately hemispherical area as the interaction space and reconstruct the three-dimensional bone model of the hand within this interaction space (as Figure 4 ). It can provide rich data of the moving hand, including the palm, fingers, and speed, etc. Although the human hand is very dexterous and has many degrees of freedom, only a few degrees of freedom are monitored and used as input signals for steering control. And the stable refresh of the LMC at about 115 frames per second also provides a reliable guarantee for the real-time input of the responsive master-slave mechanism.

[0104] To provide an intuitive manipulation experience, the movement of the robot tip should correspond to the movement of the palm of either hand, that is, the palm coordinate system {P} relative to the LMC coordinate system {L} should correspond to the tip coordinate system {T} relative to the world coordinate system {W}. Since the movement range of the hand is wider than that of the robot tip, the positioning of the hand should be scaled down proportionally while the tip posture remains unchanged. Define the position relationship between the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module as and the position relationship between the tip coordinate system {T} of the robot relative to the world coordinate system {W} as​ Form a certain mapping relationship:

[0105] When all fingers are bent towards the palm, the mapping relationship is:

[0106] When at least one finger is in the straight state, the mapping relationship is:

[0107] Among them, represents the homogeneous transformation matrix of {A} relative to {B}, Γ is the position scaling matrix, and G sign is the control parameter including various gesture patterns.

[0108] It is challenging to manipulate redundant soft robots using a mouse or traditional tactile devices, especially when defining some special movements or moving in the null space. Using a series of simple shortcut gestures to define different constraints for the robot when tracking the trajectory of the input device realizes multi-mode control and greatly facilitates the remote operation of multi-segment soft robots. In this new control method, if a finger remains straight, it is considered extended, defined as the boolean value 1. On the contrary, when a finger curls towards the palm, it is considered not extended, defined as the boolean value 0. On this basis, gestures are digitized using a boolean vector:

[0109]

[0110] Among them

[0111] Although theoretically there are up to 120 gesture combinations available for a hand, the selected gestures should be comfortable and easy to remember. For example, the gesture would be a torture. As Figure 5 shown, in addition to the general palm mode, three additional simple gesture patterns are pre-assigned for the robot movement under different constraints as its shortcuts. The vector representation of the palm mode is: G 1 = [1, 1, 1, 1, 1] T , the vector representation of the V-shaped mode is: G 2 = [0, 1, 1, 0, 0] T , the vector representation of the thumb-up mode is: G 3 = [1, 0, 0, 0, 0] T , the vector representation of the fist mode is: G 4 = [0, 0, 0, 0, 0] T

[0112] Specifically, the control parameter G including various gesture patternssign The matrix representation is as follows:

[0113] In the Palm mode, the pose (position and orientation) of the palm coordinate system {P} is used to generally control the pose of the tip coordinate system {T}, and the torsional motion of the palm (i.e., rotation around the palm center) is not considered during the control process. At this time, the control parameter G including various gesture modes sign :

[0114]

[0115] In the V-sign mode, the pose of the palm coordinate system {P} is used to constraint-control the pose of the tip coordinate system {T}, and the tip position is constrained to move within the plane of , where Z is the only position variable. At this time, the control parameter G including various gesture modes sign :

[0116]

[0117] In the Thumbs-up mode, the position of the palm coordinate system {P} is used to constraint-control the position of the tip coordinate system {T}, and the tip position is constrained to move within the plane of , where Z is the only position variable. At the same time, the tip orientation is fixed to be the same as the world coordinate system {W}, that is, it becomes a straight-line configuration. At this time, the control parameter G including various gesture modes sign :

[0118]

[0119] where, I 3×3 represents a 3×3 identity matrix, 0 1×3 represents a 1×3 zero matrix, represents the rotation matrix of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, and <·> represents the normal direction operator of the rotation matrix (i.e., only the normal of the palm is extracted by using the function metaData.hands(1).palm.normal()), so as to exclude the influence of the torsional motion of the palm along its normal direction.

[0120] In the Fist-pump mode, the pose of the palm coordinate system {P} is used to constraint-control the pose of the tip coordinate system {T}. During this process, the proximal segment is locked and remains unchanged, which is the latest updated pose before the fist is detected. In other words, moving the fist in the air will only control the distal segment, while the proximal segment is frozen.

[0121] The position scaling matrix Γ is used to magnify or reduce the movement of the hand relative to the robot. This matrix should be customized according to different soft robots to achieve the desired effect, and its calculation method is as follows:

[0122]

[0123] Where, I 3×3 represents a 3×3 identity matrix, 0 1×3 represents a 1×3 zero matrix, represents the z-axis parameter in the position data of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, c x 、c y and c z represent the adjustment parameters on the x-axis, y-axis, and z-axis respectively.

[0124] The position relationship of the tip coordinate system {T} of the robot relative to the world coordinate system {W} is expressed as

[0125]

[0126] Where, represents the rotation matrix of the tip coordinate system {T} relative to the world coordinate system {W}, represents the position of the tip coordinate system {T} relative to the world coordinate system {W}, <·> represents the normal direction operator of the rotation matrix;

[0127] The position relationship of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module is expressed as

[0128]

[0129] Where, represents the rotation matrix of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the normal direction operator of the rotation matrix , represents the position of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the x-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the y-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the z-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, 0 1×3 represents a 1×3 zero matrix.

[0130] Although the LMC module has high tracking accuracy, its tracking data may still have noise when the hand makes unintentional low-frequency jitters. Since these noises follow a Gaussian distribution and their state-space model is linearly defined, a Kalman filter (KF) can be used to estimate the state of the hand position. In S1, the method for real-time tracking of the hand state within the interaction space by the optical hand tracking module to obtain gesture information includes:

[0131] S11: Using the Kalman filter as a recursive estimator, update the current estimated state x from the prior estimated state combined with the current noise measurement : t

[0132]

[0133] where K t represents the Kalman gain matrix, z t represents the measurement vector, and H represents the observation matrix;

[0134] S12: Obtain gesture information according to the current estimated state x t .

[0135] In S11, the calculation method of the Kalman gain matrix K t is as follows:

[0136]

[0137] where represents the prior error covariance matrix, F t represents the state transition matrix applied to the state vector at the previous time step.

[0138] In S2, the method for obtaining the drive instruction includes:

[0139] Define the hand pose Return an n×n rotation matrix for representing the normal direction of the palm, and use a low-pass filter with spherical linear interpolation to create a pose sequence q′ in quaternion form:

[0140] q':=SLERP(q 0 ,q 1 ,h LPF )

[0141] where SLERP(·) represents the spherical linear interpolation function, q0 and q1 are the end points of the preset interpolation range, and h LPF represents the interpolation parameter in [0,1], ‖q 0 ,q 1 ​‖ D Denote the quaternion \(q\) in radians 0 and \(q\) 1 The angular distance between them, \(h\) RL \(\in[0,1]\) represents the range limit, \(b\) low is the low-pass parameter;

[0142] Take the attitude sequence \(q'\) as the input signal and obtain the drive command according to the input signal.

[0143] In S3, the calculation method of the specified attitude of the robot tip includes:

[0144] As Figure 1 shown, the Soft-Bodied Robot is vertically mounted on the Linear slide to form an insertion-retraction motion (i.e., \(q\) 0 \(\geq0\)). Define the drive command \(q\) at the instant \(t\) as: [t]

[0145] \(q\) [t] \(=[q\) 0 , \(q\) c T

[0146] where \(q\) c represents the linear drive command, \(q\) k,i represents the drive command of the \(i\)-th cable to the \(k\)-th segment.

[0147] Based on the drive command \(q\) [t] , obtain the homogeneous transformation matrix of the robot tip coordinate system \(\{T\}\) relative to the world coordinate system \(\{W\}\):

[0148]

[0149] where represents the position change of the linear slide, and:

[0150]

[0151]

[0152] Based on the linear tip velocity in the world coordinate system \(\{W\}\), calculate the corresponding drive velocity of the robot tip as

[0153]

[0154] where \(J\in R\) 3×7 is the wide Jacobian matrix of the redundant robot, \(R\) 3×7 ​​Represents a 3×7 dimension, Represents the right pseudo-inverse operator, Represents the null space of the projection J;

[0155] In a discrete-time system, the change Δx in the tip position over an infinitesimal time period can be approximated based on the infinitesimal drive change Δq, such that Due to system redundancy, there may be multiple sets of solutions. Therefore, the damped least squares method is used to represent the optimization relationship between the change in drive speed Δq over a certain time period and the change in tip position Δx over a certain time period:

[0156]

[0157] s.t. A·Δq ≤ b and q min ≤ q ≤ q max ,

[0158] where, Represents the desired instantaneous change in the tip attitude, Ω i (Δq c ) represents the change in the optimized tip attitude, λ represents a non-zero damping constant, A = diag[10 - 3 , 1, 1, 1, 1, 1, 1], b = 10 - 2 ·ones(7, 1), Similarly, the fist gesture mentioned above can be used to formulate an optimization problem by converging the tip and midpoint positions and the changes in joint motions:

[0159]

[0160] s.t. A·Δq ≤ b and q min ≤ q ≤ q max ,

[0161] where J M Δq represents the midpoint position calculated kinematically.

[0162] By solving the optimization relationship, the latest target tip pose change guides the real-time output of the required drive, causing the soft robot to bend into a suitable configuration and then driving the tip of the robot to the specified attitude.

[0163] The control steering performance of the method proposed in the present invention is verified through a hardware-in-the-loop simulation experiment. The CDSR simulator based on MATLAB is used. The simulator takes cable and slider drive as inputs and demonstrates two-segment soft robots in real time in a 3D animation. In this experiment, the LeapC (an API) compiled by the mex file of Matleap in the Matlab environment using Microsoft Visual C++ 2019 is used to access the hand tracking data of the LMC.

[0164] To quantitatively evaluate the performance of the method described in the present invention, two error values are introduced, denoted by as the tip position error ( represents the desired tip position, represents the actual value of the simulator tip), and denoted by as the attitude error, where and are the position and attitude of the simulator tip calculated through forward kinematics from q [t] .

[0165] To verify that the filtered LMC has reliable tracking accuracy in the hand pose, an object of known size is selected as a reference, and its contour is traced with a finger. Here, we select an A4 paper for testing. The test results are as shown in Figure 6 . The paths of Indexpath and Palmpath shown in the figure respectively reflect the and tracking performance, and the angle graph shows the change in the normal direction of the palm over the tracking time. It can be seen that the sensed and can well fit the size of the paper edge, and the finger tracking accuracy is higher than that of the palm. This is because the inherent LMC tracking algorithm relies on tracking multiple features to calculate the palm position, and it is difficult for the user to determine the exact position of the palm, so there are certain differences in the measured paths.

[0166] On the simulation platform, the cable-driven soft robot is controlled based on real-time gesture control. The manipulation results are as shown in Figure 7 , indicating that the preset basic gestures can all be well completed.

[0167] Using general control (palm mode) to manipulate the robot tip to draw an "∞" pattern in the air, as shown in Figure 8 (a). The error graph shows that both e x,sim and e Θ,sim are relatively small, within the ranges of 1 mm and 2° respectively. As shown in Figure 8As shown in (b), in the world coordinate system {W}, with [-15, -20, 140] mm as the fixed point, the same general control mode is used to control the tip to rotate around this point (i.e., zero-space motion), and the error map is also satisfactory, with the deviation within the acceptable range.

[0168] To verify the excellent transition performance during the quick gesture transformation process, an attempt was made to change the gesture while continuously changing the pose of the robot tip to switch the input mode, as Figure 9 shown. The dashed line represents the filtered pose input of the gesture, and the solid line represents the pose of the robot tip calculated by forward kinematics in the joint space. It can be clearly seen that even if there is a sudden change in the gesture, a seamless and smooth transition of the mode can be achieved.

[0169] As a master-slave control system, low latency is crucial for the success of its task execution. To ensure the open-loop accuracy, servo motors (Dynamixel XM430-W350, robot IS) with 4096 pulses / revolution position feedback were used at the joints, and a hand-eye vision measurement platform was built with an Intel D435 RGB-D camera to collect experimental data for evaluation. To obtain the pointing direction of the tip, a marker A was attached to the robot, and its pointing direction can be approximately calculated as where ∥x A -x B ∥ * represents the expression of the distance between A and B in the {W} coordinate system on the subscript plane or axis. To verify the related performance of the method described in the present invention, the soft robot was subjected to random steering control for about 130 s, as Figure 10 shown, where Figure 10 (a) is a screenshot display of the robot's motion, Figure 10 (b) is the update frequency of each cycle, including inverse kinematics and motor drive data updates. It can be clearly seen that it is updated at a frequency of about 11 Hz, which means that there is only a delay of about 0.09 s between the hand motion and the robot motion, having a stable steering performance with fast response.

[0170] To verify the control accuracy of the present invention, the simulation value was used as the true value and compared with the experimental value. The experimental results are as Figure 11 shown, Figure 11 where (a) and (b) are the experimental results of manipulating the robot end to reach any position in the workspace and hover and hold still and change the pose respectively Because the hand will tremble in mid-air without support, and the acceptable error map reflects the excellent performance of the input filtering, Figure 11Figures (c) and (d) are the experimental results of the movement of the manipulation end in three-dimensional space. It can be seen that the simulated path and the actual path are similar. Under steady state, the standard positioning error can be within 4 mm (reachable within the working space), and the pointing direction error can be within 5 degrees. This error can be explained by the differences between the simulator and the prototype. The simulator is developed based on a specific soft robot model, so the real prototype may not accurately reveal the predictions of the model. In addition, the movement accuracy of the robot in free space is also evaluated by key point fitting. The schematic diagram is as shown in Figure 12 As shown. After subsequent data processing, it is found that no matter what gesture is used to control the movement of the robot, the deviation on each axis is within 2 mm. It can be seen that the present invention can accurately reproduce the predicted movement of the simulator.

[0171] Embodiment 2

[0172] The present invention also provides a gesture-based user-friendly real-time control system for redundant soft robots, which realizes the gesture-based user-friendly real-time control method for redundant soft robots as described in Embodiment 1. The control system includes the following modules:

[0173] A gesture information acquisition module, which is used to generate an interaction space within a certain range through an optical hand tracking module, and real-time track the hand state within the interaction space to obtain gesture information;

[0174] A driving instruction calculation module, which is used to, when the gesture information changes instantaneously, use the gesture attitude sequence in the form of smoothed quaternions as the input signal, and obtain a driving instruction according to the input signal;

[0175] A robot tip driving module, which is used to calculate the specified posture of the robot tip based on the driving instruction, and according to the specified posture, linearly drive the tip of the soft robot to make a turning action corresponding to the hand movement;

[0176] Among them, the positional relationship between the palm coordinate system {P} and the coordinate system {L} of the optical hand tracking module is represented by and the positional relationship between the tip coordinate system {T} of the robot and the world coordinate system {W} is represented by to form a certain corresponding relational expression:

[0177] When all fingers are bent towards the palm, the relational expression is:

[0178] When at least one finger is in a straight state, the relational expression is:

[0179] Among them, represents the homogeneous transformation matrix of {A} relative to {B}, Γ is the position scaling matrix, and G signAre control parameters including multiple gesture patterns.

[0180] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, so that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0182] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0184] Obviously, the above embodiments are merely examples given for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is not necessary and impossible to exhaustively list all the implementation manners here. And the obvious changes or variations derived therefrom are still within the protection scope of the present invention.

Claims

1. A user-friendly real-time control method for redundant soft robots based on gestures, characterized in that: The following steps are involved: S1: Generate an interactive space of a certain range through an optical hand tracking module, and track the hand state in real time in the interactive space to obtain gesture information; S2: when the gesture information is changed in real time, the gesture posture sequence in the form of a smoothed quaternion is used as an input signal, and a driving instruction is obtained according to the input signal; S3: Based on the driving instruction, a specified posture of the tip of the robot is calculated, and according to the specified posture, the tip of the wire-driven soft robot makes a turning action corresponding to the hand movement; The positional relationship between the palm coordinate system {P} and the coordinate system {L} of the optical hand tracking module is expressed as and the target homogeneous transformation matrix of the robot's tip coordinate system {T} relative to the world coordinate system {W} Into a certain mapping relationship: When all fingers are bent toward the palm, the mapping relationship is: When at least one finger is in the straight state, the mapping relationship is: in, represents the homogeneous transformation matrix of {A} relative to {B}, Γ is the position scaling matrix, G sign Contains control parameters for multiple gesture modes.

2. The user-friendly real-time control method of redundant soft robots based on gestures according to claim 1, characterized in that: The method for acquiring gesture information includes: The Kalman filter is used as a recursive estimator to estimate the state from the prior combined with the current noisy measurement. Update the current estimated state x t : Among them, K t represents the Kalman gain matrix, z t represents the measurement vector, H represents the observation matrix; According to the current estimated state x t , get gesture information.

3. The user-friendly real-time control method of a redundant soft robot based on gestures according to claim 2, characterized in that: The Kalman gain matrix K t The calculation method is: in, represents the prior error covariance matrix, F t represents the state transfer matrix applied to the state vector at the previous time step.

4. The user-friendly real-time control method of redundant soft robots based on gestures according to claim 1, characterized in that: The method for obtaining the driving instruction includes: Defining hand poses Returns an n×n rotation matrix representing the normal orientation of the palm, using a low-pass filter with spherical linear interpolation to create a quaternion pose sequence q′: q’:=SLERP(q0,q1,h LPF ) Where SLERP(·) represents the spherical linear interpolation function, q0 and q1 are the endpoints of the preset interpolation range, and h LPF represents the interpolation parameter in [0,1], ‖q0,q1‖ D represents the angular distance between the quaternions q0 and q1 in radians, h RL ∈[0,1] represents the range limit, b low is the low-pass parameter; The posture sequence q′ is used as an input signal, and a driving instruction is obtained according to the input signal.

5. The user-friendly real-time control method of redundant soft robots based on gestures according to claim 1, characterized in that: The method for calculating the specified posture of the robot tip includes: The drive command q at the instant t [t] Defined as: what [t] =[q0,q c ] T Among them, q c Indicates line drive instruction; Based on the driving instruction q [t] , we get the homogeneous transformation matrix of the actual output of the robot tip coordinate system {T} relative to the world coordinate system {W}: The soft robot is vertically mounted on a linear slider. represents the position change of the linear slider, and: Based on the linear tip velocity in the world coordinate system {W} The corresponding driving speed of the robot tip is calculated as Among them, J∈R 3×7 is the wide Jacobian matrix of the redundant robot, R 3×7 represents the dimension of 3×7, represents the right pseudo-inverse operator, represents the null space of projection J; Use the damped least squares method to represent the driving speed over a certain period of time The optimal relationship between the change Δq and the change Δx of the tip position within a certain period of time is: s.t.A·△q≤b and q min ≤q≤q max , Where ΔΩ i =Δ[αβγ] t represents the expected instantaneous change in tip attitude, Ω i (Δq c ) represents the change of the optimized tip posture, λ represents the non-zero damping constant, A = diag[10 -3 ,1,1,1,1,1,1], b=10 -2 ones(7,1),q min =-[0,2,2,2,2,2,2] t ,q max =[60,0,0,0,0,0,0] T ; By solving the optimization relationship, the specified posture of the tip of the robot is updated.

6. The user-friendly real-time control method of redundant soft robots based on gestures according to claim 1, characterized in that: The calculation method of the position scaling matrix Γ is: Among them, I 3×3 represents the 3×3 identity matrix, 0 1×3 represents a 1×3 zero matrix, represents the z-axis parameter in the position data of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, c x 、c y and c z Represent the adjustment parameters on the x-axis, y-axis and z-axis respectively; The target homogeneous transformation matrix of the robot's tip coordinate system {T} relative to the world coordinate system {W} in, represents the rotation matrix of the tip coordinate system {T} relative to the world coordinate system {W}, represents the position of the tip coordinate system {T} relative to the world coordinate system {W}, and <·> represents the normal direction operator of the rotation matrix; The positional relationship of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module is expressed as in, represents the rotation matrix of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, <·> represents the normal direction operator of the rotation matrix, represents the position of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the x-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, represents the y-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, Represents the z-axis position parameter of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, 0 1×3 Represents a 1×3 zero matrix.

7. The user-friendly real-time control method of redundant soft robots based on gestures according to claim 1, characterized in that: The gesture modes include palm mode, V-shaped mode, thumbs-up mode and fist mode; The method of digitally representing the gesture pattern using a Boolean vector includes: defining the straight state of each finger as a Boolean value of 1, and defining the curled state of the finger toward the palm as a Boolean value of 0, then the vector representation of the palm pattern is: G1 = [1, 1, 1, 1, 1] T , the vector representation of the V-shaped pattern is: G2 = [0, 1, 1, 0, 0] T , the vector representation of the thumbs-up mode is: G3 = [1, 0, 0, 0, 0] T , the vector representation of the fist pattern is: G4 = [0,0,0,0,0] T .

8. The user-friendly real-time control method of redundant soft robots based on gestures according to claim 7, characterized in that: The control parameters G comprising multiple gesture modes sign The matrix representation of is: In the palm mode, the control parameter G including multiple gesture modes sign : In the V-shaped mode, the control parameter G including multiple gesture modes sign : In the thumbs-up mode, the control parameter G including multiple gesture modes sign : Among them, I 3×3 represents the 3×3 identity matrix, 0 1×3 represents a 1×3 zero matrix, represents the rotation matrix of the palm coordinate system {P} relative to the coordinate system {L} of the optical hand tracking module, and <·> represents the normal direction operator of the rotation matrix.

9. The user-friendly real-time control method of redundant soft robots based on gestures according to claim 1, characterized in that: The kinematic model of the wire-driven soft robot is constructed based on compressible curvature, where the decoupled (n-1)th segment configuration is determined by the bending angle Bending direction and curvature constitute: Among them, s n-1 represents the compressed length of the (n-1)th segment, ζ is the mounting angle between adjacent segments, r represents the distance between the nerve axis and the cable axis, and f k,i represents the force of the ith cable acting on the kth segment, represents the angle in the bending direction of the n-1th segment, Represents the angle in the bending direction of the nth segment.

10. A user-friendly real-time control system for redundant soft robots based on gestures, characterized in that: Implementing the user-friendly real-time control method of a redundant soft robot based on gestures as claimed in any one of claims 1 to 9, comprising the following modules: A gesture information acquisition module, used to generate an interactive space of a certain range through an optical hand tracking module, and to track the hand state in real time within the interactive space to obtain gesture information; A driving instruction calculation module, used to obtain a driving instruction according to a smoothed quaternion gesture sequence as an input signal when the gesture information is changed in real time; A robot tip driving module, for calculating a specified posture of the robot tip based on the driving instruction, and for driving the tip of the soft robot to make a turning motion corresponding to the hand movement according to the specified posture; The positional relationship between the palm coordinate system {P} and the coordinate system {L} of the optical hand tracking module is expressed as and the target homogeneous transformation matrix of the robot's tip coordinate system {T} relative to the world coordinate system {W} Into a certain mapping relationship: When all fingers are bent toward the palm, the mapping relationship is: When at least one finger is in the straightened state, the mapping relationship is: in, represents the homogeneous transformation matrix of {A} relative to {B}, Γ is the position scaling matrix, G sign Contains control parameters for multiple gesture modes.

Images