Surgical robot wearable control equipment based on gesture recognition and control method

Abstract

The invention discloses surgical robot wearable control equipment based on gesture recognition and a control method. The wearable control equipment for the surgical robot is used for detecting motion information of upper limbs of a human body, and comprises an inertial sensor array which is arranged on hand backs, front arms and upper arms of left and right arms of an operator and is used for detecting motion data of the operator; the pressure-sensitive switch array is arranged at the index fingers and the palm centers of the left and right hands of the operator and is used for activating / interrupting the movement of the surgical robot; the wireless data transmitting equipment is used for transmitting the state data of the pressure-sensitive switch to the wireless data acquisition equipment in a one-way manner; the wireless data acquisition equipment is used for receiving the state data sent by the wireless data sending equipment and the motion data sent by the inertial sensor and transmitting the state data and the motion data to the data processing equipment; and the data processing equipment processes the motion information returned by the wireless data acquisition equipment to obtain the motion intention of the operator, and converts the motion intention into the tail end motion information of the surgical robot. The upper limb posture of an operator and the control intention of the operator are detected in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of wearable control devices for surgical robots, and in particular to a wearable control device for surgical robots based on gesture recognition and a control method. Background Art

[0002] Natural Orifice Transluminal Endoscopic Surgery (NOTES) refers to entering the abdominal cavity, mediastinum, thoracic cavity, etc. through natural cavities such as the mouth, esophagus, stomach, colon (rectum), etc. to perform various endoscopic operations, such as abdominal exploration, peritoneal biopsy, liver biopsy, gastrointestinal and intestinal anastomosis, tail resection, cholecystectomy, etc. It does not require opening a hole on the patient's surface to enter the human body for surgical operations. There is no wound on the body surface after surgery, which reduces the patient's pain, relieves the patient's psychological pressure, and also shortens the recovery time after surgery.

[0003] After more than ten years of research and development, more and more medical institutions are gradually carrying out clinical work with NOTES technology. Factors such as the movement limitations of traditional rigid surgical instruments and the increasingly stringent demands for laparoscopic minimally invasive surgery have promoted the development and application of flexible surgical robot system technology represented by continuum. The flexible surgical robot will enter the stomach and other areas through natural cavities such as the narrow and curved upper digestive tract, observe the lesions in real time through the camera of the flexible surgical robot, and then cooperate with the operation with the equipped flexible end to achieve the removal or sampling of the lesions. In order to minimize damage to normal tissues, the flexible surgical robot must have a high degree of bendability, and the flexible end should be more flexible to adapt to the task of removing lesions in different positions and structures.

[0004] Therefore, the flexible end of the existing flexible surgical robot often has more degrees of freedom and is relatively slender overall, so as to reach the lesion through the natural cavity, and achieve bending and opening and closing of the instrument through control wires and other structures. However, this results in insufficient rigidity of the end of the flexible surgical robot, and the movement accuracy is not as good as that of the traditional articulated robotic arm. In addition, the friction generated by the longer control wire will also produce coupling and additional resistance, affecting the movement of the end of the instrument.

[0005] In order to overcome the above defects, the commonly used control methods for flexible surgical robots are mostly master-slave control, which relies on the operator's visual information to make up for the errors in the movement process. In order to achieve master-slave control that better matches the high degree of freedom and nonlinear motion of flexible surgical robots, researchers have proposed a variety of control interfaces and methods, but there are still many shortcomings.

[0006] Existing control methods usually rely on controllers of different structures, and most controller configurations are different from the configuration of the human upper limbs, resulting in such control methods not conforming to the intuitive operation intuition developed by users in the traditional operation process, and therefore require a long period of training before they can be used.

[0007] In addition, most existing control methods require doctors to make specific movements, such as shaking their heads, blinking, moving their feet, or making sounds, to control the robot. These operations will bring additional delays to the control process, further leading to poor hand-eye coordination, and thus resulting in problems such as prolonged surgery time. Moreover, these control solutions do not support enough degrees of freedom and lack information such as depth. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of existing control methods and devices, such as few degrees of freedom, poor hand-eye coordination, and non-intuitive control, and to provide a wearable control device for a surgical robot based on gesture recognition that simultaneously supports multi-degree-of-freedom control, has a simple and lightweight structure, and has the characteristics of dexterity, high precision, and high safety, and is more intuitive to operate. A surgical robot control method and the like are provided. The wearable device can detect the operator's upper limb posture and the operator's control intention in real time.

[0009] One aspect of the present invention provides a wearable control device for a surgical robot based on gesture recognition, which is used to monitor the posture and movement intention of an operator's upper limbs in real time, comprising:

[0010] An inertial sensor array, comprising a plurality of inertial sensors, arranged at the backs of the hands, forearms and upper arms of the operator's left and right arms, to detect the operator's motion data;

[0011] A pressure switch array, including multiple pressure switches, is placed on the index fingers and palms of the operator's left and right hands, respectively, to activate / interrupt the movement of the surgical robot;

[0012] A wireless data transmission device is connected to the pressure-sensitive switch by wire, and is used to unidirectionally transmit the status data of the pressure-sensitive switch to the wireless data acquisition device;

[0013] A wireless data acquisition device, used for wirelessly receiving the status data sent by the wireless data sending device and the motion data sent by the inertial sensor, and transmitting them to a data processing device;

[0014] The data processing device is used to process the motion information returned by the wireless data acquisition device to obtain the operator's motion intention and convert it into motion information of the end of the surgical robot.

[0015] Wherein, the inertial sensor, the wireless data sending device and the wireless data acquisition device adopt Bluetooth wireless communication.

[0016] Among them, the inertial sensors located at the back of the hand and the forearm are close to the operator's wrist joint, and the inertial sensor at the upper arm is located at the midpoint of the upper arm.

[0017] Among them, the inertial measurement unit built into the inertial sensor is used to detect motion information of six degrees of freedom, including pitch angle, roll angle, yaw angle, and three-axis acceleration; the inertial sensor sends data unidirectionally to the data acquisition device, and the refresh rate of the inertial sensor is N2, and N2 is greater than or equal to 100.

[0018] Among them, the refresh rate of the pressure-sensitive switch array is N1, N1 is greater than or equal to 50, pressing it activates the movement of the surgical robot, and releasing it interrupts the movement of the surgical robot.

[0019] Among them, the wireless data sending device is arranged on the wearable back armor, the inertial sensor is installed on the wearable armor, and the pressure switch is fixed inside by two layers of inner and outer covering materials.

[0020] Among them, the wearable control device of the surgical robot based on gesture recognition includes a display device for displaying the upper limb end posture calculated by the data processing device and the movement increment of the surgical robot at the next moment.

[0021] The data processing device is connected to the controlled surgical robot for communication, controls the surgical robot to move accordingly, and realizes continuous master-slave control of the terminal.

[0022] Another aspect of the present invention provides a control method for a wearable control device of a surgical robot based on gesture recognition, comprising the steps of:

[0023] Calculating the operator's current upper limb end posture using the inertial sensor array;

[0024] The data processing device uses the obtained upper limb terminal posture and state feedback information of the pressure-sensitive switch array to map the upper limb movement to the movement of each degree of freedom of the surgical robot based on the upper limb to surgical robot mapping module, converts it into movement information of the surgical robot terminal, and controls the movement of the surgical robot;

[0025] During the control process, the maximum movement distance of each degree of freedom of the surgical robot is restricted by the preset movement space, and the left and right side controls of the surgical robot are activated or not through the pressure switches of the pressure switch array. The robot movement is activated when pressed and interrupted when released.

[0026] The step of calculating the terminal posture of the operator's current upper limb by using the inertial sensor array comprises the following steps:

[0027] Acquiring six-degree-of-freedom data obtained by the inertial sensor;

[0028] Acquire six-degree-of-freedom data of the inertial sensor for a period of time in a stationary state, and remove the offset accumulated over time;

[0029] The six-degree-of-freedom data with the offset removed is filtered out of high-frequency noise and compared with the data at the previous moment, and the current end position of the upper limb is obtained by combining the relationship between the pitch angle, roll angle, yaw angle of the inertial sensor and the upper limb posture.

[0030] The wearable control device of the present invention has a light and compact structure, is composed of an inertial sensor array, a pressure-sensitive switch array and a wireless data transmission device, does not require peripheral auxiliary equipment, can be separated from the surgical robot, and can be remotely controlled.

[0031] The wearable control device of the present invention is based on gesture recognition technology and can recognize the operator's upper limb and arm postures. It is highly intuitive and can match the operator's movement intention with the movement direction of the surgical robot. Therefore, the hand-eye coordination is good, the training cost is low, the movement space is expanded, and the movement accuracy is improved.

[0032] The wearable control device of the present invention introduces additional links such as safe movement space limitation and control interruption to address the potential risks of additional damage or contamination caused by misoperation when performing surgery in a narrow natural cavity of the human body under master-slave control. It alleviates the problem of mismatch between the master and slave hand postures caused by incremental control, improves the problem of operator fatigue after long-term operation, and reduces the probability of accidental injury during surgery or damage to the surgical robot due to obstructed vision, fatigue or other factors.

[0033] The wearable control device of the present invention has good compatibility, does not use mechanical structure to obtain motion information, is light and reliable, and has a large motion space. By combining the wearable kit with the inertial sensor array, it can realize control logic that is intuitive to human movement and can be adjusted for different operators. Through wireless data sending equipment and wireless data acquisition equipment, the operator and the surgical robot can be separated from each other. It controls translation and rotation at the same time, supports multiple degrees of freedom, and can be applied to master-slave control of flexible surgical robots through natural cavities. It also has application value for the control of humanoid or flexible robots in industrial collaboration, disaster relief, and national defense and military fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is an overall schematic diagram of a wearable control device for a surgical robot based on gesture recognition according to an embodiment of the present invention;

[0035] Figure 2a is a schematic diagram of the layout of an inertial sensor array according to an embodiment of the present invention;

[0036] Figure 2b , Figure 2cSchematic diagram of a wearable packaging structure of an inertial sensor according to an embodiment of the present invention;

[0037] Figure 2d The coordinate system definition and each degree of freedom diagram of the inertial sensor according to the embodiment of the present invention;

[0038] Figure 3a is a schematic diagram of the layout of a pressure-sensitive switch array according to an embodiment of the present invention;

[0039] Figure 3b A schematic diagram of a wearable packaging structure of a pressure-sensitive switch according to an embodiment of the present invention;

[0040] Figure 4 This is a flow chart of a surgical robot control method based on gesture recognition according to an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of an upper limb kinematic model according to an embodiment of the present invention;

[0042] Figure 6a A schematic diagram of calculating the end position of an upper limb kinematic model according to an embodiment of the present invention;

[0043] Figure 6b It is a DH parameter definition diagram of the ball joint connecting rod of the upper limb kinematics model according to an embodiment of the present invention;

[0044] Figure 7 A schematic diagram of the upper limb posture solution result according to an embodiment of the present invention;

[0045] Figure 8 Schematic diagram of a mapping model from upper limb end motion to flexible surgical robot motion according to an embodiment of the present invention.

[0046] [Description of Reference Numerals]

[0047] 1- inertial sensor array; 2- pressure switch array; 3- wearable back armor; 4- wireless data transmission device; 5- wireless data acquisition device; 6- data processing device; 7- display device; 8- surgical robot; 11- inertial sensor; 111- inertial measurement unit; 112- inertial sensor housing; 113- inertial sensor wearable kit; 21- first pressure switch; 22- second pressure switch, 23- third pressure switch; 24- fourth pressure switch; 25- fifth pressure switch ; 26-sixth pressure switch; 211-pressure switch sensing unit; 212-coating layer; 81-flexible end; 91-ball joint type upper limb kinematic model; 92-rotational joint type upper limb kinematic model; 921-first rotational joint, 922-second rotational joint, 923-third rotational joint, 924-fourth rotational joint, 925-fifth rotational joint, 926-sixth rotational joint, 927-seventh rotational joint, 928-eighth rotational joint, 929-ninth rotational joint. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] Figure 1 FIG. 1 is an overall schematic diagram of a wearable control device for a surgical robot based on gesture recognition according to an embodiment of the present invention. Figure 1 As shown, a wearable control device for a surgical robot based on gesture recognition includes:

[0050] The inertial sensor array 1 includes a plurality of inertial sensors 11, which are arranged on the back of the left and right arms of the operator, the forearm and the upper arm to detect the operator's motion data; the pressure switch array 2 includes a plurality of pressure switches, which are respectively arranged on the index fingers and palms of the left and right hands of the operator to activate / interrupt the movement of the surgical robot; the wireless data transmission device 4 is connected to the plurality of pressure switches by wire, and is used to send the state data of the pressure switches to the wireless data acquisition device in a unidirectional manner; the wireless data acquisition device 5 is used to wirelessly receive the state data sent by the wireless data transmission device 4 and the motion data sent by the inertial sensor 11, and transmit them to the data processing device 6; the data processing device 6 is used to process the motion information returned by the wireless data acquisition device to obtain the operator's motion intention, and convert it into the motion information of the end of the surgical robot 8. In endoscopic surgery through natural cavities, the surgical robot 8 reaches the lesion through the natural cavity of the human body to operate. The wearable device of this embodiment can collect the posture of the operator's arms and control the surgical robot 8.

[0051] In some embodiments, the gesture recognition-based surgical robot wearable control device also includes a display device 7, which is used to display the upper limb end posture calculated by the data processing device and the motion increment of the surgical robot at the next moment. During the use of the wearable device, the wireless data acquisition device unidirectionally receives data sent by the inertial sensor array and the wireless data sending device; the data processing device is connected to the wireless data acquisition device, receives data and analyzes and solves it, controls the robot after obtaining the operator's upper limb posture and movement intention, and displays it on the corresponding display device, that is, the wireless data acquisition device 5 receives the transmitted data and transmits it to the data processing device 6, and the data processing device 6 calculates the upper limb end posture and the motion increment of the surgical robot 8 at the next moment and displays them through the display device 7.

[0052] In some embodiments, the inertial sensor 11 includes an inertial measurement unit 111, a power supply module, a signal transmitting module, an inertial sensor housing 112 and an inertial sensor wearable kit 113. Through the inertial sensor wearable kit 113, the inertial sensor is worn and arranged at a corresponding position on the operator's arm; the inertial sensor wearable kit 113 can be a ring structure or a similar structure, or other available structures, such as a glove or sleeve structure, to facilitate wearing the inertial sensor on the operator's arm or hand.

[0053] In some embodiments, the measurement sensor of the inertial measurement unit 11 can measure its pitch angle, roll angle, yaw angle, and three-axis acceleration in real time, totaling six degrees of freedom data, and can independently send the collected real-time data to the wireless data acquisition device 4. The inertial sensor 11 preferably communicates with the wireless data acquisition device in a wireless manner, and the data processing device receives the data and solves it to obtain the operator's upper limb posture and movement intention at this time.

[0054] In a specific embodiment, preferably, the inertial sensor array 1 is composed of six inertial sensors 11. Figure 2a , shows a schematic diagram of the specific distribution of six sensors. The inertial sensors 11 are fixed on the back of the left and right hands, the forearm near the wrist joint, and the middle of the upper arm. The distance from the inertial sensor 11 on the back of the hand to the wrist joint is L palm , the forearm length is L forearm , the upper arm length is L arm , please refer to Figure 6a Preferably, the inertial sensor 11 sends data to the wireless data acquisition device 5 via Bluetooth. Figure 2b, which is a schematic diagram of the wearable packaging structure of the inertial sensor 11. The inertial measurement unit 111 and its supporting circuit board, battery module, and switch are packaged in an inertial sensor housing 112, and the inertial sensor housing 112 is fixed to the limb through an inertial sensor wearable kit 113.

[0055] It should be noted that, in the present application, the inertial sensor wearable kit 113 can be adjusted according to the body shape of different users, and the aforementioned L palm , L forearm , L arm The value can be adjusted to match different users, thereby improving the control accuracy of the control device of the embodiment of the present invention, achieving that each inertial sensor can be replaced and quickly configured individually, and extending the service life of the device of the embodiment of the present invention.

[0056] Figure 2d The coordinate system definition and degrees of freedom of the inertial sensor of the embodiment of the present invention are shown, wherein a xn 、a yn 、a zn They represent the acceleration of the inertial sensor 11 numbered n in the x-axis, y-axis, and z-axis directions at a certain moment. n 、Pitch n 、Roll n They respectively represent the rotation angles of the inertial sensor 11 numbered n around its z-axis, y-axis and x-axis relative to the initial position at a certain moment. In the inertial sensor array 1 of the embodiment of the present invention, n can be 1, 2, 3, 4, 5, 6. Each rotation is performed relative to the posture after the previous rotation, so the final posture of the inertial sensor is related to the rotation sequence.

[0057] It should be noted that the rotation sequence of the inertial sensor in the embodiment of the present invention is Yaw-Pitch-Roll, which is only exemplary and the present invention is not limited thereto.

[0058] In some embodiments, the pressure switch array 2 includes a first pressure switch 21, a second pressure switch 22, a third pressure switch 23, a fourth pressure switch 24, a fifth pressure switch 25, a sixth pressure switch 26, and a DuPont line for communication, which is used to receive additional control instructions; the first pressure switch 21, the second pressure switch 22, the third pressure switch 23, the fourth pressure switch 24, the fifth pressure switch 25, and the sixth pressure switch 26 all include a pressure switch sensing unit 211 and a coating layer 212, and the pressure switch sensing unit 211 is coated by the coating layer 212. Specifically, the pressure switch is fixed inside by two layers of inner and outer coating materials, and is bonded to the glove-type wearable component through the coating layer or fixed in other ways.

[0059] The pressure-sensitive switch array 2 of the embodiment of the present invention is composed of three pressure-sensitive switches for each hand. Figure 3a , Figure 3a The schematic diagram of the layout of the pressure switches is shown in Figure 2. The pressure switches are fixed on the index fingers and palms of the left and right hands. The first pressure switch 21 is fixed on the index finger of the right hand, the second pressure switch 22 and the third pressure switch 23 are fixed on the palm of the right hand, the fourth pressure switch 24 is fixed on the index finger of the left hand, and the fifth pressure switch 25 and the sixth pressure switch 26 are fixed on the palm of the left hand. Figure 3b , which shows a specific wearable packaging diagram of the pressure switch, the pressure switch sensing unit 211 is encapsulated on the glove-type wearable kit through the upper and lower coating layers 212. The pressure switch sensing unit 211 sends data to the wireless data sending device 4 via a wired method. Similarly, the number, fixed position, and fixed method of the pressure switch are exemplary, and the present invention is not limited thereto.

[0060] In some embodiments, the wireless data sending device 4 includes a single-chip microcomputer, a signal sending module, and a power supply module, and is connected to the first pressure-sensitive switch 21, the second pressure-sensitive switch 22, the third pressure-sensitive switch 23, the fourth pressure-sensitive switch 24, the fifth pressure-sensitive switch 25, and the sixth pressure-sensitive switch 26 through the DuPont line for communication. The first pressure-sensitive switch 21, the second pressure-sensitive switch 22, the third pressure-sensitive switch 23, the fourth pressure-sensitive switch 24, the fifth pressure-sensitive switch 25, and the sixth pressure-sensitive switch 26 will send signals to the wireless data acquisition device.

[0061] In the embodiment of the present application, a wearable back armor 3 is further included. After being worn by the operator, the wireless data sending device 4 can be arranged thereon, such as in the form of a pocket.

[0062] The embodiment of the present invention further provides a surgical robot control method based on gesture recognition, which is implemented based on the wearable device of the embodiment of the present invention. The inertial sensor array 1 is used to calculate the real-time posture of the operator's upper limbs and arms and control the surgical robot 8. The implementation steps are as follows:

[0063] S1. Data acquisition step: by placing inertial sensor arrays at various locations of the upper limbs, the limb motion data such as the pitch angle, roll angle, yaw angle, and three-axis acceleration of each major limb of the upper limb are obtained;

[0064] S2. Posture calculation step: Based on the Euler angle data in the limb motion data, the relationship between the Euler angle of the inertial sensor array 1 and the upper limb posture is used to obtain the real-time posture of the upper limb arms;

[0065] S3. Zero-speed update step: based on the geometric mean of the time variance of the three-axis acceleration, the intention of the upper limb movement is judged to improve the accuracy of the control process;

[0066] S4. Control mapping step: through the movement intention of the upper limb end, based on the type and direction of degrees of freedom supported by the controlled surgical robot, establish a master-slave control mapping formula from the upper limb movement to the movement of the surgical robot, and solve the current end position of the surgical robot based on the above mapping formula to achieve real-time control of the surgical robot based on gesture recognition; at the same time, limit the movement range and speed of the surgical robot by limiting the movement space, adjustable movement factor and control interrupt button (sensor switch) to avoid tissue and instrument damage caused by misoperation

[0067] See also Figure 4 , Figure 4 The control process of the surgical robot using the above-mentioned wearable control device is shown. The embodiment of the present invention uses an incremental control strategy. At each moment (T), the inertial sensor will read the operator's input, and obtain the posture of the control device at the current moment, that is, the upper limb posture at the current moment (T) through the upper limb posture calculation formula; then the difference calculation is performed with the upper limb posture at the previous moment (T+1); the result of the difference is the control device posture increment (that is, the upper limb posture increment at moment T+1); the increment is input into the master-slave mapping relationship formula to obtain the surgical robot posture increment at the next moment (T+1); through subsequent processing, it can be converted into the number of steps used by the surgical robot drive motor used in the embodiment of the present invention to control the surgical robot to move. Repeatedly executing this process can achieve continuous master-slave control of the end.

[0068] Furthermore, during operation, the operator can also view the movement status of the surgical robot in real time through devices such as cameras, and make real-time adjustments based on visual information to compensate for the lack of precision.

[0069] The wearable device of the embodiment of the present invention can make the operator's movements in traditional surgery completely consistent with the movement trends of the surgical robot, without causing additional interference to the operation. The incremental control strategy does not require the operator's posture and the robot's end posture to be close to or even completely consistent. Therefore, some positions that are difficult to reach during traditional surgical operations can be reached more easily. The operator can also change the operating posture at any time to adapt to different situations, which is more in line with ergonomic requirements.

[0070] Since the human upper limbs are a series structure, see Figure 5As shown, the wearable device of the embodiment of the present invention abstracts the upper limbs of the human body into a three-link structure connected in series through ball joints and establishes a ball-jointed upper limb kinematic model 91. The above-mentioned inertial sensor collects the Euler angles of the three links of the upper limb to the world coordinate system rather than the rotation angles of the three ball joints. The Euler angles obtained by the above-mentioned inertial sensor are equivalent to the ball joint rotation angles of three groups of links, each of which is fixed to the base through a ball joint. The series connection process is equivalent to sequentially translating the links of the next level connection along the direction of the links of the previous level; therefore, it is only necessary to obtain the positions of the ends of the three groups of links respectively and add them up to obtain the position of the end of the upper limb.

[0071] A ball joint can be decomposed into three revolute joints according to the rotation order. Therefore, the ball joint of the upper limb of the human body is abstracted into the corresponding three revolute joints. Figure 6a In the kinematic model 92 of the revolute joint upper limb shown in FIG, the first revolute joint 921, the second revolute joint 922, and the third revolute joint 923 collectively represent the shoulder joint, the fourth revolute joint 924, the fifth revolute joint 925, and the sixth revolute joint 926 collectively represent the elbow joint, and the seventh revolute joint 927, the eighth revolute joint 928, and the ninth revolute joint 929 collectively represent the wrist joint. The area between the third revolute joint 923 and the fourth revolute joint 924 represents the upper arm, the area between the sixth revolute joint 926 and the seventh revolute joint 927 represents the forearm, and the area after the ninth revolute joint 929 represents the hand.

[0072] It should be emphasized that the elbow and wrist joints of a person are actually closer to two-DOF universal joints, but due to tissue deformation and other reasons, there can still be a small range of third-DOF rotation. In order to facilitate subsequent calculations and improve the accuracy of calculations, they can be defined as ball joints. Similarly, this definition is not unique, but exemplary, and the present invention is not limited to this.

[0073] In this application, based on the above-mentioned definition of upper limb joints, the classic Denavit-Hartenberg (DH) method is used to analyze the kinematics and workspace of the main manipulator model.

[0074] The following is a list of DH parameters for a set of ball joint links i, i-1: Figure 6b As shown in i For z i-1 Axis and z i Axis around x i The angle between the axes, a i Yes i-1 Axis and x i The intersection of the axes along the x i The distance from the axis to the origin of the i-th coordinate system, θ i is the joint from x i-1 Axis to x i Axis around zi-1 The angle between the axes, d i From the origin of the i-1 coordinate system to z i-1 Axis and x i Axis along z i-1 The distance of the axis intersection.

[0075] Based on this, the transformation matrix from each coordinate system to the next coordinate system, that is, the DH matrix is:

[0076]

[0077] Substituting the DH parameters of each joint into equation (1) and multiplying them in sequence, the kinematic expression of the end of each ball joint link can be obtained:

[0078]

[0079] in: Refers to the transformation matrix from the 0th coordinate system to the 3rd coordinate system, Refers to the transformation matrix from the 0th coordinate system to the 1st coordinate system, Refers to the transformation matrix from the first coordinate system to the second coordinate system, Refers to the transformation matrix from the second coordinate system to the third coordinate system, n x Refers to the component of the x-axis unit vector (1,0,0) in the 0th coordinate system in the x-axis direction in the 3rd coordinate system, n y Refers to the component of the x-axis unit vector (1,0,0) in the 0th coordinate system in the y-axis direction in the 3rd coordinate system, n z Refers to the component of the x-axis unit vector (1,0,0) in the 0th coordinate system in the z-axis direction in the 3rd coordinate system, o x Refers to the component of the y-axis unit vector (0,1,0) in the 0th coordinate system in the x-axis direction in the 3rd coordinate system, o y Refers to the component of the y-axis unit vector (0,1,0) in the 0th coordinate system in the y-axis direction in the 3rd coordinate system, o z Refers to the component of the y-axis unit vector (0,1,0) in the 0th coordinate system in the z-axis direction in the 3rd coordinate system, a x Refers to the component of the z-axis unit vector (0,0,1) in the 0th coordinate system in the x-axis direction in the 3rd coordinate system, a y Refers to the component of the z-axis unit vector (0,0,1) in the 0th coordinate system in the y-axis direction in the 3rd coordinate system, a z Refers to the component of the z-axis unit vector (0,0,1) in the 0th coordinate system in the z-axis direction in the 3rd coordinate system, p x Refers to the x-axis coordinate of the origin (0,0,0) of the 0th coordinate system in the 3rd coordinate system, p y Refers to the y-axis coordinate of the origin (0,0,0) of the 0th coordinate system in the 3rd coordinate system, p zRefers to the z-axis coordinate of the origin (0,0,0) of the 0th coordinate system in the 3rd coordinate system.

[0080] This method is used to obtain the kinematic expressions of the ends of the three ball joints. Finally, the p in the kinematic expressions of the three ball joints is replaced by x ,p y ,p z The spatial position of the end of the upper limb kinematic model can be obtained by adding them up in sequence. Its expression is as follows:

[0081]

[0082] It should be noted that a ball joint can be decomposed into three rotational joints according to the rotation order, and the three ball joints have a total of 9 θ angles. When i is 1, 2, and 3, it represents the three rotational joints of the shoulder joint mentioned above, namely the first rotational joint 921, the second rotational joint 922, and the third rotational joint 923. When i is 4, 5, and 6, it represents the three rotational joints of the elbow joint mentioned above, namely the fourth rotational joint 924, the fifth rotational joint 925, and the sixth rotational joint 926. When i is 7, 8, and 9, it represents the three rotational joints of the wrist joint mentioned above, namely the seventh rotational joint 927, the eighth rotational joint 928, and the ninth rotational joint 929. arm represents the upper arm length mentioned above, L forearm represents the forearm length mentioned above, L palm Represents the hand length mentioned above.

[0083] At the same time, after the data of the above inertial sensor array is input into the above upper limb kinematic model for reconstruction, the real-time upper limb posture solution can be obtained, see Figure 7 .

[0084] Due to the inevitable physiological tremor of the upper limbs, in order to correctly identify the operator's control intention, it is necessary to introduce a zero-speed update link into the control link, that is, when the upper limb movement is very close to static, the surgical robot is kept static. n ,y n 、z n There are certain deviations in the original data, which are mainly manifested as follows: 1) There is high-frequency noise in the sensor report data, which affects the data accuracy; 2) The sensor is affected by gravity and other factors, and the acceleration value drifts. For example, when the sensor is stationary, the acceleration value still does not fluctuate around 0, but slowly increases or decreases over time; 3) Affected by the transmission quality, sometimes the received data has errors, which is manifested as a sudden appearance of a very large data value. Affected by these deviations, the speed obtained after integration will accumulate errors over time, which in turn affects the position calculation. For this reason, the original data needs to be processed.

[0085] In view of the sensor value drift and high-frequency noise in the initial startup state, the data can be centered and smoothed by taking the average value of the data during the static time, and abnormal maximum values ​​and small fluctuations can be filtered out:

[0086]

[0087] In the formula, is the original data of three-axis acceleration received by each inertial sensor in the inertial sensor array, a max ,a min The maximum and minimum values ​​used to filter out tiny noise and abnormal maximum values, is the triaxial acceleration data after filtering out outliers, mean is the sampling length of the centralization link, is the offset value used for centering, is the three-axis acceleration data after centering, a axis The final three-axis acceleration data for subsequent calculations is obtained. Based on variance analysis, a zero-speed update link is added:

[0088]

[0089] In the formula, t n The acceleration variance calculation value of a certain degree of freedom used for the zero-speed correction link at all times. When the acceleration variance within a certain period of time is less than the threshold, it is determined that the above-mentioned inertial sensor has been stationary and the movement of the surgical robot is stopped.

[0090] In the embodiment of the present invention, the surgical robot used has two sets of flexible ends 81, which are arranged opposite to each other and connected to the host device. The host device is connected to the data processing device 6 for communication, receives the control signal of the data processing device, and then drives the flexible end 81 to produce corresponding actions according to the upper limb actions of the operator. The host device includes a control module, and the flexible end 81 includes a drive motor for driving the joints of the flexible end to rotate. The control module controls the rotation of the corresponding drive motor according to the signal of the data processing device, so that the joints of the flexible end are controlled to move according to the intention of the operation, see Figure 8As shown, each group of flexible ends 81 has a total of seven degrees of freedom, including three bending degrees of freedom SX1, SY1, and SZ1 of the first bending joint group, one axial rotation degree of freedom SR1, two bending degrees of freedom SX2 and SY2 of the second bending joint group, and one instrument opening and closing degree of freedom at the end; the first bending joint group is closer to the end and can realize up and down, left and right, front and back, and axial rotation movements, and the second bending joint group is closer to the tail, that is, the driving end, and can realize up and down, left and right expansion movements; the expansion movement is similar to making the end including the first bending joint group move upward and downward and left and right as a whole, aiming to increase the distance between the two flexible ends, the opening and closing degrees of freedom of the instrument, and be used for opening and closing of clamps, etc. The upper limb kinematic model includes the first translational degree of freedom MX1, MY1, MZ1 of the end, the second translational degree of freedom MX2, MY2, MZ2, and the axial rotational degree of freedom MR, wherein the translational degree of freedom of the end of the left or right hand is defined as the first translational degree of freedom or the second translational degree of freedom through the third pressure switch 23 (right hand) or the sixth pressure switch 26 (left hand) in the embodiment of the present invention. When the first translational degree of freedom is currently in effect, the pressure switch is pressed and released once to switch to the second translational degree of freedom. When the second translational degree of freedom is currently in effect, the pressure switch is pressed and released once to switch to the first translational degree of freedom. Figure 8 , the upper limb motion obtained by the above-mentioned surgical robot control method based on gesture recognition is mapped to the motion of the above-mentioned surgical robot, and the mapping relationship of the right hand is as follows:

[0091]

[0092] In (10), Δp sx1 ,Δp sz1 ,Δp sr1 ,Δp sy1 are the three deflection degrees of freedom increments and one axial translation degree of freedom increment of the first bending joint group, Δp sx2 ,Δp sy2 are the two deflection degrees of freedom increments of the second bending joint group, Switch s For the opening and closing of the flexible end device, Switch m is the switch state of the first pressure switch 21 (right hand) or the fourth pressure switch 24 (left hand) in the embodiment of the present invention, Δp mx ,Δp my ,Δp mz are the upper limb position increments obtained, Δθ6 is the angle increment of the sixth revolute joint 926 corresponding to θ6 in the upper limb kinematic model DH parameter definition of the embodiment of the present invention, k is the motion coefficient, the unit is 1, and Δp mx1 With Δp mx2 and Δp my1 With Δp my2The third pressure-sensitive switch 23 (right hand) or the sixth pressure-sensitive switch 26 (left hand) in the embodiment of the present invention is used for switching, and the mapping relationship of the left hand is the same.

[0093] The surgical robot wearable device based on gesture recognition and the control method thereof proposed in the embodiments of the present invention realize a master-slave control function with good real-time performance, high motion accuracy, strong safety and more intuitive for the operator, thereby improving the adaptability of the traditional master-slave control method to surgical robots; and better meet the requirements of the surgical robot control field for dexterity, efficiency, high precision and high safety.

[0094] The basic principles and main features of the present invention and the advantages of the present invention are shown and described above. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.

[0095] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the claims are included in the present invention.

[0096] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A wearable control device for surgical robots based on gesture recognition, characterized in that: Used to monitor the operator's upper limb posture and movement intention in real time, including: An inertial sensor array, comprising a plurality of inertial sensors, arranged at the backs of the hands, forearms and upper arms of the operator's left and right arms, to detect the operator's motion data; A pressure switch array, including multiple pressure switches, is placed on the index fingers and palms of the operator's left and right hands, respectively, to activate / interrupt the movement of the surgical robot; A wireless data transmission device is connected to the pressure-sensitive switch by wire, and is used to unidirectionally transmit the status data of the pressure-sensitive switch to the wireless data acquisition device; A wireless data acquisition device, used for wirelessly receiving the status data sent by the wireless data sending device and the motion data sent by the inertial sensor, and transmitting them to a data processing device; The data processing device is used to process the motion information returned by the wireless data acquisition device to obtain the operator's motion intention and convert it into motion information of the end of the surgical robot.

2. The wearable control device for surgical robots based on gesture recognition according to claim 1, characterized in that: The inertial sensor, the wireless data sending device and the wireless data collecting device adopt Bluetooth wireless communication.

3. The wearable control device for surgical robots based on gesture recognition according to claim 1, characterized in that: The inertial sensors located at the back of the hand and the forearm are close to the operator's wrist joint, and the inertial sensor at the upper arm is located at the midpoint of the upper arm.

4. The wearable control device for surgical robots based on gesture recognition according to claim 1, characterized in that: The inertial measurement unit built into the inertial sensor is used to detect motion information of six degrees of freedom, including pitch angle, roll angle, yaw angle, and three-axis acceleration; the inertial sensor sends data to the data acquisition device unidirectionally, and the refresh rate of the inertial sensor is N2, and N2 is greater than or equal to 100.

5. The wearable control device for surgical robots based on gesture recognition according to claim 1, characterized in that: The refresh rate of the pressure-sensitive switch array is N1, which is greater than or equal to 50. Pressing it activates the movement of the surgical robot, and releasing it interrupts the movement of the surgical robot.

6. The wearable control device for surgical robots based on gesture recognition according to claim 1, characterized in that: The wireless data sending device is arranged on the wearable back armor, the inertial sensor is installed on the wearable armor, and the pressure switch is fixed inside by two layers of inner and outer covering materials.

7. The wearable control device for surgical robots based on gesture recognition according to claim 1, characterized in that: It includes a display device for displaying the upper limb end posture calculated by the data processing device and the movement increment of the surgical robot at the next moment.

8. The wearable control device for surgical robots based on gesture recognition according to any one of claims 1 to 7, characterized in that: The data processing device is connected to the controlled surgical robot for communication, controls the surgical robot to perform corresponding movements, and realizes continuous master-slave control of the terminal.

9. The control method of the surgical robot wearable control device based on gesture recognition according to any one of claims 1 to 8, characterized in that: Includes steps: Calculating the operator's current upper limb end posture using the inertial sensor array; The data processing device uses the obtained upper limb terminal posture and state feedback information of the pressure-sensitive switch array to map the upper limb movement to the movement of each degree of freedom of the surgical robot based on the upper limb to surgical robot mapping module, converts it into movement information of the surgical robot terminal, and controls the movement of the surgical robot; During the control process, the maximum movement distance of each degree of freedom of the surgical robot is restricted by the preset movement space, and the left and right side controls of the surgical robot are activated or not through the pressure switches of the pressure switch array. The robot movement is activated when pressed and interrupted when released.

10. The control method of the surgical robot wearable control device based on gesture recognition according to claim 9, characterized in that: The method of calculating the terminal posture of the operator's current upper limb by using the inertial sensor array comprises the steps of: Acquiring six-degree-of-freedom data obtained by the inertial sensor; Acquire six-degree-of-freedom data of the inertial sensor for a period of time in a stationary state, and remove the offset accumulated over time; The six-degree-of-freedom data with the offset removed is filtered out of high-frequency noise and compared with the data at the previous moment, and the current end position of the upper limb is obtained by combining the relationship between the pitch angle, roll angle, yaw angle of the inertial sensor and the upper limb posture.

Images