Surgical robot wearable control device and control method based on gesture recognition

Abstract

The application discloses a surgical robot wearable control device and control method based on gesture recognition. The surgical robot wearable control device is used for detecting motion information of human upper limbs and comprises: an inertial sensor array arranged at the back of hands, forearms and upper arms of left and right arms of an operator to detect operator motion data; a pressure switch array arranged at index fingers and palms of left and right hands of the operator to activate / discontinue motion of the surgical robot; a wireless data sending device used for unidirectionally sending state data of the pressure switch to a wireless data collection device; the wireless data collection device is used for receiving the state data sent by the wireless data sending device and motion data sent by the inertial sensor and transmitting the data to a data processing device; the data processing device obtains operator motion intention after processing the motion information returned by the wireless data collection device and converts the motion intention into end motion information of the surgical robot. The application can detect the posture of the upper limbs of the operator and the control intention of the operator in real time.

Description

Technical Field

[0001] This invention relates to the field of wearable control devices for surgical robots, and in particular to a wearable control device and control method for surgical robots based on gesture recognition. Background Technology

[0002] Natural Orifice Transluminal Endoscopic Surgery (NOTES) refers to various endoscopic procedures performed by entering the abdominal cavity, mediastinum, and thoracic cavity through natural orifices such as the mouth, esophagus, stomach, and colon (rectum). These procedures include abdominal exploration, peritoneal biopsy, liver biopsy, gastrointestinal and enteroenterostomy, tail resection, and cholecystectomy. It allows surgical procedures to be performed without making any incisions on the patient's body surface. Postoperatively, there are no external wounds, reducing patient pain, psychological stress, and shortening recovery time.

[0003] After more than a decade of research and development, an increasing number of medical institutions are gradually implementing NOTES technology in clinical practice. The limitations of traditional rigid surgical instruments and the increasingly stringent demands for minimally invasive laparoscopic surgery have driven the development and application of flexible surgical robot systems, represented by continuum-based systems. Flexible surgical robots enter areas such as the stomach through narrow and winding natural cavities like the upper digestive tract. Using a camera built into the robot, lesions are observed in real time, and the robot's flexible end effector assists in resection or sampling. To minimize damage to normal tissues, flexible surgical robots must possess high flexibility, and the flexible end effector must be adaptable to the resection of lesions in different locations and structures.

[0004] Therefore, existing flexible surgical robots are equipped with flexible end effectors that often have many degrees of freedom and are relatively long and slender to facilitate access to lesions through natural cavities and to achieve bending and instrument opening and closing through structures such as control wires. However, this results in insufficient end effector stiffness and lower motion precision compared to traditional articulated robotic arms. In addition, the friction generated by the longer control wires can create coupling and additional resistance, affecting the movement of the instrument end effector.

[0005] To overcome the aforementioned shortcomings, most commonly used control methods for flexible surgical robots rely on master-slave control, depending on the operator's visual information to compensate for errors during motion. To achieve master-slave control that better matches the high degrees of freedom and nonlinear motion characteristics of flexible surgical robots, researchers have proposed various control interfaces and methods, but many limitations still exist.

[0006] Existing control methods typically rely on controllers with different structures, and most controller configurations differ from the configuration of the human upper limb. This results in such control methods not conforming to the intuitive operation developed by users in traditional operation processes, thus requiring a relatively long period of training before they can be used.

[0007] Furthermore, most existing control methods require doctors to perform specific actions, such as head movements, blinking, foot movements, or vocalizations, to control the robot. These actions introduce additional delays into the control process, leading to poor hand-eye coordination and consequently, prolonged surgical time. Moreover, these control schemes lack sufficient degrees of freedom and depth information. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing control methods and devices, such as limited degrees of freedom, poor hand-eye coordination, and unintuitive control. It provides a wearable control device and method for surgical robots based on gesture recognition that simultaneously supports multi-degree-of-freedom control, has a simple and lightweight structure, and features dexterity, high precision, and high safety. This device is also more intuitive for operation. The wearable device can detect the operator's upper limb posture and control intention in real time.

[0009] One aspect of the present invention provides a wearable control device for a surgical robot based on gesture recognition, used for real-time monitoring of the operator's upper limb posture and movement intentions, comprising:

[0010] An inertial sensor array, comprising multiple inertial sensors, is used to detect the operator's motion data by placing them on the back of the hand, forearm, and upper arm of the operator's left and right arms.

[0011] The pressure-sensitive switch array, including multiple pressure-sensitive switches, is placed on the index fingers and palms of the operator's left and right hands respectively, for activating / interrupting the movement of the surgical robot;

[0012] A wireless data transmitting device is connected to the pressure-sensitive switch via a wired connection and is used to unidirectionally transmit the status data of the pressure-sensitive switch to the wireless data acquisition device.

[0013] A wireless data acquisition device is used to wirelessly receive status data sent by the wireless data transmission device and motion data sent by the inertial sensor, and transmit them to a data processing device.

[0014] The data processing device is used to process the motion information returned by the wireless data acquisition device to obtain the operator's motion intention and convert it into motion information of the surgical robot end effector.

[0015] The inertial sensor, wireless data transmission device, and wireless data acquisition device are connected via Bluetooth wireless communication.

[0016] Among them, the inertial sensors located on the back of the hand and forearm are close to the operator's wrist joint, while the inertial sensor on the upper arm is located at the midpoint of the upper arm.

[0017] The inertial sensor has a built-in inertial measurement unit for detecting motion information in six degrees of freedom, including pitch angle, roll angle, yaw angle, and triaxial acceleration. The inertial sensor sends data unidirectionally to the data acquisition device. The refresh rate of the inertial sensor is N2, where N2 is greater than or equal to 100.

[0018] The pressure-sensitive switch array has a refresh rate of N1, where N1 is greater than or equal to 50. Pressing the switch activates the movement of the surgical robot, while releasing it interrupts the movement of the surgical robot.

[0019] The wireless data transmission device is arranged on the wearable back armor, the inertial sensor is installed on the wearable armor, and the pressure-sensitive switch is fixed inside by two layers of covering material.

[0020] The wearable control device for the surgical robot based on gesture recognition includes a display device for displaying the upper limb end pose calculated by the data processing device and the motion increment of the surgical robot at the next moment.

[0021] The data processing device is communicatively connected to the controlled surgical robot, controlling the surgical robot to move accordingly, thereby achieving continuous master-slave control of the end effector.

[0022] Another aspect of the present invention provides a control method for a wearable control device for a surgical robot based on gesture recognition, comprising the steps of:

[0023] The operator's current upper limb end-effector posture is calculated using the inertial sensor array;

[0024] The data processing device takes the end-effector pose and pressure-sensitive switch array status feedback information of the upper limb and maps the upper limb movement to the movement of each degree of freedom of the surgical robot based on the upper limb to surgical robot mapping module, converts it into the movement information of the surgical robot end-effector, and controls the movement of the surgical robot.

[0025] During the control process, the maximum movement distance of each degree of freedom of the surgical robot is constrained by the preset motion space limit. The activation or deactivation of the left and right side control of the surgical robot is controlled by the pressure-sensitive switches of the pressure-sensitive switch array. Pressing the switch activates the robot's movement, and releasing the switch interrupts the robot's movement.

[0026] The step of calculating the current end-effector posture of the operator's upper limb using the inertial sensor array includes the following steps:

[0027] Acquire the six degrees of freedom data obtained from the inertial sensor;

[0028] Acquire six degrees of freedom data of the inertial sensor over a period of time while in a static state, and remove the offset accumulated over time;

[0029] After removing high-frequency noise from the six-DOF data with offset removed, the data is compared with the data from the previous moment. The current end-effector pose of the upper limb is obtained by combining the relationship between the pitch angle, roll angle, yaw angle of the inertial sensor and the upper limb attitude.

[0030] The wearable control device of the present invention is lightweight and compact, consisting of an inertial sensor array, a pressure-sensitive switch array, and a wireless data transmission device. It requires no external auxiliary equipment, can be separated from the surgical robot, and can be remotely controlled.

[0031] The wearable control device of the present invention is based on gesture recognition technology to recognize the posture of the operator's upper limbs and arms. It has strong intuitiveness, and the operator's movement intention matches the movement direction of the surgical robot. Therefore, it has good hand-eye coordination, low training cost, expands the movement space, and improves movement accuracy.

[0032] The wearable control device of the present invention addresses the potential risks of additional damage or contamination caused by misoperation during surgery in the narrow natural cavities of the human body under master-slave control. It introduces additional elements such as safe movement space restriction and control interruption, which alleviates the problem of master-slave hand posture mismatch caused by incremental control, improves the problem of operator fatigue during long-term operation, and reduces the probability of accidental damage or damage to the surgical robot during surgery due to obstructed vision, fatigue or other factors.

[0033] The wearable control device of this invention has good compatibility, does not use mechanical structures to acquire motion information, is lightweight and reliable, and has a large movement space. By combining the wearable kit with an inertial sensor array, it can realize control logic that conforms to the intuitive human movement and can be adjusted for different operators. Through wireless data transmission and wireless data acquisition devices, the operator and the surgical robot can be separated. It can control translation and rotation simultaneously, supports multiple degrees of freedom, and can be applied to the master-slave control of flexible surgical robots through natural cavities. It also has application value in the control of humanoid or flexible robots in industrial collaboration, disaster relief, and national defense and military fields. Attached Figure Description

[0034] Figure 1 This is an overall schematic diagram of a wearable control device for a surgical robot based on gesture recognition, according to an embodiment of the present invention.

[0035] Figure 2a This is a schematic diagram of the layout of the inertial sensor array according to an embodiment of the present invention;

[0036] Figure 2b , Figure 2cThis is a schematic diagram of the wearable packaging structure of an inertial sensor according to an embodiment of the present invention;

[0037] Figure 2d This is a schematic diagram of the coordinate system definition and degrees of freedom of the inertial sensor according to an embodiment of the present invention;

[0038] Figure 3a This is a schematic diagram of the layout of the pressure-sensitive switch array according to an embodiment of the present invention;

[0039] Figure 3b This is a schematic diagram of the wearable packaging structure of the pressure-sensitive switch according to an embodiment of the present invention;

[0040] Figure 4 This is a flowchart of a surgical robot control method based on gesture recognition according to an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of the upper limb kinematic model according to an embodiment of the present invention;

[0042] Figure 6a This is a schematic diagram illustrating the calculation of the distal end position of the upper limb kinematic model according to an embodiment of the present invention;

[0043] Figure 6b This is a diagram showing the DH parameter definition of the ball joint link in the upper limb kinematic model according to an embodiment of the present invention.

[0044] Figure 7 This is a schematic diagram of the upper limb pose solution results according to an embodiment of the present invention;

[0045] Figure 8 This is a schematic diagram of the mapping model from upper limb end-effector movement to flexible surgical robot movement in an embodiment of the present invention.

[0046] [Explanation of Labels in the Attached Image]

[0047] 1-Inertial sensor array; 2-Pressure-sensitive switch array; 3-Wearable back armor; 4-Wireless data transmission device; 5-Wireless data acquisition device; 6-Data processing device; 7-Display device; 8-Surgical robot; 11-Inertial sensor; 111-Inertial measurement unit; 112-Inertial sensor housing; 113-Wearable inertial sensor kit; 21-First pressure-sensitive switch; 22-Second pressure-sensitive switch; 23-Third pressure-sensitive switch; 24-Fourth pressure-sensitive switch; 25-Fifth pressure-sensitive switch ; 26-Sixth pressure-sensitive switch; 211-Pressure-sensitive switch sensing unit; 212-Covering layer; 81-Flexible end; 91-Ball joint type upper limb kinematic model; 92-Rotary joint type upper limb kinematic model; 921-First rotary joint, 922-Second rotary joint, 923-Third rotary joint, 924-Fourth rotary joint, 925-Fifth rotary joint, 926-Sixth rotary joint, 927-Seventh rotary joint, 928-Eighth rotary joint, 929-Ninth rotary joint. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0049] Figure 1 This is an overall schematic diagram of a wearable control device for a surgical robot based on gesture recognition, according to an embodiment of the present invention. Figure 1 As shown, the wearable control device for surgical robots based on gesture recognition includes:

[0050] An inertial sensor array 1, comprising multiple inertial sensors 11, is positioned on the back of the operator's left and right arms, forearms, and upper arms to detect the operator's motion data. A pressure-sensitive switch array 2, comprising multiple pressure-sensitive switches, is positioned on the index fingers and palms of the operator's left and right hands respectively, for activating / interrupting the surgical robot's movement. A wireless data transmission device 4, connected to the multiple pressure-sensitive switches via a wired connection, is used to unidirectionally transmit the status data of the pressure-sensitive switches to a wireless data acquisition device. A wireless data acquisition device 5 is used to wirelessly receive the status data transmitted by the wireless data transmission device 4 and the motion data transmitted by the inertial sensors 11, and transmit them to a data processing device 6. The data processing device 6 processes the motion information returned by the wireless data acquisition device to obtain the operator's motion intention and converts it into motion information at the end effector of the surgical robot 8. In endoscopic surgery via natural orifices, the surgical robot 8 reaches the lesion site through the body's natural orifices to perform the operation. The wearable device in this embodiment can collect the posture of the operator's arms and control the surgical robot 8.

[0051] In some embodiments, the wearable control device for the surgical robot based on gesture recognition further includes a display device 7 for displaying the upper limb end pose calculated by the data processing device and the motion increment of the surgical robot at the next moment. During the use of the wearable device, the wireless data acquisition device unidirectionally receives data sent by the inertial sensor array and the wireless data transmission device; the data processing device is connected to the wireless data acquisition device, receives the data, analyzes and calculates it, obtains the posture and movement intention of the operator's upper limb, controls the robot, and displays it on the corresponding display device. That is, the wireless data acquisition device 5 receives the transmitted data and transmits it to the data processing device 6. The upper limb end pose calculated by the data processing device 6 and the motion increment of the surgical robot 8 at the next moment are displayed through the display device 7.

[0052] In some embodiments, the inertial sensor 11 includes an inertial measurement unit 111, a power supply module, a transmission module, an inertial sensor housing 112, and an inertial sensor wearable kit 113. The inertial sensor is worn and positioned on the operator's arm at a corresponding location through the inertial sensor wearable kit 113. The inertial sensor wearable kit 113 can be a ring-shaped structure or a similar structure, or other available structures, such as the structure of a glove or sleeve, to facilitate wearing the inertial sensor on the operator's arm or hand.

[0053] In some embodiments, the measurement sensor of the inertial measurement unit 11 can measure its pitch angle, roll angle, yaw angle, and acceleration in three axes in real time, totaling six degrees of freedom, and can independently send the collected real-time data to the wireless data acquisition device 4. Preferably, the inertial sensor 11 communicates with the wireless data acquisition device wirelessly. After receiving the data, the data processing device can solve the problem to obtain the operator's upper limb posture and movement intention at this time.

[0054] In a specific embodiment, preferably, the inertial sensor array 1 consists of six inertial sensors 11. See also Figure 2a The diagram shows the specific distribution of the six sensors. The inertial sensors 11 are fixed to the back of the left and right hands, near the wrist joint of the forearm, and in the middle of the upper arm, respectively. The distance from the inertial sensor 11 on the back of the hand to the wrist joint is L. palm The forearm length is L forearm The upper arm length is L arm Please refer to Figure 6a As shown. Preferably, the inertial sensor 11 transmits data to the wireless data acquisition device 5 via Bluetooth. See also Figure 2bThis is a schematic diagram of the wearable packaging structure of the inertial sensor 11. The inertial measurement unit 111 and its supporting circuit board, battery module and switch are encapsulated in the inertial sensor housing 112. The inertial sensor housing 112 is fixed to the limb by the inertial sensor wearable kit 113.

[0055] It should be noted that, in this application, the wearable inertial sensor kit 113 can be adjusted according to the body shape of different users, while modifying the aforementioned L... palm L forearm L arm The values ​​are matched to different users, thereby improving the control accuracy of the control device in this embodiment of the invention, enabling each inertial sensor to be replaced individually and quickly configured, and extending the service life of the device in this embodiment of the invention.

[0056] Figure 2d The coordinate system definition and degrees of freedom of the inertial sensor according to an embodiment of the present invention are shown, wherein a xn a yn a zn These represent the accelerations of inertial sensor 11, numbered n, along the x, y, and z axes at a given moment. n Pitch n Roll n These represent the angles at which the inertial sensor 11, numbered n, rotates relative to its initial position around its z-axis, y-axis, and x-axis at a given moment. In the inertial sensor array 1 of this embodiment, n can be 1, 2, 3, 4, 5, or 6. Each rotation is relative to the attitude after the previous rotation; therefore, the final attitude of the inertial sensor is related to the rotation sequence.

[0057] It should be noted that the rotation sequence of the inertial sensor in this embodiment of the invention is Yaw-Pitch-Roll. This rotation sequence is merely exemplary and is not intended to limit the invention.

[0058] In some embodiments, the pressure-sensitive switch array 2 includes a first pressure-sensitive switch 21, a second pressure-sensitive switch 22, a third pressure-sensitive switch 23, a fourth pressure-sensitive switch 24, a fifth pressure-sensitive switch 25, a sixth pressure-sensitive switch 26, and a DuPont wire for communication to receive additional control commands. The first pressure-sensitive switch 21, the second pressure-sensitive switch 22, the third pressure-sensitive switch 23, the fourth pressure-sensitive switch 24, the fifth pressure-sensitive switch 25, and the sixth pressure-sensitive switch 26 each include a pressure-sensitive switch sensing unit 211 and a covering layer 212. The pressure-sensitive switch sensing unit 211 is covered by the covering layer 212. Specifically, the pressure-sensitive switch is fixed inside by two layers of covering material, and is bonded to the glove-type wearable device or fixed by other means through the covering layer.

[0059] The pressure-sensitive switch array 2 of this embodiment consists of three pressure-sensitive switches for each hand. Taking the right hand as an example, see [link to example]. Figure 3a , Figure 3a The diagram above shows the layout of the pressure-sensitive switches. The switches are fixed to the index fingers and palms of both hands. Specifically, the first pressure-sensitive switch 21 is fixed to the right index finger, the second and third pressure-sensitive switches 22 and 23 are fixed to the right palm, the fourth pressure-sensitive switch 24 is fixed to the left index finger, and the fifth and sixth pressure-sensitive switches 25 and 26 are fixed to the left palm. (See also...) Figure 3b The diagram illustrates a wearable package of a pressure-sensitive switch, where the pressure-sensitive switch sensing unit 211 is encapsulated within a glove-shaped wearable kit by two layers 212. The pressure-sensitive switch sensing unit 211 transmits data to the wireless data transmission device 4 via a wired connection. Similarly, the number, fixed position, and fixing method of the pressure-sensitive switches are exemplary and not limiting to this invention.

[0060] In some embodiments, the wireless data transmitting device 4 includes a microcontroller, a transmitting module, and a power supply module. It is connected to the first pressure-sensitive switch 21, the second pressure-sensitive switch 22, the third pressure-sensitive switch 23, the fourth pressure-sensitive switch 24, the fifth pressure-sensitive switch 25, and the sixth pressure-sensitive switch 26 via the communication DuPont wire. The first pressure-sensitive switch 21, the second pressure-sensitive switch 22, the third pressure-sensitive switch 23, the fourth pressure-sensitive switch 24, the fifth pressure-sensitive switch 25, and the sixth pressure-sensitive switch 26 will send signals to the wireless data acquisition device.

[0061] In this embodiment, a wearable backplate 3 is further included, which, when worn by the operator, allows the wireless data transmission device 4 to be placed on it, such as through a pocket.

[0062] This invention also provides a surgical robot control method based on gesture recognition, implemented using a wearable device. The method uses the inertial sensor array 1 to calculate the real-time posture of the operator's upper limbs and control the surgical robot 8. The implementation steps are as follows:

[0063] S1. Data acquisition steps: By using an array of inertial sensors placed at various locations on the upper limbs, obtain limb motion data such as pitch angle, roll angle, yaw angle, and triaxial acceleration of each major limb.

[0064] S2. Attitude calculation steps: Based on the Euler angle data in the limb motion data, the real-time attitude of the upper limb arms is obtained by using the relationship between the Euler angles of the inertial sensor array 1 and the upper limb attitude.

[0065] S3. Zero-speed update step: Based on the geometric mean of the time variance of the three-axis acceleration, determine the intention of the upper limb movement to improve the accuracy of the control process;

[0066] S4. Control Mapping Step: Based on the movement intention of the upper limb end effector and the types and directions of the degrees of freedom supported by the controlled surgical robot, a master-slave control mapping formula is established from the upper limb movement to the movement of the surgical robot. The current end effector pose of the surgical robot is then obtained based on this mapping formula, enabling real-time control of the surgical robot based on gesture recognition. Simultaneously, the range and speed of the surgical robot's movement are limited by restricting the motion space, adjusting the motion factor, and using a control interrupt button (inductive switch) to avoid tissue and instrument damage caused by misoperation.

[0067] See Figure 4 , Figure 4 The control process for a surgical robot using the aforementioned wearable control device is illustrated. This embodiment of the invention employs an incremental control strategy. At each moment (T), the inertial sensor reads the operator's input and calculates the pose of the control device at the current moment (T) using the upper limb pose calculation formula. This difference is then calculated between the upper limb pose at the previous moment (T+1) and the previous moment (T+1). The result of this difference is the pose increment of the control device (i.e., the upper limb pose increment at time T+1). This increment is input into the master-slave mapping formula to obtain the pose increment of the surgical robot at the next moment (T+1). Subsequent processing converts this into the step count used by the surgical robot's drive motor to control the robot's movement. Repeating this process allows for continuous master-slave control of the end effector.

[0068] Furthermore, during operation, the operator can also use cameras and other devices to view the movement of the surgical robot in real time and make adjustments based on visual information to compensate for insufficient precision.

[0069] The wearable device of this invention enables the operator's movements in traditional surgery to be intuitive and completely match the movement trend of the surgical robot, without causing additional interference to the operation. Moreover, the incremental control strategy does not require the operator's posture and the robot's end-effector posture to be close or even completely consistent. Therefore, some positions that are difficult to reach in traditional surgical operations can be reached more easily. The operator can also change the operation posture at any time to adapt to different situations, thus better meeting the requirements of ergonomics.

[0070] Because the human upper limbs are a series structure, see Figure 5As shown, the wearable device of this embodiment of the invention abstracts the human upper limb as a three-bar structure connected by ball joints and establishes a ball-joint type upper limb kinematic model 91. The inertial sensor collected by the above-mentioned inertial sensor is the Euler angle of the three links of the upper limb about the world coordinate system, rather than the rotation angle of the three ball joints. The Euler angles obtained by the above-mentioned inertial sensor are equivalent to the ball joint rotation angles of the three sets of links, each fixed to the base by a ball joint. The series connection process is equivalent to translating the next-level connected link in sequence along the direction of the previous-level link; therefore, it is only necessary to calculate the position of the end of the three sets of links separately and add them together to obtain the position of the upper limb end.

[0071] A ball joint can be decomposed into three rotational joints in rotational order. Therefore, the ball joints of the human upper limb can be abstracted into three corresponding rotational joints. (See [reference needed]). Figure 6a The kinematic model 92 of the upper limb with rotating joints shown is as follows: the shoulder joint is represented by the first rotating joint 921, the second rotating joint 922, and the third rotating joint 923; the elbow joint is represented by the fourth rotating joint 924, the fifth rotating joint 925, and the sixth rotating joint 926; and the wrist joint is represented by the seventh rotating joint 927, the eighth rotating joint 928, and the ninth rotating joint 929. The area between the third rotating joint 923 and the fourth rotating joint 924 represents the upper arm; the area between the sixth rotating joint 926 and the seventh rotating joint 927 represents the forearm; and the area after the ninth rotating joint 929 represents the hand.

[0072] It is important to emphasize that the human elbow and wrist joints are actually closer to two-degree-of-freedom universal joints, but due to tissue deformation and other reasons, a small range of third-degree-of-freedom rotation can still exist. For the convenience of subsequent calculations and to improve the accuracy of calculations, they can be defined as ball joints. Similarly, this definition is not unique and is merely exemplary; the present invention is not limited thereto.

[0073] In this application, based on the above definition of upper limb joints, the classic Denavit-Hartenberg (DH) method is used to analyze the kinematics and workspace of the master manipulator model.

[0074] The following lists a set of DH parameters for ball joint links i, i-1: See Figure 6b As shown, α i For z i-1 axis and z i Axis around x i The included angle of the axes, α i It is z i-1 axis and x i The intersection of the axes along the x i The distance from the axis to the origin of the i-th coordinate system, θ i Is the joint from x i-1 axis to x i Axis around zi-1 The included angle of the axes, d i From the origin of the (i-1)th coordinate system to z i-1 axis and x i axis along z i-1 The distance between the intersection points of the axes.

[0075] Therefore, the transformation matrix from each coordinate system to the next coordinate system, i.e., the DH matrix, is:

[0076]

[0077] Substituting the DH parameters of each joint into equation (1) and multiplying them sequentially yields the kinematic expressions for the ends of each ball joint link:

[0078]

[0079] in: The transformation matrix refers to the transformation matrix from the 0th coordinate system to the 3rd coordinate system. The transformation matrix refers to the transformation matrix from the 0th coordinate system to the 1st coordinate system. The transformation matrix refers to the transformation from the first coordinate system to the second coordinate system. The transformation matrix from the second coordinate system to the third coordinate system, n x The n represents the x-axis component of the unit vector (1,0,0) in the 0th coordinate system along the 3rd coordinate system. y The n represents the component of the x-axis unit vector (1,0,0) in the 0th coordinate system along the y-axis in the 3rd coordinate system. z The component of the x-axis unit vector (1,0,0) in the 0th coordinate system along the z-axis in the 3rd coordinate system, o x The component of the y-axis unit vector (0,1,0) in the 0th coordinate system along the x-axis in the 3rd coordinate system. y The component of the y-axis of the unit vector (0,1,0) in the 0th coordinate system along the y-axis in the 3rd coordinate system. z a refers to the component of the y-axis unit vector (0,1,0) in the 0th coordinate system along the z-axis in the 3rd coordinate system. x a refers to the component of the z-axis unit vector (0,0,1) in the 0th coordinate system along the x-axis in the 3rd coordinate system. y a refers to the component of the z-axis unit vector (0,0,1) in the 0th coordinate system along the y-axis in the 3rd coordinate system. z p refers to the component of the z-axis direction of the unit vector (0,0,1) in the 0th coordinate system in the 3rd coordinate system. x p refers to the x-axis coordinate of the origin (0,0,0) of the 0th coordinate system in the 3rd coordinate system. y p refers to the y-coordinate of the origin (0,0,0) of the 0th coordinate system in the 3rd coordinate system. zThe z-axis coordinate of the origin (0,0,0) of the 0th coordinate system in the 3rd coordinate system.

[0080] Using this method, the kinematic expressions for the ends of the three ball joints are obtained. Finally, the p in the kinematic expressions of the three ball joints is... x ,p y ,p z By adding them sequentially, the spatial position of the distal end of the upper limb kinematic model can be obtained. Its expression is as follows:

[0081]

[0082] It is important to note that a ball joint can be decomposed into three rotational joints in rotational order. These three ball joints have a total of 9 θ angles. When i is 1, 2, or 3, it represents the three rotational joints of the shoulder joint mentioned above: the first rotational joint 921, the second rotational joint 922, and the third rotational joint 923. When i is 4, 5, or 6, it represents the three rotational joints of the elbow joint mentioned above: the fourth rotational joint 924, the fifth rotational joint 925, and the sixth rotational joint 926. When i is 7, 8, or 9, it represents the three rotational joints of the wrist joint mentioned above: the seventh rotational joint 927, the eighth rotational joint 928, and the ninth rotational joint 929. L arm L represents the upper arm length mentioned above. forearm L represents the forearm length mentioned above. palm This represents the hand length mentioned above.

[0083] Simultaneously, by inputting the data from the aforementioned inertial sensor array into the aforementioned upper limb kinematic model for reconstruction, real-time upper limb pose determination results can be obtained. (See [link to relevant documentation]). Figure 7 .

[0084] Due to unavoidable physiological tremors in the upper limbs, a zero-velocity update loop needs to be introduced into the control system to accurately identify the operator's control intentions. This means that when the upper limb movement is very close to stillness, the surgical robot should remain stationary. Inevitably, the x-value reported by the aforementioned inertial sensor array will... n y n z n The raw data contains certain biases, mainly manifested in the following ways: 1) High-frequency noise in the sensor's reported data affects data accuracy; 2) The sensor is affected by gravity and other factors, causing acceleration values ​​to drift. For example, when the sensor is stationary, the acceleration value does not fluctuate around 0, but rather increases or decreases slowly over time; 3) Due to transmission quality issues, sometimes the received data contains errors, such as a sudden appearance of an extremely large data value. Because of these biases, the velocity obtained after integration will accumulate errors over time, thus affecting position calculation. Therefore, it is necessary to process the raw data.

[0085] To address sensor numerical drift and high-frequency noise during startup, the average value of the data over a static period can be calculated to center and smooth the data, filtering out outliers and minor fluctuations.

[0086]

[0087] In the formula, For each inertial sensor in the inertial sensor array, a represents the raw triaxial acceleration data received by the sensor. max ,a min The maxima and minima used to filter out minute noise and outlier maxima The data represents the triaxial acceleration data after filtering out outliers, where mean is the sampling length of the centering process. The offset value used for centering. For the centered triaxial acceleration data, a axis This provides the final triaxial acceleration data for subsequent calculations. Based on analysis of variance, a zero-velocity update step is added:

[0088]

[0089] In the formula, For t n The acceleration variance of a certain degree of freedom is calculated at any given time during the zero-speed correction phase. When the acceleration variance is less than a threshold within a certain time period, it is determined that the inertial sensor has come to a stop, and the movement of the surgical robot is halted.

[0090] In this embodiment of the invention, the surgical robot has two sets of flexible ends 81, arranged opposite each other and connected to a host device. The host device is communicatively connected to a data processing device 6, receives control signals from the data processing device, and then drives the flexible ends 81 to generate corresponding movements according to the operator's upper limb movements. The host device includes a control module, and each flexible end 81 includes a drive motor for rotating the joints of the flexible ends. The control module controls the rotation of the corresponding drive motors according to the signals from the data processing device, so that the joints of the flexible ends move according to the operator's intention. See [link to relevant documentation]. Figure 8As shown, each flexible end effector 81 has seven degrees of freedom, including three bending degrees of freedom (SX1, SY1, SZ1) and one axial rotational degree of freedom (SR1) of the first bending joint group, two bending degrees of freedom (SX2, SY2) of the second bending joint group, and one instrument opening and closing degree of freedom of the end effector. The first bending joint group is closer to the end effector and can realize up-down, left-right, forward-backward, and axial rotation movements. The second bending joint group is closer to the tail end, i.e., the drive end, and can realize up-down and left-right unfolding movements. This unfolding movement is similar to causing the end effector as a whole, including the first bending joint group, to translate up-down and left-right, which aims to increase the distance between the two flexible ends and the instrument opening and closing degree of freedom, used for the opening and closing of clamps, etc. The upper limb kinematic model includes the first translational degrees of freedom MX1, MY1, and MZ1 at the distal end, the second translational degrees of freedom MX2, MY2, and MZ2, and the axial rotational degree of freedom MR. The translational degrees of freedom of the left or right hand are defined as either the first or second translational degree of freedom, which is switched by the third pressure-sensitive switch 23 (right hand) or the sixth pressure-sensitive switch 26 (left hand) in this embodiment of the invention. When the current degree of freedom is the first translational degree of freedom, pressing and releasing the pressure-sensitive switch once switches to the second translational degree of freedom; conversely, when the current degree of freedom is the second translational degree of freedom, pressing and releasing the pressure-sensitive switch once switches to the first translational degree of freedom. (Refer to...) Figure 8 The upper limb movements obtained by the above gesture recognition-based surgical robot control method are mapped to the movements of the surgical robot. The mapping relationship for the right hand is as follows:

[0091]

[0092] In (10), Δp sx1 ,Δp sz1 ,Δp sr1 ,Δp sy1 These represent the three deflection degree-of-freedom increments and one axial translation degree-of-freedom increment for the first bending joint group, Δp. sx2 ,Δp sy2 These are the two deflection degree of freedom increments for the second bending joint group, Switch s Switch for opening and closing flexible end devices m In this embodiment of the invention, Δp represents the switching state of the first pressure-sensitive switch 21 (right hand) or the fourth pressure-sensitive switch 24 (left hand). mx ,Δp my ,Δp mz The values ​​are the obtained upper limb position increments, where Δθ6 is the angle increment corresponding to the sixth rotational joint 926 in the upper limb kinematic model DH parameter definition of this embodiment, k is the motion coefficient in units of 1, and Δp is the upper limb position increment. mx1 With Δp mx2 and Δp my1 With Δp my2In this embodiment of the invention, the switching is performed using the third pressure-sensitive switch 23 (right hand) or the sixth pressure-sensitive switch 26 (left hand), with the same mapping relationship for the left hand.

[0093] The surgical robot wearable device and its control method based on gesture recognition proposed in this invention achieve a master-slave control function with good real-time performance, high motion accuracy, strong safety, and more intuitive operation for the operator. This improves the adaptability of traditional master-slave control methods to surgical robots and better meets the needs of the surgical robot control field for dexterity, efficiency, high precision, and high safety.

[0094] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.

[0095] Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of the equivalents of the claims be included within the invention.

[0096] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A wearable control device for a surgical robot based on gesture recognition, characterized in that, Used for real-time monitoring of the operator's upper limb posture and movement intentions, including: An inertial sensor array, comprising multiple inertial sensors, is used to detect the operator's motion data by placing them on the back of the hand, forearm, and upper arm of the operator's left and right arms. The pressure-sensitive switch array includes multiple pressure-sensitive switches, which are respectively placed on the index fingers and palms of the operator's left and right hands, and are used to activate / interrupt the movement of the surgical robot. A wireless data transmitting device is connected to the pressure-sensitive switch via a wired connection and is used to unidirectionally transmit the status data of the pressure-sensitive switch to the wireless data acquisition device. A wireless data acquisition device is used to wirelessly receive status data sent by the wireless data transmission device and motion data sent by the inertial sensor, and transmit them to a data processing device. The data processing device processes the motion information returned by the wireless data acquisition device, calculates the current end-effector posture of the operator's upper limbs, and calculates the difference between the posture and the previous posture to obtain the positional posture increment of the control device. The increment is input into the master-slave mapping relationship formula to obtain the positional posture increment of the surgical robot at the next moment. Based on the mapping module from the operator's upper limbs to the surgical robot, the operator's upper limb movement is mapped to the movement of each degree of freedom of the surgical robot through incremental control. The surgical robot is controlled to move under the preset motion space constraints. The preset motion space constraints constrain the maximum movement distance of each degree of freedom of the surgical robot. The activation or deactivation of the left and right side control of the surgical robot is operated through the state data of the pressure-sensitive switch. Pressing activates the robot movement and releasing interrupts the robot movement. The wearable control device for the surgical robot abstracts the human upper limb as a three-bar structure connected by ball joints and establishes a ball-joint type upper limb kinematic model; the ball joint connection is to translate the next-level connected links along the direction of the previous level links in sequence; based on the relationship between the Euler angles of the three sets of links of the upper limb with respect to the world coordinate system and the upper limb posture collected by the inertial sensor, the positions of the ends of the three sets of links are obtained by calculating and adding them together to obtain the positions of the ends of the upper limb arms; the Euler angles are the ball joint rotation angles of the three sets of links, each fixed to the base by a ball joint. The surgical robot includes two sets of flexible ends arranged opposite to each other; each set of flexible ends has seven degrees of freedom, including three bending degrees of freedom of the first bending joint group, one axial rotational degree of freedom, two bending degrees of freedom of the second bending joint group, and one instrument opening and closing degree of freedom of the end.

2. The gesture recognition based surgical robotic wearable control device of claim 1, wherein, The inertial sensor, wireless data transmission device, and wireless data acquisition device use Bluetooth wireless communication.

3. The gesture recognition based surgical robotic wearable control device of claim 1, wherein, Inertial sensors located on the back of the hand and forearm are close to the operator's wrist joint, while the inertial sensor on the upper arm is located at the midpoint of the upper arm.

4. The gesture recognition based surgical robotic wearable control device of claim 1, wherein, The inertial sensor's built-in inertial measurement unit is used to detect motion information in six degrees of freedom, including pitch angle, roll angle, yaw angle, and three-axis acceleration; the inertial sensor transmits data unidirectionally to the wireless data acquisition device.

5. The gesture recognition based surgical robotic wearable control device of claim 1, wherein, Pressing the pressure-sensitive switch array activates the movement of the surgical robot, and releasing it interrupts the movement of the surgical robot.

6. The gesture recognition based surgical robotic wearable control device of claim 1, wherein, The wireless data transmission device is arranged on the wearable back armor, the inertial sensor is installed on the wearable armor, and the pressure-sensitive switch is fixed inside by two layers of covering material.

7. The gesture recognition based surgical robotic wearable control device of claim 1, wherein, It includes a display device for displaying the upper limb end pose calculated by the data processing device and the motion increment of the surgical robot at the next moment.

8. The gesture-recognized based surgical robotic wearable control device of any one of claims 1-7, wherein, The data processing device is communicatively connected to the controlled surgical robot, controlling the surgical robot to move accordingly, thereby achieving continuous master-slave control of the end effector.

9. The motion information conversion method of the gesture recognition based surgical robot wearable control device according to any one of claims 1-8, characterized in that, Including the following steps: The operator's current upper limb end-effector posture is calculated using the inertial sensor array; The data processing device takes the end-effector pose and pressure-sensitive switch array status feedback information of the upper limb and maps the upper limb movement to the movement of each degree of freedom of the surgical robot based on the upper limb-to-surgical robot mapping module, and converts it into the motion information of the surgical robot end effector. 10.The motion information conversion method of the gesture recognition based surgical robot wearable control device according to claim 9, wherein, The step of calculating the current end-effector posture of the operator's upper limb using the inertial sensor array includes the following steps: Acquire the six degrees of freedom data obtained from the inertial sensor; Acquire six degrees of freedom data of the inertial sensor over a period of time while in a static state, and remove the offset accumulated over time; After removing high-frequency noise from the six-DOF data with offset removed, the data is compared with the data from the previous moment. The current end-effector pose of the upper limb is obtained by combining the relationship between the pitch angle, roll angle, yaw angle of the inertial sensor and the upper limb attitude.

Images