Surgical robotic system based on gesture control

Abstract

The present disclosure relates to a gesture-based control surgical robot system applied to the field of surgical robots, which comprises: an exoskeleton component for wearing on at least one upper limb of an operator to support and follow the movement of the at least one upper limb; an image acquisition device for acquiring hand images of at least one hand on the at least one upper limb; at least one driven tool comprising a tool arm and a driven instrument arranged at the end of the tool arm; and a control device connected with the exoskeleton component, the image acquisition device and the at least one driven tool, which is configured to: determine hand information of the at least one hand based on the hand images and the pose of the exoskeleton component; and perform master-slave control on the at least one driven tool based on the hand information of the at least one hand and a master-slave mapping relationship between the at least one hand and the at least one driven tool. The method can achieve accurate control of the driven tools in surgery, thereby improving the efficiency and accuracy of surgical operations.

Description

Technical Field

[0001] This disclosure relates to the field of robot control, and more particularly to a surgical robot system based on gesture control. Background Technology

[0002] In existing surgical robot systems, the operator (e.g., the surgeon) needs to operate a master manipulator to control the slave tools, enabling the slave tools to perform surgical tasks within the operating area. The master manipulator is typically mounted on the main control console of the surgical robot system, which presents problems such as high manufacturing and maintenance costs, difficulties in inter-departmental scheduling, and low efficiency in preoperative preparation. Furthermore, during prolonged surgeries, the visceral experience of operating the master manipulator can exacerbate operator fatigue, thereby increasing surgical risks.

[0003] Therefore, there is a need to provide a surgical robot system based on gesture control, which enables operators to directly control the slave tools to perform surgical tasks through gesture input, so as to balance the applicability of the surgical robot system and the user experience. Summary of the Invention

[0004] In some embodiments, this disclosure provides a gesture-based surgical robot system, comprising: an exoskeleton assembly for wearing on at least one upper limb of an operator to support and follow the movement of at least one upper limb; an image acquisition device for acquiring hand images of at least one hand on at least one upper limb; at least one driven tool, including a tool arm and a driven instrument disposed at the end of the tool arm; and a control device connected to the exoskeleton assembly, the image acquisition device, and at least one driven tool, the control device being configured to: determine hand information of at least one hand based on the hand image and the pose of the exoskeleton assembly, the hand information including the pose and / or gesture of at least one hand; and based on... The control device is configured to: control the movement of the slave device of at least one slave tool based on the pose and operation mapping relationship between at least one hand and at least one slave tool, provided that at least one hand's hand information and at least one slave tool's slave mechanism are matched with a master-slave operation gesture; and / or control the slave device of at least one slave tool to perform an operation corresponding to the master-slave operation gesture based on the pose and pose mapping relationship of at least one hand, in response to the matching of at least one hand's gesture with a master-slave operation gesture.

[0005] The aforementioned gesture-controlled surgical robot system, through the acquisition of exoskeleton components and hand images, enables precise control of surgical tools, thereby improving the accuracy of surgical operations and reducing the risk of misoperation. The exoskeleton components effectively support the operator's upper limbs, reducing fatigue during prolonged surgeries and improving operator comfort and endurance. Through gesture control, the operator can interact with surgical tools in a natural manner, improving operational precision and efficiency. The control device can monitor hand information in real time and quickly adjust the movement of the driven tools, ensuring timely response to the operator's intentions even in complex surgical environments. Attached Figure Description

[0006] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below. The accompanying drawings described below only show some embodiments of this disclosure. For those skilled in the art, other embodiments can be obtained based on the content of the embodiments of this disclosure and these drawings without creative effort.

[0007] Figure 1 A schematic diagram of a surgical robot system according to some embodiments of the present disclosure is shown;

[0008] Figure 2 This diagram illustrates a coordinate system in a master-slave motion mapping of at least one hand and at least one slave tool according to some embodiments of the present disclosure.

[0009] Figure 3 A schematic diagram of a hand key point model according to some embodiments of the present disclosure is shown;

[0010] Figure 4 A block diagram of a key point detection model according to some embodiments of the present disclosure is shown;

[0011] Figure 5 A block diagram of a gesture recognition model according to some embodiments of the present disclosure is shown;

[0012] Figure 6 This disclosure shows a schematic diagram of the structure of an exoskeleton assembly according to some embodiments;

[0013] Figure 7 A schematic diagram showing the positional boundaries of an exoskeleton assembly according to some embodiments of the present disclosure;

[0014] Figure 8-1 A schematic diagram illustrating master-slave operation gestures according to some embodiments of the present disclosure;

[0015] Figure 8-2 A schematic diagram illustrating the operation of a driven device based on hand information according to some embodiments of the present disclosure is shown.

[0016] Figure 9 A schematic diagram illustrating master-slave operation gestures according to some embodiments of the present disclosure;

[0017] Figure 10 A schematic diagram illustrating master-slave operation gestures according to some embodiments of the present disclosure;

[0018] Figure 11 A schematic diagram illustrating the allocation request for at least one slave tool according to some embodiments of the present disclosure;

[0019] Figure 12 A schematic diagram illustrating the allocation request for at least one slave tool according to other embodiments of the present disclosure;

[0020] Figure 13 This diagram illustrates a matching request for a slave device of at least one driven tool according to other embodiments of the present disclosure. Detailed Implementation

[0021] To make the technical problems solved by this disclosure, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely exemplary embodiments of this disclosure, and not all embodiments.

[0022] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional mapping relationships based on the orientation or positional mapping relationships shown in the accompanying drawings, are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this disclosure, it should be noted that unless otherwise expressly specified and limited, the terms "installed," "connected," "coupled," and "coupled" should be interpreted broadly. For example, they can refer to fixed connections or detachable connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two elements. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0023] In this disclosure, the end closer to the operator (e.g., a surgeon) is defined as the proximal end, proximal or rear end, or posterior end, and the end opposite to the proximal end, proximal or rear end, or posterior end is defined as the distal end, distal or anterior end, or anterior end. Alternatively, the end closer to the person being operated on (e.g., a surgical patient) is defined as the distal end, distal or anterior end, or anterior end, and the end opposite to the distal end, distal or anterior end, or anterior end is defined as the proximal end, proximal or rear end, or posterior end. Those skilled in the art will understand that embodiments of this disclosure can be used in medical devices or surgical robots, as well as other non-medical devices.

[0024] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom can be described using variations in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In the description of this disclosure, the term "pose" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and pose of an object or part of an object, which can be described, for example, using six parameters of the six degrees of freedom mentioned above. In this disclosure, the pose of the driven mechanism of a driven tool can be determined by the drive information of the driven tool (e.g., the drive information of the tool arm of the driven tool). In this disclosure, the configuration of an exoskeleton assembly can be represented by a set of joint information (joint values) of multiple joints included in the exoskeleton assembly when the exoskeleton assembly is in that configuration (e.g., a one-dimensional matrix composed of these joint information). The pose of the actuator arm can be determined by the actuator arm's drive information. In this disclosure, the joint information of the joint may include the angle of rotation of the corresponding joint relative to the corresponding joint axis or the distance moved relative to the initial position.

[0025] In this disclosure, a reference coordinate system can be understood as a coordinate system capable of describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, or a coordinate system in the space where the operator's hand, driven tool, or image acquisition device is located, or the operator's own perceptual coordinate system, etc.

[0026] In this disclosure, an object can be understood as an object or target that needs to be located, such as an operator's hand, a tool arm, or the end effector of a tool arm. The pose of a hand can refer to the pose of a coordinate system defined by the operator's hand relative to a reference coordinate system. The pose of an exoskeleton component or a portion thereof (e.g., the end effector) can refer to the pose of a coordinate system defined by the exoskeleton component or a portion thereof relative to a reference coordinate system. The pose of a tool arm or a portion thereof (e.g., the end effector) can refer to the pose of a coordinate system defined by the tool arm or a portion thereof relative to a reference coordinate system.

[0027] Figure 1 A schematic diagram of a surgical robot system 100 according to some embodiments of the present disclosure is shown. Figure 1 As shown, the surgical robot system 100 may include an image acquisition device 110, at least one driven tool 130, an exoskeleton assembly 140, and a control device 120. The image acquisition device 110 is used to acquire hand images of at least one hand 190 on at least one upper limb 180. The at least one driven tool 130 is used to perform surgical tasks and may include a tool arm and a driven instrument disposed at the end of the tool arm. The exoskeleton assembly 140 is worn on at least one upper limb 180 of the operator to support and follow the movement of the at least one upper limb 180. The control device 120 may be connected to the image acquisition device 110, the at least one driven tool 130, and the exoskeleton assembly 140, and realizes master-slave control of at least one hand 190 of the operator over at least one driven tool 130 based on the hand images acquired by the image acquisition device 110 and the pose of the exoskeleton assembly 140.

[0028] In some embodiments, the image acquisition device (e.g., Figure 1 The image acquisition device 110 shown is or Figure 2 The image acquisition device 210 shown may include, but is not limited to, dual-lens or single-lens image acquisition devices, such as monocular cameras, binocular cameras, structured light cameras, TOF (Time of Flight) cameras, etc. Depending on the application environment, the image acquisition device may be a camera, depth camera, head-mounted camera, industrial camera, etc. At least one of the operator's hands must be within the field of view of the image acquisition device (e.g., ...). Figure 1 The observation field of view shown is 111 or Figure 2 Within the field of view 211 shown, the acquired image may include at least one hand (e.g., Figure 1 At least one hand is shown 190. Figure 2 At least one hand 290 or shown Figure 7 An image of at least one hand (790) is shown. In some embodiments, the image acquisition device can be fixed in position or variable in position. For example, the image acquisition device can be fixed at a monitoring position to acquire images within a predetermined observation area, or the image acquisition device can be mounted on a mobile base and adjusted in position or posture to follow the movement of at least one hand of the operator, keeping at least one hand within the field of view. The mobile base can be, for example, connected to a control device (e.g., Figure 1The control device 120 shown is connected to a trolley, turntable, or robotic arm. In some embodiments, the image acquisition device can perform at least one of visible light imaging, infrared imaging, etc. Depending on the type of image acquired, those skilled in the art can select different image acquisition devices. In some embodiments, the image acquisition device can also follow and capture images of at least one of the operator's hands under the control of the control device. For example, a drive signal can be sent based on the pose change of at least one hand to drive a movable base to move the image acquisition device to follow the movement of at least one hand, thereby enabling the image acquisition device to continuously capture images of at least one of the operator's hands.

[0029] In some embodiments, the driven tool (e.g., Figure 1 The driven tool 130 shown Figure 2 The driven tool 230 shown Figure 7 The driven tool 730 shown Figure 11 The first driven tool 1131, the second driven tool 1132, and the third driven tool 1133 are shown. Figure 12 The first driven tool 1231, the second driven tool 1232, and the third driven tool 1233 shown are... Figure 13 The driven tool 1330 shown is configured to enter the operating area through a sheath, wherein the sheath can be fixed to an opening (e.g., an incision or natural opening) on ​​the patient's body, and the operating area can be the area where surgery is performed. In some embodiments, the tool arm of the driven tool can be a flexible arm body, and the driven instrument can be disposed at the distal end of the flexible arm body. In some embodiments, the driven instrument can be, for example, a surgical instrument such as a surgical forceps, electrocautery, electrocautery hook, electrocoagulation, or ultrasonic scalpel. Those skilled in the art will understand that the driven tool is not limited to the above-described structure; for example, it can also be a straight-bar type driven tool or a passive flexible driven tool.

[0030] In some embodiments, the exoskeleton component (e.g., Figure 1 The exoskeleton component 140 shown Figure 2 The exoskeleton component 240 shown Figure 6 The exoskeleton component 600 shown Figure 7 The exoskeleton assembly 740 shown can be an active or passive exoskeleton, such as a powered exoskeleton assembly that provides driving assistance to the operator via a drive device, or a passive exoskeleton assembly that provides support to the operator via an elastic resistance device (such as a spring or pulley system). Depending on the power source, the exoskeleton assembly can be a single-powered exoskeleton assembly, such as a pneumatic exoskeleton, a hydraulically driven exoskeleton, or an electrically driven exoskeleton, or a hybrid exoskeleton assembly, such as a dual-powered exoskeleton including pneumatic muscles and a drive motor. In some embodiments, the exoskeleton assembly includes, but is not limited to, being worn on at least one upper limb of the operator (e.g., Figure 1 At least one arm 180 and at least one hand 190 (in which). For example, the exoskeleton assembly can be configured as a full-body exoskeleton covering the operator's limbs and torso, or it can be configured as a split exoskeleton worn on the operator's limbs. In some embodiments, the exoskeleton assembly can be fixed or movable; for example, the exoskeleton assembly can include a support for fixation (e.g., Figure 1 As shown in bracket 149, the bracket can be fixedly mounted on a base or seat, or the exoskeleton component can move in tandem with the operator's body.

[0031] In some embodiments, the surgical robot system may also include a driven vision device (e.g., Figure 2 The illustrated slave vision device 250 and display device. In some embodiments, the slave vision device can be used to acquire surgical field images of the operating area, and the acquired surgical field images are processed and displayed on the display device (e.g., a slave vision device 250). Figure 2The display device (270) is shown. The display device may be, for example, a glasses-free 3D display device, allowing the operator to observe a three-dimensional surgical field image of the operating area with the naked eye. The operator can obtain in real-time the pose of at least one slave instrument of a slave tool relative to a reference coordinate system and / or the working state of at least one slave instrument of a slave tool through the surgical field image displayed on the display device (e.g., the working state of the slave instrument can be obtained by overlaying status prompts displayed in the image). When the operator waves at least one hand to operate the tool arm, the pose and gesture changes of at least one hand perceived by the operator conform to a preset mapping relationship with the pose and state changes of at least one slave instrument of a slave tool perceived by the operator on the display device. Thus, by waving at least one hand, the pose and gesture changes of at least one hand are converted into pose and state changes of at least one slave instrument of a slave tool based on the preset mapping relationship, thereby achieving intuitive master-slave control of at least one slave instrument of a slave tool. In some embodiments, the slave vision device may include an imaging device (e.g., an endoscope camera) for capturing real-time images of the operating area. Followed vision devices may include, but are not limited to, dual-lens followed vision devices or single-lens followed vision devices, such as binocular or monocular cameras. In some embodiments, the followed vision device can realize at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image acquired, those skilled in the art can select different followed vision devices as followed vision devices. In some embodiments, the followed vision device may also be configured to be operable by the operator's gestures. For example, the followed vision device may have a structure similar to at least one followed tool, including a flexible tool arm, with the imaging device disposed at the end of the tool arm. The operator can also control the movement of the followed vision device and / or control the operating state of the imaging device through the movement of at least one hand and gestures of at least one hand.

[0032] In this disclosure, the surgical robot system also includes a control device (e.g., Figure 1 The control device 120 shown is used for master-slave control of at least one slave tool based on hand information of at least one hand of the operator. In some embodiments, the control device may be configured to determine hand information of at least one hand based on a hand image and the pose of an exoskeleton component, and to perform master-slave control of at least one slave tool based on the hand information of at least one hand and the master-slave mapping relationship between at least one hand and at least one slave tool. It is understood that the control device may be a single controller centrally configured, or may include multiple controllers distributed in a manner.

[0033] In some embodiments, the control device can be configured to perform image recognition on hand images acquired by an image acquisition device, and combine the hand information (e.g., the pose and / or gesture of at least one hand) of the operator obtained from the pose of the exoskeleton component (e.g., the pose of the exoskeleton component's end effector) to generate control signals for a slave tool with a master-slave mapping relationship to at least one hand. The control device can also communicate with and send control signals to the drive device (e.g., a servo motor) or control unit of the slave tool, thereby enabling the drive device and / or control unit to control the slave instrument of the slave tool to perform surgical operations based on the control signals. For example, the control device can send control signals to the drive device and / or control unit of the slave tool via a CAN bus to control the slave instrument of the slave tool to move to a corresponding target pose or perform a corresponding surgical operation to achieve the surgical operation. In this disclosure, the operator can use master-slave operation gestures (e.g., ... Figure 8-1 and Figure 8-2 The master-slave operation gestures shown are 800. Figure 9 The master-slave operation gesture 900 is shown. Figure 10 The master-slave operation gesture 1000 shown allows for master-slave control of at least one slave tool that has a master-slave mapping relationship with at least one hand, or through other preset or generated gestures (e.g., ...). Figure 11 The gesture shown is 1102 or Figure 12 The assigned gesture 1202 shown assigns and matches at least one slave tool, the details of which will be described later.

[0034] Figure 2 A schematic diagram of the coordinate system in the master-slave motion mapping of at least one hand 290 and at least one driven tool 230 according to some embodiments of the present disclosure is shown. Figure 2 The coordinate systems are defined as follows: At least one driven tool's base coordinate system {Tb}, with its origin located at the base of at least one driven tool (e.g., the driven tool's drive mechanism) or the outlet of the abdominal sheath. Aligned with the extension line of the base or the axial direction of the abdominal sheath. Direction such as Figure 2 or Figure 7 As shown. At least one driven tool has a driven device coordinate system {wm}, with its origin located at the end of the tool arm of the driven tool or on the driven device positioned at the end of the tool arm. It is aligned with the axis of the end effector or the axis of the driven device. Direction such as Figure 2 As shown. The base coordinate system of the driven vision device is {Eb}, with its origin located at the base of the driven vision device (e.g., the drive mechanism of the driven vision device) or the outlet of the ingress sheath. Aligned with the extension line of the base or the axial direction of the abdominal sheath. Direction such as Figure 2 As shown. The camera coordinate system {lens} of the driven vision device has its origin located at the end of the tool arm of the driven vision device or set on the endoscope (e.g., the distal surface of the endoscope, at the midpoint of the line connecting the centers of the binocular cameras). The direction of the axis of the end or the axis of the endoscope is... Direction, after the field of vision is straightened, the upper part is Direction. The monitor coordinate system {Screen} has its origin at the center of the monitor, and the direction perpendicular to the screen image inwards is... Positive direction, the top of the screen is Direction. The image acquisition device coordinate system {cam} has its origin located on the camera of the image acquisition device (for example, it could be the far end of the lens or the midpoint of the line connecting the centers of two binocular cameras), and the lens axis direction is... direction, Direction such as Figure 2 As shown. The operator's hand coordinate system {H} for at least one hand, with the origin located at the wrist key point (which can be located at the base of the palm, for example...). Figure 3 The wrist key point 301 is shown. With wrist key points and middle finger palm key points (e.g.) Figure 3 The extension line of the line connecting the key points 310 on the palm of the middle finger (as shown) is in the same direction. The direction is perpendicular to the palmar direction (the palmar plane can be, for example, by...). Figure 3 (Any three of the following key points are used to determine the wrist key point 301, index finger palmar key point 306, middle finger palmar key point 310, ring finger palmar key point 314, and little finger palmar key point 318, as shown) Figure 2 , Figure 3 or Figure 7 As shown. The exoskeleton component's end coordinate system {E} has its origin located at the end of the exoskeleton component (e.g., ...). Figure 6 The structure of the exoskeleton assembly 600 shown is worn on at least one hand. Consistent with the axial direction at the end, Direction such as Figure 2 , Figure 6 or Figure 7 As shown. The exoskeleton component's base coordinate system is {B}, with its origin located on the support or base where the exoskeleton component is situated. The direction is vertically downward. Direction such as Figure 2 As shown. The reference coordinate system {w} can be the coordinate system of the space where the operator, exoskeleton components, driven tools, or endoscope reside, such as the driven tool base coordinate system {Tb}, or the world coordinate system, such as... Figure 2 As shown. In some embodiments, the operator's tactile sensation can be used as a reference; when the operator is seated in front of the main control panel, the perceived upward direction is... Direction, the perceived forward direction is... Direction. Understandable, for clarity. Figure 2 In this diagram, the coordinate system {E} of the exoskeleton component's end effector is shown as separate from the end effector of the exoskeleton component, but they are coincident. Those skilled in the art will understand that the definition of the coordinate system is not limited to the above, and other coordinate system definitions can also be used to achieve remote teleoperation of the surgical robot system.

[0035] like Figure 2 As shown, in some embodiments, the image acquisition device can be within the field of view (e.g., Figure 1 The observation field of view shown is 111 or Figure 2 The operator acquires an image of at least one hand from the field of view 211 shown. The image of at least one hand may include a single frame or multiple consecutive frames. Control device (e.g., Figure 1 The control device 120 shown can receive hand images of at least one hand from an image acquisition device.

[0036] In some embodiments, the image acquisition device can continuously acquire actual images of the observation area. The control device can determine, based on the actual images, whether the actual images include an image of at least one of the operator's hands. In some embodiments, the control device can be configured to extract a region including at least one hand from the actual image as an image of the hand in response to detecting at least one of the operator's hands in the actual image. For example, the region including at least one hand can be detected from the actual image through image feature extraction and matching to segment the hand image from the actual image. In some embodiments, image features include, for example, pixel grayscale, target texture, skin color, motion features, contour shape, etc.

[0037] In some embodiments, the control device can also detect at least one hand of the operator in the actual image using a pre-trained hand detection model, and determine that the actual image contains at least one hand of the operator when the hand confidence score is greater than a threshold. Here, the hand confidence score can represent the probability of at least one hand being present in the actual image. In some embodiments, the control device can detect whether the actual image contains at least one hand of the operator in the first frame of a series of actual images using the hand detection model, and track at least one hand in subsequent frames of the actual image and output the hand confidence score. When the hand confidence score is lower than the threshold, the control device can again detect whether the actual image contains at least one hand of the operator. In some embodiments, the control device can also be configured to disconnect the master-slave mapping relationship between at least one hand and at least one driven tool in response to the absence of a hand image with at least one hand detected in the image of the observation area, as detailed later.

[0038] In some embodiments, the control device may be configured to determine the relative three-dimensional coordinates of multiple key points of at least one hand in a hand image; and to determine the gesture of at least one hand based on the relative three-dimensional coordinates of the multiple key points. In some embodiments, the relative three-dimensional coordinates of the multiple key points of at least one hand may be the three-dimensional coordinates of the multiple key points relative to the hand coordinate system {H}. The relative three-dimensional coordinates of the multiple key points include the relative positional relationship between the multiple key points. In this embodiment, by determining the relative three-dimensional coordinates of multiple key points of the hand, the shape and movement of the hand can be accurately captured. This high-precision recognition improves the system's ability to understand gestures and movements, ensuring the accuracy of subsequent operations.

[0039] In some embodiments, the keypoints of at least one hand may include multiple feature points in a hand keypoint model used to represent the structural features of at least one hand, such as joints, contour points, etc. The hand keypoint model is determined based on the annotation method of the hand dataset used as training data. Depending on the annotation method of the hand dataset, the hand keypoint model may include a keypoint model with 14 keypoints, a keypoint model with 16 keypoints, a keypoint model with 21 keypoints, etc. Figure 3 A schematic diagram of a hand key point model 300 according to some embodiments of the present disclosure is shown. Figure 3 As shown, the 21 key points of the hand key point model 300 include a wrist key point 301 for aligning the wrist, a thumb palm key point 302 for aligning the thumb palm, an index finger palm key point 306 for aligning the index finger palm, a middle finger palm key point 310 for aligning the middle finger palm, a ring finger palm key point 314 for aligning the ring finger palm, a little finger palm key point 318 for aligning the little finger palm, and other key points not shown. The following description uses a hand key point model with 21 key points as an example, but this is not a limitation.

[0040] In some embodiments, the control device may be configured to use a pre-trained keypoint detection model (e.g. Figure 4 The keypoint detection model 400 shown detects keypoints of at least one hand in a hand image. The keypoint detection model can be trained from a labeled hand dataset. In some embodiments, the relative three-dimensional coordinates of multiple keypoints of at least one hand can be determined based on the hand image and the pre-trained keypoint detection model. Figure 4 A block diagram of a keypoint detection model 400 according to some embodiments of the present disclosure is shown. Figure 4 As shown, the keypoint detection model 400 may include an input layer 410, a feature extraction layer 420, and a coordinate prediction layer 430. The input layer 410 is used to receive point cloud data or voxel data obtained based on hand image processing.

[0041] The feature extraction layer 420 is used to obtain extracted features of a hand image based on point cloud data or voxel data from the input layer 410. For example, extracted features can be extracted from a single point cloud data or multiple neighboring point cloud data. Extracted features include, but are not limited to, geometric features, topological features, texture features, motion features, etc. In some embodiments, the feature extraction layer 420 may include multiple parallel and / or sequentially cascaded feature sublayers to extract different types of extracted features from the point cloud data or to sequentially extract extracted features of higher dimensionality.

[0042] The coordinate prediction layer 430 is used to predict and output the relative three-dimensional coordinates of key points of at least one hand in the hand image based on the extracted features of the hand image obtained by the feature extraction layer 420. In some embodiments, the coordinate prediction layer 430 may include multiple groups of neurons corresponding one-to-one with multiple key points of at least one hand, and each group of neurons includes three neurons to predict the X, Y, and Z coordinates of each key point. In some embodiments, each neuron in the coordinate prediction layer 430 may predict the relative three-dimensional coordinates of the corresponding key point based on preset weights and biases and the input extracted features.

[0043] In some embodiments, the keypoint detection model can be based on a pre-trained deep learning neural network (NN), such as a convolutional neural network (CNN) or a recurrent neural network (RNN), to predict the relative three-dimensional coordinates of multiple keypoints of at least one hand based on a hand image. In some embodiments, the keypoint detection model can be trained using a general hand dataset as training data to obtain a detection model capable of detecting the relative three-dimensional coordinates of hand keypoints of at least one hand under any hand gesture. General hand datasets include, for example, the NYU dataset, the ICVL dataset, the MSRA dataset, etc. Alternatively, the keypoint detection model can also be trained using a pre-constructed hand dataset as training data. For example, it can be trained for a pre-determined control system for surgical robots (e.g., Figure 1 The surgical robot system 100 shown has multiple preset gestures (e.g., Figure 8-1 and Figure 8-2 The master-slave operation gestures shown are 800. Figure 9 The master-slave operation gesture 900 is shown. Figure 10 The master-slave operation gesture 1000 or shown Figure 11The gesture assignment (1102) is shown. Multiple training images of each preset gesture under different postures, lighting conditions, and hand sizes are acquired using a camera. The coordinates of hand keypoints in each training image are labeled and used as training images for the hand dataset. In some embodiments, the hand dataset may also include extended images obtained by rotating the multiple training images, and multiple generated images generated by the hand keypoint model. Compared to training a keypoint detection model using a general hand dataset, training a keypoint detection model using a hand dataset based on master-slave gestures and other gestures can shorten the training time and improve the prediction accuracy of the keypoint detection model. In some embodiments, the operator's hand information in the surgical robot system can also be sampled. Multiple training images of the operator's left and right hands used to demonstrate master-slave gestures and other gestures are collected and labeled, and used as part of the hand dataset to further improve the prediction accuracy of the keypoint detection model.

[0044] Those skilled in the art will understand that this disclosure does not limit the way in which the layers and sub-layers of the keypoint detection model implement their functions, and each layer and sub-layer can implement its functions in any suitable way. Furthermore, the keypoint detection model is not limited to the structure described above. Any detection model capable of determining the coordinates of multiple keypoints of at least one hand based on a hand image is within the scope of this disclosure.

[0045] In some embodiments, the control device may be configured to be based on the relative three-dimensional coordinates of multiple key points of at least one hand and a gesture recognition model (e.g., Figure 5 The gesture recognition model shown (500) can determine gestures of at least one hand. By combining a pre-trained gesture recognition model, user gestures can be recognized quickly and accurately, improving system response speed and interaction efficiency.

[0046] In some embodiments, determining the gestures of at least one hand may include classifying the gestures of at least one hand based on a gesture sample set to determine which gesture in the gesture sample set matches the at least one hand. The gesture sample set may include pre-determined preset gestures for controlling the surgical robot system (e.g., ...). Figure 8-1 and Figure 8-2 The master-slave operation gestures shown are 800. Figure 9 The master-slave operation gesture 900 is shown. Figure 10 The master-slave operation gesture 1000 or shown Figure 11 The gestures assigned (1102) shown and the temporary gestures generated during the control of the surgical robot system (e.g.) Figure 12The gesture assignment shown is 1202), the details of which will be described later. In some embodiments, the control device may also be configured to generate a hand skeleton model of at least one hand (e.g., based on the relative three-dimensional coordinates of multiple key points of at least one hand). Figure 3 (as shown), and based on a hand skeleton model of at least one hand and a pre-trained gesture recognition model, to recognize gestures of at least one hand from hand images.

[0047] Figure 5 A block diagram of a gesture recognition model 500 according to some embodiments of the present disclosure is shown. Figure 5 As shown, the gesture recognition model 500 may include a feature extraction layer 510 and a gesture recognition layer 520. In some embodiments, gestures of at least one hand may include static gestures and dynamic gestures.

[0048] The feature extraction layer 510 is used to obtain gesture features of a hand image based on a hand skeleton model. In some embodiments, the feature extraction layer 510 can extract gesture features from a single frame or multiple consecutive frames of hand images based on the temporal domain and / or frequency domain. Gesture features include, but are not limited to, geometric features, topological features, and joint features. Geometric features may include the relative positions of multiple key points of at least one hand and the bending angles of the finger bones. In some embodiments, the feature extraction layer 510 can be integrated with… Figure 4 The feature extraction layer 420 shown has a similar structure, including multiple parallel and / or sequentially cascaded feature sublayers. In some embodiments, the feature extraction layer 510 may further include a temporal analysis layer 530, which is used to analyze the feature patterns of hand gesture changes in consecutive frames of hand images based on the gesture features extracted by the feature extraction layer 510, in order to identify dynamic gestures in consecutive frames of hand images. In some embodiments, the temporal analysis layer 530 may include three analysis sublayers to analyze the start, movement, and termination phases of a dynamic gesture, respectively. It should be understood that in some embodiments, the temporal analysis layer 530 may be part of the feature extraction layer 510.

[0049] The gesture recognition layer 520 is used to predict and output the gesture type of at least one hand in a hand image based on the gesture features extracted by the feature extraction layer 510 (and the temporal analysis layer 530). For example, when receiving input gesture features, the gesture recognition layer 520 can determine the matching confidence of at least one hand with gestures in the gesture dataset based on the gesture features, and output gestures with matching confidence greater than a confidence threshold as the gestures of at least one hand. In some embodiments, the gesture recognition layer 520 can be a pre-trained gesture classifier, such as a Support Vector Machine (SVM) or a SoftMax classifier.

[0050] In some embodiments, training the gesture recognition model can be implemented similarly to training the keypoint detection model. In some embodiments, gesture data can be aggregated and pre-determined to control the surgical robot system using preset gestures (e.g., ...). Figure 8-1 and Figure 8-2 The master-slave operation gestures shown are 800. Figure 9 The master-slave operation gesture 900 is shown. Figure 10 The master-slave operation gesture 1000 or shown Figure 11 The surgical data associated with the gesture (as shown in gesture 1102) is used as training data to train the gesture recognition model. In some embodiments, the gesture dataset can be a hand dataset labeled with gesture type. In some embodiments, the operator's hand information can also be sampled to obtain a gesture dataset for training the gesture recognition model.

[0051] The gesture sample set of the gesture recognition model 500 is not limited to including preset gestures associated with the gesture dataset used to train the gesture recognition model. In some embodiments, the gesture sample set of the gesture recognition model 500 also includes temporary gestures generated during the control of the surgical robot system (e.g., Figure 12 (Gesture assignment 1202 shown). In some embodiments, the control device may also be configured to add the temporary gesture to the gesture sample set in response to the generation of the temporary gesture, so as to update the gesture sample set, as will be described later.

[0052] Those skilled in the art will understand that this disclosure does not limit the way in which the layers and sub-layers of the gesture recognition model implement their functions, and each layer and sub-layer can implement its functions in any suitable manner. Furthermore, the gesture recognition model is not limited to the structure described above. Any model that can determine the gesture of at least one hand based on a hand image is within the scope of this disclosure.

[0053] In some embodiments, the control device is further configured to obtain the pose of the exoskeleton assembly; and to determine the pose of at least one hand based on the pose of the exoskeleton assembly. The pose of the exoskeleton assembly can be the pose of the exoskeleton assembly's end effector relative to a reference coordinate system {w}, for example, the pose of the exoskeleton assembly's end effector coordinate system {E} relative to the reference coordinate system {w}. In some embodiments, the pose of the exoskeleton assembly's end effector coordinate system {E} relative to the reference coordinate system {w} can be obtained based on the pose of the exoskeleton assembly's end effector coordinate system {E} relative to the exoskeleton assembly's base coordinate system {B} and the transformation relationship between the exoskeleton assembly's base coordinate system {B} and the reference coordinate system {w}.

[0054] In some embodiments, the transformation relationship between the exoskeleton component's base coordinate system {B} and the reference coordinate system {w} can be predetermined. For example, the exoskeleton component's base coordinate system {B} can be set on the support or base on which the exoskeleton component is located, and the exoskeleton component's base coordinate system {B} remains unchanged during the operator's operation, while the exoskeleton component's base coordinate system {B} and the reference coordinate system {w} have a predetermined transformation relationship.

[0055] In some embodiments, the control device can determine the pose of the exoskeleton assembly based on exoskeleton sensors. In some embodiments, the exoskeleton assembly may include a wearable structure for wearing on at least one upper limb and at least one exoskeleton sensor disposed at at least one joint on the wearable structure. The at least one exoskeleton sensor is configured to acquire joint information of at least one joint. The control device is also configured to determine the pose of the exoskeleton assembly's end effector based on the joint information of at least one joint of the exoskeleton assembly, as the pose of the exoskeleton assembly. In this embodiment, joint information of each joint can be acquired in real time through the exoskeleton sensors, allowing the control device to dynamically monitor the operator's upper limb movement state and ensure accurate understanding of gesture control. By collecting joint information, the control device can calculate the precise pose of the exoskeleton assembly's end effector, ensuring more accurate positioning of the operating tool during surgery. After determining the pose of the exoskeleton assembly, the system can better coordinate the motion relationship between the exoskeleton and the driven tool, ensuring that the driven tool accurately follows the operator's hand movements, achieving natural surgical operations.

[0056] Figure 6 A schematic diagram of the structure of an exoskeleton assembly 600 according to some embodiments of this disclosure is shown. For example... Figure 6 As shown, the exoskeleton assembly 600 may include a wearable structure 601 for wearing on at least one upper limb. The wearable structure 601 may be a multi-degree-of-freedom arm structure, such as a seven-degree-of-freedom arm structure, to allow the workspace of the exoskeleton assembly 600 to cover the movement space of at least one upper limb. In some embodiments, the wearable structure 601 may include at least one link 6011 for wearing on at least one upper limb and at least one joint 6012 corresponding to the at least one arm 6011. The at least one link 6011 is telescopically adjustable and worn on at least one upper limb of the operator via a wearable device (e.g., bandage, strap, etc.). The at least one joint 6012 may include at least one ball joint and at least one rotary joint, configured such that the axis of each joint is aligned with the axis of each upper limb joint (e.g., wrist, elbow, and shoulder joint) of at least one upper limb.

[0057] In some embodiments, the exoskeleton assembly 600 further includes at least one exoskeleton sensor (not shown) disposed at at least one joint 6012 on the wearable structure 601. The exoskeleton sensor is used to acquire joint information (e.g., joint angles) corresponding to at least one joint 6012. During the movement of at least one hand of the operator, at least one upper limb drives at least one joint 6012 of the exoskeleton assembly 600 aligned with at least one upper limb joint to rotate, thereby operating the exoskeleton assembly 600. The control device can determine the pose of the end-effector coordinate system {E} of the exoskeleton assembly relative to the base coordinate system {B} of the exoskeleton assembly based on the acquired joint information. For example, the control device can be configured to calculate the position and pose of the end-effector coordinate system {E} of the exoskeleton assembly relative to the base coordinate system {B} of the exoskeleton assembly based on the angle information in the joint information acquired by the exoskeleton sensor and a forward kinematics algorithm. In some embodiments, the control device is further configured to determine the pose of the exoskeleton assembly's end coordinate system {E} relative to the reference coordinate system {w}, based on the pose of the exoskeleton assembly's end coordinate system {E} relative to the exoskeleton assembly's base coordinate system {B} and the transformation relationship between the exoskeleton assembly's base coordinate system {B} and the reference coordinate system {w}, as the pose of the exoskeleton.

[0058] In some embodiments, the control device is further configured to determine the pose of at least one hand based on the pose of the exoskeleton assembly. In some embodiments, the control device can determine the pose of the hand coordinate system {H} relative to the reference coordinate system {w} based on the pose of the exoskeleton assembly's end-effector coordinate system {E} relative to the reference coordinate system {w} and the transformation relationship between the hand coordinate system {H} of at least one hand and the exoskeleton assembly's end-effector coordinate system {E}, as the pose of at least one hand. In some embodiments, the transformation relationship between the hand coordinate system {H} and the exoskeleton assembly's end-effector coordinate system {E} can be predetermined. For example, ... Figure 6 As shown, the end of the exoskeleton component 600 can be worn on at least one hand of the operator and moves with at least one hand. The coordinate system {E} of the end of the exoskeleton component and the coordinate system {H} of the hand have a predetermined transformation relationship.

[0059] In some embodiments, the control device is further configured to perform master-slave control on at least one driven tool based on hand information of at least one hand and a master-slave mapping relationship between at least one hand and at least one driven tool. The master-slave mapping relationship between at least one hand and at least one driven tool includes the pose mapping relationship and operation mapping relationship between at least one hand and the driven mechanism of at least one driven tool.

[0060] In master-slave control, the operator needs to maintain the hand gestures of at least one hand matching the master-slave operation gestures to maintain the master-slave mapping relationship between at least one hand and at least one slave tool. In this disclosure, master-slave operation gestures refer to gestures from a set of gestures used to perform master-slave control on at least one slave tool that has a master-slave mapping relationship with at least one hand. For ease of explanation, in this disclosure, master-slave operation gestures are used as a general term, for example... Figure 8-1 , Figure 8-2 The master-slave operation gesture 800 is shown. In some embodiments, the control device can be configured to, in response to a gesture of at least one hand matching the master-slave operation gesture, perform master-slave control on at least one slave tool based on hand information of at least one hand and the master-slave mapping relationship between at least one hand and at least one slave tool, or, in response to a mismatch between the gesture of at least one hand and the master-slave operation gesture, disconnect the master-slave mapping relationship between at least one hand and at least one slave tool. By determining whether the gesture of at least one hand is valid based on whether it matches the master-slave operation gesture, master-slave control can be performed on at least one slave tool that has a master-slave mapping relationship with at least one hand, or the master-slave mapping relationship between at least one hand and at least one slave tool can be disconnected, ensuring the safe operation of at least one slave tool by the operator.

[0061] In some embodiments, master-slave control of at least one slave tool may include: in response to a gesture of at least one hand matching a master-slave operation gesture, controlling the movement of the slave device of at least one slave tool based on the pose and pose mapping relationship of at least one hand. For example, in the state where the gesture of at least one hand matches the master-slave operation gesture, a slave tool control signal may be generated based on the pose and pose mapping relationship of at least one hand to control the movement of the slave device of at least one slave tool. In some embodiments, the pose mapping relationship between at least one hand and the slave device of at least one slave tool may include a position mapping relationship and an attitude mapping relationship between at least one hand and the slave device. In some embodiments, the position mapping relationship between at least one hand and the slave device of at least one slave tool may include the relationship between the position change of at least one hand and the position change of the slave device, and the attitude mapping relationship between at least one hand and the slave device of at least one slave tool may include the relationship between the attitude change of at least one hand and the attitude change of the slave device.

[0062] In some embodiments, the control device may be configured to: determine the current pose of at least one hand; obtain the previous pose of at least one hand; obtain the initial pose of the slave device of at least one driven tool; determine the target pose of the slave device of at least one driven tool based on the previous pose of at least one hand, the current pose of at least one hand, the initial pose of the slave device of at least one driven tool, and a pose mapping relationship; and control at least one driven tool to move based on the target pose of the slave device of at least one driven tool, so that the slave device of at least one driven tool moves to the target pose.

[0063] In some embodiments, determining the current pose of at least one hand includes determining the current pose of the hand coordinate system {H} of at least one hand relative to a reference coordinate system {w}. The current pose of at least one hand includes a current position and a current orientation. In some embodiments, determining the current position of at least one hand includes determining the current position of the hand coordinate system {H} relative to the reference coordinate system {w}, and determining the current orientation of at least one hand includes determining the current orientation of the hand coordinate system {H} relative to the reference coordinate system {w}. In some embodiments, the control device may be configured to obtain the current pose of the exoskeleton assembly corresponding to the current time, and to determine the current pose of at least one hand based on the current pose of the exoskeleton assembly. In some embodiments, the control device may configure the current joint information of at least one joint of the exoskeleton component to determine the current pose of the end-effector coordinate system {E} of the exoskeleton component relative to the reference coordinate system {w}; and determine the current pose of the hand coordinate system {H} relative to the reference coordinate system {w} based on the current pose of the end-effector coordinate system {E} of the exoskeleton component relative to the reference coordinate system {w} and the transformation relationship between the hand coordinate system {H} of at least one hand and the end-effector coordinate system {E} of the exoskeleton component, as the current pose of at least one hand.

[0064] In some embodiments, similar to determining the current pose of at least one hand, obtaining the previous pose of at least one hand includes obtaining the previous pose of the hand coordinate system {H} of at least one hand relative to a reference coordinate system {w}. The previous pose of at least one hand includes a previous position and a previous orientation. In some embodiments, obtaining the previous position of at least one hand includes obtaining the previous position of the hand coordinate system {H} relative to the reference coordinate system {w}, and obtaining the current orientation of at least one hand includes obtaining the previous orientation of the hand coordinate system {H} relative to the reference coordinate system {w}.

[0065] In some embodiments, the control device may receive the current pose of at least one hand obtained in the previous control cycle as the previous pose of at least one hand in the current control cycle. For the first control cycle, the position and posture of at least one hand when it is matched with the slave mechanism of at least one driven tool or a master-slave mapping relationship is established may be used as the previous pose of the first control cycle. Alternatively, the control device may determine the previous pose of the exoskeleton assembly based on the previous joint information of at least one joint of the exoskeleton assembly, and determine the previous pose of at least one hand based on the previous pose of the exoskeleton assembly.

[0066] In some embodiments, obtaining the initial pose of the slave device of at least one slave tool may include obtaining the initial pose of the slave device of at least one slave tool relative to the slave tool base coordinate system {Tb}, for example, the initial pose of the slave device coordinate system {wm} of at least one slave tool relative to the slave tool base coordinate system {Tb}. The initial pose of the slave device of at least one slave tool includes an initial position and an initial orientation. In some embodiments, obtaining the initial position of the slave device of at least one slave tool includes obtaining the initial position of the slave device coordinate system {wm} of at least one slave tool relative to the slave tool base coordinate system {Tb}, and obtaining the initial orientation of the slave device of at least one slave tool includes obtaining the initial orientation of the slave device coordinate system {wm} of at least one slave tool relative to the slave tool base coordinate system {Tb}.

[0067] In some embodiments, the target pose of at least one slave device of a slave tool obtained from the previous control cycle can be received as the starting pose of at least one slave device of a slave tool in the current control cycle. For the first control cycle, the initial pose of at least one slave device of a slave tool (e.g., the zero position of at least one slave tool) or the pose when the master-slave mapping relationship with at least one hand is broken can be used as the starting pose of the first control cycle.

[0068] In some embodiments, the control device may also be configured to determine the pose change of at least one hand based on the previous and current poses of at least one hand. For example, the position change of at least one hand in the reference coordinate system {w} may be determined based on the previous and current positions of the hand coordinate system {H} relative to the reference coordinate system {w}, and the pose change of at least one hand in the reference coordinate system {w} may be determined based on the previous and current poses of the hand coordinate system {H} relative to the reference coordinate system {w}.

[0069] In some embodiments, the control device may also be configured to determine the pose change of at least one driven instrument of a driven tool based on the pose change of at least one hand and the pose mapping relationship between at least one hand and the driven instrument of at least one driven tool. For example, the position change of at least one driven instrument of a driven tool may be determined based on the position change of at least one hand and the position mapping relationship between at least one hand and the driven instrument, and the attitude change of at least one driven instrument of a driven tool may be determined based on the posture change of at least one hand and the attitude mapping relationship between at least one hand and the driven instrument.

[0070] In some embodiments, the control device may further be configured to determine a target pose of at least one driven tool based on the initial pose of the driven instrument of at least one driven tool and the pose change of the driven instrument of at least one driven tool. The target pose of the driven instrument of at least one driven tool includes a target position and a target orientation. In some embodiments, the target pose of the driven instrument of at least one driven tool may be the target pose of the driven instrument of at least one driven tool relative to the driven tool base coordinate system {Tb}, for example, the target pose of the driven instrument coordinate system {wm} of at least one driven tool relative to the driven tool base coordinate system {Tb}. For example, the target position of the driven instrument of at least one driven tool may be determined based on the initial position of the driven instrument of at least one driven tool and the position change of the driven instrument of at least one driven tool, and the target orientation of the driven instrument of at least one driven tool may be determined based on the initial orientation of the driven instrument of at least one driven tool and the orientation change of the driven instrument of at least one driven tool.

[0071] In some embodiments, the pose mapping relationship between at least one hand and at least one driven tool may include the position change of the image of the driven instrument of at least one driven tool in the surgical field image relative to the reference coordinate system {w} being proportional to the position change of at least one hand relative to the reference coordinate system {w}, and / or the pose change of the image of the driven instrument of at least one driven tool in the surgical field image relative to the reference coordinate system {w} being consistent with the pose change of at least one hand relative to the reference coordinate system {w}.

[0072] In some embodiments, the control device is further configured to generate a slave tool control signal based on the initial pose and target pose of the slave instrument of at least one slave tool. The slave tool control signal is used to control the movement of the slave instrument of at least one slave tool. In some embodiments, the control device may be configured to: determine a pose difference based on the initial pose and target pose of the slave instrument of at least one slave tool; and determine a drive signal for controlling the movement of the slave instrument of at least one slave tool based on the pose difference and the inverse kinematics model of the tool arm of at least one slave tool. For example, based on the difference between the target pose and the initial pose of the slave instrument of at least one slave tool in the world coordinate system, the drive values ​​of multiple joints included in the tool arm in the current motion control cycle (or the drive values ​​of multiple corresponding motors controlling the movement of the manipulator arm) can be determined using an inverse kinematics numerical iterative algorithm of the tool arm kinematics model. It should be understood that the kinematic model can represent a mathematical model of the motion relationship between the joint space and the task space of the tool arm. For example, the kinematic model can be established using methods such as the Denavit-Hartenberg (DH) parameter method and the exponential product representation method.

[0073] In some embodiments, the control device can iteratively control the motion of the driven instrument of at least one driven tool to a target pose through multiple motion control cycles with a predetermined period.

[0074] In some embodiments, the surgical robot system may further include at least one exoskeleton drive (e.g., a servo motor) for driving at least one joint of the exoskeleton assembly (e.g., Figure 6 (At least one joint 6012 of the exoskeleton assembly 600 shown). In some embodiments, the control device may also be communicatively connected to at least one exoskeleton drive device and configured to perform impedance control of the exoskeleton assembly through the at least one exoskeleton drive device to achieve zero-force control of the exoskeleton assembly and support assistance for at least one upper limb of the operator, thereby alleviating operator fatigue during surgery.

[0075] In some embodiments, a dynamic model of the exoskeleton component can be established based on its mass inertia matrix, gravity matrix, and joint friction. In some embodiments, an impedance control model for impedance control of the exoskeleton component can also be established based on its equivalent mass, equivalent damping, equivalent stiffness, desired joint variables of at least one joint (e.g., desired joint angle, desired joint angular velocity, and desired joint angular acceleration of at least one joint), and the contact force generated when the exoskeleton component contacts at least one upper limb.

[0076] In some embodiments, the control device may further be configured to: obtain the actual torque of at least one joint; determine the compensation torque of at least one joint; determine the desired torque of at least one joint; determine the torque error of at least one joint based on the actual torque, desired torque, and compensation torque of at least one joint; and determine the drive signal of at least one exoskeleton drive device based on the torque error of at least one joint. In this embodiment, by comparing the actual torque, desired torque, and compensation torque, the system obtains a true feedback on the current torque state. Based on the torque error, the control device can calculate the drive signal required by the exoskeleton drive device. This dynamic adjustment capability allows the exoskeleton components to respond to the operator's movement intentions in real time, thereby improving the accuracy and sensitivity of movement.

[0077] In some embodiments, at least one exoskeleton sensor of the exoskeleton assembly is further configured to obtain joint torque of at least one joint. A control device may be configured to acquire the joint torque of at least one joint from the at least one exoskeleton sensor as the actual torque of at least one joint. In some embodiments, the control device may be configured to determine a compensation torque for at least one joint based on the inertial torque, gravitational torque, and joint friction of the exoskeleton assembly, to achieve torque compensation for the exoskeleton assembly. In some embodiments, the control device may be configured to determine the joint variable error of at least one joint based on the desired joint variable and the actual joint variable of at least one joint; and to determine the desired torque of at least one joint based on the joint variable error of at least one joint and the impedance control model of the exoskeleton assembly. The actual joint variable of at least one joint can be obtained through at least one exoskeleton sensor, and the actual joint variable of at least one joint may include the actual joint angle, actual joint angular velocity, and actual joint angular acceleration of at least one joint. The desired joint variable of at least one joint can be pre-determined and pre-stored in the memory of the surgical robot system through motion trajectory planning of the exoskeleton assembly, and the desired joint variable of at least one joint may include the desired joint angle, desired joint angular velocity, and desired joint angular acceleration. In some embodiments, the control device may also be configured to determine the torque error of at least one joint based on the actual torque, desired torque, and compensated torque of at least one joint; and to determine the drive signal of at least one exoskeleton drive device based on the torque error of at least one joint. For example, the control device may determine the drive signal of at least one exoskeleton drive device based on the torque error of at least one joint, the dynamic model of the exoskeleton assembly, and PID (Proportional Integral Derivative) control coefficients.

[0078] In a master-slave control state where at least one hand is matched with at least one slave tool and a master-slave mapping relationship is established, it is necessary to limit the range of motion of the exoskeleton component worn on at least one upper limb in order to restrict at least one hand to the range of motion corresponding to the workspace of at least one slave tool, thereby preventing at least one slave tool from exceeding its movement limits under the master-slave control of at least one hand.

[0079] In some embodiments, the control device may also be configured to generate a braking signal in response to the exoskeleton component being located on the position boundary of the exoskeleton component, so as to prevent the exoskeleton component from moving beyond the position boundary under the action of at least one upper limb. Figure 7 A schematic diagram illustrating the positional boundaries of an exoskeleton assembly according to some embodiments of the present disclosure is shown. Figure 7 As shown, the surgical connector 770 is inserted into the patient 750's body through an opening 751 (e.g., an incision or natural opening) formed in the patient's body. At least one driven tool 730 enters the operating space 760 within the patient's body cavity through the surgical connector 770. The operator exercises master-slave control over at least one driven tool 730, which has a master-slave mapping relationship with at least one hand 790. The control device can be configured to restrict the exoskeleton assembly 740 from moving outward from its position boundary to prevent at least one driven tool 730 from exceeding its movement limits under the master-slave control of at least one hand 790.

[0080] In some embodiments, the control device is further configured to determine the position vector space of at least one slave tool based on the initial position of at least one slave tool and the workspace of at least one slave tool; determine the position vector space of at least one hand based on the matching position of at least one hand, the position vector space of at least one slave tool and the pose mapping relationship; and determine the position boundary of the exoskeleton assembly based on the position vector space of at least one hand.

[0081] In some embodiments, the workspace of at least one driven tool 730 may be pre-stored in the memory of the robot system, and the workspace of at least one driven tool 730 may include the motion limits of at least one driven tool 730. For example... Figure 7 As shown, at least one driven tool 730 enters the operating space 760 through a sheath provided on the surgical connection device 770. In some embodiments, the working space of at least one driven tool 730 is the working space of at least one driven tool 730 in the driven tool base coordinate system {Tb}. It should be understood that the working space of at least one driven tool can also be an accessible space determined comprehensively based on the distribution of cavity walls and organs in the operating space 760, the type of surgical procedure, and the operator's instructions.

[0082] When at least one hand 790 is matched with at least one slave tool 730 and a master-slave mapping relationship is established, the initial position of at least one slave tool 730 determines the distance that at least one slave tool 730 can move in each direction within the workspace from the initial position. In some embodiments, the position vector of at least one slave tool 730 can be represented as the amount of position change of at least one slave tool 730 from the initial position to each position in the workspace. These movement vectors constitute the position vector space of at least one slave tool 730, such as position vector space 731.

[0083] In some embodiments, the control device may be configured to determine the positional changes of at least one driven tool 730 in each direction within the workspace based on the workspace of at least one driven tool 730 and the initial position of at least one driven tool 730, to form a position vector space 731 for at least one driven tool 730. In some embodiments, the initial position of at least one driven tool 730 may be the position of the driven tool coordinate system {wm} of at least one driven tool relative to a reference coordinate system {w} when at least one hand 790 is matched with at least one driven tool 730.

[0084] During master-slave control, the position change of at least one hand 790 is proportional to the position change of the driven device of at least one driven tool 730. The control device can be configured to transform the position vector space 731 of at least one driven tool 730 into the position vector space 791 of at least one hand 790 at the matching position of at least one hand 790, based on the pose mapping relationship between at least one hand 790 and at least one driven tool 730 and the matching position of at least one hand 790. In some embodiments, when at least one hand is matched with at least one driven tool, the control device can determine the position of the hand coordinate system {H} of at least one hand relative to the reference coordinate system {w}, based on the position of the end-effector coordinate system {E} of the exoskeleton assembly relative to the reference coordinate system {w} and the transformation relationship between the hand coordinate system {H} of at least one hand and the end-effector coordinate system {E} of the exoskeleton assembly, as the matching position of at least one hand.

[0085] In some embodiments, the control device can transform the position vector space 791 of at least one hand 790 into the position vector space 791 of the exoskeleton component 740, for example, position vector space 741, based on the transformation relationship between the hand coordinate system {H} of at least one hand and the end-effector coordinate system {E} of the exoskeleton component. The position boundary of the exoskeleton component 740 can be the boundary of the position vector space 741. It is understood that, for clarity, Figure 7 The position vector space 741 of the exoskeleton component is shown to coincide with the position vector space 791 of at least one hand 790, but they can be separate.

[0086] In some embodiments, the exoskeleton assembly may further include at least one exoskeleton braking device for locking at least one joint of the exoskeleton assembly. In some embodiments, the control device may be configured to generate a braking signal to drive at least one exoskeleton braking device to restrict the exoskeleton assembly from moving out of the position boundary in response to detecting that the exoskeleton assembly is moving out of the position boundary driven by at least one hand when the end of the exoskeleton assembly is located on the position boundary of the exoskeleton assembly. In some embodiments, the control device may also be configured to generate a control signal to release the restriction of at least one exoskeleton braking device on the exoskeleton assembly in response to detecting that the exoskeleton assembly is moving inward of the position boundary driven by at least one hand. It should be understood that the control device is not limited to restricting the movement of the exoskeleton assembly by at least one exoskeleton braking device. For example, the control device may also be configured to disconnect the master-slave mapping between at least one hand and at least one driven tool in response to detecting that the exoskeleton assembly is moving out of the position boundary driven by at least one hand.

[0087] In some embodiments, master-slave control of at least one slave tool may include: responding to a gesture of at least one hand matching a master-slave operation gesture, and controlling the slave instrument of at least one slave tool to perform an operation corresponding to the master-slave operation gesture based on the gesture and operation mapping relationship of at least one hand. For example, in a state where the gesture of at least one hand matches the master-slave operation gesture, a slave instrument control signal may be generated based on the gesture and operation mapping relationship of at least one hand to control the slave instrument of at least one slave tool to perform an operation corresponding to the master-slave operation gesture. In some embodiments, the set of master-slave operation gestures includes multiple slave operation gestures that have an operation mapping relationship with each operation of the slave instrument. The operation mapping relationship between at least one hand and the slave instrument of at least one slave tool may include the mapping relationship between the master-slave operation gesture matched with the gesture of at least one hand and the operation of the slave instrument. In some embodiments, the set of master-slave operation gestures and the mapping relationship between the master-slave operation gestures and the operation of the slave instrument may be predetermined and pre-stored in the memory of the surgical robot system.

[0088] In some embodiments, the operation of the driven instrument of at least one driven tool includes operating on the amount of operation of the driven instrument. The amount of operation of the driven instrument may include continuously adjustable or progressively adjustable amounts, such as the opening and closing angle of the driven instrument or the output energy. In some embodiments, the master-slave operation gesture may include a first partial gesture. The first partial gesture includes gesture features that can change continuously or progressively, and these gesture features have an operation mapping relationship with the amount of operation of the driven instrument. In some embodiments, the control device may be configured to: in response to a gesture of at least one hand matching a master-slave operation gesture, control the driven instrument of at least one driven tool to perform a corresponding operation based on the gesture features of the first partial gesture and the operation mapping relationship.

[0089] Figure 8-1 This diagram illustrates master-slave operation gestures 800 according to some embodiments of the present disclosure. Figure 8-2 A schematic diagram illustrating the operation of a driven device based on hand information according to some embodiments of the present disclosure is shown. Figure 8-1 The dotted lines in the middle indicate multiple changing states of the gesture features of the first local gesture 810. Figure 8-2 Solid dashed arrows illustrate the changing relationships between hand images, master-slave gestures, and the manipulation amounts of slave instruments. Hollow dashed arrows illustrate the mapping relationships between hand images, master-slave gestures, and the manipulation amounts of slave instruments. For example... Figure 8-1 and Figure 8-2 As shown, the master-slave operation gesture 800 may include a first local gesture 810. For example... Figure 8-1 As shown, the first partial gesture 810 may include the opening and closing between a first preset finger and a second preset finger, and the gesture features of the first partial gesture 810 include the opening and closing angle between the first preset finger and the second preset finger. The first preset finger can be, for example, the thumb of at least one hand, and the second preset finger can be, for example, the index finger of at least one hand.

[0090] In some embodiments, there is a mapping relationship between the gesture features of the first local gesture 810 and the operation of the driven device. For example... Figure 8-1 As shown, the gesture characteristics of the first local gesture 810 can change continuously or gradually. In some embodiments, the surgical instrument of at least one driven tool may include an operating unit with continuously adjustable operating volume, such as a clamping unit with continuously variable opening and closing angle, an injection unit with continuously variable injection dosage, etc. The opening and closing angle between the first preset finger and the second preset finger... Continuously variable, allowing for a continuous mapping between the gesture characteristics of the first local gesture 810 and the continuous operation of the surgical instrument's operating unit. Alternatively, the surgical instrument of at least one driven tool may include an operating unit with progressively adjustable operating volume, such as an ultrasonic scalpel with progressively adjustable output power. The opening angle between the first preset finger and the second preset finger... It can include multiple angle ranges This allows for a one-to-one mapping between the gesture features of the first local gesture 810 and the hierarchical operations of the surgical instrument's operating unit.

[0091] In some embodiments, the operational mapping relationship between at least one hand and at least one driven tool may include a correspondence between the operational amount of the driven instrument of at least one driven tool and the opening and closing angle of a first partial gesture. For example, the clamping angle as the operational amount of the driven instrument. The opening angle between the first preset finger and the second preset finger, which are gesture features of the first partial gesture. In master-slave control, consistency or proportionality can be maintained, such as... Figure 8-2 As shown.

[0092] In some embodiments, the control device is further configured to: in response to a gesture of at least one hand matching a master-slave operation gesture, determine a target operation amount of a slave device of at least one slave tool based on the opening and closing angle and operation mapping relationship of a first local gesture; and generate a first control signal of the slave device of at least one slave tool based on the target operation amount of the slave device of at least one slave tool. Figure 8-2 As shown, based on the master-slave operation gestures 800 matched with the gestures of at least one hand in hand image 891 and the operation mapping relationship, the initial operation amount of the slave device of at least one slave tool 830 can be determined; based on the master-slave operation gestures 800 matched with the gestures of at least one hand in hand image 892 and the operation mapping relationship, the target operation amount of the slave device of at least one slave tool 830 can be determined; and based on the initial operation amount and the target operation amount of the slave device of at least one slave tool 830, a first control signal of the slave device of at least one slave tool 830 can be generated to control the clamp of the slave device to move to the opening angle of the first local gesture 810 in the master-slave operation gestures 800. Clamping angle with mapping relationship In some embodiments, the target operating quantity of at least one driven instrument of a driven tool obtained from the previous control cycle may also be received as the starting operating quantity of at least one driven instrument of a driven tool in the current control cycle. Those skilled in the art will understand that... Figure 8-2The variations of master-slave gestures 800 shown are merely examples. One or more transitional master-slave gestures 800 may be included between the master-slave gestures 800 matching the gestures of at least one hand in hand image 891 and the master-slave gestures 800 matching the gestures of at least one hand in hand image 892, to correspond to continuous or gradual changes in the gesture features of the first local gesture 810.

[0093] It should be understood that the gesture features of the first partial gesture are not limited to the opening angle between the first preset finger and the second preset finger. Any gesture feature that can change continuously or gradually is within the scope of this disclosure. For example, the gesture features of the first partial gesture may also be the opening angle between other fingers, or the bending angle of the joint of a certain preset finger, etc.

[0094] In some embodiments, the operation of the driven instrument of at least one driven tool further includes operating the working state of the driven instrument. The working state of the driven instrument may include a first working state and a second working state, such as an on or off working mode or two working modes that can be switched between each other, such as the on / off state of an electrosurgical instrument or monopolar / bipolar switching. In some embodiments, the master-slave operation gesture may further include a second partial gesture. The second partial gesture includes gesture features that can change between two states, and these gesture features have an operation mapping relationship with the working state of the driven instrument. In some embodiments, the control device may be configured to: in response to a gesture of at least one hand matching the master-slave operation gesture, control the driven instrument of at least one driven tool to perform a corresponding operation based on the gesture features of the second partial gesture and the operation mapping relationship.

[0095] Figure 9 A schematic diagram of a master-slave operation gesture 900 according to some embodiments of the present disclosure is shown. Figure 9 Solid dashed arrows illustrate the changing relationships between hand images and between master and slave gestures, while hollow dashed arrows illustrate the mapping relationship between hand images and master and slave gestures. For example... Figure 9 As shown, the master-slave operation gesture 900 may include a first local gesture 910 and a second local gesture 920. The first local gesture 910 and... Figure 8-1 The first partial gesture 810 shown is similar, and its description is omitted here. In some embodiments, the second partial gesture 920 may include the opening and closing of a third preset finger, a fourth preset finger, and a fifth preset finger as a whole. The gesture features of the second partial gesture 920 include the opening and closing state of the third preset finger, the fourth preset finger, and the fifth preset finger as a whole. The third preset finger may be, for example, the middle finger of at least one hand, the fourth preset finger may be, for example, the ring finger of at least one hand, and the fifth preset finger may be, for example, the little finger of at least one hand.

[0096] In some embodiments, the operational mapping relationship between at least one hand and at least one driven tool may further include the working state of the driven device of at least one driven tool being consistent with the opening and closing state of the second local gesture. For example... Figure 9 As shown, the gesture features of the second local gesture 920 may include a first state and a second state. The first state may be, for example, a closed state in which the third preset finger, the fourth preset finger, and the fifth preset finger are bent as a whole (e.g., Figure 9 As shown in the left image (corresponding to hand image 991), the second state could be, for example, an open state where the third, fourth, and fifth preset fingers are all unfolded (e.g., ...). Figure 9 As shown in the right figure (corresponding to hand image 992), in some embodiments, there is a mapping relationship between the gesture features of the second local gesture 920 and the operating state of the driven device. For example, the operation mapping relationship may include a first state of the gesture features of the second local gesture 920 corresponding to a first operating state of the driven device, and a second state of the gesture features of the second local gesture 920 corresponding to a second operating state of the driven device.

[0097] In some embodiments, the control device may further be configured to: determine a target operating state of a driven instrument of at least one driven tool based on the opening and closing state and operation mapping relationship of a second local gesture, in response to a gesture of at least one hand matching a master-slave operation gesture; and generate a second control signal for the driven instrument of at least one driven tool based on the target operating state of the driven instrument of at least one driven tool. For example, as Figure 9 As shown, in response to a master-slave operation gesture 900 where the gesture of at least one hand in hand image 992 is in an open state, the target working state of at least one slave tool's slave device can be determined based on the open state of the second local gesture 920 of the master-slave operation gesture 900 and the operation mapping relationship between the open state of the second local gesture 920 and the working state of the slave device. Based on the target working state of the slave device of at least one slave tool, a second control signal of the slave device of at least one slave tool is generated to control the slave device to switch from an initial working state associated with the gesture of at least one hand in hand image 991 to the target working state. In some embodiments, the initial working state of the slave device may correspond to a first working state, and the target working state of the slave device may correspond to a second working state.

[0098] Those skilled in the art will understand that Figure 9The master-slave operation gesture 900 shown is merely an example. The first local gesture 910 and the second local gesture 920 of the master-slave operation gesture 900 are independent of each other. The control device can simultaneously or sequentially control the operation amount and working state of the driven instrument of at least one driven tool based on the gesture characteristics and operation mapping relationship of the first local gesture 910 and the gesture characteristics and operation mapping relationship of the second local gesture 920, when the gesture of at least one hand matches the master-slave operation gesture.

[0099] It should be understood that the gesture features of the second partial gesture are not limited to the opening and closing state of the third, fourth, and fifth preset fingers as a whole. Any gesture feature that can switch between two states is within the scope of this disclosure. For example, the gesture features of the second partial gesture may also be the opening and closing state of multiple other fingers as a whole, or it may be the opening and closing state of a certain preset finger, etc.

[0100] In some embodiments, master-slave control of at least one slave tool further includes disconnecting the master-slave mapping between at least one hand and at least one slave tool. In some embodiments, the master-slave operation gesture may further include a third local gesture. The third local gesture includes gesture features for indicating the disconnection of the master-slave mapping of at least one slave tool.

[0101] Figure 10 A schematic diagram of a master-slave operation gesture 1000 according to some embodiments of the present disclosure is shown. Figure 10 Solid dashed arrows illustrate the changing relationships between hand images and between master and slave gestures, while hollow dashed arrows illustrate the mapping relationship between hand images and master and slave gestures. For example... Figure 10 As shown, the master-slave operation gesture 1000 may include a first local gesture 1010 and a second local gesture 1020. The first local gesture 1010 and... Figure 8-1 The first local gesture 810 shown and Figure 9 The first partial gesture 910 shown is similar to the second partial gesture 1020. Figure 9 The second partial gesture 920 shown is omitted here. The master-slave operation gesture 1000 may also include a third partial gesture 1030. In some embodiments, the third partial gesture 1030 may include a fifth preset finger opening relative to a fourth preset finger. The fourth preset finger may be, for example, the ring finger of at least one hand, and the fifth preset finger may be, for example, the little finger of at least one hand. In some embodiments, the opening of the fifth preset finger relative to the fourth preset finger may include an opening angle of the fifth preset finger relative to the fourth preset finger that is greater than an angle threshold. ,like Figure 10 As shown.

[0102] In some embodiments, method 200 may include: in a state where the gesture of at least one hand matches a master-slave operation gesture, in response to the master-slave operation gesture including a third local gesture, disconnecting the master-slave mapping between at least one hand and at least one slave tool. For example, Figure 10 As shown, in response to a gesture of at least one hand in hand image 1092 matching with a master-slave operation gesture 1000 including a third local gesture 1030, the master-slave mapping between at least one hand and at least one slave tool can be disconnected. It should be understood that the gesture features of the third local gesture are not limited to the fifth preset finger being open relative to the fourth preset finger; any gesture that can be opened and closed independently based on the second local gesture is within the scope of this disclosure.

[0103] In this disclosure, before or during a surgical procedure, at least one slave tool needs to be assigned to at least one hand of the operator to establish an association between at least one hand and at least one slave tool, thereby allowing at least one hand to match with at least one slave tool and even establish a master-slave mapping relationship.

[0104] In some embodiments, prior to surgical procedures, at least one slave tool may be assigned to at least one hand based on a pre-set configuration. In some embodiments, the control device may also be configured to: in response to a gesture of at least one hand matching a ready gesture indicating that the operator is in a ready state to perform surgical procedures, assign at least one slave tool to at least one hand based on a pre-set configuration; and generate assignment status information to display the assignment status of at least one hand and at least one slave tool. In some embodiments, the pre-set configuration may include the assignment relationship between at least one slave tool and at least one hand, which may be pre-set and stored in the memory of the surgical robot system. The assignment status information may include, for example, image information, text information, or audio information. In some embodiments, the assignment status information may be overlaid on the surgical field image; for example, a highlighted frame may be generated in the surgical field image to surround the tool arm of at least one slave tool, highlighting at least one slave tool assigned to at least one hand. The ready gesture may include, but is not limited to, a gesture with the palm facing inward and the fingers extended.

[0105] In some embodiments, surgical robot systems (e.g.) Figure 1 The surgical robot system 100 shown may include multiple driven tools (e.g., Figure 1 The driven tool 130 shown Figure 2 The driven tool 230 shown Figure 7 The driven tool 730 shown Figure 11 The first driven tool 1131, the second driven tool 1132, and the third driven tool 1133 are shown. Figure 12The first driven tool 1231, the second driven tool 1232, and the third driven tool 1233 shown are... Figure 13 (See slave tool 1330). In some embodiments, the preset may further include a tool allocation pattern corresponding to the surgical procedure, allowing at least one slave tool from a plurality of slave tools to be automatically allocated to at least one hand after detecting a match between a gesture of at least one hand and a ready gesture. In some embodiments, the tool allocation pattern may be preset based on various surgical procedures and stored in the memory of the surgical robot system. In response to a match between a gesture of at least one hand and a ready gesture, at least one slave tool may be allocated to the corresponding at least one hand based on the tool allocation pattern corresponding to the current surgical procedure. In some embodiments, the tool allocation pattern may also be preset based on the operator's operating habits and stored in the memory of the surgical robot system in association with the operator's identification.

[0106] In some embodiments, before or during a surgical procedure, at least one slave tool can be assigned to at least one hand based on an operator's assignment request. In some embodiments, the control device can be configured to: generate assignment prompt information in response to at least one slave tool being in an unassigned state, prompting the operator to request assignment of at least one slave tool; assign at least one slave tool corresponding to at least one assignment gesture to at least one hand in response to a gesture of at least one hand matching at least one assignment gesture; and generate assignment status information to display the assignment status of at least one hand and at least one slave tool.

[0107] The allocation prompt message is used to prompt the operator to request the allocation of at least one slave tool. In some embodiments, if at least one hand has not established a master-slave mapping with at least one slave tool or the master-slave mapping is broken, an allocation prompt message may be generated in response to at least one slave tool being in an unallocated state, to prompt the operator to request the allocation of at least one slave tool. In some embodiments, the allocation prompt message may include image information, text information, or audio information, such as a virtual image superimposed on the surgical field image. In some embodiments, the allocation request may include at least one allocation gesture.

[0108] Figure 11 This diagram illustrates the allocation request for at least one slave tool according to some embodiments of the present disclosure. Figure 11As shown, the surgical robot system may include at least one slave tool, such as a first slave tool 1131, a second slave tool 1132, and a third slave tool 1133. In some embodiments, in response to the first slave tool 1131, the second slave tool 1132, and the third slave tool 1133 being in an unassigned state, a corresponding assignment prompt message may be generated near each slave tool to prompt the operator to make an assignment request to the corresponding slave tool according to the assignment prompt message. The assignment prompt message may be, for example, a numerical prompt message associated with the serial number of the slave tool, such as the first assignment prompt message 11310, the second assignment prompt message 11320, and the third assignment prompt message 11330. In some embodiments, the serial number of the slave tool may be determined by the arm body information of the motion arm used to carry the slave tool or arranged clockwise or counterclockwise. In some embodiments, the control device may also be configured to: obtain the initial position of at least one slave tool, and generate assignment prompt messages based on the initial position of at least one slave tool.

[0109] In some embodiments, at least one assignment gesture is associated with at least one slave tool. In some embodiments, at least one assignment gesture may be predetermined and pre-stored in the memory of the surgical robot system in association with at least one slave tool. In some embodiments, at least one assignment gesture may be a static gesture associated with the serial number of at least one slave tool, such as a digital gesture. Figure 11 As shown, at least one allocation gesture may include an allocation gesture 1102 that has an allocation association with the second slave tool 1132. The operator may request the allocation of the second slave tool 1132 based on allocation prompt information. In response to a gesture of at least one hand matching the allocation gesture 1102, the second slave tool 1132 corresponding to the allocation gesture 1102 may be allocated to at least one hand.

[0110] In some embodiments, at least one assignment gesture may also be a temporarily generated gesture. For example, in response to at least one slave tool being in an unassigned state, at least one assignment gesture with an assignment association with at least one slave tool may be generated to update the gesture sample set; in response to a gesture of at least one hand matching at least one assignment gesture, at least one slave tool corresponding to at least one assignment gesture may be assigned to at least one hand; and at least one assignment gesture may be deleted from the gesture sample set to update the gesture sample set again.

[0111] In some embodiments, at least one allocation gesture with an allocation association to at least one driven tool can be generated in real time based on the position of at least one driven tool in the surgical field image. In some embodiments, the control device can also be configured to: obtain the initial position of the driven instrument of at least one driven tool; and generate at least one allocation gesture based on the initial position of the driven instrument of at least one driven tool, an allocation reference point, and a preset allocation gesture. The allocation reference point is the starting reference point of at least one allocation gesture, and may be, for example, the center point of the surgical field image (e.g., Figure 12 The allocation reference point (P) or edge midpoint, etc., is shown. The initial position of the driven instrument of at least one driven tool is the termination reference point of at least one allocation gesture. A preset allocation gesture is used to generate the allocation gesture and can be predetermined and stored in the memory of the surgical robot system. In some embodiments, the gesture movement direction can be determined based on the initial position of the driven instrument of at least one driven tool and the position of the allocation reference point, and at least one allocation gesture can be generated based on the gesture movement direction and the preset allocation gesture.

[0112] Figure 12 This diagram illustrates a request to assign at least one slave tool according to other embodiments of the present disclosure. Figure 12 The dashed arrows indicate the relationship between the preset assigned gestures and the assigned gestures, while the hollow arrows indicate the movement process of the dynamic gestures. For example... Figure 12 As shown, the surgical robot system may include at least one driven tool, such as a first driven tool 1231, a second driven tool 1232, and a third driven tool 1233. In some embodiments, in response to the second driven tool 1232 being in an unassigned state, the initial position of the driven instrument of the second driven tool 1232 can be obtained. Based on the initial position of the driven instrument of the second driven tool 1232 and the position of the assignment reference point P in the surgical field image, the gesture movement direction corresponding to the second driven tool 1232 can be determined. Based on the gesture movement direction and a preset assignment gesture 1200, a dynamic assignment gesture 1202 can be generated.

[0113] In some embodiments, the control device may also be configured to generate allocation prompt information based on the initial position of the driven instrument of at least one driven tool and an allocation reference point. The allocation prompt information may be, for example, a virtual image overlaid on the surgical field image, or a virtual arrow pointing from the allocation reference point to the initial position of the driven instrument of at least one driven tool, to prompt for at least one allocation gesture associated with at least one driven tool. Figure 12As shown, allocation prompt information corresponding to each slave tool can be generated based on the initial position of the slave device of the first slave tool 1231, the initial position of the slave device of the second slave tool 1232, the initial position of the slave device of the third slave tool 1233, and the position of the allocation reference point P. For example, the first allocation prompt information 12310, the second allocation prompt information 12320, and the third allocation prompt information 12330 can be generated.

[0114] In some embodiments, after at least one slave tool is assigned to at least one hand, assignment status information can be generated to indicate to the operator that an assignment relationship has been established between at least one hand and at least one slave tool. In some embodiments, the assignment status information can be overlaid on the surgical field image; for example, a highlighted frame can be generated in the surgical field image to surround the tool arm of at least one slave tool, highlighting at least one slave tool that has an assignment relationship with at least one hand.

[0115] In surgical procedures, the operator exercises master-slave control over at least one slave tool by waving at least one hand. During the initiation of master-slave control, if the posture (e.g., orientation or angle) or gesture of at least one hand does not match the posture (e.g., orientation or angle) or working state of the slave instrument of at least one slave tool, the operator will find it difficult to experience intuitive control, reducing the operator's precision in controlling the slave tool and thus posing surgical risks. In some embodiments, after assigning at least one slave tool to at least one hand, the operator may be prompted to match the at least one slave tool.

[0116] In some embodiments, the control device may be configured to: generate a matching prompt message in response to at least one slave tool having an assignment relationship with at least one hand being in an unmatched state, prompting the operator to request a matching of at least one slave tool; and match at least one slave tool with at least one hand in response to at least one hand's posture matching at least one matching posture and at least one hand's gesture matching at least one matching gesture; and generate matching status information to indicate that at least one hand and at least one slave tool are in a matched state.

[0117] In some embodiments, if a master-slave mapping is not established or is broken between at least one hand and at least one slave tool, a matching prompt message can be generated in response to a mismatch between at least one slave tool and at least one hand, prompting the operator to request a matching of at least one slave tool. In some embodiments, a mismatch or non-match between at least one slave tool and at least one hand may include: the posture of the slave device of at least one slave tool is inconsistent with the posture of at least one hand and / or the working state of the slave device of at least one slave tool is inconsistent with the gesture of at least one hand. The control device may also be configured to: determine whether the postures of the slave device of at least one slave tool and at least one hand match based on the postures of the slave device of at least one slave tool and at least one hand; and determine whether the states of the slave device of at least one slave tool and at least one hand match based on the states of the slave device of at least one slave tool and at least one hand.

[0118] Figure 13 This diagram illustrates a matching request for a driven instrument of at least one driven tool according to other embodiments of the present disclosure. For ease of understanding, Figure 13 The diagram only shows a state where the posture of at least one driven tool 1330's driven instrument does not match the posture of at least one hand, but these could also be states where the working state of at least one driven tool 1330's driven instrument does not match the gesture of at least one hand. For example... Figure 13 As shown, in response to at least one slave tool 1330 having an assignment relationship with at least one hand being in an unmatched state, a matching prompt message can be generated to prompt the operator to make a matching request for at least one slave tool 1330.

[0119] In some embodiments, the matching prompt information may include image information, text information, or audio information, such as a virtual image superimposed on the surgical field image. Figure 13As shown, the matching prompt information includes a virtual image 1391 of at least one hand and / or a virtual image 1331 of a driven instrument of at least one driven tool. Virtual images 1391 and 1331 can be matched with and driven by at least one hand. In some embodiments, a virtual image of at least one hand and / or a virtual image of at least one driven tool can be generated in the surgical field image based on hand information of at least one hand. For example, the current posture and current gesture of at least one hand can be obtained; and based on the current posture and current gesture of at least one hand, a virtual image of at least one hand that matches and is driven by at least one hand / or a virtual image of a driven instrument of at least one driven tool can be generated in the surgical field image. In some embodiments, obtaining the current posture of at least one hand can include the current posture of the hand coordinate system {H} of at least one hand relative to a reference coordinate system {w}.

[0120] By generating a virtual image in the surgical field that matches and has a virtual master-slave mapping relationship with at least one hand, the operator can intuitively match the driven instrument of at least one driven tool with at least one hand based on the surgical field image. It should be understood that the method of generating a virtual image in the surgical field image is not limited to the matching process of at least one hand with at least one driven tool; it can also be generated during the allocation process of at least one hand with at least one driven tool or during the establishment of a master-slave mapping.

[0121] In some embodiments, the control device may further be configured to: obtain the initial posture of the driven instrument of at least one driven tool as at least one matched posture. Obtaining the initial posture of the driven instrument of at least one driven tool may include obtaining the initial posture of the driven instrument of at least one driven tool relative to the driven tool base coordinate system {Tb}, for example, the initial pose of the driven instrument coordinate system {wm} of at least one driven tool relative to the driven tool base coordinate system {Tb}. In some embodiments, whether the posture of at least one hand matches the at least one matched posture may be determined based on the posture matching degree between the posture of at least one hand and the at least one matched posture.

[0122] In some embodiments, the control device may further be configured to: obtain an initial state of the slave device of at least one slave tool; and determine at least one matching gesture based on the initial state of the slave device of at least one slave tool. In some embodiments, a master-slave operation gesture that has a mapping relationship with the initial state of the slave device of at least one slave tool may be searched in a gesture sample set based on the initial state of the slave device of at least one slave tool, and used as at least one matching gesture. In some embodiments, the initial state of the slave device of at least one slave tool may be obtained based on a preset state of the slave device or a previous state of the slave device of at least one slave tool. Alternatively, the initial state of the slave device of at least one slave tool may also be obtained based on the control signal of at least one slave tool.

[0123] In some embodiments, the control device may generate matching status information to indicate to the operator that at least one hand has established a matching relationship with at least one driven tool in response to at least one hand matching with at least one driven tool. In some embodiments, the matching status information may be overlaid on the surgical field image, for example, a highlighted frame may be generated in the surgical field image to surround the driven instrument of at least one driven tool, highlighting the driven instrument of at least one driven tool that has a matching relationship with at least one hand.

[0124] In some embodiments, after at least one hand and at least one driven tool enter a matching state, a master-slave mapping relationship can be established between at least one hand and at least one driven tool. By establishing a master-slave mapping relationship, it can be ensured that the pose changes of the driven instruments of at least one driven tool perceived by the operator in the surgical field image maintain a preset pose mapping relationship with the pose changes of at least one hand perceived by the operator, and that the working state of the driven instruments of at least one driven tool perceived by the operator maintains a preset operation mapping relationship with the gesture changes of at least one hand perceived by the operator, thereby achieving intuitive operation.

[0125] In some embodiments, the control device may be configured to generate a mapping prompt message in response to at least one slave tool having a matching relationship with at least one hand being in an unmapped state, prompting the operator to request mapping of at least one slave tool; and to establish a master-slave mapping between at least one hand and at least one slave tool in response to the master-slave mapping condition being met.

[0126] In some embodiments, in response to at least one slave tool being matched with at least one hand and at least one slave tool not having established a master-slave mapping relationship with at least one hand, a mapping prompt message may be generated to prompt the operator to request mapping of at least one slave tool. The mapping prompt message may include image information, text information, or audio information, etc.

[0127] In some embodiments, the mapping prompts may include prompts associated with the master-slave mapping conditions to inform the operator of the conditions that need to be met to establish the master-slave mapping. In some embodiments, the master-slave mapping conditions include: at least one hand maintaining a posture and gesture in a matching state for more than a preset time threshold. The mapping prompts may include, for example, countdown voice prompts, countdown timers, etc., to allow the operator to intuitively establish the master-slave mapping. For example, when the operator maintains the posture and gesture of at least one hand for more than a preset time threshold, a master-slave mapping between at least one hand and at least one slave tool can be established, and the operator can be informed that a master-slave mapping relationship has been established by highlighting or displaying a virtual image, indicating that master-slave control can be performed on at least one slave tool using at least one hand.

[0128] Those skilled in the art will understand that surgical robot systems can also be controlled by gestures from the operator's two hands. In some embodiments, hand information of the operator's two hands can be determined based on hand images, and master-slave control can be performed on the slave tools that have established master-slave mappings with the two hands respectively, based on the hand information of the two hands and the master-slave mapping relationship.

[0129] Furthermore, in surgical robot systems, it is not limited to operating at least one driven tool through gestures of at least one hand; it is also possible to operate a driven vision device through gestures of at least one hand to adjust the surgical field. In some embodiments, the driven vision device can be assigned, matched, and a master-slave mapping can be established with at least one hand, and the driven vision device can be controlled in a master-slave manner based on the hand information of at least one hand and the master-slave mapping relationship.

[0130] In some embodiments, in response to a gesture of at least one hand matching a master-slave operation gesture, the movement of the slave vision device can be controlled based on the pose and pose mapping relationship of at least one hand; and / or in response to a gesture of at least one hand matching a master-slave operation gesture, the slave vision device can be controlled to perform an operation corresponding to the master-slave operation gesture based on the gesture and operation mapping relationship of at least one hand. In some embodiments, the master-slave operation gesture includes a first partial gesture, which can be used to adjust the depth of field of the endoscope mounted on the slave vision device or the light intensity of the illumination device, etc. In some embodiments, the master-slave operation gesture also includes a second partial gesture, which can be used to turn the fluorescence mode on or off, etc.

[0131] Note that the above are merely exemplary embodiments and technical principles of this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, this disclosure is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this disclosure, the scope of which is determined by the scope of the appended claims.

Claims

1. A surgical robot system based on gesture control, characterized in that, include: An exoskeleton assembly for wearing on at least one upper limb of an operator to support and follow the movement of the at least one upper limb; The exoskeleton assembly includes a wearable structure for wearing on the at least one upper limb and at least one exoskeleton sensor disposed at at least one joint on the wearable structure, the at least one exoskeleton sensor being configured to obtain joint information of the at least one joint; An image acquisition device for acquiring images of the hand of at least one hand on at least one side of the upper limb; At least one driven tool, including a tool arm and a driven device disposed at the end of the tool arm; A control device is connected to the exoskeleton assembly, the image acquisition device, and the at least one slave tool. The control device is configured to: determine hand information of the at least one hand based on the hand image and the pose of the exoskeleton assembly, the hand information including the pose and / or gesture of the at least one hand; and perform master-slave control on the at least one slave tool based on the hand information of the at least one hand and the master-slave mapping relationship between the at least one hand and the at least one slave tool, the master-slave mapping relationship including the pose mapping relationship and operation mapping relationship between the at least one hand and the slave device of the at least one slave tool. The control device is also configured to: In response to the gesture of the at least one hand matching the master-slave operation gesture, the movement of the slave mechanism of the at least one slave tool is controlled based on the pose of the at least one hand and the pose mapping relationship; And / or, in response to the gesture of the at least one hand matching the master-slave operation gesture, based on the gesture of the at least one hand and the operation mapping relationship, control the slave device of the at least one slave tool to perform the operation corresponding to the master-slave operation gesture; The control device is further configured to determine the pose of the end of the exoskeleton assembly based on joint information of at least one joint of the exoskeleton assembly, as the pose of the exoskeleton assembly.

2. The surgical robot system according to claim 1, characterized in that, The system also includes: A driven vision device for acquiring surgical field images, the surgical field images including images of the at least one driven tool; and Display device for displaying the surgical field image.

3. The surgical robot system according to claim 2, characterized in that, The pose mapping relationship includes at least one of the following: The change in position of the at least one driven instrument in the surgical field image relative to the reference coordinate system is proportional to the change in position of the at least one hand relative to the reference coordinate system; or The pose change of the at least one driven instrument in the surgical field image relative to the reference coordinate system is consistent with the pose change of the at least one hand relative to the reference coordinate system.

4. The surgical robot system according to claim 3, characterized in that, The control device is also configured to: Determine the current pose of at least one hand; Obtain the previous pose of at least one hand; Obtain the initial pose of the driven device of the at least one driven tool; Based on the previous pose of the at least one hand, the current pose of the at least one hand, the initial pose of the slave device of the at least one driven tool, and the pose mapping relationship, the target pose of the slave device of the at least one driven tool is determined. as well as Based on the target pose of the slave device of the at least one slave tool, the at least one slave tool is controlled to move, so that the slave device of the at least one slave tool moves to the target pose.

5. The surgical robot system according to claim 1, characterized in that, The system also includes at least one exoskeleton drive for driving the at least one joint of the exoskeleton assembly, and the at least one exoskeleton sensor is further configured to obtain joint torque of the at least one joint; The control device is also configured to: Obtain the actual torque of the at least one joint; Determine the compensating torque of the at least one joint; Determine the desired torque of the at least one joint; Based on the actual torque, the expected torque, and the compensation torque of the at least one joint, the torque error of the at least one joint is determined; as well as The drive signal of the at least one exoskeleton drive device is determined based on the torque error of the at least one joint.

6. The surgical robot system according to claim 1, characterized in that, The control device is also configured to: In response to the exoskeleton component being positioned on the position boundary of the exoskeleton component, a braking signal is generated to prevent the exoskeleton component from moving beyond the position boundary under the influence of the at least one upper limb.

7. The surgical robot system according to claim 6, characterized in that, The control device is also configured to: Based on the initial position of the at least one driven tool and the workspace of the at least one driven tool, determine the position vector space of the at least one driven tool; The position vector space of the at least one hand is determined based on the matching position of the at least one hand, the position vector space of the at least one driven tool, and the pose mapping relationship; as well as The positional boundaries of the exoskeleton component are determined based on the position vector space of the at least one hand.

8. The surgical robot system according to claim 1, characterized in that, The master-slave operation gesture includes a first partial gesture, which includes opening and closing between a first preset finger and a second preset finger. The operation mapping relationship includes: The amount of operation of the driven instrument of the at least one driven tool corresponds to the opening and closing angle of the first local gesture.

9. The surgical robot system according to claim 8, characterized in that, The control device is also configured to: In response to the matching of the gesture of the at least one hand with the master-slave operation gesture, the target operation amount of the slave instrument of the at least one slave tool is determined based on the opening and closing angle of the first local gesture and the operation mapping relationship; as well as Based on the target operating quantity of the driven instrument of the at least one driven tool, a first control signal of the driven instrument of the at least one driven tool is generated.

10. The surgical robot system according to claim 8, characterized in that, The master-slave operation gesture also includes a second partial gesture, which includes the opening and closing of a third preset finger, a fourth preset finger, and a fifth preset finger relative to the second preset finger. The operation mapping relationship also includes: The working state of the driven instrument of the at least one driven tool is consistent with the opening and closing state of the second local gesture.

11. The surgical robot system according to claim 10, characterized in that, The control device is also configured to: In response to the matching of the gesture of the at least one hand with the master-slave operation gesture, the target working state of the slave device of the at least one slave tool is determined based on the opening and closing state of the second local gesture and the operation mapping relationship; as well as Based on the target working state of the driven instrument of the at least one driven tool, a second control signal for the driven instrument of the at least one driven tool is generated.

12. The surgical robot system according to claim 8, characterized in that, The master-slave operation gesture also includes a third partial gesture, which includes a fifth preset finger opening relative to a fourth preset finger. The control device is further configured to: When the gesture of the at least one hand matches the master-slave operation gesture, in response to the master-slave operation gesture including the third local gesture, the master-slave mapping between the at least one hand and the at least one slave tool is disconnected.

13. The surgical robot system according to claim 1, characterized in that, The control device is also configured to: In response to a gesture of the at least one hand matching a ready gesture indicating that the operator is in a ready state to perform surgical procedures, the at least one slave tool is assigned to the at least one hand based on a preset setting; as well as Generate allocation status information to display the allocation status of the at least one hand and the at least one driven tool.

14. The surgical robot system according to claim 1, characterized in that, The control device is also configured to: In response to the fact that the at least one slave tool is in an unassigned state, an assignment prompt message is generated to prompt the operator to make an assignment request for the at least one slave tool; In response to a gesture of the at least one hand matching at least one assignment gesture, the at least one slave tool corresponding to the at least one assignment gesture is assigned to the at least one hand; as well as Generate allocation status information to display the allocation status of the at least one hand and the at least one driven tool.

15. The surgical robot system according to claim 14, characterized in that, The control device is also configured to: To obtain the initial position of the driven mechanism of the at least one driven tool; and The at least one allocation gesture is generated based on the initial position of the slave device of the at least one slave tool, the allocation reference point, and the preset allocation gesture.

16. The surgical robot system according to claim 1, characterized in that, The control device is also configured to: In response to the fact that at least one slave tool having an allocation relationship with the at least one hand is in an unmatched state, a matching prompt message is generated to prompt the operator to make a matching request for the at least one slave tool; as well as In response to the at least one hand's posture matching at least one matching posture and the at least one hand's gesture matching at least one matching gesture, the at least one slave tool is matched with the at least one hand; as well as Generate matching status information to show that the at least one hand is in a matching state with the at least one driven tool.

17. The surgical robot system according to claim 16, characterized in that, The control device is also configured to: Obtain the initial state of the driven mechanism of the at least one driven tool; and The at least one matching gesture is determined based on the initial state of the driven instrument of the at least one driven tool.

18. The surgical robot system according to claim 16, characterized in that, The control device is also configured to: The initial posture of the driven instrument of the at least one driven tool is obtained as the at least one matched posture.

19. The surgical robot system according to claim 1, characterized in that, The control device is also configured to: In response to the fact that at least one slave tool that is matched with at least one hand is in an unmapped state, a mapping prompt message is generated to prompt the operator to make a mapping request for the at least one slave tool; as well as In response to the fulfillment of the master-slave mapping condition, a master-slave mapping is established between the at least one hand and the at least one slave tool.

20. The surgical robot system according to claim 19, characterized in that, The master-slave mapping condition includes: at least one hand maintains its posture and gesture in the matching state for more than a preset time threshold.

21. The surgical robot system according to any one of claims 2, 8 to 20, characterized in that, Also includes: Based on the hand information of the at least one hand, a virtual image of the at least one hand and / or a virtual image of the at least one driven tool are generated in the surgical field image.

22. The surgical robot system according to any one of claims 1 to 20, characterized in that, The control device is also configured to: Determine the relative three-dimensional coordinates of multiple key points of the at least one hand in the hand image; The gesture of at least one hand is determined based on the relative three-dimensional coordinates of the multiple key points.

23. The surgical robot system according to any one of claims 1 to 20, characterized in that, The control device is also configured to: Obtain the pose of the exoskeleton assembly; and The pose of the at least one hand is determined based on the pose of the exoskeleton component.

Images