Gesture-based robot system motion control method and robot system

Abstract

The invention relates to the field of robot control, and discloses a motion control method of a robot system based on gestures, the robot system, computer equipment and a storage medium. The robot system comprises a mobile station and at least one moving arm arranged on the mobile station, and the method comprises the steps that a hand image set of at least one hand is obtained; obtaining gesture information of at least one hand based on the hand image set; and controlling the movement of the mobile station or the at least one movement arm based on the gesture information.

Description

Technical Field

[0001] The present disclosure relates to the field of medical devices, and in particular to a motion control method of a gesture-based robot system, a robot system, a computer device, and a storage medium. Background Art

[0002] Laparoscopic surgery is a surgical form that has gradually developed and been widely used in recent years. It has advantages such as small incisions, which greatly reduces patients' recovery time, discomfort experience and post-healing side effects.

[0003] The existing laparoscopic surgical robot system uses a moving arm to carry surgical instruments. Depending on the patient and surgical procedure, the moving arm needs to be adjusted and positioned before or during surgery to adjust the surgical instruments to the specified surgical position. During the operation, the surgeon uses the teleoperation mode to control the surgical actuator at the end of the surgical instrument to perform surgical operations on different parts of the body.

[0004] The existing surgical robot system is moved by human dragging, which is inconvenient and time-consuming. The position adjustment of the motion arm is usually adjusted automatically by the system, and it takes a long time to adjust some specific positions. The mobility of the surgical robot system, as well as the position adjustment and positioning capabilities of the external motion arm in space, are directly related to the preoperative preparation time and operation experience. Summary of the invention

[0005] In some embodiments, the present disclosure provides a motion control method for a gesture-based robot system, wherein the robot system includes a mobile station and at least one motion arm disposed on the mobile station, and the method includes: obtaining a hand image set of at least one hand; based on the hand image set, obtaining gesture information of at least one hand; and based on the gesture information, controlling the movement of the mobile station or at least one motion arm.

[0006] In some embodiments, the present disclosure provides a robot system, comprising: a mobile station; at least one motion arm, arranged on the mobile station; an image acquisition device, for acquiring hand images of at least one hand; and a control device, communicatively connected to the mobile station, at least one motion arm and the image acquisition device, for executing any one of the gesture-based motion control methods of the robot system in some embodiments of the present disclosure.

[0007] In some embodiments, the present disclosure provides a computer device, comprising: a memory for storing at least one instruction; and a processor, coupled to the memory and for executing at least one instruction to execute the motion control method of a gesture-based robotic system of any one of some embodiments of the present disclosure.

[0008] In some embodiments, the present disclosure provides a computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes the computer to implement a motion control method for a gesture-based robotic system as described in any one of some embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments of the present disclosure. The drawings described below only illustrate some embodiments of the present disclosure. For ordinary technicians in this field, other embodiments can be obtained based on the contents of the embodiments of the present disclosure and these drawings without creative work.

[0010] Figure 1 A flowchart of a motion control method for a gesture-based robotic system according to some embodiments of the present disclosure is shown.

[0011] Figure 2 A structural block diagram of a robot system according to some embodiments of the present disclosure is shown.

[0012] Figure 3 A schematic diagram of the structure of a surgical robot system according to some embodiments of the present disclosure is shown.

[0013] Figure 4 A schematic diagram showing a hand key point model according to some embodiments of the present disclosure.

[0014] Figure 5 A schematic diagram showing a hand coordinate system according to some embodiments of the present disclosure.

[0015] Figure 6A-6D Schematic diagrams showing various hand configurations according to some embodiments of the present disclosure.

[0016] Figure 7 A flowchart of a motion control method for a gesture-based robotic system according to some embodiments of the present disclosure is shown.

[0017] Figure 8 A simplified schematic diagram illustrating control points of a mobile station according to some embodiments of the present disclosure.

[0018] Fig. 9A A schematic diagram showing gesture operations for controlling the forward movement of a mobile station according to some embodiments of the present disclosure is shown.

[0019] Fig. 9B A schematic diagram showing gesture operations for back control of a mobile station according to some embodiments of the present disclosure is shown.

[0020] Fig.10 A simplified schematic diagram showing a positioning device in a mobile station according to some embodiments of the present disclosure.

[0021] FIG. 11 shows a flow chart of a motion control method for a gesture-based robotic system according to some embodiments of the present disclosure.

[0022] Fig.12 A simplified schematic diagram showing control points of a motion arm according to some embodiments of the present disclosure.

[0023] Fig.13A A schematic diagram of gesture operations for controlling the position of at least one motion arm of some embodiments of the present disclosure is shown.

[0024] Fig. 13B A schematic diagram of gesture operations for controlling the posture of at least one motion arm of some embodiments of the present disclosure is shown.

[0025] Fig. 13C A schematic diagram of gesture operations for controlling the position of at least one motion arm of some embodiments of the present disclosure is shown.

[0026] Fig.14 A schematic structural diagram of a mobile station according to some other embodiments of the present disclosure is shown.

[0027] Fig.15 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0028] In order to make the technical problems solved by the present disclosure, the technical solutions adopted and the technical effects achieved more clearly, the technical solutions of the embodiments of the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only exemplary embodiments of the present disclosure, rather than all embodiments.

[0029] In the description of the present disclosure, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present disclosure, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "coupled" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present disclosure can be understood according to the specific circumstances. In the present disclosure, the end close to the operator (e.g., medical staff) is defined as the proximal end, near part, or rear end, rear part, and the end opposite to the proximal end, near part, or rear end, rear part is defined as the distal end, far end, or front end, front part. Alternatively, the end close to the operator (e.g., surgical patient) is defined as the distal end, far end, or front end, front part, and the end opposite to the distal end, far end, or front end, front part is defined as the proximal end, near part, or rear end, rear part. Those skilled in the art will appreciate that the embodiments of the present disclosure may be used for medical instruments or surgical robots, and may also be used for other non-medical devices.

[0030] In the present disclosure, the term "position" refers to the positioning of an object or a part of an object in three-dimensional space (for example, three translational degrees of freedom can be described using changes in Cartesian X, Y and Z coordinates, such as three translational degrees of freedom along Cartesian X, Y and Z axes, respectively). In the present disclosure, the term "posture" refers to the rotation setting of an object or a part of an object (for example, three rotational degrees of freedom, which can be described using roll, pitch and yaw). In the present disclosure, the term "pose" refers to a combination of the position and pose of an object or a part of an object, for example, it can be described using six parameters of the six degrees of freedom mentioned above. In the present disclosure, the configuration or posture of a moving arm or a part thereof can be represented by a set of joint values ​​of the joints of the moving arm (for example, a one-dimensional matrix composed of these joint values). In the present disclosure, the joint values ​​of the joints can include the angles rotated by the corresponding joints relative to the corresponding joint axes or the distances moved relative to the initial positions. In the present disclosure, a reference coordinate system can be understood as a coordinate system capable of describing the pose of an object. According to the actual positioning requirements, the reference coordinate system can choose to use the origin of the virtual reference object or the origin of the physical reference object as the origin of the coordinate system.

[0031] Some embodiments of the present disclosure provide a motion control method for a gesture-based robotic system. Figure 1 1 is a flowchart of a method 100 (hereinafter referred to as “method 100”) for motion control of a robot system based on gestures according to some embodiments of the present disclosure. Method 100 may be implemented or executed at least in part by hardware, software, or firmware. In some embodiments, method 100 may be implemented or executed at least in part by a robot system (e.g., Figure 2 The robotic system 200 shown in the figure) is performed. The robotic system can be a surgical robot system (for example, Figure 2 The robot system 200 shown in FIG. 1 may also include a dedicated or general-purpose robot system for use in other fields (e.g., manufacturing, machinery, etc.). In some embodiments, the method 100 may be at least partially implemented by Figure 2 The method 100 is performed by the control device 240 of the robot system 200 shown. In some embodiments, the method 100 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 2 For example, the control device of the robot system (e.g., Figure 2 The control device 240 shown may include a processor configured to execute the method 100. In some embodiments, the instructions may be stored on a computer-readable medium.

[0032] Some embodiments of the present disclosure provide a robotic system. Figure 2 FIG. 2 shows a block diagram of a robot system 200 according to some embodiments of the present disclosure. Figure 2As shown, the robot system 200 may include a mobile station 210, at least one motion arm 220, an image acquisition device 230 and a control device 240. The at least one motion arm 220 may be arranged on the mobile station 210 to perform a work task (e.g., a surgical task). In some embodiments, the mobile station 210 may be a mobile platform with wheels installed at the bottom for wheeled movement, such as a steering wheel trolley, a differential trolley, an omnidirectional trolley, etc. In some embodiments, the distal end of at least one motion arm 220 may be a telecentric motion mechanism, which may determine the mechanical fixed point of the robot system 200, and use the mechanical fixed point as the remote center of motion (RCM) to allow the distal end of at least one motion arm 220 to rotate around the remote center of motion. In some embodiments, the image acquisition device 230 is used to acquire a hand image of at least one hand, so that the gesture information of at least one hand can be obtained based on the hand image set of at least one hand, and the movement of the mobile station 210 and / or at least one motion arm 220 can be controlled based on the gesture information. In some embodiments, the image acquisition device 230 may be a depth camera. The depth camera mainly includes sensors based on TOF (Time offlight, time of flight), structured light or multi-angle imaging. In some embodiments, the image acquisition device 230 can be set in the environment. For example, the image acquisition device 230 can be set on one side of the room or on the ceiling of the room, or it can also be set near a workbench (for example, an operating table). Alternatively, the image acquisition device 230 can also be set in the robot system 200. For example, the robot system 200 can also include an equipment trolley for carrying items related to the work task (for example, surgery-related items, such as energy generating devices, cold light sources, etc.), and the image acquisition device 230 can be set on the equipment trolley. In some embodiments, the image acquisition device 230 can be connected to the control device 240 via a data transmission method such as a cable or wireless, and transmit data in real time.

[0033] In some embodiments, the control device 240 may be connected to the mobile station 210 and / or the at least one motion arm 220, for example, by a cable connection or by a wireless connection. In some embodiments, the control device 240 is used to execute the motion control method (e.g., method 100) of the gesture-based robot system in some embodiments of the present disclosure to control the motion of the mobile station 210 and / or the at least one motion arm 220.

[0034] In some embodiments, the mobile station 210 and / or at least one motion arm 220 include at least one joint, each joint may include a joint motor for driving at least one joint, and under the control of the control device 240, the corresponding joint is driven to rotate, so that at least one motion arm 220 moves in space to form a desired configuration, so as to achieve the deployment and positioning of at least one motion arm 220. In the present disclosure, positioning refers to the adjustment of the configuration of at least one motion arm 220 of the robot system to achieve a positioning configuration that can perform work. In some embodiments, the positioning of at least one motion arm 220 may include the deployment, adjustment and positioning of at least one motion arm.

[0035] In some embodiments, the robot system 200 may be any suitable surgical robot system including a laparoscopic surgical robot (eg, Figure 3 The surgical robot system 300 shown). Figure 3 FIG. 2 is a schematic diagram showing the structure of a surgical robot system 300 according to some embodiments of the present disclosure. Figure 3 As shown, the surgical robot system may include a surgical trolley 310, a main control trolley 320, and an image acquisition device ( Figure 3 ) and a control device ( Figure 3 The surgical trolley 310 may include a mobile station 311 and at least one motion arm 312, and the at least one motion arm 312 may be movably disposed on the mobile station 311. In some embodiments, the at least one motion arm 312 may be a positioning arm of the surgical robot system 300, and the distal end of the at least one motion arm 312 may carry at least one surgical instrument ( Figure 3 not shown), such as clamps, scissors, endoscopes, etc.

[0036] In some embodiments, the main control vehicle 320 may include at least one main operator 321. The at least one main operator 321 is disposed on the main control vehicle 320 and is used to receive an operation of the at least one main operator 321 by an operator (eg, medical staff, etc.). Figure 3 As shown, in some embodiments, at least one main operator 321 may include a left main operator 321a for receiving operations of the left hand of a user (e.g., a medical staff, etc.) and a right main operator 321b for receiving operations of the right hand of the user. During surgery, the operator may control at least one surgical instrument carried by at least one motion arm 312 to perform a surgical operation by operating at least one main operator 321 in the main control trolley 320.

[0037] In some embodiments, the image acquisition device can be used to acquire an image of at least one hand of an operator (e.g., a medical staff, etc.) to obtain gesture information of the operator. In some embodiments, the control device can be set in the mobile station 311 or in any suitable position in the surgical robot system 300. In some embodiments, the control device can be communicated with the mobile station 311, at least one motion arm 312, and the image acquisition device, respectively. The control device is used to execute the motion control method of the gesture-based robot system in some embodiments of the present disclosure (e.g., method 100, method 700, or method 1100) to control the motion of the mobile station 311 or at least one motion arm 312.

[0038] Below, combined with the attached Figure 1 The specific implementation process of the control method 100 for a robot system provided by the present disclosure is specifically described.

[0039] See also Figure 1 , in step 101, a hand image set of at least one hand is obtained.

[0040] In some embodiments, Figure 2 As shown, the robot system 200 may include an image acquisition device 230, and the image acquisition device 230 may be used to acquire a hand image of at least one hand of an operator (e.g., a medical staff, etc.) to obtain a hand image set of at least one hand. In the present disclosure, the hand image set of at least one hand refers to a set of continuous multiple frames of hand images of at least one hand or multiple frames of hand images with fixed intervals acquired by the image acquisition device. For example, the hand image set of at least one hand may be a set of 1st, 2nd, 3rd, 4th, 5th, ... frames of hand images acquired by the image acquisition device, or the hand image set of at least one hand may be a set of 1st, 3rd, 5th, 7th, ... frames of hand images acquired by the image acquisition device.

[0041] Continue to view Figure 1 In step 103, gesture information of at least one hand is obtained based on the hand image set.

[0042] In some embodiments, gesture information of at least one hand may be obtained based on at least one frame of hand image in the hand image set. In some embodiments, the gesture information of at least one hand includes one or more of hand configuration, hand position, and hand posture.

[0043] In some embodiments, obtaining gesture information of at least one hand based on at least one frame of hand image in a hand image set may include: determining the three-dimensional coordinates of multiple hand key points of at least one hand in the hand image; and determining the gesture information of at least one hand based on the three-dimensional coordinates of the multiple hand key points, the gesture information including one or more of hand configuration, hand position, and hand posture.

[0044] Figure 4 A schematic diagram of a hand key point model according to some embodiments of the present disclosure is shown. In some embodiments, the hand key point model may adopt a model including 21 hand key points, and the distribution and sequence of the 21 key points in the hand region are as follows: Figure 4 As shown. In some embodiments, a plurality of hand key points in a hand image can be detected by a hand key point detection model to obtain the three-dimensional coordinates of the plurality of hand key points. The three-dimensional coordinates of each hand key point can be expressed as (x, y, z), where x represents the horizontal coordinate of the pixel of the hand key point in the image frame, y represents the vertical coordinate of the pixel of the hand key point in the image frame, and z represents the depth of key point No. 0 relative to the wrist. In some embodiments, the hand key point detection model can adopt a machine learning model, for example, a neural network model, etc.

[0045] In some embodiments, the hand configuration of at least one hand can be identified through a gesture recognition model based on the three-dimensional coordinates of multiple hand key points of at least one hand. In some embodiments, the gesture recognition model can adopt a machine learning model, such as a neural network model.

[0046] In some embodiments, the hand position includes the position of the hand coordinate system in the reference coordinate system. In some embodiments, the hand posture includes the posture of the hand coordinate system in the reference coordinate system. Figure 5 Schematic diagram showing a hand coordinate system according to some embodiments of the present disclosure. Figure 5 , for example Figure 5 For the hand configuration shown with the palm open and five fingers open, the hand coordinate system {h} can be represented by the wrist joint key points (e.g., Figure 4 0 key point shown in the figure) is the origin, the wrist joint key point and the middle finger root key point (for example, Figure 4 The line connecting the key point No. 9 shown in Axis, the direction from the wrist joint key point to the middle finger root key point is Axis positive direction; axis( Figure 5 ) is perpendicular to the plane where the palm is located. When the right hand is used as the control hand, the back of the hand direction is the positive direction. When the left hand is used as the control hand, the palm direction is the positive direction. The plane where the palm is located can be based on at least three hand key points (for example, the wrist joint key point, the index finger root key point and the middle finger root key point, for example, Figure 4 0, 5 and 9 as shown in the figure); Axis based on Axis and The axes are determined by the right-handed definition. In some embodiments, the reference coordinate system can be an image acquisition device (e.g., Figure 2The reference coordinate system may also be a coordinate system of the space where the robot system is located, such as a world coordinate system, for example, Figure 5 The world coordinate system {w} is shown in . Figure 5 As shown, the world coordinate system {w} can take any point in the space where the robot system is located as the origin. The axis is perpendicular to the horizontal plane (e.g., the ground) and points vertically upward. Axis perpendicular to Axis, in some embodiments, The axis can point directly in front of the operator, Axis based on Axis and The axis is determined by the right hand definition. In some embodiments, the position and / or posture of the hand coordinate system of at least one hand in the reference coordinate system can be calculated based on the three-dimensional coordinates of multiple hand key points of at least one hand, for example, Figure 5 The position and / or posture of the hand coordinate system {h} in the world coordinate system {w} is shown.

[0047] Those skilled in the art will appreciate that, under different gesture configurations, corresponding hand coordinate systems (for example, Figure 5 The hand coordinate system {h} shown, Fig. 9A The hand coordinate system {h1} shown, Fig. 9B The hand coordinate system {h2} shown, Fig.13A The hand coordinate system {h3} shown, Fig. 13B The hand coordinate system {h4} shown). The hand position (e.g., the position of the hand coordinate system in the reference coordinate system) and / or the hand posture (e.g., the posture of the hand coordinate system in the reference coordinate system) can be calculated based on the three-dimensional coordinates of the hand key points.

[0048] Continue to view Figure 1 In step 105, based on the gesture information, the mobile station or at least one motion arm is controlled to move.

[0049] In some embodiments, the gesture information may include a hand configuration and a hand posture, and a motion control mapping relationship between at least one hand and a mobile station and / or at least one motion arm may be established based on the hand configuration and the hand posture. In some embodiments, method 100 may include: in response to the hand configuration matching the mobile station mapping configuration in the gesture configuration set and the hand posture matching the mapping posture, establishing a motion control mapping relationship between at least one hand and the mobile station; and / or, in response to the hand configuration matching the motion arm mapping configuration in the gesture mapping configuration set and the hand posture matching the mapping posture, establishing a motion control mapping relationship between at least one hand and at least one motion arm. In some embodiments, the hand configuration matching the mobile station mapping configuration includes that the hand configuration is consistent or substantially consistent with the mobile station mapping configuration. The hand configuration matching the motion arm mapping configuration includes that the hand configuration is consistent or substantially consistent with the motion arm mapping configuration. The hand posture matching the mapping posture includes that the hand posture is consistent or substantially consistent with the mapping posture.

[0050] In some embodiments, the gesture configuration set includes at least one preset configuration for at least one motion control mode of the mobile station and / or at least one motion arm. In some embodiments, at least one motion control mode of the mobile station and / or at least one motion arm includes one or more of a forward control mode of the mobile station, a backward control mode of the mobile station, a position control mode of the motion arm, and a posture control mode of the motion arm. Correspondingly, at least one preset configuration in the gesture configuration set includes: one or more of a first preset configuration for forward control of the mobile station, a second preset configuration for backward control of the mobile station, a third preset configuration for position control of at least one motion arm, and a fourth preset configuration for posture control of at least one motion arm. In some embodiments, the mobile station mapping configuration includes a first preset configuration for forward control of the mobile station and / or a second preset configuration for backward control of the mobile station, and the motion arm mapping configuration includes a third preset configuration for position control of at least one motion arm and / or a fourth preset configuration for posture control of at least one motion arm.

[0051] In some embodiments, the mapping posture is a hand setting posture for establishing a motion control mapping relationship between at least one hand and at least one motion arm. In some embodiments, the mapping posture can be a hand coordinate system of at least one hand (for example, Figure 5 The coordinate system {h} shown in the figure is in the reference coordinate system (e.g., Figure 5 In some embodiments, the mapping posture may include that the hand coordinate system of at least one hand may have a hand setting posture in the reference coordinate system {w}), such as a zero posture that is consistent with the reference coordinate system. In some embodiments, the mapping posture may include that the hand setting posture of the hand coordinate system of at least one hand in the reference coordinate system may be 0, so that when the hand posture of at least one hand matches the mapping posture, the coordinate axis of the hand coordinate system is consistent or substantially consistent with the direction of the world coordinate system.

[0052] In some embodiments, method 100 may include: in response to the hand configuration matching the first preset configuration and the hand posture matching the mapping posture, establishing a motion control mapping relationship between at least one hand and the forward control mode of the mobile station; or in response to the hand configuration matching the second preset configuration and the hand posture matching the mapping posture, establishing a motion control mapping relationship between at least one hand and the backward control mode of the mobile station; or in response to the hand configuration matching the third preset configuration and the hand posture matching the mapping posture, establishing a motion control mapping relationship between at least one hand and the position control mode of the motion arm; or in response to the hand configuration matching the fourth preset configuration and the hand posture matching the mapping posture, establishing a motion control mapping relationship between at least one hand and the posture control mode of the motion arm.

[0053] Figure 6A-6D Schematic diagram showing various hand configurations according to some embodiments of the present disclosure. The first preset configuration for the forward control of the mobile station can be as follows Fig. 6A The hand configuration shown is that the index finger is extended and the other four fingers are bent and clenched. The second preset configuration for the mobile station back control can be as follows Figure 6B The third preset configuration for at least one motion arm position control may be as follows: Figure 6C The thumb is extended and the other four fingers are bent and clenched. The fourth preset configuration for at least one motion arm posture control can be as follows Fig.6D The hand configuration shown is that the thumb and index finger are spread out and the other three fingers are bent and clenched. Those skilled in the art can understand that the preset configurations involved in the present disclosure are not limited to the hand configurations listed above, but can also include other suitable hand configurations.

[0054] Those skilled in the art will appreciate that, when a motion control mapping relationship between at least one hand and a mobile station is established (for example, a motion mapping relationship of a forward control mode of the mobile station, or a motion mapping relationship of a backward control mode of the mobile station), in step 105, the motion of the mobile station can be controlled based on gesture information (for example, the forward movement of the mobile station, or the backward movement of the mobile station); when a motion control mapping relationship between at least one hand and at least one motion arm is established (for example, a motion control mapping relationship of a position control mode of the motion arm, or a motion control mapping relationship of a posture control mode of the motion arm), in step 105, the motion of at least one motion arm can be controlled based on gesture information (for example, position control of at least one motion arm, or posture control of at least one motion arm).

[0055] In some embodiments, the method 100 may further include: in response to a hand gesture matching the mapping configuration of the mobile station or the mapping configuration of the motion arm and matching the mapping gesture being maintained for a preset time (e.g., a first preset time, such as 5 seconds, 10 seconds, etc.), triggering motion control of the mobile station or at least one motion arm. In this way, false triggering of the mobile station or at least one motion arm can be avoided.

[0056] In some embodiments, the method 100 further includes: in response to the hand configuration not matching any preset configuration in the gesture configuration set, disconnecting the motion control mapping relationship between at least one hand and the mobile station or at least one motion arm to end the current control of the mobile station or at least one motion arm. Those skilled in the art will appreciate that when it is necessary to control the mobile station or at least one motion arm again, it is necessary to re-establish the motion control mapping relationship between at least one hand and the mobile station or at least one motion arm.

[0057] In some embodiments, the gesture information may include hand gestures, and motion control instructions for the mobile station and / or at least one motion arm may be generated based on the hand gestures, and the movement of the mobile station and / or at least one motion arm may be controlled based on the motion control instructions.

[0058] In some embodiments, Figure 3 As shown, the mobile station 311 may include at least one caster (e.g., Figure 3 Casters 3114a and 3114b are shown. Figure 3 As shown, the mobile station 311 may include a base 3113. In some embodiments, as Figure 3 As shown, at least one caster can be disposed at the bottom of the base 3113 of the mobile station 311 to enable the movement of the mobile station 311. The at least one caster can include at least one active caster. In some embodiments, the at least one active caster can be a front caster of the mobile station 311 (e.g., Figure 3 The caster 3114b shown in the figure and the caster arranged opposite to the caster 3114b), or the rear caster of the mobile station 311 (such as Figure 3 In some embodiments, the control device of the surgical robot system 300 may control the movement of at least one active caster included in the mobile station 311 based on the gesture information, thereby controlling the movement of the mobile station 311. In some embodiments, the control device of the surgical robot system 300 may be connected to the at least one active caster in communication with the at least one active caster to control the movement of the mobile station 311.

[0059] In some embodiments, Figure 3As shown, at least one motion arm 312 of the surgical robot system 300 may include at least one arm body (e.g., Figure 3 In some embodiments, the control device of the surgical robot system 300 may control the movement of at least one joint (e.g., joints 3122a to 3122c, etc.) based on the gesture information to achieve the control of the movement of at least one motion arm 312. In some embodiments, the control device of the surgical robot 300 may be connected to at least one joint of at least one motion arm 312 to control the movement of at least one motion arm 312.

[0060] Based on the method 100, the operator (e.g., medical staff, etc.) can control the movement of the mobile station 311 or at least one motion arm 312 through gestures. Before surgery, the operator can control the mobile station 311 to be in place (e.g., to place the mobile station 311 close to the operating table and in a position suitable for performing surgery) or control the at least one motion arm 312 to be in place (e.g., to place the remote center of motion RCM of at least one motion arm 312 at the center of the patient's opening, suitable for connecting the surgical instrument to the motion arm 312 and feeding it into the patient's body) through gesture control.

[0061] Figure 7 1 is a flowchart of a method 700 (hereinafter referred to as “method 700”) for controlling a motion of a robot system based on gestures according to some embodiments of the present disclosure. The method 700 may be implemented or executed at least in part by hardware, software, or firmware. In some embodiments, the method 700 may be implemented or executed at least in part by a robot system (e.g., Figure 2 The robotic system 200 shown in the figure) is performed. The robotic system can be a surgical robot system (for example, Figure 2 The robot system may also include a dedicated or general-purpose robot system for other fields (e.g., manufacturing, machinery, etc.). In some embodiments, the method 700 may be at least partially performed by Figure 2 The method 700 is performed by the control device 240 of the robot system 200 shown. In some embodiments, the method 700 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 2 For example, the control device of the robot system (e.g., Figure 2 The control device 240 shown may include a processor configured to execute the method 700. In some embodiments, these instructions may be stored on a computer-readable medium.

[0062] See also Figure 7 In step 701, based on the hand gesture, a target velocity and / or a target angular velocity of a control point of the mobile station is determined.

[0063] Figure 8 A simplified schematic diagram of a control point c1 of a mobile station 810 according to some embodiments of the present disclosure is shown. Figure 8 As shown, at least one active caster of the mobile station 810 may include a first caster 814b and a second caster 814c that are symmetrically arranged. Figure 8 As shown, at least one active caster of the mobile station 810 may be a pair of front casters of the mobile station 810. In some embodiments, the rear casters ( Figure 8 In some embodiments, the control point of the mobile station 810 may include the midpoint of the first caster 814b and the second caster 814c (e.g. Figure 8 Point c1) is shown.

[0064] In some embodiments, the target speed and / or target angular velocity of the control point of the mobile station may be the target speed and / or target angular velocity of the coordinate system of the control point of the mobile station relative to the reference coordinate system (e.g., the world coordinate system). Figure 8 The coordinate system {c1} is shown. Figure 8 As shown, the mobile station control point coordinate system {c1} may take the control point c1 of the mobile station 810 as the origin, and may also include a longitudinal coordinate axis (not shown in the figure) along the vertical direction and a first transverse coordinate axis (e.g., Figure 8 x-axis shown) and a second horizontal coordinate axis (e.g., Figure 8 In some embodiments, the second horizontal coordinate axis (e.g., Figure 8 The y-axis shown in FIG. 8 may take the moving direction of the mobile station 810 as the positive direction. Those skilled in the art will appreciate that the coordinate system used to determine the movement of the control point of the mobile station is not limited to the coordinate system shown in FIG. Figure 8 The mobile station control point coordinate system {c1} shown may be any suitable coordinate system.

[0065] The casters of the mobile station 810 can only move on the ground. Based on this, for example, when the four casters included in the mobile station 810 are all active casters and the four active casters are all universal wheels, the movement of the control point of the mobile station 810 may include movement in three degrees of freedom: along the Figure 8The linear motion of the x-axis and y-axis of the coordinate system {c1} shown in FIG. 1 , and the rotational motion around the longitudinal coordinate axis (along the vertical direction, not shown in the figure). Based on this, in step 701, it can be determined that the target motion of the control point of the mobile station can be expressed as t c1 =[v c1 T ω c1 T ] T Indicates that v c1 =[v c1x v c1y 0] T ,ω c1 =[00ω c1z ] T In some embodiments, the movement of the control point of the mobile station in three degrees of freedom can be controlled by changing the hand gestures.

[0066] In some embodiments, the two front casters included in the mobile station 810 can be active casters, and both active casters are directional casters. Based on this, the mobile station 810 can achieve movement in two degrees of freedom, including along the Figure 8 The target motion of the mobile station 810 can be expressed as t c1 =[v c1 T ω c1 T ] T Indicates that v c1 =[0v c1y 0] T ,ω c1 =[00ω c1z ] T In some embodiments, the movement of the control point of the mobile station in two degrees of freedom can be controlled by changing the hand gestures.

[0067] In some embodiments, the gesture information may include a hand posture. In step 701, the hand posture may be mapped to a target velocity and / or a target angular velocity of a control point of the mobile station.

[0068] In some embodiments, method 700 may include: determining a hand posture change based on an initial hand posture and a hand manipulation posture. In some embodiments, the initial hand posture may be a hand posture when establishing a motion mapping relationship between at least one hand and a mobile station, for example, the initial hand posture may be a hand posture that matches a mapping posture. In some embodiments, the hand manipulation posture may be a posture after the continuous movement of the hand stops. For example, after the motion control is triggered, the operator's hand starts to move, stops moving after moving to an appropriate posture, for example, remains stationary for a preset time (e.g., a second preset time, such as 5 seconds, 10 seconds, etc.), and the posture after stopping may be regarded as a hand manipulation posture.

[0069] In some embodiments, in response to a hand configuration matching the mapping configuration of the mobile station and a hand posture matching the mapping posture being maintained for a preset time (e.g., a first preset time), motion control of the mobile station is triggered. After the motion control of the mobile station is triggered, the movement of the control point of the mobile station in different degrees of freedom can be controlled by changing the hand posture. The hand control posture can be a hand posture that remains stationary for a preset time (e.g., a second preset time, such as 5 seconds, 10 seconds, etc.) after the motion control of the mobile station is triggered, so as to determine the hand posture change based on the hand control posture and the initial hand posture. In some embodiments, after the motion control of the mobile station is triggered, the hand posture in the hand image captured by the image acquisition device can be detected at a predetermined period. If the hand postures in multiple consecutive frames of hand images detected for the preset time (e.g., the second preset time) remain unchanged, the hand posture of at least one hand is used as the hand enabling posture. In some embodiments, the hand posture change includes the hand coordinate system (e.g., Figure 5 At least one coordinate axis (e.g., axis, axis, The rotation information of at least one coordinate axis may include rotation direction and / or rotation angle of at least one coordinate axis.

[0070] In some embodiments, method 700 may include: determining mobile station posture control information of the mobile station based on the hand posture change. In some embodiments, the hand posture change may be the rotation direction and / or rotation angle of at least one coordinate axis of the hand coordinate system in some of the above embodiments. The mobile station posture control information includes the mobile station posture control direction and / or the posture control angular velocity magnitude. In some embodiments, the mobile station posture control direction may be determined based on the rotation direction of at least one coordinate axis of the hand coordinate system, and the mobile station posture control angular velocity magnitude of the mobile station may be determined based on the rotation angle of at least one coordinate axis of the hand coordinate system. In some embodiments, the rotation direction of at least one coordinate axis of the hand coordinate system may be mapped to the mobile station posture control direction according to a predetermined mapping rule. For example, for example Figure 8 As shown in the mobile station 810, the posture control direction of the mobile station 810 includes a first direction (for example, Figure 8 counterclockwise rotation of the mobile station 810 about the z-axis of the coordinate system {c1}) and a second direction (e.g., Figure 8 The mobile station 810 shown rotates clockwise around the z-axis of the coordinate system {c1}), and the attitude control direction of the mobile station can be expressed as d c1R In some embodiments, the rotation angle of at least one coordinate axis of the hand coordinate system can be mapped to the angular velocity of the mobile station posture control at a predetermined ratio, for example, the angular velocity of the mobile station posture control ω c1adj It can be expressed as ω c1adj = k·α1, where k is a proportionality coefficient and α1 is a rotation angle of at least one coordinate axis. The mobile station attitude control angular velocity can be obtained based on the mobile station attitude control direction and the mobile station attitude control angular velocity. In some embodiments, method 700 may further include: determining a target angular velocity of a control point of the mobile station based on the mobile station attitude control direction and the mobile station attitude control angular velocity. In some embodiments, the target angular velocity ω of the control point of the mobile station can be obtained based on formula (1): c1 :

[0071] ω c1 =ω c1adj d c1R (1)

[0072] In some embodiments, the target speed of the control point of the mobile station may be preset. Figure 8 The two front casters included in the mobile station 810 shown can be active casters, and both active casters are directional casters. Based on this, the target speed of the control point of the mobile station 810 can be set to v c1 =[0v c1y 0] T , v c1y is the preset speed.

[0073] In some embodiments, the target speed of the control point of the mobile station may also be determined based on the gesture information of at least one hand. In some embodiments, method 700 may further include: determining the acceleration and deceleration control information of the mobile station based on the hand posture change. In some embodiments, the hand posture change may be the rotation direction and / or rotation angle of at least one coordinate axis of the hand coordinate system in some of the above embodiments. The mobile station acceleration and deceleration control information of the mobile station may include an acceleration and deceleration mode (e.g., an acceleration mode or a deceleration mode) and / or an acceleration magnitude. In some embodiments, the movement direction of the mobile station may include a forward direction and a backward direction, and the mobile station position control direction is determined by the hand configuration (e.g., such as Fig. 6A The hand configuration for the forward motion control of the mobile station shown or as Figure 6B In some embodiments, the acceleration and deceleration mode of the mobile station can be determined based on the rotation direction of at least one coordinate axis of the hand coordinate system, and the acceleration magnitude of the mobile station can be determined based on the rotation angle of at least one coordinate axis of the hand coordinate system. In some embodiments, the rotation direction of at least one coordinate axis of the hand coordinate system can be mapped to the acceleration and deceleration mode of the mobile station according to a predetermined mapping rule, for example, the clockwise rotation or counterclockwise rotation of at least one coordinate axis is mapped to the acceleration mode, and the rotation in another direction of at least one coordinate axis is mapped to the deceleration mode. In some embodiments, the rotation angle of at least one coordinate axis of the hand coordinate system can be mapped to the acceleration magnitude of the mobile station in a predetermined ratio. In some embodiments, the method 700 may further include: determining the target speed of the control point of the mobile station based on the acceleration and deceleration control information of the mobile station. In some embodiments, the target speed of the control point of the mobile station can be determined based on the acceleration and deceleration mode of the mobile station, the acceleration magnitude of the mobile station, and the current speed of the control point of the mobile station.

[0074] As an example, in some embodiments, the hand configuration for mobile station forward control may be as follows: Fig. 6A Hand configuration shown. Fig. 9A FIG. 2 is a schematic diagram showing a gesture operation for controlling the forward movement of a mobile station according to some embodiments of the present disclosure. Fig. 9A As shown, with the right hand as the control hand, the hand coordinate system {h1} can be represented by the wrist joint key points (e.g., Figure 4 0 key point shown in the figure) is the origin, the wrist joint key point and the middle finger root key point (for example, Figure 4 The line connecting the key point No. 9 shown in Axis, the direction from the wrist joint key point to the middle finger root key point is Axis positive direction; The axis is perpendicular to the plane where the palm is located, and the back of the hand is the positive direction, wherein the plane where the palm is located can be based on at least three hand key points (for example, the wrist joint key point, the index finger root key point and the middle finger root key point, for example, Figure 4 0, 5 and 9 as shown in the figure); Axis based on Axis and The axes are defined in a right-handed system.

[0075] In some embodiments, when establishing a motion mapping relationship between at least one hand and the forward control mode of the mobile station, the initial hand gesture of at least one hand may be as follows: Fig. 9A The initial posture of the hand coordinate system {h1} shown in FIG. The initial posture of the hand coordinate system {h1} can be the initial posture of the hand coordinate system {h1} The plane is parallel to the horizontal plane, The axis points directly in front of the operator. The posture when the axis is perpendicular to the horizontal plane and points vertically upward. Fig. 9A The coordinate axis direction of the hand coordinate system {h1} in the initial posture shown is consistent with the coordinate axis definition direction of the reference coordinate system (e.g., the world coordinate system). Therefore, in the following description, for simplicity, the hand posture change is described with reference to the initial posture of the hand coordinate system {h1}. In the actual control process, the hand posture change is the posture change of the hand coordinate system {h1} in the reference coordinate system (e.g., the world coordinate system).

[0076] In some embodiments, when the mobile station forward control is triggered, the mobile station moves in the forward direction (eg Figure 8 In some embodiments, when the mobile station moves forward, the hand coordinate system {h1} may be based on the first coordinate axis (for example, axis) to determine the direction of the mobile station's attitude control, for example, the hand coordinate system {h1} When the shaft rotates counterclockwise, the mobile station turns left (e.g. Figure 8 The vertical axis of the coordinate system {c1} shown in the figure is counterclockwise), and the vertical axis of the hand coordinate system {h1} is counterclockwise. When the axis rotates clockwise, the mobile station turns right (for example Figure 8 The clockwise direction of the longitudinal coordinate axis of the coordinate system {c1} shown in FIG. 1 ). The first coordinate axis of the hand coordinate system (eg, The rotation angle of the hand coordinate system {h1} is determined by determining the angular velocity of the mobile station attitude control of the mobile station. For example, the angular velocity of the mobile station attitude control is related to the hand coordinate system {h1} The rotation angle of the axis is directly proportional.

[0077] In some embodiments, during the moving process of the mobile station, the hand coordinate system {h1} may be further based on a second coordinate axis (eg, The rotation direction of the hand coordinate system {h1} determines the acceleration and deceleration mode of the mobile station. When the axis rotates counterclockwise, the mobile station is in acceleration mode, and the hand coordinate system {h1} When the axis rotates clockwise, the mobile station is in deceleration mode and can be based on the second coordinate axis of the hand coordinate system (for example, The rotation angle of the hand coordinate system {h1} is used to determine the acceleration of the mobile station. The rotation angle of the axis is directly proportional. Therefore, the target speed of the control point of the mobile station can be determined based on the acceleration and deceleration mode of the mobile station, the acceleration magnitude, and the current speed of the control point of the mobile station. Those skilled in the art can understand that in the forward control mode of the mobile station, the forward initial speed of the control point of the mobile station can be predetermined.

[0078] As an example, in some embodiments, the hand configuration for mobile station back control may be as follows: Figure 6B Hand configuration shown. Fig. 9B FIG. 2 is a schematic diagram showing a gesture operation of a mobile station back control according to some embodiments of the present disclosure. Fig. 9B As shown, with the right hand as the control hand, the hand coordinate system {h2} can be represented by the wrist joint key points (e.g., Figure 4 0 key point shown in the figure) is the origin, the wrist joint key point and the middle finger root key point (for example, Figure 4 The line connecting the key point No. 9 shown in Axis, the direction from the wrist joint key point to the middle finger root key point is Axis positive direction; The axis is perpendicular to the plane where the palm is located, and the back of the hand is the positive direction, wherein the plane where the palm is located can be based on at least three hand key points (for example, the wrist joint key point, the index finger root key point and the middle finger root key point, for example, Figure 4 0, 5 and 9 as shown in the figure); Axis based on Axis and The axes are defined in a right-handed system.

[0079] In some embodiments, when establishing a motion mapping relationship between at least one hand and the backward control mode of the mobile station, the initial hand gesture of at least one hand may be as follows: Fig. 9B The initial posture of the hand coordinate system {h2} shown in FIG. The initial posture of the hand coordinate system {h2} can be the initial posture of the hand coordinate system {h2} The plane is parallel to the horizontal plane, The axis points directly in front of the operator. The hand posture when the axis is perpendicular to the horizontal plane and pointing upward. Fig. 9B The coordinate axis direction of the hand coordinate system {h2} in the initial posture shown is consistent with the coordinate axis definition direction of the reference coordinate system (e.g., the world coordinate system). Therefore, in the following description, for simplicity, the hand posture change is described with reference to the initial posture of the hand coordinate system {h2}. In the actual control process, the hand posture change is the posture change of the hand coordinate system {h2} in the reference coordinate system (e.g., the world coordinate system).

[0080] In some embodiments, when the mobile station back-up control is triggered, the mobile station moves in a back-up direction (eg Figure 8 In some embodiments, similar to the forward motion control method of the mobile station, during the backward motion of the mobile station, the hand coordinate system {h2} may be controlled based on the first coordinate axis (e.g., axis) to determine the direction of the mobile station's attitude control, for example, the hand coordinate system {h2} When the shaft rotates counterclockwise, the mobile station turns left (e.g. Figure 8 The vertical axis of the coordinate system {c1} shown in the figure is counterclockwise), and the vertical axis of the hand coordinate system {h2} is counterclockwise. When the axis rotates clockwise, the mobile station turns right (for example Figure 8 The vertical axis of the coordinate system {c1} shown in FIG. 1 is a clockwise direction), and the vertical axis of the coordinate system {c1} shown in FIG. 1 may be based on at least one coordinate axis of the hand coordinate system (eg, The rotation angle of the hand coordinate system {h2} is determined by determining the angular velocity of the mobile station attitude control of the mobile station. For example, the angular velocity of the mobile station attitude control is related to the hand coordinate system {h2} The rotation angle of the axis is directly proportional.

[0081] In some embodiments, during the backward movement of the mobile station, the hand coordinate system {h2} may be further based on at least one coordinate axis (e.g., axis) to determine the acceleration and deceleration mode of the mobile station. For example, the hand coordinate system {h2} When the axis rotates counterclockwise, the mobile station is in acceleration mode, and the hand coordinate system {h2} When the axis rotates clockwise, the mobile station is in a deceleration mode and can be based on at least one coordinate axis of the hand coordinate system (for example, The rotation angle of the hand coordinate system {h2} is used to determine the acceleration of the mobile station. The rotation angle of the axis is directly proportional. Therefore, the target speed of the control point of the mobile station can be determined based on the acceleration and deceleration mode of the mobile station, the acceleration magnitude, and the current speed of the control point of the mobile station. Those skilled in the art can understand that in the backward control mode of the mobile station, the backward initial speed of the control point of the mobile station can be predetermined. The forward initial speed and the backward initial speed of the control point of the mobile station can be the same or different.

[0082] Continue to see Figure 7 In step 703, a motion control instruction for the mobile station is generated based on the target speed and / or target angular velocity of the control point of the mobile station. In step 703, the control device may generate a motion control instruction for controlling the motion of the mobile station. In some embodiments, the mobile station (e.g., the mobile station 810) may include at least one active caster (e.g., Figure 8 The mobile station 810 may include a first caster 814b and a second caster 814c as shown in the figure), and at least one caster motor for driving at least one active caster (not shown in the figure, and may be arranged at a position close to the active caster in the mobile station 810). In some embodiments, generating a motion control instruction for the mobile station may include: generating a drive control instruction for at least one caster motor of the mobile station.

[0083] In some embodiments, method 700 may further include: determining a target speed of at least one active caster based on a target speed and / or target angular speed of a control point of the mobile station; and determining a drive control instruction of at least one caster motor based on the target speed and radius of the at least one active caster.

[0084] In some embodiments, for example, when the two front casters of the mobile station 810 are active casters and directional casters, the relationship between the movement amount (for example, movement speed and / or movement angular velocity) of the control point of the mobile station and the movement amount (for example, movement speed and / or movement angular velocity) of the caster motor of the mobile station can be described by the following formula (2):

[0085]

[0086] Among them, v c1 is the target speed of the control point of the mobile station, ω c1 is the target angular velocity of the control point of the mobile station, v cr For the right active caster of the mobile station (e.g. Figure 8 The target speed of caster 814b) is shown, v cl For the left active caster of the mobile station (e.g. Figure 8 The target speed v of the caster 814c shown is shown, and D is the distance between the left active caster and the right active caster. According to formula (2), in some embodiments, the target speed v of the control point based on the mobile station c1and / or target angular velocity ω c1 , the target speed of at least one active caster can be determined using the following formula (3):

[0087]

[0088] Among them, v cr For the right active caster of the mobile station (e.g. Figure 8 The target speed of caster 814b) is shown, v cl For the left active caster of the mobile station (e.g. Figure 8 The target speed of caster 814c) is shown.

[0089] In some embodiments, after determining the target speed of at least one active caster, the motion control instruction of at least one active caster can be determined using the following formula (4) and formula (5), and then the drive control instruction of the caster motor for driving the active caster can be determined based on the motion control instruction of at least one active caster:

[0090]

[0091]

[0092] Among them, rpm cl is the motion control instruction of the left active caster of the mobile station (such as RPM (revolutions per minute) instruction), v cl is the target speed of the left active caster of the mobile station, rpm cr is the motion control instruction (for example, RPM instruction) of the right active caster of the mobile station, v cr is the target speed of the right active caster of the mobile station, r wheel is the radius of the active caster.

[0093] Continue to see Figure 7 In step 705, the mobile station is controlled to move based on the motion control instruction of the mobile station. For example, in step 705, the control device of the surgical robot system 300 may use the motion control instruction of the active caster (for example, the motion control instruction rpm of the left active caster obtained by formula (4) and formula (5)) to control the motion of the mobile station. cl and the motion control command rpm of the right active caster cr ) controls the active caster movement of the mobile station so that the control point of the mobile station moves at a target speed and / or a target angular velocity, thereby enabling the mobile station to move following the gestures of an operator (eg, a medical staff, etc.).

[0094] In some embodiments, the operator can control the movement of the mobile station through gestures while observing the mobile station, and determine the change of the gesture posture according to the movement of the mobile station to control the movement direction and / or movement speed of the mobile station. In some embodiments, the operator can observe the mobile station by standing near the mobile station, or observe the mobile station by viewing images captured by an image acquisition device in the surgical robot system (for example, set in an operating room), etc.

[0095] In some embodiments, the operator moves the mobile station into position through gesture control, for example, to a position near the operating table suitable for the movement arm to be connected to the sheath. After the operator moves the mobile station into position through gesture control, a movement control relationship between at least one hand and at least one movement arm can be established by changing the hand configuration (for example, a hand configuration matching the third preset configuration or the fourth preset configuration) to adjust the position of at least one movement arm through gestures.

[0096] The operator can judge whether the mobile station is moved to the position in a variety of ways. In some embodiments, the mobile station can include at least one positioning device for the operator to judge whether the mobile station is in position. Fig.10 A simplified schematic diagram of a positioning device in a mobile station according to some embodiments of the present disclosure is shown. Fig.10 As shown, the mobile station may include a column 1020, a beam 1030, and a positioning device disposed in the mobile station, for example, a first positioning device 1040 and a second positioning device 1050. In some embodiments, the positioning device may be a laser positioning device or other types of positioning devices. The positioning device may be disposed on the lower surface of the beam 1030, and the positioning device may be located on the center line of the beam 1030. In some embodiments, as Fig.10 As shown, the first positioning device 1040 projects a circular cursor downward, and the second positioning device 1050 projects a cross cursor downward.

[0097] In some embodiments, whether the mobile station is in place can be determined based on the distance between the projection of the first positioning device 1040 or the second positioning device 1050 on the patient on the operating bed 1010 and the patient's opening. For example, when the center of the cross-shaped cursor projected on the patient by the second positioning device 1050 is located at the patient's opening or relatively close to the opening, it can be determined that the mobile station is in place. In some embodiments, the operator can directly check next to the mobile station to see whether the center of the cross-shaped cursor is located at the patient's opening to determine whether the mobile station is in place. In some embodiments, the mobile station can also include an image acquisition device, which can be disposed below the crossbeam 1030, for example, disposed at Fig.10The position 10a between the first positioning device 1040 and the second positioning device 1050 is indicated in the position. The camera optical axis of the image acquisition device can be vertically downward so that the cursor projected by the first positioning device 1040 and / or the second positioning device 1050 falls within the field of view of the first image acquisition device. Based on this, the operator can view the cursor projected by the first positioning device 1040 and / or the second positioning device 1050 in the display of the surgical robot, and determine whether the mobile station is in place based on the cursor position.

[0098] Fig.11A 1 is a flowchart of a method 1100A (hereinafter referred to as “method 1100A”) for controlling motion of a robot system based on gestures according to some embodiments of the present disclosure. The method 1100A may be implemented or executed at least in part by hardware, software, or firmware. In some embodiments, the method 1100A may be implemented or executed at least in part by a robot system (e.g., Figure 2 The robotic system 200 shown in the figure) is performed. The robotic system can be a surgical robot system (for example, Figure 2 The robot system 1100A may be at least partially implemented by the following method: Figure 2 The control device 240 of the robot system 200 shown in FIG. 1100A is executed. In some embodiments, the method 1100A can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 2 For example, the control device of the robot system (e.g., Figure 2 The control device 240 shown may include a processor configured to execute the method 1100A. In some embodiments, these instructions may be stored on a computer-readable medium.

[0099] See also Fig.11A In step 1101A, based on the hand posture, a target velocity and / or a target angular velocity of a control point of at least one motion arm is determined.

[0100] Fig.12 A simplified schematic diagram showing the control points of the motion arm 1210 according to some embodiments of the present disclosure. Fig.12 As shown, at least one moving arm 1210 is disposed on the mobile station 1200. The mobile station 1200 may further include a base 1220, a column 1230 disposed on the base 1220, and a beam 1240 disposed on the column 1230. Fig.12 As shown, the mobile station 1200 may include a lifting joint J1, and the motion arm 1210 may include joints J2 to J6.

[0101] In some embodiments, Fig.12 As shown, at least one control point of the motion arm may include a sheath entry point, such as a remote center of motion (RCM), and the RCM may be Fig.12 The origin of the coordinate system {RCM} shown in FIG. Fig.12 As shown, the rotation axis of the joint J4, the rotation axis of the joint J5 and the rotation axis of the joint J6 intersect at the remote center of motion (RCM), as shown in FIG. Fig.12 The origin of the coordinate system {RCM} is shown. Those skilled in the art will appreciate that during surgery, the remote motion center may be located at an opening of the patient (e.g., an abdominal inlet, etc.), and the motion arm carries the surgical instrument to move the surgical instrument around the RCM to adjust the position or perform a surgical operation.

[0102] In some embodiments, the target speed and / or target angular velocity of the control point of at least one moving arm determined based on step 1101 may be the target speed and / or target angular velocity of the control point coordinate system of the control point of at least one moving arm relative to a reference coordinate system (e.g., a world coordinate system). In some embodiments, the control point coordinate system may be as follows: Fig.12 The coordinate system {RCM} shown in FIG. 1 may be based on the remote motion center of at least one motion arm as its origin, and the coordinate system {RCM} may also include a longitudinal coordinate axis extending from the proximal end to the distal end (e.g. Fig.12 The coordinate system {RCM} shown has a coordinate axis y) and a first transverse coordinate axis and a second transverse coordinate axis (eg Fig.12 The coordinate system {RCM} shown in the figure has a coordinate axis x and a coordinate axis z). Those skilled in the art will appreciate that the control point coordinate system is not limited to the coordinate system {RCM} described above, but may be any suitable coordinate system, for example, a coordinate system having a remote motion center of at least one motion arm as its origin and including a vertical coordinate axis along a vertical direction and two horizontal coordinate axes perpendicular to the vertical coordinate axis.

[0103] In some embodiments, at least one of the control points of the motion arm may include a sheath fixing point. For example, the sheath is fixedly connected to the end of the motion arm through a sheath fixing clamp to form a sheath fixing point. In this way, the sheath fixing point may be the center of the end of the sheath fixing clamp. For example, the sheath fixing point may be as follows: Fig.12 The origin of the {TRB} coordinate system shown.

[0104] In some embodiments, at least one motion arm motion control mode may include a motion arm position control mode and a motion arm posture control mode. The motion arm position control mode may be reflected as a control point of at least one motion arm (e.g., Fig.12The RCM of the motion arm 1210 shown in the figure can adjust the position of the motion arm in three degrees of freedom, and the posture control mode of the motion arm is reflected in the control point of at least one motion arm (for example, Fig.12 The RCM of the motion arm 1210 shown in FIG. 12A is a three-degree-of-freedom posture adjustment. Thus, at least one control point of the motion arm (e.g., Fig.12 The RCM of the motion arm 210 shown can move in six degrees of freedom in space, and the target motion of at least one control point of the motion arm can be expressed as t c2 =[v c2 T ω c2 T ] T Indicates that v c2 =[v c2x v c2y v c2z ] T ,ω c2 =[ω c2x ω c2y ω c2z ] T In some embodiments, the movement of the control point of at least one motion arm in six degrees of freedom can be controlled by changing the gesture posture.

[0105] In some embodiments, the gesture information may include a hand posture, and in step 1101A, the hand posture may be mapped to a target velocity and / or a target angular velocity of a control point of at least one motion arm.

[0106] In some embodiments, method 1100A may include: determining a hand posture change based on an initial hand posture and a hand manipulation posture. In some embodiments, the initial hand posture may be a hand posture when a motion mapping relationship between at least one hand and at least one motion arm is established, for example, the initial hand posture may be a hand posture that matches the mapping posture. In some embodiments, the hand manipulation posture may be a posture after the continuous movement of the hand stops. For example, after the motion control is triggered, the operator's hand starts to move, and stops moving after moving to an appropriate posture, for example, keeping still for a preset time (for example, a second preset time, such as 5 seconds, 10 seconds, etc.), and the posture after stopping may be regarded as a hand manipulation posture.

[0107] In some embodiments, in response to a hand configuration matching the mapping configuration of the motion arm and a hand posture matching the mapping posture being maintained for a preset time (for example, a first preset time, such as 5 seconds, 10 seconds, etc.), motion control of at least one motion arm is triggered. After the motion control of at least one motion arm is triggered, the movement of the control point of at least one motion arm in different degrees of freedom can be controlled by changing the hand posture. The hand control posture can be a hand posture that remains stationary for a preset time (for example, a second preset time, such as 5 seconds, 10 seconds, etc.) after the motion control of at least one motion arm is triggered, so as to determine the hand posture change based on the hand control posture and the initial hand posture. In some embodiments, after the motion control of at least one motion arm is triggered, the hand posture in the hand image captured by the image acquisition device can be detected at a predetermined period. If the hand postures in multiple consecutive frames of hand images detected for a preset time (for example, the second preset time) remain unchanged, the hand posture of at least one hand is used as a hand enabling posture. In some embodiments, the hand posture change includes a hand coordinate system (for example, Figure 5 At least one coordinate axis (e.g., Figure 5 shown axis, axis, The rotation information of at least one coordinate axis may include rotation direction and / or rotation angle of at least one coordinate axis.

[0108] In some embodiments, the method 1100A may include: determining the motion arm position control information of at least one motion arm based on the hand posture change. In some embodiments, the hand posture change may be a rotation direction and / or rotation angle of at least one coordinate axis of the hand coordinate system. The motion arm position control information of at least one motion arm includes the motion arm position control direction (for example, including Fig.12 In some embodiments, the rotation direction of at least one coordinate axis of the hand coordinate system can be mapped to the motion arm position control direction of at least one motion arm according to a predetermined mapping rule. For example, for example Fig.12 The at least one moving arm 1210 shown in the figure, the moving arm position control direction of the at least one moving arm 1210 includes the positive direction and negative direction of the x, y, and z axes of the coordinate system {RCM}, and the moving arm position control direction of the at least one moving arm 1210 can be expressed as d c2p. In some embodiments, the rotation information of at least one coordinate axis also includes a rotation angle. In response to the rotation angle of at least one coordinate axis being not less than a preset angle, the motion arm position adjustment direction of at least one motion arm is determined based on the rotation direction of at least one coordinate axis. In some embodiments, the hand coordinate system includes three coordinate axes, and the motion arm position control directions of at least one motion arm in six directions on three degrees of freedom can be determined based on the rotation information of the three coordinate axes respectively. In some embodiments, when the rotation angle of at least one coordinate axis of the hand coordinate system is less than a preset angle, the motion arm position control of at least one motion arm is stopped.

[0109] In some embodiments, the method 1100A may further include: determining a target speed of a control point of at least one moving arm based on the moving arm position control direction and the speed magnitude of the moving arm position control. In some embodiments, the moving arm position control direction may be denoted as, and the speed magnitude of the moving arm position control may be the control reference point of at least one moving arm based on the moving arm position control direction d c2p The speed of movement and the speed of the position control of the moving arm can be preset and can be expressed as. The target speed of the control point of the moving arm can be obtained based on formula (6):

[0110] v c2 =v c2adj d c2p (6)

[0111] In some embodiments, the method 1100A may further include: determining the motion arm posture control information of at least one motion arm based on the hand posture change. In some embodiments, the hand posture change may be the rotation direction and / or rotation angle of at least one coordinate axis of the hand coordinate system in some of the above embodiments. The motion arm posture control information of at least one motion arm includes the motion arm posture control direction (for example, including the rotation direction of the hand coordinate system). Fig.12 In some embodiments, the rotation direction of at least one coordinate axis of the hand coordinate system can be mapped to the motion arm posture control direction of at least one motion arm according to a predetermined mapping rule. For example, for example Fig.12 The at least one moving arm 1210 shown in the figure has a moving arm posture control direction including a clockwise direction and a counterclockwise direction around the x, y, and z axes of the coordinate system {RCM}. The moving arm posture control direction of the at least one moving arm 1210 can be expressed as d c2R. In some embodiments, the rotation information of at least one coordinate axis also includes a rotation angle. In response to the rotation angle of at least one coordinate axis being not less than a preset angle, the motion arm posture adjustment direction of at least one motion arm is determined based on the rotation direction of at least one coordinate axis. In some embodiments, the hand coordinate system includes three coordinate axes, and the motion arm posture control directions of at least one motion arm in six directions on three degrees of freedom can be determined based on the rotation information of the three coordinate axes respectively. In some embodiments, when the rotation angle of at least one coordinate axis of the hand coordinate system is less than a preset angle, the motion arm posture control of at least one motion arm is stopped.

[0112] In some embodiments, the method 1100A may further include: determining a target angular velocity of a control point of at least one moving arm based on the moving arm posture control direction and the angular velocity magnitude of the moving arm posture control. In some embodiments, the moving arm posture control direction may be denoted as, and the angular velocity magnitude of the moving arm posture control may be the control reference point of at least one moving arm based on the moving arm posture control direction d c2R The angular velocity of the rotation, the angular velocity of the motion arm posture control can be preset, which can be expressed as, and the target angular velocity of the control point of the motion arm can be obtained based on formula (7):

[0113] ω c2 =ω c2adj d c2R (7)

[0114] Those skilled in the art will appreciate that the target angular velocity and / or target velocity of the control points (the control points of the mobile station and / or the control points of the motion arms) involved in the present disclosure should be understood as the target angular velocity and / or target velocity of the control point coordinate system with the control point as the origin.

[0115] As an example, in some embodiments, the hand configuration for the motion arm position control of at least one motion arm may be as follows: Figure 6C Hand configuration shown. Fig.13A A schematic diagram of gesture operation for controlling the position of at least one moving arm of some embodiments of the present disclosure is shown. Fig.13A As shown, with the right hand as the control hand, the hand coordinate system {h3} can be represented by the wrist joint key points (e.g., Figure 4 0 key point shown in the figure) is the origin, the wrist joint key point and the middle finger root key point (for example, Figure 4 The line connecting the key point No. 9 shown in Axis, the direction from the wrist joint key point to the middle finger root key point is Axis positive direction; The axis is perpendicular to the plane where the palm is located, and the back of the hand is the positive direction, wherein the plane where the palm is located can be based on at least three hand key points (for example, the wrist joint key point, the index finger root key point and the middle finger root key point, for example, Figure 4 0, 5 and 9 as shown in the figure); Axis based on Axis and The axes are defined in a right-handed system.

[0116] In some embodiments, when establishing a motion mapping relationship between at least one hand and a motion arm position control mode of at least one motion arm, the initial hand posture of at least one hand may be as follows: Fig.13A The initial posture of the hand coordinate system {h3} shown in FIG. The initial posture of the hand coordinate system {h3} can be the initial posture of the hand coordinate system {h3} The plane is parallel to the horizontal plane, The axis points directly in front of the operator. The posture when the axis is perpendicular to the horizontal plane and points vertically upward. Fig.13A The coordinate axis direction of the hand coordinate system {h3} in the initial posture shown is consistent with the coordinate axis definition direction of the reference coordinate system (e.g., the world coordinate system). Therefore, in the following description, for simplicity, the hand posture change is described with reference to the initial posture of the hand coordinate system {h3}. In the actual control process, the hand posture change is the posture change of the hand coordinate system {h3} in the reference coordinate system (e.g., the world coordinate system).

[0117] In some embodiments, the motion arm position control direction of at least one motion arm can be determined based on the rotation information of the three coordinate axes of the hand coordinate system {h3}, for example, The axis rotates counterclockwise and the rotation angle is greater than the preset angle, and the motion arm position control direction of at least one motion arm corresponds to the negative direction of the z-axis of the coordinate system {RCM}, so that at least one motion arm can move in the negative direction of the z-axis of the coordinate system {RCM}; the hand coordinate system {h3} The axis rotates clockwise and the rotation angle is greater than the preset angle. The position control direction of at least one moving arm corresponds to the positive direction of the z-axis of the coordinate system {RCM}, so that at least one moving arm can move in the positive direction of the z-axis of the coordinate system {RCM}; the hand coordinate system {h3} The axis rotates counterclockwise and the rotation angle is greater than the preset angle, and the motion arm position control direction of at least one motion arm corresponds to the negative direction of the x-axis of the coordinate system {RCM}, so that at least one motion arm can move in the negative direction of the x-axis of the coordinate system {RCM}; the hand coordinate system {h3} The axis rotates clockwise and the rotation angle is greater than the preset angle. The motion arm position control direction of at least one motion arm corresponds to the positive direction of the x-axis of the coordinate system {RCM}, so that at least one motion arm can move in the positive direction of the x-axis of the coordinate system {RCM}; the hand coordinate system {h3} When the axis rotates counterclockwise, the position control direction of at least one moving arm corresponds to the negative direction of the y-axis of the coordinate system {RCM}, so that at least one moving arm can move in the negative direction of the y-axis of the coordinate system {RCM}; The axis rotates clockwise and the rotation angle is greater than the preset angle, and the motion arm position control direction of at least one motion arm corresponds to the positive direction of the y-axis of the coordinate system {RCM}, so that at least one motion arm can move in the positive direction of the y-axis of the coordinate system {RCM}.

[0118] As an example, in some embodiments, the hand configuration for the motion arm posture control of at least one motion arm can be as follows: Fig.6D Hand configuration shown. Fig. 13B A schematic diagram of gesture operation for controlling the posture of at least one moving arm of some embodiments of the present disclosure is shown. Fig. 13B As shown, with the right hand as the control hand, the hand coordinate system {h4} can be represented by the wrist joint key points (e.g., Figure 4 0 key point shown in the figure) is the origin, the wrist joint key point and the middle finger root key point (for example, Figure 4 The line connecting the key point No. 9 shown in Axis, the direction from the wrist joint key point to the middle finger root key point is Axis positive direction; The axis is perpendicular to the plane where the palm is located, and the back of the hand is the positive direction, wherein the plane where the palm is located can be based on at least three hand key points (for example, the wrist joint key point, the index finger root key point and the middle finger root key point, for example, Figure 4 0, 5 and 9 as shown in the figure); Axis based on Axis and The axes are defined in a right-handed system.

[0119] In some embodiments, when establishing a motion mapping relationship between at least one hand and a motion arm gesture control mode of at least one motion arm, the initial hand gesture of at least one hand may be as follows: Fig. 13B The initial posture of the hand coordinate system {h4} shown in FIG. The initial posture of the hand coordinate system {h4} can be the initial posture of the hand coordinate system {h4} The plane is parallel to the horizontal plane, The axis points directly in front of the operator. The posture when the axis is perpendicular to the horizontal plane and points vertically upward. Fig. 13B The coordinate axis direction of the initial posture of the hand coordinate system {h4} shown is consistent with the coordinate axis definition direction of the reference coordinate system (e.g., the world coordinate system). Therefore, in the following description, for simplicity, the hand posture change is described with reference to the initial posture of the hand coordinate system {h4}. In the actual control process, the hand posture change is the posture change of the hand coordinate system {h4} in the reference coordinate system (e.g., the world coordinate system).

[0120] In some embodiments, the motion arm posture control direction of at least one motion arm can be determined based on the rotation information of the three coordinate axes of the hand coordinate system {h4}, for example, The axis rotates counterclockwise and the rotation angle is greater than the preset angle. The motion arm posture control direction of at least one motion arm corresponds to the counterclockwise direction of the x-axis of the coordinate system {RCM}, which enables at least one motion arm to rotate counterclockwise around the x-axis of the coordinate system {RCM} (downward movement); the hand coordinate system {h4} The axis rotates clockwise and the rotation angle is greater than the preset angle. The motion arm posture control direction of at least one motion arm corresponds to the clockwise direction of the x-axis of the coordinate system {RCM}, which enables at least one motion arm to rotate clockwise (upward movement) around the x-axis of the coordinate system {RCM}; the hand coordinate system {h4} The axis rotates counterclockwise and the rotation angle is greater than the preset angle. The motion arm posture control direction of at least one motion arm corresponds to the counterclockwise direction of the y-axis of the coordinate system {RCM}, which enables at least one motion arm to rotate counterclockwise (roll counterclockwise) around the y-axis of the coordinate system {RCM}; the hand coordinate system {h4} The axis rotates clockwise and the rotation angle is greater than the preset angle. The motion arm posture control direction of at least one motion arm corresponds to the clockwise direction of the y-axis of the coordinate system {RCM}, which enables at least one motion arm to rotate clockwise (clockwise roll) around the y-axis of the coordinate system {RCM}; the hand coordinate system {h4} When the axis rotates counterclockwise, the motion arm posture control direction of at least one motion arm corresponds to the counterclockwise direction of the z-axis of the coordinate system {RCM}, which enables at least one motion arm to rotate counterclockwise (counterclockwise deflection) around the z-axis of the coordinate system {RCM}; The axis rotates clockwise and the rotation angle is greater than the preset angle. The motion arm posture control direction of at least one motion arm corresponds to the clockwise direction of the z-axis of the coordinate system {RCM}, which can enable at least one motion arm to rotate clockwise (clockwise deflection) around the z-axis of the coordinate system {RCM}.

[0121] Continue to see Fig.11AIn step 1103A, a motion control instruction for at least one motion arm is generated based on a target velocity and / or a target angular velocity of a control point of at least one motion arm.

[0122] In some embodiments, Fig.12 As shown, the mobile station 1200 may include a moving arm 1210, a base 1220, a column 1230, and a beam 1240. The mobile station 1200 and the moving arm 1210 may include at least one joint, such as Fig.12 The joints J1 to J6 are shown. Fig.12 The motion arm 1210 shown in the figure, generating the motion control instruction of the motion arm may include: determining at least one joint (for example, Fig.12 In some embodiments, each joint motor (not shown) in at least one joint motor can be arranged at the joint it drives.

[0123] In some embodiments, Fig.12 As shown, the motion arm 1210 may include at least one arm body, for example, Fig.12 The first horizontal arm B1, the second horizontal arm B2, the first arc arm B3, the second arc arm B4 and the third arc arm B5 are shown. Fig.12 As shown, the proximal ends and distal ends of the multiple arms are respectively connected in rotation in sequence to form a joint, and the rotation axes of the first arc arm B3, the second arc arm B4 and the third arc arm B5 intersect at the remote motion center. Fig.12As shown, at least one joint included in the mobile station 1200 and the motion arm 1210 may include: a lifting joint J1 of the mobile station 1200, a first cross arm rotation joint J2, a second cross arm rotation joint J3, a first arc arm rotation joint J4, a second arc arm rotation joint J5 and a third arc arm rotation joint J6 of the motion arm 1210. Among them, the lifting joint J1 is arranged in the main column 1230 and connected to the main cross beam 1240, and is used to make the main cross beam 1240 realize lifting and lowering, thereby making the control point of the motion arm 1210 realize the change of lifting position; the first cross arm rotation joint J2 is connected between the main cross beam 1240 and the first cross arm B1; the second cross arm rotation joint J3 is connected between the first cross arm B1 and the second cross arm B2; the first arc arm rotation joint J4 is connected between the second cross arm B2 and the first arc arm B3; the second arc arm rotation joint J5 is connected between the first arc arm B3 and the second arc arm B4; the third arc arm rotation joint J6 is connected between the second arc arm B4 and the third arc arm B5. For the motion arm 1210, when the remote motion center RCM remains stationary, the first arc arm B3, the second arc arm B4 and the third arc arm B5 can move arbitrarily in the motion space, for example, they can be converted between fully folded, fully unfolded and intermediate positions.

[0124] In some embodiments, the method 1100 may further include: determining the joint values ​​and joint speeds of the lifting joint J1 of the mobile station 1200, the first cross-arm rotation joint J2, and the second cross-arm rotation joint J3 of the moving arm 910 based on the target speed of the control point of the moving arm 1210; and determining the driving control instructions for the joint motors for driving the lifting joint J1 of the mobile station, the first cross-arm rotation joint J2, and the second cross-arm rotation joint J3 of the moving arm based on the joint values ​​and joint speeds of the lifting joint J1 of the mobile station, the first cross-arm rotation joint J2, and the second cross-arm rotation joint J3 of the moving arm. Based on this, the control device of the surgical robot system can drive the lifting joint J1 of the mobile station 1200, the first cross-arm rotation joint J2, and the second cross-arm rotation joint J3 of the moving arm 910 to move based on the target speed of the control point of the moving arm 1210, thereby adjusting the position of the control point of the moving arm 1210.

[0125] Fig.14 FIG. 1 is a schematic diagram showing a structure of a mobile station 1400 according to some other embodiments of the present disclosure. Fig.14 As shown, the mobile station 1400 may include a motion arm 1410. The motion arm 1410 may include at least one joint, for example, Fig.14 The joints 14111-14161 are shown. Fig.14The motion arm 1410 shown, generating the motion control instruction of the motion arm may also include: determining the joint value and joint speed of at least one joint (e.g., joints 14111 to 14161) based on the target speed and / or target angular speed of the control point of the motion arm 1410; and determining the drive control instruction of at least one joint motor based on the joint value and joint speed of at least one joint. In some embodiments, each joint motor (not shown in the figure) in at least one joint motor may be arranged at the joint driven by it. The control device of the surgical robot system may drive at least one joint motor based on the drive control instruction of at least one joint motor, so that at least one joint motor drives the corresponding joint to realize the determined joint value and joint speed, based on which, the control point of the motion arm can reach the target speed and / or target angular speed, thereby realizing the control of the motion of the motion arm.

[0126] The target velocity and target angular velocity of at least one control point of the moving arm are expressed as t c2 =[v c2 T ω c2 T ] T , using the differential kinematics model, we can get formula (8) as follows:

[0127]

[0128] in, is the Jacobian matrix, J vi represents the partial derivative of the control point velocity vector with respect to the velocity of the i-th joint, J ωi represents the partial derivative of the control point angular velocity vector with respect to the velocity of the i-th joint, represents the joint velocity of the i-th joint. The i-th joint can include the 1st to 6th joints, which can be Fig.14 The joints 14111 to 14161 are shown. For example, J v1 Represents the target velocity vector v of the control point c2 right Fig.14 The partial derivative of the joint velocity of joint 14111, J ω1 represents the target angular velocity vector ω of the control point c2 right Fig.14 The partial derivative of the joint velocity of joint 14111, J v2 Represents the target velocity vector v of the control point c2 right Fig.14 The partial derivative of the joint velocity of joint 14121, J ω2 represents the target angular velocity vector ω of the control point c2 right Fig.14 The partial derivative of the joint velocity of joint 14121 is shown, express Fig.14 The joint velocity of joint 14121 is shown, express Fig.14 The joint velocity of joint 14121 is shown, and the same applies to other joints.

[0129] After determining the target velocity and target angular velocity of the control point of the motion arm, the joint value and joint velocity of at least one joint can be calculated by Jacobian pseudo-inverse. In some embodiments, the joint value and joint velocity of at least one joint can be determined based on the target velocity and / or target angular velocity of the control point of the motion arm by the following formula (9) and formula (10):

[0130]

[0131]

[0132] Among them, q is the joint value vector of at least one joint, is the joint velocity vector of at least one joint, J + is the pseudo-inverse matrix of the Jacobian matrix J.

[0133] In some embodiments, Fig.14 As shown, the motion arm 1410 may include at least one arm body, such as Fig.14 The first horizontal arm 1411, the second horizontal arm 1412, the vertical arm 1413, the first arc arm 1414, the second arc arm 1415 and the third arc arm 1416 are shown. In some embodiments, Fig.14As shown, the at least one joint included in the motion arm 1410 may include: a first cross arm rotation joint 14111, a second cross arm rotation joint 14121, a lifting joint 14131, a first arc arm rotation joint 14141, a second arc arm rotation joint 14151 and a third arc arm rotation joint 14161. Among them, the first cross arm rotation joint 14111 is used to connect the first cross arm 1411 and the main cross beam 1420 of the mobile station 1400; the second cross arm rotation joint 14121 is used to connect the second cross arm 1412 and the first cross arm 1411; the lifting joint 14131 is set on the vertical arm 1413, and is used to realize the lifting and lowering of the control point of the motion arm; the first arc arm rotation joint 14141 is used to connect the vertical arm 1413 and the first arc arm 1414; the second arc arm rotation joint 14151 is used to connect the first arc arm 1414 and the second arc arm 1415; the third arc arm rotation joint 14161 is used to connect the second arc arm 1415 and the third arc arm 1416. Those skilled in the art can understand that the robot arm to which the motion control method of the robot system provided by the present disclosure is applicable is not limited to the above-mentioned robot arm 1210 or 1410, and the structure of the applicable robot arm is not limited to the structure of the robot arm 1210 or 1410, and can also be a robot arm of any suitable structure.

[0134] In some embodiments, the method 1100 may further include: determining the joint values ​​and joint speeds of the first cross arm rotation joint 14111, the second cross arm rotation joint 14121, and the lifting joint 14131 of the moving arm 1410 based on the target speed of the control point of the moving arm 1410; and determining the driving control instructions for driving the joint motors of the first cross arm rotation joint 14111, the second cross arm rotation joint 14121, and the lifting joint 14131 of the moving arm 1401 based on the joint values ​​and joint speeds of the first cross arm rotation joint 14111, the second cross arm rotation joint 14121, and the lifting joint 14131 of the moving arm 1401. Based on this, the control device of the surgical robot system can drive the first cross arm rotation joint 14111, the second cross arm rotation joint 14121, and the lifting joint 14131 of the moving arm 1410 to move based on the target speed of the control point of the moving arm 1410, thereby adjusting the position of the control point of the moving arm 1410.

[0135] In some embodiments, method 1100 may also include: determining the joint values ​​and joint velocities of the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint (for example, joints 14141-14161 of the motion arm 1410 or joints J4-J6 of the motion arm 1210) of the motion arm based on the target angular velocity of the control point of the motion arm (for example, the motion arm 1210 or the motion arm 1410); and determining the drive control instructions for the joint motors of the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint of the motion arm based on the joint values ​​and joint velocities of the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint of the motion arm. Based on this, the control device of the surgical robot system can drive the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint (for example, joints 14141-14161 of the moving arm 1410 or joints J4-J6 of the moving arm 1210) of the moving arm based on the target angular velocity of the control point of the moving arm (for example, the moving arm 1210 or the moving arm 1410), thereby adjusting the posture of the control point of the moving arm.

[0136] In some embodiments, for example, when the control point of the motion arm is a fixed point of the sheath tube, when determining the target angular velocity ω of the control point of the motion arm c2 In the case of c2 Determine the drive control instructions of the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint (for example, the joints 14141-14161 of the motion arm 1410 or the joints J4-J6 of the motion arm 1210); and based on the target speed v c2 =0 determines the drive control instructions of the first three joints of the motion arm (for example, joints 14111-14131 of the motion arm 1410 or joints J1-J3 of the motion arm 1210). Based on this, the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint of the motion arm move to achieve the target angular velocity ω of the control point of the motion arm. c2 At the same time, the front three joints of the motion arm can move in coordination so that the position of the control point of the motion arm does not change. In some embodiments, when determining the target speed v of the control point of the motion arm c2 In the case of c2 Determine the drive control instructions for the first three joints of the motion arm; and based on the target angular velocity ω c2 = 0 determines the drive control instructions of the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint of the motion arm. Thus, the control point of the motion arm is moved at the target speed v c2 The movement occurs without changing the pose of the motion arm (eg, the control points of the motion arm).

[0137] Continuing to refer to Fig. 11, in step 1105A, based on the motion control instruction of the at least one motion arm, the at least one motion arm is controlled to move. For example, in step 1105A, the control device of the robot system can control the at least one joint motor with the drive control instruction of the at least one joint motor, so that the at least one joint moves at a target speed and / or a target angular velocity, thereby achieving the movement of the at least one motion arm following the gesture of the operator.

[0138] In some embodiments, the gesture information may include a hand position. In some embodiments, in a motion arm position control mode of at least one motion arm, a motion control instruction of at least one motion arm may be generated based on the hand position, and the motion of at least one motion arm may be controlled based on the motion control instruction.

[0139] Fig. 11B 1 is a flowchart of a motion control method 1100B (hereinafter also referred to as “method 1100B”) for a gesture-based robotic system according to some embodiments of the present disclosure. The method 1100B may be implemented or executed at least in part by hardware, software, or firmware. In some embodiments, the method 1100B may be implemented or executed at least in part by a robotic system (e.g., Figure 2 The robotic system 200 shown in the figure) is performed. The robotic system can be a surgical robot system (for example, Figure 2 The robot system may also include a dedicated or general-purpose robot system for other fields (e.g., manufacturing, machinery, etc.). In some embodiments, the method 1100B may be at least partially performed by the following method: Figure 2 The control device 240 of the robot system 200 shown in FIG. 1100B is executed. In some embodiments, the method 1100B can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 2 For example, the control device of the robot system (e.g., Figure 2 The control device 240 shown may include a processor configured to execute the method 1100B. In some embodiments, these instructions may be stored on a computer-readable medium.

[0140] See also Fig. 11B In step 1101B, based on the hand position, a target position of a control point of at least one motion arm is determined.

[0141] In some embodiments, the control point of at least one motion arm may include the sheath entry point in some of the above embodiments, such as the remote center of motion (RCM).

[0142] In some embodiments, the target position of the control point of at least one moving arm determined based on step 1101B may be the target position of the control point coordinate system of the control point of at least one moving arm relative to the reference coordinate system (e.g., the world coordinate system). In some embodiments, similar to some of the above embodiments, the control point coordinate system may be as follows: Fig.12 The coordinate system {RCM} shown, the reference coordinate system can be as follows Figure 5 The world coordinate system {w} is shown.

[0143] In some embodiments, method 1100B may include: determining a change in hand position based on a previous hand position and a current hand position. In some embodiments, the hand position (e.g., the previous hand position and / or the current hand position) may be a hand coordinate system (e.g., Figure 5 The hand coordinate system {h} shown is relative to the reference coordinate system (e.g., Figure 5 The hand position change includes the position change of the hand coordinate system relative to the reference coordinate system.

[0144] In some embodiments, method 1100B may include determining a target position of a control point of at least one motion arm based on a change in hand position, a current position of a control point of at least one motion arm, and a positional relationship between at least one hand and the control point of at least one motion arm.

[0145] In some embodiments, the current position of the control point of at least one motion arm can be obtained based on the kinematic model of the robot system. In some embodiments, the kinematic model of the robot system can be constructed based on the DH parameter method or the exponential product representation method. In some embodiments, at least one motion arm and the mobile station include at least one joint, such as Fig.12 As shown, the mobile station 1200 may include a lifting joint J1, the motion arm 1210 may include joints J2 to J6, or at least one motion arm may include at least one joint, for example, Fig.14 As shown, the motion arm 1410 may include joints 14111-14161. In some embodiments, the current joint value of at least one joint (e.g., joints J1-J6 or joints 14111-14161) may be obtained, and the current position of the control point of at least one motion arm may be calculated based on the current joint value of at least one joint.

[0146] In some embodiments, the positional relationship between the control points of at least one hand and at least one motion arm includes the relationship between the position of the control point coordinate system of at least one motion arm relative to the reference coordinate system and the position of the hand coordinate system of at least one hand relative to the reference coordinate system. In some embodiments, the positional relationship between the control points of at least one hand and at least one motion arm includes the amount of position change of the control point coordinate system of at least one motion arm relative to the reference coordinate system is proportional to the amount of position change of the hand coordinate system of at least one hand relative to the reference coordinate system.

[0147] In some embodiments, method 1100B may include: executing multiple motion control cycles of at least one moving arm at a predetermined period, and determining the current position of the hand of at least one hand and the current position of the control point of at least one moving arm in the motion control cycle. For example, the hand image captured by the image acquisition device may be acquired at a predetermined period, the current position of the hand may be determined based on the hand image, and the current position of the control point of at least one moving arm may be determined based on the joint value of at least one joint of the mobile station and / or at least one moving arm. In some embodiments, the current position of the hand of at least one hand obtained in the previous control cycle may be determined as the previous position of the hand of at least one hand in the current control cycle. In some embodiments, in each control cycle, the target position of the control point of at least one moving arm is determined based on the current position of the hand of at least one hand, the previous position of the hand, the current position of the control point of at least one moving arm, and the positional relationship between the control points of at least one hand and at least one moving arm, so as to control the at least one moving arm to move to the target posture, thereby realizing the position adjustment of at least one moving arm.

[0148] As an example, in some embodiments, the hand configuration for the motion arm position control of at least one motion arm may be as follows: Figure 6C Hand configuration shown. Fig. 13C A schematic diagram of gesture operation for controlling the position of at least one moving arm of some embodiments of the present disclosure is shown. Fig. 13C As shown, the hand coordinate system {h3} and Fig.13A The hand coordinate system {h3} in is consistent.

[0149] In some embodiments, when establishing a motion mapping relationship between at least one hand and a motion arm position control mode of at least one motion arm, the initial hand posture of at least one hand may be as follows: Fig. 13C The initial posture of the hand coordinate system {h3} shown in FIG. The initial posture of the hand coordinate system {h3} can be the initial posture of the hand coordinate system {h3} The plane is parallel to the horizontal plane, The axis points directly in front of the operator. The attitude when the axis is perpendicular to the horizontal plane and pointing vertically upward.

[0150] In some embodiments, the hand coordinate {h3} may be based on the hand position change (eg, Fig. 13C The hand position change p) shown controls at least one movement arm for position adjustment.

[0151] In some embodiments, the hand coordinate system of at least one hand may be based on a previous position of the hand coordinate system relative to the reference coordinate system. w p h3(t0) and current location w p h3 Determine the change in position of at least one hand's hand coordinate system relative to a reference coordinate system w Δp h3 For example, when the motion control of at least one motion arm is triggered or in the last control cycle, represented as time t0, the previous position of the hand coordinate system of at least one hand relative to the reference coordinate system can be determined based on the hand image captured by the image acquisition device. w p h3(t0) In the current control cycle, represented as time t1, based on the hand image captured by the image acquisition device, the current position of the hand coordinate system of at least one hand relative to the reference coordinate system is determined. w p h3 Based on the previous position of the hand coordinate system of at least one hand relative to the reference coordinate system at time t0 w p h3(t0) and the current position of the hand coordinate system of at least one hand relative to the reference coordinate system at time t1 w p h3 , get the position change of at least one hand's hand coordinate system relative to the reference coordinate system w Δp h3 In some embodiments, at time t0, the current position of the control reference point coordinate system of at least one motion arm relative to the reference coordinate system may be determined based on the current joint value of at least one joint of the mobile station and / or at least one motion arm. w p RCM(t0) In some embodiments, the hand coordinate system of at least one hand may be based on the position change relative to the reference coordinate system. w Δp h3 and the current position of the control reference point coordinate system of at least one moving arm relative to the reference coordinate system w p RCM(t0) , determine the target position of the control reference point coordinate system of at least one moving arm relative to the reference coordinate system w p RCM .

[0152] Continue to see Fig. 11BIn step 1103B, a motion control instruction of at least one motion arm is generated based on the target position of the control point of at least one motion arm.

[0153] In some embodiments, at least one of the motion arms and the mobile station includes at least one joint, e.g. Fig.12 As shown, the mobile station 1200 may include a lifting joint J1, the motion arm 1210 may include joints J2 to J6, or at least one motion arm may include at least one joint, for example, Fig.14 As shown, the motion arm 1410 may include joints 14111-14161. In some embodiments, method 1100B may include determining the joint value of at least one joint based on the target position of the control point of at least one motion arm; and determining the drive control instruction of at least one joint motor based on the joint value of at least one joint. In some embodiments, the joint value of at least one joint may be determined based on the target position of the control point of at least one motion arm and the inverse kinematics model of the robot system. In some embodiments, each joint motor (not shown) in at least one joint motor may be arranged at the joint driven by it.

[0154] Continue to see Fig. 11B In step 1105B, based on the motion control instruction of the at least one motion arm, the at least one motion arm is controlled to move. For example, in step 1105B, the control device of the robot system may control the at least one joint motor with the drive control instruction of the at least one joint motor in the robot system to move the at least one motion arm to the target position, thereby achieving the movement of the at least one motion arm following the gesture of the operator.

[0155] In some embodiments, the operator can operate the operating handle to control the movement of the moving arm while observing the moving arm, and determine how to operate the operating handle subsequently according to the movement of the moving arm. In some embodiments, the operator can stand near the end of the moving arm to observe the position of the control point of the moving arm and determine whether the control point of the moving arm is in place.

[0156] In some embodiments, the mobile station may include an image acquisition device. For example, the image acquisition device may be disposed on a distal arm body (eg, Fig.12 The third arc arm B5 shown) provides a distal end of the platform (eg, Fig.12 In some embodiments, the camera optical axis of the image acquisition device (eg, Fig.12 The straight line p shown can be drawn from the camera reference point of the image acquisition device (e.g., Fig.12 The position indicated by 12a in FIG. 1 is directed to the remote motion center of the motion arm (eg Fig.12The operator can judge whether the control point of the moving arm is in place by observing the field of view provided by the image acquisition device in the display of the surgical robot. In some embodiments, the display can also display at least one auxiliary line while displaying the field of view of the image acquisition device, and the operator can judge whether the moving arm is in place with the help of at least one auxiliary line. For example, the operator can judge that the moving arm has moved into place when observing that at least one auxiliary line displayed in the display coincides with the auxiliary line on the inner wall of at least one channel provided by the sheath. When at least one auxiliary line displayed in the display coincides with the auxiliary line on the inner wall of at least one channel provided by the sheath (at this time, the end of the sheath can be extended into the opening of the patient), it means that in this state, the surgical instrument carried by the distal end of the moving arm can enter the patient's body through the channel provided by the sheath, thereby indicating that the moving arm is in place.

[0157] The operator can determine whether the height of the control point of the motion arm is in place or appropriate in a variety of ways. In some embodiments, the operator can determine whether the height of the control point of the motion arm is in place or appropriate in a variety of ways. Fig.10 The first positioning device 1040 and the second positioning device 1050 shown determine whether the height of the moving arm is appropriate. Fig.12 The lifting joint J1 shown or Fig.14 During the lifting and lowering of the lifting joint 14131 shown in the figure, the distances between the first positioning device 1040 and the second positioning device 1050 and the operating table are different, and the sizes of the cursors projected on the patient on the operating table are also different. Based on this, during the process of adjusting the height of the control point of the moving arm, if it is observed that the circular cursor projected on the patient by the first positioning device 1040 is tangent to the cross-shaped cursor projected by the second positioning device 1050 (at a position such as Fig.10 By looking at the status of the circular cursor and the cross cursor shown in FIG. 1 , it can be determined that the control point of the moving arm is at a suitable height.

[0158] In some embodiments, when the posture control of at least one moving arm of the surgical robot system is performed by hand gestures (for example, including moving arm position control and / or moving arm posture control) and it is difficult for the operator to move the at least one moving arm into position by hand gesture control, the operator can establish a motion control relationship between at least one hand and the mobile station by changing the gesture (for example, a gesture configuration matching the first preset configuration or the second preset configuration), switch to the motion control mode of the mobile station (for example, the forward control mode of the mobile station and / or the backward control mode of the mobile station), and further adjust the posture of the mobile station by hand gestures, and then establish a motion control relationship between at least one hand and at least one moving arm by changing the gesture again (for example, a gesture configuration matching the third preset configuration or the fourth preset configuration), and switch to the motion control mode of at least one moving arm (for example, the position control mode of the moving arm and / or the posture control mode of the moving arm) again, so as to adjust the posture of the control point of at least one moving arm by hand gestures until the control point is in position again.

[0159] In some embodiments of the present disclosure, the present disclosure further provides a computer device, the computer device comprising a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory and is used to execute at least one instruction to execute some or all steps in the method of the present disclosure, for example, Figure 1 Some or all of the steps in the method 100 disclosed in Figure 7 Some or all of the steps in the method 700 disclosed in Fig.11A Some or all of the steps in method 1100A disclosed in Fig. 11B Part or all of the steps in method 1100B disclosed in.

[0160] Fig.15 1 is a schematic block diagram of a computer device 1500 according to some embodiments of the present disclosure. Fig.10 The computer device 1500 may include a central processing unit (CPU) 1501, a system memory 1504 including a random access memory (RAM) 1502 and a read-only memory (ROM) 1503, and a system bus 1505 connecting the various components. The computer device 1500 may also include an input / output system 1506, and a large-capacity storage device 1507 for storing an operating system 1512, application programs 1513, and other program modules 1514. The input / output system includes an input / output controller 1506 mainly composed of a display 1508 and an input device 1509.

[0161] The mass storage device 1507 is connected to the central processing unit 1501 through a mass storage controller (not shown) connected to the system bus 1505. The mass storage device 1507 or computer readable medium provides non-volatile storage for the computer device. The mass storage device 1507 may include a computer readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0162] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory and mass storage devices may be collectively referred to as memory.

[0163] The computer device 1500 can be connected to a network 1511 through a network interface unit 1510 connected to a system bus 1505. The system memory 1504 or the mass storage device 1507 is also used to store one or more instructions. The central processing unit 1501 implements all or part of the steps of the method in some embodiments of the present disclosure by executing the one or more instructions, for example, Figure 1 Some or all of the steps in the method 100 disclosed in Figure 7 Some or all of the steps in the method 700 disclosed in Fig.11A Some or all of the steps in method 1100A disclosed in Fig. 11B Part or all of the steps in method 1100B disclosed in.

[0164] In some embodiments of the present disclosure, the present disclosure further provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor to enable a computer to execute some or all steps of the method in some embodiments of the present disclosure, such as Figure 1 Some or all of the steps in the method 100 disclosed in Figure 7 Some or all of the steps in the method 700 disclosed in Fig.11A Some or all of the steps in method 1100A disclosed in Fig. 11BExamples of computer-readable storage media include storage of computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0165] Note that the above are only exemplary embodiments of the present disclosure and the technical principles used. It should be understood by those skilled in the art that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. A motion control method for a robot system based on gestures, characterized in that: The robot system comprises a mobile station and at least one motion arm disposed on the mobile station, and the method comprises: obtaining a set of hand images of at least one hand; Based on the hand image set, obtaining gesture information of the at least one hand; and Based on the gesture information, the mobile station or the at least one motion arm is controlled to move.

2. The method according to claim 1, characterized in that The gesture information of at least one hand includes a hand configuration and a hand posture, and the method includes: In response to the hand configuration matching a mobile station mapping configuration in a gesture configuration set and the hand posture matching a mapping posture, establishing a motion control mapping relationship between the at least one hand and the mobile station; and / or In response to the hand configuration matching the motion arm mapping configuration in the gesture mapping configuration set and the hand posture matching the mapping posture, a motion control mapping relationship between the at least one hand and the at least one motion arm is established.

3. The method according to claim 2, characterized in that The gesture configuration set includes at least one preset configuration for at least one motion control mode of the mobile station and / or the at least one motion arm, the mobile station mapping configuration includes a first preset configuration for forward control of the mobile station and / or a second preset configuration for backward control of the mobile station, and the motion arm mapping configuration includes a third preset configuration for position control of the at least one motion arm and / or a fourth preset configuration for posture control of the at least one motion arm.

4. The method according to claim 2, characterized in that: The method further comprises: In response to a hand gesture matching the mobile station mapping configuration or the motion arm mapping configuration and the mapping gesture being maintained for a preset time period, motion control of the mobile station or the at least one motion arm is triggered.

5. The method according to claim 3, characterized in that: The method further comprises: In response to the hand configuration not matching any preset configuration in the gesture configuration set, the motion control mapping relationship between the at least one hand and the mobile station or the at least one motion arm is disconnected.

6. The method according to claim 2, characterized in that The method comprises: Determining a target velocity and / or a target angular velocity of a control point of the mobile station or a control point of the at least one motion arm based on the hand gesture; generating a motion control instruction for the mobile station or the at least one motion arm based on a target velocity and / or a target angular velocity of a control point of the mobile station or a control point of the at least one motion arm; and Based on the motion control instruction of the mobile station or the at least one motion arm, the movement of the mobile station or the at least one motion arm is controlled.

7. The method according to claim 6, characterized in that The hand posture includes the posture of the hand coordinate system in the reference coordinate system, and the mapping posture is the set posture of the hand in the reference coordinate system.

8. The method according to claim 7, characterized in that The method further comprises: Determine a hand posture change based on the initial hand posture and the hand manipulation posture; Determining mobile station posture control information and / or mobile station acceleration and deceleration control information of the mobile station based on the hand posture change; and Based on the mobile station attitude control information and / or the mobile station acceleration and deceleration control information, a target speed and / or a target angular speed of a control point of the mobile station is determined.

9. The method according to claim 7, characterized in that: The method further comprises: Determine a hand posture change based on the initial hand posture and the hand manipulation posture; Determining the motion arm position control information and / or the motion arm posture control information of the at least one motion arm based on the hand posture change; and Based on the motion arm position control information and / or the motion arm posture control information, a target velocity and / or a target angular velocity of a control point of at least one motion arm is determined.

10. The method according to claim 8 or 9, characterized in that: The hand posture change includes rotation information of at least one coordinate axis of the hand coordinate system, and the rotation information of at least one coordinate axis includes a rotation direction and / or a rotation angle.

11. The method according to claim 10, characterized in that The mobile station attitude control information includes the mobile station attitude control direction and the mobile station attitude control angular velocity; The method further comprises: Determining a posture control direction of the mobile station based on a rotation direction of the at least one coordinate axis; Determining the magnitude of the attitude control angular velocity of the mobile station based on the rotation angle of the at least one coordinate axis; and Based on the attitude control direction of the mobile station and the magnitude of the attitude control angular velocity of the mobile station, a target angular velocity of a control point of the mobile station is determined.

12. The method according to claim 10, characterized in that The mobile station acceleration and deceleration control information includes an acceleration and deceleration mode and an acceleration magnitude; The method further comprises: Determining the acceleration / deceleration mode based on the rotation direction of the at least one coordinate axis; Determining the magnitude of the acceleration based on the rotational angular velocity of the at least one coordinate axis; and A target speed of the control point of the mobile station is determined based on the acceleration / deceleration mode, the acceleration magnitude, and the current speed of the control point of the mobile station.

13. The method according to claim 10, characterized in that The motion arm position control information includes the motion arm position control direction, and the motion arm posture control information includes the motion arm posture control direction; The method further comprises: Determining the position control direction of the movable arm or the posture control direction of the moving arm based on the rotation direction of the at least one coordinate axis; and Based on the motion arm position control direction and the speed of the motion arm position control, the target speed of the control point of at least one motion arm is determined; or, based on the motion arm posture control direction and the angular speed of the motion arm posture control, the target angular speed of the control point of at least one motion arm is determined.

14. The method according to claim 13, characterized in that The method further comprises: Based on the fact that the rotation angle of the at least one coordinate axis is not less than a preset angle, and based on the rotation direction of the at least one coordinate axis, the position adjustment direction of the motion arm or the posture adjustment direction of the motion arm is determined.

15. The method according to claim 2, characterized in that The gesture information of at least one hand also includes a hand position, and the method includes: determining a target position of a control point of the at least one motion arm based on the hand position; and generating a motion control instruction for the at least one motion arm based on a target position of a control point of the at least one motion arm; and Based on the motion control instruction of the at least one motion arm, the at least one motion arm is controlled to move.

16. The method according to claim 15, characterized in that The method further comprises: Determining a change in hand position based on a previous hand position and a current hand position; and Based on the hand position change, the current position of the control point of at least one motion arm, and the positional relationship between the at least one hand and the control point of the at least one motion arm, a target position of the control point of the at least one motion arm is determined.

17. The method according to claim 16, characterized in that The positional relationship includes that the position change amount of the control point coordinate system of the at least one motion arm relative to the reference coordinate system is proportional to the position change amount of the hand coordinate system of the at least one hand relative to the reference coordinate system.

18. The method according to claim 17, characterized in that The method further comprises: A plurality of motion control cycles of the at least one motion arm are executed in a predetermined period, in which the current hand position of the at least one hand obtained in the previous control cycle is determined as the previous hand position of the at least one hand in the current control cycle.

19. The control method according to claim 6, characterized in that: The mobile station includes at least one active caster and at least one caster motor for driving the at least one active caster, and the method further includes: determining a target speed of the at least one active caster based on a target speed and / or a target angular speed of a control point of the mobile station; and Based on the target speed and radius of the at least one active caster, a driving control instruction of the at least one caster motor is determined.

20. The method according to claim 19, characterized in that The at least one active caster includes a first caster and a second caster that are symmetrically arranged, and the control point of the mobile station includes a midpoint between the first caster and the second caster.

21. The method according to claim 6 or 15, characterized in that The at least one motion arm, or the at least one motion arm and the mobile station, comprises at least one joint and at least one joint motor for driving the at least one joint, and the control method further comprises: determining a joint value and a joint speed of the at least one joint based on a target speed and / or a target angular speed of a control point of the at least one motion arm; and Determining a drive control instruction of the at least one joint motor based on the joint value and the joint speed of the at least one joint; or determining a joint value of the at least one joint based on a target position of a control point of the at least one motion arm; and Based on the joint value of the at least one joint, a drive control instruction of the at least one joint motor is determined.

22. The method according to claim 21, characterized in that The at least one joint comprises: a lifting joint of the mobile station; and The first cross arm rotation joint, the second cross arm rotation joint, the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint of the at least one moving arm; or The at least one joint comprises: The at least one moving arm comprises a first cross arm rotation joint, a second cross arm rotation joint, a lifting joint, a first arc arm rotation joint, a second arc arm rotation joint and a third arc arm rotation joint.

23. The method according to claim 22, characterized in that The method comprises: Determining joint values ​​and joint velocities of a lifting joint of the mobile station, a first cross-arm rotation joint, and a second cross-arm rotation joint of the at least one motion arm based on a target velocity of a control point of the at least one motion arm; and Determine a drive control instruction for driving a joint motor of the lifting joint of the mobile station, the first cross-arm rotation joint of the at least one motion arm, and the second cross-arm rotation joint based on the joint values ​​and joint speeds of the lifting joint of the mobile station, the first cross-arm rotation joint of the at least one motion arm, and the second cross-arm rotation joint; or Determining joint values ​​and joint velocities of a first cross-arm rotation joint, a second cross-arm rotation joint, and a lifting joint of the at least one motion arm based on a target velocity of a control point of the at least one motion arm; and Determine a drive control instruction for driving a joint motor of the first cross arm rotation joint, the second cross arm rotation joint, and the lifting joint of the at least one motion arm based on the joint values ​​and joint speeds of the first cross arm rotation joint, the second cross arm rotation joint, and the lifting joint of the at least one motion arm; or determining joint values ​​and joint velocities of a first arc-arm rotation joint, a second arc-arm rotation joint, and a third arc-arm rotation joint of the at least one motion arm based on a target angular velocity of a control point of the at least one motion arm; and Based on the joint values ​​and joint velocities of the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint of the at least one moving arm, the drive control instructions for the joint motors driving the first arc arm rotation joint, the second arc arm rotation joint and the third arc arm rotation joint of the at least one moving arm are determined.

24. A robot system, characterized in that: include: Mobile station; at least one moving arm disposed on the mobile station; An image acquisition device, used for acquiring a hand image of at least one hand; as well as A control device, which is in communication with the mobile station, the at least one moving arm and the image acquisition device, is used to execute the method as described in any one of claims 1-23.

25. A computer device, characterized in that: The computer device comprises: A memory for storing at least one instruction; and A processor, coupled to the memory and configured to execute the at least one instruction to perform the method according to any one of claims 1-23.

26. A computer-readable storage medium for storing at least one instruction, characterized in that: When the at least one instruction is executed by a computer device, it causes the computer to implement the method according to any one of claims 1 to 23.

Images