Method for matching master and slave motions based on a positioning image and surgical robot system

Abstract

The present disclosure relates to the field of robotics, and discloses a control method of master-slave motion based on positioning images, comprising: acquiring a positioning image; determining a current pose of a slave tool relative to a reference coordinate system based on the positioning image; determining a target pose of a handle of a master manipulator based on the current pose of the slave tool; and generating a control signal for the master manipulator based on the target pose of the handle of the master manipulator.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and more particularly to a master-slave motion control method based on positioning images and a surgical robot system. Background Technology

[0002] With the development of technology, the use of medical robots to assist medical staff in performing surgeries has developed rapidly. Medical robots can not only help medical staff perform a series of medical diagnoses and auxiliary treatments, but also effectively alleviate the problem of strain on medical resources.

[0003] Typically, a medical robot consists of a driven tool for performing operations and a master manipulator for controlling the movement of the driven tool. In a real-world scenario, the driven tool is configured to enter the operating area, and medical personnel remotely operate the master manipulator to control the movement of the driven tool within the operating area to perform the medical procedure.

[0004] However, the number of remotely controlled slave tools is generally greater than the number of master tools. Therefore, during surgery, there may be situations where the slave tools controlled by the master tool are altered. Furthermore, at the start or during the operation, the master tool needs to establish a mapping with the slave tools before performing master-slave control. Because the master tool and its corresponding controlled slave tool are not pre-matched in terms of posture, a mismatch (such as orientation or angle) can occur between them. Directly matching them in master-slave mapping would reduce the control precision of the slave tools and degrade the human-computer interaction experience for medical personnel (such as surgeons). Therefore, after the master tool and slave tool are matched and before remote operation, the posture of the master tool needs to be matched with the posture of the slave tool to improve the accuracy of the master tool's posture control over the slave tools. Summary of the Invention

[0005] In some embodiments, this disclosure provides a master-slave motion control method, including: acquiring a positioning image; determining the current posture of a slave tool relative to a reference coordinate system based on the positioning image; determining the target posture of a master operator's handle based on the current posture of the slave tool; and generating a control signal for the master operator based on the target posture of the master operator's handle.

[0006] In some embodiments, this disclosure provides a robot system, including: a master manipulator, comprising a multi-degree-of-freedom robotic arm, a handle disposed on the multi-degree-of-freedom robotic arm, and at least one motor and at least one master manipulator sensor disposed at at least one joint on the multi-degree-of-freedom robotic arm, wherein the at least one master manipulator sensor is used to obtain joint information of at least one joint; a slave tool, comprising a manipulator arm and an end effector disposed at the end of the manipulator arm; an image acquisition device for acquiring positioning images; and a control device communicatively connected to the image acquisition device and the master manipulator, wherein the control device is configured to perform a master-slave motion control method according to some embodiments of this disclosure.

[0007] In some embodiments, this disclosure provides a computer device, the computer device including: a memory for storing at least one instruction; and a processor coupled to the memory and configured to execute at least one instruction to perform a master-slave motion control method according to some embodiments of this disclosure.

[0008] In some embodiments, this disclosure provides a computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes a robot system to implement a master-slave motion control method according to some embodiments of this disclosure. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below. The accompanying drawings described below only show some embodiments of this disclosure. For those skilled in the art, other embodiments can be obtained based on the content of the embodiments of this disclosure and these drawings without creative effort.

[0010] Figure 1 Schematic diagrams of robot systems according to some embodiments of the present disclosure are shown;

[0011] Figure 2 A schematic diagram of the structure of an operating arm according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of the structure of an operating arm according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram illustrating a label including multiple pose markers and multiple angle markers according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic diagram shows a label that is disposed on the periphery of the end of the operating arm and forms a cylindrical shape;

[0015] Figure 6 Schematic diagrams illustrating implementation scenarios according to some embodiments of the present disclosure;

[0016] Figure 7 A flowchart illustrating a master-slave motion control method according to some embodiments of the present disclosure is shown;

[0017] Figure 8 A flowchart illustrating a method for determining the attitude of a driven tool coordinate system relative to a reference coordinate system according to some embodiments of the present disclosure;

[0018] Figure 9 A schematic diagram showing multiple pose markers on a cross-sectional circle according to some embodiments of the present disclosure;

[0019] Figure 10 A flowchart illustrating a method for determining the attitude of a driven tool coordinate system relative to a reference coordinate system, according to other embodiments of the present disclosure;

[0020] Figure 11 A flowchart illustrating a method for identifying pose identifiers according to some embodiments of the present disclosure;

[0021] Figure 12 A schematic diagram showing pose identification patterns according to some embodiments of the present disclosure;

[0022] Figure 13 A flowchart illustrating a method for searching pose identifiers according to some embodiments of the present disclosure;

[0023] Figure 14 A schematic diagram illustrating a search pose identifier according to some embodiments of the present disclosure;

[0024] Figure 15 A flowchart illustrating a method for identifying angle markers according to some embodiments of the present disclosure;

[0025] Figure 16 A schematic diagram of a master operator according to some embodiments of the present disclosure is shown;

[0026] Figure 17 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown;

[0027] Figure 18 Schematic diagrams of surgical robot systems according to some embodiments of the present disclosure are shown;

[0028] Figure 19 A schematic diagram of a surgical instrument according to some embodiments of the present disclosure is shown;

[0029] Figure 20 A schematic diagram of a main control trolley according to some embodiments of the present disclosure is shown;

[0030] Figure 21 A schematic diagram of an operating table according to some embodiments of the present disclosure is shown. Detailed Implementation

[0031] To make the technical problems solved by this disclosure, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely exemplary embodiments of this disclosure, and not all embodiments.

[0032] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this disclosure, it should be noted that unless otherwise expressly specified and limited, the terms "installed," "connected," "coupled," and "coupled" should be interpreted broadly. For example, they can refer to fixed connections or detachable connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances. In this disclosure, the end closer to the operator (e.g., a doctor) is defined as the proximal end, proximal or rear end, or posterior end, and the end closer to the surgical patient is defined as the distal end, distal or anterior end, or anterior end. Those skilled in the art will understand that embodiments of this disclosure can be used in medical devices or surgical robots, as well as in other non-medical devices.

[0033] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., variations in Cartesian X, Y, and Z coordinates can be used to describe three translational degrees of freedom, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "attitude" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and attitude of an object or part of an object, which can be described, for example, using six parameters from the six degrees of freedom mentioned above. In this disclosure, the pose of the handle of the master manipulator can be represented by a set of joint information of the master manipulator joints (e.g., a one-dimensional matrix composed of this joint information). In this disclosure, the joint information of a joint can include the angle of rotation of the respective joint relative to the corresponding joint axis or the distance moved relative to the initial position.

[0034] In this disclosure, a reference coordinate system can be understood as a coordinate system capable of describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own perception coordinate system, etc. In this disclosure, an object can be understood as an object or target that needs to be positioned, such as the operating arm of a driven tool or the end effector of the operating arm. In this disclosure, the pose of the driven tool or a portion thereof refers to the pose of the driven tool coordinate system defined by the driven tool or a portion thereof relative to the reference coordinate system.

[0035] Figure 1 A schematic diagram of a robot system 100 according to some embodiments of the present disclosure is shown. Figure 1 As shown, the robot system 100 may include: an image acquisition device 110, a control device 120, at least one driven tool (150a, 150b), and a master manipulator 180. In some embodiments, the image acquisition device 110 and the master manipulator 180 are respectively communicatively connected to the control device 120.

[0036] In some embodiments, the image acquisition device 110 can be used to acquire positioning images. The positioning images may include part or all of the image of the driven tools (150a, 150b). In some embodiments, positioning markers are provided on the driven tools (150a, 150b). In some embodiments, the positioning markers include pose markers, based on which the position or orientation of the driven tools (150a, 150b) can be determined. In some embodiments, the positioning markers may include pose markers and angle markers (detailed below), based on which the position and orientation of the driven tools (150a, 150b) can be determined. Figure 1 As shown, if the driven tools (150a, 150b) are within the field of view of the image acquisition device 110, the acquired positioning image may include a partial image of the driven tools (150a, 150b). In some embodiments, the image acquisition device 110 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. Depending on the application scenario, the image acquisition device 110 may be an industrial camera, an underwater camera, a miniature electronic camera, an endoscope camera, etc. In some embodiments, the image acquisition device 110 may be fixed in position or changeable in position, for example, an industrial camera fixed at a monitoring position or an endoscope camera whose position or attitude can be adjusted. In some embodiments, the image acquisition device 110 may realize at least one of visible light band imaging, infrared band imaging, CT (Computed Tomography) imaging, and acoustic imaging. In some embodiments, the image acquisition device 110 may be, for example, a... Figure 21 The imaging module 2160b shown in the figure.

[0037] In some embodiments, the control device 120 is configured to execute at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 7 , Figure 8 , Figure 10 , Figure 11 , Figure 13 and Figure 15 The method disclosed herein may include some or all of the steps. In some embodiments, the control device 120 may receive a positioning image from the image acquisition device 110 and process the positioning image. For example, the control device 120 may identify multiple positioning markers located on at least one slave tool (150a, 150b) in the positioning image. In some embodiments, the control device 120 may determine the posture of the slave tools (150a, 150b) based on the positioning image, for example, determining the current posture of the slave tools (150a, 150b) relative to a reference coordinate system based on the positioning image. In some embodiments, the control device 120 may also determine the posture of the master operator 180 based on the posture of the slave tools (150a, 150b), for example, determining the target posture of the handle of the master operator 180 based on the current posture of the slave tools (150a, 150b). The target posture of the handle of the master operator 180 is mapped to the current posture of the slave tools (150a, 150b). In some embodiments, the control device 120 may also generate control signals for the master operator 180 based on the target posture of the handle of the master operator 180. In some embodiments, the control device 120 may send the control signals of the master operator 180 to the drive motors of multiple joints of the master operator 180.

[0038] In some embodiments, the main manipulator 180 includes a multi-degree-of-freedom robotic arm (e.g., a six-degree-of-freedom robotic arm), with joint sensors disposed at some joints of the multi-degree-of-freedom robotic arm to generate joint information (such as joint angle data). In some embodiments, the joint sensors are potentiometers and / or encoders. In some embodiments, the main manipulator may be, for example, a... Figure 16 The main manipulator 1600 is shown in the figure. In some embodiments, the main manipulator 1600 may be equipped with a controller, which can calculate the attitude data of the main manipulator 1600 based on the joint information obtained from each joint sensor, and send the calculated attitude data to the control device 120. In other embodiments, the control device 120 may also calculate the attitude data of the main manipulator based on the joint information sent by the joint sensors.

[0039] In some embodiments, the driven tool 150a is taken as an example. Figure 1As shown, the driven tool 150a includes an operating arm 140. In some embodiments, the driven tool 150a further includes an end effector 160 disposed at the end 130 of the operating arm. In some embodiments, the operating arm 140 may be a rigid arm or a deformable arm. In some embodiments, the operating arm 140 may include a continuous deformable arm. A continuous deformable arm is, for example, as shown... Figure 3 The manipulator 300 is shown. In some embodiments, the manipulator 140 may include a multi-degree-of-freedom manipulator composed of multiple joints. For example, a manipulator capable of 4 to 7 degrees of freedom of movement. For instance, a manipulator capable of 6 degrees of freedom of movement. In some embodiments, the end effector 160 may include, but is not limited to, surgical forceps, an electrosurgical unit, an electrohook, etc.

[0040] Figure 2 A schematic diagram of a segment 200 of an operating arm according to some embodiments of the present disclosure is shown. In some embodiments, the operating arm of a driven tool may include at least one deformable segment 200. Figure 2 As shown, the deformable segment 200 includes a fixed disk 210 and multiple structural bones 220. The first end of each structural bone 220 is fixedly connected to the fixed disk 210, and the second end is connected to a driving unit (not shown). In some embodiments, the fixed disk 210 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and its cross-section may be circular, rectangular, polygonal, or various other shapes. In some embodiments, the driving unit deforms the segment 200 by driving the structural bones 220. For example, the driving unit deforms the segment 200 by driving the structural bones 220 to a position such as... Figure 2 The bending state is shown. In some embodiments, the second end of the multiple structural bones 220 passes through the base plate 230 and is connected to the drive unit. In some embodiments, similar to the fixed plate 210, the base plate 230 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and the cross-section may be a circle, rectangle, polygon, etc. The drive unit may include a linear motion mechanism, a drive segment, or a combination of both. The linear motion mechanism may be connected to the structural bones 220 to push or pull the structural bones 220, thereby driving the segment 200 to bend. The drive segment may include a fixed plate and multiple structural bones, wherein one end of the multiple structural bones is fixedly connected to the fixed plate. The other end of the multiple structural bones of the drive segment is connected to or integrally formed with the multiple structural bones 220 to drive the bending of the segment 200 by bending the drive segment. In some embodiments, a spacer plate 240 is also included between the fixed plate 210 and the base plate 230, through which the multiple structural bones 220 pass. Similarly, the drive segment may also include a spacer plate.

[0041] Figure 3 A schematic diagram of the structure of an operating arm 300 according to some embodiments of the present disclosure is shown. For example... Figure 3As shown, the operating arm 300 is a deformable operating arm, which may include an operating arm end cap 310 and an operating arm body 320. The operating arm body 320 may include one or more segments, such as a first segment 3201 and a second segment 3202. In some embodiments, the structures of the first segment 3201 and the second segment 3202 may be similar to those of... Figure 2 The shown component 200 is similar. In some implementations, such as... Figure 3 As shown, the main body 320 of the operating arm also includes a first straight rod segment 3203 located between the first segment 3201 and the second segment 3202. The first end of the first straight rod segment 3203 is connected to the base plate of the second segment 3202, and the second end is connected to the fixed plate of the first segment 3201. In some embodiments, such as... Figure 3 As shown, the main body 320 of the operating arm also includes a second straight rod segment 3204, the first end of which is connected to the base plate of the first component 3201. Figure 3 As shown, each segment (first segment 3201 and second segment 3202) may include a base plate, a fixed plate, and multiple structural bones penetrating the base plate and the fixed plate. The multiple structural bones may be fixedly connected to the fixed plate and slidably connected to the base plate. The continuous deformable arm and its included segments can be described by a kinematic model (detailed below).

[0042] In some embodiments, the structure of each segment of the manipulator 300 may be as follows: Figure 2 The shown component is 200. (As shown in the image) Figure 2 As shown, the base disk coordinate system It is attached to the base disk of the t-th (t=1,2,3…) continuous segment, with its origin located at the center of the base disk, and the XY plane coinciding with the plane of the base disk. Pointing from the center of the base plate to the first structural bone (the first structural bone can be understood as any one of multiple structural bones chosen as a reference). Bending plane coordinate system 1 Its origin coincides with the origin of the base disk coordinate system, and the XY plane coincides with the bending plane. and Coincident. Fixed disk coordinate system. It is attached to the fixed disk of the t-th continuous segment, with its origin located at the center of the fixed disk, and the XY plane coinciding with the plane of the fixed disk. Pointing from the center of the fixed plate to the first structural bone. Curved plane coordinate system 2 Its origin is located at the center of the fixed disk, and the XY plane coincides with the bending plane. and coincide.

[0043] like Figure 2 The single segment 200 shown can be represented by a kinematic model. The end of segment t (fixed disk coordinate system) Relative to the base disk coordinate system Location ,attitude It can be determined based on the following formulas (1) and (2):

[0044] (1)

[0045] (2)

[0046] in, For the t-th segment, construct a virtual structural bone (e.g., Figure 2 The length of the virtual structure bone 221 shown in the figure, In the t-th section, about or Rotate to Required rotation angle Let t be the bending plane coordinate system of the segment. Relative to the base disk coordinate system posture, 2. The bending plane coordinate system of the t-th segment Relative to the curved plane coordinate system 1 posture, For the fixed disk coordinate system of the t-th section Relative to the curved plane coordinate system 2 The posture.

[0047] , and It can be based on the following formulas (3), (4) and (5):

[0048] (3)

[0049] (4)

[0050] (5)

[0051] in, For the t-th segment, the bending plane and The included angle.

[0052] like Figure 2 Joint parameters of a single segment 200 shown It can be determined based on the following formula (6):

[0053] (6)

[0054] In some embodiments, the driving amount of multiple bone structures has a known mapping relationship with joint parameters. Based on the target joint parameters of the segment and the mapping relationship, the driving amount of the multiple bone structures can be determined. The driving amount of the multiple bone structures can be understood as moving a single segment from its initial state (e.g., The length of the structural bone subjected to push or tension when bent to the target bending angle. In some embodiments, the mapping relationship between the driving amount of multiple structural bones and joint parameters can be determined based on the following formula (7):

[0055] (7)

[0056] in, For the t-th section, the first section is constructed. The distance from the root bone structure to the virtual bone structure. For the t-th section, the first section is constructed. The angle between the root bone structure and the first root bone structure. For the first The driving force of the root bone structure, based on the first The driving force of the root bone structure can determine the driving signal of the driving unit.

[0057] In some embodiments, the entire deformable arm can be described by a kinematic model. For example... Figure 3 As shown, transformations can be performed between multiple coordinate systems located at multiple positions of the deformable arm. For example, the end effector of the continuous deformable arm can be determined in the world coordinate system {w} based on the following formula (8):

[0058] (8)

[0059] in, The homogeneous transformation matrix of the end effector of the deformable arm of a continuum relative to the world coordinate system; Represents the homogeneous transformation matrix of the base disk of the first continuum segment relative to the world coordinate system; Represents the homogeneous transformation matrix of the fixed disk of the first continuous segment relative to the base disk of the first continuous segment; Represents the homogeneous transformation matrix of the base disk of the second continuous segment relative to the fixed disk of the first continuous segment; Represents the homogeneous transformation matrix of the fixed disk of the second continuum segment relative to the base disk of the second continuum segment; This represents the homogeneous transformation matrix of the end effector of the continuous deformable arm relative to the fixed disk of the second continuous segment. In some embodiments, the end effector is fixedly mounted on the fixed disk, therefore... It is known or predetermined.

[0060] Those skilled in the art will understand that the deformable arm has different joint parameters in different working states. For example, Figure 3 The manipulator 300 shown includes at least four operating states. The four operating states of the manipulator 300 are described below:

[0061] First working state: Only the second segment 3202 participates in the pose control of the end effector (e.g., only the second segment 3202 enters the workspace), at which time the joint parameters of the manipulator 300 can be determined based on the following formula (9):

[0062] (9)

[0063] in, These are the joint parameters of the manipulator 300 in its first working state. The rotation angle of the manipulator 300 around its axis. , , With Figure 2 In the structure 200 shown , and They have the same physical meaning.

[0064] Second working state: The second segment 3202 and the first straight segment 3203 participate in the pose control of the end effector (for example, the second segment 3202 is fully in the workspace, and the first straight segment 3203 is partially in the workspace). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (10):

[0065] (10)

[0066] in, These are the joint parameters of the manipulator 300 in the second working state. This is the feed rate for the first straight segment 3203.

[0067] Third working state: The second segment 3202, the first straight segment 3203, and the first segment 3201 participate in the pose control of the end effector (for example, the second segment 3202 is fully in the working space, the first straight segment 3203 is fully in the working space, and the first segment 3201 is partially in the working space). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (11):

[0068] (11)

[0069] in, These are the joint parameters of the manipulator 300 in the third working state. , and With Figure 2 In the structure 200 shown , and They have the same physical meaning.

[0070] Fourth working state: The second segment 3202, the first straight segment 3203, the first segment 3201, and the second straight segment 3204 participate in the pose control of the end effector (for example, the second segment 3202 is fully in the working space, the first straight segment 3203 is fully in the working space, the first segment 3201 is fully in the working space, and the second straight segment 3204 is partially in the working space). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (12):

[0071] (12)

[0072] in, These are the joint parameters of the manipulator 300 in the fourth working state. This is the feed rate for the second straight segment 3204.

[0073] In some embodiments, the driven tool is provided with multiple pose markers. For example, multiple pose markers are distributed on the manipulator arm of the driven tool. For example, the multiple pose markers are distributed circumferentially on the end of the manipulator arm, or the multiple pose markers are distributed axially along the manipulator arm. Based on an image of the multiple pose markers, the pose of the driven tool can be determined. In some embodiments, the multiple pose markers may include a single row or multiple rows of pose markers.

[0074] In some embodiments, the driven tool is provided with a plurality of pose markers and at least one angle marker. For example, the operating arm of the driven tool has a plurality of pose markers and at least one angle marker distributed thereon. For example, the plurality of pose markers are distributed circumferentially on the end portion 310 of the operating arm, and the plurality of angle markers are distributed circumferentially on the end portion 310 of the operating arm. The plurality of pose markers and the plurality of angle markers are arranged side by side axially on the end portion 310 of the operating arm. For example, the plurality of pose markers and the plurality of angle markers are disposed on the outer surface of the columnar portion of the end portion 310 of the operating arm.

[0075] In some embodiments, each angle marker has a positional association with one of the pose markers. Based on this positional association, the possible distribution area of ​​the angle markers can be determined by the position of the pose markers. Alternatively, the possible distribution area of ​​the pose markers can be determined by the position of the angle markers. The positional association can be determined according to the specific arrangement of the pose markers and angle markers, and can be pre-designed.

[0076] In some embodiments, the positional association may include an axial correspondence between angle markers and pose markers. For example, the positional association may include an axial offset. Based on the axial correspondence, given that the positions of one or more pose markers on the end effector of the manipulator are known, an axial offset by a certain distance can determine the area where the angle markers may exist. For example, the positional association may also include axial oblique alignment, etc.

[0077] In some embodiments, multiple pose markers and multiple angle markers can be set on a label (e.g., Figure 4 The label shown is 400 or Figure 5 The label 500 shown is affixed to the driven tool. In some embodiments, the label is affixed to the periphery of the end of the operating arm.

[0078] In some embodiments, a pose identifier may include a pose identifier pattern and pose identifier pattern corner points, and an angle identifier may include an angle identifier pattern and angle identifier pattern corner points. In some embodiments, the pose identifier pattern and angle identifier pattern may be disposed on a label attached to the end of the operating arm, or may be printed on the end of the operating arm, or may be patterns formed by the physical structure of the end of the operating arm itself, for example, including recesses or protrusions and combinations thereof. In some embodiments, the pose identifier pattern or angle identifier pattern may include patterns formed with brightness, grayscale, color, etc. In some embodiments, the pose identifier pattern and angle identifier pattern may include patterns that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provide information to be detected by the image acquisition module. Those skilled in the art will understand that in some embodiments, the pose of the pose identifier may be represented by the pose of the pose identifier pattern corner point coordinate system, and the pose of the angle identifier may be represented by the pose of the angle identifier pattern corner point coordinate system.

[0079] In some embodiments, the pose marking pattern or angle marking pattern is set on the end of the manipulator in an area suitable for image acquisition by an image acquisition device, such as an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily disturbed or obstructed during operation.

[0080] Figure 4 A schematic diagram of a label 400 including multiple pose identifiers and multiple angle identifiers according to some embodiments is shown. Figure 5 A schematic diagram is shown of a label 500 disposed on the periphery of the end of the operating arm and forming a cylindrical shape. It can be understood that, for simplicity, label 400 may include the same pose marking pattern and angle marking pattern as label 500.

[0081] See Figure 4Multiple pose markers (represented by the symbol "○" for corner points of pose marker patterns in this disclosure) and multiple angle markers (represented by the symbol "△" for corner points of angle marker patterns in this disclosure) are arranged side by side. The multiple pose marker patterns 411 may be identical or similar, and the corner points of the multiple pose marker patterns are located within the multiple pose marker patterns 411. The multiple angle marker patterns 421-426 may be different, and the corner points of the multiple angle marker patterns are located within the multiple angle marker patterns 421-426.

[0082] Each angle marker and one of the pose markers can have a positional association. For example, such as Figure 4 As shown, in the direction indicated by the arrow, some pose markers (e.g., pose marker pattern 411) and corresponding angle markers (e.g., angle marker pattern 421) are arranged along the arrow direction and have a spacing d1. See also Figure 5 In the circumferential setting state, label 400 becomes label 500 with a spatial structure of a cylinder. The positional association between each angle identifier and one of the pose identifiers can include the angle identifier and the pose identifier in the axial direction (e.g., Figure 5 The correspondence between the angle marker and the pose marker in the positive Z-axis direction. Based on the axial correspondence, given the known positions of one or more pose markers on the end of the manipulator, the area where the angle marker may exist can be determined by offsetting a certain distance (e.g., distance d1) along the axial direction. In some embodiments, the axial correspondence between the angle marker and the pose marker can be represented by the axial correspondence between the corner points of the angle marker pattern and the corner points of the pose marker pattern. In some embodiments, based on the axial correspondence between the angle marker and the pose marker, the projection of one of the corner points of the angle marker pattern and the corner point of the pose marker pattern along the Z-axis direction coincides.

[0083] In some embodiments, the about-axis angle or roll angle of the angle identifier or pose identifier can be represented by the about-axis angle of the corner point of the angle identifier pattern or the corner point of the pose identifier pattern. The corner point of the angle identifier pattern is relative to the driven tool coordinate system (e.g., a coordinate system established at the end of the manipulator, such as...). Figure 5 The angles of the XY coordinate system shown are known or predetermined, for example... Figure 5 In the XY coordinate system, the angle between corner point R5 of the angle marker pattern and the X-axis is θ. Based on the positional relationship, the angle between corner point P5 of the pose marker pattern associated with its position and the X-axis can be obtained as angle θ. It should be understood that the angle θ corresponding to corner point R5 of the angle marker pattern and corner point P5 of the pose marker pattern can be called the axial angle or roll angle of the angle marker or pose marker about the Z-axis. In this disclosure, the axial angle or roll angle refers to the angle about the Z-axis. It is understood that, for clarity, Figure 5 The corner point R5 of the angle marker pattern and the corner point P5 of the pose marker pattern are shown as separate, but they are overlapping.

[0084] Figure 6 A schematic diagram illustrating an implementation scenario 600 according to some embodiments of the present disclosure is shown. Figure 6 As shown, the driven tool includes a manipulator arm and an end effector 660 disposed at the end of the manipulator arm. Multiple pose markers and angle markers can be circumferentially disposed on the end effector 630 of the manipulator arm. For example, as... Figure 4 The label 400 shown is circumferentially disposed on the end of the operating arm 630, forming a cylindrical angle marking pattern strip 610 and a pose marking pattern strip 620. Multiple pose marking pattern corner points are distributed on the cross-sectional circle 621 of the pose marking pattern strip 620 at the end of the operating arm 630, and multiple angle marking pattern corner points are distributed on the cross-sectional circle 611 of the angle marking pattern strip 610 at the end of the operating arm 630.

[0085] In some embodiments, the multiple angle marker patterns are different patterns. Each angle marker pattern is used to indicate or identify a different angle about the axis. In some embodiments, there is a one-to-one correspondence between each angle marker pattern and the identified angle about the axis, and the identified angle about the axis can be determined based on the angle marker pattern.

[0086] For example, such as Figure 6 As shown, multiple different angle marking patterns (such as...) Figure 4 Multiple angle marker patterns (421-426) shown are evenly distributed circumferentially along the cylindrical structure, forming angle marker pattern corner points AF. The angle marker pattern corresponding to corner point A is designated as a reference pattern (e.g., the angle marker pattern corresponding to corner point A is used to mark a 0° angle around the axis). A plane coordinate system {wm1} is established. Then, based on the positional relationship between the remaining angle marker patterns and the angle marker pattern corresponding to corner point A, the angle around the axis marked by the corner points of the remaining angle marker patterns can be determined. For example, see... Figure 6 When the angle marker pattern corresponding to corner point B is identified, based on the positional relationship between the angle marker pattern corresponding to corner point B and the angle marker pattern corresponding to corner point A, the angle around the axis of corner point B in the two-dimensional plane coordinate system of section circle 611 can be determined to be 60°. The origin of the two-dimensional plane coordinate system of section circle 611 is the center of section circle 611, the X-axis direction is from the origin to corner point A of the angle marker pattern, and the Y-axis is perpendicular to the X-axis.

[0087] In some embodiments, the pose of the end effector 660 can be determined by translating a predetermined distance from the driven tool coordinate system {wm} (e.g., the end effector coordinate system). Alternatively, the pose of the end effector 660 can be approximately equal to the pose of the driven tool coordinate system {wm}.

[0088] In some embodiments, the pose of the end effector 660 relative to the reference coordinate system (e.g., the reference coordinate system is the world coordinate system {w}) is determined based on the pose of the driven tool coordinate system relative to the reference coordinate system. The specific calculation formula is as follows:

[0089] (13)

[0090] in, This refers to the attitude of the end effector relative to the world coordinate system. This refers to the position of the end effector relative to the world coordinate system. This refers to the attitude of the end effector relative to the world coordinate system. This refers to the position of the end effector relative to the world coordinate system. This refers to the attitude of the driven tool's coordinate system relative to the world coordinate system. This represents the position of the driven tool's coordinate system relative to the world coordinate system.

[0091] Some embodiments of this disclosure provide a method for controlling master-slave motion. Figure 7 A flowchart illustrating a master-slave motion control method 700 according to some embodiments of the present disclosure is shown. Method 700 can be used in robotic systems, such as… Figure 1 The robot system 100 shown or Figure 18 The surgical robot system 1800 is shown. (As shown...) Figure 7 As shown, some or all of the steps in method 700 can be executed by a control device (e.g., control device 120) of robot system 100. Control device 120 can be configured on a computing device. Method 700 can be implemented by software, firmware, and / or hardware. In some embodiments, method 700 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0092] See Figure 7 In step 701, a positioning image is acquired. In some embodiments, the positioning image includes a portion of the driven tool and multiple pose markers and at least one angle marker on the driven tool. For example, the positioning image includes a portion of the manipulator and multiple pose markers and at least one angle marker on the manipulator. In some embodiments, the positioning image can be obtained from, for example... Figure 1 The image acquisition device 110 shown receives a positioning image. For example, the control device 120 can receive a positioning image actively sent by the image acquisition device 110. Alternatively, the control device 120 can send an image request command to the image acquisition device 110, and the image acquisition device 110 responds to the image request command by sending a positioning image to the control device 120.

[0093] In step 703, the current pose of the driven tool relative to the reference coordinate system is determined based on the positioning image. In some embodiments, method 700 further includes identifying a plurality of pose identifiers located on the driven tool in the positioning image; and determining the current pose of the driven tool relative to the reference coordinate system based on the plurality of pose identifiers. In other embodiments, method 700 further includes identifying a plurality of pose identifiers and angle identifiers located on the driven tool in the positioning image, wherein the angle identifiers have a positional association with a first pose identifier among the plurality of pose identifiers. In some embodiments, an exemplary method for identifying a plurality of pose identifiers located on the driven tool may include, for example... Figure 11 and Figure 13 The method is illustrated. In some embodiments, the control device 120 can identify some or all of the pose markers in the positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm, which can extract or recognize features of the pose markers. For example, the image processing algorithm may include a corner detection algorithm for detecting corner points of the pose marker pattern. The corner detection algorithm may be one of, but not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the pose marker pattern. Another example is a contour detection algorithm for detecting contour features of the pose marker pattern. In some embodiments, the control device can identify some or all of the pose markers in the positioning image using a recognition model. In some embodiments, method 700 further includes identifying an angle marker located on the driven tool based on multiple pose markers. In some embodiments, after identifying multiple pose markers, an angle marker located on the driven tool is identified according to a positional association. In some embodiments, the positional association between the angle marker and the first pose marker may be as follows: Figure 4 or Figure 5 The positional relationships are shown in the diagram. In some embodiments, the first pose identifier (e.g., a first pose identifier pattern or a corner point of a first pose identifier pattern) refers to the pose identifier among a plurality of pose identifiers that has a positional relationship with the angle identifier. In some embodiments, an exemplary method for identifying the angle identifier includes, as shown in the diagram... Figure 15 The method shown.

[0094] In some embodiments, method 700 further includes determining the current pose of the driven tool relative to a reference coordinate system based on angle identifiers and multiple pose identifiers. In some embodiments, exemplary methods for determining the pose of the driven tool relative to a reference coordinate system include, for example... Figure 8 or Figure 10The method is illustrated. In some embodiments, the orientation of the slave tool relative to a reference coordinate system can be determined based on an angle identifier, a first pose identifier, and multiple pose identifiers. In some embodiments, the current orientation of the slave tool is the current orientation of the slave tool relative to its base coordinate system. The slave tool includes a manipulator and an end effector disposed at the end of the manipulator. The current orientation of the slave tool includes the orientation of the end effector relative to the base coordinate system of the slave tool or the orientation of the manipulator end relative to the base coordinate system of the slave tool. In some embodiments, the base coordinate system of the slave tool can be the coordinate system of the base on which the slave tool is mounted (e.g., the end effector of the surgical robot's manipulator), the coordinate system of the sheath through which the slave tool passes (e.g., the coordinate system of the sheath outlet), the coordinate system of the remote center of motion (RCM) of the slave tool, etc. For example, the base coordinate system of the slave tool can be set at the sheath outlet position and remains fixed during teleoperation. The current orientation of the end effector can be transformed to obtain its orientation relative to other coordinate systems. In some embodiments, the current pose of the slave tool is the current pose of the image of the slave tool on the display relative to the world coordinate system. In some embodiments, the world coordinate system may be the coordinate system of the space where the operator or the master operator is located. Therefore, the pose of the image of the slave tool on the display relative to the world coordinate system is the pose perceived by the operator. The slave tool includes surgical tools and vision tools. During surgery, the surgical tools perform surgery inside the patient's body, and the vision tool uses a camera to acquire images inside the patient's body and transmits the acquired images to the operating table. After the images are processed by the video processing module in the operating table, they are displayed on the display of the master control table. The operator obtains the current pose of the slave tool through the image on the display. In some embodiments, the current pose of the image of the slave tool on the display relative to the world coordinate system can be obtained through coordinate transformation. For example, the current pose of the image of the slave tool on the display relative to the world coordinate system can be obtained based on the base coordinate system of the slave tool, the coordinate system of the camera of the vision tool, the base coordinate system of the vision tool, the coordinate system of the display, and the world coordinate system.

[0095] Continue reading Figure 7 In step 705, the target pose of the master operator's handle is determined based on the current pose of the slave tool. In some embodiments, the current pose of the slave tool is its current pose relative to its base coordinate system, or the current pose of the slave tool's image on the display relative to the world coordinate system. The target pose of the master operator's handle is its pose relative to the master operator's base coordinate system. The master operator's base coordinate system may be the coordinate system of the base to which the master operator is connected. In some embodiments, the master operator's base coordinate system and the slave tool's base coordinate system have a defined transformation relationship.

[0096] In some embodiments, the current orientation of the slave tool is matched with the target orientation of the handle, for example, being the same, proportional, or having a fixed difference. For example, before teleoperation, the current orientation of the slave tool is kept unchanged and used as the target orientation of the handle, and the current orientation of the handle is adjusted to the target orientation to achieve orientation matching between the handle and the slave tool.

[0097] Continue reading Figure 7 In step 707, a control signal for the master controller is generated based on the target orientation of the master controller's handle. In some embodiments, method 700 further includes: determining the current orientation of the master controller's handle; and generating a control signal for the master controller based on the target orientation and the current orientation of the master controller's handle. The current orientation of the master controller's handle is the orientation of the master controller's handle relative to the master controller's base coordinate system. In some embodiments, the control signal corresponding to the handle reaching the target orientation from the current orientation is determined based on the handle's current orientation and the target orientation.

[0098] In some embodiments, method 700 further includes: determining the transformation relationship between the driven tool coordinate system and the pose identifier coordinate system based on angle identifiers and multiple pose identifiers. In some embodiments, according to the transformation relationship between the driven tool coordinate system and the pose identifier coordinate system, the three-dimensional coordinates in the pose identifier coordinate system can be converted into the corresponding three-dimensional coordinates in the driven tool coordinate system. In some embodiments, according to the transformation relationship between the driven tool coordinate system and the pose identifier coordinate system and the pose of the pose identifier coordinate system relative to the reference coordinate system, the attitude of the driven tool coordinate system relative to the reference coordinate system is obtained.

[0099] In some embodiments, the transformation relationship between the driven tool coordinate system and the pose identifier coordinate system may include the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system. In some embodiments, the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system may be determined based on the angle identifier and the first pose identifier. It should be understood that the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system may be the angle of rotation of the pose identifier coordinate system about the Z-axis of the driven tool coordinate system.

[0100] In some embodiments, the driven tool coordinate system may be a fixed coordinate system set on the manipulator arm of the driven tool based on multiple pose markers or multiple angle markers. In some embodiments, the Z-axis of the driven tool coordinate system is parallel to the axial direction of the manipulator arm, and the XY plane of the driven tool coordinate system is in the same plane as the corner points of the multiple pose marker patterns, or in the same plane as the corner points of the multiple angle marker patterns.

[0101] In some embodiments, a pose identifier coordinate system can be determined to facilitate the determination of the positions of multiple pose identifiers. In some embodiments, the position of the pose identifier can be represented by the position of the corner points of the pose identifier pattern. In some embodiments, the Z-axis of the pose identifier coordinate system is parallel to or coincides with the axis of the manipulator, and the XY plane of the pose identifier coordinate system is in the same plane as the corner points of the multiple pose identifier patterns.

[0102] For example, see Figure 6 Follower tool coordinate system The origin is the center of the cross-sectional circle 621 containing the corner points of multiple pose marker patterns. The X-axis points from the origin to one of the pose marker pattern corner points. The Z-axis is parallel to the axis of the end effector 630 of the operating arm, and the Y-axis is perpendicular to the XZ plane. The X-axis of the driven tool coordinate system {wm} is parallel to the two-dimensional plane coordinate system of the cross-sectional circle 611. The X-axis of the driven tool coordinate system is parallel to the Y-axis of the two-dimensional plane coordinate system {wm1} of the cross-section circle 611. The angle marking pattern corner point marked with an axis in the two-dimensional plane coordinate system {wm1} of the cross-section circle 611 can be equal to its axis marking angle in the driven tool coordinate system {wm}. Pose marking coordinate system The origin is the center of the cross-sectional circle 621 containing the corner points of multiple pose marker patterns. The X-axis points from the origin to one of the corner points of the pose marker pattern. The Z-axis is parallel to the axis of the end of the manipulator arm 630. The Y-axis is perpendicular to the XZ plane. (Continue reading...) Figure 6 The Z-axis of the driven tool coordinate system {wm} coincides with the Z-axis of the pose identifier coordinate system {wm0}. The transformation relationship between the driven tool coordinate system {wm} and the pose identifier coordinate system {wm0} can be determined by the roll angle of the pose identifier coordinate system {wm0} relative to the driven tool coordinate system {wm}. Confirmed. Roll angle. It can refer to the rotation angle of the pose identifier coordinate system {wm0} relative to the driven tool coordinate system {wm} around the Z-axis.

[0103] In some embodiments, see Figure 6 Roll angle Calculated using the following formula:

[0104] (14)

[0105] in The first angle around the axis, The second axis angle is the axis angle identified by the corner point of the angle marker pattern (e.g., corner point R6) in the driven tool coordinate system. The second axis angle is the axis angle identified by the corner point of the first pose marker pattern (e.g., pose marker pattern corner point P6) in the pose marker coordinate system.

[0106] Figure 8 A flowchart illustrating a method 800 for determining the attitude of a driven tool coordinate system relative to a reference coordinate system, according to some embodiments of the present disclosure. Figure 3 The actuator arm 300 shown may have a driven tool coordinate system that can be established at the end of the actuator arm. For example... Figure 8 As shown, some or all of the steps in method 800 can be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 800 can be implemented by software, firmware, and / or hardware. In some embodiments, method 800 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 18 The surgical robot system 1800 is shown. In some embodiments, method 800 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0107] See Figure 8 In step 801, based on the angle identifier and multiple pose identifiers, the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system is determined. In some embodiments, a first axis angle identified by the angle identifier in the driven tool coordinate system is determined. A second axis angle identified by the first pose identifier in the pose identifier coordinate system is determined. Based on the first axis angle and the second axis angle, the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system is determined. In some embodiments, the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system can be determined based on formula (14).

[0108] In step 803, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on multiple pose identifiers. The coordinates of the pose identifiers in the corresponding coordinate systems can be represented by the coordinates of the corner points of the pose identifier patterns in the corresponding coordinate systems. For example, the two-dimensional coordinates of the pose identifiers in the positioning image and the three-dimensional coordinates of the pose identifiers in the pose identifier coordinate system can be represented by the coordinates of the corner points of the pose identifier patterns. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image and the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image, the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system.

[0109] In some embodiments, the three-dimensional coordinates of the corner points of multiple pose marker patterns in the pose marker coordinate system are determined based on the distribution of multiple pose markers. For example, see... Figure 9 Each pose marker corner point is located on the circumference of cross-section circle 922, and the center and radius r of cross-section circle 922 are known. The center of cross-section circle 922 is set as the origin of the pose marker coordinate system. The XY plane lies on cross-section circle 922, and the X-axis can be specified as pointing from the origin to any known pose marker corner point (e.g., pose marker corner point P9). Therefore, based on the distribution of multiple pose markers, the three-dimensional coordinates of each pose marker corner point in the pose marker coordinate system can be determined. For example, as... Figure 9 As shown, the three-dimensional coordinates of corner point P9 of the pose marker pattern in the pose marker coordinate system are: The three-dimensional coordinates of the remaining pose marker corner points in the pose marker coordinate system can be calculated using the following formula:

[0110] (15)

[0111] in, Starting from the corner point P9 of the pose marker pattern, the first... The three-dimensional coordinates of the corner points of the pose marker pattern in the pose marker coordinate system; The angle around the axis between adjacent pose marker corner points.

[0112] In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system can be known. For example, the reference coordinate system is the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system can be determined based on the camera's pose. In other embodiments, the reference coordinate system can also be the camera coordinate system itself, depending on actual needs.

[0113] In some embodiments, based on the camera imaging principle and projection model, the pose of the pose marker coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of the corner points of multiple pose marker patterns in the positioning image and the three-dimensional coordinates of the corner points of multiple pose marker patterns in the pose marker coordinate system. Based on the transformation relationship between the pose of the pose marker coordinate system relative to the camera coordinate system and the camera coordinate system relative to the reference coordinate system, the pose of the pose marker coordinate system relative to the reference coordinate system can be obtained. In some embodiments, the camera's intrinsic parameters can also be considered. For example, the camera's intrinsic parameters can be as follows: Figure 1 The image acquisition device 110 shown or Figure 21 The image shows the camera intrinsic parameters of the imaging module 2160b. The camera intrinsic parameters can be known or obtained through calibration.

[0114] In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the optical center of the camera as the origin or a coordinate system established with the center of the camera lens as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, or the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of the line).

[0115] See Figure 8 In step 805, the attitude of the slave tool coordinate system relative to the reference coordinate system is determined based on the roll angle of the pose identifier coordinate system relative to the slave tool coordinate system and the attitude of the pose identifier coordinate system relative to the reference coordinate system. In some embodiments, the attitude of the slave tool coordinate system relative to the reference coordinate system can be used as the current attitude of the slave tool relative to the reference coordinate system.

[0116] Those skilled in the art will understand that some embodiments of this disclosure can also determine the pose of the slave tool coordinate system relative to the reference coordinate system based on the roll angle of the pose identifier coordinate system relative to the slave tool coordinate system and the pose of the pose identifier coordinate system relative to the reference coordinate system. For example, taking the world coordinate system as the reference coordinate system, the pose of the slave tool coordinate system relative to the world coordinate system is as follows:

[0117] (16)

[0118] in, This refers to the attitude of the driven tool's coordinate system relative to the world coordinate system. This represents the position of the driven tool's coordinate system relative to the world coordinate system. The pose of the position coordinate system relative to the world coordinate system. This represents the position of the pose coordinate system relative to the world coordinate system. This represents the roll angle about the Z-axis of the driven tool coordinate system. .

[0119] In some embodiments, the specific formula for calculating the pose of the driven tool coordinate system relative to the world coordinate system is as follows:

[0120] (17)

[0121] in, The pose of the camera coordinate system relative to the world coordinate system. This represents the position of the camera coordinate system relative to the world coordinate system. The pose identifier coordinate system is positioned relative to the camera coordinate system. The pose identifier coordinate system is positioned relative to the camera coordinate system. The attitude of the slave tool's coordinate system relative to the pose identifier coordinate system. The position of the relative pose identifier coordinate system of the slave tool coordinate system.

[0122] Figure 10 A flowchart illustrating a method 1000 for determining the attitude of a driven tool coordinate system relative to a reference coordinate system, according to other embodiments of this disclosure, is shown. Method 1000 may be... Figure 8 Alternative embodiments of method 800. For example... Figure 10 As shown, some or all of the steps in method 1000 can be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 1000 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1000 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 18 The surgical robot system 1800 is shown. In some embodiments, method 1000 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0123] See Figure 10 In step 1001, based on the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system and the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system, the three-dimensional coordinates of the multiple pose identifiers in the driven tool coordinate system are determined. It can be understood that, given the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system, the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system can be transformed into their three-dimensional coordinates in the driven tool coordinate system through coordinate transformation.

[0124] In step 1003, the pose of the slave tool coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple pose identifiers in the positioning image and the three-dimensional coordinates of the multiple pose identifiers in the slave tool coordinate system. In some embodiments, step 1003 can be implemented similarly to steps 803 and 805 in method 800.

[0125] Figure 11 A flowchart illustrating a method 1100 for identifying pose identifiers according to some embodiments of the present disclosure is shown. Figure 12 As shown, some or all of the steps in method 1100 can be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 1100 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1100 can be used in a robot system, such as... Figure 1 The robot system 100 shown or Figure 18The surgical robot system 1800 is shown. In some embodiments, method 1100 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0126] See Figure 11 In step 1101, multiple candidate pose identifiers are determined from the localization image. In some embodiments, candidate pose identifiers can be represented by candidate pose identifier pattern corner points. In some embodiments, candidate pose identifier pattern corner points can refer to possible pose identifier pattern corner points obtained after preliminary processing or preliminary identification of the localization image. In some embodiments, a Region of Interest (ROI) can be extracted from the localization image first, and multiple candidate pose identifiers can be determined from the ROI. The ROI can be the entire localization image or a partial region. For example, the ROI of the current frame can be extracted based on a certain range of multiple pose identifier pattern corner points determined in the previous frame image (e.g., the localization image of the previous image processing cycle). For localization images that are not the first frame, the ROI can be a region within a certain distance centered on a virtual point formed by the coordinates of multiple pose identifier pattern corner points from the previous image processing cycle. The certain distance range can be a fixed multiple of the average interval distance of the pose identifier pattern corner points, such as twice. It should be understood that the predetermined multiple can also be a variable multiple of the average interval distance of the multiple candidate pose identifier pattern corner points in the previous image processing cycle.

[0127] In some embodiments, method 1100 may include: determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of: image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include: cropping a Region of Interest (ROI) from the localization image and converting the ROI to a corresponding grayscale image.

[0128] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be calculated according to the following formula:

[0129] (18)

[0130] in, This is a set constant, for example, set to 2; , , , These are the first derivatives of the pixel in the four directions of 0, π / 4, π / 2, and -π / 4, respectively. and These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively.

[0131] In some embodiments, the Region of Interest (ROI) is divided into multiple sub-images. For example, a non-maximum suppression method can be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI can be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the location image or ROI can also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels. The pixel with the largest CL value in each sub-image can be determined, and the pixel with the largest CL value in each sub-image can be compared with a first threshold to determine the set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold can be set to 0.06. It should be understood that the first threshold can also be set to other values. In some embodiments, pixels with a CL value greater than the first threshold can be used as candidate pose identifier pattern corner points.

[0132] See Figure 11 In step 1103, an initial pose identifier is identified from multiple candidate pose identifiers based on a pose pattern matching template. In some embodiments, the pose pattern matching template is used to match the image at one of the corner points of the candidate pose identifier pattern to determine the corner point of the candidate pose identifier pattern that meets the preset pose pattern matching degree standard as the corner point of the initial pose identifier pattern.

[0133] In some embodiments, the pose pattern matching template and the image of the region near the corner point of the pose identifier pattern have the same or similar features. If the matching degree between the pose pattern matching template and the image of the region near the corner point of the candidate pose identifier pattern reaches a preset pose pattern matching degree standard (e.g., the matching degree is higher than a threshold), it can be considered that the pattern of the region near the corner point of the candidate pose identifier pattern has the same or similar features as the pose pattern matching template, and thus the current candidate pose identifier pattern corner point can be considered as the pose identifier pattern corner point.

[0134] In some embodiments, the pixel with the largest CL value in the pixel set is determined as the corner point of the candidate pose identifier pattern to be matched. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value is selected as the corner point of the candidate pose identifier pattern to be matched. After the corner point of the candidate pose identifier pattern to be matched is determined, the pose pattern matching template is used to match the pattern at the corner point of the candidate pose identifier pattern. If the preset pose pattern matching degree standard is met, the corner point of the candidate pose identifier pattern to be matched is determined as the identified initial pose identifier pattern corner point. If the corner point of the candidate pose identifier pattern to be matched does not meet the preset matching degree standard, the pixel with the second largest CL value is selected as the corner point of the candidate pose identifier pattern to be matched, and the pose pattern matching template is used to match the image at the corner point of the candidate pose identifier pattern. This process is repeated until the initial pose identifier pattern corner point is identified.

[0135] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, therefore the pose pattern matching template can be the same checkerboard pattern, utilizing the grayscale distribution of the pose pattern matching template. Pixel neighborhood grayscale distribution of pixels corresponding to corner points of candidate pose identifier patterns The correlation coefficient (CC) between pixels is used for matching. The grayscale distribution of the pixel neighborhood is also considered. This represents the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The specific formula is as follows:

[0136] (19)

[0137] Where Var is the variance function and Cov is the covariance function. In some embodiments, when the CC value is less than 0.8, the grayscale distribution in the pixel neighborhood has a low correlation with the pose pattern matching template. In this case, the candidate pose pattern corner with the highest likelihood value is determined to be the pose pattern corner. Otherwise, the candidate pose pattern corner with the highest likelihood value is considered to be the pose pattern corner.

[0138] In some embodiments, method 1100 includes: determining the edge orientation of corner points of candidate pose identifier patterns. For example, such as Figure 12 As shown, Figure 12 It includes a pose identifier pattern 1201, and the corner points of the candidate pose identifier pattern are... Figure 12 Corner point P in 12 Then the corner point P 12 The edge direction can refer to the corner point P. 12 The direction of the edge, such as Figure 12The direction indicated by the dashed arrow.

[0139] In some embodiments, the edge direction can be determined by the first-order derivative values ​​in the X and Y directions of the planar coordinate system for each pixel in a certain neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate pose pattern. and The direction of an edge can be determined using the following formula:

[0140] (20)

[0141] Among them, the first derivative ( and This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by convolving the edge directions of each pixel within its neighborhood range. and corresponding weights Clustering calculations are performed to obtain the edge direction of the pixel, and weights are selected. The class with the largest proportion As the edge direction. It should be noted that if multiple edge directions exist, a weight is selected. The largest proportion of the multiple categories corresponding to As the edge direction.

[0142] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0143] In some embodiments, method 1100 includes: rotating a pose pattern matching template according to an edge direction. Rotating the pose pattern matching template according to an edge direction can align the pose pattern matching template with the image at the corner point of a candidate pose identifier pattern.

[0144] The edge direction of the corner point of the candidate pose marker pattern can be used to determine the setting orientation of the image at the corner point of the candidate pose marker pattern in the positioning image. In some embodiments, rotating the pose pattern matching template according to the edge direction can adjust the pose pattern matching template to be the same as or nearly the same as the image orientation at the corner point of the candidate pose marker pattern to facilitate image matching.

[0145] See Figure 11 In step 1105, starting from the initial pose identifier, the pose identifier is searched. For example, Figure 13 A flowchart illustrating a method 1300 for searching pose identifiers according to some embodiments of the present disclosure is shown. Figure 13 As shown, some or all of the steps in method 1300 can be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 1300 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1300 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 18 The surgical robot system 1800 is shown. In some embodiments, method 1300 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0146] See Figure 13 In step 1301, a second pose identifier is determined, starting from the initial pose identifier. In some embodiments, the corner point of the initial pose identifier pattern is used as the starting point, and the corner point of the second pose identifier pattern is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following: directly in front of the corner point of the initial pose identifier pattern (corresponding to the 0° angle direction), directly behind (corresponding to the 180° angle direction), directly above (90° angle direction), directly below (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0147] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions, with each search direction v sn It can be calculated using the following formula:

[0148] (twenty one)

[0149] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent pose marker corner points among the multiple pose marker corner points determined in the previous frame. For example, the predetermined search direction can be calculated using the following formula:

[0150] (twenty two)

[0151] in, Two-dimensional coordinates of the corner points of multiple pose marker patterns determined in the previous frame (or the previous image processing cycle); The number of corner points of the multiple pose marker patterns determined in the previous frame; This is the first search direction set. This is the second search direction set.

[0152] In some embodiments, such as Figure 14 As shown, the corner point P of the pattern is identified by its initial pose. 1401 Using the coordinates of the given location as the starting point, search for the second pose marker corner point P in the set search direction. 1402 Specifically, the coordinate position can include: identifying the corner point P of the pattern in the initial pose. 1401 Using the coordinates as the starting point for the search, the search box (e.g., ...) Figure 14 The dashed box in the image (within the image) moves in the set search direction V with a certain search step size. 1401 Search for the corner points of the pose marker pattern. If there is at least one candidate corner point of the pose marker pattern within the search box, then the candidate corner point with the highest likelihood value within the search box is selected as the second corner point of the pose marker pattern, P. 1402 With the search box constrained to a suitable size, the pattern corner point P is identified in its initial pose. 1401 The coordinates of the point are used as the starting point for the second pose identification pattern corner point P. 1402 During the search, the candidate pose marker corner with the highest corner likelihood value among the candidate pose marker corners appearing in the search box is more likely to be the actual pose marker corner. Therefore, it can be considered that the candidate pose marker corner with the highest corner likelihood value in the search box is the second pose marker corner P. 1402 To improve data processing speed. In other embodiments, to improve the accuracy of pose marker pattern corner point recognition, when at least one candidate pose marker pattern corner point exists in the search box, the candidate pose marker pattern corner point with the highest corner point likelihood value among the candidate pose marker pattern corner points appearing in the search box is selected for corner point recognition to determine whether the candidate pose marker pattern corner point with the highest corner point likelihood value is a pose marker pattern corner point. For example, the pose pattern matching template is matched with the image within a certain range of the candidate pose marker pattern corner point with the highest corner point likelihood value. The candidate pose marker pattern corner point that meets the preset pose pattern matching degree standard can be considered as the second pose marker pattern corner point P found. 1402 .

[0153] In some embodiments, continue reading Figure 14 The size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0154] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, and the correlation coefficient CC in formula (19) can be used for pattern matching. If CC is greater than the threshold, the candidate pose identification pattern corner with the largest corner likelihood value is considered to be the pose identification pattern corner, and is recorded as the second pose identification pattern corner.

[0155] See Figure 13 In step 1303, a search direction is determined based on the initial pose identifier and the second pose identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of a corner point of the initial pose identifier pattern and moving away from the corner point of the second pose identifier pattern. The second search direction may be a direction starting from the coordinate position of a corner point of the second pose identifier pattern and moving away from the corner point of the first pose identifier pattern. For example, Figure 14 The search direction V shown 1402 .

[0156] In step 1305, starting from the initial pose identifier or the second pose identifier, a search for pose identifiers is performed in the search direction. In some embodiments, if the first pose identifier pattern corner point is used as the new starting point, the first search direction described above can be used as the search direction for the pose identifier pattern corner point. If the second pose identifier pattern corner point is used as the new starting point, the second search direction described above can be used as the search direction for the pose identifier pattern corner point. In some embodiments, a new pose identifier pattern corner point is searched (e.g., Figure 14 The third pose marker pattern corner point P in 1403 This can be performed similarly to step 1301. In some embodiments, the search step size can be the distance L1 between the corner points of the initial pose identifier pattern and the corner points of the second pose identifier pattern.

[0157] In some embodiments, the search for pose identifier pattern corner points is stopped in response to the number of corner points found being greater than or equal to a pose identifier pattern corner point number threshold. For example, the search for pose identifier pattern corner points is stopped when four pose identifier pattern corner points are found (identified).

[0158] In some embodiments, the search for the Nth pose marker corner point is stopped when the search distance is greater than a set multiple of the distance between the (N-1)th and (N-2)th pose marker corner points, where N ≥ 3. For example, the search termination condition could be that the search distance is greater than twice the distance between the first two pose marker corner points. Thus, the maximum search distance for the third pose marker corner point is twice the distance between the initial pose marker corner point and the second pose marker corner point. If no pose marker corner point is found after reaching this search distance, it is considered that the third pose marker corner point has not been found and the search ends.

[0159] In some embodiments, if the total number of pose marker corner points found is greater than or equal to a set threshold (e.g., the set threshold is 4), then it is considered that enough pose marker corner points have been successfully identified. If the total number of pose marker corner points found is less than the set value, then the search based on the initial pose marker corner point in the above steps is considered unsuccessful. In the case of unsuccessful search, a new initial pose marker corner point is determined from the candidate pose marker corner points, and then the remaining pose marker corner points are searched based on the newly determined initial pose marker corner point as the search starting point. Similar to method 1100, a new initial pose marker corner point can be determined, and similar to method 1300, the remaining pose marker corner points can be searched based on the new pose marker corner point as the search starting point.

[0160] In some embodiments, after the corner points of the pose marker pattern are searched or identified, sub-pixel positioning can be performed on the determined corner points of the pose marker pattern to improve the positional accuracy of the corner points of the pose marker pattern.

[0161] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of the corner points of the pose identifier pattern after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, where the extreme points of the function are subpixel points. The fitting function can be as follows:

[0162] (twenty three)

[0163] (twenty four)

[0164] in, The function is used to fit the CL values ​​of all pixels in each ROI, with a, b, c, d, e, and f as coefficients. The x-coordinate is the pose identifier. The y-coordinate is the pose identifier.

[0165] Figure 15A flowchart illustrating a method 1500 for identifying angle markers according to some embodiments of the present disclosure is shown. Figure 15 As shown, some or all of the steps in method 1500 can be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 1500 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1500 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 18 The surgical robot system 1800 is shown. In some embodiments, method 1500 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0166] See Figure 15 In step 1501, an imaging transformation relationship is determined based on the two-dimensional coordinates of multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system. In some embodiments, the pose marker coordinate system may be the pose marker coordinate system detailed in the embodiment shown in method 700. For example, the pose marker coordinate system is as follows: Figure 6 As shown. In some embodiments, the imaging transformation relationship can refer to the transformation relationship between the three-dimensional coordinates in the pose identifier coordinate system and the two-dimensional coordinates in the positioning image. It should be understood that, based on the imaging transformation relationship, the two-dimensional coordinates in the positioning image can also be transformed into the three-dimensional coordinates in the pose identifier coordinate system. In some embodiments, the three-dimensional coordinates of multiple pose identifiers in the pose identifier coordinate system can be determined based on formula (15). In some embodiments, the number of multiple pose identifiers can be greater than or equal to four. For example, the imaging transformation relationship can be obtained based on the two-dimensional coordinates of four pose identifiers in the positioning image and the four corresponding three-dimensional coordinates in the pose identifier coordinate system.

[0167] See Figure 15 In step 1503, based on the imaging transformation relationship, the three-dimensional coordinates and positional relationships of multiple pose markers in the pose marker coordinate system, multiple candidate regions for angle markers are determined in the positioning image. In some embodiments, the candidate regions for angle markers may represent candidate regions for angle marker patterns. In some embodiments, based on the three-dimensional coordinates and positional relationships of the corner points of multiple pose marker patterns in the pose marker coordinate system, multiple candidate three-dimensional coordinates of the corner points of multiple angle marker patterns are determined in the pose marker coordinate system. For example, based on the three-dimensional coordinates of the corner points of multiple pose marker patterns in the pose marker coordinate system, a certain distance can be offset along the axial direction to determine multiple three-dimensional coordinates in the pose marker coordinate system. These three-dimensional coordinates are represented by the candidate three-dimensional coordinates of the corner points of multiple angle marker patterns. For example, see... Figure 4The positional relationship is that the angle marker and the corresponding pose marker are spaced a certain distance along the Z-axis of the pose marker coordinate system. Given the position of the corner point of the pose marker pattern, the position obtained by moving a certain distance along the positive or negative direction of the Z-axis can be considered as the candidate position of the corner point of the angle marker pattern in the pose marker coordinate system.

[0168] In some embodiments, multiple candidate regions for angle markers are determined in the positioning image based on the imaging transformation relationship and the three-dimensional coordinates of multiple candidate corner points for angle marker patterns. For example, multiple two-dimensional coordinates of candidate corner points for angle marker patterns are obtained in the positioning image based on the imaging transformation relationship and the three-dimensional coordinates of multiple candidate corner points for angle marker patterns. In some embodiments, multiple candidate regions for angle marker patterns are determined based on the two-dimensional coordinates of multiple candidate corner points for angle marker patterns. For example, a region of a certain size (e.g., 5×5 pixels, 10×10 pixels, etc.) is determined in the positioning image centered on each candidate corner point coordinate of an angle marker pattern as a candidate region for angle markers. In some embodiments, the region of a certain size is greater than or equal to the size of the angle marker pattern after imaging. The size of the angle marker pattern after imaging can be obtained based on the actual size of the angle marker pattern and the imaging transformation relationship.

[0169] See Figure 15 In step 1505, candidate regions are identified from multiple angles to identify angle markers. In some embodiments, the angle marker includes an angle marker pattern and corner points of the angle marker pattern. In some embodiments, method 1500 may include determining the pixel with the largest corner likelihood value in each candidate region of the angle marker to form a pixel set. In some embodiments, the corner likelihood value of the pixel may be calculated when performing method 1100, or it may be recalculated based on formula (18). Method 1500 also includes determining the candidate region of the angle marker corresponding to the pixel with the largest corner likelihood value in the pixel set as the candidate region of the angle marker to be identified. Method 1500 also includes matching the candidate region of the angle marker to be identified with multiple angle pattern matching templates respectively to identify the angle marker. In some embodiments, the angle marker pattern is a pattern with different graphic features. Multiple angle pattern matching templates may refer to standard angle pattern templates with the same or similar graphic features corresponding to the multiple angle marker patterns respectively. In some embodiments, by determining multiple candidate regions of the angle marker, the angle marker can be identified in multiple candidate regions of the angle marker, avoiding the identification of the angle marker in the entire image range and improving the speed of data processing.

[0170] In some embodiments, any one of the template matching algorithms, such as the squared difference matching method, normalized squared difference matching method, correlation matching method, normalized correlation matching method, correlation coefficient matching method, and normalized correlation coefficient matching method, can be used to perform matching operations between the angle pattern matching template and the angle identifier candidate region.

[0171] In some embodiments, since the angle pattern matching template and the angle marker pattern have the same or similar graphic features, the pattern information of the angle marker may include the pattern information of the corresponding angle pattern matching template. For example, the shape of the angle pattern matching template, identifiable features in the image, etc. In some embodiments, each angle pattern matching template has a one-to-one correspondence with the axial angle identified by the corresponding angle marker pattern. The first axial angle is determined based on a specific angle pattern matching template or the pattern information of the angle marker pattern corresponding to the identified angle marker.

[0172] In some embodiments, method 1500 may include, in response to a matching failure, determining the candidate region of the angle identifier corresponding to the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as the candidate region of the angle identifier to be identified. In some embodiments, after determining the new candidate region of the angle identifier to be identified, multiple angle pattern matching templates are used to match the candidate region of the angle identifier to be identified, respectively, to identify the angle identifier.

[0173] In some embodiments, a first pose marker that has a positional relationship with the angle marker is determined based on the angle marker candidate region where the identified angle marker is located. In some embodiments, the multiple angle marker candidate regions respectively correspond to at least one of the multiple identified pose marker pattern corner points. After determining the angle marker candidate region where the identified angle marker is located, the first pose marker pattern corner point can be determined based on the correspondence between the multiple angle marker candidate regions and the multiple pose marker pattern corner points.

[0174] Figure 16 A schematic diagram of a master operator 1600 according to some embodiments of this disclosure is shown. For example... Figure 16 As shown, in some embodiments, the main manipulator 1600 includes a multi-degree-of-freedom robotic arm 1610 and a handle 1620. The multi-degree-of-freedom robotic arm 1610 includes multiple joints (16101-16107). The joints of the multi-degree-of-freedom robotic arm 1610 include position joints and attitude joints. The attitude joints serve as the orientation module of the main manipulator 1600, controlling the handle 1620 to achieve a target attitude through one or more attitude joints. The position joints serve as the positioning module of the main manipulator 1600, controlling the handle 1620 to achieve a target position through one or more position joints.

[0175] In some embodiments, determining the current orientation of the handle of the master operator includes: obtaining joint information of at least one orientation joint; and determining the current orientation of the master operator based on the joint information of at least one orientation joint.

[0176] In some embodiments, master manipulator sensors are disposed at the attitude joints of a multi-degree-of-freedom robotic arm to acquire joint information (such as angles) corresponding to the attitude joints, and to determine the current attitude of the master manipulator's handle relative to the master manipulator's base coordinate system based on the acquired joint information. In some embodiments, joint information is acquired through the master manipulator sensors at the attitude joints, and the current attitude of the master manipulator is calculated based on a forward kinematics algorithm. In some embodiments, the master manipulator includes at least one attitude joint for controlling the attitude of the master manipulator's handle, and the control signal includes a control signal for controlling one or more of the at least one attitude joint. By adjusting one or more attitude joints, the attitude of the master manipulator's handle is adjusted, achieving attitude matching between the master manipulator's handle and the driven tool.

[0177] In some embodiments, the control signals include control signals for controlling one or more of at least one attitude joint, wherein one or more of the at least one attitude joint includes uncoupled attitude joints. Coupled joints may refer to joints used to adjust the position and attitude of the master manipulator. Uncoupled joints may refer to joints that can only be used to adjust the position (referred to in this disclosure as uncoupled position joints) or attitude (referred to in this disclosure as uncoupled attitude joints). In some embodiments, the master manipulator may include at least one coupled joint. For example, Figure 16 The master manipulator 1600 shown has a first joint 16101, a second joint 16102, and a third joint 16103 that are position joints, and a first joint 16101, a second joint 16102, a fifth joint 16105, a sixth joint 16106, and a seventh joint 16107 that are attitude joints. The first joint 16101 and the second joint 16102 are coupled joints that can adjust both the position and attitude of the master manipulator 1600, while the fifth joint 16105, the sixth joint 16106, and the seventh joint 16107 are uncoupled attitude joints that can only adjust the attitude of the master manipulator 1600. In some embodiments, the attitude adjustment of the handle 1620 of the master operator 1600 can be achieved by calculating the control signals of the uncoupled attitude joints (e.g., the fifth joint 16105, the sixth joint 16106, and the seventh joint 16107), thereby achieving attitude matching between the handle 1620 of the master operator 1600 and the driven tool, and providing conditions for subsequent teleoperation.

[0178] In some embodiments, the base coordinate system of the main actuator is b, and the coordinate system of the handle is d. In some embodiments, the base coordinate system b is a coordinate system established with the base as a virtual point, and its orientation can be determined based on its physical structure. Similarly, the coordinate system d of the handle is a coordinate system established with the handle as a virtual point, and its orientation can be determined based on its physical structure. In some embodiments, the origin of the coordinate system d of the handle can coincide with the origin of the coordinate systems of the fifth, sixth, and seventh joints. Those skilled in the art will understand that the position and orientation of the coordinate system d of the handle relative to the base coordinate system of the main actuator can be determined by the joint information of the first to seventh joints.

[0179] In some embodiments, the master actuator sensor acquires joint information of the master actuator. (j is the joint number). In some embodiments, the information of the j-th joint... This can include the angle values ​​of the corresponding joints. For example, obtaining joint information of the first joint. Joint information of the second joint Joint information of the third joint Joint information of the fourth joint Joint information of the fifth joint Joint information of the sixth joint Joint information of the seventh joint In some embodiments, the fourth joint is a driven joint of the third joint, and the absolute value of the joint angle of the fourth joint is the same as that of the joint angle of the third joint, but the direction is opposite. Therefore, the angles of the six joints of the master manipulator are represented by a 6*1 matrix, and the joint angle of the fourth joint may not be in the matrix. This is reflected in the information of each joint. Can Represented as The main manipulator has six degrees of freedom, as shown in formula (25):

[0180] (25)

[0181] The first, second, and third joints are position joints. , , This determines the position of the main manipulator's handle. The first, second, fifth, sixth, and seventh joints are attitude joints. , , , , The orientation of the handle is determined. In some embodiments, determining the orientation of the main controller's handle may disregard the positions controlled by the first, second, and third joints, and focus instead on the orientation (e.g., direction) determined by the first, second, fifth, sixth, and seventh joints. In some embodiments, during motor drive, the first, second, and third joints are kept stationary, and the fifth, sixth, and seventh joints are determined based on the target pose or target orientation. , , ,according to , , Calculate control signals to adjust the handle's posture.

[0182] Those skilled in the art will understand that there are many solutions for a multi-joint master manipulator to achieve a certain target posture. In some embodiments, the posture of the master manipulator handle can be adjusted by adjusting one or more of the at least one posture joint. For example, in one embodiment, the posture of the master manipulator handle can be adjusted by keeping the coupled posture first joint, coupled posture second joint, and uncoupled posture third joint unchanged, and adjusting the uncoupled posture fifth joint, sixth joint, and seventh joint.

[0183] In some embodiments, method 700 further includes: obtaining joint information of other posture joints besides the one or more posture joints to be adjusted in at least one posture joint; and determining the transformation matrix of the other posture joints based on the joint information of the other posture joints. For example, the joint information of the other posture joints is obtained based on the master manipulator sensor, and the transformation matrix of the other posture joints is determined based on the joint information of the other posture joints. The joint information of the coupled posture first joint and second joint can be obtained, and the transformation matrix is ​​calculated. In some embodiments, one or more uncoupled posture joints (e.g., the fifth joint, the sixth joint, and the seventh joint) can be adjusted without adjusting other posture joints, such as coupled joints (e.g., the first joint and the second joint). In some embodiments, the transformation matrix can be determined based on the joint information of the first joint and the second joint. as well as Determine the transformation matrices for other pose joints (e.g., the transformation matrices for other pose joints relative to joint origin 0). In some embodiments, method 700 further includes generating the control signal for the master manipulator based on the transformation matrix of the target pose of the master manipulator's handle and the other pose joints. For example, the transformation matrix of the target pose of the master manipulator's handle and the other pose joints... Generate control signals for the master operator, as shown in formulas (26) to (28).

[0184] In some embodiments, the third and fourth joints are uncoupled position joints, based on , , Transformation matrices for other pose joints With based , Transformation matrices for other pose joints Consistent.

[0185] (26)

[0186] In formula (26), the transformation matrix From input , or , , Let b be the base coordinate system of the master operator, and d be the coordinate system of the master operator's handle. The attitude of the master controller handle relative to the master controller base coordinate system. b R0 represents the existing angular relationship between the base and the joint starting point, and is a structural constant. The existing angular relationship between the seventh joint and the handle is a structural constant.

[0187] (27)

[0188] (28)

[0189] In formula (28), The current posture of the driven tool and its relationship with same, , and Corresponding to the quantities to be solved , , Based on the obtained , , A control signal is determined, and the attitude of the master operator is adjusted based on the control signal to achieve master-slave attitude matching. Those skilled in the art will understand that... It can be the current pose of the end effector of the driven tool relative to the base coordinate system of the driven tool, or the current pose of the image of the end effector of the driven tool on the display relative to the world coordinate system. Can be with Consistent, for example, identical or having a specific ratio or difference. In some embodiments, the joint target values ​​of one or more posture joints of the handle are determined according to the control signal, and the joint target values ​​are converted into drive quantities and sent to the drive device. The drive device drives the motors of one or more posture joints of the master manipulator to move, thereby achieving the matching of the posture of the handle of the master manipulator with the posture of the end effector of the driven tool.

[0190] In some embodiments, the mathematical structure model of the master operator can be constructed based on the DH parameter method or the exponential product representation method. For example, the DH matrix corresponding to the joints of the master operator is determined, and the mathematical structure model of the master operator is determined based on the DH matrix of the joints. The DH matrix of each joint of the master operator is represented by formula (29).

[0191] (29)

[0192] The correspondence between the DH matrix and joint information is shown in Table 1.

[0193] Table 1. Correspondence between D-H matrix and joint information.

[0194]

[0195] In formula (29), Rotation about the x-axis angle, Rotation about the z-axis angle, Move in the x direction , Move in the z direction .like Figure 16 The main manipulator 1600 shown has its z-axis as the rotation axis of the joint, its x-axis pointing to the next joint, and its y-axis direction can be determined according to the left / right-hand rule of the Cartesian coordinate system. , A fourth-order matrix represents a rotation around a direction by a certain angle or a translation along a direction by a certain distance.

[0196] In some embodiments, the mathematical structural model of the master operator is described by multiplying the DH matrices of all joints, as shown in formula (30):

[0197] (30)

[0198] In some embodiments, the DH matrix of the joint in formula (30) can be determined based on formula (29).

[0199] Those skilled in the art will understand that when initiating teleoperation, if the handle's orientation (e.g., direction or angle) is inconsistent with the orientation (e.g., direction or angle) of the corresponding controlled slave tool, the operator's (e.g., surgeon's) human-computer interaction experience during operation will be poor, affecting the accuracy of the slave tool's operation. Therefore, after the master operator and slave tool are matched and connected, and before the master operator performs teleoperation on the slave tool (e.g., when the operator holds the master operator's handle to gain control of the corresponding slave tool but has not yet started master-slave teleoperation), the handle's orientation is matched and adjusted with the slave tool's orientation. When both orientations are consistent, the master operator can perform teleoperation on the slave tool, improving the accuracy and experience of subsequent teleoperations.

[0200] In some embodiments, method 700 further includes: determining the attitude matching degree between the handle of the master operator and the driven tool in response to the satisfaction of a predetermined condition. In some embodiments, the predetermined condition includes the triggering of remote operation control. In some embodiments, the triggering of remote operation control can be achieved through a triggering device. The triggering device may be a switch located on the master operator or display for easy access, touch, press, or slide by the operator. Triggering methods include, but are not limited to, maintaining proximity, touching, sliding, tapping, or long-pressing. The triggering method of the triggering device may be proximity to a sensor, flipping a switch on the master operator, touching a sensing position on the master operator, long-pressing or tapping a button on the master operator, stepping on the foot pedal of the main control panel, operating the display screen of the main control panel, etc. In some embodiments, matching means that the attitude of the handle and the attitude of the driven tool meet a preset relationship (e.g., consistency), and the attitude matching degree refers to the degree of matching between the current attitude of the handle and the current attitude of the driven tool. In some embodiments, the attitude matching degree between the master operator and the driven tool is determined based on the current attitude of the handle of the master operator and the current attitude of the driven tool. When the attitude matching degree is lower than a preset threshold, a control signal is generated to adjust the current attitude of the master controller's handle so that the attitude matching degree is higher than or equal to the preset threshold. This allows for automatic attitude adjustment to achieve consistency between the two attitudes when they do not match. When the current attitudes of the two are consistent or nearly consistent (attitude matching degree higher than or equal to the preset threshold), a master-slave mapping is established between the master controller and the slave tool, allowing the next teleoperation procedure to be executed.

[0201] In some embodiments, adjusting the orientation of the master controller's handle to match the orientation of the slave tool includes: keeping the current orientation of the slave tool unchanged, and adjusting the orientation of the master controller's handle to make the orientation of the master controller's handle match the orientation of the slave tool.

[0202] In some embodiments, the target posture of the master controller's handle is consistent with the current posture of the slave tool, establishing a master-slave mapping between the master controller and the slave tool. This allows the master controller to perform teleoperations on the slave tool, improving the accuracy and user experience of teleoperation. Those skilled in the art will understand that posture consistency means that the postures are essentially the same. There may be some error between the target posture of the master controller's handle and the current posture of the slave tool, but the error must be within an acceptable range.

[0203] In some of the above embodiments, the orientation of the handle is matched with the orientation of the slave tool before teleoperation. When the operator begins operation (e.g., pressing the clamp button on the master operator's handle), the master-slave mapping can be quickly established, and the master operator and the slave tool enter teleoperation mode. Furthermore, only the current orientation of the slave tool is maintained; the operator can still move the master operator's handle to a suitable position before performing teleoperation matching even when not in operation, greatly increasing the movement range of the master operator's handle. Moreover, the master-slave motion control method provided above is applicable to various slave ends with different principles and forms, and the calculation process is highly targeted and computationally inefficient, also reducing the driving force required to adjust the master operator's handle to the target orientation.

[0204] In some of the above embodiments, by establishing a connection between the master operator and the slave tool and transferring control, the attitude matching degree between the master operator's handle and the slave tool is determined under the connected and control-transferred state. If the attitude matching degree meets a preset threshold condition, a master-slave mapping is established between the master operator and the slave tool, and teleoperation steps are executed. If the attitude matching degree does not meet the preset threshold condition, the attitude of the master operator's handle needs to be adjusted to match the current attitude of the slave tool before establishing the master-slave mapping between the master operator and the slave tool, and teleoperation is executed through the master operator's handle. Adjusting the attitude of the master operator's handle to match the attitude of the slave tool before establishing the teleoperation relationship ensures the accuracy of the master-slave mapping between the master operator's handle and the slave tool, improves the operator's experience during teleoperation, achieves high-precision matching between the operational actions and actual actions, and avoids operational limitations caused by inconsistent motion control boundaries between the master operator and the slave tool.

[0205] In some of the above embodiments, when the controlled object of the master manipulator (e.g., the slave tool) is changed, the orientation of the front end of the slave tool entering the machine may be different from the current orientation of the handle of the master manipulator. The method provided in this disclosure can adjust the posture of the handle of the master manipulator to match the current posture of the slave tool before the master manipulator and the slave tool establish a master-slave mapping relationship and before the operator actually operates, so as to achieve a good operating experience for the operator and a high-precision match between the expected action and the actual action, while avoiding the operational limitations caused by the inconsistency of the motion control boundaries between the master manipulator and the slave tool.

[0206] In some embodiments of this disclosure, a computer device is also provided, including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 7 , Figure 8 , Figure 10 , Figure 11 , Figure 13 and Figure 15 Some or all of the steps in the method disclosed herein.

[0207] Figure 17 A schematic block diagram of a computer device 1700 according to some embodiments of the present disclosure is shown. See also Figure 17 The computer device 1700 may include a central processing unit (CPU) 1701, a system memory 1704 including random access memory (RAM) 1702 and read-only memory (ROM) 1703, and a system bus 1705 connecting the various components. The computer device 1700 may also include input / output devices 1706 and a mass storage device 1707 for storing the operating system 1713, application programs 1714, and other program modules 1715. The input / output device 1706 includes an input / output controller 1710, primarily composed of a display 1708 and input devices 1709.

[0208] Mass storage device 1707 is connected to central processing unit 1701 via a mass storage controller (not shown) connected to system bus 1705. Mass storage device 1707 or computer-readable media provides non-volatile storage for computer devices. Mass storage device 1707 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0209] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include read-only memory, random access memory, flash memory or other solid-state storage technologies, read-only optical discs or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0210] Computer device 1700 can be connected to network 1712 via network interface unit 1711 connected to system bus 1705.

[0211] The system memory 1704 or mass storage device 1707 is also used to store one or more instructions. The central processing unit 1701 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0212] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 7 , Figure 8 , Figure 10 , Figure 11 , Figure 13 and Figure 15 Some or all of the steps in the disclosed method. Examples of computer-readable storage media include memory for computer programs (instructions), such as read-only memory, random access memory, read-only optical disk, magnetic tape, floppy disk, and optical data storage devices.

[0213] Figure 18 A schematic diagram of a surgical robot system 1800 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 18 The surgical robot system 1800 may include a surgical tool 1801, a main control carriage 1802, and a surgical carriage 1803. A drive module is mounted on the surgical carriage 1803 to drive the surgical tool 1801, which is mounted on the surgical carriage 1803 and connected to the drive module. The main control carriage 1802 is communicatively connected to the surgical carriage 1803 and is used to control the surgical tool 1801 to perform surgical operations. In some embodiments, the controller in the main control carriage 1802 or the controller in the surgical carriage 1803 may be used to execute some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 7 , Figure 8 , Figure 10 , Figure 11 , Figure 13 and Figure 15 Some or all of the steps in the disclosed method. In some embodiments, the main control carriage 1802 and the operating carriage 1803 are connected by wired or wireless transmission. For example, the main control carriage 1802 and the operating carriage 1803 can be connected by a cable.

[0214] In some embodiments, the surgical tool 1801 includes a manipulator arm and an end effector disposed at the end of the manipulator arm. In some embodiments, the surgical robot system 1800 may include a surgical cart 1803. In some embodiments, the surgical robot system 1800 may include at least two surgical carts 1803, each surgical cart 1803 mounting a surgical tool 1801. In some embodiments, the surgical robot system 1800 may further include an imaging tool 1804. The imaging tool 1804 may include a manipulator arm and an imaging module disposed at the end of the manipulator arm. The imaging tool 1804 may be disposed on the surgical cart 1803 and driven by a corresponding drive module. Images of the manipulator arm and its end effector of the surgical tool 1801 acquired by the imaging module may be transmitted to the master control cart 1802. In some embodiments, a portion of the surgical tool 1801 or a portion of the imaging tool 1804 may serve as a driven tool. In some embodiments, the master control cart 1802 includes a master manipulator for remotely operating the surgical tool 1801 or the imaging tool 1804. In some embodiments, the surgical tool 1801 is, for example, a Figure 19 The surgical instrument 1900 is shown in the image. In some embodiments, the main control carriage 1802 is, for example, a surgical instrument 1900. Figure 20 The main control carriage 2000 is shown in the figure. In some embodiments, the surgical carriage 1803 is, for example, a Figure 21 The surgical cart 2100 shown in the image.

[0215] Figure 19 A schematic diagram of a surgical instrument 1900 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 19The surgical tool 1900 includes a drive transmission device 1990, an operating arm 1940, and an end effector 1960 disposed at the end of the operating arm. In some embodiments, the drive transmission device 1990 can cooperate with a drive module to drive the operating arm 1940 to move. The drive transmission device 1990 is used to connect to the drive module, and the driving force of the drive module is transmitted to the operating arm 1940 through the drive transmission device 1990, thereby driving the operating arm 1940 to achieve multi-degree-of-freedom movement. The drive module can also control the end effector 1960 to perform surgical operations. In some embodiments of this disclosure, the end effector 1960 may include, but is not limited to, a bipolar curved dissecting forceps actuator, a bipolar curved grasping forceps actuator, a unipolar curved scissors actuator, a unipolar electric hook actuator, a bipolar grasping forceps actuator, a needle holder actuator, and a tissue grasping forceps actuator. In some embodiments, the surgical tool 1900 may be, for example, mounted on... Figure 18 The surgical cart 1803 shown in the image or Figure 21 The surgical cart 2100 shown in the image.

[0216] Figure 20 A schematic diagram of a main control carriage 2000 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 20 The main control carriage 2000 includes: a controller (which can be configured on a computer device and is located inside the main control carriage 2000), a main operator 2001, a main control carriage display (e.g., displays 2002-2004), and pedals (e.g., pedals 2005-2007). The controller is communicatively connected to the main operator 2001, the main control carriage display, and the pedals, respectively, for signal interaction with the main operator 2001, the main control carriage display, and the pedals, and for generating corresponding control commands based on collected control information. In some embodiments, the controller is also communicatively connected to a surgical carriage, for example, with... Figure 18 The surgical cart 1803 shown is connected for communication and is used to control the surgical instrument 1801 to perform surgical operations or to control the imaging instrument 1804 to operate. In some embodiments, the controller of the main control cart 2000 can also be used to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 7 , Figure 8 , Figure 10 , Figure 11 , Figure 13 and Figure 15 Some or all of the steps in the method disclosed herein.

[0217] In some embodiments, the master manipulator 2001 typically includes a left master manipulator (e.g., for controlling the first manipulator arm) for operation with the left hand of the medical worker and a right master manipulator (e.g., for controlling the second manipulator arm) for operation with the right hand. In practical scenarios, the master manipulator 2001 is used to collect the operation input of the medical worker, who then remotely operates the master manipulator 2001 to control the movement of surgical or imaging tools within the operating area to perform medical operations. In some embodiments, the master manipulator 2001 includes a multi-degree-of-freedom robotic arm 20011, with a master manipulator sensor located at each joint of the multi-degree-of-freedom robotic arm 20011. Joint information (such as joint angle data) is generated through the master manipulator sensor at each joint. In some embodiments, the multi-degree-of-freedom robotic arm 20011 has six degrees of freedom. In some embodiments, the pose of the master manipulator 2001 can be represented by a set of joint information of the master manipulator joints (e.g., a one-dimensional matrix composed of this joint information). In some embodiments, the master manipulator 2001 also includes a clamp 20012, which can be used to control the opening and closing angle of the end effector. In some embodiments, the main operator 2001 may specifically be Figure 16 The main operator 1600 is shown. In some embodiments, the main control carriage display includes a stereoscopic display 2002, a main control external display 2003, and a main control touch display 2004. The stereoscopic display 2002 displays surgical images and system status prompts, the main control external display 2003 displays surgical images and system status prompts, and the touch display 2004 displays the software user interface of the main control carriage 2000. In some embodiments, the images displayed by the stereoscopic display 2002 or the main control external display 2003 can be determined based on images acquired by the imaging module, for example... Figure 21 The imaging module 2160b shown is illustrated. In some embodiments, the main control carriage pedal is used to collect input from the feet of medical staff and includes structures such as an electrocautery pedal 2005, an electrocoagulation pedal 2006, and a clutch pedal 2007.

[0218] Figure 21 A schematic diagram of a surgical cart 2100 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 21The surgical cart 2100 includes components such as a controller (which can be configured on a computer device and is located inside the surgical cart 2100), a surgical cart chassis 2102, a surgical cart housing 2103, a system status display 2105, a main column 2106, a main crossbeam 2107, positioning arms 2108, and a drive module 2109. The surgical cart chassis 2102 is used to enable the movement and fixation of the surgical cart 2100. The surgical cart housing 2103 integrates the electrical components of the surgical cart internally. The system status display 2105 displays the surgical cart system user interface and receives user input. The main column 2106 is height-adjustable, and its top is fixed to the main crossbeam 2107. The end of the main crossbeam 2107 has a crossbeam platform, and multiple positioning arms 2108 are fixed to the lower end of the crossbeam platform. The positioning arm 2108 is equipped with a drive module 2109, which is used to load surgical tools 2101 or imaging tools 2104 (the imaging tool 2104 may be, for example, a 3D electronic endoscope). In some embodiments, the surgical cart 2100 integrates multiple positioning arms 2108, each positioning arm 2108 having multiple motion joints. In some embodiments, the surgical cart 2100 integrates multiple surgical tools 2101 and imaging tools 2104, with some operating arms 2140a and end instruments 2160a of the multiple surgical tools 2101 and some operating arms 2140b and imaging modules 2160b of the imaging tools 2104 entering the workspace via a sheath 2110. In some embodiments, the controller of the surgical cart 2100 can also be used to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 7 , Figure 8 , Figure 10 , Figure 11 , Figure 13 and Figure 15 Some or all of the steps in the method disclosed herein.

[0219] Note that the above are merely exemplary embodiments and technical principles of this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, this disclosure is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this disclosure, the scope of which is determined by the scope of the appended claims.

Claims

1. A master-slave motion matching method, comprising: Based on the localization image, determine the current pose of the driven tool relative to the reference coordinate system; Based on the current posture of the driven tool, determine the target posture of the master operator's handle; as well as Based on the target posture of the handle of the main operator, a control signal for the main operator is generated; Determining the current attitude of the driven tool relative to the reference coordinate system includes: In the positioning image, multiple pose markers located on the driven tool are identified; Based on the plurality of pose identifiers, an angle identifier located on the driven tool is identified, wherein the angle identifier has a positional association with the first pose identifier among the plurality of pose identifiers; and Based on the angle identifier and the multiple pose identifiers, the current pose of the driven tool relative to the reference coordinate system is determined.

2. The matching method according to claim 1, comprising: Based on the angle identifier and the multiple pose identifiers, the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system is determined.

3. The matching method according to claim 2, comprising: Based on the multiple pose identifiers, the orientation of the pose identifier coordinate system relative to the reference coordinate system is determined; as well as The attitude of the slave tool relative to the reference coordinate system is determined based on the roll angle of the pose identifier coordinate system relative to the slave tool coordinate system and the attitude of the pose identifier coordinate system relative to the reference coordinate system.

4. The matching method according to claim 3, comprising: Based on the two-dimensional coordinates of the plurality of pose markers in the positioning image and the three-dimensional coordinates of the plurality of pose markers in the pose marker coordinate system, the orientation of the pose marker coordinate system relative to the reference coordinate system is determined.

5. The matching method according to claim 2, comprising: Based on the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system and the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system, the three-dimensional coordinates of the multiple pose identifiers in the driven tool coordinate system are determined; as well as Based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the driven tool coordinate system, the attitude of the driven tool relative to the reference coordinate system is determined.

6. The matching method according to any one of claims 2-5, comprising: Determine the first angle around the axis marked by the angle marker in the driven tool coordinate system; Determine the second angle around the axis marked by the first pose identifier in the pose identifier coordinate system; as well as Based on the first and second axial angles, the roll angle of the pose identifier coordinate system relative to the driven tool coordinate system is determined.

7. The matching method according to any one of claims 1-5, wherein the location association includes: The axial correspondence between the angle identifier and the first pose identifier.

8. The matching method according to claim 1, comprising: Based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system, the imaging transformation relationship is determined; Based on the imaging transformation relationship, the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system, and the positional association relationship, multiple angle marker candidate regions are determined in the positioning image. as well as Candidate regions are identified from the multiple angles, and the angle identifiers are recognized.

9. The matching method according to claim 8, comprising: Based on the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system and the positional relationship, multiple candidate three-dimensional coordinates of angle markers are determined in the pose marker coordinate system. as well as Based on the imaging transformation relationship and the three-dimensional coordinates of the multiple angle marker candidates, the multiple angle marker candidate regions are determined in the positioning image.

10. The matching method according to claim 8 or 9, comprising: The pixel with the largest corner likelihood value in each of the candidate regions of the angle identifier is determined to form a pixel set; The candidate region for the angle identifier corresponding to the pixel with the largest corner likelihood value in the pixel set is determined as the candidate region for the angle identifier to be identified. as well as Multiple angle pattern matching templates are used to match the candidate regions of the angle identifiers to be identified, so as to identify the angle identifiers.

11. The matching method according to claim 10, comprising: In response to a matching failure, the candidate region for the angle identifier corresponding to the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as the candidate region for the angle identifier to be identified.

12. The matching method according to claim 8 or 9, comprising: Based on the candidate region of the angle identifier where the angle identifier is located, the first pose identifier that has a positional relationship with the angle identifier is determined.

13. The matching method according to any one of claims 1-5, 8, and 9, characterized in that, Also includes: Determine the current orientation of the handle of the main controller; as well as Based on the target posture and current posture of the handle of the main operator, control signals for the main operator are generated.

14. The matching method according to claim 13, characterized in that, The main actuator includes at least one attitude joint for controlling the attitude of the handle, and the method includes: Obtain joint information of the at least one posture joint; and The current posture of the master manipulator is determined based on the joint information of the at least one posture joint.

15. The matching method according to any one of claims 1-5, 8, and 9, characterized in that, The driven tool includes a manipulator arm and an end effector disposed at the end of the manipulator arm. Determining the current orientation of the driven tool relative to the reference coordinate system includes: Determine the current orientation of the end effector relative to the base coordinate system of the driven tool; or Determine the current orientation of the end effector image on the display relative to the world coordinate system.

16. The matching method according to any one of claims 1-5, 8, and 9, characterized in that, The master manipulator includes at least one attitude joint for controlling the attitude of the handle of the master manipulator, and the control signal includes a control signal for controlling one or more of the at least one attitude joint.

17. The matching method according to claim 16, characterized in that, The at least one attitude joint includes one or more uncoupled attitude joints, and the method further includes: Obtain joint information of the other posture joints besides the one or more posture joints in the at least one posture joint; and Based on the joint information of the other posture joints, the transformation matrix of the other posture joints is determined.

18. The matching method according to claim 17, characterized in that, Also includes: The control signal of the main manipulator is generated based on the target posture of the handle and the transformation matrix of the other posture joints.

19. The matching method according to any one of claims 1-5, 8, and 9, characterized in that, Also includes: In response to the fulfillment of predetermined conditions, the attitude matching degree between the handle of the master operator and the slave tool is determined, wherein the predetermined conditions include the triggering of teleoperation control.

20. The matching method according to claim 19, characterized in that, Also includes: Based on the current posture of the handle of the master operator and the current posture of the slave tool, the posture matching degree between the handle of the master operator and the slave tool is determined.

21. The matching method according to claim 19, characterized in that, Also includes: In response to the attitude matching degree being lower than a preset threshold, the control signal of the handle of the master operator is generated so that the attitude matching degree is higher than or equal to the preset threshold.

22. The matching method according to claim 19, characterized in that, Also includes: In response to the attitude matching degree being higher than or equal to a preset threshold, a master-slave mapping is established between the master operator and the slave tool.

23. The matching method according to any one of claims 1-5, 8, and 9, characterized in that, The target orientation of the master controller's handle is consistent with the current orientation of the slave tool.

24. A robot system, comprising: The main manipulator includes a multi-degree-of-freedom robotic arm, a handle disposed on the multi-degree-of-freedom robotic arm, and at least one motor and at least one main manipulator sensor disposed at at least one joint on the multi-degree-of-freedom robotic arm, wherein the at least one main manipulator sensor is used to obtain joint information of the at least one joint; A driven tool includes a manipulator and an end effector disposed at the end of the manipulator; Image acquisition device, used to acquire positioning images; as well as A control device, communicatively connected to the image acquisition unit and the master operator, is configured to perform the matching method as described in any one of claims 1-23.

25. A computer device, the computer device comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory and configured to execute the at least one instruction to perform the matching method as described in any one of claims 1-23.

26. A computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes a robot system to perform the matching method as described in any one of claims 1-23.

Images