Master-slave motion control method based on pose identification and surgical robot system

Abstract

The present disclosure relates to the field of robotics, and discloses a control method of master-slave motion based on pose markers, comprising: acquiring a positioning image; in the positioning image, identifying a plurality of pose markers located on a slave tool, the plurality of pose markers comprising different pose marker patterns; determining a current pose of the slave tool relative to a reference coordinate system based on the plurality of pose markers; determining a target pose of a handle of a master manipulator based on the current pose of the slave tool; and generating a control signal for the master manipulator based on the target pose of the handle of the master manipulator.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and more particularly to a master-slave motion control method based on pose identification and a surgical robot system. Background Technology

[0002] With the development of technology, the use of medical robots to assist medical staff in performing surgeries has developed rapidly. Medical robots can not only help medical staff perform a series of medical diagnoses and auxiliary treatments, but also effectively alleviate the problem of strain on medical resources.

[0003] Typically, a medical robot consists of a driven tool for performing operations and a master manipulator for controlling the movement of the driven tool. In a real-world scenario, the driven tool is configured to enter the operating area, and medical personnel remotely operate the master manipulator to control the movement of the driven tool within the operating area to perform the medical procedure.

[0004] However, the number of remotely controlled slave tools is generally greater than the number of master tools. Therefore, during surgery, there may be situations where the slave tools controlled by the master tool are altered. Furthermore, at the start or during the procedure, the master tool needs to establish a mapping with the slave tools before performing master-slave control. Because the master tool and its corresponding slave tool are not pre-matched in terms of posture, a mismatch (such as orientation or angle) can occur between them. Directly matching them in master-slave mapping would reduce the control precision of the slave tools and degrade the human-computer interaction experience for medical personnel (such as surgeons). Therefore, after the master tool and slave tool are matched and before remote operation, the posture of the master tool needs to be matched with the posture of the slave tool to improve the accuracy of the master tool's posture control over the slave tools. Summary of the Invention

[0005] In some embodiments, this disclosure provides a master-slave motion control method, comprising: acquiring a positioning image; identifying multiple pose markers located on a slave tool in the positioning image, the multiple pose markers including different pose marker patterns; determining the current posture of the slave tool relative to a reference coordinate system based on the multiple pose markers; determining the target posture of the handle of a master operator based on the current posture of the slave tool; and generating a control signal for the master operator based on the target posture of the handle of the master operator.

[0006] In some embodiments, this disclosure provides a robot system, including: a master manipulator, comprising a multi-degree-of-freedom robotic arm, a handle disposed on the multi-degree-of-freedom robotic arm, and at least one motor and at least one master manipulator sensor disposed at at least one joint on the multi-degree-of-freedom robotic arm, wherein the at least one master manipulator sensor is used to obtain joint information of at least one joint; a slave tool, comprising a manipulator arm and an end effector disposed at the end of the manipulator arm; an image acquisition device for acquiring positioning images; and a control device communicatively connected to the image acquisition device and the master manipulator, wherein the control device is configured to perform a master-slave motion control method according to some embodiments of this disclosure.

[0007] In some embodiments, this disclosure provides a computer device, the computer device including: a memory for storing at least one instruction; and a processor coupled to the memory and configured to execute at least one instruction to perform a master-slave motion control method according to some embodiments of this disclosure.

[0008] In some embodiments, this disclosure provides a computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes a robot system to implement a master-slave motion control method according to some embodiments of this disclosure. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below. The accompanying drawings described below only show some embodiments of this disclosure. For those skilled in the art, other embodiments can be obtained based on the content of the embodiments of this disclosure and these drawings without creative effort.

[0010] Figure 1 Schematic diagrams of robot systems according to some embodiments of the present disclosure are shown;

[0011] Figure 2 A schematic diagram of the structure of an operating arm according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of the structure of an operating arm according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram showing a label including multiple pose identifiers according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic diagram showing a label disposed on the periphery of the end of an operating arm and formed into a cylindrical shape according to some embodiments of the present disclosure;

[0015] Figure 6A flowchart illustrating a master-slave motion control method according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A flowchart illustrating a method for determining the three-dimensional coordinates of a plurality of pose identifiers relative to a slave tool coordinate system according to some embodiments of the present disclosure;

[0017] Figure 8 A flowchart illustrating a method for determining the three-dimensional coordinates of a plurality of pose markers relative to a slave tool coordinate system according to other embodiments of the present disclosure;

[0018] Figure 9 A flowchart illustrating a method for identifying pose identifiers according to some embodiments of the present disclosure is shown.

[0019] Figure 10 A schematic diagram showing pose identification patterns according to some embodiments of the present disclosure;

[0020] Figure 11 A flowchart illustrating a method for searching pose identifiers according to some embodiments of the present disclosure;

[0021] Figure 12 A schematic diagram illustrating a search pose identifier according to some embodiments of the present disclosure;

[0022] Figure 13 A flowchart illustrating a method for searching a second pose identifier according to some embodiments of the present disclosure is shown;

[0023] Figure 14 A flowchart illustrating a method for searching pose identifiers according to some embodiments of the present disclosure;

[0024] Figure 15 A schematic diagram of a master operator according to some embodiments of the present disclosure is shown;

[0025] Figure 16 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown;

[0026] Figure 17 Schematic diagrams of surgical robot systems according to some embodiments of the present disclosure are shown;

[0027] Figure 18 A schematic diagram of a surgical instrument according to some embodiments of the present disclosure is shown;

[0028] Figure 19 A schematic diagram of a main control trolley according to some embodiments of the present disclosure is shown;

[0029] Figure 20 A schematic diagram of an operating table according to some embodiments of the present disclosure is shown. Detailed Implementation

[0030] To make the technical problems solved by this disclosure, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely exemplary embodiments of this disclosure, and not all embodiments.

[0031] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this disclosure, it should be noted that unless otherwise expressly specified and limited, the terms "installed," "connected," "coupled," and "coupled" should be interpreted broadly. For example, they can refer to fixed connections or detachable connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances. In this disclosure, the end closer to the operator (e.g., a doctor) is defined as the proximal end, proximal or rear end, or posterior end, and the end closer to the surgical patient is defined as the distal end, distal or anterior end, or anterior end. Those skilled in the art will understand that embodiments of this disclosure can be used in medical devices or surgical robots, as well as in other non-medical devices.

[0032] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., variations in Cartesian X, Y, and Z coordinates can be used to describe three translational degrees of freedom, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "attitude" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and attitude of an object or part of an object, which can be described, for example, using six parameters from the six degrees of freedom mentioned above. In this disclosure, the pose of the handle of the master manipulator can be represented by a set of joint information of the master manipulator joints (e.g., a one-dimensional matrix composed of this joint information). In this disclosure, the joint information of a joint can include the angle of rotation of the respective joint relative to the respective joint axis or the distance moved relative to the initial position.

[0033] In this disclosure, a reference coordinate system can be understood as a coordinate system capable of describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own perception coordinate system, etc. In this disclosure, an object can be understood as an object or target that needs to be positioned, such as the operating arm of a driven tool or the end effector of the operating arm. In this disclosure, the pose of the driven tool or a portion thereof refers to the pose of the driven tool coordinate system defined by the driven tool or a portion thereof relative to the reference coordinate system.

[0034] Figure 1 A schematic diagram of a robot system 100 according to some embodiments of the present disclosure is shown. Figure 1 As shown, the robot system 100 may include: an image acquisition device 110, a control device 120, at least one driven tool (150a, 150b), and a master manipulator 180. In some embodiments, the image acquisition device 110 and the master manipulator 180 are respectively communicatively connected to the control device 120.

[0035] In some embodiments, the image acquisition device 110 can be used to acquire positioning images. The positioning images may include partial or complete images of the driven tools (150a, 150b). In some embodiments, positioning markers are provided on the driven tools (150a, 150b). In some embodiments, the positioning markers include pose markers, based on which the position or orientation of the driven tools (150a, 150b) can be determined. In some embodiments, multiple pose markers include different pose marker patterns (described in detail below). Figure 1 As shown, if the driven tools (150a, 150b) are within the field of view of the image acquisition device 110, the acquired positioning image may include a partial image of the driven tools (150a, 150b). In some embodiments, the image acquisition device 110 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. Depending on the application scenario, the image acquisition device 110 may be an industrial camera, an underwater camera, a miniature electronic camera, an endoscope camera, etc. In some embodiments, the image acquisition device 110 may be fixed in position or changeable in position, for example, an industrial camera fixed at a monitoring position or an endoscope camera whose position or attitude can be adjusted. In some embodiments, the image acquisition device 110 may realize at least one of visible light band imaging, infrared band imaging, CT (Computed Tomography) imaging, and acoustic imaging. In some embodiments, the image acquisition device 110 may be, for example, a... Figure 20 The imaging module 2060b is shown in the figure.

[0036] In some embodiments, the control device 120 is configured to execute at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 11 , Figure 13 and Figure 14 The method disclosed herein may include some or all of the steps. In some embodiments, the control device 120 may receive a positioning image from the image acquisition device 110 and process the positioning image. For example, the control device 120 may identify multiple positioning markers located on at least one driven tool (150a, 150b) in the positioning image. In some embodiments, the control device 120 may determine the posture of the driven tools (150a, 150b) based on the positioning image, for example, determining the current posture of the driven tools (150a, 150b) relative to a reference coordinate system based on the positioning image. In some embodiments, the control device 120 may also determine the posture of the master operator 180 based on the posture of the driven tools (150a, 150b), for example, determining the target posture of the handle of the master operator 180 based on the current posture of the driven tools (150a, 150b). The target posture of the handle of the master operator 180 is mapped to the current posture of the driven tools (150a, 150b). In some embodiments, the control device 120 may also generate control signals for the master operator 180 based on the target posture of the handle of the master operator 180. In some embodiments, the control device 120 may send the control signals of the master operator 180 to the drive motors of multiple joints of the master operator 180.

[0037] In some embodiments, the main manipulator 180 includes a multi-degree-of-freedom robotic arm (e.g., a six-degree-of-freedom robotic arm), with joint sensors disposed at some joints of the multi-degree-of-freedom robotic arm to generate joint information (such as joint angle data). In some embodiments, the joint sensors are potentiometers and / or encoders. In some embodiments, the main manipulator may be, for example, a... Figure 15 The main manipulator 1500 is shown in the figure. In some embodiments, the main manipulator 1500 may include a controller, which can calculate the attitude data of the main manipulator 1500 based on the joint information obtained from each joint sensor, and send the calculated attitude data to the control device 120. In other embodiments, the control device 120 may also calculate the attitude data of the main manipulator based on the joint information sent by the joint sensors.

[0038] In some embodiments, the driven tool 150a is taken as an example. Figure 1As shown, the driven tool 150a includes an operating arm 140. In some embodiments, the driven tool 150a further includes an end effector 160 disposed at the end 130 of the operating arm. In some embodiments, the operating arm 140 may be a rigid arm or a deformable arm. In some embodiments, the operating arm 140 may include a continuous deformable arm. A continuous deformable arm is, for example, as shown... Figure 3 The manipulator 300 is shown. In some embodiments, the manipulator 140 may include a multi-degree-of-freedom manipulator composed of multiple joints. For example, a manipulator capable of 4 to 7 degrees of freedom of movement. For instance, a manipulator capable of 6 degrees of freedom of movement. In some embodiments, the end effector 160 may include, but is not limited to, surgical forceps, an electrosurgical unit, an electrohook, etc.

[0039] Figure 2 A schematic diagram of a segment 200 of an operating arm according to some embodiments of the present disclosure is shown. In some embodiments, the operating arm of a driven tool may include at least one deformable segment 200. Figure 2 As shown, the deformable segment 200 includes a fixed disk 210 and multiple structural bones 220. The first end of each structural bone 220 is fixedly connected to the fixed disk 210, and the second end is connected to a driving unit (not shown). In some embodiments, the fixed disk 210 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and its cross-section may be circular, rectangular, polygonal, or various other shapes. In some embodiments, the driving unit deforms the segment 200 by driving the structural bones 220. For example, the driving unit deforms the segment 200 by driving the structural bones 220 to a position such as... Figure 2 The bending state is shown. In some embodiments, the second end of the multiple structural bones 220 passes through the base plate 230 and is connected to the drive unit. In some embodiments, similar to the fixed plate 210, the base plate 230 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and the cross-section may be a circle, rectangle, polygon, etc. The drive unit may include a linear motion mechanism, a drive segment, or a combination of both. The linear motion mechanism may be connected to the structural bones 220 to push or pull the structural bones 220, thereby driving the segment 200 to bend. The drive segment may include a fixed plate and multiple structural bones, wherein one end of the multiple structural bones is fixedly connected to the fixed plate. The other end of the multiple structural bones of the drive segment is connected to or integrally formed with the multiple structural bones 220 to drive the bending of the segment 200 by bending the drive segment. In some embodiments, a spacer plate 240 is also included between the fixed plate 210 and the base plate 230, through which the multiple structural bones 220 pass. Similarly, the drive segment may also include a spacer plate.

[0040] Figure 3 A schematic diagram of the structure of an operating arm 300 according to some embodiments of the present disclosure is shown. For example... Figure 3As shown, the operating arm 300 is a deformable operating arm, which may include an operating arm end cap 310 and an operating arm body 320. The operating arm body 320 may include one or more segments, such as a first segment 3201 and a second segment 3202. In some embodiments, the structures of the first segment 3201 and the second segment 3202 may be similar to those of... Figure 2 The shown component 200 is similar. In some implementations, such as... Figure 3 As shown, the main body 320 of the operating arm also includes a first straight rod segment 3203 located between the first segment 3201 and the second segment 3202. The first end of the first straight rod segment 3203 is connected to the base plate of the second segment 3202, and the second end is connected to the fixed plate of the first segment 3201. In some embodiments, such as... Figure 3 As shown, the main body 320 of the operating arm also includes a second straight rod segment 3204, the first end of which is connected to the base plate of the first component 3201. Figure 3 As shown, each segment (first segment 3201 and second segment 3202) may include a base plate, a fixed plate, and multiple structural bones penetrating the base plate and the fixed plate. The multiple structural bones may be fixedly connected to the fixed plate and slidably connected to the base plate. The continuous deformable arm and its included segments can be described by a kinematic model (detailed below).

[0041] In some embodiments, the structure of each segment of the manipulator 300 may be as follows: Figure 2 The shown component is 200. (As shown in the image) Figure 2 As shown, the base disk coordinate system The base disk is attached to the t-th (t=1,2,3…) section of the continuum, with its origin located at the center of the base disk, and the XY plane coinciding with the plane of the base disk. Pointing from the center of the base plate to the first structural bone (the first structural bone can be understood as any one of multiple structural bones chosen as a reference). Bending plane coordinate system 1 Its origin coincides with the origin of the base disk coordinate system, and the XY plane coincides with the bending plane. and Coincident. Fixed disk coordinate system. The origin of the fixed disk is located at the center of the fixed disk, and the XY plane coincides with the plane of the fixed disk. Pointing from the center of the fixed plate to the first structural bone. Curved plane coordinate system 2 Its origin is located at the center of the fixed disk, and the XY plane coincides with the bending plane. and coincide.

[0042] like Figure 2The single segment 200 shown can be represented by a kinematic model. The position of the end of segment t (fixed disk coordinate system {te}) relative to the base disk coordinate system {tb}. tb P te ,attitude tb R te It can be determined based on the following formulas (1) and (2):

[0043]

[0044] tb R te = tb R tl t1 R t2 t2 R te (2)

[0045] Among them, L t For the t-th segment, construct a virtual structural bone (e.g., Figure 2 The length of the virtual structural bone 221 shown in the figure, θ t In the t-th section, about or Rotate to Required rotation angle tb R t1 Let {t1} be the orientation of the bending plane coordinate system of segment t relative to the base disk coordinate system {tb}. t1 R t2 Let t be the orientation of the bending plane coordinate system 2{t2} of the t-th segment relative to the bending plane coordinate system 1{t1}. t2 R te Let {te} be the orientation of the fixed disk coordinate system {t2} of the t-th segment relative to the curved plane coordinate system 2{t2}.

[0046] tb R t1 , t1 R t2 and t2 R te It can be based on the following formulas (3), (4) and (5):

[0047]

[0048]

[0049]

[0050] Where, δ t For the t-th segment, the bending plane and The included angle.

[0051] like Figure 2 The joint parameter Ψ of the single segment 200 shown t It can be determined based on the following formula (6):

[0052] ψ t =[θ t δ t ] T (6)

[0053] In some embodiments, the driving amount of multiple bone structures has a known mapping relationship with joint parameters. Based on the target joint parameters of the segment and the mapping relationship, the driving amount of the multiple bone structures can be determined. The driving amount of the multiple bone structures can be understood as moving a single segment from its initial state (e.g., θ) t =0) The length of the structural bone subjected to push or tension when bent to the target bending angle. In some embodiments, the mapping relationship between the driving amount of multiple structural bones and joint parameters can be determined based on the following formula (7):

[0054] q i_tool ≡-r ti_tool θ t cos(δ t +β ti_tool (7)

[0055] Where, r ti_tool Let β be the distance from the i-th structural bone in the t-th segment to the virtual structural bone. ti_tool Let q be the angle between the i-th structural bone and the first structural bone in the t-th segment. i_tool Let be the driving quantity of the i-th structural bone. The driving signal of the driving unit can be determined based on the driving quantity of the i-th structural bone.

[0056] In some embodiments, the entire deformable arm can be described by a kinematic model. For example... Figure 3 As shown, transformations can be performed between multiple coordinate systems located at multiple positions of the deformable arm. For example, the end effector of the continuous deformable arm can be determined in the world coordinate system {w} based on the following formula (8):

[0057] W T tip = W T 1b 1b T 1e 1e T 2b 2b T 2e 2e T tip (8)

[0058] in,W T tip The homogeneous transformation matrix of the end effector of the deformable arm of a continuum relative to the world coordinate system; W T 1b Represents the homogeneous transformation matrix of the base disk of the first continuum segment relative to the world coordinate system; 1b T le Represents the homogeneous transformation matrix of the fixed disk of the first continuous segment relative to the base disk of the first continuous segment; 1e T 2b Represents the homogeneous transformation matrix of the base disk of the second continuous segment relative to the fixed disk of the first continuous segment; 2b T 2e Represents the homogeneous transformation matrix of the fixed disk of the second continuum segment relative to the base disk of the second continuum segment; 2e T tip This represents the homogeneous transformation matrix of the end effector of the continuous deformable arm relative to the fixed disk of the second continuous segment. In some embodiments, the end effector is fixedly mounted on the fixed disk, therefore... 2e T tip It is known or predetermined.

[0059] Those skilled in the art will understand that the deformable arm has different joint parameters in different working states. For example, Figure 3 The manipulator 300 shown includes at least four operating states. The four operating states of the manipulator 300 are described below:

[0060] First working state: Only the second segment 3202 participates in the pose control of the end effector (e.g., only the second segment 3202 enters the workspace), and the joint parameters of the manipulator 300 can be determined based on the following formula (9):

[0061]

[0062] Where, ψ c1 These are the joint parameters of the manipulator 300 in its first working state. Let L2, θ2, δ2 be the rotation angle of the manipulator 300 around its axis, and let L2, θ2, δ2 be the angle of rotation of the manipulator 300 around its axis. Figure 2 In the structure shown in section 200, L t θ t and δ t They have the same physical meaning.

[0063] Second working state: The second segment 3202 and the first straight segment 3203 participate in the pose control of the end effector (for example, the second segment 3202 is fully in the workspace, and the first straight segment 3203 is partially in the workspace). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (10):

[0064]

[0065] Where, ψ c2 For the joint parameters of the manipulator 300 in the second working state, L r This is the feed rate for the first straight segment 3203.

[0066] Third working state: The second segment 3202, the first straight segment 3203, and the first segment 3201 participate in the pose control of the end effector (for example, the second segment 3202 is fully in the working space, the first straight segment 3203 is fully in the working space, and the first segment 3201 is partially in the working space). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (11):

[0067]

[0068] Where, ψ c3 For the joint parameters of the manipulator 300 in the third working state, L1, θ1, and δ1 are as follows: Figure 2 In the structure shown in section 200, L t θ t and δ t They have the same physical meaning.

[0069] Fourth working state: The second segment 3202, the first straight segment 3203, the first segment 3201, and the second straight segment 3204 participate in the pose control of the end effector (for example, the second segment 3202 is fully in the working space, the first straight segment 3203 is fully in the working space, the first segment 3201 is fully in the working space, and the second straight segment 3204 is partially in the working space). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (12):

[0070]

[0071] Where, ψ c4 For the joint parameters of the manipulator 300 in the fourth working state, L s This is the feed rate for the second straight segment 3204.

[0072] In some embodiments, the driven tool is provided with a plurality of pose markers. For example, a plurality of pose markers are distributed on the operating arm of the driven tool. In some embodiments, the plurality of pose markers are disposed on the outer surface of the cylindrical portion of the operating arm. For example, the plurality of pose markers are distributed circumferentially on the end cap 310 of the operating arm. For example, the plurality of pose markers are disposed on the outer surface of the cylindrical portion of the end cap 310 of the operating arm. In some embodiments, the pose of the driven tool can be determined based on an image of the plurality of pose markers. In some embodiments, a positioning label including a plurality of pose markers (e.g., ...) is disposed on the outer surface of the cylindrical portion of the operating arm. Figure 4The label 400 shown is... Figure 5 The label 500 shown includes multiple pose identifiers, including multiple different pose identifier patterns distributed circumferentially along the columnar portion on the positioning label, and pose identifier pattern corner points within the pose identifier patterns.

[0073] In some embodiments, the pose identifier may include a pose identifier pattern and pose identifier pattern corner points within the pose identifier pattern. In some embodiments, the pose identifier pattern may be disposed on a label on the end of the operating arm, or may be printed on the end of the operating arm, or may be a pattern formed by the physical structure of the end of the operating arm itself, for example, it may include recesses or protrusions and combinations thereof. In some embodiments, the pose identifier pattern may include a pattern formed with brightness, grayscale, color, etc. In some embodiments, the pose identifier pattern may include a pattern that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provides information that can be detected by an image acquisition device. Those skilled in the art will understand that in some embodiments, the pose of the pose identifier or the pose of the pose identifier pattern may be represented by the pose of the pose identifier pattern corner point coordinate system. In some embodiments, the pose identifier pattern is disposed on an area on the end of the operating arm suitable for image acquisition by an image acquisition device, for example, an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily disturbed or obstructed during operation.

[0074] Figure 4 A schematic diagram of a tag 400 including multiple pose identifiers according to some embodiments is shown. Figure 5 A schematic diagram is shown of a label 500 disposed on the periphery of the end of the manipulator and forming a cylindrical shape. It can be understood that, for simplicity, label 400 may include the same pose marking pattern as label 500.

[0075] See Figure 4 Multiple pose identifiers may include multiple different pose identifier patterns 410. Multiple pose identifiers may also include multiple pose identifier pattern corner points P4 within the multiple different pose identifier patterns 410, which are represented by the symbol "○" in this disclosure. In some embodiments, pose identifiers can be determined by identifying the pose identifier pattern 410 or the pose identifier pattern corner points P4 therein.

[0076] See Figure 5In the circumferential setting state, label 400 becomes label 500 with a cylindrical spatial structure. In some embodiments, the axial angle or roll angle of the pose identifier can be represented by the axial angle of the pose identifier pattern or the corner point of the pose identifier pattern. The axial angle of each pose identifier pattern or corner point is known or predetermined. In some embodiments, the axial angle identified by each pose identifier can be determined based on the distribution of multiple pose identifiers (e.g., pose identifier patterns or corner points of pose identifier patterns). In some embodiments, the multiple pose identifiers can be uniformly distributed (e.g., the corner points of the pose identifier patterns in label 400 are evenly spaced, and the corner points of the pose identifier patterns in label 500 are evenly distributed). In other embodiments, the multiple pose identifiers can be non-uniformly distributed. In some embodiments, based on the distribution of multiple pose identifiers, each pose identifier pattern can be used to identify a specific axial angle, and each pose identifier pattern has a one-to-one correspondence with the identified axial angle. In this disclosure, the angle about the axis or roll angle refers to the angle about the Z-axis (e.g., the Z-axis of the driven tool coordinate system {wm}). In some embodiments, the manipulator is a deformable manipulator, and the Z-axis is along the tangential direction of the manipulator.

[0077] like Figure 5 As shown, multiple different pose marker patterns 510 in the label 500 are uniformly distributed circumferentially along the cylindrical structure. The corner points of multiple pose marker patterns are uniformly distributed on the cross-sectional circle 520 of the XY plane of the driven tool coordinate system {wm}. Then, the distribution angle (e.g., angle α0) of any adjacent pose marker pattern corner points is equal. Set the pose marker pattern corner point P5 pointing to the X-axis. P5 is used as the reference corner point for marking 0° around the axis (the pose marker pattern where the pose marker pattern corner point P5 is located is used as the reference pattern). Then, the around-axis angle of the pose marker pattern corner point can be determined according to the positional relationship between any pose marker pattern corner point and the pose marker pattern corner point P5. In some embodiments, the around-axis angle of the pose marker pattern corner point can be determined based on the following formula (13):

[0078] α m =α0(m-1) (13)

[0079] Where, α m Let P5 be the first pose marker corner point, and the angle around the axis of the m-th pose marker corner point be in the clockwise direction of the cross-sectional circle 320.

[0080] Some embodiments of this disclosure provide a method for controlling master-slave motion. Figure 6 A flowchart illustrating a master-slave motion control method 600 according to some embodiments of the present disclosure is shown. Method 600 can be used in robotic systems, such as… Figure 1 The robot system 100 shown or Figure 17 The surgical robot system 1700 is shown. Some or all of the steps in method 600 can be performed by a control device (e.g., control device 120) of the robot system 100. Control device 120 can be configured on a computing device. Method 600 can be implemented by software, firmware, and / or hardware. In some embodiments, method 600 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0081] See Figure 6 In step 601, a positioning image is acquired. In some embodiments, the positioning image includes a portion of the driven tool and multiple pose markers and at least one angle marker on the driven tool. For example, the positioning image includes a portion of the manipulator and multiple pose markers and at least one angle marker on the manipulator. In some embodiments, the positioning image can be obtained from, for example... Figure 1 The image acquisition device 110 shown receives a positioning image. For example, the control device 120 can receive a positioning image actively sent by the image acquisition device 110. Alternatively, the control device 120 can send an image request command to the image acquisition device 110, and the image acquisition device 110 responds to the image request command by sending a positioning image to the control device 120.

[0082] Continue reading Figure 6 In step 603, in the positioning image, multiple pose identifiers located on the slave tool are identified, the multiple pose identifiers including different pose identifier patterns. For example, an exemplary method for identifying multiple pose identifiers located on the slave tool may include, as... Figure 9 , Figure 11 , Figure 13 and Figure 14 The method is illustrated. In some embodiments, the control device can identify some or all of the pose markers in the positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm that can extract or recognize features of the pose markers. For example, the image processing algorithm may include a corner detection algorithm for detecting corner points of the pose marker pattern. This corner detection algorithm may be, but is not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the pose marker pattern. As another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the pose marker pattern. In some embodiments, the control device can identify some or all of the pose markers in the positioning image using a recognition model.

[0083] Continue reading Figure 6In step 605, the current pose of the driven tool relative to the reference coordinate system is determined based on multiple pose identifiers. In some embodiments, method 600 further includes: determining the two-dimensional coordinates of the multiple pose identifiers in the positioning image; and determining the current pose of the driven tool relative to the reference coordinate system based on the two-dimensional coordinates of the multiple pose identifiers in the positioning image and the three-dimensional coordinates of the multiple pose identifiers relative to the driven tool coordinate system. In some embodiments, the coordinates of the pose identifiers can be represented by the coordinates of the corner points of the pose identifier pattern. For example, the two-dimensional coordinates of the pose identifiers in the positioning image and the three-dimensional coordinates in the driven tool coordinate system can be represented by the coordinates of the corner points of the pose identifier pattern. In some embodiments, the pose of the driven tool coordinate system relative to the reference coordinate system can be determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image and the three-dimensional coordinates in the driven tool coordinate system as the current pose of the driven tool relative to the reference coordinate system.

[0084] In some embodiments, method 600 may further include: determining the pose of the slave tool coordinate system relative to the reference coordinate system based on the two-dimensional coordinates of multiple pose marker corner points in the positioning image, the three-dimensional coordinates of the multiple pose marker corner points in the slave tool coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system. In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system may be known. For example, the reference coordinate system is the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system can be determined according to the pose of the camera. In other embodiments, the reference coordinate system may also be the camera coordinate system itself, depending on actual needs. In some embodiments, based on the camera imaging principle and projection model, the pose of the slave tool coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of multiple pose marker corner points in the positioning image and the three-dimensional coordinates of the multiple pose marker corner points in the slave tool coordinate system. Based on the pose of the slave tool coordinate system relative to the camera coordinate system and the transformation relationship between the camera coordinate system and the reference coordinate system, the pose of the slave tool coordinate system relative to the reference coordinate system can be obtained. In some embodiments, the intrinsic parameters of the camera may also be considered. For example, the intrinsic parameters of the camera may be as follows: Figure 1 The image acquisition device 110 shown has camera intrinsic parameters. These parameters can be known or obtained through calibration. In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the camera's optical center as the origin or a coordinate system established with the camera's lens center as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of that line).

[0085] In some embodiments, the pose of the driven tool coordinate system {wm} relative to the reference coordinate system (e.g., the world coordinate system) can be determined based on the following formula (14):

[0086]

[0087] in, w R wm This refers to the orientation of the driven tool's coordinate system relative to the world coordinate system. w P wm This represents the position of the driven tool's coordinate system relative to the world coordinate system. w R lens The pose of the camera coordinate system relative to the world coordinate system. w P lens This represents the position of the camera coordinate system relative to the world coordinate system. lens R wm The orientation of the driven tool coordinate system relative to the camera coordinate system. lens P wm This represents the position of the driven tool coordinate system relative to the camera coordinate system.

[0088] This disclosure provides several embodiments of a method for determining the three-dimensional coordinates of a plurality of pose markers relative to a driven tool coordinate system. In some embodiments, the three-dimensional coordinates of the plurality of pose markers relative to the driven tool coordinate system are determined based on the distribution of the plurality of pose markers. For example, the three-dimensional coordinates of the corner points of the plurality of pose marker patterns in the driven tool coordinate system are determined based on the distribution of the corner points of the plurality of pose marker patterns.

[0089] In some embodiments, the current pose of the slave tool is the current pose of the slave tool relative to its base coordinate system. The slave tool includes a manipulator arm and an end effector disposed at the end of the manipulator arm. The current pose of the slave tool includes the pose of the end effector relative to the base coordinate system of the slave tool or the pose of the manipulator arm end relative to the base coordinate system of the slave tool. In some embodiments, the base coordinate system of the slave tool can be the coordinate system of the base on which the slave tool is mounted (e.g., the end effector of a surgical robot's manipulator arm), the coordinate system of the sheath through which the slave tool passes (e.g., the coordinate system of the sheath outlet), the coordinate system of the remote center of motion (RCM) of the slave tool, etc. For example, the base coordinate system of the slave tool can be set at the sheath outlet position and remains fixed during teleoperation. The current pose of the end effector can be transformed to obtain its pose relative to other coordinate systems. In some embodiments, the current pose of the slave tool is the current pose of the image of the slave tool on the display relative to the world coordinate system. In some embodiments, the world coordinate system can be the coordinate system of the space where the operator or master manipulator is located. Therefore, the pose of the slave tool's image on the display relative to the world coordinate system is the pose perceived by the operator. Slave tools include surgical instruments and vision tools. During surgery, surgical instruments perform procedures inside the patient's body, while vision tools use cameras to acquire images inside the patient's body and transmit these images to the operating table. The images are processed by a video processing module in the operating table and displayed on the monitor of the main control unit. The operator obtains the current pose of the slave tool through the image on the display. In some embodiments, the current pose of the slave tool's image on the display relative to the world coordinate system can be obtained through coordinate transformation. For example, based on the slave tool's base coordinate system, the camera's coordinate system, the display's coordinate system, and the world coordinate system, the current pose of the slave tool's image on the display relative to the world coordinate system can be obtained.

[0090] In step 607, the target pose of the master operator's handle is determined based on the current pose of the slave tool. In some embodiments, the current pose of the slave tool is its pose relative to its base coordinate system, or the current pose of the slave tool's image on the display relative to the world coordinate system. The target pose of the master operator's handle is its pose relative to the master operator's base coordinate system. The master operator's base coordinate system may be the coordinate system of the base to which the master operator is connected. In some embodiments, the master operator's base coordinate system and the slave tool's base coordinate system have a defined transformation relationship.

[0091] In some embodiments, the current orientation of the slave tool is matched with the target orientation of the handle, for example, being the same, proportional, or having a fixed difference. For example, before teleoperation, the current orientation of the slave tool is kept unchanged and used as the target orientation of the handle, and the current orientation of the handle is adjusted to the target orientation to achieve orientation matching between the handle and the slave tool.

[0092] In step 609, a control signal for the main controller is generated based on the target pose of the main controller's handle. In some embodiments, method 600 further includes: determining the current pose of the main controller's handle; and generating a control signal for the main controller based on the target pose and the current pose of the main controller's handle. The current pose of the main controller's handle is the pose of the main controller's handle relative to the main controller's base coordinate system. In some embodiments, the control signal corresponding to the handle reaching the target pose from the current pose is determined based on the handle's current pose and the target pose.

[0093] Figure 7 A flowchart illustrates a method 700 for determining the three-dimensional coordinates of a plurality of pose markers relative to a driven tool coordinate system according to some embodiments of the present disclosure. Some or all of the steps in method 700 may be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 700 can be implemented by software, firmware, and / or hardware. In some embodiments, method 700 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 17 The surgical robot system 1700 is shown. In some embodiments, method 700 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0094] See Figure 7 In step 701, based on the distribution of multiple pose markers, the axial angles of the multiple pose markers relative to the Z-axis of the driven tool coordinate system are determined. In some embodiments, the axial angles of the multiple pose markers relative to the Z-axis of the driven tool coordinate system can be determined based on multiple pose marker patterns. For example, each pose marker pattern can identify a specific axial angle, and different pose marker patterns correspond one-to-one with the identified axial angles. Based on the identification of the pose marker patterns and the correspondence between the pose marker patterns and the axial angles, the axial angles identified by the identified pose marker patterns can be determined. It should be understood that the distribution of each pose marker pattern is known or predetermined. In some embodiments, the distribution of multiple pose marker patterns or the corner points of multiple pose marker patterns can be as follows: Figure 5The distribution is shown. In some embodiments, the angle around the axis of each pose marker corner marker can also be determined based on formula (13).

[0095] See Figure 7 In step 703, based on the about-axis angles of the multiple pose markers, the three-dimensional coordinates of the multiple pose markers relative to the driven tool coordinate system are determined. In some embodiments, such as Figure 5 As shown, each pose marker corner point is located on the circumference of the cross-sectional circle 520, and the center and radius r of the cross-sectional circle 520 are known. Taking the pose marker corner point P5 as the reference corner point, the three-dimensional coordinates of the pose marker corner point P5 in the driven tool coordinate system {wm} are (r,0,0). In some embodiments, the three-dimensional coordinates of each pose marker corner point in the driven tool coordinate system {wm} can be determined based on the following formula (15):

[0096] C m =[r·cosα] m r·sinα m 0] T (15)

[0097] Among them, C m With pose marker corner point P5 as the first pose marker corner point, the specific angle around the axis of the m-th pose marker corner point can be based on the three-dimensional coordinates of multiple pose markers in the slave tool coordinate system {wm}, following the clockwise direction of the cross-sectional circle 520.

[0098] In some embodiments, the axial angle α of the corner point marker of the m-th pose marker pattern is determined based on formula (13). m Then, the angle α around the axis determined by formula (13) m The three-dimensional coordinates C are determined by formula (15). m .

[0099] Figure 8 A flowchart illustrates a method 800 for determining the three-dimensional coordinates of a plurality of pose markers relative to a driven tool coordinate system according to other embodiments of the present disclosure. Some or all of the steps in method 800 may be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 800 can be implemented by software, firmware, and / or hardware. In some embodiments, method 800 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 17The surgical robot system 1700 is shown. In some embodiments, method 800 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0100] See Figure 8 In step 801, the arrangement order of the multiple pose identifiers is determined based on at least two of them. In some embodiments, the arrangement order of the multiple pose identifiers can be represented by the arrangement order of multiple pose identifier patterns. In some embodiments, the arrangement order of the multiple pose identifiers is determined by identifying any two pose identifier patterns. It should be understood that the multiple pose identifiers include different pose identifier patterns, and when any two pose identifier patterns are known, the arrangement order can be determined based on the known distribution of the multiple pose identifier patterns (e.g., Figure 4 The distribution of different pose marker patterns in label 400 shown, or Figure 5 The distribution of different pose marker patterns in the label 500 shown determines the arrangement order of multiple pose markers in the positioning image, such as clockwise or counterclockwise arrangement.

[0101] See Figure 8 In step 803, the three-dimensional coordinates of the multiple pose markers are determined based on their arrangement order. In some embodiments, based on the known distribution of the multiple pose markers, the three-dimensional coordinates of each pose marker in the slave tool coordinate system can be determined. The three-dimensional coordinates of each pose marker can be represented by the three-dimensional coordinates of the corner points of the pose marker pattern in the slave tool coordinate system, and each pose marker pattern corresponds to a coordinate point in the slave tool coordinate system. After determining the arrangement order of the multiple pose marker patterns, the remaining pose marker patterns can be determined based on the identified pose marker patterns, and thus the three-dimensional coordinates of each pose marker pattern in the slave tool coordinate system can be determined. In some embodiments, multiple pose marker corner points in the positioning image are identified, and any two corresponding pose marker patterns among the multiple pose marker corner points are determined. Based on the two identified pose marker patterns, the arrangement order of the corner points of the multiple pose marker patterns is determined, and thus the three-dimensional coordinates of each pose marker pattern corner point in the slave tool coordinate system can be determined. Furthermore, based on the arrangement order, the distribution of all pose marker patterns can be determined, thereby matching a specific pose pattern matching template with the pose marker pattern at the corresponding position on the positioning image, improving data processing speed. In some embodiments, the pattern matching between the pose pattern matching template and the pattern at the corner of the pose marker pattern can be implemented similarly to step 903 in method 900.

[0102] In some embodiments, the end effector is located at the end of the manipulator arm, therefore the position of the end effector is known or determinable. The pose transformation relationship of the end effector relative to the driven tool coordinate system is also known or predetermined. In some embodiments, taking the reference coordinate system as the world coordinate system as an example, the pose of the end effector of the driven tool relative to the reference coordinate system can be determined based on the following formula (16):

[0103] w R tip = w R wm wm R tip

[0104] w P tip = w R wm wm P tip + w P wm (16)

[0105] in, w R tip This refers to the attitude of the end effector relative to the world coordinate system. w P tip This refers to the position of the end effector relative to the world coordinate system. wm R tip The attitude of the end effector relative to the driven tool coordinate system. wm P tip This represents the position of the end effector relative to the driven tool coordinate system.

[0106] In some embodiments, the attitude of the driven tool coordinate system relative to the world coordinate system is determined based on formula (14). w R wm and location w P wm Then the attitude is determined based on formula (14). w R wm and location w P wm Formula (16) determines the attitude of the end effector relative to the world coordinate system. w R tip and location w P tip .

[0107] This disclosure provides some embodiments of a method for identifying pose markers. Figure 9 A flowchart illustrating a method 900 for identifying pose identifiers according to some embodiments of the present disclosure is shown. Some or all of the steps in method 900 may be controlled by a control device (e.g., Figure 1The control device 120 shown is used to execute the steps. Some or all of the steps in method 900 can be implemented by software, firmware, and / or hardware. In some embodiments, method 900 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 17 The surgical robot system 1700 is shown. In some embodiments, method 900 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0108] refer to Figure 9 In step 901, multiple candidate pose identifiers are determined from the positioning image. In some embodiments, the pose identifier may include a corner point of the pose identifier pattern in the pose identifier pattern. The coordinates or origin of the coordinate system of the candidate pose identifier can be represented by the corner point of the candidate pose identifier pattern. In some embodiments, the corner point of the candidate pose identifier pattern may refer to the possible corner point of the pose identifier pattern obtained after preliminary processing or preliminary identification of the positioning image.

[0109] In some embodiments, method 900 may further include determining a region of interest (ROI) in the localization image. For example, the ROI may be cropped from the localization image, and multiple candidate pose markers may be determined from the ROI. The ROI may be the entire localization image or a partial region. For example, the ROI of the current frame may be cropped based on a region within a certain range of the corner points of multiple pose marker patterns determined in the previous frame (e.g., the localization image of the previous image processing cycle). For localization images that are not the first frame, the ROI may be a region within a certain distance centered on a virtual point formed by the coordinates of the corner points of multiple pose marker patterns from the previous image processing cycle. The certain distance range may be a fixed multiple of the average spacing distance of the corner points of the pose marker patterns, such as twice. It should be understood that the predetermined multiple may also be a variable multiple of the average spacing distance of the corner points of multiple candidate pose marker patterns in the previous image processing cycle.

[0110] In some embodiments, method 900 may further include determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of: image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include: cropping a Region of Interest (ROI) from the localization image and converting the ROI to a corresponding grayscale image.

[0111] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be determined based on the following formula (17):

[0112]

[0113] Where τ is a set constant, for example, set to 2; I x I 45 I y I n45 These are the first derivatives of the pixel in the four directions: 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively.

[0114] In some embodiments, method 900 may further include dividing the ROI into multiple sub-regions. For example, a non-maximum suppression method can be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI can be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the location image or ROI can also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels.

[0115] In some embodiments, method 900 may further include determining the pixel with the largest corner likelihood value in each sub-region to form a pixel set. In some embodiments, the pixel set serves as a plurality of candidate identifiers determined from the localization image. For example, the pixel with the largest CL value in each sub-image may be determined, and the pixel with the largest CL value in each sub-image may be compared with a first threshold to determine a set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold may be set to 0.06. It should be understood that the first threshold may also be set to other values.

[0116] See Figure 9Step 903: Based on multiple different pose pattern matching templates, identify the first pose identifier from the candidate pose identifiers. In some embodiments, the multiple different pose pattern matching templates are matched with the patterns at the corner points of the candidate pose identifier patterns to identify the first pose identifier. For example, the corner points of the candidate pose identifier patterns that meet a preset pose pattern matching degree standard are determined as the corner points of the first pose identifier pattern. In some embodiments, the pose pattern matching template and the pattern in the vicinity of the corner point of the pose identifier pattern have the same or similar features. If the matching degree between the pose pattern matching template and the pattern in the vicinity of the corner point of the candidate pose identifier pattern reaches a preset pose pattern matching degree standard (e.g., the matching degree is higher than a threshold), it can be considered that the pattern in the vicinity of the corner point of the candidate pose identifier pattern has the same or similar features as the pose pattern matching template, and thus the current corner point of the candidate pose identifier pattern can be considered as the corner point of the pose identifier pattern.

[0117] In some embodiments, the pixel with the largest CL value in the pixel set is identified as a candidate pose identifier pattern corner point. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value can be selected as the candidate pose identifier pattern corner point. In some embodiments, after determining the candidate pose identifier pattern corner point, a pose pattern matching template is matched with the pattern at the candidate pose identifier pattern corner point. If a preset pose pattern matching degree standard is met, the candidate pose identifier pattern corner point is determined as the first identified pose identifier pattern corner point.

[0118] In some embodiments, method 900 may further include, in response to a matching failure, determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a candidate pose identifier pattern corner point. For example, if the candidate pose identifier pattern corner point does not meet a preset matching degree standard, then the pixel with the second largest CL value (the pixel with the second largest CL value) is selected as the candidate pose identifier pattern corner point, and the pose pattern matching template is matched with the pattern at the candidate pose identifier pattern corner point, and so on, until the first pose identifier pattern corner point is identified.

[0119] In some embodiments, the pose identification pattern can be a black and white alternating pattern (e.g., a checkerboard pattern), therefore the pose pattern matching template can be the same pattern, utilizing the grayscale distribution G of the pose pattern matching template. M The pixel neighborhood grayscale distribution G of the pixel corresponding to the corner point of the candidate pose identifier pattern image The correlation coefficient (CC) between pixels is used for matching. The grayscale distribution G of the pixel neighborhood is also considered. image This refers to the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The correlation coefficient can be determined based on the following formula (18):

[0120]

[0121] Where Var() is the variance function and Cov() is the covariance function. In some embodiments, when the correlation coefficient is less than 0.8, the gray-level distribution in the pixel neighborhood has a low correlation with the pose pattern matching template. In this case, the candidate pose pattern corner with the highest corner likelihood value is determined not to be a pose pattern corner. Otherwise, the candidate pose pattern corner with the highest corner likelihood value is considered to be a pose pattern corner.

[0122] In some embodiments, method 900 may further include determining the edge orientation of the corner points of the candidate pose identifier pattern. For example, such as Figure 10 As shown, the corner point of the candidate pose identifier pattern is corner point P in pose identifier pattern 1000. 1001 Corner point P 1001 The edge direction can refer to the direction of the corner point P. 1001 The direction of the edge, such as Figure 10 The direction indicated by the dashed arrow.

[0123] In some embodiments, the edge direction can be determined by the first-order derivative (I0) of each pixel in the X and Y directions of the planar coordinate system with respect to a certain neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate pose identifier pattern. x and I y The edge direction can be determined based on the following formula (19):

[0124]

[0125] Among them, the first derivative (I) x and I y This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by performing a convolution operation on the edge direction I of each pixel within the neighborhood range. angle and the corresponding weight I weight Clustering calculations are performed to obtain the edge direction of the pixel, and weight I is selected. weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if multiple edge directions exist, then weight I is selected. weight The I corresponding to the largest proportion of multiple classes angle As the edge direction.

[0126] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0127] In some embodiments, method 900 may further include rotating the pose pattern matching template based on the edge direction. Rotating the pose pattern matching template according to the edge direction allows the template to be aligned with the image at the corner point of a candidate pose identifier pattern. The edge direction of the corner point of the candidate pose identifier pattern can be used to determine the orientation of the image at that corner point in the positioning image. In some embodiments, rotating the pose pattern matching template according to the edge direction adjusts it to be the same as or nearly the same as the image orientation at the corner point of the candidate pose identifier pattern to facilitate image matching.

[0128] See Figure 9 Step 905: Starting from the first pose identifier, search for pose identifiers. For example, Figure 11 A flowchart illustrating a method 1100 for searching pose identifiers according to some embodiments of the present disclosure is shown. Some or all of the steps in method 1100 may be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 1100 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1100 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 17 The surgical robot system 1700 is shown. In some embodiments, method 1100 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0129] See Figure 11In step 1101, the second pose identifier is searched starting from the first pose identifier. In some embodiments, the corner point of the first pose identifier pattern is used as the starting point, and the corner point of the second pose identifier pattern is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following directions: directly in front of the corner point of the first pose identifier pattern (corresponding to the 0° angle direction), directly behind (corresponding to the 120° angle direction), directly above (90° angle direction), directly below (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0130] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions, with each search direction v sn It can be determined based on the following formula (20):

[0131] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (20)

[0132] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent pose identifier corner points among the multiple pose identifier corner points determined in the previous frame. For example, the predetermined search direction is determined based on the following formula (21):

[0133]

[0134] Among them, (x j ,y j ) represents the two-dimensional coordinates of the corner points of multiple pose marker patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple pose marker patterns determined in the previous frame; v s1 The first set search direction; v s2 This is the second search direction set.

[0135] In some embodiments, such as Figure 12 As shown, the first pose is used to identify the corner point P of the pattern. 1201 Using the coordinates of the given location as the starting point, search for the corner point P of the second pose marker pattern in the set search direction. 1202 The coordinate position can specifically include: identifying the corner point P of the pattern using the first pose. 1201 Using the coordinates as the starting point for the search, the search box (e.g., ...) Figure 12 (The dashed box in the image) moves in the set search direction V with a certain search step size. 1201 Search for the corner points of the pose marker pattern. If there is at least one candidate corner point of the pose marker pattern within the search box, then the candidate corner point with the highest likelihood value within the search box is selected as the second pose marker pattern corner point P.1202 With the search box limited to a suitable size, the first pose is used to identify the corner point P of the pattern. 1201 The coordinates of the point are used as the starting point for the second pose identification pattern corner point P. 1202 During the search, the candidate pose marker corner with the highest corner likelihood value among the candidate pose marker corners appearing in the search box is more likely to be the actual pose marker corner. Therefore, it can be considered that the candidate pose marker corner with the highest corner likelihood value in the search box is the second pose marker corner P. 1202 To improve data processing speed. In other embodiments, to improve the accuracy of pose marker pattern corner point recognition, when at least one candidate pose marker pattern corner point exists in the search box, the candidate pose marker pattern corner point with the highest corner point likelihood value among the candidate pose marker pattern corner points appearing in the search box is selected for corner point recognition to determine whether the candidate pose marker pattern corner point with the highest corner point likelihood value is a pose marker pattern corner point. For example, the pose pattern matching template is matched with the image within a certain range of the candidate pose marker pattern corner point with the highest corner point likelihood value. The candidate pose marker pattern corner point that meets the preset pose pattern matching degree standard can be considered as the searched second pose marker pattern corner point P. 1202 .

[0136] In some embodiments, continue reading Figure 12 The size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0137] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, and pattern matching can be performed based on the correlation coefficient in formula (18). If the correlation coefficient is greater than the threshold, the candidate pose identification pattern corner with the largest corner likelihood value is considered to be the pose identification pattern corner, and is denoted as the second pose identification pattern corner.

[0138] Figure 13 A flowchart is shown of a method 1300 for searching a second pose identifier according to some embodiments of the present disclosure. Some or all of the steps in method 1300 may be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 1300 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1300 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 17The surgical robot system 1700 is shown. In some embodiments, method 1300 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium. In some embodiments, step 1101 in method 1100 can be implemented similarly to method 1300.

[0139] See Figure 13 In step 1301, starting from the first pose identifier, candidate pose identifier pattern corner points of the second pose identifier are searched. In some embodiments, the search for candidate pose identifier pattern corner points of the second pose identifier can be combined with... Figure 12 The search is shown for the corner point P of the second pose identifier pattern. 1202 Similarly, implement it.

[0140] In step 1303, based on the distribution of multiple pose identifiers, a first pose pattern matching template and a second pose pattern matching template are determined. The first pose pattern matching template and the second pose pattern matching template correspond to pose identifiers adjacent to the first pose identifier. In some embodiments, step 1303 can be performed before or after step 1301, or step 1303 can be performed synchronously with step 1301. In some embodiments, the pose identifier patterns included in the pose identifiers included in the first pose identifier and the distribution of multiple pose identifier patterns can be used to determine the pose identifier patterns included in the pose identifiers adjacent to the first pose identifier, thereby determining the first pose pattern matching template and the second pose pattern matching template.

[0141] In step 1305, the first pose pattern matching template and / or the second pose pattern matching template are matched with the patterns at the corner positions of the candidate pose identifier patterns of the second pose identifier to identify the second pose identifier. In some embodiments, the first pose pattern matching template and / or the second pose pattern matching template can be matched with the patterns at the corner positions of the candidate pose identifier patterns of the second pose identifier based on the correlation coefficient in formula (18). If the correlation coefficient is greater than a threshold, the corner points of the candidate pose identifier patterns of the second pose identifier are determined as the corner points of the pose identifier patterns of the second pose identifier, and the patterns corresponding to the pose pattern matching templates (first pose pattern matching template or second pose pattern matching template) with a correlation coefficient greater than the threshold are determined as the pose identifier patterns of the second pose identifier.

[0142] See Figure 11In step 1103, a search direction is determined based on the first pose identifier and the second pose identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of a corner point of the first pose identifier pattern and moving away from the corner point of the second pose identifier pattern. The second search direction may be a direction starting from the coordinate position of a corner point of the second pose identifier pattern and moving away from the corner point of the first pose identifier pattern. For example, Figure 12 The search direction V shown 1202 .

[0143] See Figure 11 In step 1105, starting with either the first pose identifier or the second pose identifier, a search for pose identifiers is performed in the search direction. In some embodiments, if the first pose identifier pattern corner point is used as the new starting point, the first search direction described above can be used as the search direction for the pose identifier pattern corner point. If the second pose identifier pattern corner point is used as the new starting point, the second search direction described above can be used as the search direction for the pose identifier pattern corner point. In some embodiments, a new pose identifier pattern corner point is searched (e.g., Figure 12 The third pose marker pattern corner point P in 1203 This can be performed similarly to step 1101 in method 1100 or method 1400. In some embodiments, the search step size can be the first pose identifier pattern corner point P. 1201 Second pose identifier pattern corner point P 1202 The distance between them is L1.

[0144] Figure 14 A flowchart illustrating a method 1400 for searching pose identifiers according to some embodiments of the present disclosure is shown. Some or all of the steps in method 1400 may be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 1400 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1400 can be used in a robot system, for example, Figure 1 The robot system 100 shown or Figure 17 The surgical robot system 1700 is shown. In some embodiments, method 1400 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium. In some embodiments, step 1105 in method 1100 can be implemented similarly to method 1400.

[0145] See Figure 14In step 1401, starting from the first pose identifier or the second pose identifier, candidate pose identifier pattern corner points of the third pose identifier are searched. In some embodiments, the search for candidate pose identifier pattern corner points of the third pose identifier can be combined with... Figure 12 The search is shown for the corner point P of the third pose identifier pattern. 1203 Similarly, implement it.

[0146] In step 1403, a third pose pattern matching template is determined based on the distribution of multiple pose identifiers. The third pose pattern matching template corresponds to a pose identifier adjacent to the first pose identifier or adjacent to the second pose identifier. In some embodiments, the pose identifier pattern included in the pose identifier pattern included in the first pose identifier or the second pose identifier and the distribution of multiple pose identifier patterns can be used to determine the pose identifier pattern included in the pose identifier adjacent to the first pose identifier or the second pose identifier, thereby determining the third pose pattern matching template.

[0147] In step 1405, the third pose pattern matching template is matched with the pattern at the corner position of the candidate pose identifier pattern of the third pose identifier to identify the third pose identifier. In some embodiments, step 1405 can be implemented similarly to step 1305.

[0148] In some embodiments, in response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate pose identifier pattern corner point; and multiple different pose pattern matching templates are matched with the patterns at the corner point positions of the candidate pose identifier pattern to identify the first pose identifier. In some embodiments, after determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a new candidate pose identifier pattern corner point, a new first pose identifier can be identified based on a method similar to step 903. In some embodiments, a search distance greater than a search distance threshold can be understood as a search distance greater than a search distance threshold in some or all search directions. In some embodiments, the search distance threshold may include a set multiple of the distance between the (N-1)th pose identifier pattern corner point and the (N-2)th pose identifier pattern corner point, where N≥3.

[0149] For example, the search distance threshold is twice the distance between the first two pose marker corner points. Thus, the maximum search distance for the third pose marker corner point is twice the distance between the first and second pose marker corner points. If no pose marker corner point is found within this search distance in the search direction, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is identified as a new candidate pose marker corner point, and a new first pose marker is identified. The current search process then stops. In some embodiments, similar to method 900, a new first pose marker corner point can be determined, and similar to method 1100, the remaining pose marker corner points can be searched starting from this new corner point.

[0150] In some embodiments, in response to the number of identified pose marker pattern corner points being greater than or equal to a pose marker number threshold, the current relative pose of the manipulator relative to the reference coordinate system can be determined based on the search for the pose markers, and the search for pose marker pattern corner points will stop accordingly. For example, when four pose marker pattern corner points are identified, the search for pose marker pattern corner points will stop.

[0151] In some embodiments, in response to the number of identified pose identifiers being less than a pose identifier number threshold, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate pose identifier pattern corner point; and multiple different pose pattern matching templates are matched with the patterns at the corner point positions of the candidate pose identifier patterns to identify the first pose identifier. In some embodiments, if the total number of identified pose identifiers (e.g., pose identifier pattern corner points) is less than a set pose identifier number threshold, the search based on the first pose identifier in the above steps is considered to have failed. In some embodiments, in the case of search failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate pose identifier pattern corner point, and then a new first pose identifier can be identified based on a method similar to step 903. In some embodiments, similar to method 900, a new first pose identifier pattern corner point can be re-determined, and similar to method 1100, the remaining pose identifier pattern corner points can be searched starting from the new pose identifier pattern corner point.

[0152] In some embodiments, after the corner points of the pose marker pattern are searched or identified, sub-pixel positioning can be performed on the determined corner points of the pose marker pattern to improve the positional accuracy of the corner points of the pose marker pattern.

[0153] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of the corner points of the pose identifier pattern after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, the extreme points of which are subpixel points. The fitting function can be determined based on the following formulas (22) and (23):

[0154] S(x,y)=ax 2 +by 2 +cx+dy+exy+f (22)

[0155]

[0156] Where S(x,y) is the fitting function for the CL values ​​of all pixels in each ROI, and a, b, c, d, e, and f are coefficients; x c The x-coordinate and y-coordinate of the pose identifier c The y-coordinate is the pose identifier.

[0157] Figure 15 A schematic diagram of a master operator 1500 according to some embodiments of this disclosure is shown. For example... Figure 15 As shown, in some embodiments, the main manipulator 1500 includes a multi-degree-of-freedom robotic arm 1510 and a handle 1520. The multi-degree-of-freedom robotic arm 1510 includes multiple joints (15101-15107). The joints of the multi-degree-of-freedom robotic arm 1510 include position joints and attitude joints. The attitude joints serve as the orientation module of the main manipulator 1500, controlling the handle 1520 to achieve a target attitude through one or more attitude joints. The position joints serve as the positioning module of the main manipulator 1500, controlling the handle 1520 to achieve a target position through one or more position joints.

[0158] In some embodiments, determining the current orientation of the handle of the master operator includes: obtaining joint information of at least one orientation joint; and determining the current orientation of the master operator based on the joint information of at least one orientation joint.

[0159] In some embodiments, master manipulator sensors are disposed at the posture joints of a multi-degree-of-freedom robotic arm to acquire joint information (such as angles) corresponding to the posture joints, and to determine the current posture of the master manipulator's handle relative to the master manipulator's base coordinate system based on the acquired joint information. In some embodiments, joint information is acquired through the master manipulator sensors at the posture joints, and the current posture of the master manipulator is calculated based on a forward kinematics algorithm. In some embodiments, the master manipulator includes at least one posture joint for controlling the posture of the master manipulator's handle, and the control signal includes a control signal for controlling one or more of the at least one posture joint. By adjusting one or more posture joints, the posture of the master manipulator's handle is adjusted, achieving posture matching between the master manipulator's handle and the driven tool.

[0160] In some embodiments, the control signals include control signals for controlling one or more of at least one attitude joint, wherein the one or more of the at least one attitude joint includes uncoupled attitude joints. Coupled joints may refer to joints used to adjust the position and attitude of the master manipulator. Uncoupled joints may refer to joints that can only be used to adjust the position (referred to in this disclosure as uncoupled position joints) or attitude of the master manipulator (referred to in this disclosure as uncoupled attitude joints). In some embodiments, the master manipulator may include at least one coupled joint. For example, Figure 15 The master manipulator 1500 shown has a first joint 15101, a second joint 15102, and a third joint 15103 that are position joints, and a first joint 15101, a second joint 15102, a fifth joint 15105, a sixth joint 15106, and a seventh joint 15107 that are attitude joints. The first joint 15101 and the second joint 15102 are coupled joints that can adjust both the position and attitude of the master manipulator 1500, while the fifth joint 15105, the sixth joint 15106, and the seventh joint 15107 are uncoupled attitude joints that can only adjust the attitude of the master manipulator 1500. In some embodiments, the attitude adjustment of the handle 1520 of the master operator 1500 can be achieved by calculating the control signals of the uncoupled attitude joints (e.g., the fifth joint 15105, the sixth joint 15106, and the seventh joint 15107), thereby achieving attitude matching between the handle 1520 of the master operator 1500 and the driven tool, and providing conditions for subsequent teleoperation.

[0161] In some embodiments, the base coordinate system of the main actuator is b, and the coordinate system of the handle is d. In some embodiments, the base coordinate system b is a coordinate system established with the base as a virtual point, and its orientation can be determined based on its physical structure. Similarly, the coordinate system d of the handle is a coordinate system established with the handle as a virtual point, and its orientation can be determined based on its physical structure. In some embodiments, the origin of the coordinate system d of the handle can coincide with the origin of the coordinate systems of the fifth, sixth, and seventh joints. Those skilled in the art will understand that the position and orientation of the coordinate system d of the handle relative to the base coordinate system of the main actuator can be determined by the joint information of the first to seventh joints.

[0162] In some embodiments, the master actuator sensor acquires the joint information q of the master actuator. j_mp (j is the joint number). In some embodiments, the information q of the j-th joint... j_mp This can include the angle value θ of the corresponding joint. j_mp For example, obtaining the joint information q of the first joint. 1_mp Joint information q of the second joint 2_mp Joint information q of the third joint 3_mp Joint information q of the fourth joint 4_mp Joint information q of the fifth joint 5_mp Joint information q of the sixth joint 6_mp Joint information q of the seventh joint 7_mp In some embodiments, the fourth joint is a driven joint of the third joint, and the absolute values ​​of the joint angles of the fourth joint are the same as those of the third joint, but the directions are opposite. Therefore, the angles of the six joints of the master manipulator are represented by a 6*1 matrix q. manipulator The joint angle of the fourth joint may not be in matrix q. manipulator This is reflected in the information q of each joint. j_mp It can be represented as θ j_mp The main manipulator has six degrees of freedom, as shown in formula (24):

[0163] q manipulator =(q 1_mp q 2_mp q 3_mp q 5_mp q 6_mp q 7_mp ) T (twenty four)

[0164] The first, second, and third joints are position joints, q 1_mp q 2_mp q 3_mp This determines the position of the main manipulator's handle. Joints one, two, five, six, and seven are attitude joints.1_mp q 2_mp q 5_mp q 6_mp q 7_mp The orientation of the handle is determined. In some embodiments, determining the orientation of the main controller's handle may disregard the positions controlled by the first, second, and third joints, and focus instead on the orientation (e.g., direction) determined by the first, second, fifth, sixth, and seventh joints. In some embodiments, during motor drive, the first, second, and third joints are kept stationary, and the q corresponding to the fifth, sixth, and seventh joints is determined based on the target pose or target orientation. 5_mp q 6_mp q 7_mp According to q 5_mp q 6_mp q 7_mp Calculate control signals to adjust the handle's posture.

[0165] Those skilled in the art will understand that there are many solutions for a multi-joint master manipulator to achieve a certain target posture. In some embodiments, the posture of the master manipulator handle can be adjusted by adjusting one or more of the at least one posture joint. For example, in one embodiment, the posture of the master manipulator handle can be adjusted by keeping the coupled posture first joint, coupled posture second joint, and uncoupled posture third joint unchanged, and adjusting the uncoupled posture fifth joint, sixth joint, and seventh joint.

[0166] In some embodiments, method 600 further includes: obtaining joint information of other posture joints besides the one or more posture joints to be adjusted in at least one posture joint; and determining the transformation matrix of the other posture joints based on the joint information of the other posture joints. For example, the joint information of the other posture joints is obtained based on the master manipulator sensor, and the transformation matrix of the other posture joints is determined based on the joint information of the other posture joints. The joint information of the coupled posture first joint and second joint can be obtained, and the transformation matrix is ​​calculated. In some embodiments, one or more uncoupled posture joints (e.g., the fifth joint, the sixth joint, and the seventh joint) can be adjusted without adjusting other posture joints, such as coupled joints (e.g., the first joint and the second joint). In some embodiments, the transformation matrix can be determined based on the q corresponding to the first joint and the second joint. 1_mp and q 2_mp Determine the transformation matrices for other pose joints (e.g., the transformation matrices for other pose joints relative to joint origin 0). 0R4). In some embodiments, method 600 further includes generating the control signal of the master manipulator based on the transformation matrix of the target pose of the master manipulator's handle and the other pose joints. For example, the transformation matrix of the target pose of the master manipulator's handle and the other pose joints. 0 R4 generates the control signals for the master operator, as shown in formulas (25) to (27).

[0167] In some embodiments, the third and fourth joints are uncoupled position joints, based on q 1_mp q 2_mp q 3_mp Transformation matrices for other pose joints 0 R4 and q-based 1_mp q 2_mp Transformation matrices for other pose joints 0 R4 is the same.

[0168] 4 R7 = 0 R4 T · b R0 T · b R d · 7 R d T (25)

[0169] In formula (25), the transformation matrix 0 R4 is derived from input q 1_mp q 2_mp Or q 1_mp q 2_mp q 3_mp Let b be the base coordinate system of the master operator, and d be the coordinate system of the master operator's handle. b R d The attitude of the master controller handle relative to the master controller base coordinate system. b R0 represents the existing angular relationship between the base and the joint starting point, and is a structural constant. 7 R d The existing angular relationship between the seventh joint and the handle is a structural constant.

[0170] 4 R7 = 4 R5· 5 R6· 6 R7 (26)

[0171] R(q 5_mp ,q 6_mp ,q 7_mp )= 0 R4 T ·b R0 T ·R t · 7 R d T (27)

[0172] In formula (27), R t The current posture of the driven tool and its relationship with b R d Similarly, in formula (26), 4 R5 5 R6 and 6 R7 corresponds to the unsolved quantity q 5_mp q 6_mp q 7_mp Based on the obtained q 5_mp q 6_mp q 7_mp A control signal is determined, and the attitude of the master manipulator is adjusted based on the control signal to achieve master-slave attitude matching. Those skilled in the art will understand that R... t This could be the current pose of the end effector of the driven tool relative to the base coordinate system of the driven tool, or the current pose of the image of the end effector of the driven tool on the display relative to the world coordinate system. t Can be with b R d Consistent, for example, identical or having a specific ratio or difference. In some embodiments, the joint target values ​​of one or more posture joints of the handle are determined according to the control signal, and the joint target values ​​are converted into drive quantities and sent to the drive device. The drive device drives the motors of one or more posture joints of the master manipulator to move, thereby achieving the matching of the posture of the handle of the master manipulator with the posture of the end effector of the driven tool.

[0173] In some embodiments, the mathematical structure model of the master operator can be constructed based on the DH parameter method or the exponential product representation method. For example, the DH matrix corresponding to the joints of the master operator is determined, and the mathematical structure model of the master operator is determined based on the DH matrix of the joints. The DH matrix of each joint of the master operator is represented by formula (28).

[0174]

[0175] The correspondence between the DH matrix and joint information is shown in Table 1.

[0176] Table 1. Correspondence between DH matrix and joint information

[0177]

[0178] In formula (28), Rot(x,α)j_mp ) represents a rotation α about the x-axis j_mp Angle, Rot(z,θ) j_mp ) represents a rotation θ about the z-axis j_mp Angle, Trans(x,a) j_mp Move a in the x direction j_mp Trans(z,d) j_mp Move d in the z direction j_mp .like Figure 15 The master manipulator 1500 shown has its z-axis as the rotation axis of this joint, and its x-axis pointing to the next joint. The y-axis direction can be determined using the left / right-hand rule of the Cartesian coordinate system. Rot(x,α) j_mp ), Trans(x,a j_mp A fourth-order matrix represents a rotation around a direction by a certain angle or a translation along a direction by a certain distance.

[0179] In some embodiments, the mathematical structural model of the master operator is described by multiplying the DH matrices of all joints, as shown in equation (29):

[0180] 0 T 7_mp = 0 T 1_mp · 1_mp T 2_mp · 2_mp T 3_mp · 3_mp T 4_mp · 4_mp T 5_mp · 5_mp T 6_mp · 6_mp T 7_mp (29)

[0181] In some embodiments, the DH matrix of the joint in formula (29) can be determined based on formula (28).

[0182] Those skilled in the art will understand that when initiating teleoperation, if the handle's posture (e.g., orientation or angle) is inconsistent with the posture (e.g., orientation or angle) of the corresponding controlled slave tool, the operator's (e.g., surgeon's) human-computer interaction experience during operation will be poor, affecting the accuracy of the slave tool's operation. Therefore, after the master operator and slave tool are matched and connected, and before the master operator performs teleoperation on the slave tool (e.g., when the operator holds the master operator's handle to gain control of the corresponding slave tool but has not yet started master-slave teleoperation), the handle's posture and the slave tool's posture are matched and adjusted. When their postures are consistent, the master operator can perform teleoperation on the slave tool, improving the accuracy and experience of subsequent teleoperations.

[0183] In some embodiments, method 600 further includes: determining the attitude matching degree between the handle of the master operator and the driven tool in response to the satisfaction of a predetermined condition. In some embodiments, the predetermined condition includes the triggering of remote operation control. In some embodiments, the triggering of remote operation control can be achieved through a triggering device. The triggering device may be a switch located on the master operator or display for easy access, touch, press, or slide by the operator. Triggering methods include, but are not limited to, maintaining proximity, touching, sliding, tapping, or long-pressing. The triggering method of the triggering device may be proximity to a sensor, flipping a switch on the master operator, touching a sensing position on the master operator, long-pressing or tapping a button on the master operator, stepping on the foot pedal of the main control panel, operating the display screen of the main control panel, etc. In some embodiments, matching means that the attitude of the handle and the attitude of the driven tool meet a preset relationship (e.g., consistency), and the attitude matching degree refers to the degree of matching between the current attitude of the handle and the current attitude of the driven tool. In some embodiments, the attitude matching degree between the master operator and the driven tool is determined based on the current attitude of the handle of the master operator and the current attitude of the driven tool. When the attitude matching degree is lower than a preset threshold, a control signal is generated to adjust the current attitude of the master controller's handle so that the attitude matching degree is higher than or equal to the preset threshold. This allows for automatic attitude adjustment to achieve consistency between the two attitudes when they do not match. When the current attitudes of the two are consistent or nearly consistent (attitude matching degree higher than or equal to the preset threshold), a master-slave mapping is established between the master controller and the slave tool, allowing the next teleoperation procedure to be executed.

[0184] In some embodiments, adjusting the orientation of the master controller's handle to match the orientation of the slave tool includes: keeping the current orientation of the slave tool unchanged, and adjusting the orientation of the master controller's handle to make the orientation of the master controller's handle match the orientation of the slave tool.

[0185] In some embodiments, the target posture of the master controller's handle is consistent with the current posture of the slave tool, establishing a master-slave mapping between the master controller and the slave tool. This allows the master controller to perform teleoperations on the slave tool, improving the accuracy and user experience of teleoperation. Those skilled in the art will understand that posture consistency means that the postures are essentially the same. There may be some error between the target posture of the master controller's handle and the current posture of the slave tool, but the error must be within an acceptable range.

[0186] In some of the above embodiments, the orientation of the handle is matched with the orientation of the slave tool before teleoperation. When the operator begins operation (e.g., pressing the clamp button on the master operator's handle), the master-slave mapping can be quickly established, and the master operator and the slave tool enter teleoperation mode. Furthermore, only the current orientation of the slave tool is maintained; the operator can still move the master operator's handle to a suitable position before performing teleoperation matching even when not in operation, greatly increasing the movement range of the master operator's handle. Moreover, the master-slave motion control method provided above is applicable to various slave ends with different principles and forms, and the calculation process is highly targeted and computationally inefficient, also reducing the driving force required to adjust the master operator's handle to the target orientation.

[0187] In some of the above embodiments, by establishing a connection between the master operator and the slave tool and transferring control, the attitude matching degree between the master operator's handle and the slave tool is determined under the connected and control-transferred state. If the attitude matching degree meets a preset threshold condition, a master-slave mapping is established between the master operator and the slave tool, and teleoperation steps are executed. If the attitude matching degree does not meet the preset threshold condition, the attitude of the master operator's handle needs to be adjusted to match the current attitude of the slave tool before establishing the master-slave mapping between the master operator and the slave tool, and teleoperation is executed through the master operator's handle. Adjusting the attitude of the master operator's handle to match the attitude of the slave tool before establishing the teleoperation relationship ensures the accuracy of the master-slave mapping between the master operator's handle and the slave tool, improves the operator's experience during teleoperation, achieves high-precision matching between the operational actions and actual actions, and avoids operational limitations caused by inconsistent motion control boundaries between the master operator and the slave tool.

[0188] In some of the above embodiments, when the controlled object of the master manipulator (e.g., the slave tool) is changed, the orientation of the front end of the slave tool entering the machine may be different from the current orientation of the handle of the master manipulator. The method provided in this disclosure can adjust the posture of the handle of the master manipulator to match the current posture of the slave tool before the master manipulator and the slave tool establish a master-slave mapping relationship and before the operator actually operates, so as to achieve a good operating experience for the operator and a high-precision match between the expected action and the actual action, while avoiding the operational limitations caused by the inconsistency of the motion control boundaries between the master manipulator and the slave tool.

[0189] In some embodiments of this disclosure, a computer device is also provided, including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 6 , Figure 7 , Figure 8, Figure 9 , Figure 11 , Figure 13 and Figure 14 Some or all of the steps in the method disclosed herein.

[0190] Figure 16 A schematic block diagram of a computer device 1600 according to some embodiments of the present disclosure is shown. See also Figure 16 The computer device 1600 may include a central processing unit (CPU) 1601, a system memory 1604 including random access memory (RAM) 1602 and read-only memory (ROM) 1603, and a system bus 1605 connecting the various components. The computer device 1600 may also include input / output devices 1606 and a mass storage device 1607 for storing the operating system 1613, application programs 1614, and other program modules 1615. The input / output device 1606 includes an input / output controller 1610, primarily composed of a display 1608 and input devices 1609.

[0191] Mass storage device 1607 is connected to central processing unit 1601 via a mass storage controller (not shown) connected to system bus 1605. Mass storage device 1607 or computer-readable media provides non-volatile storage for computer devices. Mass storage device 1607 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0192] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include read-only memory, random access memory, flash memory or other solid-state storage technologies, read-only optical discs or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0193] Computer device 1600 can be connected to network 1612 via network interface unit 1611 connected to system bus 1605.

[0194] The system memory 1604 or mass storage device 1607 is also used to store one or more instructions. The central processing unit 1601 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0195] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 11 , Figure 13 and Figure 14 Some or all of the steps in the disclosed method. Examples of computer-readable storage media include memory for computer programs (instructions), such as read-only memory, random access memory, read-only optical disk, magnetic tape, floppy disk, and optical data storage devices.

[0196] Figure 17 A schematic diagram of a surgical robot system 1700 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 17 The surgical robot system 1700 may include a surgical tool 1701, a main control carriage 1702, and a surgical carriage 1703. A drive module is mounted on the surgical carriage 1703 to drive the surgical tool 1701, which is mounted on the surgical carriage 1703 and connected to the drive module. The main control carriage 1702 is communicatively connected to the surgical carriage 1703 and is used to control the surgical tool 1701 to perform surgical operations. In some embodiments, the controller in the main control carriage 1702 or the controller in the surgical carriage 1703 may be used to execute some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 11 , Figure 13 and Figure 14 Some or all of the steps in the disclosed method. In some embodiments, the main control carriage 1702 and the operating carriage 1703 are connected by wired or wireless transmission. For example, the main control carriage 1702 and the operating carriage 1703 can be connected by a cable.

[0197] In some embodiments, the surgical tool 1701 includes a manipulator arm and an end effector disposed at the end of the manipulator arm. In some embodiments, the surgical robot system 1700 may include a surgical cart 1703. In some embodiments, the surgical robot system 1700 may include at least two surgical carts 1703, each surgical cart 1703 mounting a surgical tool 1701. In some embodiments, the surgical robot system 1700 may further include an imaging tool 1704. The imaging tool 1704 may include a manipulator arm and an imaging module disposed at the end of the manipulator arm. The imaging tool 1704 may be disposed on the surgical cart 1703 and driven by a corresponding drive module. Images of the manipulator arm and its end effector of the surgical tool 1701 acquired by the imaging module may be transmitted to the master control cart 1702. In some embodiments, a portion of the surgical tool 1701 or a portion of the imaging tool 1704 may serve as a driven tool. In some embodiments, the master control cart 1702 includes a master manipulator for remotely operating the surgical tool 1701 or the imaging tool 1704. In some embodiments, the surgical tool 1701 is, for example, a Figure 18 The surgical instrument 1800 is shown in the image. In some embodiments, the main control carriage 1702 is, for example, a surgical instrument 1800. Figure 19 The main control carriage 1900 is shown in the figure. In some embodiments, the surgical carriage 1703 is, for example, a... Figure 20 The surgical cart 2000 shown in the image.

[0198] Figure 18 A schematic diagram of a surgical tool 1800 according to some embodiments of the present disclosure is shown. See also: [Image of embodiments of the present disclosure] Figure 18 The surgical tool 1800 includes a drive transmission device 1890, an operating arm 1840, and an end effector 1860 disposed at the end of the operating arm. In some embodiments, the drive transmission device 1890 can cooperate with a drive module to drive the operating arm 1840 to move. The drive transmission device 1890 is used to connect to the drive module, and the driving force of the drive module is transmitted to the operating arm 1840 through the drive transmission device 1890, thereby driving the operating arm 1840 to achieve multi-degree-of-freedom movement. The drive module can also control the end effector 1860 to perform surgical operations. In some embodiments of this disclosure, the end effector 1860 may include, but is not limited to, a bipolar curved dissecting forceps actuator, a bipolar curved grasping forceps actuator, a unipolar curved scissors actuator, a unipolar electric hook actuator, a bipolar grasping forceps actuator, a needle holder actuator, and a tissue grasping forceps actuator. In some embodiments, the surgical tool 1800 may be mounted, for example, on a drive transmission device 1890. Figure 17 The surgical cart 1703 shown in the image or Figure 20 The surgical cart 2000 shown in the image.

[0199] Figure 19 A schematic diagram of a main control carriage 1900 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 19 The main control carriage 1900 includes: a controller (which can be configured on a computer device and is located inside the main control carriage 1900), a main operator 1901, a main control carriage display (e.g., displays 1902-1904), and pedals (e.g., pedals 1905-1907). The controller is communicatively connected to the main operator 1901, the main control carriage display, and the pedals, respectively, for signal interaction with the main operator 1901, the main control carriage display, and the pedals, and for generating corresponding control commands based on collected control information. In some embodiments, the controller is also communicatively connected to a surgical carriage, for example, with... Figure 17 The surgical cart 1703 shown is connected for communication and is used to control the surgical instrument 1701 to perform surgical operations or to control the imaging instrument 1704 to operate. In some embodiments, the controller of the main control cart 1900 can also be used to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 11 , Figure 13 and Figure 14 Some or all of the steps in the method disclosed herein.

[0200] In some embodiments, the master manipulator 1901 typically includes a left master manipulator (e.g., for controlling the first manipulator arm) for operation with the left hand of the medical worker and a right master manipulator (e.g., for controlling the second manipulator arm) for operation with the right hand. In practical scenarios, the master manipulator 1901 is used to collect the operation input of the medical worker, who then remotely operates the master manipulator 1901 to control the movement of surgical or imaging tools within the operating area to perform medical operations. In some embodiments, the master manipulator 1901 includes a multi-degree-of-freedom robotic arm 19011, with a master manipulator sensor located at each joint of the multi-degree-of-freedom robotic arm 19011. Joint information (such as joint angle data) is generated through the master manipulator sensor at each joint. In some embodiments, the multi-degree-of-freedom robotic arm 19011 has six degrees of freedom. In some embodiments, the pose of the master manipulator 1901 can be represented by a set of joint information of the master manipulator joints (e.g., a one-dimensional matrix composed of this joint information). In some embodiments, the master manipulator 1901 also includes a clamp 19012, which can be used to control the opening and closing angle of the end effector. In some embodiments, the master operator 1901 may specifically be Figure 15The main operator 1500 is shown. In some embodiments, the main control carriage display includes a stereoscopic display 1902, a main control external display 1903, and a main control touch display 1904. The stereoscopic display 1902 displays surgical images and system status prompts, the main control external display 1903 displays surgical images and system status prompts, and the touch display 1904 displays the software user interface of the main control carriage 1900. In some embodiments, the images displayed by the stereoscopic display 1902 or the main control external display 1903 can be determined based on images acquired by the imaging module, for example... Figure 20 The imaging module 2060b is shown in the figure. In some embodiments, the main control carriage pedal is used to collect input from the feet of medical staff and includes structures such as an electrocautery pedal 1905, an electrocoagulation pedal 1906, and a clutch pedal 1907.

[0201] Figure 20 A schematic diagram of a surgical cart 2000 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 20 The operating trolley 2000 includes: a controller (which can be configured on a computer device and is located inside the operating trolley 2000), an operating trolley chassis 2002, an operating trolley housing 2003, a system status display 2005, a main column 2006, a main crossbeam 2007, positioning arms 2008, and a drive module 2009. The operating trolley chassis 2002 is used to enable the moving and fixing of the operating trolley 2000. The operating trolley housing 2003 integrates the electrical components of the operating trolley. The system status display 2005 displays the operating trolley system user interface and receives user input. The main column 2006 is height-adjustable, and its top is fixed to the main crossbeam 2007. The end of the main crossbeam 2007 has a crossbeam platform, and multiple positioning arms 2008 are fixed to the lower end of the crossbeam platform. The positioning arm 2008 is equipped with a drive module 2009, which is used to load surgical tools 2001 or imaging tools 2004 (the imaging tool 2004 may be, for example, a 3D electronic endoscope). In some embodiments, the operating cart 2000 integrates multiple positioning arms 2008, each positioning arm 2008 having multiple motion joints. In some embodiments, the operating cart 2000 integrates multiple surgical tools 2001 and imaging tools 2004, with some operating arms 2040a and end instruments 2060a of the multiple surgical tools 2001 and some operating arms 2040b and imaging modules 2060b of the imaging tools 2004 entering the workspace via a sheath 2010. In some embodiments, the controller of the operating cart 2000 can also be used to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 11 , Figure 13 and Figure 14 Some or all of the steps in the method disclosed herein.

[0202] Note that the above are merely exemplary embodiments and technical principles of this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, this disclosure is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this disclosure, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for controlling master-slave motion, comprising: Acquire the location image; In the positioning image, multiple pose markers located on the slave tool are identified, and the multiple pose markers include different pose marker patterns; Based on the multiple pose identifiers, the current pose of the driven tool relative to the reference coordinate system is determined; Based on the current posture of the driven tool, determine the target posture of the master operator's handle; as well as Based on the target posture of the handle of the main operator, a control signal for the main operator is generated; Determining the current pose of the follower tool relative to the reference coordinate system based on the multiple pose identifiers includes: Based on the distribution of the multiple pose markers, the axial angles of the multiple pose markers relative to the Z-axis of the driven tool coordinate system are determined. Based on the axial angles of the plurality of pose markers, the three-dimensional coordinates of the plurality of pose markers relative to the driven tool coordinate system are determined; Determine the two-dimensional coordinates of the plurality of pose markers in the positioning image; and Based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers relative to the slave tool coordinate system, the current posture of the slave tool relative to the reference coordinate system is determined.

2. The control method according to claim 1 further includes: Multiple candidate pose identifiers are determined from the positioning image; Based on multiple different pose pattern matching templates, the first pose identifier is identified from the multiple candidate pose identifiers; as well as Starting from the first pose identifier, search for pose identifiers.

3. The control method according to claim 2, wherein the pose identifier includes a corner point of the pose identifier pattern in the pose identifier pattern, and the method further includes: Determine the region of interest in the positioning image; The region of interest is divided into multiple sub-regions; The pixel with the largest corner likelihood value in each sub-region is determined to form a pixel set; The pixel with the largest corner likelihood value in the pixel set is selected as the candidate corner point of the pose identifier pattern; as well as The multiple different pose pattern matching templates are matched with the patterns at the corner positions of the candidate pose identifier pattern to identify the first pose identifier.

4. The control method according to claim 2, further comprising: The arrangement order of the plurality of pose identifiers is determined based on at least two of them; as well as Based on the arrangement order of the multiple pose markers, the three-dimensional coordinates of the multiple pose markers relative to the driven tool coordinate system are determined.

5. The control method according to any one of claims 1-4, characterized in that, Also includes: Determine the current orientation of the handle of the main controller; as well as Based on the target posture and current posture of the handle of the main operator, control signals for the main operator are generated.

6. The control method according to claim 5, characterized in that, The main actuator includes at least one attitude joint for controlling the attitude of the handle, and the control method includes: Obtain joint information of the at least one posture joint; and The current posture of the master manipulator is determined based on the joint information of the at least one posture joint.

7. The control method according to any one of claims 1-4, characterized in that, The driven tool includes a manipulator arm and an end effector disposed at the end of the manipulator arm. Determining the current orientation of the driven tool relative to the reference coordinate system includes: Determine the current orientation of the end effector relative to the base coordinate system of the driven tool; or Determine the current orientation of the end effector image on the display relative to the world coordinate system.

8. The control method according to any one of claims 1-4, characterized in that, The master manipulator includes at least one attitude joint for controlling the attitude of the handle of the master manipulator, and the control signal includes a control signal for controlling one or more of the at least one attitude joint.

9. The control method according to claim 8, characterized in that, The at least one attitude joint includes one or more uncoupled attitude joints, and the control method further includes: Obtain joint information of the other posture joints besides the one or more posture joints in the at least one posture joint; and Based on the joint information of the other posture joints, the transformation matrix of the other posture joints is determined.

10. The control method according to claim 9, characterized in that, Also includes: The control signal of the main manipulator is generated based on the target posture of the handle and the transformation matrix of the other posture joints.

11. The control method according to any one of claims 1-4, characterized in that, Also includes: In response to the fulfillment of predetermined conditions, the attitude matching degree between the handle of the master operator and the slave tool is determined, wherein the predetermined conditions include the triggering of teleoperation control.

12. The control method according to claim 11, characterized in that, Also includes: Based on the current posture of the handle of the master operator and the current posture of the slave tool, the posture matching degree between the handle of the master operator and the slave tool is determined.

13. The control method according to claim 11, characterized in that, Also includes: In response to the attitude matching degree being lower than a preset threshold, the control signal of the handle of the master operator is generated so that the attitude matching degree is higher than or equal to the preset threshold.

14. The control method according to claim 11, characterized in that, Also includes: In response to the attitude matching degree being higher than or equal to a preset threshold, a master-slave mapping is established between the master operator and the slave tool.

15. The control method according to any one of claims 1-4, characterized in that, The target orientation of the handle of the master operator is consistent with the current orientation of the slave tool.

16. A robot system, comprising: The main manipulator includes a multi-degree-of-freedom robotic arm, a handle disposed on the multi-degree-of-freedom robotic arm, and at least one motor and at least one main manipulator sensor disposed at at least one joint on the multi-degree-of-freedom robotic arm, wherein the at least one main manipulator sensor is used to obtain joint information of the at least one joint; A driven tool includes a manipulator and an end effector disposed at the end of the manipulator; Image acquisition device, used to acquire positioning images; as well as A control device, communicatively connected to the image acquisition unit and the main operator, is configured to perform the control method as described in any one of claims 1-15.

17. A computer device, the computer device comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory and configured to execute the at least one instruction to perform the control method as described in any one of claims 1-15.

18. A computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes a robot system to perform the control method as described in any one of claims 1-15.

Images