Control method of robot system based on positioning image and robot system

Abstract

The present disclosure relates to the field of control technology and includes a control method for a robotic system, a computer device, a computer-readable storage medium, and a robotic system. The control method for the robotic system includes: acquiring a positioning image; identifying, in the positioning image, multiple position markers located on an auxiliary connecting device; determining, based on the multiple position markers, a position of a first sheath of the auxiliary connecting device relative to a reference coordinate system; determining a first target position of a first motion arm based on the position of the first sheath; and controlling the first motion arm to move to the first target position to connect with the first sheath of the auxiliary connecting device.

Description

Technical Field

[0001] The present disclosure relates to the field of control technology, and in particular to a control method for a robot system based on a positioning image and a robot system. Background Art

[0002] Laparoscopic surgery is a widely used surgical procedure, offering advantages such as minimal invasiveness. In recent years, surgical robots have used motion arms to achieve greater stability and precision in surgical procedures. During surgery, these arms deliver surgical instruments through auxiliary connectors (e.g., sheaths) into the body (e.g., of a human or animal) to perform the surgical procedure.

[0003] Currently, surgical procedures performed using surgical robots primarily involve preoperative positioning (also known as preoperative positioning), intraoperative manipulation, and postoperative maintenance. Before surgery, a surgical assistant (such as an assistant doctor or nurse) typically adjusts the motion arm to the appropriate position based on the type of surgery and surgical posture, securely connects the motion arm to an auxiliary connection device, and then places surgical instruments at the end of the motion arm, allowing them to enter the body through the channel of the auxiliary connection device.

[0004] In the above applications, since the motion arm may be large in size and weight, the preoperative positioning operation performed by the surgical assistant or doctor may have problems with the stability of the motion arm and the preoperative adjustment time may be too long. Summary of the Invention

[0005] In some embodiments, the present disclosure provides a control method for a robotic system, wherein the robotic system includes multiple motion arms, the multiple motion arms including a first motion arm, the method including: acquiring a positioning image; identifying multiple posture identifiers located on an auxiliary connecting device in the positioning image; determining the posture of a first sheath of the auxiliary connecting device relative to a reference coordinate system based on the multiple posture identifiers; determining a first target posture of the first motion arm based on the posture of the first sheath; and controlling the first motion arm to move to the first target posture to connect with the first sheath of the auxiliary connecting device.

[0006] In some embodiments, the present disclosure provides a computer device, comprising: a memory for storing at least one instruction; and a processor, coupled to the memory, for executing the at least one instruction to perform the method in some embodiments of the present disclosure.

[0007] In some embodiments, the present disclosure provides a computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and the at least one instruction is executed by a processor to enable a computer to perform the method in some embodiments of the present disclosure.

[0008] In some embodiments, the present disclosure provides a robot system, comprising: a plurality of motion arms, the plurality of motion arms including a first motion arm; and a control device configured to execute the method in some embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A structural block diagram of a robot system according to some embodiments of the present disclosure is shown;

[0010] Figure 2 A schematic diagram showing the three-dimensional structure of a robot system according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A schematic diagram showing the structure of multiple motion arms of a robot system according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A partial cross-sectional view showing an auxiliary connection device according to some embodiments of the present disclosure;

[0013] Figure 5 A schematic diagram showing a connection state between a moving arm and an auxiliary connecting device according to some embodiments of the present disclosure;

[0014] Figure 6 A schematic diagram showing a label including multiple pose identifiers and multiple angle identifiers according to some embodiments of the present disclosure is shown;

[0015] Figure 7 A schematic diagram showing a cylindrical label formed according to some embodiments of the present disclosure;

[0016] Figure 8 A schematic diagram showing a positioning mark provided on an auxiliary connection device according to some embodiments of the present disclosure;

[0017] Figure 9 A flowchart illustrating a control method for a robot system according to some embodiments of the present disclosure is shown;

[0018] Figure 10 A flowchart illustrating a method for determining a position and posture of a first sheath relative to a reference coordinate system according to some embodiments of the present disclosure is shown;

[0019] Figure 11 A schematic diagram showing multiple pose markers on a cross-sectional circle according to some embodiments of the present disclosure;

[0020] Figure 12 A flowchart illustrating a method for determining a posture of a first sheath tube relative to a reference coordinate system according to other embodiments of the present disclosure is shown;

[0021] Figure 13 A flowchart illustrating a method for identifying a pose identifier according to some embodiments of the present disclosure is shown;

[0022] Figure 14 A schematic diagram illustrating a posture identification pattern according to some embodiments of the present disclosure;

[0023] Figure 15 A flowchart illustrating a method for searching for pose identifiers according to some embodiments of the present disclosure is shown;

[0024] Figure 16 A schematic diagram illustrating searching for a pose identifier according to some embodiments of the present disclosure is shown;

[0025] Figure 17 A flowchart illustrating a method for identifying an angle identifier according to some embodiments of the present disclosure is shown;

[0026] Figure 18 A flowchart illustrating a method for determining a motion path of a motion arm according to some embodiments of the present disclosure;

[0027] Figure 19 A schematic block diagram showing a computer device according to some embodiments of the present disclosure;

[0028] Figure 20 A schematic diagram showing a surgical robot system according to some embodiments of the present disclosure;

[0029] Figure 21 A schematic diagram illustrating a surgical tool according to some embodiments of the present disclosure;

[0030] Figure 22 A schematic diagram showing a master console cart according to some embodiments of the present disclosure;

[0031] Figure 23 A schematic diagram illustrating a surgical trolley according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0032] In order to make the technical problems solved by the present disclosure, the technical solutions adopted and the technical effects achieved more clear, the technical solutions of the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only exemplary embodiments of the present disclosure, rather than all embodiments.

[0033] In the description of the present disclosure, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present disclosure, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected" and "coupled" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a communication between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present disclosure can be understood according to the specific circumstances. In this disclosure, the end closest to the user (e.g., a doctor) is defined as the proximal end, near end, or rear end, and the end closest to the patient being operated on is defined as the distal end, far end, or front end. Those skilled in the art will appreciate that the embodiments of this disclosure can be used in medical devices or surgical robots, as well as other non-medical devices (e.g., industrial robots).

[0034] In the present disclosure, the term "position" refers to the positioning of an object or a part of an object in three-dimensional space (for example, the three translational degrees of freedom can be described using changes in the Cartesian X, Y and Z coordinate directions, such as the three translational degrees of freedom along the Cartesian X axis, Y axis and Z axis respectively). In the present disclosure, the term "posture" refers to the rotational setting of an object or a part of an object (that is, one or all of the three rotational degrees of freedom, for example, the three rotational degrees of freedom can be described using roll, pitch and yaw). In the present disclosure, the term "pose" refers to the combination of the position and pose of an object or a part of an object, for example, it can be described using six parameters of the six degrees of freedom mentioned above. In the present disclosure, the pose of a part of a moving arm refers to the pose of the coordinate system defined by the moving arm or a part thereof relative to the coordinate system defined by the bracket, base or world coordinate system where the moving arm is located. In the present disclosure, the pose of a moving arm can be represented by a set of joint values ​​of a plurality of joints included in the moving arm when the moving arm is in this pose (for example, a one-dimensional matrix composed of these joint values). In the present disclosure, the joint value of a joint indicates the angle of rotation of the corresponding joint relative to the corresponding joint axis or the distance moved relative to the initial position. In the present disclosure, the motion path of a motion arm or a portion thereof refers to the path along which the motion arm or a portion thereof moves from one posture to another.

[0035] In this disclosure, a reference coordinate system can be understood as a coordinate system that can describe the position and posture of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected to have the origin of a virtual reference object or the origin of a physical reference object as the coordinate system origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or an operator's own perceived coordinate system.

[0036] Figure 1 FIG. 1 shows a block diagram of a robot system 100 according to some embodiments of the present disclosure. Figure 1 As shown, the robot system 100 may include a control device 110, an image acquisition device 130, and a plurality of motion arms connected to the control device 110. In some embodiments, as shown in FIG. Figure 1 As shown, the multiple motion arms may include a first motion arm 120a and a second motion arm 120b. The control device 110 may be used to control the first motion arm 120a and the second motion arm 120b. For example, the control device 110 may adjust the movement, posture, mutual coordination, etc. of the first motion arm 120a and the second motion arm 120b. In some embodiments, the control device 110 may control the motion arm (e.g., the first motion arm 120a or the second motion arm 120b) to move to a target posture. In some embodiments, the ends of the first motion arm 120a and the second motion arm 120b may respectively include a first end (e.g., the end arm 1201a or the distal end of the end arm 1201a) and a second end (e.g., the end arm 1201b or the distal end of the end arm 1201b). The control device 110 may control the first motion arm 120a or the second motion arm 120b to move to a target posture so that the corresponding end moves to a desired position and posture.

[0037] In some embodiments, the image acquisition device 130 is communicatively connected to the control device 110. In some embodiments, the image acquisition device 130 can be used to acquire a positioning image. In some embodiments, a positioning device including a positioning identifier is provided on the auxiliary connecting device (for example, a positioning device is provided on the sheath of the auxiliary connecting device or on the main body lumen). The positioning image may include an image of part or all of the positioning device (for example, the positioning image includes a part of multiple positioning identifiers). In some embodiments, the positioning identifier includes a posture identifier, and the position or posture of the auxiliary connecting device can be determined based on the posture identifier. In some embodiments, the positioning identifier may include a posture identifier and an angle identifier (described in detail below), and the position and posture of the auxiliary connecting device can be determined based on the posture identifier and the angle identifier.

[0038] In some embodiments, a portion of the sheath or main lumen of the auxiliary connecting device is within the observation field of view of the image acquisition device 130, and the positioning device can be set on the sheath or main lumen. In some embodiments, the image acquisition device 130 may include but is not limited to a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. Depending on different application scenarios, the image acquisition device 130 can be an industrial camera, an underwater camera, a micro electronic camera, an endoscopic camera, etc. In some embodiments, the image acquisition device 130 can be fixed in position or variable in position, for example, an industrial camera fixed at a monitoring position or an endoscopic camera with adjustable position or posture. In some embodiments, the image acquisition device 130 can realize at least one of visible light band imaging, infrared band imaging, CT (Computed Tomography) imaging, and acoustic wave imaging.

[0039] In some embodiments, the control device 110 is configured to execute at least one instruction to perform some or all of the steps in the method of the present disclosure, such as Figure 9-10 、 Figure 12-13 、 Figure 15 and Figure 17-18 In some or all of the steps in the method disclosed in . In some embodiments, the control device 110 can receive a positioning image from the image acquisition device 130 and process the positioning image. For example, the control device 110 can identify a positioning mark located on the auxiliary connecting device in the positioning image. In some embodiments, the control device 110 can determine the position of the first sheath of the auxiliary connecting device based on the positioning image, for example, determine the position of the first sheath relative to the reference coordinate system based on the positioning image. In some embodiments, the robotic system 100 can also include three, four or more motion arms. The robotic system 100 can include a surgical robotic system, such as a laparoscopic surgical robotic system (for example, Figure 2 The robotic system 200 shown, or Figure 20 1. It should be understood that the robotic system 100 may also include specialized or general-purpose robotic systems used in other fields (eg, manufacturing, machinery, etc.).

[0040] Figure 2 FIG. 2 shows a schematic diagram of a three-dimensional structure of a robot system 200 according to some embodiments of the present disclosure. Figure 2As shown, the robotic system 200 may include a surgical trolley 230 and a first motion arm 220a and a second motion arm 220b disposed on the trolley 230. In some embodiments, the trolley 230 may include a base 2301 and a crossbeam 2302. In some embodiments, the first motion arm 220a and the second motion arm 220b may be movably disposed on the crossbeam 2302. It should be understood that the multiple motion arms of the robotic system 200 may also be disposed on multiple surgical trolleys. For example, each motion arm may be disposed on a corresponding surgical trolley. Alternatively, one motion arm may be disposed on one surgical trolley, while the remaining multiple motion arms may be disposed on another surgical trolley.

[0041] In some embodiments, each motion arm of the robotic system (e.g., Figure 2 The first motion arm 220a or the second motion arm 220b shown may include multiple connecting rods and multiple joints connected in series. In some embodiments, each joint of each motion arm may include a motor for driving the corresponding joint to rotate, thereby driving the corresponding connecting rod to rotate.

[0042] Figure 3 FIG. 3 shows a schematic diagram of the structure of multiple motion arms of a robot system 300 according to some embodiments of the present disclosure. Figure 3 As shown, taking the second motion arm 320b as an example, the second motion arm 320b may include joints 32011b-32081b and connecting rods 3201b-3208b. The proximal end of the connecting rod 3201b is connected to the crossbeam 3302, and the connecting rods 3201b-3207b are connected in series in sequence. Joint 32011b may be located at the proximal connection between beam 3302 and link 3201b, joint 32021b may be located at the connection between link 3201b and link 3202b, joint 32031b may be located at the connection between link 3202b and link 3203b, joint 32041b may be located at the connection between link 3203b and link 3204b, joint 32051b may be located at the connection between link 3204b and link 3205b, joint 32061b may be located at the connection between link 3205b and link 3206b, joint 32071b may be located at the connection between link 3206b and link 3207b, and joint 32081b may be located at the connection between link 3207b and link 3208b. The connecting rod 3208b is the most distal connecting rod of the second motion arm 320b, forming the end arm of the second motion arm 320b. The position and posture of the end arm need to be determined and represented by each of the aforementioned joints.

[0043] In some embodiments, the robotic system may also include one or more surgical instruments (e.g., Figure 2 As shown in FIG. 2 , a first surgical instrument 260a and a second surgical instrument 260b are provided. Figure 2As shown, the first surgical instrument 260a can be detachably mounted on the first end arm 2208a of the first moving arm 220a, and the second surgical instrument 260b can be detachably mounted on the second end arm 2208b of the second moving arm 220b. It should be understood that the first surgical instrument 260a and the second surgical instrument 260b can include but are not limited to clamps for performing surgery, electric knives, or image capture devices for performing illumination imaging (such as endoscopic tools), etc. A portion of the first surgical instrument 260a and the second surgical instrument 260b can enter a certain body part of a human or animal to perform a medical operation, such as surgery. In some embodiments, the surgical instrument may include, for example, Figure 21 Surgical tool 2100 is shown.

[0044] In some embodiments, as Figure 2 As shown, the robotic system 200 may further include an auxiliary connecting device 250, such as a sheath. A portion of the auxiliary connecting device 250 may be positioned at a body part of a human or animal requiring surgery, such as a surgical opening (e.g., an incision or a natural cavity), and another portion may be detachably connected to a motion arm (e.g., the first end arm 22081a or the second end arm 22081b of the first motion arm 220a or the second motion arm 220b) to better facilitate surgery.

[0045] Figure 4 FIG. 4 shows a partial cross-sectional view of an auxiliary connection device 400 according to some embodiments of the present disclosure. Figure 4 As shown, the auxiliary connection device 400 may include multiple sheaths, such as a first sheath 451 and a second sheath 452. In some embodiments, the auxiliary connection device 250 may also include multiple connecting portions (e.g., a first connecting portion 4511 and a second connecting portion 4521) disposed on the multiple sheaths. The connecting portions may include, but are not limited to, clamps, snap-fit ​​structures, adhesive structures, plug-in structures, and suction structures. The first connecting portion 4511 and the second connecting portion 4521 may be fixedly disposed on the first sheath 451 and the second sheath 452, respectively. In some embodiments, the auxiliary connection device 400 also includes a main lumen 453 connected to at least one sheath. In some embodiments, the main lumen 453 may include multiple tool channels 4531, each tool channel 4531 being connected to a corresponding sheath. In some embodiments, the auxiliary connection device 400 is further provided with a positioning device, which includes a positioning mark. The positioning device may, for example, be a positioning device 4512 disposed on the first sheath 451 or a positioning device 4532 disposed on the main lumen 453. In some embodiments, the positioning device 4512 or the positioning device 4532 may be, for example, a positioning tag. Figure 6 The label 600 shown or Figure 7Label 700 is shown.

[0046] In some embodiments, after the surgical instrument is mounted on the distal arm, the surgical instrument can pass through the sheath of the auxiliary connecting device into the surgical operating space (e.g., the patient's abdominal cavity). For example, a surgical tool or imaging tool can enter the surgical operating space through the sheath and tool channel.

[0047] It should be understood that Figure 4 The auxiliary connecting device 400 shown is merely exemplary. In some embodiments, the robotic system may include other numbers of motion arms, such as one, three, four or more motion arms, and the auxiliary connecting device may include other numbers of sheaths, such as one, three, four or more sheaths, each of which includes a corresponding connecting portion so that each sheath is connected to each motion arm and constrains the relative posture relationship between the ends of the multiple motion arms. In addition, surgical instruments and sheaths do not necessarily correspond one to one. For example, in some embodiments, multiple surgical instruments can enter the surgical operating space through the same sheath.

[0048] Figure 5 Schematic diagram showing a state 500 of a movement arm connected to an auxiliary connection device according to some embodiments of the present disclosure. In some embodiments, each movement arm may include a connection piece (e.g., Figure 2 The first connecting member 22081a or the second connecting member 22081b shown or Figure 5 Connector 55082 shown). See Figure 5 The auxiliary connecting device 550 includes four sheaths (eg, sheath 551 and sheath 552 ) and a main lumen 553 connected to the sheaths. Figure 5 The number of moving arms shown in FIG is four. Taking the moving arm 520 as an example, the connecting piece 52082 of the moving arm 520 is provided at the end of the end arm 5208. The connecting piece 52082 is connected to the connecting portion 5511 of the sheath tube 551 to realize the connection between the moving arm 520 and the auxiliary connecting device 550. In some embodiments, the connecting piece 52082 is detachably fixedly connected to the connecting portion 5511 of the sheath tube 551. In some embodiments, the auxiliary connecting device 550 can be connected to the connecting portion 5511 of the sheath tube 551. Figure 4 The auxiliary connecting device 400 shown has the same construction.

[0049] In some embodiments, if there are multiple surgical instruments, the multiple surgical instruments can be simultaneously or sequentially passed through corresponding sheaths into the surgical operating space. For example, a first surgical instrument and a second surgical instrument can be passed through a first sheath and a second sheath, respectively, and enter the surgical operating space simultaneously. Alternatively, the first surgical instrument can pass through the first sheath into the surgical operating space first, and then the second surgical instrument can pass through the second sheath into the surgical operating space. In some embodiments, the sheath of the auxiliary connecting device can be flexible, and the portion of the surgical instrument extending through the auxiliary connecting device can also be flexible.

[0050] It should be understood that the end of the motion arm can be, for example, the distal end of the end arm, the remote center of motion (RCM), or the connecting member of the motion arm. The end posture of the motion arm can be the posture of the end coordinate system relative to the coordinate system of the support or base on which the motion arm is located, or the world coordinate system. In some embodiments, the relative posture relationship of the ends of multiple motion arms can be determined based on the current type of surgery or the configuration of the auxiliary connection device. For example, based on the current type of surgery, the configuration of the auxiliary connection device can be determined or an auxiliary connection device suitable for the current surgery can be selected. Based on the configuration of the auxiliary connection device, the shape and relative position relationship between the multiple sheaths of the auxiliary connection device are determined to determine the relative posture relationship of the ends of the multiple motion arms. For example, the relative posture relationship between the end of the first motion arm and the end of the second motion arm can be determined based on the shape and relative position relationship of the first sheath and the second sheath. In some embodiments, the relative posture relationship between the ends of the first motion arm and the second motion arm can indicate the relative position relationship and relative posture relationship between the ends of the first motion arm and the second motion arm. For example, the relative posture relationship of the end may include, for example, the relative posture relationship between the first end arm of the first motion arm or a part of the first end arm and the second end arm of the second motion arm or a part of the second end arm. Alternatively, the relative posture relationship of the end may also include the relative posture relationship between the first surgical instrument (mounted on the first end arm) and the second surgical instrument (mounted on the second end arm). Alternatively, the relative posture relationship of the end may also include the relative posture relationship between the first connector and the second connector fixedly disposed on the first end arm and the second end arm. In some embodiments, the relative posture relationship may be stored in an associated relative posture model for calculating the target posture of the end of the second motion arm. Since the first connector and the second connector are respectively fixed on the first end arm and the second end arm, when the first end arm and the second end arm conform to the relative posture relationship of the end, the first connector and the second connector may be connected to the first connecting portion and the second connecting portion, respectively.

[0051] In some embodiments of the present disclosure, the target position or posture of the end of each motion arm can be achieved by one or more joints among the multiple joints included in the corresponding motion arm. In some embodiments, the multiple joints of the motion arm used to achieve the target posture are closer to the distal end of the motion arm relative to the multiple joints of the motion arm used to achieve the target position. It should be understood that the multiple joints for achieving the target posture and target position of the end of the motion arm can also include other settings and can be set according to specific needs.

[0052] During the preoperative preparation phase, the auxiliary connecting device is relatively fixed in position (for example, the auxiliary connecting device is connected to the patient's abdominal wall and is relatively fixed to avoid pulling on the patient's abdominal wall). In some embodiments, the motion arm can be preoperatively positioned based on the position of the auxiliary connecting device to ensure that the motion arm is in a suitable position for connection to the sheath of the auxiliary connecting device.

[0053] In some embodiments, a plurality of posture identifiers are provided on the auxiliary connecting device. For example, a plurality of posture identifiers are distributed on the first sheath or the main lumen of the auxiliary connecting device. For example, a plurality of posture identifiers are distributed circumferentially on the first sheath or the main lumen (for example, around all or part of the circumference). Based on the images of the plurality of posture identifiers, the posture of the first sheath can be determined. In some embodiments, the plurality of posture identifiers can include a single row or multiple rows of posture identifiers.

[0054] In some embodiments, the auxiliary connecting device is provided with a plurality of posture identifiers and at least one angle identifier. For example, a plurality of posture identifiers and at least one angle identifier are distributed on the first sheath or the main lumen of the auxiliary connecting device. Taking the main lumen as an example, a plurality of posture identifiers are distributed circumferentially on the main lumen, and a plurality of angle identifiers are distributed circumferentially on the main lumen. The plurality of posture identifiers and the plurality of angle identifiers are arranged axially side by side on the main lumen. For example, the plurality of posture identifiers and the plurality of angle identifiers are arranged on the outer surface of the columnar portion of the main lumen.

[0055] In some embodiments, each angle identifier has a positional association with one of the pose identifiers. Based on this positional association, the position of the pose identifiers can be used to determine the area where the angle identifiers are likely to be distributed. Alternatively, the position of the angle identifiers can be used to determine the area where the pose identifiers are likely to be distributed. The positional association can be determined based on the specific arrangement of the pose identifiers and the angle identifiers and can be pre-designed.

[0056] In some embodiments, the position association relationship may include an axial correspondence between the angle identifier and the posture identifier. For example, the position association relationship may include an axial offset. Based on the axial correspondence, when the position of one or more posture identifiers on the first sheath or the main body lumen is known, the area where the angle identifier may exist can be determined by offsetting a certain distance along the axial direction. For example, the position association relationship may also include an axial oblique alignment, etc.

[0057] In some embodiments, the positioning device may be, for example, a tag (e.g., Figure 6 The label 600 shown or Figure 7 In some embodiments, the label may be attached to the auxiliary connecting device. In some embodiments, the label may be attached to the circumference of the first sheath or the main lumen.

[0058] In some embodiments, the positioning device includes a posture identifier or an angle identifier. In some embodiments, the posture identifier may include a posture identifier pattern and a posture identifier pattern corner point, and the angle identifier may include an angle identifier pattern and an angle identifier pattern corner point. In some embodiments, the posture identifier pattern and the angle identifier pattern may be set on a label attached to the first sheath or the main body lumen, or may be printed on the first sheath or the main body lumen, or may be a pattern formed by the physical structure of the first sheath or the main body lumen itself, for example, it may include a depression or a protrusion and a combination thereof. In some embodiments, the posture identifier pattern or the angle identifier pattern may include a pattern formed by brightness, grayscale, color, etc. In some embodiments, the posture identifier pattern and the angle identifier pattern may include a pattern that actively (e.g., self-luminous) or passively (e.g., reflected light) provides information detected by the image acquisition module. Those skilled in the art will understand that in some embodiments, the posture of the posture identifier can be represented by the posture of the posture identifier pattern corner point coordinate system, and the posture of the angle identifier can be represented by the posture of the angle identifier pattern corner point coordinate system.

[0059] In some embodiments, the posture identification pattern or angle identification pattern is set on the first sheath or the main lumen in an area suitable for image capture by the image capture device, for example, an area that can be covered by the field of view of the image capture device during operation or an area that is not easily interfered with or blocked during operation.

[0060] Figure 6 A schematic diagram of a tag 600 including multiple pose identifiers and multiple angle identifiers according to some embodiments is shown. Figure 7Schematic diagram of a cylindrical label 700 disposed on the circumference of the first sheath or the main body lumen is shown. It is understood that for simplicity, the label 600 and the label 700 may include the same posture identification pattern and angle identification pattern.

[0061] See Figure 6 , multiple pose identifiers (in this disclosure, the corner points of the pose identifier pattern are represented by the "○" symbol) and multiple angle identifiers (in this disclosure, the corner points of the angle identifier pattern are represented by the "△" symbol) are arranged side by side. The multiple pose identifier patterns 611 can be the same or similar, and the multiple pose identifier pattern corner points are located in the multiple pose identifier patterns 611. The multiple angle identifier patterns 621-626 can be different, and the multiple angle identifier pattern corner points are located in the multiple angle identifier patterns 621-626.

[0062] In some embodiments, each angle identifier may have a positional association relationship with one of the posture identifiers. Figure 6 As shown, in the direction indicated by the arrow, some posture markers (eg, posture marker pattern 611) and corresponding angle markers (eg, angle marker pattern 621) are arranged along the arrow direction and have an interval distance d1. Figure 7 In the circumferential arrangement, the tag 600 becomes a cylindrical tag 700, and the positional association relationship between each angle mark and one of the posture marks may include the angle mark and the posture mark in the axial direction (such as Figure 7 Based on the axial correspondence, when the positions of one or more posture markers on the first sheath or the main lumen are known, the area where the angle marker may exist can be determined by offsetting a certain distance (such as distance d1) in the axial direction. In some embodiments, the axial correspondence between the angle marker and the posture marker can be represented by the axial correspondence between the angle marker pattern corner point and the posture marker pattern corner point. In some embodiments, based on the axial correspondence between the angle marker and the posture marker, the projection of the angle marker pattern corner point and one of the posture marker pattern corner points along the Z-axis direction coincides.

[0063] In some embodiments, the angle around the axis or the roll angle of the angle mark or the posture mark can be represented by the angle around the axis of the angle mark pattern corner point or the posture mark pattern corner point. The angle mark pattern corner point is relative to the auxiliary connection device coordinate system (for example, the coordinate system established on the first sheath or the main body lumen, such as Figure 7 The angles of the XY coordinate system shown are known or predetermined, e.g. Figure 7The angle of the angle identification pattern corner point R7 in the XY coordinate system with the X-axis is θ. Based on the position association relationship, it can be obtained that the angle of the posture identification pattern corner point P7 associated with its position relative to the X-axis is angle θ. It should be understood that the angle θ corresponding to the angle identification pattern corner point R7 and the posture identification pattern corner point P7 can be called the axis angle or roll angle around the Z axis of the angle identification or posture identification. In this disclosure, the axis angle or roll angle refers to the angle around the Z axis. It can be understood that for the sake of clarity, Figure 7 The angle identification pattern corner point R7 and the pose identification pattern corner point P7 are shown as separated in FIG, but they are coincident.

[0064] Figure 8 Schematic diagram showing the auxiliary connection device 800 according to some embodiments of the present disclosure setting the positioning mark. Figure 8 As shown, part of the main lumen 853 of the auxiliary connecting device is located in the patient's body (for example, part of the main lumen 853 is located in the abdominal cavity 830). A plurality of posture markers and angle markers can be circumferentially arranged on the main lumen 853 of the auxiliary connecting device 800. For example, Figure 6 The illustrated label 600 is circumferentially disposed on the main lumen 853, forming a cylindrical angle identification pattern band 810 and a posture identification pattern band 820. Multiple posture identification pattern corner points are distributed on the cross-sectional circle 821 of the posture identification pattern band 820 of the main lumen 853, and multiple angle identification pattern corner points are distributed on the cross-sectional circle 811 of the angle identification pattern band 810 of the main lumen 853. In some embodiments, multiple posture identifiers and angle identifiers can also be circumferentially disposed on the sheath. For example, multiple posture identifiers and angle identifiers can be disposed on the sheath 851 using a scheme similar to that used for the identification pattern band 810 and the posture identification pattern band 820.

[0065] In some embodiments, the multiple angle identification patterns are different patterns. Each angle identification pattern is used to indicate or identify a different rotation angle. In some embodiments, each angle identification pattern has a one-to-one correspondence with the identified rotation angle, and the identified rotation angle can be determined based on the angle identification pattern.

[0066] For example, Figure 8 As shown, multiple different angle identification patterns (such as Figure 6The multiple angle identification patterns 621-626 shown in FIG are evenly distributed along the circumference of the cylindrical structure to form angle identification pattern corner points AF. The angle identification pattern corresponding to the angle identification pattern corner point A is set as a reference pattern (for example, the angle identification pattern corresponding to the angle identification pattern corner point A is set to identify the 0° around-axis angle), and a plane coordinate system {wm1} is established. Then, the around-axis angles identified by the angle identification pattern corner points included in the remaining angle identification patterns can be determined based on the positional relationship between the remaining angle identification patterns and the angle identification pattern corresponding to the angle identification pattern corner point A. For example, see Figure 8 When the angle identification pattern corresponding to angle identification pattern corner point B is identified, based on the positional relationship between the angle identification pattern corresponding to angle identification pattern corner point B and the angle identification pattern corresponding to angle identification pattern corner point A, it can be determined that the angle around the axis indicated by angle identification pattern corner point B within the two-dimensional plane coordinate system of cross-sectional circle 811 is 60°. The origin of the two-dimensional plane coordinate system of cross-sectional circle 811 is the center of cross-sectional circle 811, the X-axis points from the origin to angle identification pattern corner point A, and the Y-axis is perpendicular to the X-axis.

[0067] Some embodiments of the present disclosure provide a control method for a robot system. In some embodiments, the robot system includes a plurality of motion arms, such as Figure 2 or Figure 5 shown. Figure 9 FIG. 9 is a flow chart showing a control method 900 for a robot system according to some embodiments of the present disclosure. Some or all of the steps in the method 900 may be performed by a control device (e.g., control device 110) or a controller of the robot system 100. Figure 20 The method 600 may be performed by a controller of the control cart 2040 or the surgical cart 2030 shown. The control device 110 may be configured on a computing device. The method 600 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 900 may be implemented as computer-readable instructions. These instructions may be read and executed by a general-purpose processor or a dedicated processor. In some embodiments, these instructions may be stored on a computer-readable medium.

[0068] See Figure 9 In step 901, a positioning image is acquired. In some embodiments, the positioning image includes a portion of the auxiliary connecting device and a plurality of position marks on the auxiliary connecting device. In some embodiments, the positioning image may also include a portion of the auxiliary connecting device and at least one angle mark on the auxiliary connecting device. For example, the positioning image includes a portion of the main body lumen and a plurality of position marks and at least one angle mark on the main body lumen. In some embodiments, the positioning image may be obtained from Figure 1The image acquisition device 130 shown receives the positioning image. For example, the control device 110 can receive the positioning image actively sent by the image acquisition device 130. Alternatively, the control device 110 can send an image request instruction to the image acquisition device 130, and the image acquisition device 130 sends the positioning image to the control device 110 in response to the image request instruction.

[0069] In step 903, in the positioning image, a plurality of posture markers located on the auxiliary connection device are identified. In some embodiments, an exemplary method for identifying a plurality of posture markers located on the auxiliary connection device may include: Figure 13 and Figure 15 The method shown. In some embodiments, the control device 110 may identify part or all of the posture identifiers in the positioning image through an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm, and the image processing algorithm may extract or identify features of the posture identifier. For example, the image processing algorithm may include a corner detection algorithm for detecting corners of the posture identification pattern. The corner detection algorithm may be one including but not limited to grayscale image-based corner detection, binary image-based corner detection, and contour curve-based corner detection. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the posture identification pattern. For another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the posture identification pattern. In some embodiments, the control device may identify part or all of the posture identifiers in the positioning image through a recognition model.

[0070] In some embodiments, the method 900 further includes identifying an angle identifier located on the auxiliary connection device based on the multiple posture identifiers. In some embodiments, after identifying the multiple posture identifiers, the angle identifier located on the auxiliary connection device is identified based on the position association relationship. In some embodiments, the position association relationship between the angle identifier and the first posture identifier can be as follows: Figure 6 or Figure 7 In some embodiments, the first pose identifier (e.g., the first pose identifier pattern or the first pose identifier pattern corner point) refers to a pose identifier in a plurality of pose identifiers that has a positional association relationship with the angle identifier. In some embodiments, an exemplary method for identifying an angle identifier includes: Figure 17 The method shown.

[0071] At step 905, the position of the first sheath of the auxiliary connecting device relative to the reference coordinate system is determined based on the multiple position identifiers. In other embodiments, method 900 further includes identifying, in the positioning image, multiple position identifiers and angle identifiers located on the auxiliary connecting device, the angle identifier being positionally associated with a first position identifier among the multiple position identifiers; and determining the position of the first sheath relative to the reference coordinate system based on the angle identifier and the multiple position identifiers.

[0072] In some embodiments, an exemplary method of determining the position of the first sheath relative to the reference coordinate system includes: Figure 10 or Figure 12 The method shown. In some embodiments, the posture of the first sheath relative to the reference coordinate system can be determined based on the angle identifier, the first posture identifier and multiple posture identifiers. In some embodiments, the posture of the first sheath can be represented by the posture of the coordinate system of the first sheath. For example, the posture of the coordinate system of the first sheath relative to the reference coordinate system can be used as the posture of the first sheath. The reference coordinate system can be, for example, a base coordinate system of a robot system or a motion arm or a world coordinate system. In some embodiments, the origin of the coordinate system of the first sheath can be set at the entrance position of the first sheath, and can also be set on the connecting portion of the first sheath. In some embodiments, the auxiliary connecting device can be, for example, Figure 2 The auxiliary connecting device 250 shown, or Figure 4 The auxiliary connecting device 400 shown, or Figure 5 Auxiliary connecting device 550 is shown.

[0073] In some embodiments, method 900 further includes determining a transformation relationship between the auxiliary connection device coordinate system and the posture identification coordinate system based on the angle identifier and the plurality of posture identifiers. In some embodiments, based on the transformation relationship between the auxiliary connection device coordinate system and the posture identification coordinate system, three-dimensional coordinates in the posture identification coordinate system can be converted to corresponding three-dimensional coordinates in the auxiliary connection device coordinate system. In some embodiments, based on the transformation relationship between the auxiliary connection device coordinate system and the posture identification coordinate system and the posture of the posture identification coordinate system relative to the reference coordinate system, the posture of the auxiliary connection device coordinate system relative to the reference coordinate system is obtained.

[0074] In some embodiments, the transformation relationship between the auxiliary connection device coordinate system and the posture identification coordinate system may include a roll angle of the posture identification coordinate system relative to the auxiliary connection device coordinate system. In some embodiments, the roll angle of the posture identification coordinate system relative to the auxiliary connection device coordinate system may be determined based on the angle identifier and the first posture identifier. It should be understood that the roll angle of the posture identification coordinate system relative to the auxiliary connection device coordinate system may be the angle of rotation of the posture identification coordinate system around the Z-axis of the auxiliary connection device coordinate system.

[0075] In some embodiments, the auxiliary connection device coordinate system can be a fixed coordinate system set on the sheath or main lumen of the auxiliary connection device based on multiple pose identifiers or multiple angle identifiers. In some embodiments, the Z axis of the auxiliary connection device coordinate system is parallel to the axial direction of the sheath or main lumen, and the XY plane of the auxiliary connection device coordinate system is coplanar with the corner points of the multiple pose identifier patterns, or coplanar with the corner points of the multiple angle identifier patterns.

[0076] In some embodiments, a pose identifier coordinate system can be determined to facilitate determining the positions of multiple pose identifiers. In some embodiments, the positions of the pose identifiers can be represented by the positions of the corner points of the pose identifier pattern. In some embodiments, the Z axis of the pose identifier coordinate system is parallel to or coincides with the axial direction of the sheath or main body lumen, and the XY plane of the pose identifier coordinate system is in the same plane as the corner points of the multiple pose identifier patterns.

[0077] For example, see Figure 8 , auxiliary connection device coordinate system {wm}≡[X wm Y wm Z wm ] T The origin is the center of the cross-section circle 821 where the corner points of the multiple pose identification patterns are located. The X-axis direction is from the origin to one of the corner points of the pose identification pattern. The Z-axis direction is parallel to the axial direction of the sheath or the main body lumen. The Y-axis is perpendicular to the XZ plane. The X-axis of the auxiliary connection device coordinate system {wm} and the two-dimensional plane coordinate system {wm1} of the cross-section circle 811 ≡ [X wm1 Y wm1 ] T The X-axis of the auxiliary connection device coordinate system is parallel to the Y-axis of the two-dimensional plane coordinate system {wm1} of the cross-sectional circle 811. The angle around the axis of the corner point of the angle identification pattern in the two-dimensional plane coordinate system {wm1} of the cross-sectional circle 811 can be equal to the angle around the axis of the corner point of the angle identification pattern in the two-dimensional plane coordinate system {wm} of the cross-sectional circle 811. wm0 Y wm0 Z wm0 ] T The origin is the center of the cross-section circle 821 where the corner points of the multiple pose identification patterns are located. The X-axis direction is from the origin to one of the corner points of the pose identification pattern. The Z-axis direction is parallel to the axial direction of the sheath or the main body lumen. The Y-axis is perpendicular to the XZ plane. Figure 8 The Z axis of the auxiliary connection device coordinate system {wm} coincides with the Z axis of the posture identification coordinate system {wm0}. The transformation relationship between the auxiliary connection device coordinate system {wm} and the posture identification coordinate system {wm0} can be determined by the roll angle α0 of the posture identification coordinate system {wm0} relative to the auxiliary connection device coordinate system {wm}. The roll angle α0 can refer to the rotation angle of the posture identification coordinate system {wm0} relative to the auxiliary connection device coordinate system {wm} around the Z axis.

[0078] In some embodiments, see Figure 8 , the roll angle α0 is calculated by the following formula (1):

[0079] α0=α1-α2 (1)

[0080] Where α1 is the first rotation angle, and α2 is the second rotation angle. The first rotation angle is the rotation angle of the angle identification pattern corner point (e.g., angle identification pattern corner point R8) in the auxiliary connection device coordinate system. The second rotation angle is the rotation angle of the first pose identification pattern corner point (e.g., pose identification pattern corner point P8) in the pose identification coordinate system.

[0081] Figure 10 FIG. 1 is a flow chart showing a method 1000 for determining the position of a first sheath tube relative to a reference coordinate system according to some embodiments of the present disclosure. Figure 10 As shown, some or all of the steps in the method 1000 may be controlled by a control device (e.g., Figure 1 Some or all of the steps in method 1000 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1000 may be used in a robotic system, for example, Figure 1 The robot system 100 shown, Figure 2 The surgical robot system 200 shown or Figure 20 The surgical robot system 2000 is shown. In some embodiments, the method 1000 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0082] See Figure 10 In step 1001, based on the angle identifier and multiple posture identifiers, the roll angle of the posture identifier coordinate system relative to the auxiliary connection device coordinate system is determined. In some embodiments, a first rotation angle identified by the angle identifier in the auxiliary connection device coordinate system is determined. A second rotation angle identified by the first posture identifier in the posture identifier coordinate system is determined. Based on the first rotation angle and the second rotation angle, the roll angle of the posture identifier coordinate system relative to the auxiliary connection device coordinate system is determined. In some embodiments, the roll angle of the posture identifier coordinate system relative to the coordinate system can be determined based on formula (1).

[0083] In step 1003, based on multiple pose identifiers, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined. The coordinates of the pose identifier in the corresponding coordinate system can be represented by the coordinates of the corner points of the pose identifier pattern in the corresponding coordinate system. For example, the two-dimensional coordinates of the pose identifier in the positioning image and the three-dimensional coordinates in the pose identifier coordinate system can be represented by the coordinates of the corner points of the pose identifier pattern. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image and the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image, the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system.

[0084] In some embodiments, based on the distribution of multiple pose identifiers, the three-dimensional coordinates of multiple pose identifier pattern corner points in the pose identifier coordinate system are determined. Figure 11 Each corner point of the pose identification pattern is located on the circumference of the cross-sectional circle 1122. The center and radius r of the cross-sectional circle 1122 are both known. The center of the cross-sectional circle 1122 is set as the origin of the pose identification coordinate system. The XY plane is located on the cross-sectional circle 1122. The X axis can be specified to point from the origin to any determined corner point of the pose identification pattern (for example, the corner point P of the pose identification pattern). 11 ), and then the three-dimensional coordinates of each corner point of the pose identification pattern in the pose identification coordinate system can be determined based on the distribution of multiple pose identifications. Figure 11 As shown, the pose identification pattern corner point P 11 The three-dimensional coordinates of the pose identification coordinate system are (r, 0, 0), and the three-dimensional coordinates of the other pose identification pattern corner points in the pose identification coordinate system can be calculated according to the following formula (2):

[0085] C m =[r·cos((m-1)·χ)r·sin((m-1)·χ)0] T (2)

[0086] Among them, C m To identify the pattern corner point P by pose 11 As the starting point, the three-dimensional coordinates of the mth pose identification pattern corner point in the pose identification coordinate system; χ is the angle around the axis between adjacent pose identification pattern corner points.

[0087] In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system can be known. For example, if the reference coordinate system is the world coordinate system, the transformation relationship between the camera coordinate system and the world coordinate system can be determined based on the camera's position. In other embodiments, depending on actual needs, the reference coordinate system can also be the camera coordinate system itself.

[0088] In some embodiments, based on the camera imaging principle and projection model, the pose of the pose identification coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identification patterns in the positioning image and the three-dimensional coordinates of the corner points of the multiple pose identification patterns in the pose identification coordinate system. Based on the pose of the pose identification coordinate system relative to the camera coordinate system and the transformation relationship of the camera coordinate system relative to the reference coordinate system, the pose of the pose identification coordinate system relative to the reference coordinate system can be obtained. In some embodiments, the intrinsic parameters of the camera can also be considered. For example, the intrinsic parameters of the camera can be such as Figure 1 The image acquisition device 130 shown or Figure 23 The camera intrinsic parameters of the imaging tool 2360b are shown in FIG. The camera intrinsic parameters may be known or obtained through calibration.

[0089] In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the optical center of the camera as the origin, or a coordinate system established with the center of the camera lens as the origin. When the camera is a binocular camera, the origin of the camera coordinate system can be the center of the left lens, the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of the line).

[0090] See Figure 10 In step 1005, the posture of the first sheath tube relative to the reference coordinate system is determined based on the roll angle of the posture identification coordinate system relative to the auxiliary connection device coordinate system and the posture of the posture identification coordinate system relative to the reference coordinate system.

[0091] Those skilled in the art will appreciate that in some embodiments of the present disclosure, the posture of the auxiliary connection device coordinate system relative to the reference coordinate system can be determined as the posture of the sheath or main body lumen relative to the reference coordinate system based on the roll angle of the posture identification coordinate system relative to the auxiliary connection device coordinate system and the posture of the posture identification coordinate system relative to the reference coordinate system. For example, taking the reference coordinate system as the world coordinate system, the posture of the auxiliary connection device coordinate system relative to the world coordinate system is as follows:

[0092] w R wm = w R wm0 ·rot z (α0)

[0093] w P wm =w P wm0 (3)

[0094] in, w R wm is the posture of the auxiliary connection device coordinate system relative to the world coordinate system, w P wm is the position of the auxiliary connection device coordinate system relative to the world coordinate system, w R wm0 is the posture of the pose coordinate system relative to the world coordinate system, w P wm0 is the position of the pose coordinate system relative to the world coordinate system, rot z (α0) represents the roll angle α0 around the Z axis of the auxiliary connection device coordinate system.

[0095] In some embodiments, the specific calculation formula (4) of the position of the auxiliary connection device coordinate system relative to the world coordinate system is as follows:

[0096] w R wm = w R lens lens R wm0 wm0 R wm

[0097] w P wm = w R lens ( lens R wm0 wm0 P wm + lens P wm0 )+ w P lens (4)

[0098] in, w R lens is the posture of the camera coordinate system relative to the world coordinate system, w P lens is the position of the camera coordinate system relative to the world coordinate system, lens R wm0 is the pose of the pose identification coordinate system relative to the camera coordinate system, lens P wm0 is the position of the pose identification coordinate system relative to the camera coordinate system, wm0 R wm The posture of the auxiliary connection device coordinate system relative to the pose identification coordinate system, wm0 P wm The position of the auxiliary connection device coordinate system relative to the pose identification coordinate system.

[0099] Figure 12 A flowchart of a method 1200 for determining the posture of a first sheath tube relative to a reference coordinate system according to some other embodiments of the present disclosure is shown. The method 1200 may be Figure 10 An alternative embodiment of method 1000 is provided. Figure 12 As shown, some or all of the steps in the method 1200 may be controlled by a control device (e.g., Figure 1 Some or all of the steps in method 1200 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1200 may be used in a robotic system, for example, Figure 1 The robot system 100 shown, Figure 2 The robotic system 200 shown, or Figure 20 The surgical robot system 2000 is shown. In some embodiments, the method 1200 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0100] See Figure 12 In step 1201, the three-dimensional coordinates of the multiple pose identifiers in the auxiliary connection device coordinate system are determined based on the roll angle of the pose identifier coordinate system relative to the auxiliary connection device coordinate system and the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system. It will be understood that, given the roll angle of the pose identifier coordinate system relative to the auxiliary connection device coordinate system, the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system can be transformed into three-dimensional coordinates in the auxiliary connection device coordinate system using a coordinate transformation.

[0101] In step 1203, the position of the first sheath relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple position identifiers in the positioning image and the three-dimensional coordinates of the multiple position identifiers in the auxiliary connection device coordinate system. In some embodiments, step 1203 can be implemented similarly to steps 1003 and 1005 in method 1000.

[0102] Figure 13 FIG. 1 is a flow chart of a method 1300 for identifying a pose identifier according to some embodiments of the present disclosure. Figure 13 As shown, some or all of the steps in the method 1300 may be controlled by a control device (e.g., Figure 1 Some or all of the steps in method 1300 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1300 may be used in a robotic system, such as Figure 1 The robot system 100 shown, Figure 2 The robot system 200 shown or Figure 20 The surgical robot system 2000 is shown. In some embodiments, the method 1300 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0103] See Figure 13 In step 1301, multiple candidate pose identifiers are determined from the positioning image. In some embodiments, the candidate pose identifiers can be represented by corner points of a candidate pose identifier pattern. In some embodiments, the candidate pose identifier pattern corner points can refer to possible pose identifier pattern corner points obtained through preliminary processing or preliminary identification of the positioning image. In some embodiments, a region of interest (ROI) can be first captured from the positioning image, and multiple candidate pose identifiers can be determined from the ROI. The ROI can be the entire positioning image or a partial region. For example, the ROI of the current frame can be captured based on an area within a certain range of the multiple pose identifier pattern corner points determined in the previous frame (e.g., the positioning image from the previous image processing cycle). For positioning images other than the first frame, the ROI can be an area within a certain distance range centered on a virtual point formed by the coordinates of the multiple pose identifier pattern corner points from the previous image processing cycle. The certain distance range can be a fixed multiple of the average spacing between the pose identifier pattern corner points, such as twice. It should be understood that the predetermined multiple can also be a variable multiple of the average spacing between the multiple candidate pose identifier pattern corner points from the previous image processing cycle.

[0104] In some embodiments, method 1300 may include determining a corner likelihood (CL) value for each pixel in the positioning image. In some embodiments, the corner likelihood value of a pixel may be a numerical value that characterizes the likelihood of the pixel being a feature point (e.g., a corner point). In some embodiments, the positioning image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may, for example, include at least one of image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include extracting a ROI from the positioning image and converting the ROI into a corresponding grayscale image.

[0105] In some embodiments, the method for determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first-order and / or second-order derivatives of each pixel. The first-order and / or second-order derivatives of each pixel within the ROI are used to calculate the corner likelihood value of each pixel. For example, the corner likelihood value of each pixel can be calculated according to the following formula (5):

[0106] CL=max(c xy ,c 45 )

[0107] c xy =τ 2 ·|I xy |-1.5·τ·(|I 45 |+|I n45 |)

[0108] c 45 =τ 2 ·|I 45_45 |-1.5·τ·(|I x |+|I y |) (5)

[0109] Wherein, τ is a set constant, for example, set to 2; I x , I 45 , I y , I n45 are the first-order derivatives of the pixel points in the directions of 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 are the second-order derivatives of the pixel in the directions of 0, π / 2, π / 4, and -π / 4 respectively.

[0110] In some embodiments, the ROI is divided into multiple sub-images. For example, a non-maximum suppression method can be used to evenly divide multiple sub-images within an ROI range. In some embodiments, the ROI can be evenly divided into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not restrictive. It should be understood that the positioning image or ROI can also be divided into multiple sub-images of other sizes, for example, into multiple sub-images of 9×9 pixels. The pixel point with the largest CL value in each sub-image can be determined, and the pixel point with the largest CL value in each sub-image can be compared with a first threshold to determine a set of pixel points with a CL value greater than the first threshold. In some embodiments, the first threshold can be set to 0.06. It should be understood that the first threshold can also be set to other values. In some embodiments, pixel points with a CL value greater than the first threshold can be used as candidate pose identification pattern corner points.

[0111] See Figure 13 In step 1303, an initial pose identifier is identified from the plurality of candidate pose identifiers based on the pose pattern matching template. In some embodiments, the pose pattern matching template is matched with an image at one of the corner points of the candidate pose identifier pattern, and the corner point of the candidate pose identifier pattern that meets a preset pose pattern matching standard is determined as the initial pose identifier pattern corner point.

[0112] In some embodiments, the pose pattern matching template and the image of the area near the corner point of the pose identification pattern have the same or similar features. If the degree of matching between the pose pattern matching template and the image of the area near the corner point of the candidate pose identification pattern meets a preset pose pattern matching standard (for example, the matching degree is higher than a threshold), it can be considered that the pattern of the area near the corner point of the candidate pose identification pattern and the pose pattern matching template have the same or similar features, and the current candidate pose identification pattern corner point can be considered as the pose identification pattern corner point.

[0113] In some embodiments, the pixel point with the largest CL value in the pixel point set is determined as the corner point of the candidate pose identification pattern to be matched. For example, all the pixel points in the pixel point set can be sorted in descending order of CL value, and the pixel point with the largest CL value is used as the corner point of the candidate pose identification pattern to be matched. After the corner point of the candidate pose identification pattern to be matched is determined, the pose pattern matching template is used to match the pattern at the corner point of the candidate pose identification pattern to be matched. If the preset pose pattern matching standard is reached, the corner point of the candidate pose identification pattern to be matched is determined to be the identified initial pose identification pattern corner point. If the corner point of the candidate pose identification pattern to be matched does not meet the preset matching standard, the pixel point with the secondary CL value (the pixel point with the second largest CL value) is selected as the corner point of the candidate pose identification pattern to be matched, and the pose pattern matching template is used to match the image at the corner point of the candidate pose identification pattern, and so on, until the initial pose identification pattern corner point is identified.

[0114] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, so the pose pattern matching template can be the same checkerboard pattern, and the grayscale distribution G of the pose pattern matching template is used. M Grayscale distribution G of the pixel neighborhood corresponding to the corner point of the candidate pose identification pattern image The grayscale distribution G of the pixel neighborhood is used to match the pixel. image is the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the pixel point. The specific calculation formula (6) is as follows:

[0115]

[0116] Wherein, Var is the variance function, and Cov is the covariance function. In some embodiments, when the CC value is less than 0.8, the grayscale distribution within the pixel area has a low correlation with the pose pattern matching template, and the candidate pose identification pattern corner point with the maximum corner likelihood value is determined to be the pose identification pattern corner point; otherwise, the candidate pose identification pattern corner point with the maximum corner likelihood value is determined to be the pose identification pattern corner point.

[0117] In some embodiments, method 1300 includes determining the edge directions of corner points of the candidate pose identification pattern. Figure 14 As shown, Figure 14 It includes a pose identification pattern 1401, and the candidate pose identification pattern corner points are Figure 14 Corner point P in 14 , then the corner point P 14 The edge direction can refer to the corner point P 14 The direction of the edge, such as Figure 14 The dotted arrow indicates the direction.

[0118] In some embodiments, the edge direction can be obtained by calculating the first-order derivative value (I) of each pixel in a certain range of neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate pose identification pattern in the X direction and the Y direction of the plane coordinate system. x and I y ). For example, the edge direction can be calculated using the following formula (7):

[0119]

[0120] Among them, the first-order derivative (I x and I y ) can be obtained by performing a convolution operation on each pixel point within a certain range of neighborhood. In some embodiments, by calculating the edge direction I of the pixel points within each range of neighborhood, angle and the corresponding weight I weight Perform clustering calculation to obtain the edge direction of the pixel point and select weight I weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if there are multiple edge directions, the weight I is selected. weight I corresponding to the largest number of classes angle as the edge direction.

[0121] In some embodiments, the method used for clustering calculation can be any one of the K-means method, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies) method, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, and GMM (Gaussian Mixed Model) method.

[0122] In some embodiments, the method 1300 includes: rotating the pose pattern matching template according to the edge direction. Rotating the pose pattern matching template according to the edge direction can align the pose pattern matching template with the image at the corner points of the candidate pose identification pattern.

[0123] The edge orientation of the corner points of the candidate pose identification pattern can be used to determine the orientation of the image at the corner points of the candidate identification pattern in the positioning image. In some embodiments, the pose pattern matching template can be rotated based on the edge orientation to adjust the pose pattern matching template to the same or nearly the same orientation as the image at the corner points of the candidate pose identification pattern to facilitate image matching.

[0124] See Figure 13 In step 1305, the initial pose identifier is used as a starting point to search for the pose identifier. For example, Figure 15 FIG. 1 is a flow chart of a method 1500 for searching for a pose identifier according to some embodiments of the present disclosure. Figure 15 As shown, some or all of the steps in the method 1500 may be controlled by a control device (e.g., Figure 1 Some or all of the steps in method 1500 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1500 may be used in a robotic system, for example, Figure 1 The robot system 100 shown, Figure 2 The robot system 200 shown or Figure 20 The surgical robot system 2000 is shown. In some embodiments, the method 1500 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0125] See Figure 15 In step 1501, a second pose identifier is determined with the initial pose identifier as a starting point. In some embodiments, a second pose identifier pattern corner point is searched for in a set search direction with the initial pose identifier pattern corner point as a starting point. In some embodiments, the set search direction may include at least one of: a direction directly in front of the initial pose identifier pattern corner point (corresponding to a 0° angle direction), a direction directly behind (corresponding to a 180° angle direction), a direction directly above (a 90° angle direction), a direction directly below (a -90° angle direction), and an oblique direction (e.g., a ±45° angle direction).

[0126] In some embodiments, the number of search directions is n, for example, searching in 8 directions, each search direction v sn It can be calculated according to the following formula (8):

[0127] v sn=[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (8)

[0128] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent pose identification pattern corner points among the multiple pose identification pattern corner points determined in the previous frame. For example, the predetermined search direction can be calculated according to the following formula (9):

[0129]

[0130]

[0131] Among them, (x j ,y j ) are the two-dimensional coordinates of the corner points of the multiple pose identification patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple pose identification patterns determined for the previous frame; v s1 The search direction for the first setting; v s2 The search direction for the second setting.

[0132] In some embodiments, as Figure 16 As shown, the pattern corner point P is identified with the initial pose 1601 The coordinate position of the second pose identification pattern corner point P is used as the search starting point and searched in the set search direction. 1602 The coordinate position of the pattern corner point P can be specifically included: 1601 The coordinate position of the search is used as the search starting point, and the search box (for example, Figure 16 The dotted box in the figure) searches in the set search direction V with a certain search step size. 1601 If there is at least one candidate pose identification pattern corner point in the search box, the candidate pose identification pattern corner point with the largest corner likelihood value in the search box is preferentially selected as the second pose identification pattern corner point P 1602 When the search box is limited to a suitable size, the pattern corner point P is identified with the initial pose 1601 The coordinate position of the pattern corner point P is used as the search starting point for the second pose identification 1602 During the search, the candidate pose identification pattern corner point with the largest likelihood value among the candidate pose identification pattern corner points in the search box is more likely to be the pose identification pattern corner point. Therefore, it can be considered that the candidate pose identification pattern corner point with the largest likelihood value in the search box is the second pose identification pattern corner point P 1602, in order to improve the data processing speed. In other implementations, in order to improve the accuracy of pose identification pattern corner point recognition, when there is at least one candidate pose identification pattern corner point in the search box, the candidate pose identification pattern corner point with the largest corner point likelihood value among the candidate pose identification pattern corner points appearing in the search box is selected for corner point recognition to determine whether the candidate pose identification pattern corner point with the largest corner point likelihood value is the pose identification pattern corner point. For example, the pose pattern matching template is matched with the image within a certain range of the candidate pose identification pattern corner point with the largest corner point likelihood value, and the candidate pose identification pattern corner point that meets the preset pose pattern matching degree standard can be considered to be the second pose identification pattern corner point P searched. 1602 .

[0133] In some embodiments, continue to see Figure 16 , the size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can be changed synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0134] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, and the correlation coefficient CC in formula (6) can be used for pattern matching. If CC is greater than a threshold, the candidate pose identification pattern corner point with the largest corner likelihood value is considered to be the pose identification pattern corner point and is recorded as the second pose identification pattern corner point.

[0135] See Figure 15 In step 1503, based on the initial pose identifier and the second pose identifier, a search direction is determined. In some embodiments, the search direction includes: a first search direction and a second search direction. The first search direction can be a direction starting from the coordinate position of the corner point of the initial pose identifier pattern and away from the corner point of the second pose identifier pattern. The second search direction can be a direction starting from the coordinate position of the corner point of the second pose identifier pattern and away from the corner point of the first pose identifier pattern. For example, Figure 16 The search direction V shown in 1602 .

[0136] In step 1505, the pose identifier is searched in the search direction with the initial pose identifier or the second pose identifier as the starting point. In some embodiments, if the first pose identifier pattern corner point is used as the new starting point, the first search direction in the above embodiment can be used as the search direction to search for the pose identifier pattern corner point. If the second pose identifier pattern corner point is used as the new search starting point, the second search direction in the above embodiment can be used as the search direction to search for the pose identifier pattern corner point. In some embodiments, the new pose identifier pattern corner point (for example, Figure 16 The third pose in the pattern corner P 1603) can be performed similarly to step 1501. In some embodiments, the search step size can be the distance L1 between the corner point of the initial pose identification pattern and the corner point of the second pose identification pattern.

[0137] In some embodiments, in response to the number of corner points of the pose marker pattern being greater than or equal to a threshold number of corner points of the pose marker pattern, the search for corner points of the pose marker pattern is stopped. For example, when four corner points of the pose marker pattern are searched (identified), the search for corner points of the pose marker pattern is stopped.

[0138] In some embodiments, in response to the search distance being greater than a set multiple of the distance between the N-1th pose identification pattern corner point and the N-2th pose identification pattern corner point, the search for the Nth pose identification pattern corner point is stopped, where N≥3. For example, the end condition of the search may be that the search distance is greater than twice the distance between the first two pose identification pattern corner points. In this way, the maximum search distance for searching the third pose identification pattern corner point is twice the distance between the initial pose identification pattern corner point and the second pose identification pattern corner point. If the pose identification pattern corner point has not been found after reaching the search distance, it is deemed that the third pose identification pattern corner point has not been found and the search ends.

[0139] In some embodiments, if the total number of pose identification pattern corner points searched is greater than or equal to a set threshold value (for example, the set threshold value is 4), it is considered that sufficient pose identification pattern corner points have been successfully identified. If the total number of pose identification pattern corner points found is less than the set value, it is considered that the search based on the initial pose identification pattern corner points in the above steps is unsuccessful. In the event of an unsuccessful search, new initial pose identification pattern corner points are re-determined from the candidate pose identification pattern corner points, and then the remaining pose identification pattern corner points are searched based on the re-determined initial pose identification pattern corner points as the search starting point. Similar to method 1300, new initial pose identification pattern corner points can be re-determined, and similar to method 1500, the remaining pose identification pattern corner points can be searched with the new pose identification pattern corner points as the search starting point.

[0140] In some embodiments, after searching for or identifying the corner points of the pose identification pattern, sub-pixel positioning may be performed on the determined corner points of the pose identification pattern to improve the position accuracy of the corner points of the pose identification pattern.

[0141] In some embodiments, the CL values ​​of the pixels can be fitted based on a model to determine the coordinates of the corner points of the pose identification pattern after sub-pixel positioning. For example, the fitting function of the CL value of each pixel in the ROI can be a quadratic surface function, where the extreme points of the function are sub-pixel points. The fitting function can be as follows:

[0142] S(x,y)=ax 2 +by2 +cx+dy+exy+f (10)

[0143]

[0144] Among them, S(x,y) is the CL value fitting function of all pixels in each ROI, a, b, c, d, e, f are coefficients; x c is the x coordinate of the pose marker, y c is the y coordinate of the pose identifier.

[0145] Figure 17 FIG. 1 is a flow chart illustrating a method 1700 for identifying an angle marker according to some embodiments of the present disclosure. Figure 17 As shown, some or all of the steps in the method 1700 may be controlled by a control device (e.g., Figure 1 Some or all of the steps in method 1700 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1700 may be used in a robotic system, for example, Figure 1 The robot system 100 shown, Figure 2 The robot system 200 shown or Figure 20 The surgical robot system 2000 is shown. In some embodiments, the method 1700 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0146] See Figure 17 In step 1701, an imaging transformation relationship is determined based on the two-dimensional coordinates of the plurality of posture identifiers in the positioning image and the three-dimensional coordinates of the plurality of posture identifiers in the posture identifier coordinate system. In some embodiments, the posture identifier coordinate system may be the posture identifier coordinate system described in detail in the embodiment shown in method 900. For example, the posture identifier coordinate system is as follows: Figure 8 As shown. In some embodiments, the imaging transformation relationship may refer to the transformation relationship between the three-dimensional coordinates in the pose identification coordinate system and the two-dimensional coordinates in the positioning image. It should be understood that based on the imaging transformation relationship, the two-dimensional coordinates in the positioning image may also be transformed into three-dimensional coordinates in the pose identification coordinate system. In some embodiments, the three-dimensional coordinates of multiple pose identifiers in the pose identification coordinate system may be determined based on formula (2). In some embodiments, the number of multiple pose identifiers may be greater than or equal to 4. For example, the imaging transformation relationship may be obtained based on the two-dimensional coordinates of the four pose identifiers in the positioning image and the corresponding four three-dimensional coordinates in the pose identification coordinate system.

[0147] See Figure 17In step 1703, based on the imaging transformation relationship, the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system and the position association relationship, multiple angle identifier candidate areas are determined in the positioning image. In some embodiments, the angle identifier candidate area may represent a candidate area of ​​the angle identifier pattern. In some embodiments, based on the three-dimensional coordinates of the multiple pose identifier pattern corner points in the pose identifier coordinate system and the position association relationship, multiple angle identifier pattern corner point candidate three-dimensional coordinates are determined in the pose identifier coordinate system. For example, based on the three-dimensional coordinates of the multiple pose identifier pattern corner points in the pose identifier coordinate system, multiple three-dimensional coordinates in the pose identifier coordinate system can be determined by axially offsetting a certain distance. These three-dimensional coordinates are represented by multiple angle identifier pattern corner point candidate three-dimensional coordinates. For example, see Figure 6 The positional association relationship is that the angle marker and the corresponding pose marker are separated by a certain distance along the Z axis of the pose marker coordinate system. Given the position of the corner point of the pose marker pattern, the position obtained by moving a certain distance along the positive or negative direction of the Z axis can be considered as the candidate position of the angle marker pattern corner point in the pose marker coordinate system.

[0148] In some embodiments, based on the imaging transformation relationship and the three-dimensional coordinates of the multiple angle identification pattern corner point candidates, multiple angle identification pattern candidate areas are determined in the positioning image. For example, based on the imaging transformation relationship and the three-dimensional coordinates of the multiple angle identification pattern corner point candidates, multiple angle identification pattern corner point candidate two-dimensional coordinates are obtained in the positioning image. In some embodiments, based on the two-dimensional coordinates of the multiple angle identification pattern corner point candidates, multiple angle identification pattern candidate areas are determined. For example, with each angle identification pattern corner point candidate two-dimensional coordinate as the center, an area of ​​a certain range size (for example, 5×5 pixels, 10×10 pixels, etc.) is determined in the positioning image as the angle identification candidate area. In some embodiments, the area of ​​a certain range size is greater than or equal to the size of the angle identification pattern after imaging. The size of the angle identification pattern after imaging can be obtained based on the actual size of the angle identification pattern and the imaging transformation relationship.

[0149] See Figure 17In step 1705, the angle identification candidate regions are identified from multiple angles. In some embodiments, the angle identification includes an angle identification pattern and an angle identification pattern corner point. In some embodiments, method 1700 may include determining the pixel point with the largest corner likelihood value in each angle identification candidate region to form a pixel set. In some embodiments, the corner likelihood value of the pixel point may be calculated when executing method 1300, or may be recalculated based on formula (5). Method 1700 also includes determining the angle identification candidate region corresponding to the pixel point with the largest corner likelihood value in the pixel set as the angle identification candidate region to be identified. Method 1700 also includes using multiple angle pattern matching templates to match the angle identification candidate regions to be identified respectively to identify the angle identification. In some embodiments, the angle identification pattern is a pattern with different graphic features. Multiple angle pattern matching templates may refer to standard angle pattern templates with the same or similar graphic features corresponding to the multiple angle identification patterns. In some embodiments, by determining multiple angle identification candidate regions, angle identification can be identified in multiple angle identification candidate regions, avoiding the need to identify angle identification in the entire image range and improving the speed of data processing.

[0150] In some embodiments, any template matching algorithm including square difference matching method, normalized square difference matching method, correlation matching method, normalized correlation matching method, correlation coefficient matching method and normalized correlation coefficient matching method can be used to perform matching operation on the angle pattern matching template and the angle identification candidate area.

[0151] In some embodiments, because the angle pattern matching template and the angle identification pattern have the same or similar graphical features, the pattern information of the angle identification may include the pattern information of the corresponding angle pattern matching template. For example, the shape of the angle pattern matching template, identifiable features in the image, etc. In some embodiments, each angle pattern matching template has a one-to-one correspondence with the rotation angle identified by the corresponding angle identification pattern. The first rotation angle is determined based on the pattern information of the angle identification pattern corresponding to the specific angle pattern matching template or the identified angle identification.

[0152] In some embodiments, method 1700 may include, in response to a matching failure, determining an angle identifier candidate region corresponding to a pixel having a maximum corner likelihood value among the remaining pixels in the pixel set as the angle identifier candidate region to be identified. In some embodiments, after determining a new angle identifier candidate region to be identified, multiple angle pattern matching templates are used to match the angle identifier candidate regions to be identified to identify the angle identifier.

[0153] In some embodiments, a first pose identifier having a positional association with the angle identifier is determined based on an angle identifier candidate region where the identified angle identifier is located. In some embodiments, multiple angle identifier candidate regions each correspond to at least one of multiple identified pose identifier pattern corner points. After determining the angle identifier candidate region where the identified angle identifier is located, a first pose identifier pattern corner point can be determined based on the correspondence between the multiple angle identifier candidate regions and the multiple pose identifier pattern corner points.

[0154] return Figure 9 In method 900, at step 907, a first target pose of the first motion arm is determined based on the pose of the first sheath. In some embodiments, the first target pose of the first motion arm can be represented by a set of target joint values ​​of multiple joints included in the first motion arm (e.g., a one-dimensional matrix composed of these target joint values).

[0155] In some embodiments, method 900 further includes: determining a target pose of a first swing point of the first motion arm; and determining the first target pose of the first motion arm based on the target pose of the first swing point and the kinematic model of the first motion arm. In some embodiments, the target pose of the first swing point can be determined based on the pose of the first sheath. In the present disclosure, the swing point of the motion arm can be a pose reference point at the location where the motion arm is connected to the sheath during setup.

[0156] When the swing point is in a predetermined target position, the connection position of the motion arm with the sheath is in a position suitable for connection with the sheath. In the present disclosure, the position of the swing point can be represented by the position of the coordinate system of the swing point. For example, the position of the coordinate system of the swing point relative to the reference coordinate system can be used as the position of the swing point. In some embodiments, the swing point of the motion arm can be set on the connecting part of the motion arm, or the swing point of the motion arm can also be set at the end of the motion arm, or the swing point of the motion arm is the distal motion center of the motion arm. For example, the swing point of the motion arm can be set at the end of the motion arm, and the position of the swing point can be the same as the position of the end of the motion arm (for example, the coordinate system of the end of the motion arm is the same as the coordinate system of the swing point) or have a predetermined transformation relationship with the position of the end of the motion arm. In some embodiments, the motion arm can be described by a kinematic model, and the kinematic model of the motion arm can be determined based on the structure of the motion arm. In some embodiments, the kinematic model of the motion arm can be constructed based on the DH parameter method. For example, the DH matrix corresponding to the joints of the motion arm is determined, and the kinematic model of the motion arm is determined based on the DH matrix of the joints. Based on the target pose of the first swing point and the kinematic model of the first motion arm, an inverse kinematics algorithm can be used to calculate a first target pose of the first motion arm. In some embodiments, based on the target pose of the first swing point and the kinematic model of the first motion arm, an inverse kinematics algorithm can be used to calculate joint values ​​of some or all joints of the first motion arm as the first target pose.

[0157] In some embodiments, the method 900 further includes: determining a predetermined target joint value for a characteristic joint among the plurality of joints of the first moving arm; and determining target joint values ​​for other joints of the first moving arm based on the target pose of the first swing point, the predetermined target joint value, and the kinematic model of the first moving arm. In some embodiments, the characteristic joint among the plurality of joints of the moving arm may be a joint among the plurality of joints that is likely to collide with other moving arms or structures, for example, Figure 3 The joints 32051b or 32061b shown. It should be understood that when the robot system includes multiple motion arms (for example, three or four motion arms), the predetermined target joint values ​​of the characteristic joints of different motion arms may be different. In some embodiments, the target joint values ​​of other joints may include the target joint values ​​of all other joints of the first motion arm except the characteristic joints. In some embodiments, based on the target pose of the first swing position, the predetermined target joint values ​​and the kinematic model of the first motion arm, the target joint values ​​of the other joints of the first motion arm can be calculated by an inverse kinematics algorithm.

[0158] In some embodiments, method 900 further includes: determining whether the target joint values ​​of other joints are within the joint motion range of the corresponding joint; and in response to at least one of the target joint values ​​of the other joints not being within the joint motion range of the corresponding joint, increasing or decreasing the predetermined target joint value by a preset adjustment value to adjust the predetermined target joint value. It should be understood that each joint of the moving arm has a certain range of motion, and the joint motion range of each joint includes the range between the minimum limit joint value and the maximum limit joint value of the corresponding joint. In some embodiments, the minimum limit joint value and the maximum limit joint value may not be within the joint motion range. For example, and not by way of limitation, some joints move between 18 degrees and 45 degrees, some between 45 degrees and 90 degrees, and some between -90 degrees and -45 degrees, etc. In some embodiments, the preset adjustment value may be set to, for example, 0.2° or 0.5° to adjust the predetermined target joint value. It should be understood that 0.2° or 0.5° are merely examples, and the adjustment value may also be set to other values. In some embodiments, the preset adjustment value is increased or decreased until a solution is found or the joint motion range of the characteristic joint (which may not include the limit value) is reached. For example, a solution may indicate that the predetermined target joint value or the target joint value of another joint is within the joint motion range of the corresponding joint. In some embodiments, method 900 further includes determining whether the adjusted predetermined target joint value is within the joint motion range of the feature joint.

[0159] In some embodiments, the method 900 further includes: in response to the target joint values ​​of the other joints being within the joint motion range of the corresponding joint, determining a first target pose of the first moving arm based on the predetermined target joint value and the target joint values ​​of the other joints. For example, in response to the target joint values ​​of the other joints of the first moving arm being within the joint motion range of the corresponding joint, selecting a set of the predetermined target joint value and the target joint values ​​of the other joints as the first target pose of the first moving arm.

[0160] In step 909, the first motion arm is controlled to move to a first target position to connect with the first sheath of the auxiliary connection device. In some embodiments, the control device can control the movement of multiple joints of the first motion arm to move the first motion arm from a first initial position of the first motion arm to a first target position to connect with the first sheath. For example, in the first target position, the first end arm is connected to the first sheath of the auxiliary connection device.

[0161] In some embodiments, the plurality of motion arms further includes a second motion arm. Method 900 further includes: determining a second target pose of the second motion arm based on the pose of the first sheath or the first target pose. In some embodiments, method 900 further includes: determining a target pose of a second swing point on the second motion arm based on the relative pose relationship between the first sheath and the second sheath and the pose of the first sheath or the first target pose and the relative pose relationship between the first motion arm and the second motion arm, wherein the relative pose relationship between the first sheath and the second sheath or the relative pose relationship between the first motion arm and the second motion arm is determined based on the configuration of the auxiliary connection device; and determining the second target pose of the second motion arm based on the target pose of the second swing point on the second motion arm. After the configuration or structure of the auxiliary connection device is determined, the relative pose relationship between the first sheath and the second sheath or the relative pose relationship between the first motion arm and the second motion arm can be determined based on the configuration or structure of the auxiliary connection device. The pose transformation relationship between the coordinate system of the second sheath and the coordinate system of the first sheath or the relative pose relationship between the first motion arm and the second motion arm can be substantially determined based on the configuration of the auxiliary connection device. For example, the position transformation relationship between the coordinate system of the second sheath and the coordinate system of the first sheath or the relative position transformation relationship between the first motion arm and the second motion arm can be predetermined. In actual operation, the predetermined position transformation relationship between the coordinate system of the second sheath and the coordinate system of the first sheath or the relative position transformation relationship between the first motion arm and the second motion arm can be conveniently called. In some embodiments, the position of the second sheath can be determined based on the relative position relationship between the first sheath and the second sheath and the position of the first sheath. In some embodiments, the target position of the second swing point can be determined based on the position of the second sheath. In some embodiments, similar to the first motion arm, the second target position of the second motion arm can be determined based on the target position of the second swing point and the kinematic model of the second motion arm. For example, based on the target position of the second swing point and the kinematic model of the second motion arm, the joint values ​​of some or all joints of the second motion arm can be solved by an inverse kinematics algorithm as the second target position. In some embodiments, method 900 also includes: controlling the second motion arm to move to the second target position to connect with the second sheath. In some embodiments, the control device can control the movement of multiple joints of the second motion arm to move the second motion arm from the second initial position of the second motion arm to the second target position to connect with the second sheath. For example, in the second target position, the second end arm is connected to the second sheath of the auxiliary connection device.

[0162] In some embodiments, the multiple motion arms further include a second motion arm, and method 900 further includes: obtaining a position and posture of a second sheath on the auxiliary connecting device based on a positioning image, wherein the positioning image includes a positioning device disposed on the second sheath of the auxiliary connecting device; and determining a second target position and posture of the second motion arm based on the position and posture of the second sheath. In some embodiments, the positioning device on the second sheath is, for example, a sheath positioning device including a sheath positioning marker. In some embodiments, the sheath positioning marker of the sheath positioning device can be disposed at a location where the coordinate system of the second sheath is located, or the sheath positioning device and the coordinate system of the second sheath have a defined position and posture transformation relationship. In some embodiments, the position and posture of the second sheath can be represented by the position and posture of the coordinate system of the second sheath. For example, the position and posture of the coordinate system of the second sheath relative to the reference coordinate system can serve as the position and posture of the second sheath. In some embodiments, the origin of the coordinate system of the second sheath can be disposed at the entrance of the second sheath, or can be disposed at the connection portion of the second sheath.

[0163] In some embodiments, the sheath positioning identifier includes multiple pose identifiers or angle identifiers. For example, the sheath positioning identifier may include a pose identifier or angle identifier similar to that in label 600 or label 700. Method 900 further includes: identifying multiple pose identifiers located on the second sheath in the positioning image; based on the multiple pose identifiers identified on the second sheath, identifying an angle identifier located on the second sheath, the angle identifier having a positional association relationship with a first pose identifier among the multiple pose identifiers; and determining the pose of the second sheath relative to the reference coordinate system based on the angle identifier and the multiple pose identifiers.

[0164] In some embodiments, similar to the above-mentioned determination of the posture of the first sheath, method 900 further includes: determining the posture of the second sheath based on the posture of the main lumen of the auxiliary connecting device and the configuration of the auxiliary connecting device. The posture of the main lumen can be determined based on a positioning device including a main positioning identifier. For example, in some embodiments, the main positioning identifier of the positioning device can be set at the position where the coordinate system of the main lumen is located, and the posture of the main lumen is determined based on the main positioning identifier. In some embodiments, the main positioning identifier includes multiple posture identifiers or angle identifiers. For example, the main positioning identifier can include a posture identifier or angle identifier similar to that in label 600 or label 700.

[0165] In some embodiments, method 900 further includes: controlling the first moving arm to connect with the first sheath. In some embodiments, the control device controls the first moving arm to connect with the first sheath. In some embodiments, the first connecting piece of the first moving arm is controlled to connect with the first connecting portion of the first sheath so that the first moving arm is connected to the first sheath. In some embodiments, method 900 further includes: controlling the second moving arm to connect with the second sheath. Similar to the first moving arm, the second moving arm can be controlled by the control device to connect with the second sheath. In some embodiments, the second connecting piece of the second moving arm is controlled to connect with the second connecting portion of the second sheath so that the second moving arm is connected to the second sheath. In some embodiments, the first moving arm and the second moving arm are controlled to connect with the first sheath and the second sheath respectively synchronously or sequentially. For example, the control device controls the first moving arm and the second moving arm to move simultaneously so as to connect with the first sheath and the second sheath respectively. For example, the control device can first control the first moving arm to connect with the first sheath, and in response to the first moving arm being connected with the first sheath, the control device controls the second moving arm to connect with the second sheath.

[0166] In some embodiments, a first motion path of the first moving arm can be determined based on the first target posture. In some embodiments, method 900 further includes: controlling the first moving arm to move to the first target posture based on the first motion path. In some embodiments, a second motion path of the second moving arm can be determined based on the second target posture. Method 900 further includes: controlling the second moving arm to move to the second target posture based on the second motion path. In some embodiments, method 900 further includes synchronously or sequentially controlling the first moving arm to move to the first target posture and the second moving arm to move to the second target posture based on the first motion path and the second motion path. It should be understood that the motion path may include the paths of multiple joints of the moving arm. In some embodiments, the motion of multiple joints of the moving arm can be controlled based on the motion path so that the moving arm moves to the target posture according to the set motion path. For example, the control device can control the motion of multiple joints of the first moving arm based on the first motion path so that the first moving arm moves to the first target posture according to the set first motion path.

[0167] Figure 18 FIG1 is a flowchart of a method 1800 for determining a motion path of a motion arm according to some embodiments of the present disclosure. Some or all of the steps in the method 1800 may be performed by a control device (e.g., control device 110) or a controller of the robot system 100. Figure 20The method 1800 may be performed by a controller of the master control cart 2040 or the surgical cart 2030 shown. The method 1800 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1800 may be implemented as computer-readable instructions. These instructions may be read and executed by a general-purpose processor or a dedicated processor. In some embodiments, these instructions may be stored on a computer-readable medium. In some embodiments, the method 1800 may be used to determine a first motion path of a first motion arm or a second motion path of a second motion arm.

[0168] See Figure 18 In step 1801, the initial posture of the motion arm is obtained. In some embodiments, the initial posture of the motion arm can be obtained by using sensors installed at each joint of the motion arm to obtain initial joint values ​​of each joint of the motion arm. It should be understood that the initial posture may also include the current posture of the motion arm.

[0169] In step 1803, a motion path of the motion arm is determined based on the initial and target poses of the motion arm. In some embodiments, for example, an interpolation method can be used to plan the motion path of the motion arm from its initial pose to its target pose based on the initial and target poses of the motion arm.

[0170] In some embodiments, method 1800 may further include determining whether an interference relationship will form between the current moving arm and other moving arms in the plurality of moving arms. For example, during the process of planning a second motion path for a second moving arm, determining whether an interference relationship will form between the first moving arm and the second moving arm. The interference relationship may include, for example, the possibility of collision between the second moving arm and the first moving arm. In some embodiments, in response to the target joint values ​​of all other joints of the second moving arm being within the joint motion range of the corresponding joint, determining whether an interference relationship will form between the second moving arm and the first moving arm is determined. The joint motion range may be determined based on the posture of the first moving arm to avoid interference with the first moving arm. In some embodiments, in response to the fact that no interference relationship will form between the second moving arm and the first moving arm, the second moving arm is controlled to move to a second target posture based on the second motion path. In some embodiments, when there are multiple sets of solutions for the joint values ​​of the moving arms that meet a condition (e.g., multiple sets of target joint values ​​for the second moving arm that meet the condition), the set of solutions with the lowest probability of interference between the joints of the second moving arm and the first moving arm may be selected as the target joint values ​​for the second moving arm. It should be understood that the probability of interference and whether interference has occurred can be determined based on the predetermined target joint value of the second moving arm and the distance between the target joint values ​​of the other joints and the corresponding joint values ​​of the first moving arm. For example, a larger distance indicates a lower probability of interference.

[0171] In some embodiments, method 900 further includes: determining the position of the main lumen of the auxiliary connecting device based on the positioning image; and determining the position of the first sheath relative to a reference coordinate system based on the position of the main lumen of the auxiliary connecting device and the configuration of the auxiliary connecting device. In some embodiments, the position of the main lumen of the auxiliary connecting device can be represented by the position of the main lumen's coordinate system relative to the reference coordinate system. In some embodiments, the position of the main lumen can serve as the position of the auxiliary connecting device. In some embodiments, the positioning device is, for example, a main body positioning device including a main body positioning marker (the main body positioning device can be, for example, positioning device 4532 disposed on main body lumen 453). The main body positioning marker of the main body positioning device can be disposed at a location corresponding to the coordinate system of the main lumen, or the main body positioning device can have a defined position transformation relationship with the coordinate system of the main lumen. It should be understood that after the configuration or structure of the auxiliary connecting device is determined, the relative position relationship between the first sheath and the main lumen can be determined. The position transformation relationship between the coordinate system of the first sheath and the coordinate system of the main lumen can be substantially determined based on the configuration of the auxiliary connecting device. In some embodiments, the main body positioning marker includes multiple position markers or angle markers. For example, the subject positioning mark may include a posture mark or an angle mark similar to that in the tag 600 or the tag 700. In some embodiments, the posture of the subject lumen can be determined based on the angle mark and multiple posture marks in the subject positioning mark. In some embodiments, the method for determining the posture of the subject lumen can be similar to Figure 10 The method 1000 shown in or Figure 12 A similar implementation is shown for method 1200 .

[0172] In some embodiments, the control device of the robotic system is configured to perform the method of the present disclosure. Figure 9-10 、 Figure 12-13 、 Figure 15 and Figure 17-18 In some embodiments, the robotic system may be, for example, Figure 1 The robot system 100 shown, Figure 2 The robot system 200 or Figure 20 A surgical robotic system 2000 is shown.

[0173] In some embodiments, the robot system further comprises: an auxiliary connection device comprising at least one sheath for connecting with a plurality of motion arms (eg, Figure 5 The sheath 551 shown is connected to the corresponding motion arm 520); and a posture detection device, including: at least one positioning device, the positioning device including multiple posture identifiers and at least one angle identifier; and an image collector, connected to the control device, for collecting positioning images including at least one positioning device. In some embodiments, the auxiliary connection device can be, for example, Figure 2 The auxiliary connecting device 250 shown, or Figure 4 The auxiliary connection device 400 shown, or Figure 5 The auxiliary connection device 550 is shown. In some embodiments, at least one positioning device can be, for example, Figure 4 In some embodiments, at least one positioning device may also be, for example, Figure 5 In some embodiments, the image collector can be set on the main body or beam of the operating table. For example, the image collector is Figure 2 The image acquisition device 23021 or the image acquisition device 23021 provided on the beam 2302 is shown Figure 3 The image acquisition device 33021 is shown as being arranged on the crossbeam 3302. In some embodiments, the image acquisition device can be arranged on a moving arm, for example, Figure 5 In some embodiments, the auxiliary connecting device includes a main body lumen connected to at least one sheath tube, and the at least one positioning device includes a main body positioning device (e.g., Figure 4 In some embodiments, at least one positioning device includes a sheath positioning device (e.g., a sheath positioning device provided on the first sheath) Figure 4 The sheath positioning device 4512 shown in the figure includes a sheath positioning marker.

[0174] In some embodiments of the present disclosure, the present disclosure further provides a computer device, the computer device including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory and is used to execute at least one instruction to perform some or all steps in the method of the present disclosure, such as Figure 9-10 、 Figure 12-13 、 Figure 15 and Figure 17-18 Some or all of the steps in the method disclosed in.

[0175] Figure 19 1900 is a schematic block diagram of a computer device according to some embodiments of the present disclosure. Figure 19The computer device 1900 may include a central processing unit (CPU) 1901, a system memory 1904 including a random access memory (RAM) 1902 and a read-only memory (ROM) 1903, and a system bus 1905 connecting the various components. The computer device 1900 may also include input / output devices 1906 and a mass storage device 1907 for storing an operating system 1913, application programs 1914, and other program modules 1915. The input / output devices 1906 include an input / output controller 1910 mainly composed of a display 1908 and input devices 1909.

[0176] The mass storage device 1907 is connected to the central processing unit 1901 through a mass storage controller (not shown) connected to the system bus 1905. The mass storage device 1907 or computer-readable medium provides non-volatile storage for the computer device. The mass storage device 1907 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0177] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include read-only memory, random access memory, flash memory or other solid-state storage technologies, read-only optical disks or other optical storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media is not limited to the aforementioned types. The above-mentioned system memory and mass storage devices may be collectively referred to as memory.

[0178] The computer device 1900 can be connected to a network 1912 via a network interface unit 1911 connected to the system bus 1905 .

[0179] The system memory 1904 or the mass storage device 1907 is further configured to store one or more instructions. The central processing unit 1901 implements all or part of the steps of the method in some embodiments of the present disclosure by executing the one or more instructions, such as Figure 9-10 、 Figure 12-13 、 Figure 15 and Figure 17-18 Some or all of the steps in the method disclosed in.

[0180] In some embodiments of the present disclosure, the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is executed by a processor to enable a computer to perform some or all steps of the method of some embodiments of the present disclosure, such as Figure 9-10 、 Figure 12-13 、 Figure 15 and Figure 17-18 Examples of computer-readable storage media include storage of computer programs (instructions), such as read-only memory, random access memory, read-only optical discs, magnetic tapes, floppy disks, and optical data storage devices.

[0181] Figure 20 A schematic diagram of a surgical robot system 2000 according to some embodiments of the present disclosure is shown. Figure 20 The surgical robot system 2000 may include a surgical tool 2060a, a main control trolley 2040, and an operating trolley 2030. The operating trolley 2030 is provided with a drive module for driving the surgical tool 2060a. The surgical tool 2060a is mounted on the operating trolley 2030 and connected to the drive module. The main control trolley 2040 is in communication with the operating trolley 2030 to control the surgical tool 2060a to perform surgical operations. In some embodiments, the control device of the main control trolley 2040 or the operating trolley 2030 can be used to perform some or all of the steps in the method of some embodiments of the present disclosure, such as Figure 9-10 、 Figure 12-13 、 Figure 15 and Figure 17-18 Some or all of the steps in the method disclosed in . In some embodiments, the main control trolley 2040 and the operating trolley 2030 are connected by wired transmission or wireless transmission. For example, the main control trolley 2040 and the operating trolley 2030 can be connected by a cable. In some embodiments, the surgical robot system 2000 may further include an imaging tool 2060b. The imaging tool 2060b may include an operating arm and an imaging module provided at the end of the operating arm. The imaging tool 2060b may be provided on the operating trolley 2030 and driven by a corresponding driving module. The image of the operating arm and the actuator of the surgical tool 2060a acquired by the imaging module may be transmitted to the main control trolley 2040. In some embodiments, the surgical tool 2060a is, for example, Figure 21 In some embodiments, the main control trolley 2040 is, for example, Figure 22 In some embodiments, the operating table 2030 is, for example, Figure 23 The surgical trolley 2300 is shown in FIG.

[0182] In some embodiments, a surgical robot or system includes at least two surgical tools, each of which includes a manipulator arm and an actuator disposed at the distal end of the manipulator arm. In some embodiments, the surgical robotic system may include a surgical trolley capable of mounting at least two surgical tools. In some embodiments, the surgical robotic system may include at least two surgical trolleys, each mounting a surgical tool.

[0183] Figure 21 Schematic diagram showing a surgical tool 2100 according to some embodiments of the present disclosure. Figure 21 , the surgical tool 2100 includes a drive transmission device 2190, an operating arm 2140 and an actuator 2160 arranged at the end of the operating arm. In some embodiments, the drive transmission device 2190 can cooperate with the drive module to drive the operating arm 2140 to move. The drive transmission device 2190 is used to connect with the drive module, and the driving force of the drive module is transmitted to the operating arm 2140 through the drive transmission device 2190, thereby driving the operating arm 2140 to achieve multi-degree-of-freedom movement. The drive module can also control the actuator 2160 to perform surgical operations. In some embodiments of the present disclosure, the actuator 2160 may include but is not limited to a bipolar curved separation forceps actuator, a bipolar elbow grasping forceps actuator, a monopolar curved shears actuator, a monopolar electric hook actuator, a bipolar grasping forceps actuator, a needle holding forceps actuator and a tissue grasping forceps actuator. In some embodiments, the surgical tool 2100 can be installed, for example, Figure 20 The surgical trolley 2030 shown in or Figure 23 The surgical trolley 2300 is shown in FIG.

[0184] Figure 22 Schematic diagram of the main control vehicle 2200 of some embodiments of the present disclosure is shown. Figure 22 The main control trolley 2200 includes: a controller (the controller can be configured on a computer device and set inside the main control trolley 2200), a main operator 2201, a main control trolley display (such as displays 2202-2204) and pedals (such as pedals 2205-2207). The controller is respectively connected to the main operator 2201, the main control trolley display and the pedals for communicating with the main operator 2201, the main control trolley display and the pedals, and generates corresponding control instructions based on the collected control information. In some embodiments, the controller is also connected to the operating trolley, for example, Figure 20 The operating table 2030 shown in FIG is connected to the operating table 2030 for controlling the surgical tool 2060b to perform the surgical operation or controlling the imaging tool 2060a to work. In some embodiments, the controller of the main control table 2200 can also be used to perform some or all steps in the method of some embodiments of the present disclosure, such as Figure 9-10 、 Figure 12-13 、 Figure 15 and Figure 17-18 Some or all of the steps in the method disclosed in.

[0185] In some embodiments, the main manipulator 2201 typically includes a left main manipulator (e.g., for controlling a first manipulator arm) and a right main manipulator (e.g., for controlling a second manipulator arm) corresponding to the left hand of the medical staff member and the right main manipulator (e.g., for controlling a second manipulator arm) operated by the right hand. In actual scenarios, the main manipulator 2201 is used to collect the operation input of the medical staff member, and the medical staff member controls the movement of the surgical tool or imaging tool in the operation area by remotely operating the main manipulator 2201 to perform medical operations. In some embodiments, the main manipulator 2201 includes a multi-degree-of-freedom robotic arm 22011, and a main manipulator sensor is provided at each joint of the multi-degree-of-freedom robotic arm 22011, and joint information (such as joint angle data) is generated by the main manipulator sensor of each joint. In some embodiments, the main manipulator sensor adopts a potentiometer and / or an encoder. In some embodiments, the multi-degree-of-freedom robotic arm 22011 has six degrees of freedom. In some embodiments, the posture of the main manipulator 2201 can be represented by a set of joint information of the main manipulator joints (e.g., a one-dimensional matrix composed of these joint information). In some embodiments, the main operator 2201 further includes a clamp 22012, which can be used to control the opening and closing angle of the actuator. In some embodiments, the main control trolley display includes a stereoscopic display 2202, a main control external display 2203, and a main control touch display 2204. The stereoscopic display 2202 displays the surgical image and system status prompts, the main control external display 2203 displays the surgical image and system status prompts, and the touch display 2204 displays the software user interface of the main control trolley 2200. In some embodiments, the image displayed by the stereoscopic display 2202 or the main control external display 2203 can be determined based on the image acquired by the imaging module, for example Figure 23 In some embodiments, the main control trolley pedal is used to collect the input of the medical staff's feet, including the electro-resection pedal 2205, the electro-coagulation pedal 2206, the clutch pedal 2207 and other structures.

[0186] Figure 23 Schematic diagram of an operating table trolley 2300 according to some embodiments of the present disclosure. Figure 23The operating trolley 2300 includes: a controller (the controller can be configured on a computer device and set inside the operating trolley 2300), an operating trolley chassis 2302, an operating trolley case 2303, a system status display 2305, a main column 2306, a main crossbeam 2307, a motion arm 2320, a drive module 2309 and other components. The operating trolley chassis 2302 is used to realize the movement and fixing functions of the operating trolley 2300. The operating trolley case 2303 is used to integrate the electrical components of the operating trolley internally. The system status display 2305 is used to display the operating trolley system user interface and receive user input. The main column 2306 can be raised and lowered, and its top end is fixed to the main crossbeam 2307. There is a crossbeam platform at the end of the main crossbeam 2307, and a plurality of motion arms 2320 are fixed to the lower end of the crossbeam platform. The motion arm 2320 is equipped with a drive module 2309, and the drive module 2309 is used to load a surgical tool 2360a or an imaging tool 2360b (the imaging tool 2360b can be, for example, a 3D electronic endoscope). In some embodiments, the operating trolley 2300 integrates multiple motion arms 2320, and each motion arm 2320 has multiple motion joints. In some embodiments, the operating trolley 2300 is integrated with multiple surgical tools 2360a and imaging tools 2360b, and the partial operating arms 2361a and actuators 2362a of the multiple surgical tools 2360a and the partial operating arms 2361b and imaging modules 2362b of the imaging tools 2360b enter the working space through the sheath of the auxiliary connecting device and the main body lumen 2353. In some embodiments, the controller of the operating trolley 2300 can also be used to execute some or all of the steps in the method of some embodiments of the present disclosure, such as Figure 9-10 、 Figure 12-13 、 Figure 15 and Figure 17-18 Some or all of the steps in the method disclosed in.

[0187] Although specific embodiments of the present invention have been illustrated and described, it will be apparent to those skilled in the art that various other changes and modifications may be made without departing from the spirit and scope of the present invention. Therefore, it is intended that all such changes and modifications that fall within the scope of the present invention be included in the appended claims.

Claims

1. A control method for a robotic system, the robotic system comprising a plurality of motion arms, the plurality of motion arms including a first motion arm, the method comprising: Acquire positioning images; In the positioning image, identifying a plurality of posture markers located on the auxiliary connecting device; Based on the multiple posture identifiers, identifying an angle identifier located on the auxiliary connecting device, wherein the angle identifier has a positional association relationship with a first posture identifier among the multiple posture identifiers; Determining the posture of the first sheath tube of the auxiliary connecting device relative to a reference coordinate system based on the angle identifier and the multiple posture identifiers; determining a first target posture of the first motion arm based on the posture of the first sheath; as well as The first motion arm is controlled to move to the first target posture to connect with the first sheath of the auxiliary connection device.

2. The control method according to claim 1, comprising: Determining a roll angle of a posture identifier coordinate system relative to an auxiliary connection device coordinate system based on the angle identifier and the plurality of posture identifiers; Determining the pose of the pose identifier coordinate system relative to the reference coordinate system based on the multiple pose identifiers; as well as The posture of the first sheath tube relative to the reference coordinate system is determined based on the roll angle of the posture identification coordinate system relative to the auxiliary connection device coordinate system and the posture of the posture identification coordinate system relative to the reference coordinate system.

3. The control method according to claim 2, comprising: Based on the two-dimensional coordinates of the multiple pose identifiers in the positioning image and the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined.

4. The control method according to claim 2, comprising: determining the three-dimensional coordinates of the plurality of posture identifiers in the auxiliary connection device coordinate system based on a roll angle of the posture identifier coordinate system relative to the auxiliary connection device coordinate system and the three-dimensional coordinates of the plurality of posture identifiers in the posture identifier coordinate system; as well as The posture of the first sheath tube relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple posture identifiers in the positioning image and the three-dimensional coordinates of the multiple posture identifiers in the auxiliary connection device coordinate system.

5. The control method according to any one of claims 2 to 4, comprising: Determining a first angle around the axis identified by the angle identifier in the auxiliary connecting device coordinate system; Determine a second angle around an axis identified by the first posture identifier in the posture identifier coordinate system; as well as Based on the first rotation angle and the second rotation angle, a roll angle of the posture identification coordinate system relative to the auxiliary connection device coordinate system is determined.

6. The control method according to any one of claims 1 to 4, wherein the position association relationship comprises: The corresponding relationship between the angle identifier and the first posture identifier in the axial direction.

7. The control method according to claim 1, comprising: Determining an imaging transformation relationship based on the two-dimensional coordinates of the multiple posture identifiers in the positioning image and the three-dimensional coordinates of the multiple posture identifiers in the posture identifier coordinate system; Determining a plurality of candidate three-dimensional coordinates of angle identifiers in the pose identifier coordinate system based on the three-dimensional coordinates of the plurality of pose identifiers in the pose identifier coordinate system and the position association relationship; Determining the plurality of angle identification candidate regions in the positioning image based on the imaging transformation relationship and the plurality of angle identification candidate three-dimensional coordinates; as well as Candidate regions are identified from the multiple angles, and the angle identifiers are recognized.

8. The control method according to claim 7, comprising: Determine the pixel with the largest corner likelihood value in each of the angle marker candidate regions to form a pixel set; Determine an angle marker candidate region corresponding to a pixel having the largest corner point likelihood value in the pixel set as the angle marker candidate region to be identified; as well as A plurality of angle pattern matching templates are used to respectively match the angle marker candidate areas to be identified, so as to identify the angle marker.

9. The control method according to claim 8, comprising: In response to a matching failure, an angle marker candidate region corresponding to a pixel having the largest corner likelihood value among the remaining pixels in the pixel set is determined as the angle marker candidate region to be identified.

10. The control method according to claim 7, comprising: Based on the angle identifier candidate area where the angle identifier is located, the first posture identifier having a position association relationship with the angle identifier is determined.

11. The control method according to claim 1, comprising: determining the position and posture of the main lumen of the auxiliary connecting device based on the positioning image; as well as The position of the first sheath tube relative to the reference coordinate system is determined based on the position of the main lumen of the auxiliary connecting device and the configuration of the auxiliary connecting device.

12. The control method according to any one of claims 1-4 and 7-11, further comprising controlling the first motion arm to be connected to the first sheath.

13. The control method according to claim 1, comprising: Acquiring a first initial posture of the first motion arm; determining a first motion path of the first motion arm based on the first initial posture and the first target posture; as well as Based on the first motion path, the first motion arm is controlled to move to the first target posture.

14. The control method according to claim 1, comprising: determining a target pose of a first swing position of the first motion arm; as well as Based on the target pose of the first swing point and the kinematic model of the first motion arm, a first target pose of the first motion arm is determined.

15. The control method according to claim 14, comprising: determining a predetermined target joint value of a characteristic joint among the plurality of joints of the first motion arm; as well as Based on the target pose of the first swing point, the predetermined target joint values, and the kinematic model of the first motion arm, target joint values ​​of other joints of the first motion arm are determined.

16. The control method according to claim 15, comprising: Determining whether the target joint values ​​of the other joints are within the joint motion range of the corresponding joints; as well as In response to at least one of the target joint values ​​of the other joints being outside the joint motion range of the corresponding joint, increasing or decreasing the predetermined target joint value by a preset adjustment value to adjust the predetermined target joint value; or In response to the target joint values ​​of the other joints being within the joint motion range of the corresponding joints, a first target posture of the first motion arm is determined based on the predetermined target joint value and the target joint values ​​of the other joints.

17. The control method according to claim 1, wherein the plurality of motion arms further includes a second motion arm, the method comprising: determining a second target posture of the second motion arm based on the posture of the first sheath or the first target posture; as well as The second motion arm is controlled to move to the second target posture to be connected with the second sheath.

18. The control method according to claim 17, further comprising: Determining a target posture of a second swing point on the second motion arm based on a relative posture relationship between the first sheath tube and the second sheath tube and the posture of the first sheath tube or a first target posture and a relative posture relationship between the first motion arm and the second motion arm, wherein the relative posture relationship between the first sheath tube and the second sheath tube or the relative posture relationship between the first motion arm and the second motion arm is determined based on a configuration of the auxiliary connecting device; and Based on the target pose of the second swing point on the second motion arm, a second target pose of the second motion arm is determined.

19. The control method according to claim 17 or 18, comprising: determining a first motion path of the first motion arm based on the first target posture; determining a second motion path of the second motion arm based on the second target posture; as well as Based on the first motion path and the second motion path, the first motion arm is controlled to move to the first target posture and the second motion arm is controlled to move to the second target posture synchronously or sequentially.

20. A computer device comprising: A memory for storing at least one instruction; as well as A processor, coupled to the memory, is configured to execute the at least one instruction to perform the control method according to any one of claims 1 to 19.

21. A computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and the at least one instruction is executed by a processor to enable a computer to execute the control method according to any one of claims 1 to 19.

22. A robotic system comprising: a plurality of motion arms, the plurality of motion arms including a first motion arm; as well as A control device configured to execute the control method according to any one of claims 1 to 19.

23. The robotic system of claim 22, further comprising: an auxiliary connecting device, comprising at least one sheath tube, for connecting with the plurality of moving arms; as well as The posture detection device includes: At least one positioning device, the positioning device comprising a plurality of posture identifiers and at least one angle identifier; as well as An image collector is connected to the control device and is used to collect a positioning image including the at least one positioning device.

24. The robotic system according to claim 23, wherein the auxiliary connecting device comprises a main body lumen connected to the at least one sheath tube, and the at least one positioning device comprises a main body positioning device disposed on the main body lumen.

25. The robotic system of claim 23, wherein the at least one positioning device comprises a sheath positioning device disposed on the first sheath.

Images