Method for preventing collision of operating arm based on association identifier and surgical robot system

Abstract

The present disclosure relates to the field of control technology, and discloses a collision avoidance control method for an operating arm of a robotic system, a computer device, a computer-readable storage medium, and a surgical robot system. The collision avoidance control method comprises: acquiring a positioning image; in the positioning image, identifying a plurality of first operating arm posture identifiers located on a first operating arm end of a first operating arm of at least two operating arms; based on the plurality of first operating arm posture identifiers, identifying a first operating arm angle identifier located on the first operating arm end, the first operating arm angle identifier having a positional association relationship with a first posture identifier among the plurality of first operating arm posture identifiers; based on the first operating arm angle identifier and the plurality of first operating arm posture identifiers, determining a first posture of the first operating arm end relative to a reference coordinate system; and determining a first collision avoidance operation for the first operating arm based on the first posture.

Description

Technical Field

[0001] The present disclosure belongs to the field of control technology, and in particular relates to an anti-collision control method for a manipulator arm based on association identification and a surgical robot system. Background Art

[0002] With the development of technology, it is becoming more and more popular to control a robot system manually or by a computer to perform desired actions to assist or replace an operator. A collaborative robot system usually includes at least two operating arms.

[0003] In the above applications, it is necessary to prevent the operating arm from colliding when it is working. Summary of the Invention

[0004] In some embodiments, the present disclosure provides a control method for an operating arm of a robot system, wherein the robot system includes at least two operating arms, and the control method includes: acquiring a positioning image; in the positioning image, identifying a plurality of first operating arm posture identifiers on an end of a first operating arm of a first operating arm among the at least two operating arms; based on the plurality of first operating arm posture identifiers, identifying a first operating arm angle identifier located on an end of the first operating arm, the first operating arm angle identifier having a positional association relationship with a first posture identifier among the plurality of first operating arm posture identifiers; based on the first operating arm angle identifier and the plurality of first operating arm posture identifiers, determining a first posture of the end of the first operating arm relative to a reference coordinate system; and based on the first posture, determining a first anti-collision operation for the first operating arm.

[0005] In some embodiments, the present disclosure provides a computer device, comprising: a memory for storing at least one instruction; and a processor, coupled to the memory, for executing the at least one instruction to perform the method of the present disclosure.

[0006] In some embodiments, the present disclosure provides a computer-readable storage medium having at least one instruction stored therein. The at least one instruction is executed by a processor to enable a computer to perform the method of the present disclosure.

[0007] In some embodiments, the present disclosure provides a surgical robot system comprising: at least two surgical tools, a first surgical tool among the at least two surgical tools comprising a first manipulator arm, an actuator arranged at the first manipulator arm end of the first manipulator arm, and at least one first manipulator arm angle identifier and multiple first manipulator arm posture identifiers arranged on the end of the first manipulator arm, at least one first manipulator arm angle identifier having a positional association relationship with a first posture identifier among the multiple first manipulator arm posture identifiers; an image collector for acquiring a positioning image of the manipulator arm; and a control device connected to the image collector for executing the method of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A schematic diagram showing a control system of a manipulator arm according to some embodiments of the present disclosure is shown;

[0009] Figure 2 A schematic diagram showing a structure of a manipulator arm according to some embodiments of the present disclosure is shown;

[0010] Figure 3 A schematic structural diagram of an operating arm according to some embodiments of the present disclosure is shown;

[0011] Figure 4 A schematic diagram of a label including multiple pose identifiers and multiple angle identifiers is shown;

[0012] Figure 5 A schematic diagram showing a cylindrical label formed by setting the label on the peripheral side of the end of the operating arm;

[0013] Figure 6 Showing a schematic diagram of an implementation scenario according to some embodiments of the present disclosure;

[0014] Figure 7 A flow chart showing a method for controlling a manipulator arm according to some embodiments of the present disclosure;

[0015] Figure 8 A flowchart illustrating a method for determining a first collision avoidance operation or a second collision avoidance operation according to some embodiments of the present disclosure is shown;

[0016] Figure 9 A schematic diagram of a bounding box according to some embodiments of the present disclosure is shown;

[0017] Figure 10 A schematic diagram illustrating bounding box updating according to some embodiments of the present disclosure is shown;

[0018] Figure 11 A flowchart illustrating a method for determining a first collision avoidance operation or a second collision avoidance operation according to some other embodiments of the present disclosure is shown;

[0019] Figure 12 shows an envelope schematic diagram according to some embodiments of the present disclosure;

[0020] Figure 13 A schematic diagram illustrating envelope update according to some embodiments of the present disclosure;

[0021] Figure 14 A flowchart illustrating a method for determining a pose of a manipulator coordinate system relative to a reference coordinate system according to some embodiments of the present disclosure is shown;

[0022] Figure 15 A flowchart illustrating a method for determining the pose of a manipulator coordinate system relative to a reference coordinate system according to other embodiments of the present disclosure is shown;

[0023] Figure 16 A schematic diagram showing multiple pose markers on a cross-sectional circle according to some embodiments of the present disclosure;

[0024] Figure 17 A flowchart illustrating a method for identifying a pose identifier according to some embodiments of the present disclosure is shown;

[0025] Figure 18 A schematic diagram illustrating a posture identification pattern according to some embodiments of the present disclosure;

[0026] Figure 19 A flowchart illustrating a method for searching for pose identifiers according to some embodiments of the present disclosure is shown;

[0027] Figure 20 A schematic diagram illustrating searching for a pose identifier according to some embodiments of the present disclosure is shown;

[0028] Figure 21 A flowchart illustrating a method for identifying an angle identifier according to some embodiments of the present disclosure is shown;

[0029] Figure 22 A schematic block diagram showing a computer device according to some embodiments of the present disclosure;

[0030] Figure 23 A schematic diagram showing a surgical robot system according to some embodiments of the present disclosure;

[0031] Figure 24 A schematic diagram showing a surgical robot system according to some embodiments of the present disclosure;

[0032] Figure 25 A schematic diagram illustrating a surgical tool according to some embodiments of the present disclosure;

[0033] Figure 26 A schematic diagram showing a master console cart according to some embodiments of the present disclosure is shown;

[0034] Figure 27 A schematic diagram illustrating a surgical trolley according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0035] The exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings. It will be understood by those skilled in the art that the scope of the present disclosure is not limited to these embodiments. Various improvements and variations can be made to the present disclosure based on the following embodiments. These improvements and variations are all within the scope of the present disclosure. Similar reference numerals indicate similar parts between the various embodiments shown in the accompanying drawings of the present disclosure.

[0036] In this disclosure, the term "position" refers to the positioning of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom that can be described using changes in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "pose" refers to the rotational setting of an object or a portion of an object (e.g., three rotational degrees of freedom that can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to the combination of the position and pose of an object or a portion of an object, such as six of the six degrees of freedom mentioned above.

[0037] In the present disclosure, the reference coordinate system can be understood as a coordinate system that can describe the posture of an object. According to the actual positioning requirements, the reference coordinate system can choose the origin of the virtual reference object or the origin of the physical reference object as the origin of the coordinate system. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own perception coordinate system, etc. In the present disclosure, an object can be understood as an object or target that needs to be positioned, such as an operating arm or the end of an operating arm or an actuator arranged at the far end of the end of an operating arm. Among them, the operating arm can be a rigid arm or a deformable arm. In the present disclosure, the posture of an operating arm or a part thereof refers to the posture of the operating arm coordinate system defined by the operating arm or a part thereof relative to the reference coordinate system.

[0038] Figure 1 FIG2 shows a schematic diagram of a manipulator control system 100 according to some embodiments of the present disclosure. The manipulator control system 100 can be applied to a robotic system. In some embodiments, the robotic system can be a surgical robot system, for example, Figure 23 The surgical robot system 2300 and Figure 24 The surgical robot system 2400 is shown. It should be understood that the robot system can also be a dedicated or general-purpose robot system in other fields (e.g., manufacturing, machinery, etc.). In some embodiments, the robot system includes at least two operating arms, such as a first operating arm 140-1 and a second operating arm 140-2. Figure 1 As shown, the manipulator control system 100 may include an image acquisition device 110 and a control device 120. The image acquisition device 110 is in communication with the control device 120. In some embodiments, the first manipulator arm 140-1 may include a first manipulator arm end 130-1 at its distal end or distal end. In some embodiments, a first actuator 160-1 may be provided at the distal end of the first manipulator arm end 130-1. Similar to the first manipulator arm 140-1, in some embodiments, the second manipulator arm 140-2 may include a second manipulator arm end 130-2 at its distal end or distal end. A second actuator 160-2 may be provided at the distal end of the second manipulator arm end 130-2.

[0039] In some embodiments, as Figure 1 As shown, the control device 120 can be used to control the movement of the first manipulator arm 140-1 and / or the second manipulator arm 140-2 to adjust the position and posture of the first manipulator arm 140-1 or the second manipulator arm 140-2, coordinate with each other, etc. In some embodiments, the control device 120 can control the movement of the first manipulator arm 140-1 to move the first manipulator arm end 130-1 or the first actuator 160-1 to a desired position and posture. The control device 120 can also control the movement of the second manipulator arm 140-2 to move the second manipulator arm end 130-2 or the second actuator 160-2 to a desired position and posture. In some embodiments, the control device 120 can determine a collision avoidance operation for the first manipulator arm 140-1 or the second manipulator arm 140-2 based on the position and / or posture of the first manipulator arm 140-1 or the second manipulator arm 140-2. For example, the control device 120 can determine a collision avoidance operation for the first manipulator arm 140-1 based on the position and / or posture of the first manipulator arm end 130-1. In some embodiments, the first operating arm 140-1 and the second operating arm 140-2 may be prevented from colliding by performing a collision avoidance operation. For example, the first operating arm end 130-1 or the first actuator 160-1 may be prevented from colliding with the second operating arm end 140-2 or the second actuator 160-2.

[0040] In the present disclosure, the control device 120 can be connected to a drive unit 150 (e.g., a motor) and send a drive signal to the drive unit 150, so that the drive unit 150 controls the first manipulator 140-1 or the second manipulator 140-2 to move to a corresponding target position based on the drive signal. For example, the drive unit 150 can be a servo motor that can receive instructions from the control device 120 to control the movement of the first manipulator 140-1 or the second manipulator 140-2. The control device 120 can also be connected to a sensor coupled to the drive unit 150, for example, via a communication interface, to receive motion data of the first manipulator 140-1 or the second manipulator 140-2, thereby monitoring the motion state of the first manipulator 140-1 or the second manipulator 140-2. In one example of the present disclosure, the communication interface can be a CAN (Controller Area Network) bus communication interface, which enables the control device 120 to communicate with the drive unit 150 and the sensor via the CAN bus. In some embodiments, the first manipulator 140-1 and the second manipulator 140-2 can be driven by different drive units, respectively.

[0041] In some embodiments, the first operating arm 140-1 or the second operating arm 140-2 may include a continuous deformable arm. Figure 3The operating arm 300 is shown. In some embodiments, the first operating arm 140-1 or the second operating arm 140-2 may include an operating arm with multiple degrees of freedom composed of multiple joints, such as an operating arm that can achieve 4 to 7 degrees of freedom. For example, an operating arm that can achieve 6 degrees of freedom.

[0042] In some embodiments, the image acquisition device 110 may include but is not limited to a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. In some embodiments, the image acquisition device 110 may be used to acquire a positioning image. The positioning image may include an image of part or all of the first operating arm 140-1. In some embodiments, the image acquisition device 110 may be used to acquire an image of the first operating arm end 130-1, and a plurality of identifiers may be provided on the first operating arm end 130-1. In some embodiments, the plurality of identifiers may include a posture identifier and an angle identifier (described in detail below). For example, a first operating arm positioning tag 170-1 may be provided on the first operating arm end 130-1 (the first operating arm positioning tag 170-1 may be, for example, Figure 4 400 shown). The first operating arm positioning label 170-1 may include multiple identifiers, including identifier patterns (described in detail below). In some embodiments, the positioning image may also include an image of part or all of the second operating arm 140-2. For example, the positioning image also includes an image of the second operating arm end 130-2. Similarly, multiple identifiers may be set on the second operating arm end 130-2. For example, a second operating arm positioning label 170-2 may be set on the second operating arm end 130-2. Similarly, the second operating arm positioning label 170-2 may include multiple identifiers, including posture identifiers and angle identifiers (described in detail below). In some embodiments, the identifier pattern set on the second operating arm 140-2 may be the same as or different from the identifier pattern set on the first operating arm 140-1.

[0043] like Figure 1As shown, if the first manipulator arm end 130-1 is within the observation field of view of the image acquisition device 110, the captured positioning image may include an image of the first manipulator arm end 130-1. In some embodiments, the image acquisition device 110 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. Depending on the application scenario, the image acquisition device 110 may be an industrial camera, an underwater camera, a microelectronic camera, an endoscopic camera, etc. In some embodiments, the image acquisition device 110 may be fixed or variable in position, such as an industrial camera fixed at a monitoring location or an endoscopic camera with adjustable position or posture. In some embodiments, the image acquisition device 110 may implement at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image to be acquired, those skilled in the art may select different image acquisition devices as the image acquisition device 110.

[0044] In some embodiments, the control device 120 may receive and process the positioning image from the image acquisition device 110. For example, the control device 120 may identify multiple markers on the first manipulator 140-1 in the positioning image and determine the position and pose of the first manipulator 140-1 or the first actuator 160-1 relative to a reference coordinate system (e.g., a world coordinate system).

[0045] Figure 2 Schematic diagram of a segment 200 of an operating arm according to some embodiments of the present disclosure is shown. The operating arm (eg, the first operating arm 140-1 or the second operating arm 140-2) may include at least one deformable segment 200. Figure 2 As shown, the deformable segment 200 includes a fixed plate 210 and a plurality of structural bones 220. The first ends of the plurality of structural bones 220 are fixedly connected to the fixed plate 210, and the second ends are connected to a driving unit (not shown). In some embodiments, the fixed plate 210 can be a ring-shaped structure, a disc-shaped structure, etc., and the cross-section can be a circular, rectangular, polygonal, etc. The driving unit drives the structural bones 220 to deform the segment 200. For example, the driving unit drives the structural bones 220 to make the segment 200 be in the following shape: Figure 2The bent state shown. In some embodiments, the second ends of the multiple structural bones 220 pass through the base plate 230 and are connected to the drive unit. In some embodiments, similar to the fixed plate 210, the base plate 230 may be a ring-shaped structure, a disc-shaped structure, etc., and the cross-section may be a circular, rectangular, polygonal, and other shapes. The drive unit may include a linear motion mechanism, a drive segment, or a combination of the two. The linear motion mechanism may be connected to the structural bone 220 to push or pull the structural bone 220, thereby driving the segment 200 to bend. The drive segment may include a fixed plate and multiple structural bones, wherein one end of the multiple structural bones is fixedly connected to the fixed plate. The other ends of the multiple structural bones of the drive segment are connected to or integrally formed with the multiple structural bones 220 to drive the bending of the segment 200 by driving the bending of the segment.

[0046] In some embodiments, a spacer disk 240 is further included between the fixing disk 210 and the base disk 230, and the plurality of structural bones 220 pass through the spacer disk 240. Similarly, the driving segment may also include a spacer disk.

[0047] Figure 3 FIG. 3 is a schematic diagram showing a structure of an operating arm 300 according to some embodiments of the present disclosure. In some embodiments, the first operating arm 140-1 or the second operating arm 140-2 may be as follows: Figure 3 The operating arm 300 shown. Figure 3 As shown, the operating arm 300 is a deformable operating arm, and the operating arm 300 may include an operating arm end 310 and an operating arm body 320. The operating arm body 320 may include one or more segments, such as a first segment 3201 and a second segment 3202. In some embodiments, the structures of the first segment 3201 and the second segment 3202 may be the same as those of the embodiment of FIG. Figure 2 The structure section 200 shown is similar. In some embodiments, as Figure 3 As shown, the operating arm body 320 further includes a first straight rod segment 3203 located between the first segment 3201 and the second segment 3202. The first end of the first straight rod segment 3203 is connected to the base plate of the second segment 3202, and the second end is connected to the fixed plate of the first segment 3201. In some implementations, such as Figure 3 As shown, the operating arm body 320 further includes a second straight rod segment 3204, the first end of the second straight rod segment 3204 is connected to the base plate of the first structural section 3201. Figure 3 As shown, each segment (first segment 3201 and second segment 3202) can include a base plate, a fixed plate, and multiple structural bones extending through the base plate and the fixed plate. The multiple structural bones can be fixedly connected to the fixed plate and slidably connected to the base plate. The continuum deformable arm and its included segments can be described by a kinematic model (described in detail below).

[0048] In some embodiments, the structure of each segment of the operating arm 300 can be as follows: Figure 2 The structure section 200 shown. Figure 2 As shown, the base plate coordinate system Attached to the base plate of the tth (t=1, 2, 3...) continuum segment, its origin is located at the center of the base plate, and the XY plane coincides with the base plate plane. Point from the center of the base plate to the first structural bone (the first structural bone can be understood as a structural bone arbitrarily designated as a reference from multiple structural bones). Bending plane coordinate system 1 Its origin coincides with the origin of the base plate coordinate system, and the XY plane coincides with the bending plane. and Coincident. Fixed disk coordinate system Attached to the fixed disk of the t-th continuum segment, its origin is located at the center of the fixed disk, and the XY plane coincides with the plane of the fixed disk. From the center of the fixed plate to the first structural bone. Bending plane coordinate system 2 Its origin is located at the center of the fixed disk, and the XY plane coincides with the bending plane. and coincide.

[0049] like Figure 2 The single segment 200 shown can be represented by a kinematic model. The position of the segment tip of the t-th segment (fixed disk coordinate system {te}) relative to the base disk coordinate system {tb} tb P te ,attitude tb R te It can be determined based on the following formulas (1) and (2):

[0050]

[0051] tb R te = tb R t1 t1 R t2 t2 R te (2)

[0052] Among them, L t is the virtual structural bone of the tth segment (e.g., Figure 2 The length of the virtual structural bone 221) shown in t In the tth section, about or Rotate to The required rotation angle, tb R t1is the posture of the bending plane coordinate system 1 {t1} of the t-th structural segment relative to the base plate coordinate system {tb}, t1 R t2 is the posture of the bending plane coordinate system 2 {t2} of the t-th structural section relative to the bending plane coordinate system 1 {t1}, t2 R te is the posture of the fixed disk coordinate system {te} of the t-th node relative to the curved plane coordinate system 2 {t2}.

[0053] tb R t1 、 t1 R t2 and t2 R te It can be based on the following formulas (3), (4) and (5):

[0054]

[0055]

[0056]

[0057] Among them, δ t In the tth structural section, the bending plane and Angle.

[0058] like Figure 2 The joint parameters Ψ of a single segment 200 are shown t It can be determined based on the following formula (6):

[0059] ψ t =[θ t , δ t ] T (6)

[0060] In some embodiments, the driving amount of multiple structural bones has a known mapping relationship with the joint parameters. Based on the target joint parameters of the joint and the mapping relationship, the driving amount of multiple structural bones can be determined. The driving amount of multiple structural bones can be understood as the amount of each structural bone moving from the initial state (e.g., θ t =0) is the length of the structural bone pushed or pulled when the structural bone is bent to the target bending angle. In some embodiments, the mapping relationship between the driving amount of multiple structural bones and the joint parameters can be determined based on the following formula (7):

[0061] q i ≡-r ti θ t cos(δ t +β ti ) (7)

[0062] Among them, rti is the distance from the i-th structural bone to the virtual structural bone in the t-th structural segment, β ti is the angle between the i-th structural bone and the first structural bone in the t-th segment, q i is the driving amount of the i-th structural bone. The driving signal of the driving unit can be determined based on the driving amount of the i-th structural bone.

[0063] In some embodiments, the entire deformable arm can be described by a kinematic model. Figure 3 As shown, multiple coordinate systems at multiple locations of the deformable arm can be transformed. For example, the actuator of the continuous deformable arm can be determined in the world coordinate system {w} based on the following formula (8):

[0064] W T tip = W T 1b 1b T 1e 1e T 2b 2b T 2e 2e T tip (8)

[0065] in, W T tip The homogeneous transformation matrix representing the actuator of the continuum deformable arm relative to the world coordinate system; w T 1b a homogeneous transformation matrix representing the base plate of the first continuum segment relative to the world coordinate system; 1b T 1e a homogeneous transformation matrix representing the fixed disk of the first continuum segment relative to the base disk of the first continuum segment; 1e T 2b a homogeneous transformation matrix representing the base plate of the second continuum segment relative to the fixed plate of the first continuum segment; 2b T 2e a homogeneous transformation matrix representing the fixed disk of the second continuum segment relative to the base disk of the second continuum segment; 2e T tip The homogeneous transformation matrix of the actuator of the continuum deformable arm relative to the fixed disk of the second continuum segment is represented. In some embodiments, the actuator is fixedly mounted on the fixed disk, so 2e T tip is known or predetermined.

[0066] It should be understood that the deformable arm has different joint parameters in different working states. For example, Figure 3 The operating arm 300 shown includes at least four working states. The four working states of the operating arm 300 are as follows:

[0067] First working state: Only the second segment 3202 participates in the posture control of the actuator (for example, only the second segment 3202 enters the working space). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (9):

[0068]

[0069] Among them, ψ c1 are the joint parameters of the operating arm 300 in the first working state, is the rotation angle of the operating arm 300, L2, θ2, δ2 and Figure 2 The L in the structure section 200 shown t ,θ t and δ t The physical meaning is the same.

[0070] Second working state: The second segment 3202 and the first straight segment 3203 participate in the posture control of the actuator (for example, the second segment 3202 enters the workspace in its entirety, and the first straight segment 3203 partially enters the workspace). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (10):

[0071]

[0072] Among them, ψ c2 is the joint parameter of the operating arm 300 in the second working state, L r is the feed amount of the first straight line segment 3203.

[0073] The third working state: the second segment 3202, the first straight segment 3203, and the first segment 3201 participate in the posture control of the actuator (for example, the second segment 3202 enters the workspace in its entirety, the first straight segment 3203 enters the workspace in its entirety, and the first segment 3201 partially enters the workspace). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (11):

[0074]

[0075] Among them, ψ c3 are the joint parameters of the operating arm 300 in the third working state, L1, θ1 and δ1 are the same as Figure 2 The L in the structure section 200 shown t ,θ t and δ t The physical meaning is the same.

[0076] Fourth working state: the second segment 3202, the first straight segment 3203, the first segment 3201, and the second straight segment 3204 participate in the posture control of the actuator (for example, the second segment 3202 enters the workspace in its entirety, the first straight segment 3203 enters the workspace in its entirety, the first segment 3201 enters the workspace in its entirety, and the second straight segment 3204 partially enters the workspace). At this time, the joint parameters of the manipulator 300 can be determined based on the following formula (12):

[0077]

[0078] Among them, ψ c4 is the joint parameter of the operating arm 300 in the fourth working state, L s is the feed amount of the second straight line segment 3204.

[0079] In some embodiments, the manipulator arm (e.g., Figure 1 The first operating arm 140 - 1 or the second operating arm 140 - 2 shown, Figure 3 The illustrated operating arm body 320 is provided with multiple position markers and at least one angle marker. For example, the multiple position markers are circumferentially distributed on the operating arm end 310, and the multiple angle markers are circumferentially distributed on the operating arm end 310. The multiple position markers and the multiple angle markers are arranged axially side by side on the operating arm end 310. For example, the multiple position markers and the multiple angle markers are arranged on the outer surface of the cylindrical portion of the operating arm end 310.

[0080] In some embodiments, each angle identifier has a positional association with one of the pose identifiers. Based on this positional association, the position of the pose identifiers can be used to determine the area where the angle identifiers are likely to be distributed. Alternatively, the position of the angle identifiers can be used to determine the area where the pose identifiers are likely to be distributed. The positional association can be determined based on the specific arrangement of the pose identifiers and the angle identifiers and can be pre-designed.

[0081] In some embodiments, the position association relationship may include an axial correspondence between the angle identifier and the posture identifier. For example, the position association relationship may include an axial offset. Based on the axial correspondence relationship, when the position of one or more posture identifiers on the end of the manipulator is known, the area where the angle identifier may be located can be determined by offsetting the axial position by a certain distance. For example, the position association relationship may also include an axial oblique alignment, etc.

[0082] In some embodiments, multiple posture identifiers and multiple angle identifiers can be set on a label attached to the peripheral side of the end of the operating arm.

[0083] In some embodiments, the posture identification may include a posture identification pattern and a posture identification pattern corner point, and the angle identification may include an angle identification pattern and an angle identification pattern corner point. In some embodiments, the posture identification pattern and the angle identification pattern may be set on a label attached to the end of the operating arm, or may be printed on the end of the operating arm, or may be a pattern formed by the physical structure of the end of the operating arm itself, for example, it may include a depression or a protrusion and a combination thereof. In some embodiments, the posture identification pattern or the angle identification pattern may include a pattern formed with brightness, grayscale, color, etc. In some embodiments, the posture identification pattern and the angle identification pattern may include a pattern that actively (for example, self-luminous) or passively (for example, reflected light) provides information detected by the image acquisition device. Those skilled in the art will understand that in some embodiments, the posture of the posture identification can be represented by the posture of the posture identification pattern corner point coordinate system, and the posture of the angle identification can be represented by the posture of the angle identification pattern corner point coordinate system.

[0084] In some embodiments, the posture identification pattern or angle identification pattern is set in an area on the end of the manipulator arm that is suitable for image capture by the image capture device, for example, an area that can be covered by the field of view of the image capture device during operation or an area that is not easily interfered with or blocked during operation.

[0085] Figure 4 A schematic diagram of a tag 400 including multiple pose identifiers and multiple angle identifiers according to some embodiments is shown. Figure 5 FIG2 shows a schematic diagram of a cylindrical tag 500 disposed on the peripheral side of the end of the operating arm. It is understood that for simplicity, the tag 400 and the tag 500 may include the same posture identification pattern and angle identification pattern.

[0086] See Figure 4 , multiple pose identifiers (in this disclosure, the corner points of the pose identifier pattern are represented by the symbol "○") and multiple angle identifiers (in this disclosure, the corner points of the angle identifier pattern are represented by the symbol "△") are arranged side by side. The multiple pose identifier patterns 411 can be the same or similar, and the multiple pose identifier pattern corner points are located in the multiple pose identifier patterns 411. The multiple angle identifier patterns 421-426 can be different, and the multiple angle identifier pattern corner points are located in the multiple angle identifier patterns 421-426.

[0087] Each angle identifier may have a positional association relationship with one of the pose identifiers. Figure 4 As shown, in the direction indicated by the arrow, some posture markers (such as the posture marker pattern 411) and corresponding angle markers (such as the angle marker pattern 421) are arranged along the arrow direction and have an interval distance d1. Figure 5In the circumferential arrangement, the tag 400 becomes a cylindrical tag 500, and the positional association relationship between each angle mark and one of the posture marks may include the angle mark and the posture mark in the axial direction (such as Figure 5 Based on the axial correspondence, when the positions of one or more posture identifiers on the end of the manipulator are known, the area where the angle identifier may exist can be determined by offsetting a certain distance (such as distance d1) along the axial direction. In some embodiments, the axial correspondence between the angle identifier and the posture identifier can be represented by the axial correspondence between the angle identifier pattern corner point and the posture identifier pattern corner point. In some embodiments, based on the axial correspondence between the angle identifier and the posture identifier, the projection of the angle identifier pattern corner point along the Z-axis direction coincides with that of one of the posture identifier pattern corner points.

[0088] In some embodiments, the angle around the axis or the roll angle of the angle mark or the posture mark can be represented by the angle around the axis of the angle mark pattern corner point or the posture mark pattern corner point. The angle mark pattern corner point is relative to the operating arm coordinate system (for example, the coordinate system established at the end of the operating arm, such as Figure 5 The angles of the XY coordinate system shown are known or predetermined, e.g. Figure 5 The angle of the angle identification pattern corner point R5 in the XY coordinate system with the X-axis is θ. Based on the position association relationship, it can be obtained that the angle of the posture identification pattern corner point P5 associated with its position relative to the X-axis is angle θ. It should be understood that the angle θ corresponding to the angle identification pattern corner point R5 and the posture identification pattern corner point P5 can be called the axis angle or roll angle around the Z axis of the angle identification or posture identification. In the present disclosure, the axis angle or roll angle refers to the angle around the Z axis. It can be understood that for the sake of clarity, Figure 5 The angle identification pattern corner point R5 and the pose identification pattern corner point P5 are shown as separated, but they are coincident.

[0089] Figure 6 Schematic diagram of an implementation scenario 600 according to some embodiments of the present disclosure is shown. Figure 6 As shown, the operating arm 640 includes an end 630 and an actuator 660 at the distal end, and a plurality of posture marks and angle marks can be circumferentially arranged on the end 630. For example, Figure 4 The illustrated label 400 is circumferentially arranged on the end 630 of the operating arm, forming a cylindrical angle identification pattern band 610 and a posture identification pattern band 620. Multiple posture identification pattern corner points are distributed on a cross-sectional circle 621 of the posture identification pattern band 620 of the operating arm end 630, and multiple angle identification pattern corner points are distributed on a cross-sectional circle 611 of the angle identification pattern band 610 of the operating arm end 630.

[0090] In some embodiments, the multiple angle identification patterns are different patterns. Each angle identification pattern is used to indicate or identify a different rotation angle. In some embodiments, each angle identification pattern has a one-to-one correspondence with the identified rotation angle, and the identified rotation angle can be determined based on the angle identification pattern.

[0091] For example, Figure 6 As shown, multiple different angle identification patterns (such as Figure 4 The multiple angle identification patterns 421-426 shown in FIG are evenly distributed along the circumference of the cylindrical structure to form angle identification pattern corner points AF. The angle identification pattern corresponding to the angle identification pattern corner point A is set as a reference pattern (for example, the angle identification pattern corresponding to the angle identification pattern corner point A is set to identify the 0° around-axis angle), and a plane coordinate system {wm1} is established. Then, the around-axis angles identified by the angle identification pattern corner points included in the remaining angle identification patterns can be determined based on the positional relationship between the remaining angle identification patterns and the angle identification pattern corresponding to the angle identification pattern corner point A. For example, see Figure 6 When the angle identification pattern corresponding to angle identification pattern corner point B is identified, based on the positional relationship between the angle identification pattern corresponding to angle identification pattern corner point B and the angle identification pattern corresponding to angle identification pattern corner point A, it can be determined that the angle around the axis indicated by angle identification pattern corner point B within the two-dimensional plane coordinate system of cross-sectional circle 611 is 60°. The origin of the two-dimensional plane coordinate system of cross-sectional circle 611 is the center of cross-sectional circle 611, the X-axis points from the origin to angle identification pattern corner point A, and the Y-axis is perpendicular to the X-axis.

[0092] In some embodiments, the pose of the actuator 660 can be determined by translating the manipulator coordinate system {wm} (e.g., the manipulator end coordinate system) by a predetermined distance. Alternatively, the pose of the actuator 660 can be approximately equal to the pose of the manipulator end coordinate system {wm}.

[0093] In some embodiments, the pose of the actuator 660 relative to the reference coordinate system (for example, the reference coordinate system is the world coordinate system {w}) is determined based on the pose of the manipulator coordinate system relative to the reference coordinate system. The specific calculation formula is as follows:

[0094]

[0095] in, w R tip is the posture of the actuator relative to the world coordinate system, w P tip is the position of the actuator relative to the world coordinate system, wm R tip is the posture of the actuator relative to the world coordinate system, wm P tip is the position of the actuator relative to the world coordinate system,w R wm is the posture of the manipulator coordinate system relative to the world coordinate system, w P wm is the position of the manipulator coordinate system relative to the world coordinate system.

[0096] Some embodiments of the present disclosure provide a method for controlling a manipulator arm of a robotic system. In some embodiments, the robotic system includes at least two manipulator arms. Figure 7 FIG. 7 is a flow chart showing a method 700 for controlling an operating arm according to some embodiments of the present disclosure. Some or all of the steps in the method 700 may be performed by a control device (e.g., control device 120) or a controller of the operating arm control system 100. Figure 24 The method 700 is executed by the controller of the main control trolley 2402 and the operating trolley 2403 shown. The control device 120 can be configured on a computing device. The method 700 can be implemented by software, firmware, and / or hardware. In some embodiments, the method 700 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0097] See Figure 7 In step 701, a positioning image is acquired. In some embodiments, the positioning image includes multiple position markers and at least one angle marker on the first operating arm. In some embodiments, the positioning image can be obtained from Figure 1 The image acquisition device 110 shown receives the positioning image. For example, the control device 120 can receive the positioning image actively sent by the image acquisition device 110. Alternatively, the control device 120 can send an image request instruction to the image acquisition device 110, and the image acquisition device 110 sends the positioning image to the control device 120 in response to the image request instruction.

[0098] Continue to read Figure 7 In step 703, in the positioning image, a plurality of first manipulator pose identifiers located on a first manipulator end of a first manipulator of the at least two manipulators are identified. For example, an exemplary method of identifying a plurality of pose identifiers located on a manipulator may include: Figure 17 and Figure 19The method shown. In some embodiments, the control device 120 may identify part or all of the posture identifiers in the positioning image through an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm, and the image processing algorithm may extract or identify features of the posture identifier. For example, the image processing algorithm may include a corner detection algorithm for detecting corners of the posture identification pattern. The corner detection algorithm may be one including but not limited to grayscale image-based corner detection, binary image-based corner detection, and contour curve-based corner detection. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the posture identification pattern. For another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the posture identification pattern. In some embodiments, the control device may identify part or all of the posture identifiers in the positioning image through a recognition model.

[0099] Continue to read Figure 7 In step 705, based on the plurality of first manipulator arm posture identifiers, a first manipulator arm angle identifier located at the end of the first manipulator arm is identified, and the first manipulator arm angle identifier has a position association relationship with the first posture identifier among the plurality of first manipulator arm posture identifiers. In some embodiments, after identifying the plurality of first manipulator arm posture identifiers, the first manipulator arm angle identifier is identified based on the position association relationship. In some embodiments, the position association relationship between the first manipulator arm angle identifier and the first posture identifier can be as follows: Figure 4 or Figure 5 In some embodiments, the first pose identifier (e.g., the first pose identifier pattern or the first pose identifier pattern corner point) refers to a pose identifier among the plurality of first operating arm pose identifiers that has a positional association relationship with the first operating arm angle identifier. An exemplary method for identifying the first operating arm angle identifier includes: Figure 21 The method shown.

[0100] Continue to read Figure 7 In step 707, based on the first manipulator arm angle identifier and the plurality of first manipulator arm posture identifiers, a first posture of the first manipulator arm end relative to the reference coordinate system is determined. An exemplary method for determining the posture of the first manipulator arm end relative to the reference coordinate system includes: Figure 14 or Figure 15 In some embodiments, the posture of the first manipulator arm end relative to the reference coordinate system can be determined based on the first manipulator arm angle identifier, the first posture identifier, and the plurality of first manipulator arm posture identifiers.

[0101] In some embodiments, method 700 further includes: determining a transformation relationship between the first manipulator coordinate system and the first manipulator pose identification coordinate system based on the first manipulator angle identifier and the plurality of first manipulator pose identifiers. In some embodiments, based on the transformation relationship between the first manipulator coordinate system and the first manipulator pose identification coordinate system, the three-dimensional coordinates in the first manipulator pose identification coordinate system can be converted into corresponding three-dimensional coordinates in the first manipulator coordinate system. In some embodiments, based on the transformation relationship between the first manipulator coordinate system and the first manipulator pose identification coordinate system and the pose of the first manipulator pose identification coordinate system relative to the reference coordinate system, the pose of the first manipulator coordinate system relative to the reference coordinate system is obtained.

[0102] In some embodiments, the transformation relationship between the first manipulator arm coordinate system and the first manipulator arm posture identification coordinate system may include a roll angle of the first manipulator arm posture identification coordinate system relative to the first manipulator arm coordinate system. In some embodiments, the roll angle of the first manipulator arm posture identification coordinate system relative to the first manipulator arm coordinate system may be determined based on the first manipulator arm angle identifier and the first posture identifier. It should be understood that the roll angle of the first manipulator arm posture identification coordinate system relative to the first manipulator arm coordinate system may be the angle of rotation of the first manipulator arm posture identification coordinate system around the Z-axis of the first manipulator arm coordinate system.

[0103] In some embodiments, the first manipulator coordinate system may be a fixed coordinate system set on the object based on a plurality of first manipulator pose identifiers or a plurality of first manipulator angle identifiers. In some embodiments, the Z axis of the first manipulator coordinate system is parallel to the axial direction of the first manipulator, and the XY plane of the first manipulator coordinate system is coplanar with the corner points of the plurality of first manipulator pose identifier patterns, or is coplanar with the corner points of the plurality of first manipulator angle identifier patterns.

[0104] In some embodiments, a first manipulator arm pose identification coordinate system can be determined to facilitate determining the positions of multiple first manipulator arm pose identifications. In some embodiments, the positions of the first manipulator arm pose identifications can be represented by the positions of the corner points of the first manipulator arm pose identification pattern. In some embodiments, the Z axis of the first manipulator arm pose identification coordinate system is parallel to or coincides with the axial direction of the first manipulator, and the XY plane of the first manipulator arm pose identification coordinate system is coplanar with the corner points of the multiple first manipulator arm pose identification patterns.

[0105] For example, see Figure 6 , taking the manipulator 640 as an example, the manipulator coordinate system {wm}≡[X wm Y wm Z wm ] TThe origin is the center of the cross-section circle 621 where the corner points of the multiple pose identification patterns are located. The X-axis direction is from the origin to one of the corner points of the pose identification pattern. The direction of the Z-axis is parallel to the axial direction of the manipulator end 630. The Y-axis is perpendicular to the XZ plane. The X-axis of the manipulator coordinate system {wm} and the two-dimensional plane coordinate system {wm1} of the cross-section circle 611 ≡ [X wm1 Y wm1 ] T The X-axis of the manipulator coordinate system is parallel to the Y-axis of the two-dimensional plane coordinate system {wm1} of the cross-sectional circle 611. The angle around the axis indicated by the corner point of the angle identification pattern in the two-dimensional plane coordinate system {wm1} of the cross-sectional circle 611 can be equal to the angle around the axis indicated by the corner point in the manipulator coordinate system {wm}. The pose identification coordinate system {wm0}≡[X wm0 Y wm0 Z wm0 ] T The origin is the center of the cross-section circle 621 where the corner points of the multiple pose identification patterns are located. The X-axis direction is from the origin to one of the corner points of the pose identification pattern. The Z-axis direction is parallel to the axial direction of the object manipulation arm end 630. The Y-axis is perpendicular to the XZ plane. Figure 6 , the Z axis of the manipulator coordinate system {wm} coincides with the Z axis of the pose identification coordinate system {wm0}. The transformation relationship between the manipulator coordinate system {wm} and the pose identification coordinate system {wm0} can be determined by the roll angle α0 of the pose identification coordinate system {wm0} relative to the manipulator coordinate system {wm}. The roll angle α0 can be the rotation angle around the Z axis of the pose identification coordinate system {wm0} relative to the manipulator coordinate system {wm}.

[0106] In some embodiments, see Figure 6 , the roll angle α0 is calculated by the following formula:

[0107] α0=α1-α2 (14)

[0108] Where α1 is the first rotation angle, and α2 is the second rotation angle. The first rotation angle is the rotation angle of the angle identification pattern corner point (for example, angle identification pattern corner point R6) in the manipulator coordinate system. The second rotation angle is the rotation angle of the first pose identification pattern corner point (for example, pose identification pattern corner point P6) in the pose identification coordinate system.

[0109] Continue to read Figure 7, in step 709, based on the first posture, a first anti-collision operation for the first operating arm is determined. In some embodiments, method 700 may further include determining a drive signal for the first operating arm based on the first posture to drive the first operating arm to perform the first anti-collision operation. In some embodiments, the first anti-collision operation may include stopping the movement of the first operating arm or generating a collision alarm message. In some embodiments, the first anti-collision operation may also include reducing the movement speed of the first operating arm or controlling the first operating arm to move in the opposite direction of the current movement direction. In some embodiments, the collision alarm information of the first anti-collision operation includes collision alarm information of multiple different alarm levels. Different levels of alarm information correspond to different levels of collision risks, which can be represented by different sounds or different colors of light.

[0110] In some embodiments, method 700 may further include: in response to not identifying the first manipulator arm posture identifier in the positioning image, determining the first kinematic posture of the end of the first manipulator arm as the first posture based on the drive information of the first manipulator arm and the kinematic model of the first manipulator arm. In some embodiments, the drive information of the first manipulator arm may be, for example, the drive value of the drive unit of the first manipulator arm. The drive value of the drive unit of the first manipulator arm may be obtained based on the encoder value of the drive motor. It should be understood that the kinematic model may represent a mathematical model of the motion relationship between the joint space and the task space of the manipulator arm. For example, the kinematic model may be established by methods such as the DH (Denavit-Hartenberg) parameter method and the exponential product representation method.

[0111] In some embodiments, the robotic system can be remotely operated by a master manipulator. It will be understood by those skilled in the art that there is a mapping relationship between the posture of the master manipulator and the posture of the manipulator arm in the remote operation state. This mapping relationship is, for example, a master-slave mapping relationship determined based on the configuration of the master manipulator and the configuration of the manipulator arm. In some embodiments, the posture of the master manipulator can be determined, for example, based on the drive value of the joint drive motor of the master manipulator or the numerical value of the displacement sensor on some or all joints of the master manipulator. Method 700 may also include: in response to not identifying the first manipulator arm posture identifier in the positioning image, determining the first kinematic posture of the end of the first manipulator arm as the first posture based on the posture of the master manipulator. For example, based on the master-slave mapping relationship between the posture of the master manipulator and the first manipulator arm, determining the first kinematic posture of the end of the first manipulator arm as the first posture. In some embodiments, the master manipulator can be, for example, Figure 26 The main operator 2601 is shown in FIG.

[0112] In some embodiments, at least two manipulators of the robotic system include a second manipulator. Method 700 may further include: determining a second posture of the second manipulator end of the second manipulator relative to the reference coordinate system; and based on the first posture and the second posture, determining a first anti-collision operation and / or a second anti-collision operation for the second manipulator. In some embodiments, the second anti-collision operation may include stopping the movement of the second manipulator or generating a collision alarm message. In some embodiments, the second anti-collision operation may further include reducing the movement speed of the second manipulator or controlling the second manipulator to move in the opposite direction of the current movement direction. In some embodiments, the collision alarm information of the second anti-collision operation includes collision alarm information of multiple different alarm levels.

[0113] In some embodiments, a method for determining a second posture of a second manipulator end relative to a reference coordinate system is provided. Method 700 may further include determining a second kinematic posture of the second manipulator end as a second posture based on drive information of the second manipulator and a kinematic model of the second manipulator. Similar to the first manipulator, in some embodiments, the drive information of the second manipulator may be, for example, a drive value of a drive unit of the second manipulator. The drive value of the drive unit of the second manipulator may be obtained based on an encoder value of a drive motor.

[0114] In some embodiments, another method for determining a second pose of the second manipulator end relative to the reference coordinate system is provided. Method 700 may further include determining a second kinematic pose of the second manipulator end as the second pose based on the pose of the master manipulator. For example, based on a master-slave mapping relationship between the pose of the master manipulator and the second manipulator, determining the second kinematic pose of the second manipulator end as the second pose. In some embodiments, the master manipulator may be, for example, Figure 26 The main operator 2601 is shown in FIG.

[0115] In some embodiments, another method for determining a second pose of the second manipulator end relative to a reference coordinate system is provided. Figure 1 , a plurality of different second operating arm identifiers may be provided on the second operating arm end 130-2, and these second operating arm identifiers include different second operating arm identifier patterns. Method 700 may also include: in the positioning image, identifying a plurality of second operating arm identifiers located on the second operating arm end, the plurality of second operating arm identifiers including different second operating arm identifier patterns; and determining a second posture based on the plurality of second operating arm identifiers. In some embodiments, the method for determining the second posture is similar to the method for determining the first posture. For example, steps 703, 705, 707, Figure 14 and Figure 15 The second pose is determined using the method shown in .

[0116] In some embodiments, method 700 may further include: in response to not identifying the second manipulator arm posture identifier in the positioning image, determining a second kinematic posture of the end of the second manipulator arm as the second posture based on the driving information of the second manipulator arm and the kinematic model of the second manipulator arm.

[0117] In some embodiments, the positioning image includes multiple first operating arm identifiers (for example, multiple first operating arm posture identifiers and first operating arm angle identifiers) and multiple second operating arm identifiers (for example, multiple second operating arm posture identifiers and second operating arm angle identifiers). It should be understood that two postures can be determined based on the multiple first operating arm identifiers and multiple second operating arm identifiers in the positioning image, corresponding to the postures of the first operating arm end and the second operating arm end relative to the reference coordinate system, i.e., the first posture and the second posture. The present disclosure also provides a method for determining the first posture or the second posture from two postures. In some embodiments, method 700 may further include: determining the first kinematic posture of the first operating arm end based on the driving information and kinematic model of the first operating arm, and determining the first posture from the first posture and the second posture based on the first kinematic posture. Those skilled in the art will understand that the first kinematic posture is a posture similar to the first posture, the posture close to the first kinematic posture is the first posture, and the other posture is the second posture. In some embodiments, method 700 may further include determining a second kinematic posture of the end of the second manipulator based on the drive information and kinematic model of the second manipulator, and determining a second posture from the first posture and the second posture based on the second kinematic posture. The second kinematic posture is similar to the second posture, the posture close to the second kinematic posture is the second posture, and the other posture is the first posture. In some embodiments, the first kinematic posture or the second kinematic posture may also be determined based on the posture of the master manipulator. For example, the first kinematic posture is determined based on the master-slave mapping relationship between the posture of the master manipulator and the first manipulator. In some embodiments, the first posture and the second posture can be distinguished by the spatial position of the first manipulator and the second manipulator. For example, the first manipulator and the second manipulator extend from channels arranged on the left and right, respectively, and operate on the left and right sides of the image, respectively. In this way, it can be determined that the identifier on the left side of the image is the first manipulator identifier, and the identifier on the right side of the image is the second manipulator identifier.

[0118] In some embodiments, collision detection of the first manipulator or the second manipulator may be implemented using a bounding box method. Figure 8 FIG. 8 is a flow chart showing a method 800 for determining a first collision avoidance operation or a second collision avoidance operation according to some embodiments of the present disclosure. Figure 8 As shown, some or all of the steps in the method 800 may be performed by, for example Figure 1 The control device (eg, control device 120) of the manipulator control system 100 shown or Figure 24 The method 800 is executed by the controller of the main control trolley 2402 and the operating trolley 2403 shown. The control device 120 can be configured on a computing device. The method 800 can be implemented by software, firmware, and / or hardware. In some embodiments, the method 800 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0119] See Figure 8 In step 801, based on the first pose, a first bounding box of a first actuator disposed at the end of a first manipulator is determined, where the first bounding box includes one or more first sub-bounding boxes.

[0120] In step 803 , based on the second pose, a second bounding box of a second actuator disposed on the end of the second manipulator is determined, where the second bounding box includes one or more second sub-bounding boxes.

[0121] In step 805 , a first anti-collision operation and / or a second anti-collision operation is determined based on the first bounding box and the second bounding box.

[0122] Figure 9 Schematic diagram of bounding box in some embodiments of the present disclosure is shown. Figure 9 As shown, the system includes: a first manipulator arm 940-1, a first manipulator arm end 930-1, a first actuator 960-1, a second manipulator arm 940-2, a second manipulator arm end 930-2, and a second actuator 960-2. The first actuator 960-1 is represented by a first bounding box 980-1, and the second actuator 960-2 is represented by a second bounding box 980-2. Figure 9 The bounding box is shown in a two-dimensional diagram, and those skilled in the art will understand that Figure 9 The bounding boxes shown in FIG. 9 (eg, the first bounding box 980 - 1 , the second bounding box 980 - 2 ), the first manipulator end 930 - 1 , the first actuator 960 - 1 , the second manipulator end 930 - 2 , and the second actuator 960 - 2 may also be shown in a three-dimensional structure.

[0123] In some embodiments, the first bounding box or the second bounding box can be generated based on any one of the Spheres bounding box detection method, the Aligned Axis Bounding Box (AABB) detection method, the Oriented Bounding Box (OBB) detection method, the Discrete Orientation Polytope (k-DOPs) detection method and the Fixed Direction Hull (FDH) detection method.

[0124] In some embodiments, the method 800 may further include controlling the first manipulator arm or the second manipulator arm to perform a corresponding collision avoidance operation in response to the first bounding box intersecting the second bounding box.

[0125] In some embodiments, method 800 may further include determining a virtual model corresponding to the first manipulator based on the first pose, and constructing a first bounding box including one or more first sub-bounding boxes for the virtual model of the first manipulator in the first pose state using the above-mentioned bounding box detection method. Determining a virtual model corresponding to the second manipulator based on the second pose, and constructing a second bounding box including one or more second sub-bounding boxes for the virtual model of the second manipulator in the second pose state using the above-mentioned bounding box detection method.

[0126] In some embodiments, method 800 may further include: updating a first sub-bounding box of the first bounding box and / or a second sub-bounding box of the second bounding box in response to the first bounding box intersecting the second bounding box; and determining a first anti-collision operation and / or a second anti-collision operation based on the updated first bounding box and the second bounding box.

[0127] In some embodiments, updating the first sub-bounding box of the first bounding box and / or the second sub-bounding box of the second bounding box may be, for example, constructing a smaller bounding box to characterize the structure enclosed by the first bounding box or the second bounding box. In some embodiments, constructing a smaller bounding box may be, for example, segmenting the first bounding box or the second bounding box. In some embodiments, a hierarchical first bounding box or a second bounding box may be constructed by updating the first sub-bounding box of the first bounding box or the second sub-bounding box of the second bounding box. In some embodiments, a first anti-collision operation or a second anti-collision operation may be determined based on the hierarchy or level of the intersecting first bounding box or the second bounding box, for example, determining the level of collision risk and generating an alarm signal of a corresponding level. The size of the bounding box after each update is smaller, and thus the accuracy of collision detection based on the updated bounding box is also higher.

[0128] Figure 10 Schematic diagram showing the update of bounding boxes in some embodiments of the present disclosure. Figure 10 As shown, the actuator 1060 (eg, the first actuator or the second actuator) may be, for example, a clamp. Figure 10 The process of updating the bounding box three times is shown, from the bounding box 1080a to the bounding box 1080b, to the bounding box 1080c, and then to the bounding box 1080d. The size of the bounding box gradually decreases, which can more finely represent the structure of the actuator 1060.

[0129] In other embodiments, a multi-level bounding box algorithm can be used to directly construct a bounding box. For example, a hierarchical bounding box of the first manipulator end or actuator can be constructed using an AABB detection method (in some embodiments, the hierarchical bounding box can also be called a bounding box tree).

[0130] In some embodiments, the present disclosure provides a method for determining a first collision avoidance operation and / or a second collision avoidance operation based on a collision evaluation index. The collision evaluation index can be used to characterize the proximity between the end of a first manipulator arm and the end of a second manipulator arm, or to characterize the proximity between a first actuator and a second actuator. In some embodiments, method 800 may further include determining the hierarchy or level of the intersecting first bounding box or second bounding box. Based on the determined hierarchy or level of the bounding box, a collision evaluation index is determined. Based on the collision evaluation index, the first collision avoidance operation and / or the second collision avoidance operation are determined. Those skilled in the art will appreciate that bounding boxes of different hierarchies correspond to different sizes, and a collision evaluation index determined based on a hierarchy of smaller bounding boxes represents a higher degree of proximity. In some embodiments, a specific first collision avoidance operation or second collision avoidance operation can be determined based on the collision evaluation index. For example, the collision evaluation index can be used to determine whether to stop the movement of the manipulator arm, reduce the movement speed of the manipulator arm, or generate a collision warning message. For example, the collision evaluation index can be used to determine whether to trigger collision warning messages of different alarm levels.

[0131] Figure 11 FIG. 1 is a flow chart showing a method 1100 for determining a first anti-collision operation or a second anti-collision operation according to some other embodiments of the present disclosure. Figure 11 As shown, some or all of the steps in the method 1100 may be performed by, for example Figure 1 The control device (eg, control device 120) of the manipulator control system 100 shown or Figure 24 The method 1100 is executed by the controller of the main control trolley 2402 and the operating trolley 2403 shown. The control device 120 can be configured on a computing device. The method 1100 can be implemented by software, firmware, and / or hardware. In some embodiments, the method 1100 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0132] See Figure 11 In step 1101, based on the first pose, a first envelope of a first actuator disposed at the end of a first manipulator is determined.

[0133] In step 1103 , based on the second pose, a second envelope of a second actuator disposed on the end of the second manipulator arm is determined.

[0134] At step 1105 , a first collision avoidance operation and / or a second collision avoidance operation is determined based on the first envelope and the second envelope.

[0135] Figure 12 Schematic diagram of envelope in some embodiments of the present disclosure is shown. Figure 12 As shown, the system includes: a first operating arm 1240-1, a first operating arm end 1230-1, a first actuator 1260-1, a second operating arm 1240-2, a second operating arm end 1230-2, and a second actuator 1260-2. The first actuator 1260-1 is represented by a first envelope 1280-1, and the second actuator 1260-2 is represented by a second envelope 1280-2.

[0136] In some embodiments, the edge of the virtual model of the first actuator is determined based on the first pose as the first envelope, or the edge of the virtual model of the second actuator is determined based on the second pose as the second envelope. In other embodiments, the corresponding envelope (e.g., the first envelope or the second envelope) can be obtained by expanding the edge of the virtual model by a certain distance.

[0137] In some embodiments, method 1100 may further include updating a first envelope of the first actuator in response to first control information from the master operator, wherein the first control information is used to adjust the operating state of the first actuator. Method 1100 may further include updating a second envelope of the second actuator in response to second control information from the master operator, wherein the second control information is used to adjust the operating state of the second actuator.

[0138] In some embodiments, the first actuator or the second actuator may be, for example, an actuator having a clamping or cutting function. The first control information or the second control information may be, for example, control information for opening or closing the first actuator or the second actuator (for example, Figure 26 (e.g., information about the opening and closing angle of clamp 26012 shown in FIG. ). In some embodiments, taking the first actuator as an example, method 1100 may further include updating the virtual model of the first actuator in response to first control information from the master operator; and determining a new first envelope based on the updated virtual model of the first actuator. For example, if the first actuator is a clamp, the master operator issues first control information to control the clamp to open. In response to the first control information, the virtual model of the clamp in the open state is updated, and a new first envelope is determined based on the virtual model of the clamp in the open state.

[0139] Figure 13 Schematic diagram showing envelope update in some embodiments of the present disclosure. Figure 13 As shown, the actuator 1360 (eg, the first actuator or the second actuator) may be, for example, a clamp. Figure 13The actuator 1360 is shown as being updated from a closed state to an open state, and the envelope of the actuator 1360 is also updated from envelope 1380a to a new envelope 1380b. The updated envelope 1380b can more accurately represent the actuator 1360 in the open working state.

[0140] In some embodiments, the present disclosure also provides another method for determining a first anti-collision operation and / or a second anti-collision operation based on a collision evaluation index. In some embodiments, method 1100 may further include: determining an overlapping range between the first envelope and the second envelope. Based on the overlapping range, determining a collision evaluation index. Based on the collision evaluation index, determining the first anti-collision operation and / or the second anti-collision operation. In some embodiments, the overlapping range of the first envelope and the second envelope can be represented by the overlapping area / volume of the first envelope and the second envelope, or the maximum width / depth of the overlapping portion.

[0141] Figure 14 A flow chart of a method 1400 for determining the pose of a manipulator coordinate system relative to a reference coordinate system according to some embodiments of the present disclosure is shown. In some embodiments, the method 1400 can be used to determine a first pose of a first manipulator end relative to a reference coordinate system or a second pose of a second manipulator end relative to a reference coordinate system. In some embodiments, the manipulator coordinate system can include a coordinate system of the manipulator end. For example, the coordinate system of the manipulator end 310. Some or all of the steps in the method 1400 can be performed by a control device (e.g., control device 120) or a control device of the manipulator control system 100. Figure 24 The method 1400 is performed by the controller of the main control trolley 2402 and the operating trolley 2403 shown. The control device 120 can be configured on a computing device. The method 1400 can be implemented by software, firmware, and / or hardware. In some embodiments, the method 1400 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0142] See Figure 14 In step 1401, based on the angle identifier and the plurality of posture identifiers, a roll angle of the posture identifier coordinate system relative to the manipulator coordinate system is determined. In some embodiments, a first rotation angle identified by the angle identifier in the manipulator coordinate system is determined. A second rotation angle identified by the first posture identifier in the posture identifier coordinate system is determined. Based on the first rotation angle and the second rotation angle, a roll angle of the posture identifier coordinate system relative to the manipulator coordinate system is determined. In some embodiments, the roll angle of the posture identifier coordinate system relative to the manipulator coordinate system can be determined based on formula (14).

[0143] In step 1403, based on multiple pose identifiers, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined. The coordinates of the pose identifier in the corresponding coordinate system can be represented by the coordinates of the corner points of the pose identifier pattern in the corresponding coordinate system. For example, the two-dimensional coordinates of the pose identifier in the positioning image and the three-dimensional coordinates in the pose identifier coordinate system can be represented by the coordinates of the corner points of the pose identifier pattern. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image and the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image, the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system.

[0144] In some embodiments, based on the distribution of multiple pose identifiers, the three-dimensional coordinates of multiple pose identifier pattern corner points in the pose identifier coordinate system are determined. Figure 16 Each corner point of the pose identification pattern is located on the circumference of the cross-sectional circle 1622. The center and radius r of the cross-sectional circle 1622 are both known. The center of the cross-sectional circle 1622 is set as the origin of the pose identification coordinate system. The XY plane is located on the cross-sectional circle 1622. The X axis can be specified to point from the origin to any determined corner point of the pose identification pattern (for example, the corner point P of the pose identification pattern). 16 ), and then the three-dimensional coordinates of each corner point of the pose identification pattern in the pose identification coordinate system can be determined based on the distribution of multiple pose identifications. Figure 16 As shown, the pose identification pattern corner point P 16 The three-dimensional coordinates of the pose marker coordinate system are (r, 0, 0), and the three-dimensional coordinates of the other pose marker pattern corners in the pose marker coordinate system can be calculated according to the following formula:

[0145] C m =[r·cos((m-1)·χ)r·sin((m-1)·χ)0] T (15)

[0146] Among them, C m To identify the pattern corner point P by pose 16 As the starting point, the three-dimensional coordinates of the mth pose identification pattern corner point in the pose identification coordinate system; χ is the angle around the axis between adjacent pose identification pattern corner points.

[0147] In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system can be known. For example, if the reference coordinate system is the world coordinate system, the transformation relationship between the camera coordinate system and the world coordinate system can be determined based on the camera's position. In other embodiments, depending on actual needs, the reference coordinate system can also be the camera coordinate system itself.

[0148] In some embodiments, based on the camera imaging principle and projection model, the pose of the pose identification coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identification patterns in the positioning image and the three-dimensional coordinates of the corner points of the multiple pose identification patterns in the pose identification coordinate system. Based on the pose of the pose identification coordinate system relative to the camera coordinate system and the transformation relationship of the camera coordinate system relative to the reference coordinate system, the pose of the pose identification coordinate system relative to the reference coordinate system can be obtained. In some embodiments, the intrinsic parameters of the camera can also be considered. For example, the intrinsic parameters of the camera can be such as Figure 1 The image acquisition device 110 shown or Figure 27 The camera intrinsic parameters of the imaging module 2760b are shown. The camera intrinsic parameters can be known or obtained through calibration.

[0149] In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the optical center of the camera as the origin, or a coordinate system established with the center of the camera lens as the origin. When the camera is a binocular camera, the origin of the camera coordinate system can be the center of the left lens, the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of the line).

[0150] See Figure 14 In step 1405, the pose of the manipulator coordinate system relative to the reference coordinate system is determined based on the roll angle of the pose identification coordinate system relative to the manipulator coordinate system and the pose of the pose identification coordinate system relative to the reference coordinate system. In some embodiments, the pose of the manipulator coordinate system relative to the reference coordinate system can serve as the current relative pose of the manipulator relative to the reference coordinate system.

[0151] For example, taking the world coordinate system as the reference coordinate system, the pose of the manipulator coordinate system relative to the world coordinate system is as follows:

[0152] w R wm = w R wm0 ·rot z (α0)

[0153] w P wm = w P wm0 (16)

[0154] in,w R wm is the posture of the manipulator coordinate system relative to the world coordinate system, w P wm is the position of the manipulator coordinate system relative to the world coordinate system, w R wm0 is the posture of the pose coordinate system relative to the world coordinate system, w P wm0 is the position of the pose coordinate system relative to the world coordinate system, rot z (α0) represents the roll angle α0 around the Z axis of the manipulator coordinate system.

[0155] In some embodiments, the specific calculation formula for the position of the manipulator coordinate system relative to the world coordinate system is as follows:

[0156] w R wm = w R lens lens R wm0 wm0 R wm

[0157] w P wm = w R lens ( lens R wm0 wm0 P wm + lens P wm0 )+ w P lens (17)

[0158] in, w R lens is the posture of the camera coordinate system relative to the world coordinate system, w P lens is the position of the camera coordinate system relative to the world coordinate system, lens R wm0 is the pose of the pose identification coordinate system relative to the camera coordinate system, lens P wm0 is the position of the pose identification coordinate system relative to the camera coordinate system, wm0 R wm is the posture of the manipulator coordinate system relative to the pose identification coordinate system, wm0 P wm The position of the manipulator coordinate system relative to the pose identification coordinate system.

[0159] Figure 15A flowchart of a method 1500 for determining the pose of a manipulator coordinate system relative to a reference coordinate system according to some other embodiments of the present disclosure is shown. The method 1500 may be Figure 14 Some or all of the steps in method 1500 may be performed by a control device (e.g., control device 120) or a controller of the manipulator control system 100. Figure 24 The method 1500 is executed by the controller of the main control trolley 2402 and the operating trolley 2403 shown. The control device 120 can be configured on a computing device. The method 1500 can be implemented by software, firmware, and / or hardware. In some embodiments, the method 1500 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0160] See Figure 15 In step 1501, the three-dimensional coordinates of the multiple pose identifiers in the manipulator coordinate system are determined based on the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system and the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system. It will be understood that, given the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system, the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system can be transformed into three-dimensional coordinates in the manipulator coordinate system using a coordinate transformation.

[0161] In step 1503, the pose of the manipulator coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple pose identifiers in the positioning image and the three-dimensional coordinates of the multiple pose identifiers in the manipulator coordinate system. In some embodiments, step 1503 can be implemented similarly to steps 1403 and 1405 in method 1400.

[0162] Figure 17 FIG1 is a flowchart of a method 1700 for identifying a posture identifier according to some embodiments of the present disclosure. Some or all of the steps in the method 1700 may be performed by a control device (e.g., control device 120) or a controller of the manipulator control system 100. Figure 24 The method 1700 is executed by the controller of the main control trolley 2402 and the operating trolley 2403 shown. The control device 120 can be configured on a computing device. The method 1700 can be implemented by software, firmware, and / or hardware. In some embodiments, the method 1700 can be implemented as computer-readable instructions. These instructions can be read and executed by a general-purpose processor or a special-purpose processor. In some embodiments, these instructions can be stored on a computer-readable medium.

[0163] See Figure 17In step 1701, multiple candidate pose identifiers are determined from the positioning image. In some embodiments, the candidate pose identifiers can be represented by corner points of a candidate pose identifier pattern. In some embodiments, the candidate pose identifier pattern corner points can refer to possible pose identifier pattern corner points obtained through preliminary processing or preliminary identification of the positioning image. In some embodiments, a region of interest (ROI) can be first captured from the positioning image, and multiple candidate pose identifiers can be determined from the ROI. The ROI can be the entire positioning image or a partial region. For example, the ROI of the current frame can be captured based on an area within a certain range of multiple pose identifier pattern corner points determined in a previous image frame (e.g., a positioning image from a previous image processing cycle). For positioning images other than the first frame, the ROI can be an area within a certain distance range centered on a virtual point formed by the coordinates of multiple pose identifier pattern corner points from the previous image processing cycle. The certain distance range can be a fixed multiple of the average spacing between the pose identifier pattern corner points, such as twice. It should be understood that the predetermined multiple can also be a variable multiple of the average spacing between the multiple candidate pose identifier pattern corner points from the previous image processing cycle.

[0164] In some embodiments, method 1700 may include determining a corner likelihood (CL) for each pixel in the positioning image. In some embodiments, the corner likelihood of a pixel may be a numerical value representing the likelihood of the pixel being a feature point (e.g., a corner point). In some embodiments, the positioning image may be preprocessed before calculating the corner likelihood of each pixel, and then the corner likelihood of each pixel in the preprocessed image may be determined. Image preprocessing may, for example, include at least one of image grayscale conversion, image denoising, and image enhancement.

[0165] For example, image preprocessing may include: extracting a ROI from the positioning image, and converting the ROI into a corresponding grayscale image.

[0166] In some embodiments, the method for determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first-order and / or second-order derivatives of each pixel. The corner likelihood value of each pixel is calculated using the first-order and / or second-order derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be calculated according to the following formula:

[0167] CL=max(c xy ,c 45 )

[0168] c xy =τ 2 ·|I xy |-1.5·τ·(|I45 |+I n45 )

[0169] c 45 =τ 2 ·|I 45_45 |-1.5·τ·(|I x |+I y ) (18)

[0170] Wherein, τ is a set constant, for example, set to 2; I x , I 45 , I y , I n45 are the first-order derivatives of the pixel points in the directions of 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 are the second-order derivatives of the pixel in the directions of 0, π / 2, π / 4, and -π / 4 respectively.

[0171] In some embodiments, the ROI is divided into multiple sub-images. For example, a non-maximum suppression method can be used to evenly divide multiple sub-images within an ROI range. In some embodiments, the ROI can be evenly divided into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not restrictive. It should be understood that the positioning image or ROI can also be divided into multiple sub-images of other sizes, for example, into multiple sub-images of 9×9 pixels. The pixel point with the largest CL value in each sub-image can be determined, and the pixel point with the largest CL value in each sub-image can be compared with a first threshold to determine a set of pixel points with a CL value greater than the first threshold. In some embodiments, the first threshold can be set to 0.06. It should be understood that the first threshold can also be set to other values. In some embodiments, pixel points with a CL value greater than the first threshold can be used as candidate pose identification pattern corner points.

[0172] See Figure 17 In step 1703, an initial pose identifier is identified from the plurality of candidate pose identifiers based on the pose pattern matching template. In some embodiments, the pose pattern matching template is matched with an image at one of the corner points of the candidate pose identifier pattern, and the corner point of the candidate pose identifier pattern that meets a preset pose pattern matching standard is determined as the initial pose identifier pattern corner point.

[0173] In some embodiments, the pose pattern matching template and the image of the area near the corner point of the pose identification pattern have the same or similar features. If the degree of matching between the pose pattern matching template and the image of the area near the corner point of the candidate pose identification pattern meets a preset pose pattern matching standard (for example, the matching degree is higher than a threshold), it can be considered that the pattern of the area near the corner point of the candidate pose identification pattern and the pose pattern matching template have the same or similar features, and the current candidate pose identification pattern corner point can be considered as the pose identification pattern corner point.

[0174] In some embodiments, the pixel point with the largest CL value in the pixel point set is determined as the corner point of the candidate pose identification pattern to be matched. For example, all the pixel points in the pixel point set can be sorted in descending order of CL value, and the pixel point with the largest CL value is used as the corner point of the candidate pose identification pattern to be matched. After the corner point of the candidate pose identification pattern to be matched is determined, the pose pattern matching template is used to match the pattern at the corner point of the candidate pose identification pattern to be matched. If the preset pose pattern matching standard is reached, the corner point of the candidate pose identification pattern to be matched is determined to be the identified initial pose identification pattern corner point. If the corner point of the candidate pose identification pattern to be matched does not meet the preset matching standard, the pixel point with the secondary CL value (the pixel point with the second largest CL value) is selected as the corner point of the candidate pose identification pattern to be matched, and the pose pattern matching template is used to match the image at the corner point of the candidate pose identification pattern, and so on, until the initial pose identification pattern corner point is identified.

[0175] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, so the pose pattern matching template can be the same checkerboard pattern, and the grayscale distribution G of the pose pattern matching template is used. M Grayscale distribution G of the pixel neighborhood corresponding to the corner point of the candidate pose identification pattern image The grayscale distribution G of the pixel neighborhood is used to match the pixel. image is the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the pixel. The specific formula is as follows:

[0176]

[0177] Wherein, Var is the variance function, and Cov is the covariance function. In some embodiments, when the CC value is less than 0.8, the grayscale distribution within the pixel area has a low correlation with the pose pattern matching template, and the candidate pose identification pattern corner point with the maximum corner likelihood value is determined to be the pose identification pattern corner point; otherwise, the candidate pose identification pattern corner point with the maximum corner likelihood value is determined to be the pose identification pattern corner point.

[0178] In some embodiments, method 1700 includes determining the edge directions of corner points of the candidate pose identification pattern. Figure 18 As shown, Figure 18 The pose identification pattern 1800 is included, and the candidate pose identification pattern corner point is the corner point P in the pose identification pattern 1800. 1801 , then the corner point P 1801 The edge direction can refer to the corner point P 1801 The direction of the edge, such as Figure 18 The dotted arrow indicates the direction.

[0179] In some embodiments, the edge direction can be obtained by calculating the first-order derivative value (I) of each pixel in a certain range of neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate pose identification pattern in the X direction and the Y direction of the plane coordinate system. x and I y ). For example, the edge direction can be calculated using the following formula:

[0180]

[0181] Among them, the first-order derivative (I x and I y ) can be obtained by performing a convolution operation on each pixel point within a certain range of neighborhood. In some embodiments, by calculating the edge direction I of the pixel points within each range of neighborhood, angle and the corresponding weight I weight Perform clustering calculation to obtain the edge direction of the pixel point and select weight I weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if there are multiple edge directions, the weight I is selected. weight I corresponding to the largest number of classes angle as the edge direction.

[0182] In some embodiments, the method used for clustering calculation can be any one of the K-means method, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies) method, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, and GMM (Gaussian Mixed Model) method.

[0183] In some embodiments, method 1700 includes: rotating the pose pattern matching template according to the edge direction. Rotating the pose pattern matching template according to the edge direction can align the pose pattern matching template with the image at the corner point of the candidate pose identification pattern.

[0184] The edge orientation of the corner points of the candidate pose identification pattern can be used to determine the orientation of the image at the corner points of the candidate identification pattern in the positioning image. In some embodiments, the pose pattern matching template can be rotated based on the edge orientation to adjust the pose pattern matching template to the same or nearly the same orientation as the image at the corner points of the candidate pose identification pattern to facilitate image matching.

[0185] See Figure 17 In step 1705, the pose identifier is searched with the initial pose identifier as the starting point.

[0186] For example, Figure 19 FIG. 1 is a flow chart showing a method 1900 for searching for a pose identifier according to some embodiments of the present disclosure. Figure 19 As shown, some or all of the steps in the method 1900 may be performed by a data processing device (e.g., Figure 1 The control device 120 shown or Figure 24 The method 1900 may be executed by a controller of the main control trolley 2402 or the operating trolley 2403 shown. The control device 120 may be configured on a computing device. The method 1900 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 1900 may be implemented as computer-readable instructions. These instructions may be read and executed by a general-purpose processor or a dedicated processor. In some embodiments, these instructions may be stored on a computer-readable medium.

[0187] See Figure 19 In step 1901, a second pose identifier is determined with the initial pose identifier as a starting point. In some embodiments, a second pose identifier pattern corner point is searched for in a set search direction with the initial pose identifier pattern corner point as a starting point. In some embodiments, the set search direction may include at least one of: a direction directly in front of the initial pose identifier pattern corner point (corresponding to a 0° angle direction), a direction directly behind (corresponding to a 180° angle direction), a direction directly above (a 90° angle direction), a direction directly below (a -90° angle direction), and an oblique direction (e.g., a ±45° angle direction).

[0188] In some embodiments, the number of search directions is n, for example, searching in 8 directions, each search direction v sn It can be calculated according to the following formula:

[0189] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (21)

[0190] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent pose identification pattern corner points among the multiple pose identification pattern corner points determined in the previous frame. For example, the predetermined search direction can be calculated according to the following formula:

[0191]

[0192]

[0193] Among them, (x j ,y j ) are the two-dimensional coordinates of the corner points of the multiple pose identification patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple pose identification patterns determined for the previous frame; v s1 The search direction for the first setting; v s2 The search direction for the second setting.

[0194] In some embodiments, as Figure 20 As shown, the pattern corner point P is identified with the initial pose 2001 The coordinate position of the second pose identification pattern corner point P is used as the search starting point and searched in the set search direction. 2002 The coordinate position of the pattern corner point P can be specifically included: 2001 The coordinate position of the search is used as the search starting point, and the search box (for example, Figure 20 The dotted box in the figure) searches in the set search direction V with a certain search step size. 2001 If there is at least one candidate pose identification pattern corner point in the search box, the candidate pose identification pattern corner point with the largest corner likelihood value in the search box is preferentially selected as the second pose identification pattern corner point P 2002 When the search box is limited to a suitable size, the pattern corner point P is identified with the initial pose 2001 The coordinate position of the pattern corner point P is used as the search starting point for the second pose identification 2002 During the search, the candidate pose identification pattern corner point with the largest likelihood value among the candidate pose identification pattern corner points in the search box is more likely to be the pose identification pattern corner point. Therefore, it can be considered that the candidate pose identification pattern corner point with the largest likelihood value in the search box is the second pose identification pattern corner point P 2002, in order to improve the data processing speed. In other implementations, in order to improve the accuracy of pose identification pattern corner point recognition, when there is at least one candidate pose identification pattern corner point in the search box, the candidate pose identification pattern corner point with the largest corner point likelihood value among the candidate pose identification pattern corner points appearing in the search box is selected for corner point recognition to determine whether the candidate pose identification pattern corner point with the largest corner point likelihood value is the pose identification pattern corner point. For example, the pose pattern matching template is matched with the image within a certain range of the candidate pose identification pattern corner point with the largest corner point likelihood value, and the candidate pose identification pattern corner point that meets the preset pose pattern matching degree standard can be considered to be the second pose identification pattern corner point P searched. 2002 .

[0195] In some embodiments, continue to see Figure 20 , the size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can be changed synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0196] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, and the correlation coefficient CC in formula (19) can be used for pattern matching. If CC is greater than a threshold, the candidate pose identification pattern corner point with the largest corner likelihood value is considered to be the pose identification pattern corner point and is recorded as the second pose identification pattern corner point.

[0197] See Figure 19 In step 1903, based on the initial pose identifier and the second pose identifier, a search direction is determined. In some embodiments, the search direction includes: a first search direction and a second search direction. The first search direction can be a direction starting from the coordinate position of the corner point of the initial pose identifier pattern and away from the corner point of the second pose identifier pattern. The second search direction can be a direction starting from the coordinate position of the corner point of the second pose identifier pattern and away from the corner point of the first pose identifier pattern. For example, Figure 20 The search direction V shown in 2002 .

[0198] In step 1905, the pose identifier is searched in the search direction with the initial pose identifier or the second pose identifier as the starting point. In some embodiments, if the first pose identifier pattern corner point is used as the new starting point, the first search direction in the above embodiment can be used as the search direction to search for the pose identifier pattern corner point. If the second pose identifier pattern corner point is used as the new search starting point, the second search direction in the above embodiment can be used as the search direction to search for the pose identifier pattern corner point. In some embodiments, the new pose identifier pattern corner point (for example, Figure 20 The third pose in the pattern corner P 2003) can be performed similarly to step 1901. In some embodiments, the search step size can be the distance L1 between the corner point of the initial pose identification pattern and the corner point of the second pose identification pattern.

[0199] In some embodiments, in response to the number of corner points of the pose marker pattern being greater than or equal to a threshold number of corner points of the pose marker pattern, the search for corner points of the pose marker pattern is stopped. For example, when four corner points of the pose marker pattern are searched (identified), the search for corner points of the pose marker pattern is stopped.

[0200] In some embodiments, in response to the search distance being greater than a set multiple of the distance between the N-1th pose identification pattern corner point and the N-2th pose identification pattern corner point, the search for the Nth pose identification pattern corner point is stopped, where N≥3. For example, the end condition of the search may be that the search distance is greater than twice the distance between the first two pose identification pattern corner points. In this way, the maximum search distance for searching the third pose identification pattern corner point is twice the distance between the initial pose identification pattern corner point and the second pose identification pattern corner point. If the pose identification pattern corner point has not been found after reaching the search distance, it is deemed that the third pose identification pattern corner point has not been found and the search ends.

[0201] In some embodiments, if the total number of pose identification pattern corner points searched is greater than or equal to a set threshold value (for example, the set threshold value is 4), it is considered that sufficient pose identification pattern corner points have been successfully identified. If the total number of pose identification pattern corner points found is less than the set value, it is considered that the search based on the initial pose identification pattern corner points in the above steps is unsuccessful. In the event of an unsuccessful search, new initial pose identification pattern corner points are re-determined from the candidate pose identification pattern corner points, and then the remaining pose identification pattern corner points are searched based on the re-determined initial pose identification pattern corner points as the search starting point. Similar to method 1700, new initial pose identification pattern corner points can be re-determined, and similar to method 1900, the remaining pose identification pattern corner points can be searched with the new pose identification pattern corner points as the search starting point.

[0202] In some embodiments, after searching for or identifying the corner points of the pose identification pattern, sub-pixel positioning may be performed on the determined corner points of the pose identification pattern to improve the position accuracy of the corner points of the pose identification pattern.

[0203] In some embodiments, the CL values of the pixels can be fitted based on a model to determine the coordinates of the corner points of the pose identification pattern after sub-pixel positioning. For example, the fitting function of the CL value of each pixel in the ROI can be a quadratic surface function, where the extreme points of the function are sub-pixel points. The fitting function can be as follows:

[0204] S(x,y)=ax 2 +by2 +cx+dy+exy+f (23)

[0205]

[0206] Among them, S(x, y) is the CL value fitting function of all pixels in each ROI, a, b, c, d, e, f are coefficients; x c is the x coordinate of the pose marker, y c is the y coordinate of the pose identifier.

[0207] Figure 21 FIG. 2 is a flow chart showing a method 2100 for identifying an angle marker according to some embodiments of the present disclosure. Figure 21 As shown, some or all of the steps in the method 2100 may be performed by a data processing device (e.g., Figure 1 The control device 120 shown or Figure 24 The method 2100 may be performed by a controller of the main control trolley 2402 and the operating trolley 2403 shown in FIG. The control device 120 may be configured on a computing device. The method 2100 may be implemented by software, firmware, and / or hardware. In some embodiments, the method 2100 may be implemented as computer-readable instructions. These instructions may be read and executed by a general-purpose processor or a dedicated processor. In some embodiments, these instructions may be stored on a computer-readable medium.

[0208] See Figure 21 In step 2101, an imaging transformation relationship is determined based on the two-dimensional coordinates of the plurality of posture identifiers in the positioning image and the three-dimensional coordinates of the plurality of posture identifiers in the posture identifier coordinate system. In some embodiments, the posture identifier coordinate system may be the posture identifier coordinate system described in detail in the embodiment shown in method 700. For example, the posture identifier coordinate system is as follows: Figure 6 As shown. In some embodiments, the imaging transformation relationship may refer to the transformation relationship between the three-dimensional coordinates in the pose identification coordinate system and the two-dimensional coordinates in the positioning image. It should be understood that based on the imaging transformation relationship, the two-dimensional coordinates in the positioning image may also be transformed into three-dimensional coordinates in the pose identification coordinate system. In some embodiments, the three-dimensional coordinates of multiple pose identifiers in the pose identification coordinate system may be determined based on formula (15). In some embodiments, the number of multiple pose identifiers may be greater than or equal to 4. For example, the imaging transformation relationship may be obtained based on the two-dimensional coordinates of the four pose identifiers in the positioning image and the corresponding four three-dimensional coordinates in the pose identification coordinate system.

[0209] See Figure 21In step 2103, based on the imaging transformation relationship, the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system and the position association relationship, multiple angle identification candidate areas are determined in the positioning image. In some embodiments, the angle identification candidate area may represent a candidate area of the angle identification pattern. In some embodiments, based on the three-dimensional coordinates of the multiple pose identifier pattern corner points in the pose identifier coordinate system and the position association relationship, multiple angle identification pattern corner point candidate three-dimensional coordinates are determined in the pose identifier coordinate system. For example, based on the three-dimensional coordinates of the multiple pose identifier pattern corner points in the pose identifier coordinate system, multiple three-dimensional coordinates in the pose identifier coordinate system can be determined by axially offsetting a certain distance. These three-dimensional coordinates are represented by multiple angle identification pattern corner point candidate three-dimensional coordinates. For example, see Figure 4 The position association relationship is that the angle marker and the corresponding pose marker are separated by a certain distance along the Z axis of the pose marker coordinate system. Given the position of the corner point of the pose marker pattern, the position obtained by moving a certain distance along the positive or negative direction of the Z axis can be considered as the candidate position of the angle marker pattern corner point in the pose marker coordinate system.

[0210] In some embodiments, based on the imaging transformation relationship and the three-dimensional coordinates of the multiple angle identification pattern corner point candidates, multiple angle identification pattern candidate areas are determined in the positioning image. For example, based on the imaging transformation relationship and the three-dimensional coordinates of the multiple angle identification pattern corner point candidates, multiple angle identification pattern corner point candidate two-dimensional coordinates are obtained in the positioning image. In some embodiments, based on the two-dimensional coordinates of the multiple angle identification pattern corner point candidates, multiple angle identification pattern candidate areas are determined. For example, with each angle identification pattern corner point candidate two-dimensional coordinate as the center, an area of a certain range size (for example, 5×5 pixels, 10×10 pixels, etc.) is determined in the positioning image as the angle identification candidate area. In some embodiments, the area of a certain range size is greater than or equal to the size of the angle identification pattern after imaging. The size of the angle identification pattern after imaging can be obtained based on the actual size of the angle identification pattern and the imaging transformation relationship.

[0211] See Figure 21In step 2105, the angle identification candidate areas are identified from multiple angles. In some embodiments, the angle identification includes an angle identification pattern and an angle identification pattern corner point. In some embodiments, the method 2100 may include determining the pixel point with the largest corner likelihood value in each angle identification candidate area to form a pixel set. In some embodiments, the corner likelihood value of the pixel point may be calculated when executing method 1700, or may be recalculated based on formula (18). The method 2100 also includes determining the angle identification candidate area corresponding to the pixel point with the largest corner likelihood value in the pixel set as the angle identification candidate area to be identified. The method 2100 also includes using multiple angle pattern matching templates to match the angle identification candidate areas to be identified respectively to identify the angle identification. In some embodiments, the angle identification pattern is a pattern with different graphic features. The multiple angle pattern matching templates may refer to standard angle pattern templates with the same or similar graphic features corresponding to the multiple angle identification patterns. In some embodiments, by determining multiple angle identification candidate areas, the angle identification can be identified in the multiple angle identification candidate areas, avoiding the angle identification identification in the entire image range and improving the speed of data processing.

[0212] In some embodiments, any template matching algorithm including square difference matching method, normalized square difference matching method, correlation matching method, normalized correlation matching method, correlation coefficient matching method and normalized correlation coefficient matching method can be used to perform matching operation on the angle pattern matching template and the angle identification candidate area.

[0213] In some embodiments, because the angle pattern matching template and the angle identification pattern have the same or similar graphical features, the pattern information of the angle identification may include the pattern information of the corresponding angle pattern matching template. For example, the shape of the angle pattern matching template, identifiable features in the image, etc. In some embodiments, each angle pattern matching template has a one-to-one correspondence with the rotation angle identified by the corresponding angle identification pattern. The first rotation angle is determined based on the pattern information of the angle identification pattern corresponding to the specific angle pattern matching template or the identified angle identification.

[0214] In some embodiments, method 2100 may include, in response to a matching failure, determining an angle identifier candidate region corresponding to a pixel having a maximum corner likelihood value among the remaining pixels in the pixel set as the angle identifier candidate region to be identified. In some embodiments, after determining a new angle identifier candidate region to be identified, multiple angle pattern matching templates are used to match the angle identifier candidate regions to be identified to identify the angle identifier.

[0215] In some embodiments, a first pose identifier having a positional association with the angle identifier is determined based on an angle identifier candidate region where the identified angle identifier is located. In some embodiments, multiple angle identifier candidate regions each correspond to at least one of multiple identified pose identifier pattern corner points. After determining the angle identifier candidate region where the identified angle identifier is located, a first pose identifier pattern corner point can be determined based on the correspondence between the multiple angle identifier candidate regions and the multiple pose identifier pattern corner points.

[0216] In some embodiments of the present disclosure, the present disclosure further provides a computer device, the computer device including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory and is used to execute at least one instruction to perform some or all steps in the method of the present disclosure, such as Figure 7 、 Figure 8 、 Figure 11 、 Figure 14 、 Figure 15 、 Figure 17 、 Figure 19 and Figure 21 Some or all of the steps in the method disclosed in.

[0217] Figure 22 2200 according to some embodiments of the present disclosure. Figure 22 The computer device 2200 may include a central processing unit (CPU) 2201, a system memory 2204 including a random access memory (RAM) 2202 and a read-only memory (ROM) 2203, and a system bus 2205 connecting the various components. The computer device 2200 may also include input / output devices 2206 and a mass storage device 2207 for storing an operating system 2213, application programs 2214, and other program modules 2215. The input / output devices 2206 include an input / output controller 2210 mainly composed of a display 2208 and input devices 2209.

[0218] The mass storage device 2207 is connected to the central processing unit 2201 through a mass storage controller (not shown) connected to the system bus 2205. The mass storage device 2207 or computer-readable medium provides non-volatile storage for the computer device. The mass storage device 2207 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0219] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include read-only memory, random access memory, flash memory or other solid-state storage technologies, read-only optical disks or other optical storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media is not limited to the aforementioned types. The above-mentioned system memory and mass storage devices may be collectively referred to as memory.

[0220] The computer device 2200 can be connected to a network 2212 via a network interface unit 2211 connected to the system bus 2205 .

[0221] The system memory 2204 or the mass storage device 2207 is further configured to store one or more instructions. The central processing unit 2201 implements all or part of the steps of the method in some embodiments of the present disclosure by executing the one or more instructions.

[0222] In some embodiments of the present disclosure, the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is executed by a processor to enable a computer to perform some or all steps of the method of some embodiments of the present disclosure, such as Figure 7 、 Figure 8 、 Figure 11 、 Figure 14 、 Figure 15 、 Figure 17 、 Figure 19 and Figure 21 Examples of computer-readable storage media include storage of computer programs (instructions), such as read-only memory, random access memory, read-only optical discs, magnetic tapes, floppy disks, and optical data storage devices.

[0223] Figure 23 A schematic diagram of a surgical robot system 2300 according to some embodiments of the present disclosure is shown. Figure 23The surgical robot system 2300 may include: at least two surgical tools 2301, an image collector 2310 and a control device 2320 (for example, a processor), the first surgical tool among the at least two surgical tools 2301 includes a first operating arm, an actuator provided at the end of the first operating arm, and a plurality of first operating arm identifiers provided on the end of the first operating arm, the plurality of first operating arm identifiers including at least one first operating arm angle identifier and a plurality of first operating arm posture identifiers, at least one first operating arm angle identifier having a positional association relationship with the first posture identifier among the plurality of first operating arm posture identifiers. The image collector 2310 can be used to collect the positioning image of the first operating arm. The control device 2320 is connected to the image collector 2310, and is used to perform some or all of the steps in the method of some embodiments of the present disclosure, such as Figure 7 、 Figure 8 、 Figure 11 、 Figure 14 、 Figure 15 、 Figure 17 、 Figure 19 and Figure 21 In some embodiments, the surgical tool 2301 may be, for example, Figure 25 Surgical tool 2500 is shown in FIG.

[0224] Figure 24 A schematic diagram of a surgical robot system 2400 according to some embodiments of the present disclosure is shown. Figure 24 The surgical robot system 2400 may include a surgical tool 2401, a main control trolley 2402, and an operating trolley 2403. The operating trolley 2403 is provided with a drive module for driving the surgical tool 2401. The surgical tool 2401 is mounted on the operating trolley 2403 and connected to the drive module. The main control trolley 2402 is in communication with the operating trolley 2403 to control the surgical tool 2401 to perform the surgical operation. In some embodiments, the main control trolley 2402 may be used to perform some or all of the steps in the method of some embodiments of the present disclosure, such as Figure 7 、 Figure 8 、 Figure 11 、 Figure 14 、 Figure 15 、 Figure 17 、 Figure 19 and Figure 21 In some embodiments, the control trolley 2402 and the operating trolley 2403 are connected by wired or wireless transmission. For example, the control trolley 2402 and the operating trolley 2403 can be connected by a cable.

[0225] In some embodiments, the surgical robot system 2400 includes at least two surgical tools 2401, and the surgical tool 2401 includes an operating arm and an actuator provided at the end of the operating arm. In some embodiments, the surgical robot system 2400 may include an operating trolley 2403, and one operating trolley 2403 can be installed with at least two surgical tools 2401. In some embodiments, the surgical robot system 2400 may include at least two operating trolleys 2403, and each operating trolley 2403 is installed with a surgical tool 2401. In some embodiments, the surgical robot system 2400 may also include an imaging tool 2404. The imaging tool 2404 may include an operating arm and an imaging module provided at the end of the operating arm. The imaging tool 2404 may be provided on the operating trolley 2403 and driven by a corresponding driving module. The image of the operating arm and the actuator of the surgical tool 2401 obtained by the imaging module can be transmitted to the main control trolley 2402. In some embodiments, the surgical tool 2401 is, for example Figure 25 In some embodiments, the main control trolley 2402 is, for example, Figure 26 In some embodiments, the operating table 2403 is, for example, Figure 27 The surgical trolley 2700 is shown in FIG.

[0226] Figure 25 Schematic diagram showing a surgical tool 2500 according to some embodiments of the present disclosure. Figure 25 , the surgical tool 2500 includes a drive transmission device 2590, an operating arm 2540 and an actuator 2560 arranged at the end of the operating arm. In some embodiments, the drive transmission device 2590 can cooperate with the drive module to drive the operating arm 2540 to move. The drive transmission device 2590 is used to connect with the drive module, and the driving force of the drive module is transmitted to the operating arm 2540 through the drive transmission device 2590, thereby driving the operating arm 2540 to achieve multi-degree-of-freedom movement. The drive module can also control the actuator 2560 to perform surgical operations. In some embodiments of the present disclosure, the actuator 2560 may include but is not limited to a bipolar curved separation forceps actuator, a bipolar elbow grasping forceps actuator, a monopolar curved shears actuator, a monopolar electric hook actuator, a bipolar grasping forceps actuator, a needle holding forceps actuator and a tissue grasping forceps actuator. In some embodiments, the surgical tool 2500 can be installed, for example Figure 24 The surgical trolley 2403 shown in or Figure 27 The surgical trolley 2700 is shown in FIG.

[0227] Figure 26 Schematic diagram of the main control trolley 2600 of some embodiments of the present disclosure is shown. Figure 26The main control trolley 2600 includes: a controller (the controller can be configured on a computer device and set inside the main control trolley 2600), a main operator 2601, a main control trolley display (such as displays 2602-2604) and pedals (such as pedals 2605-2607). The controller is respectively connected to the main operator 2601, the main control trolley display and the pedals for communicating with the main operator 2601, the main control trolley display and the pedals, and generates corresponding control instructions based on the collected control information. In some embodiments, the controller is also connected to the operating trolley, for example, Figure 24 The operating table 2403 shown in FIG is in communication connection, and is used to control the surgical tool 2401 to perform a surgical operation or to control the imaging tool 2404 to work. In some embodiments, the controller of the main control table 2600 can also be used to perform some or all steps in the method of some embodiments of the present disclosure, such as Figure 7 、 Figure 8 、 Figure 11 、 Figure 14 、 Figure 15 、 Figure 17 、 Figure 19 and Figure 21 Some or all of the steps in the method disclosed in.

[0228] In some embodiments, the master manipulator 2601 typically includes a left master manipulator (e.g., for controlling a first manipulator arm) and a right master manipulator (e.g., for controlling a second manipulator arm) corresponding to the left hand of the medical staff member and the right master manipulator (e.g., for controlling a second manipulator arm) operated by the right hand. In actual scenarios, the master manipulator 2601 is used to collect the operation input of the medical staff member, and the medical staff member controls the movement of the surgical tool or imaging tool in the operation area by remotely operating the master manipulator 2601 to perform medical operations. In some embodiments, the master manipulator 2601 includes a multi-degree-of-freedom robotic arm 26011, and a master manipulator sensor is provided at each joint of the multi-degree-of-freedom robotic arm 26011. Joint information (such as joint angle data) is generated by the master manipulator sensor of each joint. In some embodiments, the master manipulator sensor uses a potentiometer and / or an encoder. In some embodiments, the multi-degree-of-freedom robotic arm 26011 has six degrees of freedom. In some embodiments, the posture of the master manipulator 2601 can be represented by a set of joint information of the master manipulator joints (e.g., a one-dimensional matrix composed of these joint information). In some embodiments, the main operator 2601 further includes a clamp 26012, which can be used to control the opening and closing angle of the actuator. In some embodiments, the main control trolley display includes a stereoscopic display 2602, a main control external display 2603, and a main control touch display 2604. The stereoscopic display 2602 displays the surgical image and system status prompts, the main control external display 2603 displays the surgical image and system status prompts, and the touch display 2604 displays the software user interface of the main control trolley 2600. In some embodiments, the image displayed by the stereoscopic display 2602 or the main control external display 2603 can be determined based on the image acquired by the imaging module, for example Figure 27 In some embodiments, the main control trolley pedal is used to collect the input of the medical staff's feet, including the electric cutting pedal 2605, the electric coagulation pedal 2606, the clutch pedal 2607 and other structures.

[0229] Figure 27 Schematic diagram of an operating table trolley 2700 according to some embodiments of the present disclosure. Figure 27The operating trolley 2700 includes: a controller (the controller can be configured on a computer device and set inside the operating trolley 2700), an operating trolley chassis 2702, an operating trolley case 2703, a system status display 2705, a main column 2706, a main beam 2707, a positioning arm 2708, a drive module 2709 and other components. The operating trolley chassis 2702 is used to realize the movement and fixing functions of the operating trolley 2700. The operating trolley case 2703 is used to integrate the electrical components of the operating trolley internally. The system status display 2705 is used to display the operating trolley system user interface and receive user input. The main column 2706 can be raised and lowered, and its top end is fixed to the main beam 2707. There is a beam platform at the end of the main beam 2707, and multiple positioning arms 2708 are fixed to the lower end of the beam platform. The positioning arm 2708 is equipped with a driving module 2709, and the driving module 2709 is used to load the surgical tool 2701 or the imaging tool 2704 (the imaging tool 2704 can be, for example, a 3D electronic endoscope). In some embodiments, the operating trolley 2700 integrates multiple positioning arms 2708, and each positioning arm 2708 has multiple motion joints. In some embodiments, the operating trolley 2700 is integrated with multiple surgical tools 2701 and imaging tools 2704, and the partial operating arms 2740a and actuators 2760a of the multiple surgical tools 2701 and the partial operating arms 2740b and imaging modules 2760b of the imaging tools 2704 enter the working space through the sheath 2710. In some embodiments, the controller of the operating trolley 2700 can also be used to execute some or all of the steps in the method of some embodiments of the present disclosure, such as Figure 7 、 Figure 8 、 Figure 11 、 Figure 14 、 Figure 15 、 Figure 17 、 Figure 19 and Figure 21 Some or all of the steps in the method disclosed in.

[0230] During the operation of a robotic system, especially during a surgical robot operation, collisions of the manipulator arms can bring unexpected risks, cause operational failures, and even lead to irreparable losses. In an embodiment of the present disclosure, the position of the manipulator arm can be detected to detect collision risks. For example, the position of multiple manipulator arms can be detected, and if a collision risk is found, a collision warning can be issued, such as issuing an alarm, or an evasive action can be performed, such as pausing the movement or moving in the opposite direction. The embodiments of the present disclosure can reduce or even avoid the collision risk during the operation of the robotic system, thereby significantly improving the safety of the robotic system.

[0231] Although specific embodiments of the present disclosure have been illustrated and described, it will be apparent to those skilled in the art that various other changes and modifications may be made without departing from the spirit and scope of the present disclosure. It is therefore intended that all such changes and modifications as fall within the scope of the present disclosure be included in the appended claims.

Claims

1. A method for controlling a manipulator arm of a robotic system, wherein the robotic system includes at least two manipulator arms, the method comprising: Acquire positioning images; identifying, in the positioning image, a plurality of first manipulator arm position and posture identifiers located on a first manipulator arm end of a first manipulator arm of the at least two manipulator arms; Based on the multiple first operating arm posture identifiers, identifying a first operating arm angle identifier located at the end of the first operating arm, the first operating arm angle identifier having a position association relationship with a first posture identifier among the multiple first operating arm posture identifiers, wherein the position association relationship includes an axial correspondence between the first operating arm angle identifier and the first posture identifier among the multiple first operating arm posture identifiers; Determining a first pose of the first manipulator arm end relative to a reference coordinate system based on the first manipulator arm angle identifier and the plurality of first manipulator arm pose identifiers; as well as determining a first collision avoidance operation for the first manipulator based on the first pose; The control method further includes: Determining a roll angle of a first manipulator arm posture identifier coordinate system relative to a first manipulator arm coordinate system based on the first manipulator arm angle identifier and the plurality of first manipulator arm posture identifiers; The at least two operating arms include a second operating arm, and the control method further includes: determining a second pose of a second manipulator end of the second manipulator relative to the reference coordinate system; and Based on the first pose and the second pose, the first collision avoidance maneuver and / or a second collision avoidance maneuver for the second manipulator arm is determined.

2. The control method according to claim 1, further comprising: Determine, based on the first pose, a first bounding box of a first actuator disposed on an end of the first manipulator, wherein the first bounding box includes one or more first sub-bounding boxes; Determining a second bounding box of a second actuator disposed on an end of the second manipulator based on the second pose, wherein the second bounding box includes one or more second sub-bounding boxes; as well as The first collision avoidance operation and / or the second collision avoidance operation are determined based on the first bounding box and the second bounding box.

3. The control method according to claim 2, further comprising: In response to the first bounding box intersecting with the second bounding box, updating a first sub-bounding box of the first bounding box and / or a second sub-bounding box of the second bounding box; The first collision avoidance operation and / or the second collision avoidance operation are determined based on the updated first bounding box and the second bounding box.

4. The control method according to claim 1, further comprising: determining a first envelope of a first actuator disposed at a distal end of the first manipulator based on the first pose; determining a second envelope of a second actuator disposed on a distal end of the second manipulator based on the second pose; as well as The first collision avoidance operation and / or the second collision avoidance operation are determined based on the first envelope and the second envelope.

5. The control method according to claim 4, further comprising: updating a first envelope of the first actuator in response to first control information from a master operator, wherein the first control information is used to adjust a working state of the first actuator; and / or In response to second control information from the master operator, a second envelope of the second actuator is updated, wherein the second control information is used to adjust the working state of the second actuator.

6. The control method according to claim 4, further comprising: determining an overlapping range of the first envelope and the second envelope; as well as determining a collision evaluation index of the first operating arm end and the second operating arm end based on the overlapping range; as well as The first collision avoidance operation and / or the second collision avoidance operation are determined based on the collision evaluation index.

7. The control method according to claim 1, further comprising: Based on the driving information and the kinematic model of the second manipulator, a kinematic posture of the end of the second manipulator is determined as the second posture.

8. The control method according to claim 1, further comprising: In the positioning image, identifying a plurality of second manipulator arm position and posture identifiers located on an end of the second manipulator arm; Based on the plurality of second operating arm posture identifiers, identifying a second operating arm angle identifier located at an end of the second operating arm, the second operating arm angle identifier being in a positionally associated relationship with a first posture identifier among the plurality of second operating arm posture identifiers; and The second posture is determined based on the second operating arm angle identifier and the plurality of second operating arm posture identifiers.

9. The control method according to claim 8, further comprising: In response to not recognizing the first manipulator arm posture identifier in the positioning image, determining a first kinematic posture of the first manipulator arm end as the first posture based on the driving information and kinematic model of the first manipulator or based on the posture of a master manipulator; and / or In response to not identifying the second manipulator arm posture identifier in the positioning image, a second kinematic posture of the end of the second manipulator arm is determined as the second posture based on the driving information and kinematic model of the second manipulator or based on the posture of the main manipulator.

10. The control method according to claim 8, further comprising: Determining a first kinematic posture of a distal end of the first manipulator based on the driving information and a kinematic model of the first manipulator or based on a posture of a master manipulator; as well as determining the first posture from the first posture and the second posture based on the first kinematic posture, and / or Determining a second kinematic posture of a tip of the second manipulator arm based on the driving information and the kinematic model of the second manipulator arm or based on the posture of the master manipulator; as well as Based on the second kinematic posture, the second posture is determined from the first posture and the second posture.

11. The control method according to any one of claims 1 to 10, wherein the first collision avoidance operation comprises at least one of the following: stopping the movement of the first operating arm; or Generate collision warning information. 12 . The control method according to claim 11 , wherein the collision warning information comprises collision warning information of multiple different warning levels.

13. The control method according to claim 1, comprising: Determining, based on the plurality of first manipulator pose identifiers, a pose of a coordinate system of the first manipulator pose identifier relative to the reference coordinate system; as well as The posture of the first manipulator arm coordinate system relative to the reference coordinate system is determined based on the roll angle of the first manipulator arm posture identification coordinate system relative to the first manipulator arm coordinate system and the posture of the first manipulator arm posture identification coordinate system relative to the reference coordinate system.

14. The control method according to claim 13, comprising: Based on the two-dimensional coordinates of the multiple first manipulator arm posture identifiers in the positioning image and the three-dimensional coordinates of the multiple first manipulator arm posture identifiers in the first manipulator arm posture identifier coordinate system, the posture of the first manipulator arm posture identifier coordinate system relative to the reference coordinate system is determined.

15. The control method according to claim 1, comprising: Determining the three-dimensional coordinates of the multiple first manipulator arm posture identifiers in the first manipulator arm coordinate system based on the roll angle of the first manipulator arm posture identifier coordinate system relative to the first manipulator arm coordinate system and the three-dimensional coordinates of the multiple first manipulator arm posture identifiers in the first manipulator arm posture identifier coordinate system; as well as The posture of the first manipulator coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple first manipulator pose identifiers in the positioning image and the three-dimensional coordinates of the multiple first manipulator pose identifiers in the first manipulator coordinate system.

16. The control method according to any one of claims 13 to 15, comprising: determining a first angle around an axis identified by the first operating arm angle identifier in the first operating arm coordinate system; Determine a second rotation angle of a first pose identifier among the plurality of first manipulator arm pose identifiers in the first manipulator arm pose identifier coordinate system; as well as Based on the first rotational axis angle and the second rotational axis angle, a roll angle of the first operating arm posture identification coordinate system relative to the first operating arm coordinate system is determined.

17. The control method according to claim 1, comprising: Determining the three-dimensional coordinates of the plurality of first manipulator arm posture identifiers in a first manipulator arm posture identifier coordinate system based on the distribution of the plurality of first manipulator arm posture identifiers; determining an imaging transformation relationship based on the two-dimensional coordinates of the plurality of first manipulator pose identifiers in the positioning image and the three-dimensional coordinates of the plurality of first manipulator pose identifiers in the first manipulator pose identifier coordinate system; Determining a plurality of angle identification candidate areas in the positioning image based on the imaging transformation relationship, the three-dimensional coordinates of the plurality of first manipulator pose identifiers in the first manipulator pose identifier coordinate system, and the position association relationship; as well as The candidate areas are identified from the multiple angles, and the first operating arm angle identification is recognized.

18. The control method according to claim 17, comprising: Determining a plurality of candidate three-dimensional coordinates of angle identifiers in the first manipulator pose identifier coordinate system based on the three-dimensional coordinates of the plurality of first manipulator pose identifiers in the first manipulator pose identifier coordinate system and the position association relationship; as well as Based on the imaging transformation relationship and the multiple angle identification candidate three-dimensional coordinates, the multiple angle identification candidate areas are determined in the positioning image.

19. The control method according to claim 17 or 18, comprising: Determine the pixel with the largest corner likelihood value in each of the angle marker candidate regions to form a pixel set; Determine an angle marker candidate region corresponding to a pixel having the largest corner point likelihood value in the pixel set as the angle marker candidate region to be identified; as well as A plurality of first operating arm angle pattern matching templates are used to respectively match the angle identifier candidate areas to be identified, so as to identify the first operating arm angle identifier.

20. The control method according to claim 17 or 18, comprising: Based on the angle identifier candidate area, the first posture identifier having a position association relationship with the first operating arm angle identifier is determined.

21. The control method according to claim 1, comprising: determining a plurality of candidate pose identifiers from the positioning image; identifying an initial pose identifier from the plurality of candidate pose identifiers based on the first manipulator pose pattern matching template; as well as Taking the initial pose identifier as a starting point, searching for a pose identifier.

22. The control method according to any one of claims 1-10, 13-15, 17-18 and 21, wherein the plurality of first operating arm position identifiers and the first operating arm angle identifier are arranged on an outer surface of a columnar portion of the first operating arm.

23. A computer device comprising: A memory for storing at least one instruction; as well as A processor, coupled to the memory, is configured to execute the at least one instruction to perform the control method according to any one of claims 1 to 22. 24 . A computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and wherein the at least one instruction is executed by a processor to enable a computer to execute the control method according to any one of claims 1 to 22.

25. A surgical robot system comprising: At least two surgical tools, a first surgical tool of the at least two surgical tools comprising a first operating arm, an actuator disposed at a first operating arm end of the first operating arm, and at least one first operating arm angle identifier and a plurality of first operating arm posture identifiers disposed at the first operating arm end, the at least one first operating arm angle identifier being positionally associated with a first posture identifier among the plurality of first operating arm posture identifiers; An image collector, used for collecting positioning images; as well as A control device, connected to the image collector, for executing the control method according to any one of claims 1 to 22.

Images