Method of controlling an operating arm and surgical robot system

Abstract

The present disclosure relates to the technical field of control, and discloses a control method of an operating arm, a computer device, a computer readable storage medium and a surgical robot system. The control method of the operating arm comprises: acquiring a positioning image; in the positioning image, identifying a plurality of pose marks located on the operating arm; based on the plurality of pose marks, identifying an angle mark located on the operating arm, the angle mark having a positional correlation relationship with a first pose mark in the plurality of pose marks; based on the angle mark and the plurality of pose marks, determining a current relative pose of the operating arm relative to a reference coordinate system; and based on the current relative pose and a target pose of the operating arm, determining a driving signal of the operating arm.

Description

Technical Field

[0001] This disclosure belongs to the field of control technology, and in particular relates to a control method for an operating arm and a surgical robot system. Background Technology

[0002] With the development of technology, it is becoming increasingly common for machines and equipment, controlled by humans or computers, to perform desired actions in order to assist or replace operators. For example, logistics robots are used to sort express packages, and surgical robots are used to assist doctors in performing surgeries.

[0003] In the above applications, it is necessary to control the manipulator arm in order to control the machine equipment. Summary of the Invention

[0004] In some embodiments, this disclosure provides a method for controlling an operating arm, including:

[0005] Acquire the location image;

[0006] In the positioning image, identify multiple pose markers located on the manipulator;

[0007] Based on multiple pose markers, an angle marker located on the manipulator is identified. The angle marker has a positional relationship with the first pose marker among the multiple pose markers.

[0008] Based on angle identifiers and multiple pose identifiers, the current relative pose of the manipulator with respect to the reference coordinate system is determined; and

[0009] Based on the current relative pose and the target pose of the manipulator, the drive signal of the manipulator is determined.

[0010] In some embodiments, this disclosure provides a computer device, the computer device comprising:

[0011] Memory, used to store at least one instruction; and

[0012] A processor, coupled to memory, is configured to execute the at least one instruction to perform the method of this disclosure.

[0013] In some embodiments, this disclosure provides a computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the method of this disclosure.

[0014] In some embodiments, this disclosure provides a surgical robot system, including:

[0015] Surgical instruments include an operating arm, an actuator disposed at the distal end of the operating arm, and at least one angle marker and multiple pose markers disposed at the end of the operating arm.

[0016] Image acquisition device, used to acquire positioning images of the manipulator; and

[0017] A processor, connected to an image acquisition unit, is used to execute the methods of this disclosure to determine the drive signals for the manipulator. Attached Figure Description

[0018] Figure 1 A schematic diagram of an operating arm control system according to some embodiments of the present disclosure is shown;

[0019] Figure 2 A schematic diagram of the structure of an operating arm according to some embodiments of the present disclosure is shown;

[0020] Figure 3 A schematic diagram of the structure of an operating arm according to some embodiments of the present disclosure is shown;

[0021] Figure 4 A schematic diagram of a label including multiple pose markers and multiple angle markers is shown;

[0022] Figure 5 A schematic diagram showing a label disposed on a cylindrical label formed on the periphery of the end of the operating arm;

[0023] Figure 6 The diagram illustrates implementation scenarios according to some embodiments of this disclosure;

[0024] Figure 7 A flowchart illustrating a control method for an operating arm control system according to some embodiments of the present disclosure;

[0025] Figure 8 A flowchart illustrating a method for determining a drive signal according to some embodiments of the present disclosure is shown;

[0026] Figure 9 A flowchart illustrating a method for determining the pose of an arm coordinate system relative to a reference coordinate system according to some embodiments of the present disclosure;

[0027] Figure 10 A flowchart illustrating a method for determining the pose of an arm coordinate system relative to a reference coordinate system according to other embodiments of the present disclosure;

[0028] Figure 11 A schematic diagram showing multiple pose markers on a cross-sectional circle according to some embodiments of the present disclosure;

[0029] Figure 12 A flowchart illustrating a method for identifying pose identifiers according to some embodiments of the present disclosure;

[0030] Figure 13 A schematic diagram showing pose identification patterns according to some embodiments of the present disclosure;

[0031] Figure 14 A flowchart illustrating a method for searching pose identifiers according to some embodiments of the present disclosure;

[0032] Figure 15 A schematic diagram illustrating a search pose identifier according to some embodiments of the present disclosure;

[0033] Figure 16 A flowchart illustrating a method for identifying angle markers according to some embodiments of the present disclosure;

[0034] Figure 17 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown;

[0035] Figure 18 A schematic diagram of a surgical robot system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0036] Exemplary embodiments of this disclosure are described below with reference to the accompanying drawings. Those skilled in the art will understand that the scope of this disclosure is not limited to these embodiments. Various modifications and variations can be made to the embodiments described below. All such modifications and variations are included within the scope of this disclosure. Similar reference numerals indicate similar components among the various embodiments shown in the accompanying drawings.

[0037] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom can be described using variations in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "attitude" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and attitude of an object or part of an object, which can be described, for example, using six parameters from the six degrees of freedom mentioned above.

[0038] In this disclosure, a reference coordinate system can be understood as a coordinate system capable of describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own perception coordinate system, etc.

[0039] In this disclosure, an object can be understood as an object or target that needs to be located, such as a manipulator or the end effector of a manipulator.

[0040] In this disclosure, the pose of the manipulator or a portion thereof refers to the pose of the manipulator coordinate system defined by the manipulator or a portion thereof relative to a reference coordinate system.

[0041] Figure 1 A schematic diagram of a manipulator control system 100 according to some embodiments of the present disclosure is shown. For example... Figure 1 As shown, the manipulator control system 100 may include an image acquisition device 110, at least one manipulator 140, and a control device 120. The image acquisition device 110 and the at least one manipulator 140 are communicatively connected to the control device 120. In some embodiments, such as Figure 1 As shown, the control device 120 can be used to control the movement of at least one manipulator 140 to adjust the pose of the at least one manipulator 140, coordinate with each other, etc. In some embodiments, at least one manipulator 140 may include a manipulator end effector 130 at its distal end or end. The control device 120 can control the movement of at least one manipulator 140 to move the manipulator end effector 130 to a desired position and orientation. Those skilled in the art will understand that the manipulator control system 100 can be applied to surgical robot systems, such as laparoscopic surgical robot systems. For example, a surgical actuator 160 may be disposed at the distal end of the manipulator end effector 130, such as... Figure 1 As shown. It should be understood that the manipulator control system 100 can also be applied to dedicated or general-purpose robot systems in other fields (e.g., manufacturing, machinery, etc.).

[0042] In this disclosure, the control device 120 can be communicatively connected to the drive unit 150 (e.g., a motor) of at least one manipulator 140 and send drive signals to the drive unit 150, thereby enabling the drive unit 150 to control the at least one manipulator 140 to move to a corresponding target pose based on the drive signals. For example, the drive unit 150 controlling the movement of the manipulator 140 can be a servo motor, which can receive instructions from the control device to control the movement of the manipulator 140. The control device 120 can also be communicatively connected to a sensor coupled to the drive unit 150, for example, through a communication interface, to receive motion data of the manipulator 140 and monitor the motion status of the manipulator 140. In one example of this disclosure, the communication interface can be a CAN (Controller Area Network) bus communication interface, which enables the control device 120 to communicate with the drive unit 150 and the sensor via the CAN bus.

[0043] In some embodiments, the manipulator 140 may include a continuous deformable arm, such as a multi-degree-of-freedom manipulator composed of multiple joints, such as a manipulator capable of six degrees of freedom of movement. The image acquisition device 110 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera.

[0044] In some embodiments, the image acquisition device 110 can be used to acquire positioning images. The positioning images may include part or all of the image of the manipulator 140. In some embodiments, the image acquisition device 110 can be used to acquire images of the manipulator end cap 130, which is provided with positioning markers. The positioning markers may include pose markers and angle markers (described in detail below). Figure 1 As shown, if the end of the manipulator 130 is within the field of view of the image acquisition device 110, the acquired positioning image may include an image of the end of the manipulator 130.

[0045] In some embodiments, the control device 120 may receive a positioning image from the image acquisition device 110 and process the positioning image. For example, the control device 120 may identify multiple pose markers and at least one angle marker located on the manipulator 140 in the positioning image, and determine the current relative pose of the manipulator 140 relative to a reference coordinate system (e.g., the world coordinate system). The control device 120 may also determine a drive signal for the manipulator 140 based on the current relative pose and the target pose of the manipulator 140. The drive signal may be sent to the drive unit 150 to perform motion control on the manipulator 140.

[0046] Figure 2 A schematic diagram of a segment 200 of a manipulator according to some embodiments of the present disclosure is shown. The manipulator (e.g., manipulator 140) may include at least one deformable segment 200. Figure 2 As shown, the deformable segment 200 includes a fixed disk 210 and multiple structural bones 220. A first end of each structural bone 220 is fixedly connected to the fixed disk 210, and a second end is connected to a drive unit (not shown). In some embodiments, the fixed disk 210 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and its cross-section may be circular, rectangular, polygonal, or various other shapes.

[0047] The driving unit causes the segment 200 to deform by driving the structural bone 220. For example, the driving unit causes the segment 200 to be in a position such as... Figure 2The bending state is shown. In some embodiments, the second end of the multiple structural bones 220 passes through the base plate 230 and is connected to the drive unit. In some embodiments, similar to the fixed plate 210, the base plate 230 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and the cross-section may be a circle, a rectangle, a polygon, etc. The drive unit may include a linear motion mechanism, a drive segment, or a combination of both. The linear motion mechanism may be connected to the structural bones 220 to push or pull the structural bones 220, thereby driving the segment 200 to bend. The drive segment may include a fixed plate and multiple structural bones, wherein one end of the multiple structural bones is fixedly connected to the fixed plate. The other end of the multiple structural bones of the drive segment is connected to or integrally formed with the multiple structural bones 220 to drive the bending of the segment 200 by bending the drive segment.

[0048] In some embodiments, a spacer disk 240 is further included between the fixed disk 210 and the base disk 230, through which multiple structural bones 220 pass. Similarly, the drive segment may also include a spacer disk.

[0049] Figure 3 A schematic diagram of the structure of an operating arm 300 according to some embodiments of the present disclosure is shown. For example... Figure 3 As shown, the operating arm 300 is a deformable operating arm, which may include an operating arm end cap 310 and an operating arm body 320. The operating arm body 320 may include one or more segments, such as a first segment 3201 and a second segment 3202. In some embodiments, the structures of the first segment 3201 and the second segment 3202 may be similar to those of... Figure 2 The shown component 200 is similar. In some implementations, such as... Figure 3 As shown, the main body 320 of the operating arm also includes a first straight rod segment 3203 located between the first segment 3201 and the second segment 3202. The first end of the first straight rod segment 3203 is connected to the base plate of the second segment 3202, and the second end is connected to the fixed plate of the first segment 3201. In some embodiments, such as... Figure 3 As shown, the main body 320 of the operating arm also includes a second straight rod segment 3204, the first end of which is connected to the base plate of the first component 3201.

[0050] In some embodiments, a plurality of pose markers and at least one angle marker are distributed on the manipulator (e.g., on the manipulator body 320 or the manipulator end 310). For example, the plurality of pose markers are distributed circumferentially on the manipulator end 310, and the plurality of angle markers are distributed circumferentially on the manipulator end 310. The plurality of pose markers and the plurality of angle markers are arranged side by side axially on the manipulator end 310. For example, the plurality of pose markers and the plurality of angle markers are disposed on the outer surface of the columnar portion of the manipulator end 310.

[0051] In some embodiments, each angle marker has a positional association with one of the pose markers. Based on this positional association, the possible distribution area of ​​the angle markers can be determined by the position of the pose markers. Alternatively, the possible distribution area of ​​the pose markers can be determined by the position of the angle markers. The positional association can be determined according to the specific arrangement of the pose markers and angle markers, and can be pre-designed.

[0052] In some embodiments, the positional association may include an axial correspondence between angle markers and pose markers. For example, the positional association may include an axial offset. Based on the axial correspondence, given that the positions of one or more pose markers on the end effector of the manipulator are known, an axial offset by a certain distance can determine the area where the angle markers may exist. For example, the positional association may also include axial oblique alignment, etc.

[0053] In some embodiments, multiple pose markers and multiple angle markers may be set on a label attached to the periphery of the end of the manipulator.

[0054] In some embodiments, a pose identifier may include a pose identifier pattern and pose identifier pattern corner points, and an angle identifier may include an angle identifier pattern and angle identifier pattern corner points. In some embodiments, the pose identifier pattern and angle identifier pattern may be disposed on a label attached to the end of the operating arm, or may be printed on the end of the operating arm, or may be patterns formed by the physical structure of the end of the operating arm itself, for example, including recesses or protrusions and combinations thereof. In some embodiments, the pose identifier pattern or angle identifier pattern may include patterns formed with brightness, grayscale, color, etc. In some embodiments, the pose identifier pattern and angle identifier pattern may include patterns that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provide information to be detected by the image acquisition module. Those skilled in the art will understand that in some embodiments, the pose of the pose identifier may be represented by the pose of the pose identifier pattern corner point coordinate system, and the pose of the angle identifier may be represented by the pose of the angle identifier pattern corner point coordinate system.

[0055] In some embodiments, the pose marking pattern or angle marking pattern is set on the end of the manipulator in an area suitable for image acquisition by an image acquisition device, such as an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily disturbed or obstructed during operation.

[0056] Figure 4 A schematic diagram of a label 400 including multiple pose identifiers and multiple angle identifiers according to some embodiments is shown. Figure 5A schematic diagram is shown of a label 500 disposed on the periphery of the end of the operating arm and forming a cylindrical shape. It can be understood that, for simplicity, label 400 may include the same pose marking pattern and angle marking pattern as label 500.

[0057] See Figure 4 Multiple pose markers (represented by the symbol "○" for corner points in this disclosure) and multiple angle markers (represented by the symbol "△" for corner points in this disclosure) are arranged side by side. The multiple pose marker patterns 411 may be identical or similar, and the corner points of the multiple pose marker patterns are located within the multiple pose marker patterns 411. The multiple angle marker patterns 421-426 may be different, and the corner points of the multiple angle marker patterns are located within the multiple angle marker patterns 421-426.

[0058] Each angle marker and one of the pose markers can have a positional association. For example, such as Figure 4 As shown, in the direction indicated by the arrow, some pose markers (e.g., pose marker pattern 411) and corresponding angle markers (e.g., angle marker pattern 421) are arranged along the arrow direction and have a spacing d1. See also Figure 5 In the circumferential setting state, label 400 becomes label 500 with a spatial structure of a cylinder. The positional association between each angle identifier and one of the pose identifiers can include the angle identifier and the pose identifier in the axial direction (e.g., Figure 5 The correspondence between the angle markers and the pose markers along the positive Z-axis is established. Based on this axial correspondence, given the known positions of one or more pose markers at the end of the manipulator, the area where the angle markers might exist can be determined by offsetting by a certain distance (e.g., distance d1) along the axial direction. In some embodiments, the axial correspondence between the angle markers and the pose markers can be represented by the axial correspondence between the corner points of the angle marker pattern and the corner points of the pose marker pattern. In some embodiments, based on the axial correspondence between the angle markers and the pose markers, the projections of one of the corner points of the angle marker pattern and the corner points of the pose marker pattern along the Z-axis coincide.

[0059] In some embodiments, the about-axis angle or roll angle of the angle identifier or pose identifier can be represented by the about-axis angle of the corner point of the angle identifier pattern or the corner point of the pose identifier pattern. The corner point of the angle identifier pattern is relative to the manipulator coordinate system (e.g., a coordinate system established at the end of the manipulator, such as...). Figure 5 The angles of the XY coordinate system shown are known or predetermined, for example... Figure 5In the XY coordinate system, the angle between corner point R5 of the angle marker pattern and the X-axis is θ. Based on the positional relationship, the angle between corner point P5 of the pose marker pattern associated with its position and the X-axis can be obtained as angle θ. It should be understood that the angle θ corresponding to corner point R5 of the angle marker pattern and corner point P5 of the pose marker pattern can be called the axial angle or roll angle of the angle marker or pose marker about the Z-axis. In this disclosure, the axial angle or roll angle refers to the angle about the Z-axis. It is understood that, for clarity, Figure 5 The corner point R5 of the angle marker pattern and the corner point P5 of the pose marker pattern are shown as separate, but they are overlapping.

[0060] Figure 6 A schematic diagram illustrating an implementation scenario 600 according to some embodiments of the present disclosure is shown. Figure 6 As shown, the manipulator 640 includes an end effector 630 and a distal actuator 660. Multiple pose markers and angle markers can be circumferentially positioned on the end effector 630. For example, as... Figure 4 The label 400 shown is circumferentially disposed on the end of the operating arm 630, forming a cylindrical angle marking pattern strip 610 and a pose marking pattern strip 620. Multiple pose marking pattern corner points are distributed on the cross-sectional circle 621 of the pose marking pattern strip 620 at the end of the operating arm 630, and multiple angle marking pattern corner points are distributed on the cross-sectional circle 611 of the angle marking pattern strip 610 at the end of the operating arm 630.

[0061] In some embodiments, the multiple angle marker patterns are different patterns. Each angle marker pattern is used to indicate or identify a different angle about the axis. In some embodiments, there is a one-to-one correspondence between each angle marker pattern and the identified angle about the axis, and the identified angle about the axis can be determined based on the angle marker pattern.

[0062] For example, such as Figure 6 As shown, multiple different angle marking patterns (such as...) Figure 4 Multiple angle marker patterns (421-426) shown are evenly distributed circumferentially along the cylindrical structure, forming angle marker pattern corner points AF. The angle marker pattern corresponding to corner point A is designated as a reference pattern (e.g., the angle marker pattern corresponding to corner point A is used to mark a 0° angle around the axis). A plane coordinate system {wm1} is established. Then, based on the positional relationship between the remaining angle marker patterns and the angle marker pattern corresponding to corner point A, the angle around the axis marked by the corner points of the remaining angle marker patterns can be determined. For example, see... Figure 6When the angle marker pattern corresponding to corner point B is identified, based on the positional relationship between the angle marker pattern corresponding to corner point B and the angle marker pattern corresponding to corner point A, the angle around the axis of corner point B in the two-dimensional plane coordinate system of section circle 611 can be determined to be 60°. The origin of the two-dimensional plane coordinate system of section circle 611 is the center of section circle 611, the X-axis direction is from the origin to corner point A of the angle marker pattern, and the Y-axis is perpendicular to the X-axis.

[0063] In some embodiments, the pose of actuator 660 can be determined by translating the manipulator coordinate system {wm} (e.g., the manipulator end-effector coordinate system) by a predetermined distance. Alternatively, the pose of actuator 660 can be approximately equal to the pose of the manipulator end-effector coordinate system {wm}.

[0064] In some embodiments, the pose of the actuator 660 relative to the reference coordinate system (e.g., the world coordinate system {w}) is determined based on the pose of the manipulator coordinate system relative to the reference coordinate system. The specific calculation formula is as follows:

[0065] w R tip = w R wm wm R tip (1)

[0066] w P tip = w R wm wm P tip + w P wm

[0067] in, w R tip The attitude of the actuator relative to the world coordinate system. w P tip The position of the actuator relative to the world coordinate system. wm R tip The attitude of the actuator relative to the world coordinate system. wm P tip The position of the actuator relative to the world coordinate system. w R wm This refers to the orientation of the manipulator's coordinate system relative to the world coordinate system. w P wm This represents the position of the manipulator coordinate system relative to the world coordinate system.

[0068] Some embodiments of this disclosure provide a method for controlling an operating arm. Figure 7A flowchart illustrating a control method 700 of a manipulator control system (e.g., manipulator control system 100) according to some embodiments of the present disclosure is shown. Figure 7 As shown, some or all of the steps in method 700 can be executed by a control device (e.g., control device 120) of the manipulator control system 100. Control device 120 can be configured on a computing device. Method 700 can be implemented by software, firmware, and / or hardware. In some embodiments, method 700 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., [processor name missing]). Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0069] See Figure 7 In step 701, a positioning image is acquired. In some embodiments, the positioning image includes multiple pose markers on the manipulator and at least one angle marker. In some embodiments, the image can be obtained from, for example... Figure 1 The image acquisition device 110 shown receives a positioning image. For example, the control device 120 can receive a positioning image actively sent by the image acquisition device 110. Alternatively, the control device 120 can send an image request command to the image acquisition device 110, and the image acquisition device 110 responds to the image request command by sending a positioning image to the control device 120.

[0070] Continue reading Figure 7 In step 703, multiple pose markers located on the manipulator are identified in the positioning image. For example, an exemplary method for identifying multiple pose markers located on the manipulator may include, for instance... Figure 12 and Figure 14 The method is illustrated. In some embodiments, the control device 120 can identify some or all of the pose markers in the positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm, which can extract or identify features of the pose markers. For example, the image processing algorithm may include a corner detection algorithm for detecting corner points of the pose marker pattern. The corner detection algorithm may be one of, but is not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the pose marker pattern. As another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the pose marker pattern.

[0071] In some embodiments, the control device can identify some or all of the pose markers in the positioning image by recognizing the model.

[0072] Continue reading Figure 7In step 705, based on multiple pose identifiers, an angle identifier located on the manipulator is identified, and the angle identifier has a positional association with the first pose identifier among the multiple pose identifiers. In some embodiments, after identifying the multiple pose identifiers, the angle identifier located on the manipulator is identified according to the positional association. In some embodiments, the positional association between the angle identifier and the first pose identifier can be as follows: Figure 4 or Figure 5 The positional relationships are shown in the diagram. In some embodiments, the first pose identifier (e.g., a first pose identifier pattern or a corner point of a first pose identifier pattern) refers to the pose identifier among a plurality of pose identifiers that has a positional relationship with the angle identifier. Exemplary methods for identifying the angle identifier include, as shown in the diagram... Figure 16 The method shown.

[0073] Continue reading Figure 7 In step 707, based on the angle identifier and multiple pose identifiers, the current relative pose of the manipulator relative to the reference coordinate system is determined. An exemplary method for determining the relative pose of the manipulator relative to the reference coordinate system includes, for example: Figure 9 or Figure 10 The method shown. In some embodiments, the pose of the manipulator relative to a reference coordinate system can be determined based on an angle identifier, a first pose identifier, and multiple pose identifiers.

[0074] In some embodiments, method 700 further includes: determining the transformation relationship between the operator arm coordinate system and the pose identifier coordinate system based on angle identifiers and multiple pose identifiers. In some embodiments, according to the transformation relationship between the operator arm coordinate system and the pose identifier coordinate system, the three-dimensional coordinates in the pose identifier coordinate system can be converted into the corresponding three-dimensional coordinates in the operator arm coordinate system. In some embodiments, the pose of the operator arm coordinate system relative to the reference coordinate system is obtained according to the transformation relationship between the operator arm coordinate system and the pose identifier coordinate system and the pose of the pose identifier coordinate system relative to the reference coordinate system.

[0075] In some embodiments, the transformation relationship between the manipulator coordinate system and the pose identifier coordinate system may include the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system. In some embodiments, the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system may be determined based on the angle identifier and the first pose identifier. It should be understood that the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system may be the angle of rotation of the pose identifier coordinate system about the Z-axis of the manipulator coordinate system.

[0076] In some embodiments, the manipulator coordinate system may be a fixed coordinate system set on the object based on multiple pose markers or multiple angle markers. In some embodiments, the Z-axis of the manipulator coordinate system is parallel to the axial direction of the manipulator, and the XY plane of the manipulator coordinate system is in the same plane as the corner points of the multiple pose marker patterns, or in the same plane as the corner points of the multiple angle marker patterns.

[0077] In some embodiments, a pose identifier coordinate system can be determined to facilitate the determination of the positions of multiple pose identifiers. In some embodiments, the position of the pose identifier can be represented by the position of the corner points of the pose identifier pattern. In some embodiments, the Z-axis of the pose identifier coordinate system is parallel to or coincides with the axis of the manipulator, and the XY plane of the pose identifier coordinate system is in the same plane as the corner points of the multiple pose identifier patterns.

[0078] For example, see Figure 6 The coordinate system of the manipulator is {wm}≡[X]. wm Y wm Z wm ] T The origin is the center of the cross-sectional circle 621 containing the corner points of multiple pose marker patterns. The X-axis points from the origin to one of the pose marker pattern corner points. The Z-axis is parallel to the axis of the end effector 630 of the manipulator. The Y-axis is perpendicular to the XZ plane. The X-axis of the manipulator coordinate system {wm} is ≡ [X] of the two-dimensional plane coordinate system {wm1} of the cross-sectional circle 611. wm1 Y wm1 ] T The X-axis is parallel to the Y-axis of the manipulator coordinate system, and the Y-axis of the manipulator coordinate system is parallel to the Y-axis of the two-dimensional plane coordinate system {wm1} of the cross-section circle 611. The angle marking point of the angle marking pattern in the two-dimensional plane coordinate system {wm1} of the cross-section circle 611 can be equal to its angle marking in the manipulator coordinate system {wm}. The pose marking coordinate system {wm0} ≡ [X wm0 Y wm0 Z wm0 ] T The origin is the center of the cross-sectional circle 621 containing the corner points of multiple pose marker patterns. The X-axis points from the origin to one of the corner points of the pose marker pattern. The Z-axis is parallel to the axis of the end of the manipulator arm 630. The Y-axis is perpendicular to the XZ plane. (Continue reading...) Figure 6 The Z-axis of the manipulator coordinate system {wm} coincides with the Z-axis of the pose identifier coordinate system {wm0}. The transformation relationship between the manipulator coordinate system {wm} and the pose identifier coordinate system {wm0} can be determined by the roll angle α0 of the pose identifier coordinate system {wm0} relative to the manipulator coordinate system {wm}. The roll angle α0 can be the rotation angle of the pose identifier coordinate system {wm0} relative to the manipulator coordinate system {wm} around the Z-axis.

[0079] In some embodiments, see Figure 6 The roll angle α0 is calculated using the following formula:

[0080] α0=α1-α2 (2)

[0081] Where α1 is the first angle around the axis and α2 is the second angle around the axis. The first angle around the axis is the angle around the axis marked by the corner point of the angle marker pattern (e.g., corner point R6) in the manipulator coordinate system. The second angle around the axis is the angle around the axis marked by the corner point of the first pose marker pattern (e.g., corner point P6) in the pose marker coordinate system.

[0082] Continue reading Figure 7 In step 709, the drive signal of the manipulator is determined based on the current relative pose and the target pose of the manipulator. In some embodiments, method 700 may further include determining the drive signal of the manipulator at predetermined intervals to achieve real-time control through multiple motion control cycles.

[0083] In some embodiments, method 700 may further include: determining a pose difference based on the current relative pose of the manipulator and the target pose of the manipulator; and determining the drive signal of the manipulator based on the pose difference and the inverse kinematics model of the manipulator. For example, based on the difference between the target pose and the current pose of the manipulator end in the world coordinate system, the drive values ​​of multiple joints included in the manipulator in the current motion control cycle (or the drive values ​​of multiple corresponding motors controlling the movement of the manipulator) can be determined by using an inverse kinematics numerical iterative algorithm of the manipulator kinematic model. It should be understood that the kinematic model can be a mathematical model representing the motion relationship between the joint space and the task space of the manipulator. For example, the kinematic model can be established by methods such as the Denavit-Hartenberg (DH) parameter method and the exponential product representation method.

[0084] In some embodiments, the target pose of the manipulator is the target pose of the manipulator in the world coordinate system. Method 700 may further include: determining the current pose of the manipulator in the world coordinate system based on the current relative pose; and determining a pose difference based on the target pose of the manipulator and the current pose of the manipulator in the world coordinate system. In some embodiments, the pose difference includes a position difference and an orientation difference.

[0085] In the k-th motion control loop, the pose difference can be expressed as follows:

[0086]

[0087] in, Let $\frac{k}{k}$ be the position difference of the manipulator during the k-th motion control cycle. P represents the angle difference of the manipulator during the k-th motion control cycle. t k R represents the target position of the manipulator during the k-th motion control cycle. t k The target posture of the manipulator during the k-th motion control cycle. R represents the current position of the manipulator during the k-th motion control cycle. t k For the k-th motion control cycle, the manipulator's... Current posture, express With R t k The corner between them.

[0088] In some embodiments, the target pose of the manipulator is updated at predetermined intervals before or during each motion control cycle. In some embodiments, multiple motion control cycles are executed iteratively. In each motion control cycle, methods according to some embodiments of this disclosure, such as steps 701-709, can be performed to control the manipulator to move to the target pose. By iteratively executing multiple motion control cycles, real-time closed-loop control of the manipulator end-effector pose can be achieved, thereby improving the pose control accuracy of the manipulator. It should be understood that implementing manipulator pose control through the methods of this disclosure can improve the trajectory tracking error of the manipulator (e.g., a continuous deformable arm).

[0089] In some embodiments, method 700 may further include receiving a control command and determining the target pose of the manipulator based on the control command. In some embodiments, the target pose of the manipulator end effector in the world coordinate system may be input by a user via an input device. By comparison and calculation, the difference between the target pose and the current pose of the manipulator end effector can be determined. In some embodiments, the control command may be received based on a master-slave motion control method. For example, the target pose of the manipulator may be determined by acquiring the pose or joint information of the master manipulator in each motion control cycle. Real-time master-slave motion control can be performed through multiple motion control cycles.

[0090] Figure 8 A flowchart illustrating a method 800 for determining a drive signal according to some embodiments of the present disclosure is shown. Figure 8 As shown, some or all of the steps in method 800 can be executed by a control device (e.g., control device 120) of the manipulator control system 100. Control device 120 can be configured on a computing device. Method 800 can be implemented by software, firmware, and / or hardware. In some embodiments, method 800 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., [processor name missing]). Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0091] See Figure 8In step 801, the Cartesian space velocity is determined based on the pose difference. In some embodiments, the Cartesian space velocity includes Cartesian space linear velocity and Cartesian space angular velocity. Method 800 may further include: determining the Cartesian space linear velocity based on the position difference, and determining the Cartesian space angular velocity based on the pose difference. In some embodiments, the Cartesian space angular velocity may be determined based on the pose difference using a proportional-integral-derivative controller or a proportional-derivative controller. In some embodiments, the Cartesian space velocity of the k-th motion control cycle... as follows:

[0092]

[0093] Among them, v k Let ω be the Cartesian linear velocity of the k-th motion control cycle. k Let P be the Cartesian space angular velocity of the k-th motion control cycle. v D is the linear velocity proportionality coefficient. v P is the differential coefficient of linear velocity. ω D is the angular velocity proportionality coefficient. ω These are the differential coefficients of angular velocity. The position difference of the manipulator during the (k-1)th motion control cycle. The angle difference of the manipulator during the (k-1)th motion control cycle.

[0094] See Figure 8 In step 803, the parameter space velocity is determined based on the Cartesian space velocity. The joint parameter space velocity for the k-th motion control cycle is... as follows:

[0095]

[0096] J + It is the Moore-Penrose pseudo-inverse of the velocity Jacobian matrix J of the kinematic model of the manipulator. The velocity Jacobian matrix J of the kinematic model of the manipulator can be determined based on the structure of the manipulator.

[0097] See Figure 8 In step 805, the target joint parameters are determined based on the parameter space velocity and the current joint parameters. The target joint parameters for the k-th motion control cycle are... as follows:

[0098]

[0099] in, Let be the current joint parameters for the k-th motion control cycle, and Δt be the period of the motion control cycle.

[0100] It should be understood that when the manipulator has multiple components (e.g., Figure 3 The target joint parameters of the manipulator 300 shown can be the target joint parameters of all the components, or the target joint parameters of one or some of the components.

[0101] See Figure 8 In step 807, a drive signal is determined based on the target joint parameters. For example, based on the mapping relationship between the target joint parameters and the drive amount, the drive amount of multiple joints included in the manipulator in the current motion control cycle can be determined, and then the drive signal of the drive unit (e.g., motor) can be determined based on the drive amount. In some embodiments, the mapping relationship between the joint parameters of a single component and the drive amount can be the mapping relationship shown in formula (15).

[0102] In some embodiments, the manipulator is taken as a deformable kinematic arm (e.g., a continuous deformable arm) as an example. A continuous deformable arm can be as follows: Figure 3 The manipulator arm 300 is shown. (As shown in the image) Figure 3 As shown, each segment (first segment 3201 and second segment 3202) may include a base plate, a fixed plate, and multiple structural bones penetrating the base plate and the fixed plate. The multiple structural bones can be fixedly connected to the fixed plate and slidably connected to the base plate. The continuous deformable arm and its included segments can be described by a kinematic model. In some embodiments, the structure of each segment may be specifically as follows: Figure 2 The shown component is 200. (As shown in the image) Figure 2 As shown, the base disk coordinate system The base disk is attached to the t-th (t=1,2,3…) section of the continuum, with its origin located at the center of the base disk, and the XY plane coinciding with the plane of the base disk. Pointing from the center of the base plate to the first structural bone (the first structural bone can be understood as any one of multiple structural bones chosen as a reference). Curved plane coordinate system. Its origin coincides with the origin of the base disk coordinate system, and the XY plane coincides with the bending plane. and Coincident. Fixed disk coordinate system. The origin of the fixed disk is located at the center of the fixed disk, and the XY plane coincides with the plane of the fixed disk. Pointing from the center of the fixed plate to the first structural bone. Curved plane coordinate system. Its origin is located at the center of the fixed disk, and the XY plane coincides with the bending plane. and coincide.

[0103] like Figure 2The single segment 200 shown can be represented by a kinematic model. The position of the end of segment t (fixed disk coordinate system {te}) relative to the base disk coordinate system {tb}. tb P te ,attitude tb R te As shown in formulas (7) and (8) below:

[0104]

[0105] tb R te = tb R t1 t1 R t2 t2 R te (8)

[0106] Among them, L t For the t-th segment, construct a virtual structural bone (e.g., Figure 2 The length of the virtual structural bone 221 shown in the figure, θ t In the t-th section, about or Rotate to Required rotation angle tb R t1 Let {t1} be the orientation of the bending plane coordinate system of segment t relative to the base disk coordinate system {tb}. t1 R t2 Let t be the orientation of the bending plane coordinate system 2{t2} of the t-th segment relative to the bending plane coordinate system 1{t1}. t2 R te Let {te} be the orientation of the fixed disk coordinate system {t2} of the t-th segment relative to the curved plane coordinate system 2{t2}.

[0107] tb R t1 , t1 R t2 and t2 R te The results can be shown in formulas (9), (10), and (11) below:

[0108]

[0109]

[0110]

[0111] Where, δ t For the t-th segment, the bending plane and The included angle.

[0112] like Figure 2 The joint parameter Ψ of the single segment 200 shown t It can be shown in the following formula (12):

[0113] ψ t =[θ t ,δ t ] T (12)

[0114] Joint parameters Ψ t Parameter space velocity It can be shown in the following formula (13):

[0115]

[0116] in, For θ t The first derivative, For δ t The first derivative.

[0117] like Figure 2 The Cartesian space velocity at the end of the single segment 200 shown It can be shown in the following formula (14):

[0118]

[0119] Among them, J t J is the velocity Jacobian matrix of the kinematic model of a single component. tv J is the linear velocity Jacobian matrix of the kinematic model of a single component. tω It is the angular velocity Jacobian matrix of the kinematic model of a single component, v t It is the end linear velocity of a single segment, ω t It is the end angular velocity of a single segment.

[0120] In some embodiments, the driving amount of multiple bone structures has a known mapping relationship with joint parameters. Based on the target joint parameters of the segment and the mapping relationship, the driving amount of the multiple bone structures can be determined. The driving amount of the multiple bone structures can be understood as moving a single segment from its initial state (e.g., θ) t =0) The length of the structural bone subjected to push or tension when bent to the target bending angle. In some embodiments, the mapping relationship between the driving amount of multiple structural bones and joint parameters can be as shown in the following formula (15):

[0121] q i ≡-r ti θ t cos(δ t+β ti (15) Where, r ti Let β be the distance from the i-th structural bone in the t-th segment to the virtual structural bone. ti Let q be the angle between the i-th structural bone and the first structural bone in the t-th segment. i Let be the driving quantity of the i-th structural bone. The driving signal of the driving unit can be determined based on the driving quantity of the i-th structural bone.

[0122] In some embodiments, the end Cartesian space velocity of a single segment can be determined based on formulas (3) and (4), the parametric space velocity of a single segment can be determined based on formula (5), the target joint parameters of a single segment can be determined based on formula (6), the driving amount of each structural bone can be determined based on formula (15), and then the driving signal of the driving unit (e.g., motor) can be determined based on the driving amount.

[0123] In some embodiments, the entire deformable arm can be described by a kinematic model. For example... Figure 3 As shown, transformations can be performed between multiple coordinate systems located at various positions on the deformable arm. For example, the end effector of the continuum deformable arm can be represented in the world coordinate system {w} as follows:

[0124] W T tip = W T 1b 1b T 1e 1e T 2b 2b T 2e 2e T tip (16)

[0125] in, W T tip The homogeneous transformation matrix of the end effector of the deformable arm of the continuum relative to the world coordinate system; W T 1b Represents the homogeneous transformation matrix of the base disk of the first continuum segment relative to the world coordinate system; 1b T 1e Represents the homogeneous transformation matrix of the fixed disk of the first continuous segment relative to the base disk of the first continuous segment; 1e T 2b Represents the homogeneous transformation matrix of the base disk of the second continuous segment relative to the fixed disk of the first continuous segment; 2b T 2e Represents the homogeneous transformation matrix of the fixed disk of the second continuum segment relative to the base disk of the second continuum segment; 2e T tipThis represents the homogeneous transformation matrix of the end effector of the continuum deformable arm relative to the fixed disk of the second continuum segment. In some embodiments, the end effector is fixedly mounted on the fixed disk, therefore... 2e T tip It is known or predetermined.

[0126] It should be understood that deformable arms have different joint parameters in different operating states. For example, Figure 3 The manipulator 300 shown includes at least four operating states. The four operating states of the manipulator 300 are described below:

[0127] First working state: Only the second component 3202 participates in the actuator's pose control (e.g., only the second component 3202 enters the workspace), and the joint parameters of the manipulator 300 at this time are as shown in the following formula (17):

[0128]

[0129] Where, ψ c1 These are the joint parameters of the manipulator 300 in its first working state. Let L2, θ2, δ2 be the rotation angle of the manipulator 300 around its axis, and let L2, θ2, δ2 be the angle of rotation of the manipulator 300 around its axis. Figure 2 In the structure shown in section 200, L t θ t and δ t They have the same physical meaning.

[0130] Second working state: The second component 3202 and the first linear segment 3203 participate in the position control of the actuator (for example, the second component 3202 is fully in the workspace, and the first linear segment 3203 is partially in the workspace). At this time, the joint parameters of the manipulator 300 are as shown in the following formula (18):

[0131]

[0132] Where, ψ c2 For the joint parameters of the manipulator 300 in the second working state, L r This is the feed rate for the first straight segment 3203.

[0133] Third working state: The second segment 3202, the first linear segment 3203, and the first segment 3201 participate in the position control of the actuator (for example, the second segment 3202 is fully in the workspace, the first linear segment 3203 is fully in the workspace, and the first segment 3201 is partially in the workspace). At this time, the joint parameters of the manipulator 300 are as shown in the following formula (19):

[0134]

[0135] Where, ψc3 For the joint parameters of the manipulator 300 in the third working state, θ1 and δ1 are as follows: Figure 2 In the shown section 200, θ t and δ t They have the same physical meaning.

[0136] Fourth working state: Second segment 3202, first linear segment 3203, first segment 3201 and second linear segment 3204 participate in the position control of the actuator (for example, the second segment 3202 is fully in the workspace, the first linear segment 3203 is fully in the workspace, the first segment 3201 is fully in the workspace, and the second linear segment 3204 is partially in the workspace). At this time, the joint parameters of the manipulator 300 are as shown in the following formula (20):

[0137]

[0138] Where, ψ c4 For the joint parameters of the manipulator 300 in the fourth working state, L s This is the feed rate for the second straight segment 3204.

[0139] In some embodiments, similar to a single segment, the end Cartesian space velocity of the deformable arm can be determined based on formulas (3) and (4), the parametric space velocity of the deformable arm can be determined based on formula (5), and the target joint parameters of the deformable arm can be determined based on formula (6). The specific parameters included in the parametric space velocity in formula (5) and the target joint parameters in formula (6) can be determined based on formulas (17), (18), (19) or (20). The driving amount of each structural bone of each segment can be determined based on formula (15), and then the driving signal of the driving unit (e.g., motor) can be determined based on the driving amount.

[0140] Figure 9 A flowchart illustrating a method 900 for determining the pose of an arm coordinate system relative to a reference coordinate system, according to some embodiments of the present disclosure, is shown. Figure 3 The manipulator 300 shown may have a coordinate system that includes the coordinate system at the end of the manipulator. For example... Figure 9 As shown, some or all of the steps in method 900 can be controlled by a control device (e.g., Figure 1 The control device 120 shown is used to execute the steps. Some or all of the steps in method 900 can be implemented by software, firmware, and / or hardware. In some embodiments, method 900 can be executed by a robot system (e.g., Figure 18 The surgical robot system 1800 shown is executed. In some embodiments, method 900 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0141] See Figure 9 In step 901, based on the angle identifier and multiple pose identifiers, the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system is determined. In some embodiments, a first axis angle identified by the angle identifier in the manipulator coordinate system is determined. A second axis angle identified by the first pose identifier in the pose identifier coordinate system is determined. Based on the first axis angle and the second axis angle, the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system is determined. In some embodiments, the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system can be determined based on formula (2).

[0142] In step 903, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on multiple pose identifiers. The coordinates of the pose identifiers in the corresponding coordinate systems can be represented by the coordinates of the corner points of the pose identifier patterns in the corresponding coordinate systems. For example, the two-dimensional coordinates of the pose identifiers in the positioning image and the three-dimensional coordinates of the pose identifiers in the pose identifier coordinate system can be represented by the coordinates of the corner points of the pose identifier patterns. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image and the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image, the three-dimensional coordinates of the corner points of the multiple pose identifier patterns in the pose identifier coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system.

[0143] In some embodiments, the three-dimensional coordinates of the corner points of multiple pose marker patterns in the pose marker coordinate system are determined based on the distribution of multiple pose markers. For example, see... Figure 11 Each pose marker corner point is located on the circumference of cross-section circle 1122, the center and radius r of which are known. The center of cross-section circle 1122 is set as the origin of the pose marker coordinate system. The XY plane lies on cross-section circle 1122, and the X-axis can be specified as pointing from the origin to any known pose marker corner point (e.g., pose marker corner point P). 11 This allows us to determine the three-dimensional coordinates of each pose marker's corner point in the pose marker coordinate system based on the distribution of multiple pose markers. For example, such as... Figure 11 As shown, the corner point P of the pose marker pattern 11 If the three-dimensional coordinates of the pose marker are (r, 0, 0), then the three-dimensional coordinates of the corner points of the other pose marker patterns in the pose marker coordinate system can be calculated using the following formula:

[0144] Cm =[r·cos((m-1)·χ)r·sin((m-1)·χ)0] T (twenty one)

[0145] Among them, C m To identify the corner point P of the pattern by its pose 11 As the starting point, the three-dimensional coordinates of the m-th pose marker corner point in the pose marker coordinate system; χ is the angle around the axis between adjacent pose marker corner points.

[0146] In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system can be known. For example, the reference coordinate system is the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system can be determined based on the camera's pose. In other embodiments, the reference coordinate system can also be the camera coordinate system itself, depending on actual needs.

[0147] In some embodiments, based on the camera imaging principle and projection model, the pose of the pose marker coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of the corner points of multiple pose marker patterns in the positioning image and the three-dimensional coordinates of the corner points of multiple pose marker patterns in the pose marker coordinate system. Based on the transformation relationship between the pose of the pose marker coordinate system relative to the camera coordinate system and the camera coordinate system relative to the reference coordinate system, the pose of the pose marker coordinate system relative to the reference coordinate system can be obtained. In some embodiments, the camera's intrinsic parameters can also be considered. For example, the camera's intrinsic parameters can be as follows: Figure 1 The image acquisition device 110 shown has camera intrinsic parameters. The camera's intrinsic parameters can be known or obtained through calibration.

[0148] In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the optical center of the camera as the origin or a coordinate system established with the center of the camera lens as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, or the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of the line).

[0149] See Figure 9 In step 905, the pose of the manipulator coordinate system relative to the reference coordinate system is determined based on the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system and the pose of the pose identifier coordinate system relative to the reference coordinate system. In some embodiments, the pose of the manipulator coordinate system relative to the reference coordinate system can be used as the current relative pose of the manipulator relative to the reference coordinate system.

[0150] For example, taking the reference coordinate system as the world coordinate system, the pose of the manipulator coordinate system relative to the world coordinate system is as follows:

[0151]

[0152] in, w R wm This refers to the orientation of the manipulator's coordinate system relative to the world coordinate system. w P wm This represents the position of the manipulator coordinate system relative to the world coordinate system. w R wm0 The pose of the position coordinate system relative to the world coordinate system. w P wm0 rot represents the position of the pose coordinate system relative to the world coordinate system. z (α0) represents the roll angle α0 around the Z-axis of the manipulator coordinate system.

[0153] In some embodiments, the specific formula for calculating the pose of the manipulator coordinate system relative to the world coordinate system is as follows:

[0154]

[0155] in, w R lens The pose of the camera coordinate system relative to the world coordinate system. w P lens This represents the position of the camera coordinate system relative to the world coordinate system. lens R wm0 The pose identifier coordinate system is positioned relative to the camera coordinate system. lens P wm0 The pose identifier coordinate system is positioned relative to the camera coordinate system. wm0 R wm The orientation of the relative pose identifier coordinate system of the manipulator coordinate system. wm0 P wm The position of the relative pose identifier coordinate system of the manipulator coordinate system.

[0156] Figure 10 A flowchart illustrating a method 1000 for determining the pose of an arm coordinate system relative to a reference coordinate system, according to other embodiments of this disclosure, is shown. Method 1000 may be... Figure 9 Alternative embodiments of method 900, such as... Figure 10 As shown, some or all of the steps in method 1000 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 18 The processor 1820 shown is used to execute the method. Some or all of the steps in method 1000 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1000 can be executed by a robot system (e.g., Figure 18The surgical robot system 1800 shown is executed. In some embodiments, method 1000 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0157] See Figure 10 In step 1001, based on the roll angle of the pose marker coordinate system relative to the manipulator coordinate system and the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system, the three-dimensional coordinates of the multiple pose markers in the manipulator coordinate system are determined. It can be understood that, given the roll angle of the pose marker coordinate system relative to the manipulator coordinate system, the three-dimensional coordinates of the corner points of the multiple pose marker patterns in the pose marker coordinate system can be transformed into their three-dimensional coordinates in the manipulator coordinate system through coordinate transformation.

[0158] In step 1003, the pose of the manipulator coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the manipulator coordinate system. In some embodiments, step 1003 can be implemented similarly to steps 903 and 905 in method 900.

[0159] Figure 12 A flowchart illustrating a method 1200 for identifying pose identifiers according to some embodiments of the present disclosure is shown. Figure 12 As shown, some or all of the steps in method 1200 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 18 The processor 1820 shown is used to execute the method. Some or all of the steps in method 1200 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1200 can be executed by a robot system (e.g., Figure 18 The surgical robot system 1800 shown is executed. In some embodiments, method 1200 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0160] See Figure 12In step 1201, multiple candidate pose identifiers are determined from the localization image. In some embodiments, candidate pose identifiers can be represented by candidate pose identifier pattern corner points. In some embodiments, candidate pose identifier pattern corner points can refer to possible pose identifier pattern corner points obtained after preliminary processing or preliminary identification of the localization image. In some embodiments, a Region of Interest (ROI) can be extracted from the localization image first, and multiple candidate pose identifiers can be determined from the ROI. The ROI can be the entire localization image or a partial region. For example, the ROI of the current frame can be extracted based on a certain range of multiple pose identifier pattern corner points determined in the previous frame image (e.g., the localization image of the previous image processing cycle). For localization images that are not the first frame, the ROI can be a region within a certain distance centered on a virtual point formed by the coordinates of multiple pose identifier pattern corner points from the previous image processing cycle. The certain distance range can be a fixed multiple of the average interval distance of the pose identifier pattern corner points, such as twice. It should be understood that the predetermined multiple can also be a variable multiple of the average interval distance of the multiple candidate pose identifier pattern corner points in the previous image processing cycle.

[0161] In some embodiments, method 1200 may include: determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of: image grayscale conversion, image denoising, and image enhancement.

[0162] For example, image preprocessing may include: cropping the Region of Interest (ROI) from the localized image and converting the ROI into a corresponding grayscale image.

[0163] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be calculated according to the following formula:

[0164]

[0165] Where τ is a set constant, for example, set to 2; I x I 45 I y I n45These are the first derivatives of the pixel in the four directions: 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively.

[0166] In some embodiments, the Region of Interest (ROI) is divided into multiple sub-images. For example, a non-maximum suppression method can be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI can be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the location image or ROI can also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels. The pixel with the largest CL value in each sub-image can be determined, and the pixel with the largest CL value in each sub-image can be compared with a first threshold to determine the set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold can be set to 0.06. It should be understood that the first threshold can also be set to other values. In some embodiments, pixels with a CL value greater than the first threshold can be used as candidate pose identifier pattern corner points.

[0167] See Figure 12 In step 1203, an initial pose identifier is identified from multiple candidate pose identifiers based on a pose pattern matching template. In some embodiments, the pose pattern matching template is used to match the image at one of the corner points of the candidate pose identifier pattern to determine the corner point of the candidate pose identifier pattern that meets the preset pose pattern matching degree standard as the corner point of the initial pose identifier pattern.

[0168] In some embodiments, the pose pattern matching template and the image of the region near the corner point of the pose identifier pattern have the same or similar features. If the matching degree between the pose pattern matching template and the image of the region near the corner point of the candidate pose identifier pattern reaches a preset pose pattern matching degree standard (e.g., the matching degree is higher than a threshold), it can be considered that the pattern of the region near the corner point of the candidate pose identifier pattern has the same or similar features as the pose pattern matching template, and thus the current candidate pose identifier pattern corner point can be considered as the pose identifier pattern corner point.

[0169] In some embodiments, the pixel with the largest CL value in the pixel set is determined as the corner point of the candidate pose identifier pattern to be matched. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value is selected as the corner point of the candidate pose identifier pattern to be matched. After the corner point of the candidate pose identifier pattern to be matched is determined, the pose pattern matching template is used to match the pattern at the corner point of the candidate pose identifier pattern. If the preset pose pattern matching degree standard is met, the corner point of the candidate pose identifier pattern to be matched is determined as the identified initial pose identifier pattern corner point. If the corner point of the candidate pose identifier pattern to be matched does not meet the preset matching degree standard, the pixel with the second largest CL value is selected as the corner point of the candidate pose identifier pattern to be matched, and the pose pattern matching template is used to match the image at the corner point of the candidate pose identifier pattern. This process is repeated until the initial pose identifier pattern corner point is identified.

[0170] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, therefore the pose pattern matching template can be the same checkerboard pattern, utilizing the grayscale distribution G of the pose pattern matching template. M The pixel neighborhood grayscale distribution G of the pixel corresponding to the corner point of the candidate pose identifier pattern image The correlation coefficient (CC) between pixels is used for matching. The grayscale distribution G of the pixel neighborhood is also considered. image This represents the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The specific formula is as follows:

[0171]

[0172] Where Var is the variance function and Cov is the covariance function. In some embodiments, when the CC value is less than 0.8, the grayscale distribution in the pixel neighborhood has a low correlation with the pose pattern matching template. In this case, the candidate pose pattern corner with the highest likelihood value is determined to be the pose pattern corner. Otherwise, the candidate pose pattern corner with the highest likelihood value is considered to be the pose pattern corner.

[0173] In some embodiments, method 1200 includes: determining the edge orientation of corner points of candidate pose identifier patterns. For example, such as Figure 13 As shown, Figure 13 It includes a pose identifier pattern 1311, and the corner points of the candidate pose identifier pattern are... Figure 13 Corner point P in 13 Then the corner point P 13 The edge direction can refer to the corner point P. 13 The direction of the edge, such as Figure 13The direction indicated by the dashed arrow.

[0174] In some embodiments, the edge direction can be determined by the first-order derivative (I0) of each pixel in the X and Y directions of the planar coordinate system with respect to a certain neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate pose identifier pattern. x and I y The direction of an edge can be determined using the following formula:

[0175]

[0176] Among them, the first derivative (I) x and I y This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by performing a convolution operation on the edge direction I of each pixel within the neighborhood range. angle and the corresponding weight I weight Clustering calculations are performed to obtain the edge direction of the pixel, and weight I is selected. weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if multiple edge directions exist, then weight I is selected. weight The I corresponding to the largest proportion of multiple classes angle As the edge direction.

[0177] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0178] In some embodiments, method 1200 includes: rotating a pose pattern matching template according to an edge direction. Rotating the pose pattern matching template according to an edge direction can align the pose pattern matching template with the image at the corner point of a candidate pose identifier pattern.

[0179] The edge direction of the corner point of the candidate pose marker pattern can be used to determine the setting orientation of the image at the corner point of the candidate pose marker pattern in the positioning image. In some embodiments, rotating the pose pattern matching template according to the edge direction can adjust the pose pattern matching template to be the same as or nearly the same as the image orientation at the corner point of the candidate pose marker pattern to facilitate image matching.

[0180] See Figure 12 In step 1205, starting from the initial pose identifier, the pose identifier is searched.

[0181] For example, Figure 14 A flowchart illustrating a method 1400 for searching pose identifiers according to some embodiments of the present disclosure is shown. Figure 14 As shown, some or all of the steps in method 1400 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 18 The processor 1820 shown is used to execute the method. Some or all of the steps in method 1400 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1400 can be executed by a robot system (e.g., Figure 18 The surgical robot system 1800 shown is executed. In some embodiments, method 1400 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0182] See Figure 14 In step 1401, a second pose identifier is determined, starting from the initial pose identifier. In some embodiments, the corner point of the initial pose identifier pattern is used as the starting point, and the corner point of the second pose identifier pattern is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following: directly in front of the corner point of the initial pose identifier pattern (corresponding to a 0° angle direction), directly behind (corresponding to a 180° angle direction), directly above (90° angle direction), directly below (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0183] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions. The vsn of each search direction can be calculated according to the following formula:

[0184] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (27)

[0185] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent pose marker corner points among the multiple pose marker corner points determined in the previous frame. For example, the predetermined search direction can be calculated using the following formula:

[0186]

[0187] Among them, (x j ,y j ) represents the two-dimensional coordinates of the corner points of multiple pose marker patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple pose marker patterns determined in the previous frame; v s1 The first set search direction; v s2 This is the second search direction set.

[0188] In some embodiments, such as Figure 15 As shown, the corner point P of the pattern is identified by its initial pose. 151 Using the coordinates of the given location as the starting point, search for the second pose marker corner point P in the set search direction. 152 Specifically, the coordinate position can include: identifying the corner point P of the pattern in the initial pose. 151 Using the coordinates as the starting point for the search, the search box (e.g., ...) Figure 15 (The dashed box in the image) moves in the set search direction V with a certain search step size. 151 Search for the corner points of the pose marker pattern. If there is at least one candidate corner point of the pose marker pattern within the search box, then the candidate corner point with the highest likelihood value within the search box is selected as the second corner point of the pose marker pattern, P. 152 With the search box constrained to a suitable size, the pattern corner point P is identified in its initial pose. 151 The coordinates of the point are used as the starting point for the second pose identification pattern corner point P. 152 During the search, the candidate pose marker corner with the highest corner likelihood value among the candidate pose marker corners appearing in the search box is more likely to be the actual pose marker corner. Therefore, it can be considered that the candidate pose marker corner with the highest corner likelihood value in the search box is the second pose marker corner P. 152To improve data processing speed. In other embodiments, to improve the accuracy of pose marker pattern corner point recognition, when at least one candidate pose marker pattern corner point exists in the search box, the candidate pose marker pattern corner point with the highest corner point likelihood value among the candidate pose marker pattern corner points appearing in the search box is selected for corner point recognition to determine whether the candidate pose marker pattern corner point with the highest corner point likelihood value is a pose marker pattern corner point. For example, the pose pattern matching template is matched with the image within a certain range of the candidate pose marker pattern corner point with the highest corner point likelihood value. The candidate pose marker pattern corner point that meets the preset pose pattern matching degree standard can be considered as the second pose marker pattern corner point P found. 152 .

[0189] In some embodiments, continue reading Figure 15 The size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0190] In some embodiments, the pose marker pattern can be a black and white checkerboard pattern, and the correlation coefficient CC in formula (25) can be used for pattern matching. If CC is greater than the threshold, the candidate pose marker pattern corner with the largest corner likelihood value is considered to be the pose marker pattern corner, and is recorded as the second pose marker pattern corner.

[0191] See Figure 14 In step 1403, a search direction is determined based on the initial pose identifier and the second pose identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of a corner point of the initial pose identifier pattern and moving away from the corner point of the second pose identifier pattern. The second search direction may be a direction starting from the coordinate position of a corner point of the second pose identifier pattern and moving away from the corner point of the first pose identifier pattern. For example, Figure 15 The search direction V shown 152 .

[0192] In step 1405, starting from the initial pose identifier or the second pose identifier, a search for pose identifiers is performed in the search direction. In some embodiments, if the first pose identifier pattern corner point is used as the new starting point, the first search direction described above can be used as the search direction for the pose identifier pattern corner point. If the second pose identifier pattern corner point is used as the new starting point, the second search direction described above can be used as the search direction for the pose identifier pattern corner point. In some embodiments, a new pose identifier pattern corner point is searched (e.g., Figure 15 The third pose marker pattern corner point P in 153This can be performed similarly to step 1401. In some embodiments, the search step size can be the distance L1 between the corner points of the initial pose identifier pattern and the corner points of the second pose identifier pattern.

[0193] In some embodiments, the search for pose identifier pattern corner points is stopped in response to the number of corner points found being greater than or equal to a pose identifier pattern corner point number threshold. For example, the search for pose identifier pattern corner points is stopped when four pose identifier pattern corner points are found (identified).

[0194] In some embodiments, the search for the Nth pose marker corner point is stopped when the search distance is greater than a set multiple of the distance between the (N-1)th and (N-2)th pose marker corner points, where N ≥ 3. For example, the search termination condition could be that the search distance is greater than twice the distance between the first two pose marker corner points. Thus, the maximum search distance for the third pose marker corner point is twice the distance between the initial pose marker corner point and the second pose marker corner point. If no pose marker corner point is found after reaching this search distance, it is considered that the third pose marker corner point has not been found and the search ends.

[0195] In some embodiments, if the total number of pose marker corner points found is greater than or equal to a set threshold (e.g., the set threshold is 4), then it is considered that enough pose marker corner points have been successfully identified. If the total number of pose marker corner points found is less than the set value, then the search based on the initial pose marker corner point in the above steps is considered unsuccessful. In the case of unsuccessful search, a new initial pose marker corner point is determined from the candidate pose marker corner points, and then the remaining pose marker corner points are searched based on the newly determined initial pose marker corner point as the search starting point. Similar to method 1200, a new initial pose marker corner point can be determined, and similar to method 1400, the remaining pose marker corner points can be searched based on the new pose marker corner point as the search starting point.

[0196] In some embodiments, after the corner points of the pose marker pattern are searched or identified, sub-pixel positioning can be performed on the determined corner points of the pose marker pattern to improve the positional accuracy of the corner points of the pose marker pattern.

[0197] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of the corner points of the pose identifier pattern after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, where the extreme points of the function are subpixel points. The fitting function can be as follows:

[0198] S(x,y)=ax 2 +by2 +cx+dy+exy+f (29)

[0199]

[0200] Where S(x, y) is the fitting function for the CL values ​​of all pixels in each ROI, and a, b, c, d, e, and f are coefficients; x c The x-coordinate and y-coordinate of the pose identifier c The y-coordinate is the pose identifier.

[0201] Figure 16 A flowchart illustrating a method 1600 for identifying angle markers according to some embodiments of the present disclosure is shown. Figure 16 As shown, some or all of the steps in method 1600 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 18 The processor 1820 shown is used to execute the method. Some or all of the steps in method 1600 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1600 can be executed by a robot system (e.g., Figure 18 The surgical robot system 1800 shown is executed. In some embodiments, method 1600 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0202] See Figure 16 In step 1601, an imaging transformation relationship is determined based on the two-dimensional coordinates of multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system. In some embodiments, the pose marker coordinate system may be the pose marker coordinate system detailed in the embodiment shown in method 700. For example, the pose marker coordinate system is as follows: Figure 6 As shown. In some embodiments, the imaging transformation relationship can refer to the transformation relationship between the three-dimensional coordinates in the pose identifier coordinate system and the two-dimensional coordinates in the positioning image. It should be understood that, based on the imaging transformation relationship, the two-dimensional coordinates in the positioning image can also be transformed into three-dimensional coordinates in the pose identifier coordinate system. In some embodiments, the three-dimensional coordinates of multiple pose identifiers in the pose identifier coordinate system can be determined based on formula (21). In some embodiments, the number of multiple pose identifiers can be greater than or equal to four. For example, the imaging transformation relationship can be obtained based on the two-dimensional coordinates of four pose identifiers in the positioning image and the four corresponding three-dimensional coordinates in the pose identifier coordinate system.

[0203] See Figure 16In step 1603, based on the imaging transformation relationship, the three-dimensional coordinates and positional relationships of multiple pose markers in the pose marker coordinate system, multiple candidate regions for angle markers are determined in the positioning image. In some embodiments, the candidate regions for angle markers may represent candidate regions for angle marker patterns. In some embodiments, based on the three-dimensional coordinates and positional relationships of the corner points of multiple pose marker patterns in the pose marker coordinate system, multiple candidate three-dimensional coordinates of the corner points of multiple angle marker patterns are determined in the pose marker coordinate system. For example, based on the three-dimensional coordinates of the corner points of multiple pose marker patterns in the pose marker coordinate system, a certain distance can be offset along the axial direction to determine multiple three-dimensional coordinates in the pose marker coordinate system. These three-dimensional coordinates are represented by the candidate three-dimensional coordinates of the corner points of multiple angle marker patterns. For example, see... Figure 4 The positional relationship is that the angle marker and the corresponding pose marker are spaced a certain distance along the Z-axis of the pose marker coordinate system. Given the position of the corner point of the pose marker pattern, the position obtained by moving a certain distance along the positive or negative direction of the Z-axis can be considered as the candidate position of the corner point of the angle marker pattern in the pose marker coordinate system.

[0204] In some embodiments, multiple candidate regions for angle markers are determined in the positioning image based on the imaging transformation relationship and the three-dimensional coordinates of multiple candidate corner points for angle marker patterns. For example, multiple two-dimensional coordinates of candidate corner points for angle marker patterns are obtained in the positioning image based on the imaging transformation relationship and the three-dimensional coordinates of multiple candidate corner points for angle marker patterns. In some embodiments, multiple candidate regions for angle marker patterns are determined based on the two-dimensional coordinates of multiple candidate corner points for angle marker patterns. For example, a region of a certain size (e.g., 5×5 pixels, 10×10 pixels, etc.) is determined in the positioning image centered on each candidate corner point coordinate of an angle marker pattern as a candidate region for angle markers. In some embodiments, the region of a certain size is greater than or equal to the size of the angle marker pattern after imaging. The size of the angle marker pattern after imaging can be obtained based on the actual size of the angle marker pattern and the imaging transformation relationship.

[0205] See Figure 16 In step 1605, candidate regions are identified from multiple angles, and angle identifiers are recognized. In some embodiments, the angle identifier includes an angle identifier pattern and angle identifier pattern corner points. In some embodiments, method 1600 may include determining the pixel with the largest corner likelihood value in each candidate region of the angle identifier to form a pixel set. In some embodiments, the corner likelihood value of the pixel may be calculated when performing method 1200, or it may be recalculated based on formula (24). Method 1600 also includes determining the candidate region of the angle identifier corresponding to the pixel with the largest corner likelihood value in the pixel set as the candidate region of the angle identifier to be identified. Method 1600 also includes matching the candidate region of the angle identifier to be identified with multiple angle pattern matching templates respectively to identify the angle identifier.

[0206] In some embodiments, the angle marker patterns are patterns with different graphic features. Multiple angle pattern matching templates can refer to standard angle pattern templates that have the same or similar graphic features and correspond to the multiple angle marker patterns respectively.

[0207] In some embodiments, by determining multiple candidate regions for angle markers, angle markers can be identified in multiple candidate regions, avoiding the need to identify angle markers across the entire image range and improving the speed of data processing.

[0208] In some embodiments, any one of the template matching algorithms, such as the squared difference matching method, normalized squared difference matching method, correlation matching method, normalized correlation matching method, correlation coefficient matching method, and normalized correlation coefficient matching method, can be used to perform matching operations between the angle pattern matching template and the angle identifier candidate region.

[0209] In some embodiments, since the angle pattern matching template and the angle marker pattern have the same or similar graphic features, the pattern information of the angle marker may include the pattern information of the corresponding angle pattern matching template. For example, the shape of the angle pattern matching template, identifiable features in the image, etc. In some embodiments, each angle pattern matching template has a one-to-one correspondence with the axial angle identified by the corresponding angle marker pattern. The first axial angle is determined based on a specific angle pattern matching template or the pattern information of the angle marker pattern corresponding to the identified angle marker.

[0210] In some embodiments, method 1600 may include, in response to a matching failure, determining the candidate region of the angle identifier corresponding to the pixel with the highest corner likelihood value among the remaining pixels in the pixel set as the candidate region of the angle identifier to be identified. In some embodiments, after determining the new candidate region of the angle identifier to be identified, multiple angle pattern matching templates are used to match the candidate region of the angle identifier to be identified, respectively, to identify the angle identifier.

[0211] In some embodiments, a first pose marker that has a positional relationship with the angle marker is determined based on the angle marker candidate region where the identified angle marker is located. In some embodiments, the multiple angle marker candidate regions respectively correspond to at least one of the multiple identified pose marker pattern corner points. After determining the angle marker candidate region where the identified angle marker is located, the first pose marker pattern corner point can be determined based on the correspondence between the multiple angle marker candidate regions and the multiple pose marker pattern corner points.

[0212] In some embodiments of this disclosure, a computer device is also provided, including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 7 , Figure 8 , Figure 9 , Figure 10 , Figure 12 , Figure 14 and Figure 16 Some or all of the steps in the method disclosed herein.

[0213] Figure 17 A schematic block diagram of a computer device 1700 according to some embodiments of the present disclosure is shown. See also Figure 17 The computer device 1700 may include a central processing unit (CPU) 1701, a system memory 1704 including random access memory (RAM) 1702 and read-only memory (ROM) 1703, and a system bus 1705 connecting the various components. The computer device 1700 may also include an input / output system and a mass storage device 1707 for storing an operating system 1713, application programs 1714, and other program modules 1715. The input / output devices include an input / output controller 1710, primarily composed of a display 1708 and input devices 1709.

[0214] Mass storage device 1707 is connected to central processing unit 1701 via a mass storage controller (not shown) connected to system bus 1705. Mass storage device 1707 or computer-readable media provides non-volatile storage for computer devices. Mass storage device 1707 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0215] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0216] Computer device 1700 can be connected to network 1712 via network interface unit 1711 connected to system bus 1705.

[0217] The system memory 1704 or mass storage device 1707 is also used to store one or more instructions. The central processing unit 1701 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0218] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 7 , Figure 8 , Figure 9 , Figure 10 , Figure 12 , Figure 14 and Figure 16 Some or all of the steps in the disclosed method. Examples of computer-readable storage media include memory for computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0219] Figure 18 A schematic diagram of a surgical robot system 1800 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 18 The surgical robot system 1800 may include a surgical instrument 1850, an image acquisition unit 1810, and a processor 1820. The surgical instrument 1850 may include a manipulator arm 1840 and an end effector 1830 disposed at the distal end of the manipulator arm 1840. The end effector 1830 may include at least one angle marker, multiple pose markers, and actuators. The image acquisition unit 1810 may be used to acquire positioning images of the manipulator arm 1840. The processor 1820 is connected to the image acquisition unit 1810 and is used to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 7 , Figure 8 , Figure 9 , Figure 10 , Figure 12 , Figure 14 and Figure 16 Some or all of the steps in the method disclosed herein.

[0220] While specific embodiments of this disclosure have been illustrated and described by way of example, it will be apparent to those skilled in the art that many other changes and modifications can be made without departing from the spirit and scope of this disclosure. Therefore, all such changes and modifications falling within the scope of this disclosure are included in the appended claims.

Claims

1. A method for controlling an operating arm, comprising: Acquire the location image; In the positioning image, multiple pose markers located on the manipulator are identified; Based on the plurality of pose identifiers, an angle identifier located on the manipulator is identified, and the angle identifier has a positional association with the first pose identifier among the plurality of pose identifiers; Based on the angle identifier and the multiple pose identifiers, the current relative pose of the manipulator relative to the reference coordinate system is determined; as well as Based on the current relative pose and the target pose of the manipulator, the drive signal of the manipulator is determined.

2. The control method according to claim 1, comprising: Based on the current relative pose and the target pose of the manipulator, determine the pose difference; as well as Based on the pose difference and the inverse kinematics model of the manipulator, the drive signal of the manipulator is determined.

3. The control method according to claim 2, wherein the control method comprises: Based on the current relative pose, determine the current pose of the manipulator in the world coordinate system; as well as Based on the target pose of the manipulator and the current pose of the manipulator in the world coordinate system, the pose difference is determined, and the target pose of the manipulator is the target pose of the manipulator in the world coordinate system.

4. The control method according to claim 3, comprising: Based on the pose difference, determine the Cartesian space velocity; Based on the Cartesian space velocity, determine the parameter space velocity; Based on the parameter space velocity and the current joint parameters, determine the target joint parameters; as well as The drive signal is determined based on the target joint parameters.

5. The control method according to claim 4, wherein the pose difference includes position difference and attitude difference, the Cartesian space velocity includes Cartesian space linear velocity and Cartesian space angular velocity, and the control method further includes: Based on the position difference, the linear velocity in Cartesian space is determined; as well as Based on the attitude difference, the Cartesian space angular velocity is determined.

6. The control method according to claim 4, wherein the operating arm comprises: At least one component, the component including a fixing disk and multiple structural bones, a first end of the multiple structural bones being fixedly connected to the fixing disk, and a second end of the multiple structural bones being connected to a drive unit; The control method further includes: Based on the target joint parameters, the driving force of the multiple structural bones is determined; and Based on the driving amount of the multiple structural bones, the driving signal for the driving unit is determined.

7. The control method according to any one of claims 1-6, further comprising: Receive control commands; as well as Based on the control commands, the target pose of the manipulator is determined.

8. The control method according to any one of claims 1-6, further comprising: The drive signal of the operating arm is determined at a predetermined period to achieve real-time control through multiple motion control cycles.

9. The control method according to claim 1, comprising: Based on the angle identifier and the multiple pose identifiers, determine the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system; Based on the multiple pose identifiers, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined; as well as Based on the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system and the pose of the pose identifier coordinate system relative to the reference coordinate system, the pose of the manipulator coordinate system relative to the reference coordinate system is determined.

10. The control method according to claim 9, comprising: Based on the two-dimensional coordinates of the plurality of pose markers in the positioning image and the three-dimensional coordinates of the plurality of pose markers in the pose marker coordinate system, the pose of the pose marker coordinate system relative to the reference coordinate system is determined.

11. The control method according to claim 1, comprising: Based on the angle identifier and the multiple pose identifiers, determine the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system; Based on the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system and the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system, the three-dimensional coordinates of the multiple pose identifiers in the manipulator coordinate system are determined. as well as Based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the manipulator coordinate system, the pose of the manipulator coordinate system relative to the reference coordinate system is determined.

12. The control method according to any one of claims 9-11, comprising: Determine the first angle around the axis marked by the angle marker in the manipulator coordinate system; Determine the second angle around the axis marked by the first pose identifier in the pose identifier coordinate system; as well as Based on the first and second axial angles, the roll angle of the pose identifier coordinate system relative to the manipulator coordinate system is determined.

13. The control method according to any one of claims 1-3 and 9-11, wherein the position association includes: The axial correspondence between the angle identifier and the first pose identifier.

14. The control method according to claim 1, comprising: Based on the distribution of the multiple pose markers, the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system are determined; Based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system, the imaging transformation relationship is determined; Based on the imaging transformation relationship, the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system, and the positional association relationship, multiple angle marker candidate regions are determined in the positioning image. as well as Candidate regions are identified from the multiple angles, and the angle identifiers are recognized.

15. The control method according to claim 14, comprising: Based on the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system and the positional relationship, multiple candidate three-dimensional coordinates of angle markers are determined in the pose marker coordinate system. as well as Based on the imaging transformation relationship and the three-dimensional coordinates of the multiple angle marker candidates, the multiple angle marker candidate regions are determined in the positioning image.

16. The control method according to claim 14 or 15, comprising: The pixel with the largest corner likelihood value in each of the candidate regions of the angle identifier is determined to form a pixel set; The candidate region for the angle identifier corresponding to the pixel with the largest corner likelihood value in the pixel set is determined as the candidate region for the angle identifier to be identified. as well as Multiple angle pattern matching templates are used to match the candidate regions of the angle identifiers to be identified, so as to identify the angle identifiers.

17. The control method according to claim 14 or 15, comprising: Based on the candidate region of the angle identifier where the angle identifier is located, the first pose identifier that has a positional relationship with the angle identifier is determined.

18. The control method according to claim 1, comprising: Multiple candidate pose identifiers are determined from the positioning image; Based on the pose pattern matching template, an initial pose identifier is identified from the plurality of candidate pose identifiers; as well as Starting from the initial pose identifier, search for pose identifiers.

19. A computer device, the computer device comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory, is configured to execute the at least one instruction to perform the method as described in any one of claims 1-18.

20. A computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the method as described in any one of claims 1-18.

21. A surgical robot system, comprising: The surgical tool includes an operating arm, an actuator disposed at the distal end of the operating arm, and at least one angle marker and multiple pose markers disposed at the end of the operating arm. An image acquisition device is used to acquire positioning images of the operating arm; as well as A processor, connected to the image acquisition unit, is configured to perform the method as described in any one of claims 1-18 to determine the drive signal for the manipulator.

Images