Method for controlling an operating arm and surgical robotic system

Abstract

The present disclosure relates to the technical field of control, and discloses a method for controlling an operating arm, a computer device, a computer readable storage medium and a surgical robot system. The method for controlling the operating arm comprises: acquiring a positioning image; in the positioning image, identifying a plurality of marks located on the operating arm, the plurality of marks comprising a plurality of pose marks for identifying a pose and at least one composite mark for identifying a pose and an angle; determining a current relative pose of the operating arm relative to a reference coordinate system based on the at least one composite mark and the plurality of pose marks; and determining a driving signal of the operating arm based on the current relative pose and a target pose of the operating arm.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of control, and particularly relates to a method for controlling an operating arm and a surgical robot system. BACKGROUND

[0002] With the development of technology, it becomes more and more popular to use machines controlled by humans or computers to perform desired actions to assist or replace human operators. For example, using logistics robots to sort express delivery, using surgical robots to assist doctors to perform surgery, etc.

[0003] In the above applications, the operating arm needs to be controlled to achieve the control of the machine. SUMMARY

[0004] In some embodiments, the present disclosure provides a method for controlling an operating arm, comprising: acquiring a positioning image; in the positioning image, identifying a plurality of markers located on the operating arm, the plurality of markers comprising a plurality of pose markers for identifying a pose and at least one composite marker for identifying a pose and an angle; determining a current relative pose of the operating arm relative to a reference coordinate system based on the at least one composite marker and the plurality of pose markers; and determining a driving signal of the operating arm based on the current relative pose and a target pose of the operating arm.

[0005] In some embodiments, the present disclosure provides a computer device, comprising: a memory for storing at least one instruction; and a processor coupled to the memory for executing the at least one instruction to perform the method of any one of the embodiments of the present disclosure.

[0006] In some embodiments, the present disclosure provides a computer-readable storage medium having at least one instruction stored therein, the at least one instruction being executed by a processor to cause a computer to perform the method of any one of the embodiments of the present disclosure.

[0007] In some embodiments, the present disclosure provides a surgical robot system, comprising: a surgical tool comprising an operating arm, an effector arranged at a distal end of the operating arm, and at least one composite marker and a plurality of pose markers arranged on a distal end of the operating arm; an image collector for collecting a positioning image of the operating arm; and a processor connected to the image collector for executing the method of any one of the embodiments of the present disclosure to determine a driving signal of the operating arm. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 A schematic diagram of a control system according to some embodiments of the present disclosure is shown;

[0009] Figure 2 A schematic diagram of a label comprising a plurality of markers according to some embodiments of the present disclosure is shown;

[0010] Figure 3 A schematic diagram showing a label arranged on the side of the end of the operating arm and forming a cylinder according to some embodiments of the present disclosure;

[0011] Figure 4 A schematic diagram showing an implementation scenario according to some embodiments of the present disclosure;

[0012] Figure 5 A flow chart showing a control method of an operating arm control system according to some embodiments of the present disclosure;

[0013] Figure 6 A flow chart showing a method of determining a current relative pose of an operating arm relative to a reference coordinate system according to some embodiments of the present disclosure;

[0014] Figure 7 A flow chart showing a method of determining a pose of an operating arm relative to a reference coordinate system according to some embodiments of the present disclosure;

[0015] Figure 8 A flow chart showing a method for recognizing an identification according to some embodiments of the present disclosure;

[0016] Figure 9 A schematic diagram showing a pose identification pattern according to some embodiments of the present disclosure;

[0017] Figure 10 A flow chart showing a method for searching an identification according to some embodiments of the present disclosure;

[0018] Figure 11 A schematic diagram showing a search for an identification according to some embodiments of the present disclosure;

[0019] Figure 12 A schematic diagram showing a segment of an operating arm according to some embodiments of the present disclosure;

[0020] Figure 13 A schematic diagram showing a structure of an operating arm according to some embodiments of the present disclosure;

[0021] Figure 14 A flow chart showing a method for determining a driving signal according to some embodiments of the present disclosure;

[0022] Figure 15 A schematic diagram of a computer device according to some embodiments of the present disclosure;

[0023] Figure 16 A schematic diagram of a surgical robot system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. It is understood that the scope of the present disclosure is not limited to these embodiments. Various modifications and changes can be made to the embodiments described below. Such modifications and changes are intended to be included within the scope of the present disclosure. Similar reference numerals in the various embodiments of the present disclosure indicate similar components.

[0025] In the present disclosure, the term "position" refers to the positioning of an object or a part of an object in a three-dimensional space (e.g., three translational degrees of freedom can be described using variations in Cartesian X, Y, and Z coordinates, e.g., three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In the present disclosure, the term "attitude" refers to the rotational setting of an object or a part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In the present disclosure, the term "pose" refers to the combination of the position and attitude of an object or a part of an object, which can be described using six parameters out of the six degrees of freedom mentioned above.

[0026] In the present disclosure, a reference coordinate system can be understood as a coordinate system describing the pose of an object. According to the actual positioning requirements, the reference coordinate system can be selected to have the origin of a virtual reference object or the origin of a physical reference object as the origin of the coordinate system. In some embodiments, the reference coordinate system can be a world coordinate system or a camera coordinate system or a perception coordinate system of an operator, etc. In the present disclosure, an object can be understood as an object or target to be positioned, such as an operating arm or an end of an operating arm or an effector arranged at the distal end of the end of the operating arm. The operating arm can be a rigid arm or a deformable arm. In the present disclosure, the pose of the operating arm or a part thereof refers to the pose of the operating arm coordinate system defined by the operating arm or the part thereof relative to the reference coordinate system.

[0027] Figure 1 A schematic diagram of a control system 100 according to some embodiments of the present disclosure is shown. As shown, the control system 100 can include an image acquisition device 110, at least one operating arm 140, and a control device 120. The image acquisition device 110 and the at least one operating arm 140 are respectively in communication connection with the control device 120. In some embodiments, as shown, Figure 1 Figure 1 ​As shown, the control device 120 can be used to control the movement of at least one manipulator 140 to adjust the pose of the at least one manipulator 140, coordinate with each other, etc. In some embodiments, at least one manipulator 140 may include a manipulator end effector 130 at its distal end or end. The control device 120 can control the movement of at least one manipulator 140 to move the manipulator end effector 130 to a desired position and orientation. Those skilled in the art will understand that the control system 100 can be applied to surgical robot systems, such as laparoscopic surgical robot systems. For example, an actuator 160 may be disposed at the distal end of the manipulator end effector 130, such as... Figure 1 As shown. It should be understood that the control system 100 can also be applied to dedicated or general-purpose robot systems in other fields (e.g., manufacturing, machinery, etc.).

[0028] In this disclosure, the control device 120 can be communicatively connected to the drive unit 150 (e.g., a motor) of at least one manipulator 140 and send drive signals to the drive unit 150, thereby enabling the drive unit 150 to control at least one manipulator 140 to move to a corresponding target pose based on the drive signals. For example, the drive unit 150 controlling the movement of the manipulator 140 can be a servo motor, which can receive instructions from the control device to control the movement of the manipulator 140. The control device 120 can also be communicatively connected to a sensor coupled to the drive unit 150, for example, through a communication interface, to receive motion data of the manipulator 140 and monitor the motion status of the manipulator 140. In one example of this disclosure, the communication interface can be a CAN (Controller Area Network) bus communication interface, which enables the control device 120 to communicate with the drive unit 150 and the sensor via the CAN bus. In some embodiments, the control device 120 may include a local processor (e.g., a local computer device) or a cloud processor (e.g., a cloud server or cloud computing platform).

[0029] In some embodiments, the manipulator 140 may include a continuous deformable arm, such as a multi-degree-of-freedom manipulator composed of multiple joints, such as a manipulator capable of 6 degrees of freedom of motion.

[0030] In some embodiments, the image acquisition device 110 can be used to acquire positioning images. The positioning images may include part or all of the image of the manipulator arm 140. In some embodiments, the image acquisition device 110 can be used to acquire images of the manipulator arm end effector 130, which may have multiple different pose markers, including different pose marker patterns. For example, a positioning label 170 may be provided on the manipulator arm end effector 130 (the positioning label 170 may be, for example, a positioning tag 170). Figure 2The label 200 shown. The positioning label 170 may include multiple identifiers, including multiple pose identifiers for identifying pose and at least one composite identifier for identifying pose and angle (detailed below).

[0031] like Figure 1 As shown, if the end effector 130 of the manipulator is within the field of view of the image acquisition device 110, the acquired positioning image may include an image of the end effector 130 of the manipulator. In some embodiments, the image acquisition device 110 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. Depending on the application scenario, the image acquisition module 110 may be an industrial camera, an underwater camera, a miniature electronic camera, an endoscope camera, etc. In some embodiments, the image acquisition module 110 may be fixed in position or have a variable position, for example, an industrial camera fixed at a monitoring position or an endoscope camera with adjustable position or attitude. In some embodiments, the image acquisition module 110 may realize at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image acquired, those skilled in the art can select different image acquisition devices as the image acquisition module 110.

[0032] In some embodiments, the control device 120 may receive a positioning image from the image acquisition device 110 and process the positioning image. For example, the control device 120 may identify multiple markers located on the manipulator 140 in the positioning image and determine the relative pose of the manipulator 140 or actuator 160 relative to a reference coordinate system (e.g., the world coordinate system). The control device 120 may also determine a drive signal for the manipulator 140 based on the current relative pose and target pose of the manipulator 140 or actuator 160. The drive signal may be sent to the drive unit 150 to perform motion control on the manipulator 140.

[0033] In some embodiments, the manipulator (e.g., Figure 1 The operating arm 140 or the end of the operating arm 130 shown is shown. Figure 13 The manipulator body 1320 or manipulator end 1310 shown has multiple markings distributed on it. In some embodiments, the multiple markings are disposed on the outer surface of the cylindrical portion of the manipulator. For example, the multiple markings are distributed circumferentially on the manipulator end 130. For example, the multiple markings are disposed on the outer surface of the cylindrical portion of the manipulator end 130. The multiple markings may include multiple pose markings for marking pose and multiple composite markings for marking pose and angle (e.g., about-axis angle or roll angle). In some embodiments, a positioning label (e.g., ...) is disposed on the outer surface of the cylindrical portion of the manipulator. Figure 2The label 200 shown may include multiple identifier patterns distributed circumferentially on the positioning label along the columnar portion, as well as multiple identifier pattern corner points within the identifier patterns. The multiple identifier patterns include multiple different composite identifier patterns and multiple pose identifier patterns, which may be identical. The composite identifier patterns and their corner points can be used to identify pose and angle, and the pose identifier patterns and their corner points can be used to identify pose. In some embodiments, the multiple different composite identifier patterns and the multiple pose identifier patterns are located in the same pattern distribution zone. In some embodiments, at least one composite identifier pattern is included among N consecutive identifier patterns in the multiple identifier patterns, where 2 ≤ N ≤ 4, and the composite identifier pattern in the N consecutive identifier patterns is different from the pose identifier pattern. For example, the multiple identifier patterns may be evenly distributed on the outer surface of the columnar portion, and the multiple composite identifier patterns may be evenly spaced within the multiple pose identifier patterns, such as inserting one composite identifier pattern every three pose identifier patterns. Figure 2 As shown.

[0034] In some embodiments, the identification pattern may be affixed to a label on the end of the operating arm, or printed on the end of the operating arm, or may be a pattern formed by the physical structure of the end of the operating arm itself, such as recesses or protrusions and combinations thereof. In some embodiments, the identification pattern may include a pattern formed by brightness, grayscale, color, etc. In some embodiments, the identification pattern may include a pattern that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provides information that can be detected by an image acquisition device. Those skilled in the art will understand that in some embodiments, the pose of the identification or the pose of the identification pattern may be represented by the pose of the corner coordinate system of the identification pattern. In some embodiments, the identification pattern is disposed on an area on the end of the operating arm suitable for image acquisition by an image acquisition device, such as an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily disturbed or obstructed during operation.

[0035] Figure 2 A schematic diagram of a label 200 including multiple identifiers according to some embodiments is shown. Figure 3 A schematic diagram is shown of a label 300 disposed on the periphery of the end of the operating arm and forming a cylindrical shape. It can be understood that, for simplicity, label 200 may include the same marking pattern as label 300.

[0036] See Figure 2 Multiple identifiers include multiple pose identifier patterns 210 and multiple pose identifier pattern corner points P therein. 210 And composite logo pattern 220 and the corner point R of the composite logo pattern therein. 220 In some embodiments, such as Figure 2As shown, multiple pose marker patterns 210 and composite marker patterns 220 are arranged in the same pattern distribution zone. In this disclosure, the corner points of the pose marker patterns are represented by the symbol "○", and the corner points of the composite marker patterns are represented by the symbol "△". In some embodiments, the pose marker pattern 210 or the corner point P of the pose marker pattern can be identified. 210 Determine the pose identifier by recognizing the composite identifier pattern 220 or the corner point R of the composite identifier pattern. 220 Determine the composite identifier.

[0037] See Figure 3 In the circumferential setting state, label 200 becomes label 300 with a spatially constructed cylindrical shape. In some embodiments, the axial angle or roll angle of each identifier can be represented by the axial angle of the identifier pattern or the corner point of the identifier pattern, wherein the identifier pattern includes a pose identifier pattern 310 and a composite identifier pattern 320. The axial angle of each identifier pattern or the corner point of the identifier pattern is known or predetermined. In some embodiments, the axial angle identified by each identifier can be determined based on the distribution of multiple identifiers (identifier patterns or corner points of identifier patterns). In some embodiments, multiple identifiers can be evenly distributed (e.g., the corner points of the identifier patterns in label 200 are evenly spaced, and the corner points of the identifier patterns in label 300 are evenly distributed angularly). In some embodiments, based on the distribution of multiple identifiers, each identifier can be used to identify a specific axial angle, and each identifier has a one-to-one correspondence with the identified axial angle. In this disclosure, the axial angle or roll angle refers to the angle around the Z-axis (e.g., the Z-axis of the manipulator coordinate system or the identifier coordinate system). In some embodiments, the manipulator is a deformable manipulator, and the Z-axis is along the tangential direction of the manipulator.

[0038] like Figure 3 As shown, multiple identification patterns in label 300 are evenly distributed circumferentially along the cylindrical structure, and the corner points of multiple identification patterns are evenly distributed on the cross-sectional circle 330. Therefore, the distribution angle (e.g., angle α0) of any adjacent corner points of the identification patterns is equal. Let the corner point P of the identification pattern pointed to by the X-axis be... 301 P 301 As a reference corner point for marking the 0° angle around the axis (corner point P of the marking pattern) 301 (Using the logo pattern as a reference pattern), then any corner point of the logo pattern and corner point P of the logo pattern can be used as a reference. 301 The positional relationship determines the angle around the axis of the corner mark of the logo pattern.

[0039] In some embodiments, the corner points of the identification pattern are located in a set coordinate system (e.g., Figure 3 The coordinate system shown is {wm0}≡[X] wm0 Y wm0 Z wm0 ] TThe angle around the axis indicated in the figure can be determined based on the following formula (1):

[0040] α m =α0(m-1) (1)

[0041] Where, α m To select the corner point of the logo pattern (e.g., corner point P of the logo pattern). 301 As the first corner point of the identification pattern, the angle around the axis of the m-th corner point of the identification pattern is in the clockwise direction of the cross-sectional circle 330.

[0042] In some embodiments, the multiple pose marker patterns can be the same pattern or different patterns. In some embodiments, the multiple composite marker patterns are different patterns, each composite marker pattern can be used to identify a specific angle around an axis, and each composite marker pattern has a one-to-one correspondence with the identified angle around an axis.

[0043] Figure 4 A schematic diagram illustrating an implementation scenario 400 according to some embodiments of the present disclosure is shown. Figure 4 As shown, the manipulator 440 includes an end effector 430 and a distal actuator 460. Multiple markers (e.g., pose marker pattern 410 and composite marker pattern 420) can be circumferentially disposed on the end effector 430. For example, as... Figure 2 The label 200 shown is circumferentially disposed on the end of the operating arm 430. Multiple marking pattern corner points are distributed on the cross-sectional circle 431 of the end of the operating arm 430. In some embodiments, based on the identified markings, a marking coordinate system {wm0}≡[X] is established. wm0 Y wm0 Z wm0 ] T The origin of the coordinate system {wm0} is the center of the cross-sectional circle 431, and the X-axis points from the origin to one of the corner points of the marker pattern (e.g., the corner point P corresponding to one of the identified pose markers). 401 The Z-axis is parallel to the axial direction of the operating arm 440, and the Y-axis is perpendicular to the XZ plane.

[0044] In some embodiments, an operator coordinate system {wm}≡[X] is established based on multiple composite identifiers. wm Y wm Z wm ] T The origin of the operator arm coordinate system {wm} is the center of the cross-sectional circle 431, and the X-axis points to the corner point R of the composite marking pattern. 401 The Z-axis is parallel to or coincides with the axial direction of the operating arm 440, and the Y-axis is perpendicular to the XZ plane. In some embodiments, the distribution of multiple composite marking patterns can be based on, for example, the remaining composite marking patterns and the corner points R of the composite marking patterns. 401The positional relationship of the corresponding composite logo patterns determines the axial angle of the corner markers of the composite logo patterns contained within the composite logo pattern.

[0045] This disclosure provides a method for controlling a manipulator through some embodiments. Figure 5 A flowchart illustrating a method 500 for controlling a manipulator according to some embodiments of the present disclosure is shown. Some or all of the steps in method 500 may be performed by a control device (e.g., control device 120) of a control system 100. Control device 120 may be configured on a computing device. Some or all of the steps in method 500 may be implemented by software, firmware, and / or hardware. In some embodiments, method 500 may be performed by a robotic system (e.g., Figure 16 The surgical robot system 1600 shown is executed. In some embodiments, method 500 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 The processor 1620 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0046] See Figure 5 In step 501, a positioning image is acquired. In some embodiments, the positioning image includes multiple markers on the manipulator. In some embodiments, the multiple markers include multiple pose markers for identifying pose and at least one composite marker for identifying pose and angle. In some embodiments, the markers can be obtained from, for example... Figure 1 The image acquisition device 110 shown receives a positioning image. For example, the control device 120 can receive a positioning image actively sent by the image acquisition device 110. Alternatively, the control device 120 can send an image request command to the image acquisition device 110, and the image acquisition device 110 responds to the image request command by sending a positioning image to the control device 120.

[0047] Continue reading Figure 5 In step 503, multiple markers located on the manipulator are identified in the positioning image. For example, an exemplary method for identifying multiple markers located on the manipulator may include, for instance... Figure 8 and Figure 10The method is illustrated. In some embodiments, the control device 120 can identify some or all of the markers in the positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm that can extract or recognize features of the markers. For example, the image processing algorithm may include a corner detection algorithm for detecting corner points of the marker pattern. The corner detection algorithm may be one of, but is not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the marker pattern. As another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the marker pattern. In some embodiments, the control device can identify some or all of the markers in the positioning image using a recognition model.

[0048] Continue reading Figure 5 In step 505, the current relative pose of the manipulator relative to the reference coordinate system is determined based on at least one composite identifier and multiple pose identifiers. In some embodiments, the pose of the manipulator coordinate system relative to the reference coordinate system can be determined based on the two-dimensional coordinates of the at least one composite identifier and multiple pose identifiers in the positioning image and their three-dimensional coordinates in the manipulator coordinate system, which serves as the current relative pose of the manipulator relative to the reference coordinate system.

[0049] Continue reading Figure 5 In step 507, the drive signal of the manipulator is determined based on the current relative pose and the target pose of the manipulator. In some embodiments, method 500 may further include determining the drive signal of the manipulator at predetermined intervals to achieve real-time control through multiple motion control cycles.

[0050] In some embodiments, method 500 may further include receiving a control command and determining the target pose of the manipulator based on the control command. In some embodiments, the target pose of the manipulator end effector in the world coordinate system may be input by a user via an input device. By comparison and calculation, the difference between the target pose and the current pose of the manipulator end effector can be determined. In some embodiments, the control command may be received based on a master-slave motion control method. For example, the target pose of the manipulator may be determined by acquiring the pose or joint information of the master manipulator in each motion control cycle. Real-time master-slave motion control can be performed through multiple motion control cycles.

[0051] In some embodiments, method 500 may further include determining the two-dimensional coordinates of a plurality of markers in a positioning image. In some embodiments, the coordinates of the markers can be represented by the coordinates of the corner points of the marker pattern. For example, the two-dimensional coordinates of the markers in the positioning image and their three-dimensional coordinates in the manipulator coordinate system can be represented by the coordinates of the corner points of the marker pattern. In some embodiments, determining the two-dimensional coordinates of a plurality of markers in the positioning image may include determining the two-dimensional coordinates of at least one composite marker and a plurality of pose markers in the positioning image. In some embodiments, method 500 may further include determining the three-dimensional coordinates of at least one composite marker and a plurality of pose markers in the manipulator coordinate system based on at least one composite marker.

[0052] In some embodiments, method 500 may further include determining the pose of the manipulator coordinate system relative to the reference coordinate system based on the two-dimensional coordinates of at least one composite marker corner point and multiple pose marker corner points in the positioning image, their three-dimensional coordinates in the manipulator coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system. In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system may be known. For example, the reference coordinate system may be the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system may be determined based on the camera's orientation. In other embodiments, the reference coordinate system may be the camera coordinate system itself, depending on actual needs. In some embodiments, based on the camera imaging principle and projection model, the pose of the manipulator coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of at least one composite marker corner point and multiple pose marker corner points in the positioning image, and their three-dimensional coordinates in the manipulator coordinate system. Based on the pose of the manipulator coordinate system relative to the camera coordinate system and the transformation relationship between the camera coordinate system and the reference coordinate system, the pose of the manipulator coordinate system relative to the reference coordinate system can be obtained.

[0053] In some embodiments, camera intrinsic parameters may also be considered. For example, camera intrinsic parameters may be as follows: Figure 1 The image acquisition device 110 shown has camera intrinsic parameters. These parameters can be known or obtained through calibration. In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the camera's optical center as the origin or a coordinate system established with the camera's lens center as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of that line).

[0054] In some embodiments, the pose of the manipulator coordinate system relative to a reference coordinate system (e.g., the world coordinate system) can be determined based on the following formula (2):

[0055] w R wm =w R lens lens R wm

[0056] w P wm = w R lens( lens R wm + lens P wm) + w P lens (2)

[0057] in, w R wm This refers to the orientation of the manipulator's coordinate system relative to the world coordinate system. w P wm This represents the position of the manipulator coordinate system relative to the world coordinate system. w R lens The pose of the camera coordinate system relative to the world coordinate system. w P lens This represents the position of the camera coordinate system relative to the world coordinate system. lens R wm The orientation of the manipulator coordinate system relative to the camera coordinate system. lens P wm This represents the position of the manipulator coordinate system relative to the camera coordinate system.

[0058] Figure 6 A flowchart illustrating a method 600 for determining the current relative pose of a manipulator relative to a reference coordinate system according to some embodiments of the present disclosure is provided. Some or all of the steps in method 600 may be performed by a control device (e.g., control device 120) of a control system 100. Control device 120 may be configured on a computing device. Some or all of the steps in method 600 may be implemented by software, firmware, and / or hardware. In some embodiments, method 600 may be performed by a robotic system (e.g., Figure 16 The surgical robot system 1600 shown is executed. In some embodiments, method 600 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 The processor 1620 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0059] See Figure 6 In step 601, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system are determined. In some embodiments, the three-dimensional coordinates of each marker pattern corner point in the marker coordinate system {wm0} can be determined based on the following formula (3):

[0060] C m =[r·cosα] m r·sinα m 0] T (3)

[0061] Among them, C m To use the selected corner point of the marker pattern as the first corner point of the marker pattern (e.g., pose marker pattern corner point P) 401 ), in the clockwise direction of cross-section circle 431, the three-dimensional coordinates of the m-th corner point of the marking pattern in the marking coordinate system, where r is the radius.

[0062] In some embodiments, the axial angle α of the m-th corner marker of the marker pattern is determined based on formula (1). m Then, based on the angle α around the axis determined by formula (1) m Formula (3) determines the three-dimensional coordinates C of the m-th corner point of the marker pattern in the marker coordinate system {wm0}. m .

[0063] See Figure 6 In step 603, based on at least one composite identifier, the roll angle of the identifier coordinate system relative to the operator arm coordinate system is determined. In some embodiments, a first axis angle in which one of the at least one composite identifier is identified in the operator arm coordinate system, and a second axis angle in which the composite identifier is identified in the identifier coordinate system, can be determined. Based on the first axis angle and the second axis angle, the roll angle of the identifier coordinate system relative to the operator arm coordinate system can be determined. In some embodiments, see... Figure 4 The roll angle Δα can refer to the rotation angle of the identifier coordinate system {wm0} relative to the operator arm coordinate system {wm} about the Z-axis. In some embodiments, the roll angle Δα can be determined based on the following formula (4):

[0064] Δα=α1-α2 (4)

[0065] Where α1 is the first angle around the axis, and α2 is the second angle around the axis. The first angle around the axis is the corner point of the composite logo pattern (e.g., the corner point R of the composite logo pattern). 402 The first angle around the axis is marked in the operator arm coordinate system, and the second angle around the axis is the corner point of the composite marking pattern (e.g., corner point R of the composite marking pattern). 402 The angle around the axis is indicated in the coordinate system.

[0066] In some embodiments, the X-axis of the identifier coordinate system {wm0} points to the corner point of the composite identifier pattern (e.g., the corner point R of the composite identifier pattern). 402Method 600 may further include determining a first about-axis angle in which the composite identifier is identified in the operator arm coordinate system as the roll angle of the identifier coordinate system relative to the operator arm coordinate system. In some embodiments, the first about-axis angle may be determined based on the pattern included in the composite identifier.

[0067] See Figure 6 In step 605, based on the roll angle of the marker coordinate system relative to the operator arm coordinate system and the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the operator arm coordinate system are determined. It can be understood that, given the roll angle of the marker coordinate system relative to the operator arm coordinate system, the three-dimensional coordinates of multiple marker pattern corner points (e.g., corner points of composite marker patterns and corner points of pose marker patterns) in the marker coordinate system can be transformed into three-dimensional coordinates in the operator arm coordinate system through coordinate transformation.

[0068] See Figure 6 In step 607, based on the two-dimensional coordinates of at least one composite identifier and multiple pose identifiers in the positioning image and their three-dimensional coordinates in the manipulator coordinate system, the pose of the manipulator coordinate system relative to the reference coordinate system is determined as the current relative pose of the manipulator relative to the reference coordinate system. In some embodiments, step 607 in method 600 can be implemented similarly to determining the pose of the manipulator coordinate system relative to the reference coordinate system in method 500.

[0069] Figure 7 A flowchart illustrating a method 700 for determining the current relative pose of an arm relative to a reference coordinate system according to other embodiments of this disclosure is shown. Method 700 may be... Figure 6 An alternative embodiment of method 600. Some or all of the steps in method 700 may be performed by a control device (e.g., control device 120) of the control system 100. Control device 120 may be configured on a computing device. Some or all of the steps in method 700 may be implemented by software, firmware, and / or hardware. In some embodiments, method 700 may be performed by a robot system (e.g., Figure 16 The surgical robot system 1600 shown is executed. In some embodiments, method 700 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 The processor 1620 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0070] See Figure 7In step 701, the pose of the marker coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of at least one composite marker and multiple pose markers in the positioning image and their three-dimensional coordinates in the marker coordinate system. In some embodiments, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system can be implemented similarly to step 601 in method 600.

[0071] See Figure 7 In step 703, based on at least one composite identifier, the roll angle of the identifier coordinate system relative to the operator arm coordinate system is determined. In some embodiments, determining the roll angle of the identifier coordinate system relative to the operator arm coordinate system can be implemented similarly to step 603 in method 600.

[0072] See Figure 7 In step 705, based on the roll angle of the identifier coordinate system relative to the operator arm coordinate system and the pose of the identifier coordinate system relative to the reference coordinate system, the pose of the operator arm coordinate system relative to the reference coordinate system is determined as the current relative pose of the operator arm relative to the reference coordinate system.

[0073] For example, taking the reference coordinate system as the world coordinate system, the pose of the manipulator coordinate system relative to the world coordinate system can be determined based on the following formula (5):

[0074] w R wm = w R wm0 ·rot z (Δα)

[0075] w P wm = w P wm0 (5)

[0076] in, w R wm This refers to the orientation of the manipulator's coordinate system relative to the world coordinate system. w P wm This represents the position of the manipulator coordinate system relative to the world coordinate system. w R wm0 To indicate the orientation of the coordinate system relative to the world coordinate system, w P wm0 To identify the position of the coordinate system relative to the world coordinate system, rot z (Δα) represents the roll angle Δα around the Z-axis of the manipulator coordinate system.

[0077] In some embodiments, method 500 further includes: determining the current relative pose of the end effector of the manipulator relative to the reference coordinate system based on the current relative pose of the manipulator relative to the reference coordinate system; and determining a drive signal for the manipulator based on the current relative pose of the end effector and the target pose. In some embodiments, the target pose of the end effector of the manipulator can be determined based on the target pose of the manipulator, and the drive signal for the manipulator can be determined based on the current relative pose of the end effector relative to the reference coordinate system and the target pose of the end effector. Alternatively, a control command input by a user can be received to determine the target pose of the end effector. For example, in each motion control cycle, the pose of the master manipulator or the joint information of the master manipulator is acquired to determine the target pose of the end effector. In some embodiments, the end effector (e.g., Figure 1 The actuator 160 shown is located at the end of the manipulator arm, therefore the position of the end effector is known or determinable. The pose transformation relationship of the end effector relative to the manipulator arm coordinate system is also known or predetermined. In some embodiments, taking the reference coordinate system as the world coordinate system as an example, the pose of the end effector of the manipulator arm relative to the reference coordinate system can be determined based on the following formula (6):

[0078] w R tip = w R wm wm R tip

[0079] w P tip = w R wm wm P tip + w P wm (6)

[0080] in, w R tip This refers to the attitude of the end effector relative to the world coordinate system. w P tip This refers to the position of the end effector relative to the world coordinate system. wm R tip The orientation of the end effector relative to the operator arm coordinate system. wm P tip This represents the position of the end effector relative to the manipulator's coordinate system.

[0081] In some embodiments, the orientation of the manipulator coordinate system relative to the world coordinate system is determined based on formula (5). w R wm and location w P wm Then the attitude is determined based on formula (5).w R wm and location w P wm Formula (6) determines the attitude of the end effector relative to the world coordinate system. w R tip and location w P tip .

[0082] Figure 8 A flowchart illustrating a method 800 for identifying an identifier according to some embodiments of the present disclosure is shown. Figure 8 As shown, some or all of the steps in method 800 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 16 The processor 1620 shown is used to execute the method. Some or all of the steps in method 800 can be implemented by software, firmware, and / or hardware. In some embodiments, method 800 can be executed by a robot system (e.g., Figure 16 The surgical robot system 1600 shown is executed. In some embodiments, method 800 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 The processor 1620 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0083] See Figure 8 In step 801, a plurality of candidate markers are determined from the positioning image. In some embodiments, the markers may include marker pattern corner points in a marker pattern. The coordinates or origin of the coordinate system of a candidate marker can be represented by a candidate marker pattern corner point. In some embodiments, a candidate marker pattern corner point may refer to a possible marker pattern corner point obtained after preliminary processing or preliminary identification of the positioning image.

[0084] In some embodiments, method 800 may include determining a region of interest (ROI) in a localization image. For example, the ROI may be cropped from the localization image, and multiple candidate identifiers may be determined from the ROI. The ROI may be the entire localization image or a partial region. For example, the ROI of the current frame may be cropped based on a region within a certain range of multiple identifier corner points determined in the previous frame image (e.g., the localization image of the previous image processing cycle). For localization images that are not the first frame, the ROI may be a region within a certain distance centered on a virtual point formed by the coordinates of multiple identifier corner points from the previous image processing cycle. The certain distance range may be a fixed multiple of the average spacing distance of the identifier corner points, such as twice. It should be understood that the predetermined multiple may also be a variable multiple of the average spacing distance of the multiple candidate identifier corner points in the previous image processing cycle.

[0085] In some embodiments, method 800 may include determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include: cropping a Region of Interest (ROI) from the localization image and converting the ROI to a corresponding grayscale image.

[0086] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be determined based on the following formula (7):

[0087]

[0088] Where τ is a set constant, for example, set to 2; I x I 45 I y I n45 These are the first derivatives of the pixel in the four directions: 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively.

[0089] In some embodiments, method 800 may include dividing the ROI into multiple sub-regions. For example, a non-maximum suppression method may be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI may be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the location image or ROI may also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels.

[0090] In some embodiments, method 800 may include determining the pixel with the largest corner likelihood value in each sub-region to form a pixel set. For example, the pixel with the largest CL value in each sub-image may be determined, and the pixel with the largest CL value in each sub-image may be compared with a first threshold to determine a set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold may be set to 0.06. It should be understood that the first threshold may also be set to other values.

[0091] See Figure 8 In step 803, a first identifier is identified from a plurality of candidate identifiers. In some embodiments, the first identifier is identified based on an identifier pattern matching template. In some embodiments, the identifier pattern matching template includes at least one pose identifier pattern matching template and multiple composite identifier pattern matching templates with different patterns. In some embodiments, the composite identifier is identified based on multiple composite identifier pattern matching templates with different patterns. For example, if the identifier patterns of pose identifiers are the same, the pose identifier pattern matching template can be matched with the candidate identifiers first. If the matching fails, multiple different composite identifier pattern matching templates can be matched with the candidate identifiers one by one until a match is successful.

[0092] In some embodiments, a marker pattern matching template is used to match the pattern at the corner of a candidate marker pattern to identify the first marker. For example, a candidate marker pattern corner that meets a preset pose pattern matching degree standard is determined as the first marker pattern corner. In some embodiments, the marker pattern matching template and the pattern in the vicinity of the marker pattern corner have the same or similar features. If the matching degree between the marker pattern matching template and the pattern in the vicinity of the candidate marker pattern corner reaches a preset pattern matching degree standard (e.g., the matching degree is higher than a threshold), it can be considered that the pattern in the vicinity of the candidate marker pattern corner has the same or similar features as the marker pattern matching template, and thus the current candidate marker pattern corner can be considered as the marker pattern corner.

[0093] In some embodiments, the pixel with the largest CL value in the pixel set is identified as a candidate identifier pattern corner point. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value can be selected as the candidate identifier pattern corner point. In some embodiments, after the candidate identifier pattern corner point is determined, an identifier pattern matching template is used to match the pattern at the candidate identifier pattern corner point. If a preset pattern matching degree standard is met, the candidate identifier pattern corner point is determined as the first identified identifier pattern corner point.

[0094] In some embodiments, method 800 may further include, in response to a matching failure, determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a candidate identifier pattern corner point. For example, if a candidate identifier pattern corner point does not meet a preset matching degree standard, a pixel with a secondary CL value (the pixel with the second largest CL value) is selected as a candidate identifier pattern corner point, and an identifier pattern matching template is used to match the pattern at the candidate identifier pattern corner point, and so on, until the first identifier pattern corner point is identified.

[0095] In some embodiments, the logo pattern can be a black and white checkerboard pattern, therefore the logo pattern matching template can be the same checkerboard pattern, utilizing the grayscale distribution G of the logo pattern matching template. M The pixel neighborhood grayscale distribution G of the pixel corresponding to the corner point of the candidate identifier pattern image The correlation coefficient (CC) between pixels is used for matching. The grayscale distribution G of the pixel neighborhood is also considered. image This refers to the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The correlation coefficient can be determined based on the following formula (8):

[0096]

[0097] Where Var() is the variance function and Cov() is the covariance function. In some embodiments, when the correlation coefficient is less than 0.8, the gray-level distribution in the pixel neighborhood has a low correlation with the identifier pattern matching template. In this case, the candidate identifier pattern corner with the highest likelihood value is determined not to be an identifier pattern corner; otherwise, the candidate identifier pattern corner with the highest likelihood value is considered to be an identifier pattern corner.

[0098] In some embodiments, method 800 may further include determining the edge direction of the corner points of the candidate identifier pattern. For example, as Figure 9 As shown, the corner point of the candidate pose identifier pattern is corner point P in pose identifier pattern 900. 901 Then the corner point P 901 The edge direction can refer to the direction of the corner point P. 901 The direction of the edge, such as Figure 9 The direction indicated by the dashed arrow.

[0099] In some embodiments, the edge direction can be determined by the first-order derivative (I0) of each pixel in the X and Y directions of the planar coordinate system with respect to a certain neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate identifier pattern. x and I y The edge direction can be determined based on the following formula (9):

[0100]

[0101] Among them, the first derivative (I) x and I y This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by performing a convolution operation on the edge direction I of each pixel within the neighborhood range. angle and the corresponding weight I weight Clustering calculations are performed to obtain the edge direction of the pixel, and weight I is selected. weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if multiple edge directions exist, then weight I is selected. weight The I corresponding to the largest proportion of multiple classes angle As the edge direction.

[0102] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0103] In some embodiments, method 800 may include rotating a marker pattern matching template based on its edge direction. Rotating the marker pattern matching template based on its edge direction can align the template with the image at the corner point of a candidate marker pattern. The edge direction of the corner point of the candidate marker pattern can be used to determine the orientation of the image at that corner point in the positioning image. In some embodiments, rotating the marker pattern matching template based on its edge direction can adjust it to have the same or nearly the same orientation as the image at the corner point of the candidate marker pattern to facilitate image matching.

[0104] See Figure 8 In step 805, starting with the first identifier, other identifiers are searched. In some embodiments, in response to the identification of a composite identifier, other identifiers are identified based on a pose identifier pattern matching template. In some embodiments, other identifiers include pose identifiers or composite identifiers.

[0105] Figure 10 A flowchart illustrating a method 1000 for searching for identifiers according to some embodiments of the present disclosure is shown. Figure 10As shown, some or all of the steps in method 1000 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 16 The processor 1620 shown is used to execute the method. Some or all of the steps in method 1000 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1000 can be executed by a robot system (e.g., Figure 16 The surgical robot system 1600 shown is executed. In some embodiments, method 1000 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 The processor 1620 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0106] See Figure 10 In step 1001, a second identifier is determined starting from the first identifier. In some embodiments, the corner point of the first identifier pattern is used as the starting point, and the corner point of the second identifier pattern is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following directions: directly in front of the corner point of the first identifier pattern (corresponding to a 0° angle direction), directly behind it (corresponding to a 120° angle direction), directly above it (90° angle direction), directly below it (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0107] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions, with each search direction v sn It can be determined based on the following formula (10):

[0108] v sn =[cos(n·π / 4) sin(n·π / 4)], (n=1,2,…,8) (10)

[0109] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent corner points of multiple marker patterns determined in the previous frame. For example, the predetermined search direction can be determined based on the following formula (11):

[0110]

[0111] Among them, (x j ,y j ) represents the two-dimensional coordinates of the corner points of multiple marker patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple marker patterns determined in the previous frame; v s1 The first set search direction; v s2 This is the second search direction set.

[0112] In some embodiments, such as Figure 11 As shown, the first identification pattern corner point P 1101 Using the coordinates of the location as the starting point, search for the corner point P of the second marker pattern in the set search direction. 1102 The coordinates of the location. For example, the corner point P of the first identifier pattern. 1101 Using the coordinates as the starting point for the search, the search box (e.g., ...) Figure 11 (The dashed box in the image) moves in the set search direction V with a certain search step size. 1101 Search for the corner points of the icon pattern.

[0113] In some embodiments, if there is at least one candidate identifier within the search box, the candidate identifier pattern corner point with the highest corner point likelihood value within the search box is preferentially selected as the second identifier pattern corner point P. 1102 With the search box limited to a suitable size, the first identifier pattern corner point P is used. 1101 The coordinates of the second marker P are used as the starting point for the search. 1102 During the search, the candidate icon with the highest corner likelihood value among the candidate icons appearing in the search box is more likely to be the corner of the icon pattern. Therefore, it can be considered that the candidate icon with the highest corner likelihood value in the search box is the corner of the second icon pattern, P. 1102 To improve data processing speed. In other embodiments, to improve the accuracy of corner point recognition of the marker pattern, when at least one candidate marker exists in the search box, the candidate marker pattern corner point with the highest corner point likelihood value among the candidate markers appearing in the search box is selected for corner point recognition to determine whether the candidate marker pattern corner point with the highest corner point likelihood value is a marker pattern corner point. For example, a pose marker pattern matching template or a composite marker pattern matching template can be used to match the image within a certain range of the candidate marker pattern corner point with the highest corner point likelihood value. The candidate marker pattern corner point that meets the preset pattern matching degree standard can be considered as the searched second marker pattern corner point P. 1102 .

[0114] In some embodiments, continue reading Figure 11 The size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0115] In some embodiments, the identification pattern can be a black and white graphic, and pattern matching can be performed based on the correlation coefficient in formula (8). If the correlation coefficient is greater than the threshold, the candidate identification pattern corner with the largest likelihood value is considered to be the identification pattern corner, and is denoted as the second identification pattern corner.

[0116] SeeFigure 10 In step 1003, a search direction is determined based on the first identifier and the second identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of the corner point of the first identifier pattern and moving away from the corner point of the second identifier pattern. The second search direction may be a direction starting from the coordinate position of the corner point of the second identifier pattern and moving away from the corner point of the first identifier pattern. For example, Figure 11 The search direction V shown 1102 .

[0117] In step 1005, starting with the first or second identifier, an identifier is searched in the search direction. In some embodiments, if the corner point of the first identifier pattern is used as the new starting point, the first search direction described above can be used as the search direction for the identifier pattern corner point. If the corner point of the second identifier pattern is used as the new starting point, the second search direction described above can be used as the search direction for the identifier pattern corner point. In some embodiments, a new identifier pattern corner point is searched (e.g., Figure 11 The third identifier pattern corner point P 1103 This can be performed similarly to step 1001. In some embodiments, the search step size can be the first identifier pattern corner point P. 1101 Second identification pattern corner point P 1102 The distance between them is L1.

[0118] In some embodiments, in response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; and an identifier pattern matching template is matched with the identifier pattern at the candidate identifier pattern corner point location to identify a first identifier. In some embodiments, after determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a new candidate identifier pattern corner point, a new first identifier can be identified based on a method similar to step 803. In some embodiments, a search distance greater than a search distance threshold can be understood as a search distance greater than a search distance threshold in some or all search directions. In some embodiments, the search distance threshold may include a set multiple of the distance between the (N-1)th pose identifier pattern corner point and the (N-2)th pose identifier pattern corner point, where N≥3. For example, the search distance threshold is twice the distance between the first two identifier pattern corner points. Thus, the maximum search distance for the third marker corner is twice the distance between the first and second marker corners. If no marker corner is found within this search distance in the search direction, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is identified as a new candidate pose marker corner, and a new first marker is identified. The current search process then stops. In some embodiments, similar to method 800, a new first marker corner can be determined, and similar to method 1000, the remaining marker corners can be searched starting from the new marker corner.

[0119] In some embodiments, in response to the number of identified markers being greater than or equal to a marker number threshold, the current relative pose of the manipulator relative to the reference coordinate system can be determined based on the identified markers, and the search for markers can be stopped accordingly. For example, in response to the number of identified marker pattern corner points being greater than or equal to the marker number threshold, the search for marker pattern corner points can be stopped. For example, when four marker pattern corner points are identified, the search for marker pattern corner points can be stopped.

[0120] In some embodiments, in response to the number of identified identifiers being less than an identifier number threshold, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; and an identifier pattern matching template is matched with the identifier pattern at the position of the candidate identifier pattern corner point to identify the first identifier. In some embodiments, if the total number of identified identifier pattern corner points is less than the identifier number threshold, the search based on the first identifier pattern in the above steps is considered to have failed. In some embodiments, if composite identifiers are not included among all identified identifiers, for example, if the identified identifier pattern corner points do not include composite identifier pattern corner points, the search based on the first identifier pattern in the above steps is considered to have failed. In some embodiments, in the case of search failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate identifier pattern corner point, and then a new first identifier can be identified based on a method similar to step 803. In some embodiments, similar to method 800, a new first identifier pattern corner point can be re-determined, and similar to method 1000, the remaining identifier pattern corner points can be searched starting from the new identifier pattern corner point.

[0121] In some embodiments, if the identified identifiers include composite identifiers, the type of the remaining identifiers found may be uncertain (it should be understood that identifier types include pose identifiers and composite identifiers). For example, if the first identifier is a composite identifier, it may be uncertain whether the second identifier is a pose identifier or a composite identifier.

[0122] In some embodiments, if the identified identifiers do not include composite identifiers, the type of the newly found identifier is determined. For example, if the first identifier is not a composite identifier, it is necessary to determine whether the second identifier is a pose identifier or a composite identifier. If neither the first nor the second identifier is a composite identifier, it is necessary to determine whether the third identifier is a pose identifier or a composite identifier, and so on.

[0123] In some embodiments, after the corner points of the marker pattern are searched or identified, sub-pixel positioning can be performed on the identified corner points of the marker pattern to improve the positional accuracy of the corner points of the marker pattern.

[0124] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of the corner points of the identifier pattern after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, the extreme points of which are subpixel points. The fitting function can be determined based on the following formulas (12) and (13):

[0125] S(x,y)=ax 2 +by 2 +cx+dy+exy+f (12)

[0126]

[0127] Where S(x, y) is the fitting function for the CL values ​​of all pixels in each ROI, and a, b, c, d, e, and f are coefficients; x c The x-coordinate and y-coordinate of the pose identifier c The y-coordinate is the pose identifier.

[0128] Figure 12 A schematic diagram of a segment 1200 of a manipulator according to some embodiments of the present disclosure is shown. The manipulator (e.g., manipulator 140) may include at least one deformable segment 1200. Figure 12 As shown, the deformable segment 1200 includes a fixed disk 1210 and multiple structural bones 1220. A first end of each structural bone 1220 is fixedly connected to the fixed disk 1210, and a second end is connected to a drive unit (not shown). In some embodiments, the fixed disk 1210 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and its cross-section may be circular, rectangular, polygonal, or various other shapes.

[0129] The driving unit causes the segment 1200 to deform by driving the structural bone 1220. For example, the driving unit causes the segment 1200 to be in a position such as... Figure 12 The bending state is shown. In some embodiments, the second end of the multiple structural bones 1220 passes through the base plate 1230 and is connected to the drive unit. In some embodiments, similar to the fixed plate 1210, the base plate 1230 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and the cross-section may be various shapes such as circular, rectangular, polygonal, etc. The drive unit may include a linear motion mechanism, a drive segment, or a combination of both. The linear motion mechanism may be connected to the structural bone 1220 to push or pull the structural bone 1220, thereby driving the segment 1200 to bend. The drive segment may include a fixed plate and multiple structural bones, wherein one end of the multiple structural bones is fixedly connected to the fixed plate. The other end of the multiple structural bones of the drive segment is connected to or integrally formed with the multiple structural bones 1220 to drive the bending of the segment 1200 by bending the drive segment.

[0130] In some embodiments, a spacer disk 1240 is further included between the fixed disk 1210 and the base disk 1230, through which multiple structural bones 1220 pass. Similarly, the drive segment may also include a spacer disk.

[0131] Figure 13 A schematic diagram of the structure of an operating arm 1300 according to some embodiments of the present disclosure is shown. For example... Figure 13As shown, the operating arm 1300 is a deformable operating arm, which may include an operating arm end cap 1310 and an operating arm body 1320. The operating arm body 1320 may include one or more segments, such as a first segment 1321 and a second segment 1322. In some embodiments, the structures of the first segment 1321 and the second segment 1322 may be similar to those of... Figure 12 The shown component 1200 is similar. In some implementations, such as... Figure 13 As shown, the main body 1320 of the operating arm also includes a first straight rod segment 1323 located between the first segment 1321 and the second segment 1322. A first end of the first straight rod segment 1323 is connected to the base plate of the second segment 1322, and a second end is connected to the fixed plate of the first segment 1321. In some embodiments, such as... Figure 13 As shown, the main body 1320 of the operating arm also includes a second straight rod section 1324, the first end of which is connected to the base plate of the first component 1321.

[0132] In some embodiments, method 500 may further include determining a pose difference based on the current relative pose of the manipulator and the target pose of the manipulator; and determining the drive signal of the manipulator based on the pose difference and the inverse kinematics model of the manipulator. For example, based on the difference between the target pose and the current pose of the manipulator end in the world coordinate system, the drive values ​​of multiple joints included in the manipulator in the current motion control cycle (or the drive values ​​of the corresponding multiple motors controlling the movement of the manipulator) can be determined by using an inverse kinematics numerical iterative algorithm of the manipulator kinematic model. It should be understood that the kinematic model can be a mathematical model representing the motion relationship between the joint space and the task space of the manipulator. For example, the kinematic model can be established by methods such as the Denavit-Hartenberg (DH) parameter method and the exponential product representation method.

[0133] In some embodiments, the target pose of the manipulator is the target pose of the manipulator in the world coordinate system. Method 500 may further include: determining the current pose of the manipulator in the world coordinate system based on the current relative pose; and determining the pose difference in the world coordinate system based on the target pose of the manipulator and the current pose of the manipulator in the world coordinate system. In some embodiments, the pose difference in the world coordinate system includes a position difference and an orientation difference.

[0134] In the k-th motion control cycle, the pose difference can be determined based on the following formula (14):

[0135]

[0136] in, Let $\frac{ ... P represents the angle difference of the manipulator during the k-th motion control cycle. tk R represents the target position of the manipulator during the k-th motion control cycle. t k The target posture of the manipulator during the k-th motion control cycle. R represents the current position of the manipulator during the k-th motion control cycle. t k For the k-th motion control cycle, the manipulator's... Current posture, express With R t k The corner between them.

[0137] In some embodiments, the target pose of the manipulator is updated at predetermined intervals before or during each motion control cycle. In some embodiments, multiple motion control cycles are executed iteratively. In each motion control cycle, methods according to some embodiments of this disclosure, such as steps 501-507, can be performed to control the manipulator to move to the target pose. By iteratively executing multiple motion control cycles, real-time closed-loop control of the manipulator end-effector pose can be achieved, thereby improving the pose control accuracy of the manipulator. It should be understood that implementing manipulator pose control through the methods of this disclosure can improve the trajectory tracking error of the manipulator (e.g., a continuous deformable arm).

[0138] Figure 14 A flowchart illustrating a method 1400 for determining a drive signal according to some embodiments of the present disclosure is shown. Figure 14 As shown, some or all of the steps in method 1400 can be executed by a control device (e.g., control device 120) of the manipulator control system 100. Control device 120 can be configured on a computing device. Method 1400 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1400 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., [processor name missing]). Figure 16 The processor 1620 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0139] See Figure 14 In step 1401, the Cartesian space velocity is determined based on the pose difference. In some embodiments, the Cartesian space velocity includes Cartesian space linear velocity and Cartesian space angular velocity. Method 1400 may further include: determining the Cartesian space linear velocity based on the position difference, and determining the Cartesian space angular velocity based on the pose difference. In some embodiments, the Cartesian space angular velocity may be determined based on the pose difference using a proportional-integral-derivative controller or a proportional-derivative controller. In some embodiments, the Cartesian space velocity of the k-th motion control cycle... It can be determined based on the following formula (15):

[0140]

[0141] Among them, V k Let ω be the Cartesian linear velocity of the k-th motion control cycle. k Let P be the Cartesian space angular velocity of the k-th motion control cycle. v D is the linear velocity proportionality coefficient. v P is the linear velocity differential coefficient. ω D is the angular velocity proportionality coefficient. ω These are the differential coefficients of angular velocity. The position difference of the manipulator during the (k-1)th motion control cycle. The angle difference of the manipulator during the (k-1)th motion control cycle.

[0142] See Figure 14 In step 1403, the joint parameter space velocity is determined based on the Cartesian space velocity. The joint parameter space velocity of the k-th motion control cycle... It can be determined based on the following formula (16):

[0143]

[0144] J + It is the Moore-Penrose pseudo-inverse of the velocity Jacobian matrix J of the kinematic model of the manipulator. The velocity Jacobian matrix J of the kinematic model of the manipulator can be determined based on the structure of the manipulator.

[0145] See Figure 14 In step 1405, the target joint parameters are determined based on the joint parameter space velocity and the current joint parameters. The target joint parameters for the k-th motion control cycle are... It can be determined based on the following formula (17):

[0146]

[0147] in, Let be the current joint parameters for the k-th motion control cycle, and Δt be the period of the motion control cycle.

[0148] It should be understood that when the manipulator has multiple components (e.g., Figure 13 The target joint parameters of the manipulator 1300 shown can be the target joint parameters of all the components, or the target joint parameters of one or some of the components.

[0149] See Figure 14In step 1407, a drive signal is determined based on the target joint parameters. For example, based on the mapping relationship between the target joint parameters and the drive amount, the drive amount of multiple joints included in the manipulator in the current motion control cycle can be determined, and then the drive signal of the drive unit (e.g., motor) can be determined based on the drive amount. In some embodiments, the mapping relationship between the joint parameters of a single component and the drive amount can be based on the mapping relationship shown in formula (26).

[0150] In some embodiments, the manipulator is taken as a deformable kinematic arm (e.g., a continuous deformable arm) as an example. A continuous deformable arm can be as follows: Figure 13 The manipulator arm 1300 is shown. (As shown...) Figure 13 As shown, each segment (first segment 1321 and second segment 1322) may include a base plate, a fixed plate, and multiple structural bones penetrating the base plate and the fixed plate. The multiple structural bones may be fixedly connected to the fixed plate and slidably connected to the base plate. The continuous deformable arm and its included segments can be described by a kinematic model. In some embodiments, the structure of each segment may be specifically as follows: Figure 12 The shown component is 1200. (As shown...) Figure 12 As shown, the base disk coordinate system The base disk is attached to the t-th (t = 1, 2, 3...) section of the continuum, with its origin located at the center of the base disk, and the XY plane coinciding with the plane of the base disk. Pointing from the center of the base plate to the first structural bone (the first structural bone can be understood as any one of multiple structural bones chosen as a reference). Curved plane coordinate system. Its origin coincides with the origin of the base disk coordinate system, and the XY plane coincides with the bending plane. and Coincident. Fixed disk coordinate system. The origin of the fixed disk is located at the center of the fixed disk, and the XY plane coincides with the plane of the fixed disk, attached to the t-th segment of the continuum. Pointing from the center of the fixed plate to the first structural bone. Curved plane coordinate system 2 Its origin is located at the center of the fixed disk, and the XY plane coincides with the bending plane. and coincide.

[0151] like Figure 12 The single segment 1200 shown can be represented by a kinematic model. The position of the end of segment t (fixed disk coordinate system {te}) relative to the base disk coordinate system {tb}. tb P te ,attitude tb R te It can be determined based on the following formulas (18) and (19):

[0152]

[0153] tb R te = tb R t1 t1 R t2 t2 R te (19)

[0154] Among them, L t For the t-th segment, construct a virtual structural bone (e.g., Figure 12 The length of the virtual structure bone 1221 shown in the figure, θ t In the t-th section, about or Rotate to Required rotation angle tb R t1 Let {t1} be the orientation of the bending plane coordinate system of segment t relative to the base disk coordinate system {tb}. t1 R t2 Let t be the orientation of the bending plane coordinate system 2{t2} of the t-th segment relative to the bending plane coordinate system 1{t1}. t2 R te Let {te} be the orientation of the fixed disk coordinate system {te} of the t-th segment relative to the curved plane coordinate system 2{t2}.

[0155] tb R t1 , t1 R t2 and t2 R te It can be determined based on the following formulas (20), (21), and (22):

[0156]

[0157]

[0158]

[0159] Where, δ t For the t-th segment, the bending plane and The included angle.

[0160] like Figure 12 The joint parameter Ψ of the single segment 1200 shown t It can be determined based on the following formula (23):

[0161] ψ t =[θ t δt ] T (twenty three)

[0162] Joint parameters Ψ t Joint parameter space velocity It can be determined based on the following formula (24):

[0163]

[0164] in, For θ t The first derivative, For δ t The first derivative.

[0165] like Figure 12 The Cartesian space velocity at the end of the single segment 1200 shown It can be determined based on the following formula (25):

[0166]

[0167] Among them, J t J is the velocity Jacobian matrix of the kinematic model of a single component. tv J is the linear velocity Jacobian matrix of the kinematic model of a single component. tω V is the angular velocity Jacobian matrix of the kinematic model of a single component. t It is the end linear velocity of a single segment, ω t It is the end angular velocity of a single segment.

[0168] In some embodiments, the driving amount of multiple bone structures has a known mapping relationship with joint parameters. Based on the target joint parameters of the segment and the mapping relationship, the driving amount of the multiple bone structures can be determined. The driving amount of the multiple bone structures can be understood as moving a single segment from its initial state (e.g., θ) t =0) The length of the structural bone subjected to push or tension when bent to the target bending angle. In some embodiments, the mapping relationship between the driving amount of multiple structural bones and joint parameters can be determined based on the following formula (26):

[0169] q i ≡-r ti θ t coS(δ t +β ti (26)

[0170] Where, r ti Let β be the distance from the i-th structural bone in the t-th segment to the virtual structural bone. ti Let q be the angle between the i-th structural bone and the first structural bone in the t-th segment. iLet be the driving quantity of the i-th structural bone. The driving signal of the driving unit can be determined based on the driving quantity of the i-th structural bone.

[0171] In some embodiments, the end Cartesian space velocity of a single segment can be determined based on formulas (14) and (15), the joint parameter space velocity of a single segment can be determined based on formula (16), the target joint parameter of a single segment can be determined based on formula (17), the driving amount of each structural bone can be determined based on formula (26), and then the driving signal of the driving unit (e.g., motor) can be determined based on the driving amount.

[0172] In some embodiments, the entire deformable arm can be described by a kinematic model. For example... Figure 13 As shown, transformations can be performed between multiple coordinate systems located at multiple positions on the deformable arm. For example, the end effector of the continuous deformable arm can be determined in the world coordinate system {w} based on the following formula (27):

[0173] W T tip = W T 1b 1b T 1e 1e T 2b 2b T 2e 2e T tip (27)

[0174] in, W T tip The homogeneous transformation matrix of the end effector of the deformable arm of the continuum relative to the world coordinate system; W T 1b Represents the homogeneous transformation matrix of the base disk of the first continuum segment relative to the world coordinate system; 1b T le Represents the homogeneous transformation matrix of the fixed disk of the first continuous segment relative to the base disk of the first continuous segment; 1e T 2b Represents the homogeneous transformation matrix of the base disk of the second continuous segment relative to the fixed disk of the first continuous segment; 2b T 2e Represents the homogeneous transformation matrix of the fixed disk of the second continuum segment relative to the base disk of the second continuum segment; 2e T tip This represents the homogeneous transformation matrix of the end effector of the continuum deformable arm relative to the fixed disk of the second continuum segment. In some embodiments, the end effector is fixedly mounted on the fixed disk, therefore... 2e T tip It is known or predetermined.

[0175] It should be understood that deformable arms have different joint parameters in different operating states. For example, Figure 13 The manipulator 1300 shown includes at least four operating states. The four operating states of the manipulator 1300 are as follows:

[0176] First working state: Only the second component 1322 participates in the actuator's pose control (e.g., only the second component 1322 enters the workspace), at which time the joint parameters of the manipulator 1300 can be determined based on the following formula (28):

[0177]

[0178] Where, ψ c1 These are the joint parameters of the manipulator 1300 in its first working state. Let L2, θ2, δ2 be the rotation angle of the manipulator 1300 around its axis, and let L2, θ2, δ2 be the angle of rotation of the manipulator 1300 around its axis. Figure 12 In the structure shown, L in section 1200 t θ t and δ t They have the same physical meaning.

[0179] Second working state: The second component 1322 and the first linear segment 1323 participate in the position control of the actuator (for example, the second component 1322 is fully in the workspace, and the first linear segment 1323 is partially in the workspace). At this time, the joint parameters of the manipulator 1300 can be determined based on the following formula (29):

[0180]

[0181] Where, ψ c2 For the joint parameters of the manipulator 1300 in the second working state, L r This is the feed amount for the first straight segment 1323.

[0182] Third working state: Second segment 1322, first straight segment 1323 and first segment 1321 participate in the position control of the actuator (for example, the second segment 1322 is fully in the workspace, the first straight segment 1323 is fully in the workspace, and the first segment 1321 is partially in the workspace). At this time, the joint parameters of the manipulator 1300 can be determined based on the following formula (30):

[0183]

[0184] Where, ψ c3 For the joint parameters of the manipulator 1300 in the third working state, θ1 and δ1 are as follows: Figure 12 In the structure shown, θ in section 1200 t and δ t They have the same physical meaning.

[0185] Fourth working state: Second segment 1322, first linear segment 1323, first segment 1321 and second linear segment 1324 participate in the actuator's pose control (for example, second segment 1322 is fully in the workspace, first linear segment 1323 is fully in the workspace, first segment 1321 is fully in the workspace, and second linear segment 1324 is partially in the workspace). At this time, the joint parameters of the manipulator 1300 can be determined based on the following formula (31):

[0186]

[0187] Where, ψ c4 For the joint parameters of the manipulator 1300 in the fourth working state, L s This is the feed rate for the second straight segment 1324.

[0188] In some embodiments, similar to a single segment, the end Cartesian space velocity of the deformable arm can be determined based on formulas (14) and (15), the joint parameter space velocity of the deformable arm can be determined based on formula (16), and the target joint parameter of the deformable arm can be determined based on formula (17). The specific parameters included in the joint parameter space velocity in formula (16) and the target joint parameter in formula (17) can be determined based on formulas (28), (29), (30) or (31). The driving amount of each structural bone of each segment can be determined based on formula (26), and then the driving signal of the driving unit (e.g., motor) can be determined based on the driving amount.

[0189] In some embodiments of this disclosure, a computer device is also provided, including a memory for storing at least one instruction; and a processor coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 10 and Figure 14 Some or all of the steps in the method disclosed herein.

[0190] Figure 15 A schematic diagram of a computer device 1500 according to some embodiments of the present disclosure is shown. See also Figure 15The computer device 1500 includes a central processing unit (CPU) 1501, a system memory 1504 including random access memory (RAM) 1502 and read-only memory (ROM) 1503, and a system bus 1505 connecting the various components. The computer device 1500 also includes input / output devices 1506 and a mass storage device 1507 for storing the operating system 1513, application programs 1514, and other program modules 1515. The input / output device 1506 includes an input / output controller 1510, primarily composed of a display 1508 and input devices 1509.

[0191] Mass storage device 1507 is connected to central processing unit 1501 via a mass storage controller (not shown) connected to system bus 1505. Mass storage device 1507 and its associated computer-readable media provide non-volatile storage for computer devices. That is, mass storage device 1507 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0192] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0193] Computer device 1500 can be connected to network 1512 via network interface unit 1511 connected to system bus 1505.

[0194] The system memory 1504 or mass storage device 1507 is also used to store one or more instructions. The central processing unit 1501 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0195] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the positioning method of some embodiments of this disclosure, such as... Figure 5 , Figure 6 ,Figure 7 , Figure 8 , Figure 10 and Figure 14 Some or all of the steps in the method disclosed herein.

[0196] In some embodiments, this disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory including a computer program (instructions) executable by a processor of a computer device to perform the methods shown in various embodiments of this application. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0197] Figure 16 A schematic diagram of a surgical robot system 1600 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [link to relevant documentation]. Figure 16 The surgical robot system 1600 may include a surgical tool 1650, an image acquisition unit 1610, and a processor 1620. The surgical tool 1650 may include a manipulator arm 1640, an actuator 1630 disposed at the distal end of the manipulator arm 1640, and at least one composite identifier and multiple pose identifiers disposed at the distal end of the manipulator arm 1640. The image acquisition unit 1610 can be used to acquire positioning images of the manipulator arm 1640. The processor 1620 is connected to the image acquisition unit 1610 and is used to execute some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 10 and Figure 14 Some or all of the steps in the method disclosed herein.

[0198] While specific embodiments of this disclosure have been illustrated and described by way of example, it will be apparent to those skilled in the art that many other changes and modifications can be made without departing from the spirit and scope of this disclosure. Therefore, all such changes and modifications falling within the scope of this disclosure are included in the appended claims.

Claims

1. A method for controlling a manipulator, comprising: Acquire the location image; In the positioning image, multiple markers located on the manipulator are identified, including multiple pose markers for identifying pose and at least one composite marker for identifying pose and angle. Based on the at least one composite identifier and the plurality of pose identifiers, the current relative pose of the manipulator relative to the reference coordinate system is determined; Based on the current relative pose and the target pose of the manipulator, determine the pose difference; as well as Based on the pose difference and the inverse kinematics model of the manipulator, the drive signal of the manipulator is determined.

2. The method according to claim 1, further comprising: Based on the at least one composite identifier, determine the three-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the manipulator coordinate system.

3. The method according to claim 2, further comprising: Based on the two-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the positioning image and the three-dimensional coordinates in the manipulator coordinate system, the pose of the manipulator coordinate system relative to the reference coordinate system is determined as the current relative pose of the manipulator relative to the reference coordinate system.

4. The method according to claim 1, further comprising: Determine the three-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the identifier coordinate system; Based on the at least one composite identifier, determine the roll angle of the identifier coordinate system relative to the operator arm coordinate system; Based on the roll angle of the marker coordinate system relative to the manipulator coordinate system and the three-dimensional coordinates of the at least one composite marker and the plurality of pose markers in the marker coordinate system, the three-dimensional coordinates of the at least one composite marker and the plurality of pose markers in the manipulator coordinate system are determined. as well as Based on the two-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the positioning image and the three-dimensional coordinates in the manipulator coordinate system, the pose of the manipulator coordinate system relative to the reference coordinate system is determined as the pose of the manipulator relative to the reference coordinate system.

5. The method according to claim 1, further comprising: Multiple candidate identifiers are determined from the location image; Identify a first identifier from the plurality of candidate identifiers; as well as Starting from the first identifier, search for other identifiers.

6. The method according to claim 5, further comprising: In response to the recognition of the composite identifier, other identifiers are identified based on the pose identifier pattern matching template.

7. The method according to claim 5, wherein the mark includes a mark pattern and mark pattern corner dots in the mark pattern, and the method further includes: Determine the region of interest in the positioning image; The region of interest is divided into multiple sub-regions; The pixel with the largest corner likelihood value in each sub-region is determined to form a pixel set; The pixel with the highest corner likelihood value among the multiple candidate identifiers is selected as the corner point of the candidate identifier pattern. as well as The identification pattern matching template is matched with the identification pattern at the corner position of the candidate identification pattern to identify the first identification.

8. The method according to claim 7, further comprising: In response to a matching failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is identified as a candidate identifier pattern corner.

9. The method according to claim 7, further comprising: Starting from the first identifier, search for the second identifier; Based on the first identifier and the second identifier, the search direction is determined; as well as Starting from the first identifier or the second identifier, search for identifiers in the search direction.

10. The method of claim 9, further comprising: In response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; as well as The identification pattern matching template is matched with the identification pattern at the corner position of the candidate identification pattern to identify the first identification.

11. The method of claim 9, further comprising: In response to the number of identified identifiers being greater than or equal to an identifier number threshold, the current relative pose of the manipulator relative to the reference coordinate system is determined based on the identified identifiers.

12. The method of claim 9, further comprising: In response to the fact that the number of identified identifiers is less than the identifier number threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as the candidate identifier pattern corner. as well as The identification pattern matching template is matched with the identification pattern at the corner position of the candidate identification pattern to identify the first identification.

13. The method according to claim 1, further comprising: Based on the current relative pose, determine the current pose of the manipulator in the world coordinate system; as well as Based on the target pose of the manipulator and the current pose of the manipulator in the world coordinate system, the pose difference in the world coordinate system is determined, and the target pose of the manipulator is the target pose of the manipulator in the world coordinate system.

14. The method of claim 13, comprising: Based on the pose difference, determine the Cartesian space velocity; Based on the Cartesian space velocity, determine the joint parameter space velocity; Based on the joint parameter space velocity and the current joint parameters, determine the target joint parameters; as well as The drive signal is determined based on the target joint parameters.

15. The method according to claim 14, wherein the pose difference in the world coordinate system includes position difference and attitude difference in the world coordinate system, and the Cartesian space velocity includes Cartesian space linear velocity and Cartesian space angular velocity, the method further comprising: Based on the position difference, the linear velocity in Cartesian space is determined; as well as Based on the attitude difference, the Cartesian space angular velocity is determined.

16. The method of claim 14, wherein the manipulator comprises: At least one component, the component including a fixing disk and multiple structural bones, a first end of the multiple structural bones being fixedly connected to the fixing disk, and a second end of the multiple structural bones being connected to a drive unit; The method further includes: Based on the target joint parameters, the driving force of the multiple structural bones is determined; and Based on the driving amount of the multiple structural bones, the driving signal for the driving unit is determined.

17. The method according to any one of claims 1-16, further comprising: Receive control commands; as well as Based on the control commands, the target pose of the manipulator is determined.

18. The method according to any one of claims 1-16, further comprising: The drive signal of the operating arm is determined at a predetermined period to achieve real-time control through multiple motion control cycles.

19. The method according to any one of claims 1-16, further comprising: Based on the current relative pose of the manipulator relative to the reference coordinate system, determine the current relative pose of the end effector of the manipulator relative to the reference coordinate system. as well as The drive signal of the manipulator is determined based on the current relative pose and target pose of the end effector of the manipulator.

20. The method according to any one of claims 1-16, wherein a positioning label is provided on the outer surface of the columnar portion of the operating arm, the positioning label comprising a plurality of marking patterns, the plurality of marking patterns comprising a plurality of different composite marking patterns and a plurality of pose marking patterns, the plurality of different composite marking patterns and the plurality of pose marking patterns being located in the same pattern distribution zone.

21. The method according to claim 20, wherein at least one composite identification pattern is included among the N consecutive identification patterns of the plurality of identification patterns, wherein the composite identification pattern is different from the pose identification pattern, and 2≤N≤4.

22. A computer device, comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory, is configured to execute the at least one instruction to perform the method as described in any one of claims 1-21.

23. A computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the method as described in any one of claims 1-21.

24. A surgical robot system, comprising: A surgical tool, the surgical tool comprising an operating arm, an actuator disposed at the distal end of the operating arm, and at least one composite marker and multiple pose markers disposed at the end of the operating arm; An image acquisition device is used to acquire positioning images of the operating arm; as well as A processor, connected to the image acquisition unit, is configured to perform the method as described in any one of claims 1-21 to determine the drive signal for the manipulator.

Images