Method for determining a pose of an object and surgical robot system

Abstract

The present disclosure relates to the technical field of positioning, and discloses a method for determining a pose of an object, a computer device, a computer readable storage medium and a surgical robot system. The method for determining the pose of the object comprises: acquiring a positioning image; in the positioning image, identifying a plurality of pose identifiers located on the object, the plurality of pose identifiers comprising different pose identifier patterns; and determining the pose of the object relative to a reference coordinate system based on the plurality of pose identifiers.

Description

Technical Field

[0001] This disclosure belongs to the field of positioning technology, and in particular relates to a method for determining the pose of an object and a surgical robot system. Background Technology

[0002] With the development of technology, it is becoming increasingly common for machines and equipment, controlled by humans or computers, to perform desired actions in order to assist or replace operators. For example, logistics robots are used to sort express packages, and surgical robots are used to assist doctors in performing surgeries.

[0003] In the above applications, it is necessary to determine the position and orientation of movable parts such as controlled devices or structures in order to control the machine equipment. Summary of the Invention

[0004] In some embodiments, this disclosure provides a method for determining the pose of an object, comprising: acquiring a positioning image; identifying multiple pose markers located on the object in the positioning image, the multiple pose markers including different pose marker patterns; and determining the pose of the object relative to a reference coordinate system based on the multiple pose markers.

[0005] In some embodiments, this disclosure provides a computer device, the computer device comprising: a memory for storing at least one instruction; and a processor coupled to the memory for executing at least one instruction to perform the method of this disclosure.

[0006] In some embodiments, this disclosure provides a computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the method of this disclosure.

[0007] In some embodiments, this disclosure provides a surgical robot system, including: a surgical instrument, the surgical instrument including a manipulator arm, an actuator disposed at the distal end of the manipulator arm, and a plurality of pose markers disposed on the distal end of the manipulator arm, the plurality of pose markers including different pose marker patterns; an image acquisition unit for acquiring positioning images of the manipulator arm; and a processor connected to the image acquisition unit for executing the method of this disclosure to determine the pose of the actuator. Attached Figure Description

[0008] Figure 1 A schematic diagram of a control system according to some embodiments of the present disclosure is shown;

[0009] Figure 2 A schematic diagram showing a label including multiple pose identifiers according to some embodiments of the present disclosure;

[0010] Figure 3A schematic diagram showing a label disposed on the periphery of the end of an operating arm and formed into a cylindrical shape according to some embodiments of the present disclosure;

[0011] Figure 4 A flowchart illustrating a method for determining the pose of an object according to some embodiments of the present disclosure;

[0012] Figure 5 A flowchart illustrating a method for determining the three-dimensional coordinates of a plurality of pose markers relative to an object coordinate system according to some embodiments of the present disclosure;

[0013] Figure 6 A flowchart illustrating a method for determining the three-dimensional coordinates of a plurality of pose markers relative to an object coordinate system according to other embodiments of the present disclosure;

[0014] Figure 7 A flowchart illustrating a method for identifying pose identifiers according to some embodiments of the present disclosure is shown.

[0015] Figure 8 A schematic diagram showing pose identification patterns according to some embodiments of the present disclosure;

[0016] Figure 9 A flowchart illustrating a method for searching pose identifiers according to some embodiments of the present disclosure;

[0017] Figure 10 A schematic diagram illustrating a search pose identifier according to some embodiments of the present disclosure;

[0018] Figure 11 A flowchart illustrating a method for searching a second pose identifier according to some embodiments of the present disclosure is shown;

[0019] Figure 12 A flowchart illustrating a method for searching pose identifiers according to some embodiments of the present disclosure;

[0020] Figure 13 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown;

[0021] Figure 14 A schematic diagram of a surgical robot system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0022] Exemplary embodiments of this disclosure are described below with reference to the accompanying drawings. Those skilled in the art will understand that the scope of this disclosure is not limited to these embodiments. Various modifications and variations can be made to this disclosure based on the following embodiments. All such modifications and variations are included within the scope of this disclosure.

[0023] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom can be described using variations in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "attitude" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and attitude of an object or part of an object, which can be described, for example, using six parameters from the six degrees of freedom mentioned above.

[0024] In this disclosure, a reference coordinate system can be understood as a coordinate system describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own perception coordinate system, etc. In some embodiments, the pose of the object coordinate system is used to represent the pose of the object, and the pose of the object coordinate system relative to the reference coordinate system can represent the pose of the object relative to the reference coordinate system. In some embodiments, the object can be understood as an object or target that needs to be positioned, such as a manipulator or the end effector of a manipulator or an actuator located at the distal end of the manipulator. The manipulator can be a rigid arm or a deformable arm (e.g., Figure 1 (The manipulator arm 140 shown).

[0025] In some embodiments, the method for determining the pose of an object disclosed herein can be applied to application scenarios that require obtaining the pose of an object. For example, during the execution of actions such as grasping, clamping, cutting, electrocoagulation, or suturing by the actuator of a surgical robot, in order to achieve precise control of the actuator, it is necessary to obtain the actual position of the actuator relative to the world coordinate system, and also to obtain the attitude of the actuator relative to the world coordinate system (e.g., including the roll angle, pitch angle, and yaw angle of the actuator). Specifically, the surgical robot may be a laparoscopic surgical robot, an orthopedic surgical robot, or a vascular interventional surgical robot, etc.

[0026] Figure 1 A schematic diagram of a control system 100 according to some embodiments of the present disclosure is shown. Figure 1 As shown, the object whose pose needs to be determined in the control system 100 may include a manipulator arm 140. The control system 100 may include an image acquisition device 110, at least one manipulator arm 140, and a control device 120. The image acquisition device 110 and the at least one manipulator arm 140 are communicatively connected to the control device 120. In some embodiments, such as Figure 1As shown, the control device 120 can be used to control the movement of at least one manipulator 140 to adjust the pose of the at least one manipulator 140, coordinate with each other, etc. In some embodiments, at least one manipulator 140 may include a manipulator end effector 130 at its distal end or end. The control device 120 can control the movement of at least one manipulator 140 to move the manipulator end effector 130 to a desired position and orientation. Those skilled in the art will understand that the control system 100 can be applied to surgical robot systems, such as laparoscopic surgical robot systems. For example, an actuator 160 may be disposed at the distal end of the manipulator end effector 130, such as... Figure 1 As shown. It should be understood that the control system 100 can also be applied to dedicated or general-purpose robot systems in other fields (e.g., manufacturing, machinery, etc.).

[0027] In this disclosure, the control device 120 can be communicatively connected to the drive unit 150 (e.g., a motor) of at least one manipulator 140 and send drive signals to the drive unit 150, thereby enabling the drive unit 150 to control at least one manipulator 140 to move to a corresponding target pose based on the drive signals. For example, the drive unit 150 controlling the movement of the manipulator 140 can be a servo motor, which can receive instructions from the control device to control the movement of the manipulator 140. The control device 120 can also be communicatively connected to a sensor coupled to the drive unit 150, for example, through a communication interface, to receive motion data of the manipulator 140 and monitor the motion status of the manipulator 140. In one example of this disclosure, the communication interface can be a CAN (Controller Area Network) bus communication interface, which enables the control device 120 to communicate with the drive unit 150 and the sensor via the CAN bus. In some embodiments, the control device 120 may include a local processor (e.g., a local computer device) or a cloud processor (e.g., a cloud server or cloud computing platform).

[0028] In some embodiments, the manipulator 140 may include a continuous deformable arm, such as a multi-degree-of-freedom manipulator composed of multiple joints, such as a manipulator capable of 6 degrees of freedom of motion.

[0029] In some embodiments, the image acquisition device 110 can be used to acquire positioning images. The positioning images may include part or all of the image of the manipulator arm 140. In some embodiments, the image acquisition device 110 can be used to acquire images of the manipulator arm end effector 130, which may have multiple different pose markers, including different pose marker patterns. For example, a positioning label 170 may be provided on the manipulator arm end effector 130 (the positioning label 170 may be, for example, a positioning tag 170). Figure 2The label 200 shown. The positioning label 170 may include multiple pose identifiers, each including different pose identifier patterns (detailed below). For example... Figure 1 As shown, if the end of the manipulator 130 is within the field of view of the image acquisition device 110, the acquired positioning image may include an image of the end of the manipulator 130.

[0030] In some embodiments, the control device 120 may receive a positioning image from the image acquisition device 110 and process the positioning image. For example, the control device 120 may identify multiple pose markers located on the manipulator 140 in the positioning image and determine the relative pose of the manipulator 140 or the actuator 160 relative to a reference coordinate system (e.g., the world coordinate system).

[0031] In some embodiments, the image acquisition device 110 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. Depending on the application scenario, the image acquisition module 110 may be an industrial camera, an underwater camera, a miniature electronic camera, an endoscope camera, etc. In some embodiments, the image acquisition module 110 may be fixed in position or have a variable position, for example, an industrial camera fixed at a monitoring position or an endoscope camera with adjustable position or orientation. In some embodiments, the image acquisition module 110 may realize at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image acquired, those skilled in the art can select different image acquisition devices as the image acquisition module 110.

[0032] In some embodiments, the object (e.g., Figure 1 The illustrated manipulator 140 or manipulator end cap 130 has a plurality of pose markers distributed on it. In some embodiments, the plurality of pose markers are disposed on the outer surface of the cylindrical portion of the object. For example, the plurality of pose markers are distributed circumferentially on the manipulator end cap 130. For example, the plurality of pose markers are disposed on the outer surface of the cylindrical portion of the manipulator end cap 130. In some embodiments, a positioning label including a plurality of pose markers is disposed on the outer surface of the cylindrical portion of the object, the plurality of pose markers including a plurality of different pose marker patterns distributed circumferentially on the positioning label along the cylindrical portion and pose marker pattern corner points in the pose marker patterns.

[0033] In some embodiments, the pose identifier may include a pose identifier pattern and pose identifier pattern corner points within the pose identifier pattern. In some embodiments, the pose identifier pattern may be disposed on a label on the end of the operating arm, or may be printed on the end of the operating arm, or may be a pattern formed by the physical structure of the end of the operating arm itself, for example, it may include recesses or protrusions and combinations thereof. In some embodiments, the pose identifier pattern may include a pattern formed with brightness, grayscale, color, etc. In some embodiments, the pose identifier pattern may include a pattern that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provides information that can be detected by an image acquisition device. Those skilled in the art will understand that in some embodiments, the pose of the pose identifier or the pose of the pose identifier pattern may be represented by the pose of the pose identifier pattern corner point coordinate system. In some embodiments, the pose identifier pattern is disposed on an area on the end of the operating arm suitable for image acquisition by an image acquisition device, for example, an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily disturbed or obstructed during operation.

[0034] Figure 2 A schematic diagram of a tag 200 including multiple pose identifiers according to some embodiments is shown. Figure 3 A schematic diagram is shown of a label 300 disposed on the periphery of the end of the manipulator and forming a cylindrical shape. It can be understood that, for simplicity, label 200 may include the same pose marking pattern as label 300.

[0035] See Figure 2 Multiple pose identifiers may include multiple different pose identifier patterns 210. Multiple pose identifiers may also include multiple pose identifier pattern corner points within the multiple different pose identifier patterns 210, which are represented by the symbol "○" in this disclosure. In some embodiments, pose identifiers can be determined by identifying the pose identifier pattern 210. See also... Figure 3In the circumferential setting state, label 200 becomes label 300 with a spatial structure of a cylindrical shape. In some embodiments, the axial angle or roll angle of the pose identifier can be represented by the axial angle of the pose identifier pattern or the corner point of the pose identifier pattern. The axial angle of each pose identifier pattern or corner point is known or predetermined. In some embodiments, the axial angle identified by each pose identifier can be determined based on the distribution of multiple pose identifiers (e.g., pose identifier patterns or corner points of pose identifier patterns). In some embodiments, the multiple pose identifiers can be uniformly distributed (e.g., the corner points of the pose identifier patterns in label 200 are evenly spaced, and the corner points of the pose identifier patterns in label 300 are evenly distributed). In other embodiments, the multiple pose identifiers can be non-uniformly distributed. In some embodiments, based on the distribution of multiple pose identifiers, each pose identifier pattern can be used to identify a specific axial angle, and each pose identifier pattern has a one-to-one correspondence with the identified axial angle. In this disclosure, the angle about the axis or the roll angle refers to the angle about the Z-axis (e.g., the Z-axis of the object coordinate system {wm}).

[0036] like Figure 3 As shown, multiple different pose marker patterns 310 in the label 300 are uniformly distributed circumferentially along the cylindrical structure. The corner points of multiple pose marker patterns are uniformly distributed on the cross-sectional circle 320 of the XY plane of the object coordinate system. Then, the distribution angle (e.g., angle α0) of any adjacent pose marker pattern corner points is equal. Set the pose marker pattern corner point P3 pointing to the X-axis. P3 is used as the reference corner point for marking the 0° angle around the axis (the pose marker pattern where the pose marker pattern corner point P3 is located is used as the reference pattern). Then, the angle around the axis of the pose marker pattern corner point can be determined according to the positional relationship between any pose marker pattern corner point and the pose marker pattern corner point P3. In some embodiments, the angle around the axis of the pose marker pattern corner point can be determined based on the following formula (1):

[0037] α m =α0(m-1) (1)

[0038] Where, α m Let P3 be the first pose marker corner point, and let the angle around the axis of the m-th pose marker corner point be in the clockwise direction of the cross-sectional circle 320.

[0039] This disclosure provides a method for determining the pose of an object through some embodiments. Figure 4A flowchart illustrating a method 400 for determining the pose of an object according to some embodiments of the present disclosure is shown. Some or all of the steps in method 400 may be performed by a control device (e.g., control device 120) of control system 100. Control device 120 may be configured on a computing device. Method 400 may be implemented by software, firmware, and / or hardware. In some embodiments, method 400 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 14 The processor 1420 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0040] See Figure 4 In step 401, a positioning image is acquired. In some embodiments, the positioning image contains multiple pose markers on the object. In some embodiments, these markers can be obtained from, for example... Figure 1 The image acquisition device 110 shown receives a positioning image. For example, the control device 120 can receive a positioning image actively sent by the image acquisition device 110. Alternatively, the control device 120 can send an image request command to the image acquisition device 110, and the image acquisition device 110 responds to the image request command by sending a positioning image to the control device 120.

[0041] Continue reading Figure 4 In step 403, multiple pose markers located on the object are identified in the positioning image, the multiple pose markers including different pose marker patterns. For example, an exemplary method for identifying multiple pose markers located on the object may include... Figure 7 and Figure 9 The method is illustrated. In some embodiments, the control device 120 can identify some or all of the pose markers in the positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm, which can extract or recognize features of the pose markers. For example, the image processing algorithm may include a corner detection algorithm for detecting corner points of the pose marker pattern. The corner detection algorithm may be one of, but is not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the pose marker pattern. As another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the pose marker pattern. In some embodiments, the control device can identify some or all of the pose markers in the positioning image using a recognition model.

[0042] Continue reading Figure 4In step 405, the pose of the object relative to a reference coordinate system is determined based on multiple pose markers. In some embodiments, method 400 further includes: determining the two-dimensional coordinates of the multiple pose markers in a positioning image; and determining the pose of the object relative to the reference coordinate system based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers relative to the object coordinate system. In some embodiments, the coordinates of the pose markers can be represented by the coordinates of the corner points of the pose marker pattern. For example, the two-dimensional coordinates of the pose markers in the positioning image and the three-dimensional coordinates in the object coordinate system can be represented by the coordinates of the corner points of the pose marker pattern. In some embodiments, the pose of the object coordinate system relative to the reference coordinate system can be determined based on the two-dimensional coordinates of the corner points of the multiple pose marker patterns in the positioning image and the three-dimensional coordinates of the corner points of the multiple pose marker patterns in the object coordinate system.

[0043] In some embodiments, method 400 further includes: determining the pose of the object coordinate system relative to the reference coordinate system based on the two-dimensional coordinates of the corner points of the multiple pose marker patterns in the positioning image, the three-dimensional coordinates of the corner points of the multiple pose marker patterns in the object coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system. In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system may be known. For example, the reference coordinate system is the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system can be determined according to the pose of the camera. In other embodiments, the reference coordinate system may also be the camera coordinate system itself, depending on actual needs. In some embodiments, based on the camera imaging principle and projection model, the pose of the object coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of the corner points of the multiple pose marker patterns in the positioning image and the three-dimensional coordinates of the corner points of the multiple pose marker patterns in the object coordinate system. Based on the pose of the object coordinate system relative to the camera coordinate system and the transformation relationship between the camera coordinate system and the reference coordinate system, the pose of the object coordinate system relative to the reference coordinate system can be obtained. In some embodiments, the intrinsic parameters of the camera may also be considered. For example, the intrinsic parameters of the camera may be as follows: Figure 1 The image acquisition device 110 shown has camera intrinsic parameters. These parameters can be known or obtained through calibration. In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the camera's optical center as the origin or a coordinate system established with the camera's lens center as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of that line).

[0044] In some embodiments, the pose of the object coordinate system {wm} relative to a reference coordinate system (e.g., the world coordinate system) can be determined based on the following formula (2):

[0045] w R wm = w R lens lens R wm

[0046] w P wm = w R lens ( lens R wm + lens P wm )+ w P lens (2)

[0047] in, w R wm The orientation of the object's coordinate system relative to the world coordinate system. w P wm This represents the position of the object's coordinate system relative to the world coordinate system. w R lens The pose of the camera coordinate system relative to the world coordinate system. w P lens This represents the position of the camera coordinate system relative to the world coordinate system. lens R wm The pose of the object's coordinate system relative to the camera's coordinate system. lens P wm This represents the position of the object's coordinate system relative to the camera's coordinate system.

[0048] This disclosure provides several embodiments of a method for determining the three-dimensional coordinates of a plurality of pose markers relative to an object coordinate system. In some embodiments, the three-dimensional coordinates of the plurality of pose markers relative to the object coordinate system are determined based on the distribution of the plurality of pose markers. For example, the three-dimensional coordinates of the corner points of the plurality of pose marker patterns in the object coordinate system are determined based on the distribution of the corner points of the plurality of pose marker patterns.

[0049] Figure 5 A flowchart illustrating a method 500 for determining the three-dimensional coordinates of a plurality of pose identifiers relative to an object coordinate system according to some embodiments of the present disclosure. Some or all of the steps in method 500 may be performed by a control device (e.g., control device 120) of control system 100. Control device 120 may be configured on a computing device. Method 500 may be implemented by software, firmware, and / or hardware. In some embodiments, method 500 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., [specific processor name missing]). Figure 14 The processor 1420 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0050] See Figure 5In step 501, based on the distribution of multiple pose markers, the axial angles of the multiple pose markers relative to the Z-axis of the object coordinate system are determined. In some embodiments, the axial angles of the multiple pose markers relative to the Z-axis of the object coordinate system can be determined based on multiple pose marker patterns. For example, each pose marker pattern can identify a specific axial angle, and different pose marker patterns correspond one-to-one with the identified axial angles. Based on the identification of the pose marker patterns and the correspondence between the pose marker patterns and the axial angles, the axial angles identified by the identified pose marker patterns can be determined. It should be understood that the distribution of each pose marker pattern is known or predetermined. In some embodiments, the distribution of multiple pose marker patterns or the corner points of multiple pose marker patterns can be as follows: Figure 3 The distribution is shown. In some embodiments, the angle around the axis of each pose marker corner marker can also be determined based on formula (1).

[0051] See Figure 5 In step 503, based on the about-axis angles of the multiple pose markers, the three-dimensional coordinates of the multiple pose markers relative to the object coordinate system are determined. In some embodiments, such as Figure 3 As shown, each pose marker corner point is located on the circumference of the cross-sectional circle 320, and the center and radius r of the cross-sectional circle 320 are known. Taking the pose marker corner point P3 as the reference corner point, the three-dimensional coordinates of the pose marker corner point P3 in the object coordinate system {wm} are (r,0,0). In some embodiments, the three-dimensional coordinates of each pose marker corner point in the object coordinate system {wm} can be determined based on the following formula (3):

[0052] C m =[r·cosα] m r·sinα m 0] T (3)

[0053] Among them, C m With pose marker corner point P3 as the first pose marker corner point, the specific angle around the axis of the m-th pose marker corner point can be based on the three-dimensional coordinates of multiple pose markers in the object coordinate system, following the clockwise direction of the cross-sectional circle 320.

[0054] In some embodiments, the axial angle α of the corner point marker of the m-th pose marker pattern is determined based on formula (1). m Then, based on the angle α around the axis determined by formula (1) m The three-dimensional coordinates C are determined by formula (3). m .

[0055] Figure 6A flowchart illustrating a method 600 for determining the three-dimensional coordinates of a plurality of pose identifiers relative to an object coordinate system according to other embodiments of the present disclosure. Method 600 may be an alternative embodiment of method 500. Some or all of the steps in method 600 may be performed by a control device (e.g., control device 120) of control system 100. Control device 120 may be configured on a computing device. Method 600 may be implemented by software, firmware, and / or hardware. In some embodiments, method 600 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 14 The processor 1420 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0056] See Figure 6 In step 601, the arrangement order of the multiple pose identifiers is determined based on at least two of them. In some embodiments, the arrangement order of the multiple pose identifiers can be represented by the arrangement order of multiple pose identifier patterns. In some embodiments, the arrangement order of the multiple pose identifiers is determined by identifying any two pose identifier patterns. It should be understood that the multiple pose identifiers include different pose identifier patterns, and when any two pose identifier patterns are known, the arrangement order can be determined based on the known distribution of the multiple pose identifier patterns (e.g., Figure 2 The distribution of different pose marker patterns in label 200 shown, or Figure 3 The distribution of different pose marker patterns in the label 300 shown determines the arrangement order of multiple pose markers in the positioning image, such as clockwise or counterclockwise arrangement.

[0057] See Figure 6In step 603, the three-dimensional coordinates of the multiple pose markers are determined based on their arrangement order. In some embodiments, based on the known distribution of the multiple pose markers, the three-dimensional coordinates of each pose marker in the object coordinate system can be determined. The three-dimensional coordinates of each pose marker can be represented by the three-dimensional coordinates of the corner points of the pose marker pattern in the object coordinate system, and each pose marker pattern corresponds to a coordinate point in the object coordinate system. After determining the arrangement order of the multiple pose marker patterns, the remaining pose marker patterns can be determined based on the identified pose marker patterns, and thus the three-dimensional coordinates of each pose marker pattern in the object coordinate system can be determined. In some embodiments, multiple pose marker corner points in the positioning image are identified, and any two corresponding pose marker patterns among the multiple pose marker corner points are determined. The arrangement order of the corner points of the multiple pose marker patterns is determined based on the two identified pose marker patterns, and thus the three-dimensional coordinates of each pose marker pattern corner point in the object coordinate system can be determined. Furthermore, based on the arrangement order, the distribution of all pose marker patterns can be determined, thereby matching a specific pose pattern matching template with the pose marker patterns at corresponding positions on the positioning image, improving data processing speed. In some embodiments, the pattern matching between the pose pattern matching template and the corner point of the pose identifier pattern can be implemented similarly to step 703 in method 700.

[0058] In some embodiments, method 400 further includes: determining the pose of the end effector of the object relative to the reference coordinate system based on the pose of the object relative to the reference coordinate system. In some embodiments, the end effector is located at the end of the object, so the position of the end effector is known or can be determined. The pose transformation relationship of the end effector relative to the object coordinate system is also known or predetermined. In some embodiments, taking the world coordinate system as the reference coordinate system as an example, the pose of the end effector of the object relative to the reference coordinate system can be determined based on the following formula (4):

[0059] w R tip = w R wm wm R tip

[0060] w P tip = w R wm wm P tip + w P wm (4)

[0061] in, w R tip This refers to the attitude of the end effector relative to the world coordinate system. w P tipThis refers to the position of the end effector relative to the world coordinate system. wm R tip The orientation of the end effector relative to the object's coordinate system. wm P tip This represents the position of the end effector relative to the object's coordinate system.

[0062] In some embodiments, the orientation of the object coordinate system relative to the world coordinate system is determined based on formula (2). w R wm and location w P wm Then the attitude is determined based on formula (2). w R wm and location w P wm Formula (4) determines the attitude of the end effector relative to the world coordinate system. w R tip and location w P tip .

[0063] This disclosure provides some embodiments of a method for identifying pose markers. Figure 7 A flowchart illustrating a method 700 for identifying pose identifiers according to some embodiments of the present disclosure is shown. Some or all of the steps in method 700 may be performed by a control device (e.g., control device 120) of control system 100. Control device 120 may be configured on a computing device. Method 700 may be implemented by software, firmware, and / or hardware. In some embodiments, method 700 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 14 The processor 1420 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0064] refer to Figure 7 In step 701, multiple candidate pose identifiers are determined from the positioning image. In some embodiments, the pose identifier may include a corner point of the pose identifier pattern in the pose identifier pattern. The coordinates or origin of the coordinate system of the candidate pose identifier can be represented by the corner point of the candidate pose identifier pattern. In some embodiments, the corner point of the candidate pose identifier pattern may refer to the possible corner point of the pose identifier pattern obtained after preliminary processing or preliminary identification of the positioning image.

[0065] In some embodiments, method 700 may include: determining a region of interest (ROI) in a localization image. For example, the ROI may be cropped from the localization image, and multiple candidate pose markers may be determined from the ROI. The ROI may be the entire localization image or a partial region. For example, the ROI of the current frame may be cropped based on a region within a certain range of corner points of multiple pose marker patterns determined in the previous frame (e.g., the localization image of the previous image processing cycle). For localization images that are not the first frame, the ROI may be a region within a certain distance centered on a virtual point formed by the coordinates of the corner points of multiple pose marker patterns from the previous image processing cycle. The certain distance range may be a fixed multiple of the average spacing distance of the corner points of the pose marker patterns, such as twice. It should be understood that the predetermined multiple may also be a variable multiple of the average spacing distance of the corner points of multiple candidate pose marker patterns in the previous image processing cycle.

[0066] In some embodiments, method 700 may include determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include: cropping a Region of Interest (ROI) from the localization image and converting the ROI to a corresponding grayscale image.

[0067] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be determined based on the following formula (5):

[0068] CL = max(c xy ,c 45 )

[0069] c xy =τ 2 ·|I xy |-1.5·τ·(|I 45 |+|I n45 |) (5)

[0070] c 45 =τ 2 ·|I 45_45|-1.5·τ·(|I x |+|I y |)

[0071] Where τ is a set constant, for example, set to 2; I x I 45 I y I n45 These are the first derivatives of the pixel in the four directions: 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively.

[0072] In some embodiments, method 700 may include dividing the ROI into multiple sub-regions. For example, a non-maximum suppression method may be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI may be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the location image or ROI may also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels.

[0073] In some embodiments, method 700 may include: determining the pixel with the largest corner likelihood value in each sub-region to form a pixel set. In some embodiments, the pixel set serves as a plurality of candidate identifiers determined from a localization image. For example, the pixel with the largest CL value in each sub-image may be determined, and the pixel with the largest CL value in each sub-image may be compared with a first threshold to determine a set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold may be set to 0.06. It should be understood that the first threshold may also be set to other values.

[0074] See Figure 7 Step 703: Based on multiple different pose pattern matching templates, identify the first pose identifier from the candidate pose identifiers. In some embodiments, the multiple different pose pattern matching templates are matched with the patterns at the corner points of the candidate pose identifier patterns to identify the first pose identifier. For example, the corner points of the candidate pose identifier patterns that meet a preset pose pattern matching degree standard are determined as the corner points of the first pose identifier pattern. In some embodiments, the pose pattern matching template and the pattern in the vicinity of the corner point of the pose identifier pattern have the same or similar features. If the matching degree between the pose pattern matching template and the pattern in the vicinity of the corner point of the candidate pose identifier pattern reaches a preset pose pattern matching degree standard (e.g., the matching degree is higher than a threshold), it can be considered that the pattern in the vicinity of the corner point of the candidate pose identifier pattern has the same or similar features as the pose pattern matching template, and thus the current corner point of the candidate pose identifier pattern can be considered as the corner point of the pose identifier pattern.

[0075] In some embodiments, the pixel with the largest CL value in the pixel set is identified as a candidate pose identifier pattern corner point. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value can be selected as the candidate pose identifier pattern corner point. In some embodiments, after determining the candidate pose identifier pattern corner point, a pose pattern matching template is matched with the pattern at the candidate pose identifier pattern corner point. If a preset pose pattern matching degree standard is met, the candidate pose identifier pattern corner point is determined as the first identified pose identifier pattern corner point.

[0076] In some embodiments, method 700 further includes: in response to a matching failure, determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a candidate pose identification pattern corner point. For example, if the candidate pose identification pattern corner point does not meet a preset matching degree standard, then the pixel with the second largest CL value (the pixel with the second largest CL value) is selected as the candidate pose identification pattern corner point, and the pose pattern matching template is matched with the pattern at the candidate pose identification pattern corner point, and so on, until the first pose identification pattern corner point is identified.

[0077] In some embodiments, the pose identification pattern can be a black and white alternating pattern (e.g., a checkerboard pattern), therefore the pose pattern matching template can be the same pattern, utilizing the grayscale distribution G of the pose pattern matching template. M The pixel neighborhood grayscale distribution G of the pixel corresponding to the corner point of the candidate pose identifier pattern image The correlation coefficient (CC) between pixels is used for matching. The grayscale distribution G of the pixel neighborhood is also considered. image This refers to the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The correlation coefficient can be determined based on the following formula (6):

[0078]

[0079] Where Var() is the variance function and Cov() is the covariance function. In some embodiments, when the correlation coefficient is less than 0.8, the gray-level distribution in the pixel neighborhood has a low correlation with the pose pattern matching template. In this case, the candidate pose pattern corner with the highest corner likelihood value is determined not to be a pose pattern corner. Otherwise, the candidate pose pattern corner with the highest corner likelihood value is considered to be a pose pattern corner.

[0080] In some embodiments, method 700 may include determining the edge orientation of corner points of candidate pose identifier patterns. For example, such as Figure 8As shown, the corner point of the candidate pose identifier pattern is corner point P8 in pose identifier pattern 800. The edge direction of corner point P8 can refer to the direction of the edge forming corner point P8, such as... Figure 8 The direction indicated by the dashed arrow.

[0081] In some embodiments, the edge direction can be determined by the first-order derivative (I0) of each pixel in the X and Y directions of the planar coordinate system with respect to a certain neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate pose identifier pattern. x and I y The direction of an edge can be determined using the following formula:

[0082]

[0083] Among them, the first derivative (I) x and I y This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by performing a convolution operation on the edge direction I of each pixel within the neighborhood range. angle and the corresponding weight I weight Clustering calculations are performed to obtain the edge direction of the pixel, and weight I is selected. weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if multiple edge directions exist, then weight I is selected. weight The I corresponding to the largest proportion of multiple classes angle As the edge direction.

[0084] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0085] In some embodiments, method 700 may include rotating a pose pattern matching template based on edge direction. Rotating the pose pattern matching template according to the edge direction allows the template to be aligned with the image at the corner point of a candidate pose marker pattern. The edge direction of the corner point of the candidate pose marker pattern can be used to determine the orientation of the image at that corner point in the positioning image. In some embodiments, rotating the pose pattern matching template according to the edge direction adjusts it to be the same as or nearly the same as the image orientation at the corner point of the candidate pose marker pattern to facilitate image matching.

[0086] See Figure 7 Step 705: Starting from the first pose identifier, search for pose identifiers. For example, Figure 9 A flowchart illustrating a method 900 for searching pose identifiers according to some embodiments of the present disclosure is shown. Figure 9 As shown, some or all of the steps in method 900 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 14 The processor 1420 shown is used to execute the method. Some or all of the steps in method 900 can be implemented by software, firmware, and / or hardware. In some embodiments, method 900 can be executed by a robot system (e.g., Figure 14 The surgical robot system 1400 shown is executed. In some embodiments, method 900 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 14 The processor 1420 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0087] See Figure 9 In step 901, the second pose identifier is searched starting from the first pose identifier. In some embodiments, the corner point of the first pose identifier pattern is used as the starting point, and the corner point of the second pose identifier pattern is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following directions: directly in front of the corner point of the first pose identifier pattern (corresponding to the 0° angle direction), directly behind (corresponding to the 120° angle direction), directly above (90° angle direction), directly below (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0088] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions, with each search direction v sn It can be determined based on the following formula (8):

[0089] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (8)

[0090] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent pose marker corner points among the multiple pose marker corner points determined in the previous frame. For example, the predetermined search direction is determined based on the following formula (9):

[0091]

[0092] Among them, (x j ,y j ) represents the two-dimensional coordinates of the corner points of multiple pose marker patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple pose marker patterns determined in the previous frame; v s1 The first set search direction; v s2 This is the second search direction set.

[0093] In some embodiments, such as Figure 10 As shown, the first pose is used to identify the corner point P of the pattern. 1001 Using the coordinates of the given location as the starting point, search for the corner point P of the second pose marker pattern in the set search direction. 1002 The coordinate position can specifically include: identifying the corner point P of the pattern using the first pose. 1001 Using the coordinates as the starting point for the search, through the search box (e.g., ...), Figure 10 The dashed box in the image (within the image) moves in the set search direction V with a certain search step size. 1001 Search for the corner points of the pose marker pattern. If there is at least one candidate corner point of the pose marker pattern within the search box, then the candidate corner point with the highest likelihood value within the search box is selected as the second pose marker pattern corner point P. 1002 With the search box limited to a suitable size, the first pose is used to identify the corner point P of the pattern. 1001 The coordinates of the point are used as the starting point for the second pose identification pattern corner point P. 1002 During the search, the candidate pose marker corner with the highest corner likelihood value among the candidate pose marker corners appearing in the search box is more likely to be the actual pose marker corner. Therefore, it can be considered that the candidate pose marker corner with the highest corner likelihood value in the search box is the second pose marker corner P. 1002To improve data processing speed. In other embodiments, to improve the accuracy of pose marker pattern corner point recognition, when at least one candidate pose marker pattern corner point exists in the search box, the candidate pose marker pattern corner point with the highest corner point likelihood value among the candidate pose marker pattern corner points appearing in the search box is selected for corner point recognition to determine whether the candidate pose marker pattern corner point with the highest corner point likelihood value is a pose marker pattern corner point. For example, the pose pattern matching template is matched with the image within a certain range of the candidate pose marker pattern corner point with the highest corner point likelihood value. The candidate pose marker pattern corner point that meets the preset pose pattern matching degree standard can be considered as the searched second pose marker pattern corner point P. 1002 .

[0094] In some embodiments, continue reading Figure 10 The size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0095] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, and pattern matching can be performed based on the correlation coefficient in formula (6). If the correlation coefficient is greater than the threshold, the candidate pose identification pattern corner with the largest corner likelihood value is considered to be the pose identification pattern corner, and is denoted as the second pose identification pattern corner.

[0096] Figure 11 A flowchart illustrating a method 1100 for searching a second pose identifier according to some embodiments of the present disclosure is shown. Figure 11 As shown, some or all of the steps in method 1100 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 14 The processor 1420 shown is used to execute the method. Some or all of the steps in method 1100 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1100 may be executed by a robot system (e.g., Figure 14 The surgical robot system 1400 shown is executed. In some embodiments, method 1100 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 14 The processor 1420 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium. In some embodiments, step 901 in method 900 may be implemented similarly to method 1100.

[0097] See Figure 11In step 1101, starting from the first pose identifier, candidate pose identifier pattern corner points for the second pose identifier are searched. In some embodiments, the search for candidate pose identifier pattern corner points for the second pose identifier can be combined with... Figure 10 The search is shown for the corner point P of the second pose identifier pattern. 1002 Similarly, implement it.

[0098] In step 1103, based on the distribution of multiple pose identifiers, a first pose pattern matching template and a second pose pattern matching template are determined. The first pose pattern matching template and the second pose pattern matching template correspond to pose identifiers adjacent to the first pose identifier. In some embodiments, step 1103 can be performed before or after step 1101, or step 1103 can be performed synchronously with step 1101. In some embodiments, the pose identifier patterns included in the pose identifiers included in the first pose identifier and the distribution of multiple pose identifier patterns can be used to determine the pose identifier patterns included in the pose identifiers adjacent to the first pose identifier, thereby determining the first pose pattern matching template and the second pose pattern matching template.

[0099] In step 1105, the first pose pattern matching template and / or the second pose pattern matching template are matched with the patterns at the corner positions of the candidate pose identifier patterns of the second pose identifier to identify the second pose identifier. In some embodiments, the first pose pattern matching template and / or the second pose pattern matching template can be matched with the patterns at the corner positions of the candidate pose identifier patterns of the second pose identifier based on the correlation coefficient in formula (6). If the correlation coefficient is greater than a threshold, the corner points of the candidate pose identifier patterns of the second pose identifier are determined as the corner points of the pose identifier patterns of the second pose identifier, and the patterns corresponding to the pose pattern matching templates (first pose pattern matching template or second pose pattern matching template) with a correlation coefficient greater than the threshold are determined as the pose identifier patterns of the second pose identifier.

[0100] See Figure 9 In step 903, a search direction is determined based on the first pose identifier and the second pose identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of a corner point of the first pose identifier pattern and moving away from the corner point of the second pose identifier pattern. The second search direction may be a direction starting from the coordinate position of a corner point of the second pose identifier pattern and moving away from the corner point of the first pose identifier pattern. For example, Figure 10 The search direction V shown 1002 .

[0101] In step 905, starting with either the first pose identifier or the second pose identifier, a search for pose identifiers is performed in the search direction. In some embodiments, if the first pose identifier pattern corner point is used as the new starting point, the first search direction described above can be used as the search direction for the pose identifier pattern corner point. If the second pose identifier pattern corner point is used as the new starting point, the second search direction described above can be used as the search direction for the pose identifier pattern corner point. In some embodiments, a new pose identifier pattern corner point is searched (e.g., Figure 10 The third pose marker pattern corner point P in 1003 This can be performed similarly to step 901. In some embodiments, the search step size can be the first pose identifier pattern corner point P. 1001 Second pose identifier pattern corner point P 1002 The distance between them is L1.

[0102] Figure 12 A flowchart illustrating a method 1200 for searching pose identifiers according to some embodiments of the present disclosure is shown. Figure 12 As shown, some or all of the steps in method 1200 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 14 The processor 1420 shown is used to execute the method. Some or all of the steps in method 1200 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1200 can be executed by a robot system (e.g., Figure 14 The surgical robot system 1400 shown is executed. In some embodiments, method 1200 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 14 The processor 1420 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium. In some embodiments, step 905 in method 900 may be implemented similarly to method 1200.

[0103] See Figure 12 In step 1201, starting from the first pose identifier or the second pose identifier, candidate pose identifier pattern corner points of the third pose identifier are searched. In some embodiments, the search for candidate pose identifier pattern corner points of the third pose identifier can be combined with... Figure 10 The search is shown for the corner point P of the third pose identifier pattern. 1003 Similarly, implement it.

[0104] In step 1203, a third pose pattern matching template is determined based on the distribution of multiple pose identifiers. The third pose pattern matching template corresponds to a pose identifier adjacent to the first pose identifier or adjacent to the second pose identifier. In some embodiments, the pose identifier pattern included in the pose identifier pattern included in the first pose identifier or the second pose identifier and the distribution of multiple pose identifier patterns can be used to determine the pose identifier pattern included in the pose identifier adjacent to the first pose identifier or the second pose identifier, thereby determining the third pose pattern matching template.

[0105] In step 1205, the third pose pattern matching template is matched with the pattern at the corner position of the candidate pose identifier pattern of the third pose identifier to identify the third pose identifier. In some embodiments, step 1205 can be implemented similarly to step 1105.

[0106] In some embodiments, in response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate pose identifier pattern corner point; and multiple different pose pattern matching templates are matched with the patterns at the corner point positions of the candidate pose identifier pattern to identify the first pose identifier. In some embodiments, after determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a new candidate pose identifier pattern corner point, a new first pose identifier can be identified based on a method similar to step 703. In some embodiments, a search distance greater than a search distance threshold can be understood as a search distance greater than a search distance threshold in some or all search directions. In some embodiments, the search distance threshold may include a set multiple of the distance between the (N-1)th pose identifier pattern corner point and the (N-2)th pose identifier pattern corner point, where N≥3. For example, the search distance threshold is twice the distance between the first two pose identifier pattern corner points. Thus, the maximum search distance for the third pose marker corner point is twice the distance between the first and second pose marker corner points. If a pose marker corner point is not found after reaching this search distance in the search direction, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate pose marker corner point, and a new first pose marker is identified. The current search process then stops accordingly. In some embodiments, similar to method 700, a new first pose marker corner point can be determined, and similar to method 900, the remaining pose marker corner points can be searched starting from the new pose marker corner point.

[0107] In some embodiments, in response to the number of identified pose marker pattern corner points being greater than or equal to a pose marker number threshold, the pose of the object relative to the reference coordinate system can be determined based on the search for the pose markers, and the search for pose marker pattern corner points will stop accordingly. For example, when four pose marker pattern corner points are identified, the search for pose marker pattern corner points is stopped.

[0108] In some embodiments, in response to the number of identified pose identifiers being less than a pose identifier number threshold, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate pose identifier pattern corner point; and multiple different pose pattern matching templates are matched with the patterns at the corner point positions of the candidate pose identifier patterns to identify the first pose identifier. In some embodiments, if the total number of identified pose identifiers (e.g., pose identifier pattern corner points) is less than a set pose identifier number threshold, the search based on the first pose identifier in the above steps is considered to have failed. In some embodiments, in the case of search failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate pose identifier pattern corner point, and then a new first pose identifier can be identified based on a method similar to step 703. In some embodiments, similar to method 700, a new first pose identifier pattern corner point can be re-determined, and similar to method 900, the remaining pose identifier pattern corner points can be searched starting from the new pose identifier pattern corner point.

[0109] In some embodiments, after the corner points of the pose marker pattern are searched or identified, sub-pixel positioning can be performed on the determined corner points of the pose marker pattern to improve the positional accuracy of the corner points of the pose marker pattern.

[0110] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of the corner points of the pose identifier pattern after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, the extreme points of which are subpixel points. The fitting function can be determined based on the following formulas (10) and (11):

[0111] S(x, y) = ax 2 +by 2 +cx+dy+exy+f (10)

[0112]

[0113] Where S(x, y) is the fitting function for the CL values ​​of all pixels in each ROI, and a, b, c, d, e, and f are coefficients; x c The x-coordinate and y-coordinate of the pose identifier c The y-coordinate is the pose identifier.

[0114] In some embodiments of this disclosure, a computer device is also provided, including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 4 , Figure 5, Figure 6 , Figure 7 , Figure 9 , Figure 11 and Figure 12 Some or all of the steps in the method disclosed herein.

[0115] Figure 13 A schematic block diagram of a computer device 1300 according to some embodiments of the present disclosure is shown. See also Figure 13 The computer device 1300 may include a central processing unit (CPU) 1301, a system memory 1304 including random access memory (RAM) 1302 and read-only memory (ROM) 1303, and a system bus 1305 connecting the various components. The computer device 1300 may also include an input / output system and a mass storage device 1307 for storing the operating system 1313, application programs 1314, and other program modules 1315. The input / output devices include an input / output controller 1310, primarily composed of a display 1308 and input devices 1309.

[0116] Mass storage device 1307 is connected to central processing unit 1301 via a mass storage controller (not shown) connected to system bus 1305. Mass storage device 1307 or computer-readable media provides non-volatile storage for computer devices. Mass storage device 1307 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0117] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0118] Computer device 1300 can be connected to network 1312 via network interface unit 1311 connected to system bus 1305.

[0119] The system memory 1304 or mass storage device 1307 is also used to store one or more instructions. The central processing unit 1301 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0120] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 9 , Figure 11 and Figure 12 Some or all of the steps in the disclosed method. Examples of computer-readable storage media include memory for computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0121] Figure 14 A schematic diagram of a surgical robot system 1400 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [link to relevant documentation]. Figure 14 The surgical robot system 1400 may include a surgical instrument 1450, an image acquisition unit 1410, and a processor 1420. The surgical instrument 1450 may include a manipulator arm 1440, an actuator 1430 disposed at the distal end of the manipulator arm 1440, and multiple pose markers disposed at the distal end of the manipulator arm 1440, the multiple pose markers including different pose marker patterns. The image acquisition unit 1410 can be used to acquire positioning images of the manipulator arm 1440. The processor 1420 is connected to the image acquisition unit 1410 and is used to execute some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 9 , Figure 11 and Figure 12 Some or all of the steps in the methods disclosed herein. In some embodiments, processor 1420 is used to perform some or all of the steps in the methods of some embodiments of this disclosure to determine the pose of actuator 1430.

[0122] While specific embodiments of this disclosure have been illustrated and described by way of example, it will be apparent to those skilled in the art that many other changes and modifications can be made without departing from the spirit and scope of this disclosure. Therefore, all such changes and modifications falling within the scope of this disclosure are included in the appended claims.

Claims

1. A method for determining the pose of an object, comprising: Acquire the location image; In the positioning image, multiple pose markers located on the object are identified, and the multiple pose markers include different pose marker patterns; as well as Based on the multiple pose identifiers, the pose of the object relative to the reference coordinate system is determined; The identification of multiple pose markers located on the object in the positioning image includes: Multiple candidate pose identifiers are determined from the positioning image; Based on multiple different pose pattern matching templates, identify the first pose identifier from the multiple candidate pose identifiers; and Starting from the first pose identifier, search for pose identifiers; The method further includes: Based on at least two of the plurality of pose identifiers, determine the arrangement order of the plurality of pose identifiers; and The three-dimensional coordinates of the multiple pose markers are determined based on their arrangement order.

2. The method according to claim 1, comprising: Based on the distribution of the multiple pose markers, the three-dimensional coordinates of the multiple pose markers relative to the object coordinate system are determined.

3. The method according to claim 2, comprising: Based on the distribution of the multiple pose markers, the axial angles of the multiple pose markers relative to the Z-axis of the object coordinate system are determined. as well as Based on the axial angles of the plurality of pose markers, the three-dimensional coordinates of the plurality of pose markers relative to the object coordinate system are determined.

4. The method according to claim 3, comprising: Based on the multiple pose marker patterns, the axial angles of the multiple pose markers relative to the Z-axis of the object coordinate system are determined.

5. The method according to any one of claims 2-4, comprising: Determine the two-dimensional coordinates of the plurality of pose markers in the positioning image; as well as Based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers relative to the object coordinate system, the pose of the object relative to the reference coordinate system is determined.

6. The method according to claim 1, wherein the pose identifier includes a corner point of the pose identifier pattern in the pose identifier pattern, the method comprising: Determine the region of interest in the localization image; The region of interest is divided into multiple sub-regions; The pixel with the largest corner likelihood value in each sub-region is determined to form a pixel set; The pixel with the largest corner likelihood value in the pixel set is selected as the candidate corner point of the pose identifier pattern; as well as The multiple different pose pattern matching templates are matched with the patterns at the corner positions of the candidate pose identifier pattern to identify the first pose identifier.

7. The method of claim 6, comprising: In response to a matching failure, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as the candidate pose identifier pattern corner.

8. The method of claim 6, comprising: Starting from the first pose identifier, search for the second pose identifier; Based on the first pose identifier and the second pose identifier, the search direction is determined; as well as Starting from the first pose identifier or the second pose identifier, search for pose identifiers in the search direction.

9. The method according to claim 8, wherein searching for a second pose identifier using the first pose identifier as a starting point comprises: Starting from the first pose identifier, search for the corner points of the candidate pose identifier pattern of the second pose identifier; Based on the distribution of the multiple pose identifiers, a first pose pattern matching template and a second pose pattern matching template are determined, and the first pose pattern matching template and the second pose pattern matching template correspond to the pose identifiers adjacent to the first pose identifier. as well as The first pose pattern matching template and / or the second pose pattern matching template are matched with the pattern at the corner position of the candidate pose identifier pattern of the second pose identifier to identify the second pose identifier.

10. The method according to claim 8, wherein searching for a pose identifier in the search direction using the first pose identifier or the second pose identifier as a starting point comprises: Starting from the first pose identifier or the second pose identifier, search for candidate pose identifier pattern corner points of the third pose identifier; Based on the distribution of the multiple pose identifiers, a third pose pattern matching template is determined, which corresponds to a pose identifier that is adjacent to the first pose identifier or the second pose identifier. as well as The third pose pattern matching template is matched with the pattern at the corner position of the candidate pose identifier pattern of the third pose identifier to identify the third pose identifier.

11. The method of claim 8, comprising: In response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate pose identifier pattern corner point; as well as The multiple different pose pattern matching templates are matched with the patterns at the corner positions of the candidate pose identifier patterns to identify the first pose identifier.

12. The method of claim 8, comprising: In response to the fact that the number of identified pose markers is less than the pose marker number threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as the candidate pose marker pattern corner point. as well as The multiple different pose pattern matching templates are matched with the patterns at the corner positions of the candidate pose identifier patterns to identify the first pose identifier.

13. The method of claim 8, comprising: In response to the number of identified pose identifiers being greater than or equal to a pose identifier number threshold, the pose of the object relative to the reference coordinate system is determined based on the identified pose identifiers.

14. The method according to any one of claims 1-4 and 6-7, comprising: Based on the pose of the object relative to the reference coordinate system, the pose of the end effector of the object relative to the reference coordinate system is determined.

15. The method according to any one of claims 1-4 and 6-7, wherein the plurality of pose markers are disposed on the outer surface of the columnar portion of the object.

16. The method according to any one of claims 1-4 and 6-7, wherein a positioning label including the plurality of pose identifiers is provided on the outer surface of the columnar portion of the object, the plurality of pose identifiers including a plurality of different pose identifier patterns distributed circumferentially along the columnar portion on the positioning label and pose identifier pattern corner points in the pose identifier patterns.

17. A computer device, the computer device comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory, is configured to execute the at least one instruction to perform the method as described in any one of claims 1-16.

18. A computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the method as described in any one of claims 1-16.

19. A surgical robot system, comprising: A surgical instrument, the surgical instrument comprising an operating arm, an actuator disposed at the distal end of the end of the operating arm, and a plurality of position markers disposed on the end of the operating arm, the plurality of position markers comprising different position marker patterns; An image acquisition device is used to acquire positioning images of the operating arm; as well as A processor, connected to the image acquisition unit, is configured to perform the method as described in any one of claims 1-16 to determine the pose of the actuator.

Images