Method for determining a pose of an object and surgical robot system

Abstract

This disclosure relates to the field of positioning technology, and discloses a method, computer device, computer-readable storage medium, and surgical robot system for determining the pose of an object. The method for determining the pose of an object includes: acquiring a positioning image; identifying multiple markers located on the object in the positioning image, the multiple markers including multiple pose markers for identifying pose and at least one composite marker for identifying pose and angle; and determining the pose of the object relative to a reference coordinate system based on at least one composite marker and the multiple pose markers.

Description

Technical Field

[0001] This disclosure belongs to the field of positioning technology, and in particular relates to a method for determining the pose of an object and a surgical robot system. Background Technology

[0002] With the development of technology, it is becoming increasingly common for machines and equipment, controlled by humans or computers, to perform desired actions in order to assist or replace operators. For example, logistics robots are used to sort express packages, and surgical robots are used to assist doctors in performing surgeries.

[0003] In the above applications, it is necessary to determine the position and orientation of movable parts such as controlled devices or structures in order to control the machine equipment. Summary of the Invention

[0004] In some embodiments, this disclosure provides a method for determining the pose of an object, comprising: acquiring a positioning image; identifying multiple markers located on the object in the positioning image, the multiple markers including multiple pose markers for identifying pose and at least one composite marker for identifying pose and angle; and determining the pose of the object relative to a reference coordinate system based on at least one composite marker and multiple pose markers.

[0005] In some embodiments, this disclosure provides a computer device, the computer device comprising: a memory for storing at least one instruction; and a processor coupled to the memory for executing at least one instruction to perform the method of this disclosure.

[0006] In some embodiments, this disclosure provides a computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the method of this disclosure.

[0007] In some embodiments, this disclosure provides a surgical robot system, comprising: a surgical tool including a manipulator arm, an actuator disposed at the distal end of the manipulator arm, and at least one composite identifier and multiple pose identifiers disposed at the distal end of the manipulator arm; an image acquisition unit for acquiring positioning images of the manipulator arm; and a processor connected to the image acquisition unit for executing the method of this disclosure to determine the pose of the actuator. Attached Figure Description

[0008] Figure 1 A schematic diagram of a control system according to some embodiments of the present disclosure is shown;

[0009] Figure 2 A schematic diagram illustrating a label including multiple identifiers according to some embodiments of the present disclosure;

[0010] Figure 3A schematic diagram showing a label disposed on the periphery of the end of an operating arm and formed into a cylindrical shape according to some embodiments of the present disclosure;

[0011] Figure 4 Schematic diagrams illustrating implementation scenarios according to some embodiments of the present disclosure;

[0012] Figure 5 A flowchart illustrating a method for determining the pose of an object according to some embodiments of the present disclosure;

[0013] Figure 6 A flowchart illustrating a method for determining the pose of an object relative to reference coordinates according to some embodiments of the present disclosure;

[0014] Figure 7 A flowchart illustrating a method for determining the pose of an object relative to reference coordinates according to other embodiments of the present disclosure;

[0015] Figure 8 A flowchart illustrating a method for identifying an identifier according to some embodiments of the present disclosure is shown;

[0016] Figure 9 A schematic diagram showing pose identification patterns according to some embodiments of the present disclosure;

[0017] Figure 10 A flowchart illustrating a method for searching identifiers according to some embodiments of the present disclosure is shown;

[0018] Figure 11 A schematic diagram illustrating search identifiers according to some embodiments of the present disclosure;

[0019] Figure 12 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown;

[0020] Figure 13 A schematic diagram of a surgical robot system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0021] Exemplary embodiments of this disclosure are described below with reference to the accompanying drawings. Those skilled in the art will understand that the scope of this disclosure is not limited to these embodiments. Various modifications and variations can be made to the embodiments described below. All such modifications and variations are included within the scope of this disclosure. Similar reference numerals indicate similar components among the various embodiments shown in the accompanying drawings.

[0022] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom can be described using variations in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "attitude" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and attitude of an object or part of an object, which can be described, for example, using six parameters from the six degrees of freedom mentioned above.

[0023] In this disclosure, a reference coordinate system can be understood as a coordinate system describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own perception coordinate system, etc. In some embodiments, the pose of the object coordinate system is used to represent the pose of the object, and the pose of the object coordinate system relative to the reference coordinate system can represent the pose of the object relative to the reference coordinate system. In some embodiments, the object can be understood as an object or target that needs to be positioned, such as a manipulator or the end effector of a manipulator or an actuator located at the distal end of the manipulator. The manipulator can be a rigid arm or a deformable arm (e.g., Figure 1 (The manipulator arm 140 shown).

[0024] In some embodiments, the method for determining the pose of an object disclosed herein can be applied to application scenarios that require obtaining the pose of an object. For example, during the execution of actions such as grasping, clamping, cutting, electrocoagulation, or suturing by the actuator of a surgical robot, in order to achieve precise control of the actuator, it is necessary to obtain the actual position of the actuator relative to the world coordinate system, and also to obtain the attitude of the actuator relative to the world coordinate system (e.g., including the roll angle, pitch angle, and yaw angle of the actuator). Specifically, the surgical robot may be a laparoscopic surgical robot, an orthopedic surgical robot, or a vascular interventional surgical robot, etc.

[0025] Figure 1 A schematic diagram of a control system 100 according to some embodiments of the present disclosure is shown. Figure 1 As shown, the object whose pose needs to be determined in the control system 100 may include a manipulator arm 140. The control system 100 may include an image acquisition device 110, at least one manipulator arm 140, and a control device 120. The image acquisition device 110 and the at least one manipulator arm 140 are communicatively connected to the control device 120. In some embodiments, such as Figure 1As shown, the control device 120 can be used to control the movement of at least one manipulator 140 to adjust the pose of the at least one manipulator 140, coordinate with each other, etc. In some embodiments, at least one manipulator 140 may include a manipulator end effector 130 at its distal end or end. The control device 120 can control the movement of at least one manipulator 140 to move the manipulator end effector 130 to a desired position and orientation. Those skilled in the art will understand that the control system 100 can be applied to surgical robot systems, such as laparoscopic surgical robot systems. For example, an actuator 160 may be disposed at the distal end of the manipulator end effector 130, such as... Figure 1 As shown. It should be understood that the control system 100 can also be applied to dedicated or general-purpose robot systems in other fields (e.g., manufacturing, machinery, etc.).

[0026] In this disclosure, the control device 120 can be communicatively connected to the drive unit 150 (e.g., a motor) of at least one manipulator 140 and send drive signals to the drive unit 150, thereby enabling the drive unit 150 to control at least one manipulator 140 to move to a corresponding target pose based on the drive signals. For example, the drive unit 150 controlling the movement of the manipulator 140 can be a servo motor, which can receive instructions from the control device to control the movement of the manipulator 140. The control device 120 can also be communicatively connected to a sensor coupled to the drive unit 150, for example, through a communication interface, to receive motion data of the manipulator 140 and monitor the motion status of the manipulator 140. In one example of this disclosure, the communication interface can be a CAN (Controller Area Network) bus communication interface, which enables the control device 120 to communicate with the drive unit 150 and the sensor via the CAN bus. In some embodiments, the control device 120 may include a local processor (e.g., a local computer device) or a cloud processor (e.g., a cloud server or cloud computing platform).

[0027] In some embodiments, the manipulator 140 may include a continuous deformable arm, such as a multi-degree-of-freedom manipulator composed of multiple joints, such as a manipulator capable of 6 degrees of freedom of motion.

[0028] In some embodiments, the image acquisition device 110 can be used to acquire positioning images. The positioning images may include part or all of the image of the manipulator arm 140. In some embodiments, the image acquisition device 110 can be used to acquire images of the manipulator arm end effector 130, which may have multiple different pose markers, including different pose marker patterns. For example, a positioning label 170 may be provided on the manipulator arm end effector 130 (the positioning label 170 may be, for example, a positioning tag 170). Figure 2(See label 200). The positioning label 170 may include multiple identifiers, including multiple pose identifiers for identifying pose and at least one composite identifier for identifying pose and angle (detailed below). Figure 1 As shown, if the end of the manipulator 130 is within the field of view of the image acquisition device 110, the acquired positioning image may include an image of the end of the manipulator 130.

[0029] In some embodiments, the control device 120 may receive a positioning image from the image acquisition device 110 and process the positioning image. For example, the control device 120 may identify multiple markers located on the manipulator 140 in the positioning image and determine the relative pose of the manipulator 140 or the actuator 160 relative to a reference coordinate system (e.g., the world coordinate system).

[0030] In some embodiments, the image acquisition device 110 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. Depending on the application scenario, the image acquisition module 110 may be an industrial camera, an underwater camera, a miniature electronic camera, an endoscope camera, etc. In some embodiments, the image acquisition module 110 may be fixed in position or have a variable position, for example, an industrial camera fixed at a monitoring position or an endoscope camera with adjustable position or orientation. In some embodiments, the image acquisition module 110 may realize at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image acquired, those skilled in the art can select different image acquisition devices as the image acquisition module 110.

[0031] In some embodiments, the object (e.g., Figure 1 The illustrated manipulator 140 or manipulator end cap 130 has a plurality of pose markers distributed on it. In some embodiments, the plurality of pose markers are disposed on the outer surface of the cylindrical portion of the object. For example, the plurality of pose markers are distributed circumferentially on the manipulator end cap 130. For example, the plurality of pose markers are disposed on the outer surface of the cylindrical portion of the manipulator end cap 130. The plurality of markers may include a plurality of pose markers for identifying pose and a plurality of composite markers for identifying pose and angle (e.g., about-axis angle or roll angle).

[0032] In some embodiments, a positioning label (e.g., ...) is provided on the outer surface of the columnar portion of the object. Figure 2The label 200 shown may include multiple pose markers, which may include multiple marker patterns distributed circumferentially on the positioning label along the columnar portion, and multiple marker pattern corner points within the marker patterns. The multiple marker patterns include multiple different composite marker patterns and multiple pose marker patterns, and the multiple pose marker patterns may be identical. The composite marker patterns and their corner points can be used to identify pose and angle, and the pose marker patterns and their corner points can be used to identify pose. In some embodiments, the multiple different composite marker patterns and the multiple pose marker patterns are located in the same pattern distribution zone. In some embodiments, at least one composite marker pattern is included among N consecutive marker patterns in the multiple marker patterns, where 2 ≤ N ≤ 4. For example, the multiple marker patterns may be evenly distributed on the outer surface of the columnar portion, and the multiple composite marker patterns may be evenly spaced within the multiple pose marker patterns, such as inserting one composite marker pattern every three pose marker patterns. Figure 2 As shown.

[0033] In some embodiments, the identification pattern may be affixed to a label on the end of the operating arm, or printed on the end of the operating arm, or may be a pattern formed by the physical structure of the end of the operating arm itself, such as recesses or protrusions and combinations thereof. In some embodiments, the identification pattern may include a pattern formed by brightness, grayscale, color, etc. In some embodiments, the identification pattern may include a pattern that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provides information that can be detected by an image acquisition device. Those skilled in the art will understand that in some embodiments, the pose of the identification or the pose of the identification pattern may be represented by the pose of the corner coordinate system of the identification pattern. In some embodiments, the identification pattern is disposed on an area on the end of the operating arm suitable for image acquisition by an image acquisition device, such as an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily disturbed or obstructed during operation.

[0034] Figure 2 A schematic diagram of a label 200 including multiple identifiers according to some embodiments is shown. Figure 3 A schematic diagram is shown of a label 300 disposed on the periphery of the end of the operating arm and forming a cylindrical shape. It can be understood that, for simplicity, label 200 may include the same marking pattern as label 300.

[0035] See Figure 2 The multiple identifiers include multiple pose identifier patterns 210 and multiple pose identifier pattern corner points therein, as well as a composite identifier pattern 220 and composite identifier pattern corner points therein. In some embodiments, such as Figure 2As shown, multiple pose marker patterns 210 and composite marker patterns 220 are arranged in the same pattern distribution zone. In this disclosure, the corner points of the pose marker patterns are represented by the symbol "○", and the corner points of the composite marker patterns are represented by the symbol "△". In some embodiments, the pose marker pattern 210 or the corner point P of the pose marker pattern can be identified. 210 Determine the pose identifier by recognizing the composite identifier pattern 220 or the corner point R of the composite identifier pattern. 220 Determine the composite identifier.

[0036] See Figure 3 In the circumferential setting state, label 200 becomes label 300 with a spatially constructed cylindrical shape. In some embodiments, the axial angle or roll angle of each identifier can be represented by the axial angle of the identifier pattern or the corner point of the identifier pattern, wherein the identifier pattern includes a pose identifier pattern 310 and a composite identifier pattern 320. The axial angle of each identifier pattern or the corner point of the identifier pattern is known or predetermined. In some embodiments, the axial angle identified by each identifier can be determined based on the distribution of multiple identifiers (identifier patterns or corner points of identifier patterns). In some embodiments, multiple identifiers can be evenly distributed (e.g., the corner points of the identifier patterns in label 200 are evenly spaced, and the corner points of the identifier patterns in label 300 are evenly distributed angularly). In some embodiments, based on the distribution of multiple identifiers, each identifier can be used to identify a specific axial angle, and each identifier has a one-to-one correspondence with the identified axial angle. In this disclosure, the axial angle or roll angle refers to the angle around the Z-axis (e.g., the Z-axis of the object coordinate system or the identifier coordinate system).

[0037] like Figure 3 As shown, multiple identification patterns in label 300 are evenly distributed circumferentially along the cylindrical structure, and the corner points of multiple identification patterns are evenly distributed on the cross-sectional circle 330. Therefore, the distribution angle (e.g., angle α0) of any adjacent corner points of the identification patterns is equal. Let the corner point P of the identification pattern pointed to by the X-axis be... 301 P 301 As a reference corner point for marking the 0° angle around the axis (corner point P of the marking pattern) 301 (Using the logo pattern as a reference pattern), then any corner point of the logo pattern and corner point P of the logo pattern can be used as a reference. 301 The positional relationship determines the angle around the axis of the corner mark of the logo pattern.

[0038] In some embodiments, the corner points of the identification pattern are located in a set coordinate system (e.g., Figure 3 The coordinate system shown is {wm0}≡[X] wm0 Y wm0 Z wm0 ] T The angle about the axis indicated in the figure can be determined based on the following formula (1):

[0039] α m =α0(m-1) (1)

[0040] Where, α m To select the corner point of the logo pattern (e.g., corner point P of the logo pattern). 301 As the first corner point of the identification pattern, the angle around the axis of the m-th corner point of the identification pattern is in the clockwise direction of the cross-sectional circle 330.

[0041] In some embodiments, the multiple pose marker patterns can be the same pattern or different patterns. In some embodiments, the multiple composite marker patterns are different patterns, each composite marker pattern can be used to identify a specific angle around an axis, and each composite marker pattern has a one-to-one correspondence with the identified angle around an axis.

[0042] Figure 4 A schematic diagram illustrating an implementation scenario 400 according to some embodiments of the present disclosure is shown. Figure 4 As shown, the manipulator 440 includes an end effector 430 and a distal actuator 460. Multiple markers (e.g., pose marker pattern 410 and composite marker pattern 420) can be circumferentially disposed on the end effector 430. For example, as... Figure 2 The label 200 shown is circumferentially disposed on the end of the operating arm 430. Multiple marking pattern corner points are distributed on the cross-sectional circle 431 of the end of the operating arm 430. In some embodiments, based on the identified markings, a marking coordinate system {wm0}≡[X] is established. wm0 Y wm0 Z wm0 ] T The origin of the coordinate system {wm0} is the center of the cross-sectional circle 431, and the X-axis points from the origin to one of the corner points of the marker pattern (e.g., the corner point P corresponding to one of the identified pose markers). 401 The Z-axis is parallel to the axis of the object (e.g., the manipulator 440), and the Y-axis is perpendicular to the XZ plane.

[0043] In some embodiments, an object coordinate system {wm}≡[X] is established based on multiple composite identifiers. wm Y wm Z wm ] T The origin of the object coordinate system {wm} is the center of the cross-sectional circle 431, and the X-axis points to the corner point R of the composite logo pattern. 401 The Z-axis is parallel to or coincides with the axis of the object (e.g., the manipulator 440), and the Y-axis is perpendicular to the XZ plane. In some embodiments, the distribution of multiple composite marking patterns can be based on, for example, the remaining composite marking patterns and the corner points R of the composite marking patterns. 401The positional relationship of the corresponding composite logo patterns determines the axial angle of the corner markers of the composite logo patterns contained within the composite logo pattern.

[0044] This disclosure provides a method for determining the pose of an object through some embodiments. Figure 5 A flowchart illustrating a method 500 for determining the pose of an object according to some embodiments of the present disclosure is shown. Some or all of the steps in method 500 may be performed by a control device (e.g., control device 120) of a control system 100. Control device 120 may be configured on a computing device. Some or all of the steps in method 500 may be implemented by software, firmware, and / or hardware. In some embodiments, method 500 may be performed by a robotic system (e.g., Figure 13 The surgical robot system 1300 shown is executed. In some embodiments, method 500 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 13 The processor 1320 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0045] See Figure 5 In step 501, a positioning image is acquired. In some embodiments, the positioning image includes multiple markers on the object. In some embodiments, the multiple markers include multiple pose markers for identifying pose and at least one composite marker for identifying pose and angle. In some embodiments, the markers can be obtained from, for example... Figure 1 The image acquisition device 110 shown receives a positioning image. For example, the control device 120 can receive a positioning image actively sent by the image acquisition device 110. Alternatively, the control device 120 can send an image request command to the image acquisition device 110, and the image acquisition device 110 responds to the image request command by sending a positioning image to the control device 120.

[0046] Continue reading Figure 5 In step 503, multiple identifiers located on the object are identified in the positioning image. For example, exemplary methods for identifying multiple identifiers located on an object may include... Figure 8 and Figure 10The method is illustrated. In some embodiments, the control device 120 can identify some or all of the markers in the positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm that can extract or recognize features of the markers. For example, the image processing algorithm may include a corner detection algorithm for detecting corner points of the marker pattern. The corner detection algorithm may be one of, but is not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the marker pattern. As another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the marker pattern. In some embodiments, the control device can identify some or all of the markers in the positioning image using a recognition model.

[0047] Continue reading Figure 5 In step 505, the pose of the object relative to the reference coordinate system is determined based on at least one composite identifier and multiple pose identifiers. In some embodiments, the pose of the object coordinate system relative to the reference coordinate system can be determined based on the two-dimensional coordinates of at least one composite identifier and multiple pose identifiers in the positioning image and their three-dimensional coordinates in the object coordinate system, which serves as the pose of the object relative to the reference coordinate system.

[0048] In some embodiments, method 500 may further include determining the two-dimensional coordinates of a plurality of markers in a positioning image. In some embodiments, the coordinates of the markers can be represented by the coordinates of the corner points of the marker pattern. For example, the two-dimensional coordinates of the markers in the positioning image and their three-dimensional coordinates in the object coordinate system can be represented by the coordinates of the corner points of the marker pattern. In some embodiments, determining the two-dimensional coordinates of a plurality of markers in the positioning image may include determining the two-dimensional coordinates of at least one composite marker and a plurality of pose markers in the positioning image. In some embodiments, method 500 may further include determining the three-dimensional coordinates of at least one composite marker and a plurality of pose markers in the object coordinate system based on at least one composite marker.

[0049] In some embodiments, method 500 may further include determining the pose of the object coordinate system relative to the reference coordinate system based on the two-dimensional coordinates of at least one composite marker corner point and multiple pose marker corner points in the positioning image, their three-dimensional coordinates in the object coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system. In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system may be known. For example, the reference coordinate system may be the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system may be determined based on the camera's orientation. In other embodiments, the reference coordinate system may be the camera coordinate system itself, depending on actual needs. In some embodiments, based on the camera imaging principle and projection model, the pose of the object coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of at least one composite marker corner point and multiple pose marker corner points in the positioning image, their three-dimensional coordinates in the object coordinate system, and their three-dimensional coordinates in the object coordinate system. Based on the pose of the object coordinate system relative to the camera coordinate system and the transformation relationship between the camera coordinate system and the reference coordinate system, the pose of the object coordinate system relative to the reference coordinate system can be obtained.

[0050] In some embodiments, camera intrinsic parameters may also be considered. For example, camera intrinsic parameters may be as follows: Figure 1 The image acquisition device 110 shown has camera intrinsic parameters. These parameters can be known or obtained through calibration. In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the camera's optical center as the origin or a coordinate system established with the camera's lens center as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of that line).

[0051] In some embodiments, the pose of the object coordinate system relative to a reference coordinate system (e.g., the world coordinate system) can be determined based on the following formula (2):

[0052]

[0053] in, w R wm The orientation of the object's coordinate system relative to the world coordinate system. w P wm This represents the position of the object's coordinate system relative to the world coordinate system. w R lens The pose of the camera coordinate system relative to the world coordinate system. w P lens This represents the position of the camera coordinate system relative to the world coordinate system. lens R wm The pose of the object's coordinate system relative to the camera's coordinate system. lens P wmThis represents the position of the object's coordinate system relative to the camera's coordinate system.

[0054] Figure 6 A flowchart illustrating a method 600 for determining the pose of an object relative to reference coordinates according to some embodiments of the present disclosure is shown. Some or all of the steps in method 600 may be performed by a control device (e.g., control device 120) of a control system 100. Control device 120 may be configured on a computing device. Some or all of the steps in method 600 may be implemented by software, firmware, and / or hardware. In some embodiments, method 600 may be performed by a robotic system (e.g., Figure 13 The surgical robot system 1300 shown is executed. In some embodiments, method 600 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 13 The processor 1320 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0055] See Figure 6 In step 601, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system are determined. In some embodiments, the three-dimensional coordinates of each marker pattern corner point in the marker coordinate system {wm0} can be determined based on the following formula (3):

[0056] C m =[r·cosα] m r·sinα m 0] T (3)

[0057] Among them, C m To use the selected corner point of the marker pattern as the first corner point of the marker pattern (e.g., pose marker pattern corner point P) 401 ), in the clockwise direction of cross-section circle 431, the three-dimensional coordinates of the m-th corner point of the marking pattern in the marking coordinate system, where r is the radius.

[0058] In some embodiments, the axial angle α of the m-th corner marker of the marker pattern is determined based on formula (1). m Then, based on the angle α around the axis determined by formula (1) m Formula (3) determines the three-dimensional coordinates C of the m-th corner point of the marker pattern in the marker coordinate system {wm0}. m .

[0059] See Figure 6In step 603, based on at least one composite identifier, the roll angle of the identifier coordinate system relative to the object coordinate system is determined. In some embodiments, a first axis angle in which one of the at least one composite identifier is identified in the object coordinate system, and a second axis angle in which the composite identifier is identified in the identifier coordinate system, can be determined. Based on the first axis angle and the second axis angle, the roll angle of the identifier coordinate system relative to the object coordinate system can be determined. In some embodiments, see... Figure 4 The roll angle Δα can refer to the rotation angle of the identifier coordinate system {wm0} relative to the object coordinate system {wm} about the Z-axis. In some embodiments, the roll angle Δα can be determined based on the following formula (4):

[0060] Δα=α1-α2 (4)

[0061] Where α1 is the first angle around the axis, and α2 is the second angle around the axis. The first angle around the axis is the corner point of the composite logo pattern (e.g., the corner point R of the composite logo pattern). 402 The first angle is the angle around the axis as indicated in the object's coordinate system. The second angle around the axis is the corner point of the composite logo pattern (e.g., corner point R of the composite logo pattern). 402 The angle around the axis is indicated in the coordinate system.

[0062] In some embodiments, the X-axis of the identifier coordinate system {wm0} points to the corner point of the composite identifier pattern (e.g., the corner point R of the composite identifier pattern). 402 Method 600 may further include determining a first about-axis angle in which the composite identifier is identified in the object coordinate system as a roll angle of the identifier coordinate system relative to the object coordinate system. In some embodiments, the first about-axis angle may be determined based on the pattern included in the composite identifier.

[0063] See Figure 6 In step 605, based on the roll angle of the marker coordinate system relative to the object coordinate system and the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the object coordinate system are determined. It can be understood that, given the roll angle of the marker coordinate system relative to the object coordinate system, the three-dimensional coordinates of multiple marker pattern corner points (e.g., corner points of composite marker patterns and corner points of pose marker patterns) in the marker coordinate system can be transformed into three-dimensional coordinates in the object coordinate system through coordinate transformation.

[0064] See Figure 6 In step 607, based on the two-dimensional coordinates of at least one composite identifier and multiple pose identifiers in the localization image and their three-dimensional coordinates in the object coordinate system, the pose of the object coordinate system relative to the reference coordinate system is determined as the pose of the object relative to the reference coordinate system. In some embodiments, step 607 in method 600 can be implemented similarly to determining the pose of the object coordinate system relative to the reference coordinate system in method 500.

[0065] Figure 7 A flowchart illustrating a method 700 for determining the pose of an object relative to reference coordinates according to other embodiments of the present disclosure is shown. Method 700 may be... Figure 6 An alternative embodiment of method 600. Some or all of the steps in method 700 may be performed by a control device (e.g., control device 120) of the control system 100. Control device 120 may be configured on a computing device. Some or all of the steps in method 700 may be implemented by software, firmware, and / or hardware. In some embodiments, method 700 may be performed by a robot system (e.g., Figure 13 The surgical robot system 1300 shown is executed. In some embodiments, method 700 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 13 The processor 1320 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0066] See Figure 7 In step 701, the pose of the marker coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of at least one composite marker and multiple pose markers in the positioning image and their three-dimensional coordinates in the marker coordinate system. In some embodiments, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system can be implemented similarly to step 601 in method 600.

[0067] See Figure 7 In step 703, based on at least one composite identifier, the roll angle of the identifier coordinate system relative to the object coordinate system is determined. In some embodiments, determining the roll angle of the identifier coordinate system relative to the object coordinate system can be implemented similarly to step 603 in method 600.

[0068] See Figure 7 In step 705, based on the roll angle of the marker coordinate system relative to the object coordinate system and the pose of the marker coordinate system relative to the reference coordinate system, the pose of the object coordinate system relative to the reference coordinate system is determined and used as the pose of the object relative to the reference coordinate system.

[0069] For example, taking the reference coordinate system as the world coordinate system, the pose of the object coordinate system relative to the world coordinate system can be determined based on the following formula (5):

[0070]

[0071] in, w R wm The orientation of the object's coordinate system relative to the world coordinate system. w P wm This represents the position of the object's coordinate system relative to the world coordinate system.w R wm0 To indicate the orientation of the coordinate system relative to the world coordinate system, w P wm0 To identify the position of the coordinate system relative to the world coordinate system, rot z (Δα) represents the roll angle Δα around the Z-axis of the object's coordinate system.

[0072] Figure 8 A flowchart illustrating a method 800 for identifying an identifier according to some embodiments of the present disclosure is shown. Figure 8 As shown, some or all of the steps in method 800 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 13 The processor 1320 shown is used to execute the method. Some or all of the steps in method 800 can be implemented by software, firmware, and / or hardware. In some embodiments, method 800 can be executed by a robot system (e.g., Figure 13 The surgical robot system 1300 shown is executed. In some embodiments, method 800 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 13 The processor 1320 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0073] See Figure 8 In step 801, a plurality of candidate markers are determined from the positioning image. In some embodiments, the markers may include marker pattern corner points in a marker pattern. The coordinates or origin of the coordinate system of a candidate marker can be represented by a candidate marker pattern corner point. In some embodiments, a candidate marker pattern corner point may refer to a possible marker pattern corner point obtained after preliminary processing or preliminary identification of the positioning image.

[0074] In some embodiments, method 800 may include determining a region of interest (ROI) in a localization image. For example, the ROI may be cropped from the localization image, and multiple candidate identifiers may be determined from the ROI. The ROI may be the entire localization image or a partial region. For example, the ROI of the current frame may be cropped based on a region within a certain range of multiple identifier corner points determined in the previous frame image (e.g., the localization image of the previous image processing cycle). For localization images that are not the first frame, the ROI may be a region within a certain distance centered on a virtual point formed by the coordinates of multiple identifier corner points from the previous image processing cycle. The certain distance range may be a fixed multiple of the average spacing distance of the identifier corner points, such as twice. It should be understood that the predetermined multiple may also be a variable multiple of the average spacing distance of the multiple candidate identifier corner points in the previous image processing cycle.

[0075] In some embodiments, method 800 may include determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include: cropping a Region of Interest (ROI) from the localization image and converting the ROI to a corresponding grayscale image.

[0076] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be determined based on the following formula (6):

[0077]

[0078] Where τ is a set constant, for example, set to 2; I x I 45 I y I n45 These are the first derivatives of the pixel in the four directions: 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively.

[0079] In some embodiments, method 800 may include dividing the ROI into multiple sub-regions. For example, a non-maximum suppression method may be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI may be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the location image or ROI may also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels.

[0080] In some embodiments, method 800 may include determining the pixel with the largest corner likelihood value in each sub-region to form a pixel set. For example, the pixel with the largest CL value in each sub-image may be determined, and the pixel with the largest CL value in each sub-image may be compared with a first threshold to determine a set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold may be set to 0.06. It should be understood that the first threshold may also be set to other values.

[0081] See Figure 8 In step 803, a first identifier is identified from a plurality of candidate identifiers. In some embodiments, the first identifier is identified based on an identifier pattern matching template. In some embodiments, the identifier pattern matching template includes at least one pose identifier pattern matching template and multiple composite identifier pattern matching templates with different patterns. In some embodiments, the composite identifier is identified based on multiple composite identifier pattern matching templates with different patterns. For example, if the identifier patterns of pose identifiers are the same, the pose identifier pattern matching template can be matched with the candidate identifiers first. If the matching fails, multiple different composite identifier pattern matching templates can be matched with the candidate identifiers one by one until a match is successful.

[0082] In some embodiments, a marker pattern matching template is used to match the pattern at the corner of a candidate marker pattern to identify the first marker. For example, a candidate marker pattern corner that meets a preset pose pattern matching degree standard is determined as the first marker pattern corner. In some embodiments, the marker pattern matching template and the pattern in the vicinity of the marker pattern corner have the same or similar features. If the matching degree between the marker pattern matching template and the pattern in the vicinity of the candidate marker pattern corner reaches a preset pattern matching degree standard (e.g., the matching degree is higher than a threshold), it can be considered that the pattern in the vicinity of the candidate marker pattern corner has the same or similar features as the marker pattern matching template, and thus the current candidate marker pattern corner can be considered as the marker pattern corner.

[0083] In some embodiments, the pixel with the largest CL value in the pixel set is identified as a candidate identifier pattern corner point. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value can be selected as the candidate identifier pattern corner point. In some embodiments, after the candidate identifier pattern corner point is determined, an identifier pattern matching template is used to match the pattern at the candidate identifier pattern corner point. If a preset pattern matching degree standard is met, the candidate identifier pattern corner point is determined as the first identified identifier pattern corner point.

[0084] In some embodiments, method 800 may further include, in response to a matching failure, determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a candidate identifier pattern corner point. For example, if a candidate identifier pattern corner point does not meet a preset matching degree standard, a pixel with a secondary CL value (the pixel with the second largest CL value) is selected as a candidate identifier pattern corner point, and an identifier pattern matching template is used to match the pattern at the candidate identifier pattern corner point, and so on, until the first identifier pattern corner point is identified.

[0085] In some embodiments, the logo pattern can be a black and white checkerboard pattern, therefore the logo pattern matching template can be the same checkerboard pattern, utilizing the grayscale distribution G of the logo pattern matching template. M The pixel neighborhood grayscale distribution G of the pixel corresponding to the corner point of the candidate identifier pattern image The correlation coefficient (CC) between pixels is used for matching. The grayscale distribution G of the pixel neighborhood is also considered. image This refers to the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The correlation coefficient can be determined based on the following formula (7):

[0086]

[0087] Where Var() is the variance function and Cov() is the covariance function. In some embodiments, when the correlation coefficient is less than 0.8, the gray-level distribution in the pixel neighborhood has a low correlation with the identifier pattern matching template. In this case, the candidate identifier pattern corner with the highest likelihood value is determined not to be an identifier pattern corner; otherwise, the candidate identifier pattern corner with the highest likelihood value is considered to be an identifier pattern corner.

[0088] In some embodiments, method 800 may further include determining the edge direction of the corner points of the candidate identifier pattern. For example, as Figure 9 As shown, the corner point of the candidate pose identifier pattern is corner point P in pose identifier pattern 900. 901 Then the corner point P 901 The edge direction can refer to the direction of the corner point P. 901 The direction of the edge, such as Figure 9 The direction indicated by the dashed arrow.

[0089] In some embodiments, the edge direction can be determined by the first-order derivative (I0) of each pixel in the X and Y directions of the planar coordinate system with respect to a certain neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate identifier pattern. x and I y The edge direction can be determined based on the following formula (8):

[0090]

[0091] Among them, the first derivative (I) x and I y This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by performing a convolution operation on the edge direction I of each pixel within the neighborhood range. angle and the corresponding weight I weight Clustering calculations are performed to obtain the edge direction of the pixel, and weight I is selected. weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if multiple edge directions exist, then weight I is selected. weight The I corresponding to the largest proportion of multiple classes angle As the edge direction.

[0092] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0093] In some embodiments, method 800 may include rotating a marker pattern matching template based on its edge direction. Rotating the marker pattern matching template based on its edge direction can align the template with the image at the corner point of a candidate marker pattern. The edge direction of the corner point of the candidate marker pattern can be used to determine the orientation of the image at that corner point in the positioning image. In some embodiments, rotating the marker pattern matching template based on its edge direction can adjust it to have the same or nearly the same orientation as the image at the corner point of the candidate marker pattern to facilitate image matching.

[0094] See Figure 8 In step 805, starting with the first identifier, other identifiers are searched. In some embodiments, in response to the identification of a composite identifier, other identifiers are identified based on a pose identifier pattern matching template. In some embodiments, other identifiers include pose identifiers or composite identifiers. Figure 10 A flowchart illustrating a method 1000 for searching for identifiers according to some embodiments of the present disclosure is shown. Figure 10 As shown, some or all of the steps in method 1000 can be performed by a data processing device (e.g., Figure 1 The control device 120 shown, Figure 13 The processor 1320 shown is used to execute the method. Some or all of the steps in method 1000 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1000 can be executed by a robot system (e.g., Figure 13 The surgical robot system 1300 shown is executed. In some embodiments, method 1000 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 13 The processor 1320 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0095] See Figure 10 In step 1001, a second identifier is determined starting from the first identifier. In some embodiments, the corner point of the first identifier pattern is used as the starting point, and the corner point of the second identifier pattern is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following directions: directly in front of the corner point of the first identifier pattern (corresponding to a 0° angle direction), directly behind it (corresponding to a 120° angle direction), directly above it (90° angle direction), directly below it (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0096] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions, with each search direction v sn It can be determined based on the following formula (9):

[0097] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (9)

[0098] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent corner points of multiple marker patterns determined in the previous frame. For example, the predetermined search direction is determined based on the following formula (10):

[0099]

[0100] Among them, (x j ,y j ) represents the two-dimensional coordinates of the corner points of multiple marker patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple marker patterns determined in the previous frame; v s1 The first set search direction; v s2 This is the second search direction set.

[0101] In some embodiments, such as Figure 11 As shown, the first identification pattern corner point P 1101 Using the coordinates of the location as the starting point, search for the corner point P of the second marker pattern in the set search direction. 1102 The coordinates of the location. For example, the corner point P of the first identifier pattern. 1101 Using the coordinates as the starting point for the search, the search box (e.g., ...) Figure 11 (The dashed box in the image) moves in the set search direction V with a certain search step size. 1101 Search for the corner points of the icon pattern.

[0102] In some embodiments, if there is at least one candidate identifier within the search box, the candidate identifier pattern corner point with the highest corner point likelihood value within the search box is preferentially selected as the second identifier pattern corner point P. 1102 With the search box limited to a suitable size, the first identifier pattern corner point P is used. 1101 The coordinates of the second marker P are used as the starting point for the search. 1102 During the search, the candidate icon with the highest corner likelihood value among the candidate icons appearing in the search box is more likely to be the corner of the icon pattern. Therefore, it can be considered that the candidate icon with the highest corner likelihood value in the search box is the corner of the second icon pattern, P. 1102 To improve data processing speed. In other embodiments, to improve the accuracy of corner point recognition of the marker pattern, when at least one candidate marker exists in the search box, the candidate marker pattern corner point with the highest corner point likelihood value among the candidate markers appearing in the search box is selected for corner point recognition to determine whether the candidate marker pattern corner point with the highest corner point likelihood value is a marker pattern corner point. For example, a pose marker pattern matching template or a composite marker pattern matching template can be used to match the image within a certain range of the candidate marker pattern corner point with the highest corner point likelihood value. The candidate marker pattern corner point that meets the preset pattern matching degree standard can be considered as the searched second marker pattern corner point P. 1102 .

[0103] In some embodiments, continue reading Figure 11 The size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0104] In some embodiments, the identification pattern can be a black and white graphic, and pattern matching can be performed based on the correlation coefficient in formula (7). If the correlation coefficient is greater than the threshold, the candidate identification pattern corner with the largest likelihood value is considered to be the identification pattern corner, and is denoted as the second identification pattern corner.

[0105] See Figure 10In step 1003, a search direction is determined based on the first identifier and the second identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of the corner point of the first identifier pattern and moving away from the corner point of the second identifier pattern. The second search direction may be a direction starting from the coordinate position of the corner point of the second identifier pattern and moving away from the corner point of the first identifier pattern. For example, Figure 11 The search direction V shown 1102 .

[0106] In step 1005, starting with the first or second identifier, an identifier is searched in the search direction. In some embodiments, if the corner point of the first identifier pattern is used as the new starting point, the first search direction described above can be used as the search direction for the identifier pattern corner point. If the corner point of the second identifier pattern is used as the new starting point, the second search direction described above can be used as the search direction for the identifier pattern corner point. In some embodiments, a new identifier pattern corner point is searched (e.g., Figure 11 The third identifier pattern corner point P 1103 This can be performed similarly to step 1001. In some embodiments, the search step size can be the first identifier pattern corner point P. 1101 Second identification pattern corner point P 1102 The distance between them is L1.

[0107] In some embodiments, in response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; and an identifier pattern matching template is matched with the identifier pattern at the position of the candidate identifier pattern corner point to identify a first identifier. In some embodiments, after determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a new candidate identifier pattern corner point, a new first identifier can be identified based on a method similar to step 803. In some embodiments, a search distance greater than a search distance threshold can be understood as a search distance greater than a search distance threshold in some or all search directions. In some embodiments, the search distance threshold may include a set multiple of the distance between the (N-1)th pose identifier pattern corner point and the (N-2)th pose identifier pattern corner point, where N≥3. For example, the search distance threshold is twice the distance between the first two identifier pattern corner points. Thus, the maximum search distance for the third marker corner is twice the distance between the first and second marker corners. If no marker corner is found within this search distance in the search direction, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is identified as a new candidate pose marker corner, and a new first marker is identified. The current search process then stops. In some embodiments, similar to method 800, a new first marker corner can be determined, and similar to method 1000, the remaining marker corners can be searched starting from the new marker corner.

[0108] In some embodiments, in response to the number of identified markers being greater than or equal to a marker number threshold, the pose of the object relative to a reference coordinate system can be determined based on the identified markers, and the search for markers can be stopped accordingly. For example, in response to the number of identified marker pattern corner points being greater than or equal to the marker number threshold, the search for marker pattern corner points can be stopped. For example, when four marker pattern corner points are identified, the search for marker pattern corner points can be stopped.

[0109] In some embodiments, in response to the number of identified identifiers being less than an identifier number threshold, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; and an identifier pattern matching template is matched with the identifier pattern at the position of the candidate identifier pattern corner point to identify the first identifier. In some embodiments, if the total number of identified identifier pattern corner points is less than the identifier number threshold, the search based on the first identifier pattern in the above steps is considered to have failed. In some embodiments, if composite identifiers are not included among all identified identifiers, for example, if the identified identifier pattern corner points do not include composite identifier pattern corner points, the search based on the first identifier pattern in the above steps is considered to have failed. In some embodiments, in the case of search failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate identifier pattern corner point, and then a new first identifier can be identified based on a method similar to step 803. In some embodiments, similar to method 800, a new first identifier pattern corner point can be re-determined, and similar to method 1000, the remaining identifier pattern corner points can be searched starting from the new identifier pattern corner point.

[0110] In some embodiments, if the identified identifiers include composite identifiers, the type of the remaining identifiers found may be uncertain (it should be understood that identifier types include pose identifiers and composite identifiers). For example, if the first identifier is a composite identifier, it may be uncertain whether the second identifier is a pose identifier or a composite identifier.

[0111] In some embodiments, if the identified identifiers do not include composite identifiers, the type of the newly found identifier is determined. For example, if the first identifier is not a composite identifier, it is necessary to determine whether the second identifier is a pose identifier or a composite identifier. If neither the first nor the second identifier is a composite identifier, it is necessary to determine whether the third identifier is a pose identifier or a composite identifier.

[0112] In some embodiments, after the corner points of the marker pattern are searched or identified, sub-pixel positioning can be performed on the identified corner points of the marker pattern to improve the positional accuracy of the corner points of the marker pattern.

[0113] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of the corner points of the identifier pattern after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, the extreme points of which are subpixel points. The fitting function can be determined based on the following formulas (11) and (12):

[0114] S(x,y)=ax 2 +by 2 +cx+dy+exy+f (11)

[0115]

[0116] Where S(x, y) is the fitting function for the CL values ​​of all pixels in each ROI, and a, b, c, d, e, and f are coefficients; x c The x-coordinate and y-coordinate of the pose identifier c The y-coordinate is the pose identifier.

[0117] In some embodiments of this disclosure, a computer device is also provided, including a memory for storing at least one instruction; and a processor coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figure 5 , Figure 6 , Figure 7 , Figure 8 and Figure 10 Some or all of the steps in the method disclosed herein.

[0118] Figure 12 A schematic diagram of a computer device 1200 according to some embodiments of the present disclosure is shown. See also Figure 12 The computer device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including random access memory (RAM) 1202 and read-only memory (ROM) 1203, and a system bus 1205 connecting the various components. The computer device 1200 also includes an input / output system and a mass storage device 1207 for storing the operating system 1212, application programs 1214, and other program modules 1215. The input / output devices include an input / output controller 1210 mainly composed of a display 1208 and input devices 1209.

[0119] Mass storage device 1207 is connected to central processing unit 1201 via a mass storage controller (not shown) connected to system bus 1205. Mass storage device 1207 and its associated computer-readable media provide non-volatile storage for computer devices. That is, mass storage device 1207 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0120] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0121] Computer device 1200 can be connected to network 1212 via network interface unit 1211 connected to system bus 1205.

[0122] The system memory 1204 or mass storage device 1207 is also used to store one or more instructions. The central processing unit 1201 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0123] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the positioning method of some embodiments of this disclosure, such as... Figure 5 , Figure 6 , Figure 7 , Figure 8 and Figure 10 Some or all of the steps in the method disclosed herein.

[0124] In some embodiments, this disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory including a computer program (instructions) executable by a processor of a computer device to perform the methods shown in various embodiments of this application. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0125] Figure 13 A schematic diagram of a surgical robot system 1300 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [link to relevant documentation]. Figure 13The surgical robot system 1300 may include a surgical tool 1350, an image acquisition unit 1310, and a processor 1320. The surgical tool 1350 may include a manipulator arm 1340, an actuator 1330 disposed at the distal end of the manipulator arm 1340, and at least one composite identifier and multiple pose identifiers disposed at the distal end of the manipulator arm 1340. The image acquisition unit 1310 can be used to acquire positioning images of the manipulator arm 1340. The processor 1320 is connected to the image acquisition unit 1310 and is used to execute some or all of the steps in the methods of some embodiments of this disclosure, such as... Figure 5 , Figure 6 , Figure 7 , Figure 8 and Figure 10 Some or all of the steps in the method disclosed herein.

[0126] While specific embodiments of this disclosure have been illustrated and described by way of example, it will be apparent to those skilled in the art that many other changes and modifications can be made without departing from the spirit and scope of this disclosure. Therefore, all such changes and modifications falling within the scope of this disclosure are included in the appended claims.

Claims

1. A method for determining the pose of an object, comprising: Acquire the location image; In the positioning image, multiple markers located on the object are identified, including multiple pose markers for identifying pose and at least one composite marker for identifying pose and angle. as well as Based on the at least one composite identifier and the plurality of pose identifiers, the pose of the object relative to the reference coordinate system is determined; Determining the pose of the object relative to the reference coordinate system based on the at least one composite identifier and the plurality of pose identifiers includes: Determine the two-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the positioning image; Determine the three-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the identifier coordinate system; Based on the at least one composite identifier, determine the roll angle of the identifier coordinate system relative to the object coordinate system; Based on the roll angle of the marker coordinate system relative to the object coordinate system and the three-dimensional coordinates of the at least one composite marker and the plurality of pose markers in the marker coordinate system, determine the three-dimensional coordinates of the at least one composite marker and the plurality of pose markers in the object coordinate system; and Based on the two-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the positioning image and the three-dimensional coordinates in the object coordinate system, the pose of the object coordinate system relative to the reference coordinate system is determined as the pose of the object relative to the reference coordinate system. Alternatively, determining the pose of the object relative to the reference coordinate system based on the at least one composite identifier and the plurality of pose identifiers includes: Determine the two-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the positioning image; Determine the three-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the identifier coordinate system; Based on the at least one composite identifier, determine the roll angle of the identifier coordinate system relative to the object coordinate system; Based on the two-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in the positioning image and their three-dimensional coordinates in the identifier coordinate system, the pose of the identifier coordinate system relative to the reference coordinate system is determined; and Based on the roll angle of the marker coordinate system relative to the object coordinate system and the pose of the marker coordinate system relative to the reference coordinate system, the pose of the object coordinate system relative to the reference coordinate system is determined and used as the pose of the object relative to the reference coordinate system.

2. The method according to claim 1, comprising: Determine a first about-axis angle identified by one of the at least one composite identifiers in the object coordinate system; Determine the second about-axis angle of one of the at least one composite identifiers in the identifier coordinate system; as well as Based on the first and second axial angles, the roll angle of the marker coordinate system relative to the object coordinate system is determined.

3. The method according to claim 1, wherein the X-axis of the identification coordinate system points to the composite identification, the method comprising: The first angle around the axis of the composite identifier in the object coordinate system is determined as the roll angle of the identifier coordinate system relative to the object coordinate system.

4. The method according to claim 2 or 3, comprising: The first angle around the axis is determined based on the pattern included in the composite logo.

5. The method according to claim 1, comprising: Multiple candidate identifiers are determined from the location image; Identify a first identifier from the plurality of candidate identifiers; as well as Starting from the first identifier, search for other identifiers.

6. The method according to claim 5, comprising: The first identifier is identified based on the identifier pattern matching template.

7. The method according to claim 5, comprising: The composite logo is identified based on a pattern matching template with multiple different composite logo patterns.

8. The method according to claim 5, comprising: In response to the recognition of the composite identifier, other identifiers are identified based on the pose identifier pattern matching template.

9. The method according to claim 6, wherein the mark includes a mark pattern and mark pattern corner dots in the mark pattern, the method comprising: Determine the region of interest in the positioning image; The region of interest is divided into multiple sub-regions; The pixel with the largest corner likelihood value in each sub-region is determined to form a pixel set; The pixel with the highest corner likelihood value among the multiple candidate identifiers is selected as the corner point of the candidate identifier pattern. as well as The identification pattern matching template is matched with the identification pattern at the corner position of the candidate identification pattern to identify the first identification.

10. The method of claim 9, comprising: In response to a matching failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is identified as a candidate identifier pattern corner.

11. The method of claim 9, comprising: Starting from the first identifier, search for the second identifier; Based on the first identifier and the second identifier, the search direction is determined; as well as Starting from the first identifier or the second identifier, search for identifiers in the search direction.

12. The method of claim 11, comprising: In response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; as well as The identification pattern matching template is matched with the identification pattern at the corner position of the candidate identification pattern to identify the first identification.

13. The method of claim 11, comprising: In response to the number of identified identifiers being greater than or equal to an identifier number threshold, the pose of the object relative to the reference coordinate system is determined based on the identified identifiers.

14. The method of claim 11, comprising: In response to the fact that the number of identified identifiers is less than the identifier number threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as the candidate identifier pattern corner. as well as The identification pattern matching template is matched with the identification pattern at the corner position of the candidate identification pattern to identify the first identification.

15. The method according to any one of claims 1-3 and 5-14, comprising: Based on the pose of the object relative to the reference coordinate system, the pose of the end effector of the object relative to the reference coordinate system is determined.

16. The method according to any one of claims 1-3 and 5-14, wherein the plurality of markings are disposed on the outer surface of the columnar portion of the object.

17. The method according to any one of claims 1-3 and 5-14, wherein a positioning label is provided on the outer surface of the columnar portion of the object, the positioning label comprising a plurality of identification patterns, the plurality of identification patterns comprising a plurality of different composite identification patterns and a plurality of pose identification patterns, the plurality of different composite identification patterns and the plurality of pose identification patterns being located in the same pattern distribution zone.

18. The method according to claim 17, wherein at least one composite identification pattern is included among the N consecutive identification patterns of the plurality of identification patterns, where 2≤N≤4.

19. A computer device, the computer device comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory, is configured to execute the at least one instruction to perform the method as described in any one of claims 1-18.

20. A computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the method as described in any one of claims 1-18.

21. A surgical robot system, comprising: A surgical tool, the surgical tool comprising an operating arm, an actuator disposed at the distal end of the end of the operating arm, and at least one composite marker and multiple pose markers disposed at the end of the operating arm; An image acquisition device is used to acquire positioning images of the operating arm; as well as A processor, connected to the image acquisition unit, is configured to perform the method as described in any one of claims 1-18 to determine the pose of the actuator.

Images