Positioning method and surgical robotic system

Abstract

The present disclosure relates to the technical field of positioning, and discloses a positioning method for determining a pose, a computer device, a computer readable storage medium and a surgical robot system. The positioning method for determining a pose comprises: acquiring a positioning image; in the positioning image, identifying a plurality of pose marks located on an object; based on the plurality of pose marks, identifying an angle mark located on the object, the angle mark having a positional correlation relationship with a first pose mark in the plurality of pose marks; and based on the angle mark and the plurality of pose marks, determining a pose of the object relative to a reference coordinate system.

Description

Technical Field

[0001] This disclosure belongs to the field of positioning technology, and in particular relates to a positioning method and a surgical robot system. Background Technology

[0002] With the development of technology, it is becoming increasingly common for machines and equipment to be controlled by humans or computers to perform desired actions in order to assist or replace operators. For example, logistics robots are used to sort express packages, and surgical robots are used to assist doctors in performing surgeries.

[0003] In the above applications, it is necessary to position the movable parts such as the controlled device or structure in order to control the machine equipment. Summary of the Invention

[0004] In some embodiments, this disclosure provides a positioning method for determining pose, including:

[0005] Acquire the location image;

[0006] In the localization image, identify multiple pose markers located on the object;

[0007] Based on multiple pose markers, an angle marker located on an object is identified, and the angle marker has a positional association with the first pose marker among the multiple pose markers; and

[0008] Based on angle identifiers and multiple pose identifiers, the pose of the object relative to the reference coordinate system is determined.

[0009] In some embodiments, this disclosure provides a computer device, the computer device comprising:

[0010] Memory, used to store at least one instruction; and

[0011] A processor, coupled to memory, is configured to execute the at least one instruction to perform the positioning method of this disclosure.

[0012] In some embodiments, this disclosure provides a computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the positioning method of this disclosure.

[0013] In some embodiments, this disclosure provides a surgical robot system, including:

[0014] Surgical instruments include an arm body, an actuator disposed at the distal end of the arm body, and at least one angle marker and multiple pose markers disposed on the distal portion of the arm body.

[0015] Image acquisition device, used to acquire positioning images of the arm body; and

[0016] A processor, connected to an image acquisition unit, is used to execute the positioning method of this disclosure to determine the pose of the end portion of the arm. Attached Figure Description

[0017] Figure 1 A schematic diagram of a device 100 capable of performing positioning methods according to some embodiments of the present disclosure is shown;

[0018] Figure 2 A label 200 including multiple pose identifiers and multiple angle identifiers is shown;

[0019] Figure 3 A label 200 is shown disposed on a cylindrical label 300 formed around the periphery of an object;

[0020] Figure 4 This diagram illustrates an implementation scenario 400 according to some embodiments of the present disclosure;

[0021] Figure 5 A flowchart of a positioning method 500 according to some embodiments of the present disclosure is shown;

[0022] Figure 6 A flowchart of a method 600 for determining the pose of an object coordinate system relative to a reference coordinate system, according to some embodiments of the present disclosure, is shown.

[0023] Figure 7 A flowchart of a method 700 for determining the pose of an object coordinate system relative to a reference coordinate system, according to some embodiments of the present disclosure, is shown.

[0024] Figure 8 A schematic diagram of multiple pose markers on cross-sectional circle 822 according to some embodiments of the present disclosure is shown;

[0025] Figure 9 A flowchart of a method 900 for identifying pose identifiers according to some embodiments of the present disclosure is shown;

[0026] Figure 10 A schematic diagram of a pose identification pattern 1011 according to some embodiments of the present disclosure is shown;

[0027] Figure 11 A flowchart of a method 1100 for searching pose identifiers according to some embodiments of the present disclosure is shown;

[0028] Figure 12 A schematic diagram of a search pose identifier according to some embodiments of the present disclosure is shown;

[0029] Figure 13 A flowchart of a method 1300 for identifying angle markers according to some embodiments of the present disclosure is shown;

[0030] Figure 14 A schematic diagram of a computer device 1400 according to some embodiments of the present disclosure is shown;

[0031] Figure 15 A schematic diagram of a surgical robot system 1500 according to some embodiments of the present disclosure is shown. Detailed Implementation

[0032] Exemplary embodiments of this disclosure are described below with reference to the accompanying drawings. Those skilled in the art will understand that the scope of this disclosure is not limited to these embodiments. Various modifications and variations can be made to the embodiments described below. All such modifications and variations are included within the scope of this disclosure. Similar reference numerals indicate similar components among the various embodiments shown in the accompanying drawings.

[0033] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom can be described using variations in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "attitude" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and attitude of an object or part of an object, which can be described, for example, using six parameters from the six degrees of freedom mentioned above.

[0034] In some embodiments, the positioning method can be used to determine the pose of an object relative to a reference coordinate system. In some embodiments, the reference coordinate system can be understood as a coordinate system that describes the position of the object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own visual coordinate system, etc. In some embodiments, the pose of the object's coordinate system is used to represent the pose of the object, and the pose of the object relative to the reference coordinate system is the pose of the object's coordinate system relative to the reference coordinate system. In some embodiments, the object can be understood as the object or target that needs to be positioned.

[0035] In some embodiments, the positioning method can be used in any application scenario that requires obtaining the pose of an object. For example, in the process of controlling the actuator of a surgical robot to perform actions such as grasping, clamping, cutting, electrocoagulation or suturing, in order to achieve precise control of the actuator, it is necessary to obtain the actual position of the actuator relative to the world coordinate system, as well as the attitude of the actuator relative to the world coordinate system (e.g., may include the roll angle, pitch angle and yaw angle of the actuator).

[0036] In some embodiments, the definitions of the coordinate systems and poses involved in the positioning method of this disclosure can be as shown in Table 1.

[0037] Table 1

[0038]

[0039]

[0040] In some embodiments, the object may be a movable device such as a controlled moving apparatus or structure, which may be controlled by a control terminal or by an operator. For example, such an apparatus or structure may be a robotic arm (actuator) or a robotic leg.

[0041] In some embodiments, the object may also include a functional end or end effector of a movable device, such as a means or structure that is controlled to move. For example, the gripping end or gripper of a sorting robot's robotic arm, or the end effector or actuator of a surgical robot's execution arm.

[0042] In some embodiments, the aforementioned robotic arms or legs may be, for example, robotic arms or legs of medical robots (e.g., surgical robots, rehabilitation robots, assistive robots, medical service robots, etc.), industrial robots, logistics robots (e.g., sorting robots), cleaning robots, and biomimetic robots (e.g., robotic dogs, robotic cows, etc.). Specifically, surgical robots may be laparoscopic surgical robots, orthopedic surgical robots, and vascular interventional surgical robots, etc. Exemplarily, the positioning methods according to some embodiments of this disclosure can be used for, for example... Figure 15 The surgical robot system 1500 shown can specifically be used to position the arm 1540, the end effector of the arm 1540, or the actuator 1530 located at the distal end of the arm 1540. For example, the positioning method according to some embodiments of this disclosure can obtain the pose of the arm 1540, the end effector of the arm 1540, or the actuator 1530 located at the distal end of the arm 1540 relative to a reference coordinate system (e.g., world coordinate system, camera coordinate system, or the operator's own visual coordinate system).

[0043] In some embodiments, the working environment of an object may include the ground (e.g., the arm or actuator of a mechanical device working on the ground surface), underground (e.g., the arm or actuator of an underground mining device that performs digging functions such as mining), underwater (e.g., the arm or actuator of an underwater activity device such as a submersible), space (e.g., the arm or actuator of a spacecraft such as a space station), inside an animal (e.g., the arm or actuator of a surgical robot, etc.), etc.

[0044] In some embodiments, the object can be rigid or deformable. For example, a rigid robotic arm or a deformable robotic arm.

[0045] Figure 1 A schematic diagram of a device 100 capable of performing positioning methods according to some embodiments of the present disclosure is shown. See also Figure 1 The device 100 may include an image acquisition module 110 and a data processing module 120. The image acquisition module 110 is connected to the data processing module 120. For example, the image acquisition module 110 is connected to the data processing module 120 via a communication network (e.g., a wireless network or a wired network).

[0046] In some embodiments, the image acquisition module 110 can be used to acquire positioning images. The positioning image may include part or all of an image of the object 130. The object 130 may be the end effector of the actuator arm 140. Positioning markers are provided on the object 130. The positioning markers may include pose markers and angle markers (described in detail below). Figure 1 As shown, if object 130 is within the field of view of the image acquisition module, the acquired positioning image may include an image of object 130.

[0047] In some embodiments, depending on the application scenario, the image acquisition module 110 may be an industrial camera, an underwater camera, a miniature electronic camera, an endoscope camera, etc. In some embodiments, the image acquisition module 110 may be fixed in position or have a variable position; for example, an industrial camera fixed at a monitoring location or an endoscope camera with adjustable position or orientation. In some embodiments, the image acquisition module 110 may perform at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image acquired, those skilled in the art can select different image acquisition devices as the image acquisition module 110.

[0048] In some embodiments, the data processing module 120 can be used to locate the object 130 based on the positioning image to determine the pose of the object 130. The data processing module 120 can acquire the positioning image. For example, after the image acquisition module 110 acquires the positioning image, the data processing module 120 can receive the positioning image sent by the image acquisition module 110. The data processing module 120 can process the positioning image to determine the pose of the object 130.

[0049] In some embodiments, the data processing module 120 may include a local processor (e.g., a local computer device) or a cloud processor (e.g., a cloud server or cloud computing platform). In some embodiments, the data processing module 120 may be a terminal with a processor, such as a desktop computer, laptop computer, or tablet computer.

[0050] In some embodiments, an object 130 has a plurality of pose markers and at least one angle marker distributed thereon. For example, the plurality of pose markers are distributed circumferentially on the object 130, and the plurality of angle markers are distributed circumferentially on the object 130. The plurality of pose markers and the plurality of angle markers are arranged side by side on the object 130 along the axial direction. Exemplarily, the plurality of pose markers and the plurality of angle markers are disposed on the outer surface of the columnar portion of the object 130.

[0051] In some embodiments, each angle marker has a positional association with one of the pose markers. Based on this positional association, the possible distribution area of ​​the angle markers can be determined by the position of the pose markers. Alternatively, the possible distribution area of ​​the pose markers can be determined by the position of the angle markers. The positional association can be determined according to the specific arrangement of the pose markers and angle markers.

[0052] In some embodiments, the positional association may include an axial correspondence between angle markers and pose markers. For example, based on this axial correspondence, given the known positions of one or more pose markers on an object, offsetting by a certain distance along the axial direction can determine the area where angle markers may exist.

[0053] In some embodiments, multiple pose markers and multiple angle markers may be set on a label attached to the periphery of the object.

[0054] In some embodiments, the pose marker may include a pose marker pattern, and the angle marker may include an angle marker pattern. In some embodiments, the pose marker pattern and the angle marker pattern may be affixed to a label on an object, or printed on the object, or may be patterns formed by the physical structure of the object itself, such as recessed or raised structures and combinations thereof. In some embodiments, the pose marker pattern or the angle marker pattern may differ in brightness, grayscale, and color.

[0055] In some embodiments, the pose marker pattern and the angle marker pattern may be graphics that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provide information that can be detected by the image acquisition module.

[0056] In some embodiments, pose marker patterns or angle marker patterns are set on an area of ​​the object suitable for image acquisition by the image acquisition module, such as an area that can be covered by the field of view of the image acquisition module during operation or an area that is not easily disturbed or obstructed during operation.

[0057] Figure 2 A label 200 is shown, which includes multiple pose identifiers and multiple angle identifiers.

[0058] Figure 3 A label 200 is shown disposed on a cylindrical label 300 formed around the periphery of an object.

[0059] See Figure 2 Multiple pose markers (represented by the symbol "○" in this disclosure) and multiple angle markers (represented by the symbol "△" in this disclosure) are arranged in two rows. Multiple pose marker patterns 211 are used to form multiple pose markers, and multiple angle marker patterns 221-226 are used to form multiple angle markers. In some embodiments, pose markers can be determined by recognizing pose marker patterns, and angle markers can be determined by recognizing angle marker patterns. In some embodiments, pose markers may include corner points of pose marker patterns, and angle markers may include corner points of angle marker patterns.

[0060] based on Figure 2 The arrangement of the pose and angle markers determines the positional relationship between each angle marker and one of the pose markers when the corresponding labels are attached to the periphery of the object.

[0061] For example, such as Figure 2 As shown, in the direction indicated by the arrow, some pose markers and their corresponding angle markers have a spacing of d1. See also Figure 3 In the circumferential setting state, label 200 becomes label 300 with a spatial structure of a cylinder. The positional association between each angle identifier and one of the pose identifiers can include the angle identifier and the pose identifier in the axial direction (e.g., Figure 3 The correspondence between the positive Z-axis and the pose markers. Based on the axial correspondence, given the positions of one or more pose markers on an object, offsetting by a certain distance (e.g., distance d1) along the axial direction can determine the area where angle markers may exist. In some embodiments, based on the axial correspondence between angle markers and pose markers, the projections of one of the angle markers and pose markers along the Z-axis coincide. Given the angle of one of the angle markers (e.g., the positive Z-axis direction), the coordinates are determined. Figure 3Given the angle identifier R3 in the XY coordinate system (within which the angle θ is located), based on this positional relationship, the angle θ between the pose identifier P3 associated with its position and the X-axis can be obtained. It should be understood that the angle θ corresponding to the angle identifier R3 and the pose identifier P3 can be considered as the axial angle or roll angle around the Z-axis. In this disclosure, the axial angle or roll angle refers to the angle around the Z-axis.

[0062] Figure 4 A schematic diagram of an implementation scenario 400 according to some embodiments of the present disclosure is shown. For example... Figure 4 As shown, object 430 is the end effector or actuator of the actuator arm 440. For example... Figure 2 The label 200 shown is circumferentially disposed on the end of the actuator arm 440, forming a cylindrical angle marking pattern structure 410 and a pose marking pattern structure 420. Multiple pose markings are distributed on the cross-sectional circle 421 at the end of the actuator arm, and multiple angle markings are distributed on the cross-sectional circle 411 at the end of the actuator arm 440.

[0063] In some embodiments, the multiple angle marker patterns are different patterns. Each angle marker pattern is used to indicate or identify a different angle about the axis. In some embodiments, the pattern information of each angle marker has a one-to-one correspondence with the identified angle about the axis, and the identified angle about the axis can be obtained based on the pattern information of the angle marker.

[0064] For example, such as Figure 4 As shown, multiple different angle marking patterns (such as...) Figure 2 The multiple angle marker patterns (221-226) shown are evenly distributed along the circumference of the cylindrical structure, forming 6 angle markers AF. Setting the angle marker pattern corresponding to angle marker A as a reference pattern (for example, setting the angle marker pattern corresponding to angle marker A to mark a 0° angle around the axis), and establishing a plane coordinate system {wm1}, the angles around the axis marked by the other angle marker patterns can be determined based on the positional relationship between the other angle marker patterns and the angle marker pattern corresponding to angle marker A. For example, see... Figure 4 When the angle mark pattern corresponding to angle mark B is identified, based on the positional relationship between the angle mark pattern corresponding to angle mark B and the angle mark pattern corresponding to angle mark A, it can be determined that the angle around the axis of angle mark B in the two-dimensional plane coordinate system of section circle 411 is 60°. The origin of the two-dimensional plane coordinate system of section circle 411 is the center of section circle 411, the X-axis direction is from the origin to angle mark A, and the Y-axis is perpendicular to the X-axis.

[0065] Figure 5 A flowchart illustrating a positioning method 500 according to some embodiments of the present disclosure is shown. Figure 5As shown, some or all of the steps in method 500 can be performed by a data processing device (e.g., Figure 1 The data processing module 120 shown is... Figure 15 The processor 1520 shown is used to execute the method. Some or all of the steps in method 500 can be implemented by software, firmware, and / or hardware. In some embodiments, method 500 can be executed by a robot system (e.g., Figure 15 The surgical robot system 1500 shown is executed. In some embodiments, method 500 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 15 The processor 1520 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0066] See Figure 5 In step 501, a positioning image is acquired. In some embodiments, the positioning image includes multiple pose markers on the object and at least one angle marker. In some embodiments, it can be obtained from, for example... Figure 1 The data processing module 120 shown receives a positioning image. For example, the data processing module 120 can receive a positioning image actively sent by the image acquisition module 110. Alternatively, the data processing module 120 can send an image request command to the image acquisition module 110, and the image acquisition module 110 can respond to the image request command by sending a positioning image to the data processing module 120.

[0067] Continue reading Figure 5 In step 503, multiple pose markers located on the object are identified in the localization image. For example, an exemplary method for identifying multiple pose markers located on the object may include, for instance... Figure 9 The method is illustrated. In some embodiments, the data processing apparatus can identify some or all of the pose markers in a positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm, which can extract or identify features of a corresponding type depending on the type of pose marker. For example, if the pose marker is a corner feature, the image processing algorithm may include a corner detection algorithm. This corner detection algorithm may be one of, but is not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, if the pose marker is a color feature, the image processing algorithm may be a color feature extraction algorithm. In other possible embodiments, if the pose marker is a contour feature, the image processing algorithm may be a contour detection algorithm.

[0068] In some embodiments, the data processing apparatus can identify some or all of the pose markers in a positioning image by recognizing a recognition model.

[0069] Continue reading Figure 5 In step 505, based on multiple pose identifiers, an angle identifier located on the object is identified, and the angle identifier has a positional association with the first pose identifier among the multiple pose identifiers. In some embodiments, after identifying multiple pose identifiers, the angle identifier located on the object is identified according to the positional association. In some embodiments, the positional association between the angle identifier and the first pose identifier can be as follows: Figure 2 or Figure 3 The positional relationships are shown in the diagram. In some embodiments, the first pose identifier refers to the pose identifier among a plurality of pose identifiers that has a positional relationship with the angle identifier. Exemplary methods for identifying the angle identifier include, as shown in the diagram... Figure 13 The method shown.

[0070] Continue reading Figure 5 In step 507, the pose of the object relative to a reference coordinate system is determined based on the angle identifier and multiple pose identifiers. Exemplary methods for determining the pose of an object relative to a reference coordinate system include, for example... Figure 6 or Figure 7 The method shown.

[0071] In some embodiments, the pose of an object relative to a reference coordinate system can be determined based on an angle identifier, a first pose identifier, and multiple pose identifiers.

[0072] In some embodiments, method 500 includes: determining a transformation relationship between an object coordinate system and a pose identifier coordinate system based on angle identifiers and multiple pose identifiers. In some embodiments, according to the transformation relationship between the object coordinate system and the pose identifier coordinate system, three-dimensional coordinates in the pose coordinate system can be converted into corresponding three-dimensional coordinates in the object coordinate system. In some embodiments, according to the transformation relationship between the object coordinate system and the pose identifier coordinate system and the pose of the pose identifier coordinate system relative to the reference coordinate system, the pose of the object coordinate system relative to the reference coordinate system is obtained.

[0073] In some embodiments, the transformation relationship between the object coordinate system and the pose identifier coordinate system may include the roll angle of the pose identifier coordinate system relative to the object coordinate system. In some embodiments, the roll angle of the pose identifier coordinate system relative to the object coordinate system may be determined based on the angle identifier and the first pose identifier. It should be understood that the roll angle of the pose identifier coordinate system relative to the object coordinate system may be the angle of rotation of the pose identifier coordinate system about the Z-axis of the object coordinate system.

[0074] In some embodiments, method 500 includes: determining a first about-axis angle represented by an angle marker in an object coordinate system; determining a second about-axis angle represented by the first pose marker in a pose marker coordinate system; and determining a roll angle of the pose marker coordinate system relative to the object coordinate system based on the first and second about-axis angles.

[0075] In some embodiments, the object coordinate system may be a fixed coordinate system set on the object based on multiple pose markers or multiple angle markers. In some embodiments, the Z-axis of the object coordinate system is parallel to the axis of the object, and the XY plane of the object coordinate system is in the same plane as the multiple pose markers or the multiple angle markers.

[0076] In some embodiments, a pose identifier coordinate system can be determined to facilitate the determination of the positions of multiple pose identifiers. In some embodiments, the Z-axis of the pose identifier coordinate system is parallel to the axis of the object, and the XY plane of the pose identifier coordinate system is in the same plane as the multiple pose identifiers.

[0077] For example, see Figure 4 The origin of the object coordinate system {wm} is the center of the cross-sectional circle 421 containing multiple pose markers. The X-axis points from the origin to one of the pose markers, the Z-axis is parallel to the axis of object 430, and the Y-axis is perpendicular to the XZ plane. The X-axis of the object coordinate system {wm} is parallel to the X-axis of the two-dimensional plane coordinate system {wm1} of cross-sectional circle 411, and the Y-axis of the object coordinate system is parallel to the Y-axis of the two-dimensional plane coordinate system {wm1} of cross-sectional circle 411. The angle marker's angle around the axis in the two-dimensional plane coordinate system {wm1} of cross-sectional circle 411 can be considered as its angle around the axis in the object coordinate system {wm}. The origin of the pose marker coordinate system {wm0} is the center of the cross-sectional circle 421 containing multiple pose markers. The X-axis points from the origin to one of the pose markers, the Z-axis is parallel to the axis of object 430, and the Y-axis is perpendicular to the XZ plane. (Continue to see...) Figure 4 The object coordinate system {wm} and the pose identifier coordinate system {wm0} share a common Z-axis. The transformation relationship between the object coordinate system {wm} and the pose identifier coordinate system {wm0} can be determined by the roll angle α0 of the pose identifier coordinate system {wm0} relative to the object coordinate system {wm}. The roll angle α0 can be the rotation angle of the pose identifier coordinate system {wm0} relative to the object coordinate system {wm} around the Z-axis.

[0078] In some embodiments, see Figure 4 The roll angle α0 is calculated using the following formula:

[0079] α0=α1-α2 (1)

[0080] Where α1 is the first angle around the axis and α2 is the second angle around the axis. The first angle around the axis is the angle around the axis indicated by the angle marker (e.g., angle marker R4) in the object coordinate system. The second angle around the axis is the angle around the axis indicated by the first pose marker (e.g., pose marker P4) in the pose marker coordinate system.

[0081] In some embodiments, method 500 further includes: determining the pose of the object's end relative to the reference coordinate system based on the object's pose relative to the reference coordinate system. For example, taking the world coordinate system as the reference coordinate system, the specific calculation formula is as follows:

[0082]

[0083] Figure 6 A flowchart of a method 600 for determining the pose of an object's coordinate system relative to a reference coordinate system, according to some embodiments of the present disclosure, is shown. Figure 6 As shown, some or all of the steps in method 600 can be performed by a data processing device (e.g., Figure 1 The data processing module 120 shown is... Figure 15 The processor 1520 shown is used to execute the method. Some or all of the steps in method 600 can be implemented by software, firmware, and / or hardware. In some embodiments, method 600 can be executed by a robot system (e.g., Figure 15 The surgical robot system 1500 shown is executed. In some embodiments, method 600 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 15 The processor 1520 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0084] See Figure 6 In step 601, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on multiple pose identifiers. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple pose identifiers in the positioning image and the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system. In some embodiments, the pose of the pose identifier coordinate system relative to the reference coordinate system is determined based on the transformation relationship between the two-dimensional coordinates of the multiple pose identifiers in the positioning image, the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system, and the camera coordinate system relative to the reference coordinate system.

[0085] In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system can be known. For example, the reference coordinate system is the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system can be determined based on the camera's placement. In other embodiments, the reference coordinate system can also be the camera coordinate system itself, depending on actual needs.

[0086] In some embodiments, based on the camera imaging principle and projection model, and based on the two-dimensional coordinates of multiple pose markers in the positioning image and the three-dimensional coordinates of multiple pose markers in the pose marker coordinate system, the pose of the pose marker coordinate system relative to the camera coordinate system is determined. Based on the transformation relationship between the pose of the pose marker coordinate system relative to the camera coordinate system and the camera coordinate system relative to the reference coordinate system, the pose of the pose marker coordinate system relative to the reference coordinate system can be obtained.

[0087] In some embodiments, based on camera imaging principles and projection models, the pose of the pose identifier coordinate system relative to the camera coordinate system can be obtained according to the two-dimensional coordinates of multiple pose identifiers in the positioning image, the three-dimensional coordinates of multiple pose identifiers in the pose identifier coordinate system, and the camera's intrinsic parameters. In some embodiments, the camera's intrinsic parameters may be as follows: Figure 1 The image acquisition module 110 shown has camera intrinsic parameters. The camera's intrinsic parameters can be known or obtained through calibration.

[0088] In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the optical center of the camera as the origin or a coordinate system established with the center of the camera lens as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, or the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of the line).

[0089] In step 603, the pose of the object coordinate system relative to the reference coordinate system is determined based on the roll angle of the pose identifier coordinate system relative to the object coordinate system and the pose of the pose identifier coordinate system relative to the reference coordinate system.

[0090] For example, taking the reference coordinate system as the world coordinate system, the pose of the object's coordinate system relative to the world coordinate system is as follows:

[0091]

[0092] Among them, rot z (α0) represents the roll angle α0 around the Z-axis.

[0093] In some embodiments, the specific formula for calculating the pose of the object's coordinate system relative to the world coordinate system is as follows:

[0094]

[0095] Figure 7 A flowchart of a method 700 for determining the pose of an object's coordinate system relative to a reference coordinate system, according to some embodiments of the present disclosure, is shown. Figure 7 As shown, some or all of the steps in method 700 can be performed by a data processing device (e.g., Figure 1The data processing module 120 shown is... Figure 15 The processor 1520 shown is used to execute the method. Some or all of the steps in method 700 can be implemented by software, firmware, and / or hardware. In some embodiments, method 700 can be executed by a robot system (e.g., Figure 15 The surgical robot system 1500 shown is executed. In some embodiments, method 700 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 15 The processor 1520 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0096] See Figure 7 In step 701, based on the roll angle of the pose identifier coordinate system relative to the object coordinate system and the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system, the three-dimensional coordinates of the multiple pose identifiers in the object coordinate system are determined. It can be understood that, given the roll angle of the pose identifier coordinate system relative to the object coordinate system, the three-dimensional coordinates of the multiple pose identifiers in the pose identifier coordinate system can be transformed into their three-dimensional coordinates in the object coordinate system through coordinate transformation.

[0097] In step 703, the pose of the object relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the object coordinate system. The pose of the object relative to the reference coordinate system can be understood as the pose of the object coordinate system relative to the reference coordinate system. In some embodiments, the specific implementation of step 703 is similar to the specific implementation of step 601 in method 600.

[0098] In some embodiments, the three-dimensional coordinates of the multiple pose markers in a pose marker coordinate system are determined based on the distribution of the multiple pose markers. For example, see [link to example description]. Figure 8 Each pose marker is located on the circumference of section circle 822, the center and radius r of which are known. The center of section circle 822 is set as the origin of the pose marker coordinate system. The XY plane lies on section circle 822, and the X-axis can be specified as pointing from the origin to any known pose marker (e.g., pose marker P8). Therefore, the three-dimensional coordinates of each pose marker in the pose marker coordinate system can be determined based on the distribution of multiple pose markers. For example, as shown... Figure 8 As shown, the three-dimensional coordinates of pose marker P8 in the pose marker coordinate system are (r, 0, 0). The three-dimensional coordinates of the other pose markers in the pose marker coordinate system can be calculated using the following formula:

[0099] m i =[r·cos((i-1)·β)r·sin((i-1)·β)0]T (5)

[0100] Where, m i Let P8 be the starting point, and let β be the three-dimensional coordinates of the i-th pose marker in the pose marker coordinate system; β is the angle around the axis between adjacent pose markers.

[0101] Figure 9 A flowchart of a method 900 for identifying pose identifiers according to some embodiments of the present disclosure is shown. Figure 9 As shown, some or all of the steps in method 900 can be performed by a data processing device (e.g., Figure 1 The data processing module 120 shown is... Figure 15 The processor 1520 shown is used to execute the method. Some or all of the steps in method 900 can be implemented by software, firmware, and / or hardware. In some embodiments, method 900 can be executed by a robot system (e.g., Figure 15 The surgical robot system 1500 shown is executed. In some embodiments, method 900 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 15 The processor 1520 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0102] See Figure 9 In step 901, multiple candidate pose identifiers are determined from the positioning image. In some embodiments, candidate pose identifiers may refer to features of possible pose identifiers obtained after preliminary processing or preliminary identification of the positioning image.

[0103] In some embodiments, method 900 may include determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of image grayscale conversion, image denoising, and image enhancement. In other embodiments, a Region of Interest (ROI) may be extracted from the localization image first, and multiple candidate pose identifiers may be determined from the ROI.

[0104] For example, image preprocessing may include: cropping a Region of Interest (ROI) from a localization image and converting the ROI into a corresponding grayscale image. The ROI can be the entire localization image, or it can be a region within a certain range determined based on multiple pose markers from the previous frame (e.g., the localization image from the previous image processing cycle). For instance, during preprocessing of a localization image that is not the first frame, the ROI range is a region within a certain distance centered on a virtual point formed by the coordinates of multiple pose markers from the previous image processing cycle. This certain distance range can be a fixed multiple of the average spacing distance of the pose markers, such as twice. It should be understood that the predetermined multiple can also be a variable multiple of the average spacing distance of multiple candidate pose markers in the previous image processing cycle.

[0105] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be calculated according to the following formula:

[0106]

[0107] Where ε is a set constant, for example, set to 2; I x I 45 I y I n45 These are the first derivatives of the pixel in the four directions: 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively; the first and second derivatives can be obtained by performing a convolution operation on each pixel within the ROI.

[0108] In some embodiments, the Region of Interest (ROI) is divided into multiple sub-images. For example, a non-maximum suppression method can be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI can be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the localization image or ROI can also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels. The pixel with the largest CL value in each sub-image can be determined, and the pixel with the largest CL value in each sub-image can be compared with a first threshold to determine the set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold can be set to 0.06. It should be understood that the first threshold can also be set to other values. In some embodiments, pixels with a CL value greater than the first threshold can be used as candidate pose identifiers.

[0109] See Figure 9 In step 903, an initial pose identifier is identified from multiple candidate pose identifiers based on a pose pattern matching template. In some embodiments, the pose pattern matching template is used to match the image at one of the candidate pose identifiers to determine the candidate pose identifier that meets the preset pose pattern matching degree standard as the initial pose identifier.

[0110] In some embodiments, the pose pattern matching template is a standard pose pattern template, and the pose pattern matching and the image of the region near the pose identifier have the same or similar features. If the matching degree between the pose pattern matching template and the image of the region near the candidate pose identifier reaches a preset pose pattern matching degree standard, it can be considered that the image of the region near the candidate pose identifier has the same or similar features as the pose pattern matching template, and thus the current candidate pose identifier can be considered as a pose identifier.

[0111] In some embodiments, the pixel with the largest CL value in the pixel set is determined as a candidate pose identifier to be matched. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value is selected as the candidate pose identifier. After determining the candidate pose identifier, a pose pattern matching template is used to match the image at the candidate pose identifier. If a preset pose pattern matching degree standard is met, the candidate pose identifier is determined as the identified initial pose identifier. If the candidate pose identifier does not meet the preset matching degree standard, the pixel with the second largest CL value is selected as the candidate pose identifier, and the pose pattern matching template is used to match the image at the candidate pose identifier. This process is repeated until the initial pose identifier is identified.

[0112] In some embodiments, method 900 includes: determining the edge orientation of the candidate pose identifier. For example, as... Figure 10 As shown, Figure 10 It includes a pose identifier pattern 1011, and the candidate pose identifier is... Figure 10 Corner point P in 10 Then the edge direction of the corner point can refer to the direction of the edge forming the corner point, such as... Figure 10 The direction indicated by the dashed arrow.

[0113] In some embodiments, the edge direction can be determined by the first-order derivative (I0) of each pixel in the X and Y directions of the planar coordinate system with respect to a certain neighborhood (e.g., 10×10 pixels) centered on the pixel corresponding to the candidate pose identifier. x and I y The edge direction can be determined using the following formula:

[0114]

[0115] Among them, the first derivative (I) x and I y This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by performing a convolution operation on the edge direction I of each pixel within the neighborhood range. angle and the corresponding weight I weight Clustering calculations are performed to obtain the edge direction of the pixel, and weight I is selected. weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if multiple edge directions exist, then weight I is selected. weight The I corresponding to the largest proportion of multiple classes angle As the edge direction.

[0116] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0117] In some embodiments, method 900 includes: rotating a pose pattern matching template according to an edge direction. Rotating the pose pattern matching template according to an edge direction can align the pose pattern matching template with the image at the candidate pose marker.

[0118] The edge direction of the candidate pose identifier can be used to determine the orientation of the pattern corresponding to the candidate identifier in the positioning image. In some embodiments, rotating the pose pattern matching template according to the edge direction can adjust the pose pattern matching template to be the same as or nearly the same as the image orientation at the candidate pose identifier to facilitate image matching.

[0119] See Figure 9 In step 905, starting from the initial pose identifier, a pose identifier is searched from the remaining candidate pose identifiers.

[0120] Figure 11 A flowchart of a method 1100 for searching pose identifiers according to some embodiments of the present disclosure is shown. Figure 11 As shown, some or all of the steps in method 1100 can be performed by a data processing device (e.g., Figure 1 The data processing module 120 shown is... Figure 15 The processor 1520 shown is used to execute the method. Some or all of the steps in method 1100 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1100 can be executed by a robot system (e.g., Figure 15 The surgical robot system 1500 shown is executed. In some embodiments, method 1100 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 15 The processor 1520 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0121] See Figure 11 In step 1101, starting from the initial pose identifier, a second pose identifier is determined from the remaining candidate pose identifiers. In some embodiments, starting from the initial pose identifier, the second pose identifier is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following: directly in front of the initial pose identifier (corresponding to a 0° angle direction), directly behind it (corresponding to a 180° angle direction), directly above it (90° angle direction), directly below it (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0122] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions, with each search direction v sn It can be calculated using the following formula:

[0123] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (8)

[0124] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent pose identifiers among multiple pose identifiers determined in the previous frame. For example, the predetermined search direction can be calculated using the following formula:

[0125]

[0126] Among them, (x i ,y i ) represents the two-dimensional coordinates of multiple pose identifiers determined in the previous frame (or the previous image processing cycle); n last The number of multiple pose markers determined in the previous frame; v s1 The first set search direction; v s2 This is the second search direction set.

[0127] In some embodiments, such as Figure 12 As shown, P is identified by its initial pose. 121 Using the coordinates of the given location as the starting point, search for the second pose identifier P in the set search direction. 122 The coordinate position can specifically include: the initial pose identifier P 121 The coordinates are used as the starting point for the search, and the search box is used to... Figure 12 (The dashed box in the image) moves in the set search direction V with a certain search step size. 121 Search for a pose identifier. If there is at least one candidate pose identifier within the search box, then the candidate pose identifier with the highest likelihood value at the corner of the search box is selected as the second pose identifier P. 122 With the search box constrained to a suitable size, P is identified by its initial pose. 121 The coordinates of P are used as the starting point for the second pose identification. 122 During the search, the candidate pose identifier with the highest corner likelihood value among the candidate pose identifiers appearing in the search box is more likely to be the pose identifier. Therefore, the candidate pose identifier with the highest corner likelihood value in the search box can be considered the second pose identifier P. 122To improve data processing speed. In other embodiments, to improve the accuracy of pose identifier recognition, when at least one candidate pose identifier exists in the search box, the candidate pose identifier with the largest corner likelihood value among the candidate pose identifiers appearing in the search box is selected for corner identification to determine whether the candidate pose identifier with the largest corner likelihood value is a pose identifier. For example, a similarity judgment is made using a pose pattern matching template and a pattern within a certain range at the candidate pose identifier with the largest corner likelihood value. A candidate pose identifier that meets the similarity judgment criteria can be considered as the second pose identifier P found in the search. 122 .

[0128] In some embodiments, continue reading Figure 12 The size of the search box can be gradually increased (i.e., the search range gradually increases). The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0129] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, therefore the pose pattern matching template used can be the same checkerboard pattern, utilizing the grayscale distribution G of the pose pattern matching template. M The pixel neighborhood grayscale distribution G of the pixel at the candidate pose identifier image The correlation coefficient (CC) between pixels is used to determine the grayscale distribution G of the pixel neighborhood. image This represents the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The specific formula is as follows:

[0130]

[0131] Where Var is the variance function and Cov is the covariance function.

[0132] In some embodiments, when the CC value is less than 0.8, that is, the grayscale distribution in the neighborhood of the pixel is less correlated with the pose pattern matching template, the pixel at the candidate pose identifier with the largest corner likelihood value is determined not to be a pose identifier; otherwise, the pixel at the candidate pose identifier with the largest corner likelihood value is considered to be a pose identifier and is recorded as the second pose identifier.

[0133] See Figure 11In step 1103, a search direction is determined based on the initial pose identifier and the second pose identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of the initial pose identifier and moving away from the second pose identifier. The second search direction may be a direction starting from the coordinate position of the second pose identifier and moving away from the first pose identifier. For example, Figure 12 The search direction V shown 122 .

[0134] In step 1105, starting from the initial pose identifier or the second pose identifier, a pose identifier is searched from the remaining candidate pose identifiers along the search direction. In some embodiments, if the coordinate position of the first pose identifier is used as the new starting point, the first search direction described above can be used as the search direction for the pose identifier. If the coordinate position of the second pose identifier is used as the new starting point, the second search direction described above can be used as the search direction for the pose identifier. In some embodiments, a new pose identifier is searched (e.g., Figure 12 The third pose identifier P in 123 The specific implementation of the coordinate position can refer to the implementation of searching for the second pose identifier in step 1101. In some embodiments, the search step size can be the distance L1 between the initial pose identifier and the second pose identifier.

[0135] In some embodiments, the search for pose identifiers stops when the number of found pose identifiers is greater than or equal to a pose identifier count threshold. For example, the search for pose identifiers stops when four pose identifiers are found (identified).

[0136] In some embodiments, the search for the Nth pose identifier stops when the search distance is greater than a predetermined multiple of the distance between the (N-1)th and (N-2)th pose identifiers, where N ≥ 3. For example, the search termination condition could be that the search distance is greater than twice the distance between the coordinates of the first two pose identifiers. Thus, the maximum search distance for the third pose identifier is twice the distance between the initial pose identifier and the second pose identifier. If no pose identifier is found after reaching this search distance, it is considered that the third pose identifier has not been found and the search ends.

[0137] In some embodiments, if the total number of found pose identifiers is greater than or equal to a set threshold (e.g., a set value of 4), then it is considered that enough pose identifiers have been successfully identified. If the total number of found pose identifiers is less than the set value, then the search based on the initial pose identifier in the above steps is considered unsuccessful, and it is necessary to redetermine new pose identifiers from the candidate pose identifiers. Then, based on the redetermined pose identifiers, the remaining pose identifiers are searched. The method for redetermining new pose identifiers can be specifically referred to the description of method 900 in the above embodiments, and the method for searching the remaining pose identifiers based on the new pose identifiers can be specifically referred to the description of method 1100 in the above embodiments.

[0138] In some embodiments, after a pose identifier is found or identified, the process may further include subpixel localization of the identified pose identifier to improve its positional accuracy. For example, if the pose identifier is a corner point in a pose identifier pattern, subpixel localization of each corner point can improve the coordinate accuracy of each pose identifier.

[0139] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of pose identifiers (e.g., corner points) after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, where the extreme points of the function are subpixel points. The fitting function can be as follows:

[0140] S(x,y)=ax 2 +by 2 +cx+dy+exy+f (11)

[0141]

[0142] Where S(x, y) is the fitting function for the CL values ​​of all pixels in each ROI, and a, b, c, d, e, and f are coefficients; x c The x-coordinate and y-coordinate of the pose identifier c The y-coordinate is the pose identifier.

[0143] Figure 13 A flowchart of a method 1300 for identifying angle markers according to some embodiments of the present disclosure is shown. Figure 13 As shown, some or all of the steps in method 1300 can be performed by a data processing device (e.g., the data processing module 120 shown in FIG1). Figure 15 The processor 1520 shown is used to execute the method. Some or all of the steps in method 1300 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1300 may be executed by a robot system (e.g., Figure 15The surgical robot system 1500 shown is executed. In some embodiments, method 1300 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 15 The processor 1520 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0144] See Figure 13 In step 1301, an imaging transformation relationship is determined based on the two-dimensional coordinates of multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system. In some embodiments, the pose marker coordinate system may be the pose marker coordinate system detailed in the embodiment shown in method 500. For example, the pose marker coordinate system is as follows: Figure 4 As shown. In some embodiments, the imaging transformation relationship can refer to the transformation relationship between three-dimensional coordinates in the pose identifier coordinate system and two-dimensional coordinates in the positioning image. It should be understood that, based on the imaging transformation relationship, two-dimensional coordinates in the positioning image can also be transformed into three-dimensional coordinates in the pose identifier coordinate system.

[0145] In some embodiments, the number of multiple pose markers can be greater than or equal to four. For example, the imaging transformation relationship can be obtained based on the two-dimensional coordinates of the four pose markers in the positioning image and the four corresponding three-dimensional coordinates in the pose marker coordinate system.

[0146] In some embodiments, the three-dimensional coordinates in the pose identification coordinate system can be determined by the method detailed in the embodiment shown in method 700.

[0147] See Figure 13 In step 1303, based on the imaging transformation relationship, the three-dimensional coordinates and positional relationships of multiple pose markers in the pose marker coordinate system, multiple angle marker candidate regions are determined in the positioning image. In some embodiments, based on the three-dimensional coordinates and positional relationships of multiple pose markers in the pose marker coordinate system, multiple angle marker candidate three-dimensional coordinates are determined in the pose marker coordinate system.

[0148] In some embodiments, the positional association can be the positional association detailed in the above embodiments. For example, the positional association is the correspondence between angle identifiers and pose identifiers along the axial direction. For example, based on the three-dimensional coordinates of multiple pose identifiers in the pose identifier coordinate system, a certain distance along the axial direction can determine multiple three-dimensional coordinates in the pose identifier coordinate system. These three-dimensional coordinates are multiple candidate three-dimensional coordinates of angle identifiers. For example, see... Figure 2If the positional relationship is such that the angle marker and the corresponding pose marker maintain a certain distance along the Z-axis, then, given the position of the pose marker, the position obtained by moving a certain step along the positive or negative direction of the Z-axis can be considered as the approximate position of the predicted angle marker in the pose marker coordinate system.

[0149] In some embodiments, multiple candidate regions for angle markers are determined in the positioning image based on imaging transformation relationships and multiple candidate 3D coordinates for angle markers.

[0150] In some embodiments, multiple candidate two-dimensional coordinates for angle markers are obtained in the positioning image based on the imaging transformation relationship and multiple candidate three-dimensional coordinates for angle markers. In some embodiments, multiple candidate angle marker regions are determined based on the multiple candidate two-dimensional coordinates for angle markers. For example, a region of a certain size (e.g., 5×5 pixels, 10×10 pixels, etc.) is determined in the positioning image with each candidate two-dimensional coordinate for angle markers as the center as the candidate angle marker region. In some embodiments, the region of a certain size is greater than or equal to the size of the imaged angle marker pattern. The size of the imaged angle marker pattern can be obtained based on the actual size of the angle marker pattern and the imaging transformation relationship.

[0151] See Figure 13 In step 1305, candidate regions are identified from multiple angles, and the angle identifiers are recognized.

[0152] In some embodiments, method 1300 includes

[0153] The pixel with the highest corner likelihood value in each angle identifier candidate region is determined to form a pixel set. In some embodiments, the corner likelihood value of a pixel may be calculated when performing method 900, or it may be recalculated according to the method detailed in method 900. Method 1300 further includes determining the angle identifier candidate region corresponding to the pixel with the highest corner likelihood value in the pixel set as the angle identifier candidate region to be identified. Method 1300 further includes matching the angle identifier candidate regions to be identified with multiple angle pattern matching templates to identify angle identifiers.

[0154] In some embodiments of this disclosure, by determining multiple candidate regions of angular feature patterns, angular markers can be identified in multiple candidate regions of angular feature patterns, avoiding the need to identify angular markers across the entire image range and improving the speed of data processing.

[0155] In some embodiments, the angle marker patterns are patterns with different graphic features. Multiple angle pattern matching templates can refer to standard angle pattern templates that have the same or similar graphic features and correspond to the multiple angle marker patterns respectively. For example... Figure 2As shown in the image.

[0156] In some embodiments, any one of the template matching algorithms, such as the squared difference matching method, normalized squared difference matching method, correlation matching method, normalized correlation matching method, correlation coefficient matching method, and normalized correlation coefficient matching method, can be used to perform matching operations between the angle pattern matching template and the angle identifier candidate region.

[0157] In some embodiments, since the angle pattern matching template and the angle marker pattern have the same or similar graphic features, the pattern information of the angle marker may include the pattern information of the corresponding angle pattern matching template. For example, the shape of the angle pattern matching template, identifiable features in the image, etc.

[0158] In some embodiments, the corner point corresponding to the candidate region of the angle identifier that successfully matches a specific angle pattern matching template is used as the identified angle identifier. The pattern information of the specific angle pattern matching template is used as the pattern information of the angle identifier pattern corresponding to the identified angle identifier. In some embodiments, there is a one-to-one correspondence between each angle pattern matching template and the angle around the axis identified by the corresponding angle identifier pattern. The first angle around the axis is determined based on the specific angle pattern matching template or the pattern information of the angle identifier pattern corresponding to the identified angle identifier.

[0159] In some embodiments, method 1300 includes: in response to a matching failure, determining the angle identifier candidate region corresponding to the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as the angle identifier candidate region to be identified. In some embodiments, after determining the new angle identifier candidate region to be identified, multiple angle pattern matching templates are used to match the angle identifier candidate region to be identified, respectively, to identify the angle identifier.

[0160] In some embodiments, a first pose identifier that has a positional association with the angle identifier is determined based on the angle identifier candidate region where the angle identifier is located. In some embodiments, multiple angle identifier candidate regions respectively correspond to at least one of multiple identified pose identifiers. After determining the angle identifier candidate region where the angle identifier is located, the first pose identifier can then be determined according to the correspondence between the multiple angle identifier candidate regions and the multiple pose identifiers.

[0161] In some embodiments of this disclosure, a computer device is also provided, including a memory for storing at least one instruction; and a processor coupled to the memory for executing the at least one instruction to perform some or all of the steps in the positioning method of this disclosure, such as... Figure 5 , Figure 6 , Figure 7 , Figure 9 , Figure 11 and Figure 13 Some or all of the steps in the method disclosed herein.

[0162] Figure 14 A schematic diagram of a computer device 1400 according to some embodiments of the present disclosure is shown. See also Figure 14 The computer device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including random access memory (RAM) 1402 and read-only memory (ROM) 1403, and a system bus 1405 connecting the various components. The computer device 1400 also includes an input / output system and a mass storage device 1407 for storing the operating system 1412, application programs 1414, and other program modules 1415. The input / output devices include an input / output controller 1410 mainly composed of a display 1408 and input devices 1409.

[0163] Mass storage device 1407 is connected to central processing unit 1401 via a mass storage controller (not shown) connected to system bus 1405. Mass storage device 1407 and its associated computer-readable media provide non-volatile storage for computer devices. That is, mass storage device 1407 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0164] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0165] Computer device 1400 can be connected to network 1412 via network interface unit 1411 connected to system bus 1405.

[0166] The system memory 1404 or mass storage device 1407 is also used to store one or more instructions. The central processing unit 1401 implements all or part of the steps of the positioning method in some embodiments of this disclosure by executing the one or more instructions.

[0167] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the positioning method of some embodiments of this disclosure, such as... Figure 5 , Figure 6 , Figure 7 , Figure 9 Figure 11 and Figure 13 Some or all of the steps in the method disclosed herein.

[0168] In some embodiments, this disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory including a computer program (instructions) executable by a processor of a computer device to perform the methods shown in various embodiments of this application. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0169] Figure 15 A schematic diagram of a surgical robot system 1500 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [link to relevant documentation]. Figure 15 The surgical robot system 1500 may include a surgical tool 1550, an image acquisition unit 1510, and a processor 1520. The surgical tool 1550 may include an arm 1540, an actuator 1530 disposed at the distal end of the arm 1540, and at least one angle marker and multiple pose markers disposed on the distal portion of the arm 1540. The image acquisition unit 1510 can be used to acquire positioning images of the arm 1540. The processor 1520 is connected to the image acquisition unit 1510 and is used to perform some or all of the steps in the positioning methods of some embodiments of this disclosure, such as... Figure 5 , Figure 6 , Figure 7 , Figure 9 , Figure 11 and Figure 13 Some or all of the steps in the method disclosed herein.

[0170] While specific embodiments of this disclosure have been illustrated and described by way of example, it will be apparent to those skilled in the art that many other changes and modifications can be made without departing from the spirit and scope of this disclosure. Therefore, all such changes and modifications falling within the scope of this disclosure are included in the appended claims.

Claims

1. A positioning method for determining pose, comprising: Acquire the location image; In the positioning image, multiple pose markers located on the object are identified; Based on the plurality of pose identifiers, an angle identifier located on the object is identified, and the angle identifier has a positional association with the first pose identifier among the plurality of pose identifiers; as well as Based on the angle identifier and the multiple pose identifiers, the pose of the object relative to the reference coordinate system is determined; Determining the pose of the object relative to the reference coordinate system based on the angle identifier and the plurality of pose identifiers includes: Based on the angle identifier and the multiple pose identifiers, determine the roll angle of the pose identifier coordinate system relative to the object coordinate system; Based on the multiple pose identifiers, determine the pose of the pose identifier coordinate system relative to the reference coordinate system; and Based on the roll angle of the pose identifier coordinate system relative to the object coordinate system and the pose of the pose identifier coordinate system relative to the reference coordinate system, the pose of the object relative to the reference coordinate system is determined. or Based on the angle identifier and the multiple pose identifiers, determine the roll angle of the pose identifier coordinate system relative to the object coordinate system; Based on the roll angle of the pose identifier coordinate system relative to the object coordinate system and the three-dimensional coordinates of the plurality of pose identifiers in the pose identifier coordinate system, the three-dimensional coordinates of the plurality of pose identifiers in the object coordinate system are determined; and The pose of the object relative to the reference coordinate system is determined based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the object coordinate system.

2. The positioning method according to claim 1, determining the pose of the pose identifier coordinate system relative to the reference coordinate system includes: Based on the two-dimensional coordinates of the plurality of pose markers in the positioning image and the three-dimensional coordinates of the plurality of pose markers in the pose marker coordinate system, the pose of the pose marker coordinate system relative to the reference coordinate system is determined.

3. The positioning method according to claim 1 or 2, comprising: Determine the first angle around the axis that the angle marker represents in the object coordinate system; Determine the second angle around the axis marked by the first pose identifier in the pose identifier coordinate system; as well as Based on the first and second axial angles, the roll angle of the pose identifier coordinate system relative to the object coordinate system is determined.

4. The positioning method according to claim 3, comprising: Based on the pattern information of the angle marker, the first angle around the axis is determined.

5. The positioning method according to claim 1 or 2, wherein the location association includes: The axial correspondence between the angle identifier and the first pose identifier.

6. The positioning method according to claim 1, comprising: Based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system, the imaging transformation relationship is determined; Based on the imaging transformation relationship, the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system, and the positional association relationship, multiple angle marker candidate regions are determined in the positioning image. as well as Candidate regions are identified from the multiple angles, and the angle identifiers are recognized.

7. The positioning method according to claim 6, comprising: Based on the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system and the positional relationship, multiple candidate three-dimensional coordinates of angle markers are determined in the pose marker coordinate system. as well as Based on the imaging transformation relationship and the three-dimensional coordinates of the multiple angle marker candidates, the multiple angle marker candidate regions are determined in the positioning image.

8. The positioning method according to claim 6 or 7, comprising: The pixel with the largest corner likelihood value in each of the candidate regions of the angle identifier is determined to form a pixel set; The candidate region for the angle identifier corresponding to the pixel with the largest corner likelihood value in the pixel set is determined as the candidate region for the angle identifier to be identified. as well as Multiple angle pattern matching templates are used to match the candidate regions of the angle identifiers to be identified, so as to identify the angle identifiers.

9. The positioning method according to claim 8, comprising: In response to a matching failure, the candidate region for the angle identifier corresponding to the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as the candidate region for the angle identifier to be identified.

10. The positioning method according to claim 6 or 7, comprising: Based on the candidate region of the angle identifier where the angle identifier is located, the first pose identifier that has a positional relationship with the angle identifier is determined.

11. The positioning method according to claim 1 or 2, comprising: Based on the distribution of the multiple pose markers, the three-dimensional coordinates of the multiple pose markers in the pose marker coordinate system are determined.

12. The positioning method according to claim 1, comprising: Multiple candidate pose identifiers are determined from the positioning image; Based on the pose pattern matching template, an initial pose identifier is identified from the plurality of candidate pose identifiers; as well as Starting from the initial pose identifier, a pose identifier is searched from the remaining candidate pose identifiers.

13. The positioning method according to claim 12, comprising: Starting from the initial pose identifier, a second pose identifier is determined from the remaining candidate pose identifiers; Based on the initial pose identifier and the second pose identifier, the search direction is determined; and Starting from the initial pose identifier or the second pose identifier, a pose identifier is searched from the remaining candidate pose identifiers in the search direction.

14. The positioning method according to claim 12 or 13, comprising: If the number of known pose identifiers is greater than or equal to the pose identifier number threshold, the search for pose identifiers is stopped.

15. The positioning method according to any one of claims 1, 2, 6, 7, 12, and 13, comprising: Based on the pose of the object relative to the reference coordinate system, the pose of the end of the object relative to the reference coordinate system is determined.

16. The positioning method according to any one of claims 1, 2, 6, 7, 12, and 13, wherein the plurality of pose markers and the angle markers are arranged axially on the object.

17. The positioning method according to any one of claims 1, 2, 6, 7, 12, and 13, wherein the plurality of pose markers and the angle markers are disposed on the outer surface of the columnar portion of the object.

18. A computer device, the computer device comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory, is configured to execute the at least one instruction to perform the method as described in any one of claims 1-17.

19. A computer-readable storage medium storing at least one instruction, which is executed by a processor to cause a computer to perform the method as described in any one of claims 1-17.

20. A surgical robot system, comprising: A surgical tool, the surgical tool comprising an arm body, an actuator disposed at the distal end of the arm body, and at least one angle marker and multiple pose markers disposed on the distal portion of the arm body; Image acquisition device, used to acquire positioning images of the arm body; as well as A processor, connected to the image acquisition unit, is configured to perform the method as described in any one of claims 1-17 to determine the pose of the end portion of the arm.

Images