Surgical robot positioning method, apparatus, system, device, medium and product

Abstract

This invention discloses a surgical robot positioning method, device, system, equipment, medium, and product, belonging to the field of medical device technology. The method includes: determining the registration result between a current visual image and a medical image, and the surgical path therein; determining the initial mapping result of the surgical path in a first coordinate system where a target visual marker is located; based on the pose of the target visual marker and the pose of the robotic arm visual marker, determining the mapping positions of two coordinate points, including the starting point, in the initial mapping result in a second coordinate system where the robotic arm visual marker is located; determining the target offset matrix of the second coordinate system relative to the first coordinate system based on the positions of two reference points on the axis of the tubular fixed structure in the second coordinate system and the aforementioned mapping positions; determining a reference mapping result of the initial mapping result in the second coordinate system based on the target offset matrix; and determining the end-effector posture adjustment information corresponding to the reference mapping result. This invention can improve the accuracy of surgical robot positioning.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a surgical robot positioning method, device, system, equipment, medium and product. Background Technology

[0002] Surgical robots are high-tech medical devices that combine robotics, computer technology, medical imaging technology, and minimally invasive surgical techniques, and are widely used in the field of surgery.

[0003] In recent years, although the performance of surgical robots has improved significantly in all aspects, their real-time positioning accuracy still cannot meet the requirements of high-precision surgeries, such as anterior cruciate ligament reconstruction and posterior cruciate ligament reconstruction. Summary of the Invention

[0004] This invention provides a surgical robot positioning method, device, system, equipment, medium, and product to solve the problem that the positioning accuracy of existing surgical robots cannot meet the requirements of high-precision surgery.

[0005] According to one aspect of the present invention, a surgical robot positioning method is provided, comprising:

[0006] The registration result between the current visual image of the target and the medical image is determined, as well as the surgical path in the registration result, wherein the surgical path is a part of the pre-planned target surgical path in the medical image;

[0007] Determine the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the implementation target;

[0008] If the pose of the target visual marker changes, then based on the pose of the target visual marker and the pose of the robotic arm visual marker, the first mapping position and the second mapping position of the two coordinate points including the starting point in the initial mapping result are determined in the second coordinate system, where the second coordinate system is the coordinate system where the robotic arm visual marker is located.

[0009] Based on the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position, and the second mapping position, the target offset matrix of the second coordinate system relative to the first coordinate system is determined. The tubular fixing structure is used to fix surgical instruments and is set at the end of the fixing rod. The other end of the fixing rod is fixed to the end of the robotic arm.

[0010] Based on the target offset matrix, a reference mapping result of the initial mapping result in the second coordinate system is determined, and end attitude adjustment information corresponding to the reference mapping result is determined.

[0011] According to another aspect of the present invention, a surgical robot positioning device is provided, comprising:

[0012] The registration module is used to determine the registration result between the current visual image of the target and the medical image, and the surgical path in the registration result, wherein the surgical path is a part of the pre-planned target surgical path in the medical image;

[0013] The first mapping module is used to determine the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the implementation target.

[0014] The mapping position determination module is used to determine, based on the pose of the target visual marker and the pose of the robotic arm visual marker, the first mapping position and the second mapping position of the two coordinate points, including the starting point, in the initial mapping result in the second coordinate system, respectively, if the pose of the target visual marker changes; the second coordinate system is the coordinate system where the robotic arm visual marker is located.

[0015] The target offset matrix module is used to determine the target offset matrix of the second coordinate system relative to the first coordinate system based on the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position and the second mapping position, respectively. The tubular fixing structure is used to fix surgical instruments and is set at the end of the fixing rod. The other end of the fixing rod is fixed to the end of the robotic arm.

[0016] The adjustment information determination module is used to determine the reference mapping result of the initial mapping result in the second coordinate system based on the target offset matrix, and to determine the end attitude adjustment information corresponding to the reference mapping result.

[0017] According to another aspect of the present invention, a surgical robot system is provided, the system comprising:

[0018] The target visual marker is rigidly connected to the implementation target;

[0019] The robotic arm is equipped with a robotic arm visual marker and an end-effector positioning device. The end-effector positioning device is mounted on a fixed rod. One end of the fixed rod is fixed to the end of the robotic arm, and the other end is fixedly connected to a tubular fixing structure. The tubular fixing structure is used to fix surgical instruments.

[0020] A vision device is used to acquire, in real time, a current visual image including the target, the target visual marker, the robotic arm visual marker, and the end-effector positioning device;

[0021] The processor is configured to execute the method described in any embodiment and to perform attitude adjustment of the robotic arm end based on the end-effector attitude adjustment information.

[0022] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the surgical robot positioning method according to any embodiment of the present invention.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the surgical robot positioning method according to any embodiment of the present invention.

[0027] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the surgical robot positioning method according to any embodiment.

[0028] The technical solution of the surgical robot positioning method provided in this invention addresses the issue that if the pose of the target visual marker changes, it means that the target offset matrix between the first coordinate system where the target visual marker is located and the second coordinate system where the robotic arm visual marker is located changes accordingly. Based on the poses of the target visual marker and the robotic arm visual marker, the mapping positions of the starting point of the initial mapping result in the first coordinate system and another coordinate point in the second coordinate system are determined. Simultaneously, considering the position of two reference points on the axis of the tubular fixed structure in the second coordinate system (since the tubular fixed structure connects the surgical instruments inside to the end of the surgical instruments via a fixed rod), the current target offset matrix can be quickly and accurately determined. Based on this target offset matrix, the initial mapping result in the first coordinate system can be directly and accurately mapped to the second coordinate system, improving the mapping speed and accuracy of the initial mapping result. This, in turn, improves the determination speed and accuracy of the end-effector pose adjustment information, achieving accurate real-time positioning of the robotic arm end-effector. This allows for surgical operations on the patient's target without patient fixation, rapidly responding to changes in the patient's target pose during surgery, and improving the safety and success rate of the surgery.

[0029] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the surgical robot system provided in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram illustrating the positional relationship between the visual device and the visual marker provided in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the end-effector positioning device provided in an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the structure of the visual markers provided in the embodiments of the present invention;

[0035] Figure 5A This is a flowchart of a surgical robot positioning method provided according to an embodiment of the present invention;

[0036] Figure 5B This is a schematic diagram of the reference point distribution provided according to an embodiment of the present invention;

[0037] Figure 6 This is a flowchart of a method for determining the transformation matrix of a robotic arm according to an embodiment of the present invention;

[0038] Figure 7 This is a schematic diagram of the structure of a surgical robot positioning device provided according to an embodiment of the present invention;

[0039] Figure 8 This is another structural schematic diagram of the surgical robot positioning device provided according to an embodiment of the present invention;

[0040] Figure 9 This is a schematic diagram of the structure of an electronic device that implements the surgical robot positioning method of this invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] To facilitate the description of the technical solution, the surgical robot system in the embodiments of the present invention will be introduced first. For example... Figures 1-4 As shown, the system includes a target visual marker 11, a robotic arm visual marker (not shown), an end-effector positioning device 12, a robotic arm 2, a vision device 3, and a processor (not shown). The target visual marker 11 is rigidly connected to the target 01. The robotic arm 2 is equipped with an end-effector positioning device 12, which is mounted on a fixed rod 131. One end of the fixed rod 131 is fixed to the end of the robotic arm 2, and the other end is fixedly connected to a tubular fixing structure 132, which is used to fix surgical instruments. The vision device 3 is used to acquire real-time visual images including the target 01, the target visual marker 11, the robotic arm visual marker, and the end-effector positioning device 12. The processor (not shown) is configured to execute the surgical robot positioning method described in any of the following embodiments and to complete the posture adjustment of the robotic arm end based on the end-effector posture adjustment information determined by the surgical robot positioning method.

[0044] The vision device 3 can be an existing visual image acquisition device such as a binoculars.

[0045] The target of implementation refers to the surgical site, such as the surgical site corresponding to cruciate ligament surgery, specifically the surgical site corresponding to anterior cruciate ligament surgery, posterior cruciate ligament surgery, etc.

[0046] like Figure 2As shown, Kirschner wires are used to achieve a rigid connection between the target visual marker and the implementation target. Of course, other fixing methods can also be used to achieve a rigid connection between the target visual marker and the implementation target, such as bandages or tape.

[0047] Both target visual markers and robotic arm visual markers are visual markers, but their specific structures differ.

[0048] In one embodiment, the visual marker 10 includes a predetermined number of positioning balls made of reflective material, such as... Figure 4 As shown. Figure 4 , Figure 2 As shown, different visual markers all include four positioning balls, but the relative positions of the four positioning balls are different. Therefore, by binding the relative positions of the four positioning balls, it can be determined whether it is a target visual marker, a robotic arm visual marker, or even an end-effector visual marker. Figure 4 The three arrows are used to indicate the direction of the corresponding coordinate axis in the three-dimensional coordinate system.

[0049] Of course, other forms of visual markers can also be used, and examples will not be described here.

[0050] like Figure 3 As shown, the fixing rod 131 is L-shaped, and its long axis coincides with the axis of the flange at the end of the robotic arm. When a surgical instrument, such as a bone drill, is fixed inside the tubular fixing structure 132, the axis of the surgical instrument coincides with the axis of the tubular fixing structure 132.

[0051] In one embodiment, the processor is set to Figure 1 There are one or more trolleys in the cart 4.

[0052] The robotic arm also has a flange at its end, which is used to fix surgical instruments. The transformation matrix between the second coordinate system and the coordinate system where the robotic arm end flange is located can be predetermined. The coordinate system where the robotic arm end flange is located refers to the coordinate system where the center of the robotic arm end flange is located.

[0053] Figure 5A This is a flowchart of a surgical robot positioning method provided in an embodiment of the present invention. This embodiment is applicable to situations where real-time positioning of a surgical robot is achieved in a servo-driven scenario. The method can be executed by a surgical robot positioning device, which can be implemented in hardware and / or software and can be configured in the electronic equipment of the surgical robot system. Figure 5A As shown, the method includes:

[0054] S110. Determine the registration result between the current visual image of the target and the medical image, and the surgical path in the registration result, wherein the surgical path is part of the pre-planned target surgical path in the medical image.

[0055] The current visual image can be understood as an image of the current surgical area taken by the vision device of the surgical robot.

[0056] Medical images are pre-captured medical images that include the target of the procedure, such as MRI (Magnetic Resonance Imaging) images of the knee joint.

[0057] The surgical path in the registration result is part of the target surgical path planned in advance in the medical image.

[0058] It should be noted that existing image registration methods can be used to complete the image registration between the current visual image and the medical image to obtain the registration result. This embodiment does not impose any specific limitations on this.

[0059] The surgical path corresponding to the current registration result is adjacent to or partially overlaps with the surgical path corresponding to the previous registration result in the target surgical path. In the case of partial overlap, at least the starting point of the surgical path must be a point in the surgical path determined by the previous registration result.

[0060] S120. Determine the initial mapping result of the surgical path in the current first coordinate system. The first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the target.

[0061] The initial mapping result is the specific pose of the surgical path in the current first coordinate system, or in other words, the surgical path in the current first coordinate system.

[0062] After the image registration result is determined, the medical image and the surgical path within it are mapped to the coordinate system of the vision device. In one embodiment, taking a medical image including a predetermined total surgical path as an example, the total surgical path in the medical image is first removed or hidden to obtain a pure medical image. The initial registration result between this pure medical image and the current vision image is determined, and the image offset matrix is ​​determined. Then, based on the image offset matrix, the surgical path segments in the medical image corresponding to the initial registration result are mapped to the coordinate system of the vision device, thereby obtaining a registration result including the surgical path.

[0063] In one embodiment, Kirschner wires are used to achieve a rigid connection between the target visual marker and the implementation target. This embodiment can achieve a rigid connection between the target visual marker and the implementation target simply and quickly.

[0064] Of course, other fixing devices can also be used to achieve a rigid connection between the target visual marker and the target, which will not be elaborated here.

[0065] It is understandable that, since the target visual marker is rigidly connected to the target, the orientation of the target visual marker can change with the orientation of the target.

[0066] S130. If the pose of the target visual marker changes, based on the pose of the target visual marker and the pose of the robotic arm visual marker, determine the first and second mapping positions of the two coordinate points, including the starting point, in the second coordinate system, respectively. The second coordinate system is the coordinate system where the robotic arm visual marker is located.

[0067] Since the pose of the target visual marker can change with the change of the target pose, and the change of the target visual marker pose will cause the coordinate system in which it is located to change accordingly, the change of the target pose will cause the target offset matrix of the first coordinate system to change relative to the second coordinate system.

[0068] Therefore, in this embodiment, when a change in the pose of a target visual marker is detected, it is necessary to determine the current target offset matrix for the current pose of the target visual marker.

[0069] When determining the target offset matrix, it is necessary to determine the first and second mapping positions of the two coordinate points, including the starting point, in the second coordinate system based on the target visual marker pose and the robotic arm visual marker pose.

[0070] In the initial mapping result, the two coordinate points, including the starting point, are preferably the starting point and the ending point in the initial mapping result, so as to improve the determination speed of these two coordinate points.

[0071] The target visual marker posture is specifically the posture of the target visual marker under the machine vision device; the robotic arm visual marker posture is specifically the posture of the robotic visual marker under the machine vision device.

[0072] The target visual marker pose carries offset information of the first coordinate system relative to the coordinate system of the machine vision device; the robotic arm visual marker pose carries offset information of the second coordinate system relative to the coordinate system of the machine vision device. Therefore, based on the target visual marker pose and the robotic arm visual marker pose, the starting point of the initial mapping result and the mapping positions of at least one other coordinate point in the second coordinate system can be determined, namely the first mapping position and the second mapping position.

[0073] The method for determining the pose of the target visual marker is the same as that for the pose of the robotic arm visual marker. This embodiment describes the method for determining the pose of the visual marker. For the method for determining the pose of the target visual marker and the pose of the robotic arm visual marker, please refer to the method for determining the pose of the visual marker.

[0074] The vision device identifies positioning balls on a visual marker by measuring the intensity of reflected light, captures the relative positional relationship between each positioning ball and other positioning balls, and verifies this relative positional relationship with data in a predetermined attribute file to determine the attribute object of the visual marker. Simultaneously, based on the current visual image, it determines the position P(x, y, z) of the visual marker under the machine vision device and the rotation angle of the marker. , , , The attribute object refers to whether the current visual marker is the target visual marker or the robotic arm's visual marker.

[0075] The rotation angle of a visual marker is converted into a rotation matrix using a formula, as follows:

[0076] .

[0077] in, The components of the unit vector around the rotation axis. For rotation angle, , and( , , , The relationship is:

[0078]

[0079]

[0080] in, , , and It is a quaternion. Let be the real part of the quaternion. These are the components of the quaternion on the X-axis. The components of the quaternion on the Y-axis. The components of the quaternion on the Z-axis are... , , and Substituting the rotation matrix above, we get:

[0081] .

[0082] Use the position P(x, y, z) of the visual marker and the rotation matrix The pose M of a visual marker is defined as the complete pose of the corresponding visual marker in the second coordinate system.

[0083] S140. Based on the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position, and the second mapping position, respectively, the target offset matrix of the second coordinate system relative to the first coordinate system is determined. The tubular fixing structure is used to fix surgical instruments and is set at the end of the fixing rod. The other end of the fixing rod is fixed to the end of the robotic arm.

[0084] The tubular fixing structure can be a fixing mechanism such as a sleeve.

[0085] The two reference points include a first reference point and a second reference point. For example... Figure 5B As shown, the first reference point is the intersection of the axis of the tubular fixed structure 132 and the axis of the end flange of the robotic arm; starting from this intersection, the second reference point is obtained by moving L along the axis of the tubular fixed structure 132 toward the surgical instrument implementation end; L is the length of the surgical instrument, and the implementation end of the surgical instrument is the part that comes into contact with the surgical area during the operation.

[0086] The first reference point is configured to correspond to the starting point of the initial mapping result.

[0087] In one embodiment, the current target offset matrix is ​​determined by the following steps: determining the positions of a first reference point and a second reference point on the axis of the tubular fixed structure in the second coordinate system; determining the first deviation data between the position of the first reference point and the first mapped position; taking the direction from the position of the first reference point to the position of the second reference point as the first vector direction, and taking the direction from the first mapped position to the second mapped position as the second vector direction; determining the first rotation information between the first vector direction and the second vector direction; and determining the target offset matrix based on the first deviation data and the first rotation information.

[0088] Since the coordinates of the first reference point and the first mapped position are both in the second coordinate system, the deviation data between the two can be determined, and this deviation data can be used as the first deviation data.

[0089] The direction of the first vector can be represented as: The direction of the second vector can be represented as: .

[0090] In one embodiment, the first rotation information is determined through the following steps:

[0091] Step a1: Determine the first rotation angle of the second vector direction relative to the first vector direction.

[0092] The first rotation angle is determined using the following formula:

[0093] ;

[0094] in, This is the first rotation angle.

[0095] Step a2: Determine the rotation axis corresponding to the first vector direction and the second vector direction.

[0096] The formula for calculating the rotation axis between the first vector direction and the second vector direction is:

[0097] .

[0098] Step a3: Normalize the rotation axis to obtain the normalized rotation axis.

[0099] The normalization formula is:

[0100] .

[0101] Set the rotation axis It can be represented as: .in, The rotation component along the x-axis; The rotation component along the y-axis; This represents the rotational component along the z-axis.

[0102] Step a4: Determine the first rotation information corresponding to the first rotation angle and the normalized rotation axis.

[0103] In one embodiment, the first rotation information is determined using Rodriguez's formula, specifically:

[0104] ;

[0105] Where I is the identity matrix, and K is the antisymmetric matrix of the rotation axis, specifically:

[0106] ;

[0107] By substituting the identity matrix and the antisymmetric matrix K of the rotation axis into the formula for determining the first rotation information above, the required first rotation information can be obtained.

[0108] Once the first deviation data and the first rotation information are determined, the target offset matrix can be determined based on the first deviation data and the first rotation information.

[0109] Specifically, the first deviation data is set as Then the target offset matrix can be expressed as:

[0110] .

[0111] S150. Based on the target offset matrix, determine the reference mapping result of the initial mapping result in the second coordinate system, and determine the end attitude adjustment information corresponding to the reference mapping result.

[0112] In one embodiment, after the target offset matrix is ​​determined, the reference mapping result is mapped to the coordinate system of the end flange of the robotic arm based on the robotic arm transformation matrix between the predetermined second coordinate system and the coordinate system of the end flange of the robotic arm to obtain the target mapping result; based on the target mapping result, the end posture adjustment information for the end of the robotic arm is determined.

[0113] In another embodiment, after the target mapping result is determined, the target mapping result is filtered to obtain an updated target mapping result; and the end-effector posture adjustment information for the robotic arm end is determined based on the updated target mapping result.

[0114] For example, the target mapping result is filtered using the EWMA (Exponential Weighted Moving Average) filtering algorithm to obtain an updated target mapping result. This filtering process can filter out jitter and noise generated during the surgical procedure, ensuring the stability of the multi-joint robotic arm during surgical positioning. Specifically: ,in, This is the filtered target mapping result. For the target mapping result, These are the filter coefficients.

[0115] After the target mapping result is determined, the target mapping result is converted into the angle that each joint in the multi-joint robotic arm needs to reach through the inverse kinematics algorithm, thereby obtaining the end-effector posture adjustment information. Then, a combination of control commands is generated based on the end-effector posture adjustment information. The control command combination is used to adjust the angle of each joint to drive the end-effector positioning device to complete the real-time positioning of the end of the robotic arm.

[0116] In one embodiment, if the pose of the target visual marker has not changed, a reference mapping result in the second coordinate system is determined based on the latest target offset matrix, and end pose adjustment information corresponding to the reference mapping result is determined.

[0117] Specifically, if the pose of the target visual marker does not change, it means that its corresponding target offset matrix has not changed. Therefore, the latest target offset matrix can be read directly, and then the initial mapping result can be mapped from the first coordinate system to the second coordinate system based on the latest target offset matrix to improve the mapping speed of the initial mapping result.

[0118] In one embodiment, an initial coordinate system is determined for the visual markers on the robotic arm. A second coordinate system is obtained by compensating the initial coordinate system based on a pre-determined safety strategy matrix. This safety strategy matrix is ​​used to adjust the robotic arm's positioning joints to a predetermined safe posture while ensuring the end effector's posture remains unchanged. The compensated initial coordinate system is then used as the second coordinate system. This safety matrix strategy ensures that the visual markers remain visible throughout the surgical procedure, preventing occlusion and data loss, thus enhancing surgical safety.

[0119] The technical solution of the surgical robot positioning method provided in this invention addresses the issue that if the pose of the target visual marker changes, it means that the target offset matrix between the first coordinate system where the target visual marker is located and the second coordinate system where the robotic arm visual marker is located changes accordingly. Based on the poses of the target visual marker and the robotic arm visual marker, the mapping positions of the starting point of the initial mapping result in the first coordinate system and another coordinate point in the second coordinate system are determined. Simultaneously, considering the position of two reference points on the axis of the tubular fixed structure in the second coordinate system (since the tubular fixed structure connects the surgical instruments inside to the end of the surgical instruments via a fixed rod), the current target offset matrix can be quickly and accurately determined. Based on this target offset matrix, the initial mapping result in the first coordinate system can be directly and accurately mapped to the second coordinate system, improving the mapping speed and accuracy of the initial mapping result. This, in turn, improves the determination speed and accuracy of the end-effector pose adjustment information, achieving accurate real-time positioning of the robotic arm end-effector. This allows for surgical operations on the patient's target without patient fixation, rapidly responding to changes in the patient's target pose during surgery, and improving the safety and success rate of the surgery.

[0120] Figure 6 This is a flowchart illustrating a method for determining the robotic arm matrix according to an embodiment of the present invention. This embodiment is used to explain the method for determining the robotic arm transformation matrix between the second coordinate system and the coordinate system where the robotic arm's end flange is located. Figure 6 As shown, the method includes:

[0121] S210. In the coordinate system where the end flange of the robotic arm is located, the coordinate positions of the two reference points are respectively taken as the first intermediate position and the second intermediate position, and the direction from the first intermediate position to the second intermediate position is taken as the third vector direction.

[0122] Determine the coordinate positions of the two reference points in the coordinate system of the end flange of the robotic arm, and use them as the first target position and the second target position, respectively.

[0123] In this embodiment, the coordinate system of the end flange of the robotic arm specifically refers to the coordinate system of the center of the end flange of the robotic arm.

[0124] S220. Based on the first vector direction and the third vector direction, determine the second rotation angle of the second coordinate system relative to the coordinate system where the end flange of the robotic arm is located.

[0125] The rotation angle of the second coordinate system relative to the coordinate system of the end flange of the robotic arm along the x-axis can be expressed as: .

[0126] The rotation angle of the second coordinate system relative to the coordinate system of the end flange of the robotic arm can be expressed as: .

[0127] The rotation angle of the second coordinate system relative to the coordinate system of the end flange of the robotic arm along the z-axis can be expressed as:

[0128] .

[0129] Therefore, the second rotation angle of the second coordinate system relative to the coordinate system of the end flange of the robotic arm can be expressed as:

[0130] .

[0131] S230. Based on the second rotation angle, map the first intermediate position and the second intermediate position to the coordinate system where the end flange of the robotic arm is located to obtain the first target position and the second target position.

[0132] After the second rotation angle is determined, the first intermediate position and the second intermediate position are rotated using the second rotation angle respectively, so as to map them to the coordinate system where the end flange of the robotic arm is located, and thus obtain the first target position and the second target position.

[0133] S240. Based on the first target position and the first reference position, determine the second deviation data of the second coordinate system relative to the coordinate system where the end flange of the robotic arm is located.

[0134] Since both the first target position and the first reference position are in the coordinate system of the end flange of the robotic arm, the positional deviation between the two is calculated, and this positional deviation is used as the second deviation data of the second coordinate system relative to the coordinate system of the end flange of the robotic arm.

[0135] S250. Based on the second rotation angle and the second deviation data, determine the offset matrix of the second coordinate system relative to the coordinate system where the end flange of the robotic arm is located, and use the offset matrix as the transformation matrix of the robotic arm.

[0136] Based on the second rotation angle and the second deviation data, an offset matrix can be determined. This offset matrix is ​​the robot arm transformation matrix relative to the coordinate system where the end flange of the robot arm is located.

[0137] In this embodiment of the invention, based on the two reference points in the tubular fixed structure of the end-effector positioning device, the posture in the second coordinate system should be the same as the posture in the coordinate system where the end flange of the robotic arm is located. The transformation matrix of the robotic arm relative to the coordinate system where the end flange of the robotic arm is located is determined. The whole determination process is simple and fast.

[0138] Figure 7 This is a schematic diagram of the surgical robot positioning device provided in an embodiment of the present invention. Figure 7 As shown, the device includes:

[0139] The registration module 31 is used to determine the registration result between the current visual image of the target and the medical image, and the surgical path in the registration result, wherein the surgical path is a part of the pre-planned target surgical path in the medical image;

[0140] The first mapping module 32 is used to determine the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the implementation target.

[0141] The mapping position determination module 33 is used to determine, based on the pose of the target visual marker and the pose of the robotic arm visual marker, the first mapping position and the second mapping position of the two coordinate points, including the starting point, in the second coordinate system in the initial mapping result, respectively, if the pose of the target visual marker changes, the second coordinate system is the coordinate system where the robotic arm visual marker is located.

[0142] The target offset matrix module 34 is used to determine the target offset matrix of the second coordinate system relative to the first coordinate system based on the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position and the second mapping position, respectively. The tubular fixing structure is used to fix surgical instruments and is set at the end of the fixing rod. The other end of the fixing rod is fixed to the end of the robotic arm.

[0143] The adjustment information determination module 35 is used to determine the reference mapping result of the initial mapping result in the second coordinate system based on the target offset matrix, and to determine the end attitude adjustment information corresponding to the reference mapping result.

[0144] In one embodiment, such as Figure 8 As shown, the device also includes a direct mapping module 36, which is used for:

[0145] If the pose of the target visual marker has not changed, then based on the latest target offset matrix, the reference mapping result of the initial mapping result in the second coordinate system is determined;

[0146] The end-effector posture adjustment information is determined based on the reference mapping result.

[0147] In one embodiment, the target offset matrix module 34 includes:

[0148] The reference point position unit is used to determine the positions of the first reference point and the second reference point on the axis of the tubular fixed structure in the second coordinate system.

[0149] A deviation data unit is used to determine first deviation data between the position of the first reference point and the first mapped position;

[0150] The vector direction unit is used to take the direction from the position of the first reference point to the position of the second reference point as the first vector direction, and the direction from the first mapped position to the second mapped position as the second vector direction.

[0151] A rotation information unit is used to determine first rotation information between the first vector direction and the second vector direction;

[0152] The target offset matrix unit is used to determine the target offset matrix based on the first deviation data and the first rotation information.

[0153] In one embodiment, the rotation information unit is used for:

[0154] Determine a first rotation angle between the direction of the second vector and the direction of the first vector;

[0155] Determine the rotation axis corresponding to the first vector direction and the second vector direction;

[0156] The rotation axis is normalized to obtain the normalized rotation axis;

[0157] Determine the first rotation information corresponding to the first rotation angle and the normalized rotation axis.

[0158] In one embodiment, a coordinate system determination unit is further included, wherein the target offset matrix module 34 determines the second coordinate system based on the coordinate system determination unit, and the coordinate system determination unit is specifically used for:

[0159] Determine the initial coordinate system of the robotic arm's visual markers;

[0160] The initial coordinate system is compensated based on a predetermined safety strategy matrix to obtain the second coordinate system. The safety strategy matrix is ​​used to adjust the robotic arm's positioning joints to a predetermined safe posture while ensuring that the robotic arm's end-effector posture remains unchanged.

[0161] The compensated initial coordinate system is used as the second coordinate system.

[0162] In one embodiment, the adjustment information determination module 35 is specifically used for:

[0163] Based on the predetermined transformation matrix between the second coordinate system and the coordinate system where the end flange of the robotic arm is located, the reference mapping result is mapped to the coordinate system where the end flange of the robotic arm is located to obtain the target mapping result;

[0164] The target mapping result is filtered to obtain the updated target mapping result;

[0165] The end-effector attitude adjustment information is determined based on the updated target mapping results.

[0166] The technical solution of the surgical robot positioning device provided in this invention addresses the issue that changes in the pose of the target visual marker indicate a change in the target offset matrix between the first coordinate system containing the target visual marker and the second coordinate system containing the robotic arm visual marker. Based on the poses of the target visual marker and the robotic arm visual marker, the starting point of the initial mapping result in the first coordinate system and the mapping positions of another coordinate point in the second coordinate system are determined. Simultaneously, considering the position of two reference points on the axis of the tubular fixed structure in the second coordinate system (since the tubular fixed structure connects the surgical instruments within it to the end of the surgical instruments via a fixed rod), the current target offset matrix can be quickly and accurately determined. Based on this target offset matrix, the initial mapping result in the first coordinate system can be directly and accurately mapped to the second coordinate system, improving the mapping speed and accuracy of the initial mapping result. This, in turn, improves the speed and accuracy of determining the end-effector pose adjustment information, achieving accurate real-time positioning of the robotic arm end-effector. This allows for surgical operations on the patient's target without patient fixation, rapidly responding to changes in the patient's target pose during surgery, and enhancing the safety and success rate of the surgery.

[0167] The surgical robot positioning device provided in the embodiments of the present invention can execute the surgical robot positioning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0168] Figure 9A schematic diagram of an electronic device 90 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0169] like Figure 9 As shown, the electronic device 90 includes at least one processor 91 and a memory, such as a read-only memory (ROM) 92 or a random access memory (RAM) 93, communicatively connected to the at least one processor 91. The memory stores computer programs executable by the at least one processor. The processor 91 can perform various appropriate actions and processes based on the computer program stored in the ROM 92 or loaded into the RAM 93 from storage unit 98. The RAM 93 can also store various programs and data required for the operation of the electronic device 90. The processor 91, ROM 92, and RAM 93 are interconnected via a bus 94. An input / output (I / O) interface 95 is also connected to the bus 94.

[0170] Multiple components in electronic device 90 are connected to I / O interface 95, including: input unit 96, such as keyboard, mouse, etc.; output unit 97, such as various types of monitors, speakers, etc.; storage unit 98, such as disk, optical disk, etc.; and communication unit 99, such as network card, modem, wireless transceiver, etc. Communication unit 99 allows electronic device 90 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0171] Processor 91 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 91 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 91 performs the various methods and processes described above, such as surgical robot localization methods.

[0172] In some embodiments, the surgical robot positioning method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 98. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 90 via ROM 92 and / or communication unit 99. When the computer program is loaded into RAM 93 and executed by processor 91, one or more steps of the surgical robot positioning method described above may be performed. Alternatively, in other embodiments, processor 91 may be configured to perform the surgical robot positioning method by any other suitable means (e.g., by means of firmware).

[0173] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0175] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0177] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0178] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0179] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the surgical robot positioning method provided in any embodiment of this application.

[0180] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0181] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0182] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A surgical robot positioning method, characterized in that, include: The registration result between the current visual image of the target and the medical image is determined, as well as the surgical path in the registration result, wherein the surgical path is a part of the pre-planned target surgical path in the medical image; Determine the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the implementation target; If the pose of the target visual marker changes, then based on the pose of the target visual marker and the pose of the robotic arm visual marker, the first mapping position and the second mapping position of the two coordinate points including the starting point in the initial mapping result are determined in the second coordinate system, where the second coordinate system is the coordinate system where the robotic arm visual marker is located. Based on the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position, and the second mapping position, the target offset matrix of the second coordinate system relative to the first coordinate system is determined. The tubular fixing structure is used to fix surgical instruments and is set at the end of the fixing rod. The other end of the fixing rod is fixed to the end of the robotic arm. Based on the target offset matrix, determine the reference mapping result of the initial mapping result in the second coordinate system, and determine the end attitude adjustment information corresponding to the reference mapping result; The step of determining the target offset matrix of the second coordinate system relative to the first coordinate system based on the positions of two reference points on the axis of the tubular fixed structure in the second coordinate system, the first mapped position, and the second mapped position, includes: Determine the positions of the first reference point and the second reference point on the axis of the tubular fixed structure in the second coordinate system; Determine the first deviation data between the position of the first reference point and the first mapped position; The direction from the position of the first reference point to the position of the second reference point is taken as the first vector direction, and the direction from the first mapped position to the second mapped position is taken as the second vector direction. Determine the first rotation information between the first vector direction and the second vector direction; The target offset matrix is ​​determined based on the first deviation data and the first rotation information.

2. The method according to claim 1, characterized in that, After determining the initial mapping result of the surgical path in the current first coordinate system, the method further includes: If the pose of the target visual marker has not changed, then based on the latest target offset matrix, the reference mapping result of the initial mapping result in the second coordinate system is determined; The end-effector posture adjustment information is determined based on the reference mapping result.

3. The method according to claim 1, characterized in that, Determining the first rotation information between the first vector direction and the second vector direction includes: Determine a first rotation angle between the direction of the second vector and the direction of the first vector; Determine the rotation axis corresponding to the first vector direction and the second vector direction; The rotation axis is normalized to obtain the normalized rotation axis; Determine the first rotation information corresponding to the first rotation angle and the normalized rotation axis.

4. The method according to claim 1, characterized in that, The second coordinate system is determined by the following steps: Determine the initial coordinate system of the visual markers on the robotic arm; The initial coordinate system is compensated based on a predetermined safety strategy matrix to obtain the second coordinate system. The safety strategy matrix is ​​used to adjust the robotic arm's positioning joint to a predetermined safe posture while ensuring that the posture of the robotic arm's end effector remains unchanged. The compensated initial coordinate system is used as the second coordinate system.

5. The method according to claim 1, characterized in that, The determination of the end-effector attitude adjustment information corresponding to the reference mapping result includes: Based on the predetermined transformation matrix between the second coordinate system and the coordinate system where the end flange of the robotic arm is located, the reference mapping result is mapped to the coordinate system where the end flange of the robotic arm is located to obtain the target mapping result; The target mapping result is filtered to obtain the updated target mapping result; The end-effector attitude adjustment information is determined based on the updated target mapping results.

6. A surgical robot positioning device, characterized in that, include: The registration module is used to determine the registration result between the current visual image of the target and the medical image, and the surgical path in the registration result, wherein the surgical path is a part of the pre-planned target surgical path in the medical image; The first mapping module is used to determine the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the implementation target. The mapping position determination module is used to determine, based on the pose of the target visual marker and the pose of the robotic arm visual marker, the first mapping position and the second mapping position of the two coordinate points, including the starting point, in the initial mapping result in the second coordinate system, respectively, if the pose of the target visual marker changes; the second coordinate system is the coordinate system where the robotic arm visual marker is located. The target offset matrix module is used to determine the target offset matrix of the second coordinate system relative to the first coordinate system based on the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position and the second mapping position, respectively. The tubular fixing structure is used to fix surgical instruments and is set at the end of the fixing rod. The other end of the fixing rod is fixed to the end of the robotic arm. The adjustment information determination module is used to determine, based on the target offset matrix, the reference mapping result of the initial mapping result in the second coordinate system, and to determine the end attitude adjustment information corresponding to the reference mapping result; Specifically, the target offset matrix module is used for: Determine the positions of the first reference point and the second reference point on the axis of the tubular fixed structure in the second coordinate system; Determine the first deviation data between the position of the first reference point and the first mapped position; The direction from the position of the first reference point to the position of the second reference point is taken as the first vector direction, and the direction from the first mapped position to the second mapped position is taken as the second vector direction. Determine the first rotation information between the first vector direction and the second vector direction; The target offset matrix is ​​determined based on the first deviation data and the first rotation information.

7. A surgical robot system, characterized in that, The system includes: The target visual marker is rigidly connected to the implementation target; The robotic arm is equipped with a robotic arm visual marker and an end-effector positioning device. The end-effector positioning device is mounted on a fixed rod. One end of the fixed rod is fixed to the end of the robotic arm, and the other end is fixedly connected to a tubular fixing structure. The tubular fixing structure is used to fix surgical instruments. A vision device is used to acquire, in real time, a current visual image including the target, the target visual marker, the robotic arm visual marker, and the end-effector positioning device; The processor is configured to execute the method of any one of claims 1-5 and to perform attitude adjustment of the robotic arm end based on the end-effector attitude adjustment information.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the surgical robot positioning method according to any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the surgical robot positioning method according to any one of claims 1-5.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the surgical robot positioning method according to any one of claims 1-5.

Images