Surgical robot positioning method, device, system, equipment, medium and product

Abstract

The invention discloses a surgical robot positioning method, device, system and equipment, a medium and a product, and belongs to the technical field of medical equipment. The method comprises the following steps: determining a registration result of a current visual image and a medical image and an operation path therein; determining an initial mapping result of the operation path in a first coordinate system where the target visual marker is located; based on the posture of the target visual marker and the posture of the mechanical arm visual marker, the mapping positions of two coordinate points including the starting point in the initial mapping result in a second coordinate system where the mechanical arm visual marker is located are determined; determining a target offset matrix of the second coordinate system relative to the first coordinate system according to the positions of the two reference points on the axis of the tubular fixed structure in the second coordinate system and the mapping positions; and determining a reference mapping result of the initial mapping result in the second coordinate system based on the target offset matrix, and determining tail end attitude adjustment information corresponding to the reference mapping result. The positioning accuracy of the surgical robot can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and particularly to a positioning method, device, system, equipment, medium and product for a surgical robot. Background Art

[0002] A surgical robot is a high-tech medical device that combines robot technology, computer technology, medical imaging technology, and minimally invasive surgery technology, and is widely used in the field of surgical operations.

[0003] In recent years, although the performance of surgical robots has been improved to a high level in all aspects, their real-time positioning accuracy still cannot meet the requirements of highly precise surgeries, such as anterior cruciate ligament reconstruction, posterior cruciate ligament reconstruction, etc. Summary of the Invention

[0004] The present invention provides a positioning method, device, system, equipment, medium and product for a surgical robot to solve the problem that the positioning accuracy of existing surgical robots cannot meet the requirements of highly precise surgeries.

[0005] According to one aspect of the present invention, a positioning method for a surgical robot is provided, including:

[0006] Determining the registration result of the current visual image of the implementation target and the medical image, and the surgical path in the registration result, where the surgical path is a part of the target surgical path pre-planned in the medical image;

[0007] Determining the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the implementation target;

[0008] If the attitude of the target visual identifier changes, then based on the attitude of the target visual marker and the attitude of the robotic arm visual marker, determining the first mapping position and the second mapping position of two coordinate points including the starting point in the initial mapping result in the second coordinate system, where the second coordinate system is the coordinate system where the robotic arm visual marker is located;

[0009] Determining the target offset matrix of the second coordinate system relative to the first coordinate system according to the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position and the second mapping position, where the tubular fixing structure is used to fix the surgical instrument and is arranged at the end of the fixing rod, and the other end of the fixing rod is fixed to the end of the robotic arm;

[0010] Based on the target offset matrix, determining the reference mapping result of the initial mapping result in the second coordinate system, and determining the end attitude adjustment information corresponding to the reference mapping result.

[0011] According to another aspect of the present invention, there is provided a surgical robot positioning device, comprising:

[0012] A registration module for determining the registration result of the current visual image of the implementation target and the medical image, and the surgical path in the registration result, where the surgical path is a part of the target surgical path pre-planned in the medical image;

[0013] A first mapping module for determining the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the implementation target;

[0014] A mapping position determination module for, if the pose of the target visual identifier changes, determining the first mapping position and the second mapping position of two coordinate points including the starting point in the second coordinate system based on the pose of the target visual marker and the pose of the robotic arm visual marker, where the second coordinate system is the coordinate system where the robotic arm visual marker is located;

[0015] A target offset matrix module for determining the target offset matrix of the second coordinate system relative to the first coordinate system according to the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position and the second mapping position, where the tubular fixing structure is used to fix the surgical instrument and is arranged at the end of the fixing rod, and the other end of the fixing rod is fixed to the end of the robotic arm;

[0016] An adjustment information determination module for determining the reference mapping result of the initial mapping result in the second coordinate system based on the target offset matrix, and determining the end pose adjustment information corresponding to the reference mapping result.

[0017] According to another aspect of the present invention, there is provided a surgical robot system, the system comprising:

[0018] A target visual marker rigidly connected to the implementation target;

[0019] A robotic arm provided with a robotic arm visual marker and an end positioning device, where the end positioning device is arranged on a fixing rod, one end of the fixing rod is fixed to the end of the robotic arm, and the other end is fixedly connected to a tubular fixing structure for fixing the surgical instrument;

[0020] A visual device for real-time collecting the current visual image including the implementation target, the target visual marker, the robotic arm visual marker and the end positioning device;

[0021] A processor, configured to execute the method described in any embodiment, and complete the attitude adjustment of the end of the robotic arm based on the end attitude adjustment information.

[0022] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the surgical robot positioning method described in any embodiment of the present invention.

[0026] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the surgical robot positioning method described in any embodiment of the present invention when executed.

[0027] According to another aspect of the present invention, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the surgical robot positioning method described in any embodiment.

[0028] For the technical solution of the surgical robot positioning method provided by the embodiments of the present invention, if the attitude of the target visual marker changes, it means that the target offset matrix between the first coordinate system where the target visual marker is located and the second coordinate system where the robotic arm visual marker is located changes accordingly; based on the attitude of the target visual marker and the attitude of the robotic arm visual marker, the mapping positions of the starting point of the initial mapping result in the first coordinate system and another coordinate point in the second coordinate system are determined, and at the same time, in the coordinate system where the end positioning device of the robotic arm is located, since the tubular fixing structure connects the surgical instruments therein to the end of the surgical instrument through a fixing rod, the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system can be used to quickly and accurately determine the current target offset matrix; based on this target offset matrix, the initial mapping result in the first coordinate system can be directly and accurately mapped to the second coordinate system, improving the mapping speed and accuracy of the initial mapping result, thereby improving the determination speed and accuracy of the end attitude adjustment information, realizing the accurate and real-time positioning of the end of the robotic arm; allowing surgical operations on the patient's implementation target without fixing the patient, quickly responding to the attitude changes of the patient's implementation target during the operation, and improving the safety and success rate of the operation.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become readily understood from the following description. Description of the Drawings

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0031] Figure 1 is a schematic structural diagram of a surgical robot system provided by an embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of the positional relationship between a vision device and a vision marker provided by an embodiment of the present invention;

[0033] Figure 3 is a schematic structural diagram of an end positioning device provided by an embodiment of the present invention;

[0034] Figure 4 is a schematic structural diagram of a vision marker provided by an embodiment of the present invention;

[0035] Figure 5A is a flowchart of a surgical robot positioning method provided by an embodiment of the present invention;

[0036] Figure 5B is a schematic diagram of the distribution of reference points provided by an embodiment of the present invention;

[0037] Figure 6 is a flowchart of a method for determining a manipulator transformation matrix provided by an embodiment of the present invention;

[0038] Figure 7 is a schematic structural diagram of a surgical robot positioning device provided by an embodiment of the present invention;

[0039] Figure 8 is another schematic structural diagram of a surgical robot positioning device provided by an embodiment of the present invention;

[0040] Figure 9 is a schematic structural diagram of an electronic device for implementing the surgical robot positioning method of an embodiment of the present invention. Detailed Embodiments

[0041] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0043] To facilitate the description of the technical solution, the surgical robot system in the embodiments of the present invention will be introduced first. As Figures 1-4 shown, the system includes a target visual marker 11, an arm visual marker (not shown), a terminal positioning device 12, an arm 2, a visual device 3 and a processor (not shown); the target visual marker 11 is rigidly connected to the implementation target 01; the arm 2 is provided with the terminal positioning device 12, the terminal positioning device 12 is arranged on a fixed rod 131, one end of the fixed rod 131 is fixed to the end of the arm 2, and the other end is fixedly connected to a tubular fixing structure 132, and the tubular fixing structure 132 is used to fix the surgical instrument; the visual device 3 is used to collect in real time the current visual image including the implementation target 01, the target visual marker 11, the arm visual marker and the terminal positioning device 12; the processor (not shown) is configured to execute the surgical robot positioning method described in any of the following embodiments, and complete the attitude adjustment of the arm end based on the end attitude adjustment information determined by the surgical robot positioning method.

[0044] The visual device 3 can be selected from existing visual image acquisition devices such as binoculars.

[0045] Among them, the implementation target refers to the surgical site, such as the surgical site corresponding to the cruciate ligament surgery, specifically, the surgical site corresponding to the anterior cruciate ligament surgery, the surgical site corresponding to the posterior cruciate ligament surgery, etc.

[0046] As Figure 2As shown, a Kirschner wire is used to complete the rigid connection between the target visual marker and the implementation target. Of course, other fixing methods can also be adopted to complete the rigid connection between the target visual marker and the implementation target, such as bandages and tapes.

[0047] Both the target visual marker and the robotic arm visual marker are visual markers, but there are differences in their specific structures.

[0048] In one embodiment, the visual marker 10 includes a predetermined number of positioning balls made of a reflective material, such as Figure 4 as shown. As Figure 4 、 Figure 2 shown, different visual markers all include four positioning balls, but the relative positional relationship between the four positioning balls is different. Therefore, the target visual marker, the robotic arm visual marker, and even the end visual marker can be determined by binding the relative positional relationship between the four positioning balls. Among them, Figure 4 the three arrows in it are used to represent the directions of the corresponding coordinate axes in the three-dimensional coordinate system.

[0049] Of course, other forms of visual markers can also be used, and the embodiments are not elaborated here.

[0050] As Figure 3 shown, the fixing rod 131 is L-shaped, and its long axis coincides with the axis of the flange at the end of the robotic arm. When a surgical instrument, such as a bone drill, is fixed into the tubular fixing structure 132, the axis of the surgical instrument coincides with the axis of the tubular fixing structure 132.

[0051] In one embodiment, the processor is arranged in Figure 1 the trolley 4 in, and the number is one or more.

[0052] A flange is also provided at the end of the robotic arm, and this flange is used to fix the surgical instrument; the transformation matrix between the second coordinate system and the coordinate system where the flange at the end of the robotic arm is located can be determined in advance, where the coordinate system where the flange at the end of the robotic arm is located refers to the coordinate system where the center of the flange at the end of the robotic arm is located.

[0053] Figure 5A FIG. is a flowchart of the surgical robot positioning method provided by the embodiment of the present invention. This embodiment is applicable to the situation of completing the real-time positioning of the surgical robot in a follow-up scenario. This method can be executed by a surgical robot positioning device, and the surgical robot positioning device can be implemented in the form of hardware and / or software. The surgical robot positioning device can be configured in the electronic device of the surgical robot system. As Figure 5A shown, this method includes:

[0054] S110. Determine the registration result of the current visual image of the implementation target and the medical image, as well as the surgical path in the registration result, where the surgical path is a part of the target surgical path pre-planned in the medical image.

[0055] The current visual image can be understood as the current surgical area image captured by the visual device of the surgical robot.

[0056] The medical image is a pre-taken medical image including the implementation target, such as a knee joint MR (Magnetic Resonance Imaging, MRI) image, etc.

[0057] The surgical path in the registration result is a part of the target surgical path pre-planned in the medical image.

[0058] It should be noted that the image registration between the current visual image and the medical image is completed by using the existing image registration method, and the registration result can be obtained. This embodiment does not make specific limitations here.

[0059] The surgical path corresponding to the current registration result and the surgical path corresponding to the previous registration result are adjacent or partially overlapping in the target surgical path. For the case of partial overlap, at least the starting point of the surgical path is a point in the surgical path determined by the previous registration result.

[0060] S120. Determine the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system where the target visual marker is located, and the target visual marker is rigidly connected to the implementation target.

[0061] The initial mapping result is the specific pose of the surgical path in the current first coordinate system, or rather, the surgical path in the current first coordinate system.

[0062] After the image registration result is determined, the medical image and the surgical path therein are mapped to the coordinate system where the visual device is located. In one embodiment, taking the medical image including the pre-determined total surgical path as an example. First, remove or hide the total surgical path in the medical image to obtain a pure medical image, determine the initial registration result of the pure medical image and the current visual image, and determine the image offset matrix; then, based on the image offset matrix, map the surgical path segment corresponding to the initial registration result in the medical image to the coordinate system where the visual device is located, so as to obtain the registration result including the surgical path.

[0063] In one embodiment, a Kirschner wire is used to complete the rigid connection between the target visual marker and the implementation target, and this embodiment method can simply and quickly complete the rigid connection between the target visual marker and the implementation target.

[0064] Of course, other fixing devices can also be used to complete the rigid connection between the target visual marker and the target to be implemented, which will not be elaborated in this embodiment.

[0065] It can be understood that since the target visual marker is rigidly connected to the target to be implemented, the attitude of the target visual marker can change with the change of the attitude of the target to be implemented.

[0066] S130. If the attitude of the target visual identifier changes, then based on the attitude of the target visual marker and the attitude of the robotic arm visual marker, determine the first mapping position and the second mapping position of the two coordinate points including the starting point in the initial mapping result in the second coordinate system, where the second coordinate system is the coordinate system where the robotic arm visual marker is located.

[0067] Since the attitude of the target visual marker can change with the change of the attitude of the target to be implemented, and the change of the attitude of the target visual marker will cause the coordinate system where it is located to change accordingly, the change of the attitude of the target to be implemented will cause the target offset matrix of the first coordinate system relative to the second coordinate system to change.

[0068] For this reason, in this embodiment, when it is detected that the attitude of the target visual identifier changes, it is necessary to determine the current target offset matrix for the current attitude of the target visual identifier.

[0069] When determining the target offset matrix, it is necessary to determine the first mapping position and the second mapping position of the two coordinate points including the starting point in the initial mapping result (the surgical path in the first coordinate system) in the second coordinate system based on the attitude of the target visual marker and the attitude of the robotic arm visual marker.

[0070] Among them, the two coordinate points including the starting point in the initial mapping result are preferably the starting point and the ending point in the initial mapping result to improve the determination speed of these two coordinate points.

[0071] The attitude of the target visual marker is specifically the attitude of the target visual marker under the machine vision device; the attitude of the robotic arm visual marker is specifically the attitude of the mechanical visual marker under the machine vision device.

[0072] The attitude of the target visual marker carries the offset information of the first coordinate system relative to the coordinate system where the machine vision device is located; the attitude of the robotic arm visual marker carries the offset information of the second coordinate system relative to the coordinate system where the machine vision device is located. Therefore, based on the attitude of the target visual marker and the attitude of the robotic arm visual marker, the mapping positions of the starting point and at least one other coordinate point of the initial mapping result in the second coordinate system, that is, the first mapping position and the second mapping position, can be determined.

[0073] The method for determining the pose of the target visual marker is the same as that of the robotic arm visual marker. In this embodiment, the method for determining the pose of the visual marker is introduced here. For the method for determining the pose of the target visual marker and the robotic arm visual marker, refer to the method for determining the pose of this visual marker.

[0074] The vision device identifies the positioning balls on the visual marker by the reflected light intensity, captures the relative position relationship between each positioning ball and other positioning balls, and verifies the relative position relationship with the data in the predetermined attribute file, so as to determine the attribute object of the visual marker. At the same time, based on the current vision image, the position P(x, y, z) of the visual marker under the machine vision device and the marker rotation angle (Q0, Q X , Q Y , Q Z ) are determined. Among them, the attribute object refers to whether the current visual marker is the target visual marker or the robotic arm visual marker.

[0075] The rotation angle of the visual marker is converted into a rotation matrix through a formula, which is specifically as follows:

[0076]

[0077] Among them, n is the component of the unit vector of the rotation axis, θ is the rotation angle, and the relationship between n, θ and (Q0, Q X , Q Y , Q Z ) is:

[0078] Q0 = cos(θ / 2);

[0079] Q X = n x sin(θ / 2);

[0080] Q Y = n y sin(θ / 2);

[0081] Q Z = n Z sin(θ / 2);

[0082] Among them, Q0, Q X , Q Y and Q Z are quaternions. Q0 is the real part of the quaternion, Q X is the component of the quaternion on the X axis, Q Y is the component of the quaternion on the Y axis, Q Z is the component of the quaternion on the Z axis. Substitute Q0, Q X , Q Y and Q Z into the above rotation matrix to obtain:

[0083]

[0084] The position P(x, y, z) of the visual marker and the rotation matrix R (n,θ) constitute the pose M of a visual marker, and at this time, M represents the complete pose of the corresponding visual marker in the second coordinate system.

[0085] S140. Determine the target offset matrix of the second coordinate system relative to the first coordinate system according to the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position and the second mapping position. The tubular fixing structure is used to fix the surgical instrument and is arranged at the end of the fixing rod, and the other end of the fixing rod is fixed to the end of the robotic arm.

[0086] The tubular fixing structure can be a fixing mechanism such as a sleeve.

[0087] The two reference points include a first reference point and a second reference point. As Figure 5B shown, the first reference point is the intersection of the axis of the tubular fixing structure 132 and the axis of the flange at the end of the robotic arm; starting from this intersection, move L along the axis of the tubular fixing structure 132 towards the implementation end of the surgical instrument to obtain the second reference point; L is the length of the surgical instrument, and the implementation end of the surgical instrument is the part that comes into contact with the surgical area during the operation.

[0088] The first reference point is configured to correspond to the starting point of the initial mapping result.

[0089] In one embodiment, the current target offset matrix is determined through the following steps: determine the positions of the first reference point and the second reference point on the axis of the tubular fixing structure in the second coordinate system respectively; determine the first deviation data between the position of the first reference point and the first mapping position; use the direction from the position of the first reference point to the position of the second reference point as the first vector direction, and use the direction from the first mapping position to the second mapping position as the second vector direction; determine the first rotation information between the first vector direction and the second vector direction; determine the target offset matrix according to the first deviation data and the first rotation information.

[0090] Since the coordinates of the first reference point and the first mapping position are both in the second coordinate system, the deviation data between the two can be determined, and this deviation data is used as the first deviation data.

[0091] Among them, the first vector direction can be expressed as: The second vector direction can be expressed as:

[0092] In one embodiment, the first rotation information is determined through the following steps:

[0093] Step a1: Determine the first rotation angle of the second vector direction relative to the first vector direction.

[0094] Determine the first rotation angle through the following formula:

[0095]

[0096] where is the first rotation angle.

[0097] Step a2: Determine the rotation axis corresponding to the first vector direction and the second vector direction.

[0098] Among them, the calculation formula for the rotation axis between the first vector direction and the second vector direction is:

[0099]

[0100] Step a3: Normalize the rotation axis to obtain the normalized rotation axis.

[0101] The normalization formula is:

[0102]

[0103] Set the rotation axis which can be expressed as: where k target_frame_x is the rotation component under the x coordinate axis; k target_frame_y is the rotation component under the y coordinate axis; k arget_frame_z is the rotation component under the z coordinate axis.

[0104] Step a4: Determine the first rotation information corresponding to the first rotation angle and the normalized rotation axis.

[0105] In one embodiment, the Rodrigues formula is used to determine the first rotation information, specifically:

[0106]

[0107] where I is the identity matrix and K is the skew-symmetric matrix of the rotation axis, specifically:

[0108]

[0109] Substitute the identity matrix and the skew-symmetric matrix K of the rotation axis into the above first rotation information determination formula to obtain the required first rotation information.

[0110] After the first deviation data and the first rotation information are determined, the target offset matrix can be determined according to the first deviation data and the first rotation information.

[0111] Specifically, set the first deviation data as (x target_a y target_a z target_a ), then the target offset matrix can be expressed as;

[0112]

[0113] S150. Based on the target offset matrix, determine the reference mapping result of the initial mapping result in the second coordinate system, and determine the end - pose adjustment information corresponding to the reference mapping result.

[0114] In one embodiment, after the target offset matrix is determined, based on the pre - determined robotic arm transformation matrix between the second coordinate system and the coordinate system where the end - flange of the robotic arm is located, map the reference mapping result to the coordinate system where the end - flange of the robotic arm is located to obtain the target mapping result; determine the end - pose adjustment information for the end of the robotic arm based on this target mapping result.

[0115] In another embodiment, after the target mapping result is determined, filter the target mapping result to obtain an updated target mapping result; determine the end - pose adjustment information for the end of the robotic arm according to the updated target mapping result.

[0116] Exemplarily, filter the target mapping result through the EWMA filtering algorithm (Exponentially Weighted Moving Average filtering algorithm) to obtain an updated target mapping result. This filtering process can filter out the jitter and noise generated during the surgical implementation process, ensuring the stability of the multi - joint robotic arm during the surgical implementation positioning process. Specifically: y[M n = α·x[M n +(1 - α)·y[M n - 1], where y[M n is the filtered target mapping result, x[M n is the target mapping result, and α is the filtering coefficient.

[0117] After the target mapping result is determined, convert the target mapping result into the angles that each joint of the multi - joint robotic arm needs to reach through the inverse kinematics algorithm, so as to obtain the end - pose adjustment information. Then, generate a control instruction combination according to this end - pose adjustment information, and adjust each joint angle through the control instruction combination to drive the end - positioning device to complete the real - time positioning of the end of the robotic arm.

[0118] In one embodiment, if the pose of the target visual marker does not change, then based on the latest target offset matrix, determine the reference mapping result of the initial mapping result in the second coordinate system, and determine the end - pose adjustment information corresponding to the reference mapping result.

[0119] Specifically, if the pose of the target visual marker does not change, it means that the corresponding target offset matrix does not change. Therefore, the latest target offset matrix can be directly read, and then based on this latest target offset matrix, the initial mapping result is mapped from the first coordinate system to the second coordinate system to improve the mapping speed of the initial mapping result.

[0120] In one embodiment, the initial coordinate system where the robotic arm visual marker is located is determined; the initial coordinate system is compensated based on a pre-determined safety policy matrix to obtain a second coordinate system, and the safety policy matrix is used to adjust the robotic arm positioning joint to a predetermined safe pose while ensuring that the pose of the end of the robotic arm remains unchanged; the compensated initial coordinate system is used as the second coordinate system. The safety matrix policy can ensure that the visual marker is always in a visible state during the surgical procedure, avoiding occlusion and data loss, and enhancing the safety of the surgery.

[0121] For the technical solution of the surgical robot positioning method provided by the embodiments of the present invention, if the pose of the target visual marker changes, it means that the target offset matrix between the first coordinate system where the target visual marker is located and the second coordinate system where the robotic arm visual marker is located changes accordingly; based on the pose of the target visual marker and the pose of the robotic arm visual marker, the mapping positions of the starting point of the initial mapping result in the first coordinate system and another coordinate point in the second coordinate system are determined respectively. At the same time, in the coordinate system where the end positioning device of the robotic arm is located, since the tubular fixing structure connects the surgical instruments therein to the end of the surgical instrument through a fixing rod, the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system can be used to quickly and accurately determine the current target offset matrix; based on this target offset matrix, the initial mapping result in the first coordinate system can be directly and accurately mapped to the second coordinate system, improving the mapping speed and accuracy of the initial mapping result, thereby improving the determination speed and accuracy of the end pose adjustment information, achieving accurate and real-time positioning of the end of the robotic arm; allowing surgical operations on the patient's implementation target without fixing the patient, quickly responding to the pose changes of the patient's implementation target during the surgical procedure, and enhancing the safety and success rate of the surgery.

[0122] Figure 6 It is a flowchart of the robotic arm matrix determination method provided by the embodiments of the present invention. This embodiment is used to illustrate the determination method of the robotic arm transformation matrix between the second coordinate system and the coordinate system where the flange at the end of the robotic arm is located. As Figure 6 shown, the method includes:

[0123] S210. Respectively use the coordinate positions of two reference points in the coordinate system where the flange at the end of the robotic arm is located as the first intermediate position and the second intermediate position, and use the direction from the first intermediate position to the second intermediate position as the third vector direction.

[0124] Determine the coordinate positions of the two reference points in the coordinate system where the end flange of the robotic arm is located, and use them as the first target position and the second target position respectively.

[0125] In this embodiment, the coordinate system where the end flange of the robotic arm is located specifically refers to the coordinate system where the center of the end flange of the robotic arm is located.

[0126] S220. Determine the second rotation angle of the second coordinate system relative to the coordinate system where the end flange of the robotic arm is located according to the first vector direction and the third vector direction.

[0127] The rotation angle of the second coordinate system relative to the x-axis of the coordinate system where the end flange of the robotic arm is located can be expressed as:

[0128]

[0129] The rotation angle of the second coordinate system relative to the y-axis of the coordinate system where the end flange of the robotic arm is located can be expressed as:

[0130]

[0131] The rotation angle of the second coordinate system relative to the z-axis of the coordinate system where the end flange of the robotic arm is located can be expressed as:

[0132]

[0133] Therefore, the second rotation angle of the second coordinate system relative to the coordinate system where the end flange of the robotic arm is located can be expressed as:

[0134]

[0135] S230. Based on the second rotation angle, map the first intermediate position and the second intermediate position to the coordinate system where the end flange of the robotic arm is located to obtain the first target position and the second target position.

[0136] After the second rotation angle is determined, use the second rotation angle to rotate the first intermediate position and the second intermediate position respectively to map them to the coordinate system where the end flange of the robotic arm is located to obtain the first target position and the second target position.

[0137] S240. Determine the second deviation data of the second coordinate system relative to the coordinate system where the end flange of the robotic arm is located according to the first target position and the first reference position.

[0138] Since both the first target position and the first reference position are in the coordinate system where the end flange of the robotic arm is located, calculate the position deviation between the two, and use this position deviation as the second deviation data of the second coordinate system relative to the coordinate system where the end flange of the robotic arm is located.

[0139] S250. Determine the offset matrix of the second coordinate system relative to the coordinate system of the end flange of the robotic arm according to the second rotation angle and the second deviation data, and use this offset matrix as the robotic arm transformation matrix.

[0140] An offset matrix can be determined according to the second rotation angle and the second deviation data, and this offset matrix is the robotic arm transformation matrix of the second coordinate system relative to the coordinate system of the end flange of the robotic arm.

[0141] In the embodiments of the present invention, according to the poses of two reference points in the tubular fixing structure of the end positioning device being the same in the second coordinate system and the coordinate system of the end flange of the robotic arm, the robotic arm transformation matrix of the second coordinate system relative to the coordinate system of the end flange of the robotic arm is determined, and the entire determination process is simple and fast.

[0142] Figure 7 It is a schematic structural diagram of the surgical robot positioning device provided by the embodiments of the present invention. As Figure 7 shown, the device includes:

[0143] A registration module 31, configured to determine the registration result of the current visual image of the implementation target and the medical image, and the surgical path in the registration result, where the surgical path is a part of the target surgical path pre-planned in the medical image;

[0144] A first mapping module 32, configured to determine the initial mapping result of the surgical path in the current first coordinate system, where the first coordinate system is the coordinate system of the target visual marker, and the target visual marker is rigidly connected to the implementation target;

[0145] A mapping position determination module 33, configured to, if the pose of the target visual identifier changes, determine the first mapping position and the second mapping position of two coordinate points including the starting point in the second coordinate system based on the pose of the target visual marker and the pose of the robotic arm visual marker, where the second coordinate system is the coordinate system of the robotic arm visual marker;

[0146] A target offset matrix module 34, configured to determine the target offset matrix of the second coordinate system relative to the first coordinate system according to the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position and the second mapping position, where the tubular fixing structure is used to fix the surgical instrument and is arranged at the end of the fixing rod, and the other end of the fixing rod is fixed to the end of the robotic arm;

[0147] An adjustment information determination module 35, configured to determine the reference mapping result of the initial mapping result in the second coordinate system based on the target offset matrix, and determine the end pose adjustment information corresponding to the reference mapping result.

[0148] In one embodiment, as Figure 8 shown, the device further includes a direct mapping module 36, and the direct mapping module 36 is configured to:

[0149] If the pose of the target visual identifier does not change, determine a reference mapping result of the initial mapping result in the second coordinate system based on the latest target offset matrix;

[0150] Determine end pose adjustment information for the end of the robotic arm according to the reference mapping result.

[0151] In one embodiment, the target offset matrix module 34 includes:

[0152] A reference point position unit, configured to determine the positions of a first reference point and a second reference point on the axis of the tubular fixed structure in the second coordinate system respectively;

[0153] A deviation data unit, configured to determine first deviation data between the position of the first reference point and the first mapping position;

[0154] A vector direction unit, configured to use the direction from the position of the first reference point to the position of the second reference point as a first vector direction, and use the direction from the first mapping position to the second mapping position as a second vector direction;

[0155] A rotation information unit, configured to determine first rotation information between the first vector direction and the second vector direction;

[0156] A target offset matrix unit, configured to determine a target offset matrix according to the first deviation data and the first rotation information.

[0157] In one embodiment, the rotation information unit is configured to:

[0158] Determine a first rotation angle of the second vector direction relative to the first vector direction;

[0159] Determine a rotation axis corresponding to the first vector direction and the second vector direction;

[0160] Perform normalization processing on the rotation axis to obtain the normalized rotation axis;

[0161] Determine first rotation information corresponding to the first rotation angle and the normalized rotation axis.

[0162] In one embodiment, it further includes a coordinate system determination unit, and the target offset matrix module 34 determines the second coordinate system based on the coordinate system determination unit. The coordinate system determination unit is specifically configured to:

[0163] Determine the initial coordinate system where the robotic arm vision marker is located;

[0164] Compensate the initial coordinate system based on a pre-determined safety policy matrix to obtain the second coordinate system, where the safety policy matrix is used to adjust the robotic arm positioning joint to a predetermined safe posture while ensuring the posture of the robotic arm end remains unchanged;

[0165] Use the compensated initial coordinate system as the second coordinate system.

[0166] In one embodiment, the adjustment information determination module 35 is specifically configured to:

[0167] Based on the robotic arm transformation matrix between the pre-determined second coordinate system and the coordinate system where the robotic arm end flange is located, map the reference mapping result to the coordinate system where the robotic arm end flange is located to obtain a target mapping result;

[0168] Filter the target mapping result to obtain the updated target mapping result;

[0169] Determine the end posture adjustment information for the robotic arm end according to the updated target mapping result.

[0170] For the technical solution of the surgical robot positioning device provided by the embodiments of the present invention, if the posture of the target vision marker changes, it means that the target offset matrix between the first coordinate system where the target vision marker is located and the second coordinate system where the robotic arm vision marker is located changes accordingly; based on the target vision marker posture and the robotic arm vision marker posture, determine the mapping positions of the starting point of the initial mapping result in the first coordinate system and another coordinate point in the second coordinate system respectively, and at the same time, in the coordinate system where the robotic arm end positioning device is located, since the tubular fixing structure connects the surgical instruments inside it to the surgical instrument end through the fixing rod, the positions of two reference points on the axis of the tubular fixing structure in the second coordinate system can be used to quickly and accurately determine the current target offset matrix; based on this target offset matrix, the initial mapping result in the first coordinate system can be directly and accurately mapped to the second coordinate system, improving the mapping speed and accuracy of the initial mapping result, thereby improving the determination speed and accuracy of the end posture adjustment information, and realizing the accurate real-time positioning of the robotic arm end; allowing surgical operations on the patient's implementation target without fixing the patient, quickly responding to the posture changes of the patient's implementation target during the operation, and improving the safety and success rate of the operation.

[0171] The surgical robot positioning device provided by the embodiments of the present invention can execute the surgical robot positioning method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0172] Figure 9 FIG. 2 shows a schematic structural diagram of an electronic device 90 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0173] As Figure 9 shown, the electronic device 90 includes at least one processor 91 and a memory communicatively connected to the at least one processor 91, such as a read-only memory (ROM) 92, a random access memory (RAM) 93, etc. The memory stores a computer program executable by the at least one processor. The processor 91 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 92 or the computer program loaded from the storage unit 98 into the random access memory (RAM) 93. In the RAM 93, various programs and data required for the operation of the electronic device 90 can also be stored. The processor 91, the ROM 92, and the RAM 93 are connected to each other through a bus 94. An input / output (I / O) interface 95 is also connected to the bus 94.

[0174] Multiple components in the electronic device 90 are connected to the I / O interface 95, including: an input unit 96, such as a keyboard, a mouse, etc.; an output unit 97, such as various types of displays, speakers, etc.; a storage unit 98, such as a magnetic disk, an optical disc, etc.; and a communication unit 99, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 99 allows the electronic device 90 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0175] The processor 91 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 91 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 91 executes the various methods and processes described above, such as the surgical robot positioning method.

[0176] In some embodiments, the surgical robot positioning method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 98. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 90 via the ROM 92 and / or the communication unit 99. When the computer program is loaded into the RAM 93 and executed by the processor 91, one or more steps of the surgical robot positioning method described above may be performed. Alternatively, in other embodiments, the processor 91 may be configured to perform the surgical robot positioning method by any other suitable means (e.g., by means of firmware).

[0177] The various embodiments of the systems and techniques described above in this document may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on a chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] The computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs may be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on the remote machine or server.

[0179] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0181] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by any form or medium of digital data communication (e.g., a communication network). Examples of the communication network include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0182] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0183] An embodiment of the present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the surgical robot positioning method provided in any embodiment of the present application.

[0184] In the process of implementing the computer program product, computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0185] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0186] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A surgical robot positioning method, characterized in that: include: Determine a registration result of a current visual image of an implementation target and a medical image, and a surgical path in the registration result, wherein the surgical path is a portion of a target surgical path pre-planned in the medical image; Determine an initial mapping result of the surgical path in a current first coordinate system, where the first coordinate system is a coordinate system where a target visual marker is located, and the target visual marker is rigidly connected to the implementation target; If the posture of the target visual marker changes, based on the posture of the target visual marker and the posture of the robot arm visual marker, determine the first mapping position and the second mapping position of two coordinate points including the starting point in the initial mapping result in the second coordinate system, where the second coordinate system is the coordinate system where the robot arm visual marker is located; Determining a target offset matrix of the second coordinate system compared to the first coordinate system according to positions of two reference points on the axis of the tubular fixing structure in the second coordinate system, the first mapping position and the second mapping position, wherein the tubular fixing structure is used to fix the surgical instrument and is disposed at the end of a fixing rod, and the other end of the fixing rod is fixed to the end of the robotic arm; Based on the target offset matrix, a reference mapping result of the initial mapping result in the second coordinate system is determined, and terminal posture adjustment information corresponding to the reference mapping result is determined.

2. The method according to claim 1, characterized in that After determining the initial mapping result of the surgical path in the current first coordinate system, the method further includes: If the posture of the target visual marker has not changed, determining a reference mapping result of the initial mapping result in the second coordinate system based on the latest target offset matrix; The end posture adjustment information for the end of the robotic arm is determined according to the reference mapping result.

3. The method according to claim 1, characterized in that The step of determining a target offset matrix of the second coordinate system compared to the first coordinate system according to the positions of two reference points on the axis of the tubular fixed structure in the second coordinate system, the first mapping position and the second mapping position, comprises: Determine the positions of a first reference point and a second reference point on the axis of the tubular fixed structure in the second coordinate system respectively; determining first deviation data between the position of the first reference point and the first mapped position; The direction from the first reference point to the second reference point is used as a first vector direction, and the direction from the first mapping position to the second mapping position is used as a second vector direction; determining first rotation information between the first vector direction and the second vector direction; A target offset matrix is ​​determined according to the first deviation data and the first rotation information.

4. The method according to claim 3, characterized in that The determining first rotation information between the first vector direction and the second vector direction includes: determining a first rotation angle of the second vector direction relative to the first vector direction; determining a rotation axis corresponding to the first vector direction and the second vector direction; Normalizing the rotation axis to obtain the normalized rotation axis; First rotation information corresponding to the first rotation angle and the normalized rotation axis is determined.

5. The method according to claim 1, characterized in that The second coordinate system is determined by the following steps, including: Determine the initial coordinate system where the robot arm visual marker is located; The initial coordinate system is compensated based on a predetermined safety strategy matrix to obtain the second coordinate system, wherein the safety strategy matrix is ​​used to adjust the manipulator arm positioning joint to a predetermined safety posture while ensuring that the posture of the end of the manipulator remains unchanged; The compensated initial coordinate system is used as the second coordinate system.

6. The method according to claim 1, characterized in that The determining of the terminal posture adjustment information corresponding to the reference mapping result includes: Based on a predetermined robot arm transformation matrix between the second coordinate system and the coordinate system where the robot arm end flange is located, mapping the reference mapping result to the coordinate system where the robot arm end flange is located to obtain a target mapping result; Filtering the target mapping result to obtain an updated target mapping result; The end posture adjustment information for the end of the robotic arm is determined according to the updated target mapping result.

7. A surgical robot positioning device, characterized in that: include: A registration module, used to determine the registration result of the current visual image of the implementation target and the medical image, and the surgical path in the registration result, wherein the surgical path is a part of the target surgical path pre-planned in the medical image; A first mapping module, used to determine an initial mapping result of the surgical path in a current first coordinate system, where the first coordinate system is a coordinate system where a target visual marker is located, and the target visual marker is rigidly connected to the implementation target; A mapping position determination module, for determining, if the posture of the target visual marker changes, based on the posture of the target visual marker and the posture of the robotic arm visual marker, the first mapping position and the second mapping position of two coordinate points including the starting point in the initial mapping result in the second coordinate system, where the second coordinate system is the coordinate system where the robotic arm visual marker is located; a target offset matrix module, for determining a target offset matrix of the second coordinate system compared to the first coordinate system according to positions of two reference points on the axis of the tubular fixed structure in the second coordinate system, the first mapping position and the second mapping position, wherein the tubular fixed structure is used to fix the surgical instrument and is disposed at the end of a fixing rod, and the other end of the fixing rod is fixed to the end of the robotic arm; The adjustment information determination module is used to determine a reference mapping result of the initial mapping result in the second coordinate system based on the target offset matrix, and determine the terminal posture adjustment information corresponding to the reference mapping result.

8. A surgical robot system, characterized in that: The system comprises: Target visual marker, rigidly connected to the implementation target; A robotic arm, provided with a robotic arm visual marker and an end positioning device, wherein the end positioning device is arranged on a fixing rod, one end of the fixing rod is fixed to the end of the robotic arm, and the other end is fixedly connected to a tubular fixing structure, wherein the tubular fixing structure is used to fix the surgical instrument; A visual device, used for collecting in real time a current visual image including the implementation target, the target visual marker, the robot arm visual marker and the end positioning device; A processor is configured to execute the method described in any one of claims 1-7, and complete the posture adjustment of the end of the robotic arm based on the end posture adjustment information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the surgical robot positioning method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the surgical robot positioning method described in any one of claims 1-7 when executed.

11. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the surgical robot positioning method according to any one of claims 1-7.

Images