Percutaneous puncture surgery navigation method, device, electronic device and storage medium

Abstract

The embodiments of the present invention relate to the field of surgical robots, and disclose a percutaneous puncture surgical navigation method, device, electronic device and storage medium, including: obtaining a binocular image of the target surgical area captured by a binocular camera; using the binocular image to locate the puncture instrument and the target target, the puncture instrument is installed at the end of a robotic arm, and the target target is used to mark the target position of the puncture instrument; according to the error between the puncture instrument and the target target obtained by positioning, the movement of the robotic arm is controlled to keep the same distance between the puncture instrument and the target target during the percutaneous puncture surgery. The percutaneous puncture surgical navigation method disclosed in the present application solves the problem that the prior art method of using an infrared optical positioning system for surgical navigation requires calibration in advance, and the cost of surgical navigation is high and the operation is complicated. It can at least achieve the effect of visual servo navigation during percutaneous puncture surgery without calibration.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of surgical robots, and in particular to a percutaneous puncture surgical navigation method, device, electronic device and storage medium. Background Art

[0002] At present, the application of surgical robots in the field of percutaneous puncture surgery is gradually expanding. In actual operation, doctors generally use imaging equipment such as CT and ultrasound to formulate the needle insertion trajectory and realize image navigation of percutaneous puncture surgery. Then, this process requires the completion of the coordinate conversion of the target target, surgical instruments, and image navigation system, which is relatively complicated. In addition, in order to meet the accuracy requirements of surgical navigation, it is usually necessary to calibrate the infrared optical positioning system in advance, which makes the surgical navigation method costly and complicated to operate. In addition, the optical calibration ball used by the infrared optical positioning system will also interfere with the needle insertion operation. In addition, the relevant technical solutions usually adopt offline planning solutions, which makes it difficult to compensate for the body surface movement caused by breathing in real time, thereby affecting the alignment accuracy of the needle insertion point, limiting its application in operations with high requirements for the needle insertion path. Summary of the Invention

[0003] The purpose of the present invention is to at least provide a percutaneous puncture surgical navigation method, device, electronic device and storage medium, which can at least solve the problem that the existing technology of using infrared optical positioning systems for surgical navigation requires advance calibration, and the cost of surgical navigation is high and the operation is complicated. At least it can achieve the effect of visual servo navigation during percutaneous puncture surgery without the need for calibration.

[0004] To solve the above technical problems, at least one embodiment of the present application provides a percutaneous puncture surgery navigation method, including: obtaining a binocular image of the target surgical area captured by a binocular camera; using the binocular image to locate the puncture instrument and the target target in real time, the puncture instrument is installed at the end of a robotic arm, and the target target is used to mark the target position of the puncture instrument; based on the error between the puncture instrument and the target target obtained by positioning, the movement of the robotic arm is controlled to maintain the same distance between the puncture instrument and the target target during the percutaneous puncture surgery, thereby realizing dynamic compensation for the target displacement caused by respiratory movement.

[0005] At least one embodiment of the present application also provides a percutaneous puncture surgical navigation device, including: an image acquisition module, used to obtain a binocular image of the target surgical area captured by a binocular camera; a positioning module, used to use the binocular image to locate the puncture instrument and the target target in real time, the puncture instrument is installed at the end of the robotic arm, and the target target is used to mark the target position of the puncture instrument; a visual servo control module, used to control the movement of the robotic arm according to the error between the puncture instrument and the target target obtained by positioning, so that the puncture instrument and the target target maintain the same distance during the percutaneous puncture surgery, thereby realizing dynamic compensation for the target displacement caused by respiratory movement.

[0006] At least one embodiment of the present application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the percutaneous surgical navigation method.

[0007] At least one embodiment of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program implements the percutaneous puncture surgery navigation method when executed by a processor.

[0008] At least one embodiment of the present application provides a percutaneous puncture surgical navigation system, comprising: a robotic arm with a puncture instrument mounted on the end thereof; a binocular camera for capturing a binocular image of a target surgical area; and the electronic device described above.

[0009] The percutaneous puncture surgical navigation method provided in the embodiment of the present application uses a binocular camera to capture a binocular image of the target surgical area, uses the binocular image to locate the puncture instrument and the target target in real time, and controls the movement of the robotic arm based on the error between the puncture instrument and the target target obtained by positioning, so that the distance between the puncture instrument and the target target remains unchanged. The cost of this surgical navigation solution is significantly lower than the cost of surgical navigation using an infrared optical positioning system in the prior art, and it is simple to operate. Without the need for prior calibration, visual servo navigation can be achieved during percutaneous puncture surgery, so that the puncture instrument always maintains the same distance from the target target during percutaneous puncture surgery. To a certain extent, it can reduce the puncture error caused by chest displacement caused by respiratory movement, compensate for the body surface movement caused by breathing in real time, improve the alignment accuracy of the needle entry point, and thus improve the surgical accuracy of percutaneous puncture.

[0010] In some optional embodiments, the binocular image is used to locate the puncture instrument and the target point in real time, including: using the binocular image to calculate the coordinates of the top and end of the puncture instrument and the coordinates of the target point in the camera coordinate system through parallax; obtaining a first feature vector based on the coordinates of the top and end of the puncture instrument, and obtaining a second feature vector based on the coordinates of the target point in the camera coordinate system, the first feature vector being used to characterize the positioning information of the puncture instrument, and the second feature vector being used to characterize the positioning information of the target point. Using the three-dimensional coordinates of the puncture instrument and the target point in the camera coordinate system to obtain the feature vectors of the puncture instrument and the target point, the puncture instrument and the target point can be accurately located in three-dimensional space, providing accurate positioning information for the visual servoing process to improve puncture accuracy.

[0011] In some optional embodiments, the binocular image includes a left-eye image and a right-eye image; the coordinates of the top and end of the puncture instrument and the coordinates of the target target in the camera coordinate system are obtained by parallax calculation using the binocular image, including: inputting the left-eye image into a pre-trained key point detection model to obtain the coordinates of the top and end of the puncture instrument in the target surgical area, as well as the coordinates of the target target; using the right-eye image, calculating the parallax through a stereo matching algorithm, and combining the binocular camera intrinsic parameter matrix to convert the image coordinate system to the camera coordinate system. By using the left-eye image to identify the coordinates of the puncture instrument and the target target, and using the right-eye image to perform depth calculation, the coordinates of the puncture instrument and the target target can be converted from the image coordinate system to the camera coordinate system, providing accurate three-dimensional positioning information for the visual servo tracking process to improve puncture accuracy.

[0012] In some optional embodiments, based on the error between the puncture instrument and the target point obtained by positioning, the movement of the robotic arm is controlled to maintain a constant distance between the puncture instrument and the target point during the percutaneous puncture procedure, thereby achieving dynamic compensation for target point displacement caused by respiratory motion. The method includes: calculating the error between the first eigenvector and the second eigenvector; if the error does not meet a preset condition, calculating the movement speed of the robotic arm using a Jacobian matrix and the error, wherein the Jacobian matrix describes the differential of a first change with respect to a second change, the first change being the change in the first eigenvector in the camera coordinate system, and the second change being the change in the position of the robotic arm end in the robotic arm coordinate system; controlling the movement of the robotic arm based on the calculated movement speed of the robotic arm and updating the Jacobian matrix. Visual servoing is performed using the Jacobian matrix and the error between the puncture instrument and the target point, so that the robotic arm drives the puncture instrument to always accurately track changes in the target point, ensuring that the puncture instrument always maintains a constant distance from the target point. This can, to a certain extent, reduce puncture errors caused by chest displacement caused by respiratory motion and improve the surgical accuracy of percutaneous puncture.

[0013] In some optional embodiments, the percutaneous puncture surgical navigation method further includes: moving the end of the robotic arm to an initial position; moving each position component of the robotic arm end at the initial position once to obtain an offset posture representing the movement of each position component of the robotic arm end, and recording a first change after each movement; calculating a second change based on the offset posture and the initial position; and calculating the differential of the first change with respect to the second change to obtain an initial Jacobian matrix. The initialization of the robotic arm and the Jacobian matrix is ​​achieved, so that during the percutaneous puncture surgery, visual servoing is started based on the initial Jacobian matrix, the movement speed of the robotic arm is calculated using the Jacobian matrix and the error between the puncture instrument and the target point, the Jacobian matrix is ​​updated in real time, and the robotic arm drives the puncture instrument to always accurately track the target point.

[0014] In some optional embodiments, the binocular image includes a left image and a right image. After acquiring the binocular image of the target surgical area captured by the binocular camera, the method further includes: normalizing the left image; and resampling the normalized left image to adjust the resolution of the left image to the target resolution. This can provide suitable target image quality for subsequent processing, facilitating accurate positioning results.

[0015] In some optional embodiments, the positioning module is used to: use the binocular image to obtain the coordinates of the top and end of the puncture instrument and the coordinates of the target target in the camera coordinate system through parallax calculation; obtain a first eigenvector based on the coordinates of the top and end of the puncture instrument, and obtain a second eigenvector based on the coordinates of the target target in the camera coordinate system, the first eigenvector is used to characterize the positioning information of the puncture instrument, and the second eigenvector is used to characterize the positioning information of the target target; the visual servoing control module is used to: calculate the error between the first eigenvector and the second eigenvector; when the error does not meet the preset conditions, use the Jacobian matrix and the error to calculate the movement speed of the robotic arm, the Jacobian matrix is ​​used to describe the differential of the first change amount with respect to the second change amount, the first change amount is the change amount of the first eigenvector in the camera coordinate system, and the second change amount is the position change amount of the end of the robotic arm in the robotic arm coordinate system; control the movement of the robotic arm according to the calculated movement speed of the robotic arm, and update the Jacobian matrix. Visual servoing is performed using the Jacobian matrix and the error between the puncture instrument and the target. The robotic arm drives the puncture instrument to always accurately track the changes of the target, so that the puncture instrument always maintains the same distance from the target. To a certain extent, it can reduce the puncture error caused by chest displacement caused by respiratory movement, compensate for the surface movement caused by breathing in real time, improve the alignment accuracy of the needle entry point, and thus improve the surgical accuracy of percutaneous puncture. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0017] Figure 1 is a flowchart of a percutaneous puncture surgery navigation method provided in an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of target point pasting provided in an embodiment of the present application.

[0019] Figure 3 This is a schematic diagram of the visual servoing process provided by an embodiment of the present application;

[0020] Figure 4 Schematic diagram of a percutaneous surgical navigation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] To facilitate understanding of the embodiments of the present application, relevant content regarding percutaneous surgical navigation is first introduced here.

[0022] Image-guided percutaneous surgery is an important method for diagnosing and treating tumors. Compared with open surgery, percutaneous surgery has the advantages of less trauma, less pain, fewer complications, faster recovery, and lower cost. It is widely used for the diagnosis and treatment of vital thoracic and abdominal organs, but it still faces the following challenges:

[0023] 1. Significantly affected by the patient's respiratory movement: The patient's breathing causes the chest and upper abdomen to rise and fall, making it difficult to accurately puncture the designated skin entry point, thereby affecting the accuracy of the puncture procedure. Clinically, patients are often required to use techniques such as breath holding and respiratory gating to artificially reduce the interference of surface body movement, which is not friendly to some patients who cannot control their breathing.

[0024] 2. The doctor's hand tremors can reduce the accuracy of puncture and even lead to complications such as bleeding;

[0025] 3. Percutaneous puncture surgery is highly dependent on the doctor's experience: Clinically, the biopsy diagnosis rate of tumors with a diameter of 5mm to 10mm is only 52%. For tumors and nodules with a diameter of less than 5mm, manual puncture is difficult to guarantee the success rate of the operation, and the puncture accuracy and effect are heavily dependent on the doctor's experience.

[0026] At present, the application of surgical robots in the field of percutaneous puncture surgery is gradually expanding. In actual operation, doctors generally use imaging equipment such as CT and ultrasound to formulate the needle insertion trajectory and realize image navigation of percutaneous puncture surgery. Then, this process requires the completion of the coordinate conversion of the target target, surgical instruments, and image navigation system, which is relatively complicated. In addition, in order to meet the accuracy requirements of surgical navigation, it is usually necessary to calibrate the infrared optical positioning system in advance, which makes the surgical navigation method costly and complicated to operate. In addition, the optical calibration ball used by the infrared optical positioning system will also interfere with the needle insertion operation. In addition, the relevant technical solutions usually adopt offline planning solutions, which makes it difficult to compensate for the body surface movement caused by breathing in real time, thereby affecting the alignment accuracy of the needle insertion point, limiting its application in operations with high requirements for the needle insertion path.

[0027] Therefore, this field urgently needs a surgical navigation solution for percutaneous puncture surgery that is low-cost, easy to operate, and can cope with respiratory motion interference. It can solve the problem that surgical navigation using infrared optical positioning systems requires advance calibration, and the cost and operation of surgical navigation are high. At least it can achieve the effect of visual servo navigation during percutaneous puncture surgery without the need for calibration.

[0028] In order to solve the above technical problems, the present invention proposes a percutaneous puncture surgical navigation method. The implementation details of the percutaneous puncture surgical navigation method of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided by the present invention.

[0029] Example 1:

[0030] The percutaneous puncture surgical navigation method of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 1 As shown, including:

[0031] Step 101: Acquire a binocular image of a target surgical area captured by a binocular camera.

[0032] Specifically, the binocular image includes a left eye image and a right eye image. Before capturing the binocular image of the target surgical area, the binocular camera is initialized. During the percutaneous puncture surgery, the binocular camera captures the binocular image of the target surgical area in real time to achieve surgical navigation.

[0033] Step 102: Use binocular images to locate the puncture instrument and the target point in real time.

[0034] Specifically, the puncture instrument is installed at the end of the robotic arm, and the target point is used to mark the target position of the puncture instrument.

[0035] In some examples, the puncture instrument includes a puncture needle, the puncture needle includes an ablation needle, the percutaneous puncture procedure includes a percutaneous ablation procedure, the robotic arm can be a six-degree-of-freedom robotic arm, and the ablation needle is installed on the end effector of the robotic arm.

[0036] In some examples, the target point can be marked by attaching a marker to the target surgical area (the space to be punctured) to mark the location where the puncture needle needs to reach. The marker can include several independent markers, which can be any form of markers, such as Figure 2 As shown, two circular markers are used as marking points and are pasted on the target surgical area on the patient. The hollow parts of the two rings are used to expose the top and end of the puncture needle to avoid interference with the identification of the target point when the top and end of the puncture needle are blocked by the markers.

[0037] Step 103 , based on the error between the puncture instrument and the target point obtained by positioning, the movement of the robotic arm is controlled to maintain the same distance between the puncture instrument and the target point during the percutaneous puncture surgery, thereby achieving dynamic compensation for the target point displacement caused by respiratory movement.

[0038] The percutaneous puncture surgical navigation method provided in this embodiment utilizes a binocular camera to capture a binocular image of the target surgical area, uses the binocular image to locate the puncture instrument and the target point in real time, and controls the movement of the robotic arm based on the error between the located puncture instrument and the target point to maintain a constant distance between the puncture instrument and the target point. This surgical navigation solution is significantly less expensive than the conventional method of surgical navigation using infrared optical positioning systems. It is also simple to operate and can achieve visual servo navigation during percutaneous puncture surgery without the need for prior calibration. This ensures that the puncture instrument always maintains the same distance from the target point during percutaneous puncture surgery. This can, to a certain extent, reduce puncture errors caused by chest displacement due to respiratory movement, compensate for body surface movement caused by respiration in real time, improve the alignment accuracy of the needle entry point, and thereby improve the surgical accuracy of percutaneous puncture.

[0039] In some embodiments, using binocular images to locate the puncture instrument and the target point includes steps 1021 and 1022:

[0040] Step 1021 : Using the binocular image, the coordinates of the top and the end of the puncture instrument and the coordinates of the target point in the camera coordinate system are obtained through parallax calculation.

[0041] Taking the puncture instrument including the ablation needle as an example, the ablation needle has a specific geometric shape and size, and is a slender needle-like object. The ablation needle can be regarded as a rigid three-dimensional object. The ablation needle is fixed at the base of the end effector of the robotic arm. The ablation needle model can be simplified as a slender cylinder with a length of L ,radiusr The point where the ablation needle is connected to the robotic arm (the base of the end effector) is defined as the tip. The tip of the ablation needle is defined as the end , and then the ablation needle model can be expressed as:

[0042] Ablation needle feature vector: ;

[0043] , represents the coordinate of the end of the ablation needle in the camera coordinate system;

[0044] , represents the coordinate of the tip of the ablation needle in the camera coordinate system;

[0045] Satisfy constraints .

[0046] The target points include two markers for marking the top target position and the end target position of the ablation needle. The positions of these two markers (target points) are respectively recorded as and , Indicates the target position that the ablation needle tip should reach. Indicates the target position that the ablation needle tip (end) should reach, satisfying the constraints .

[0047] In a specific implementation, step 1021 may further include:

[0048] In step 1021a, the left eye image is input into a pre-trained key point detection model to obtain the coordinates of the top and end of the puncture instrument in the target surgical area, as well as the coordinates of the target point.

[0049] In some examples, the key point detection model can use the HRNet model to extract the end of the ablation needle from the left image ( ) and top ( ), target and The image coordinates of 、 、 The HRNet model can accurately extract the locations of key points in the left eye image and maintain good stability and accuracy in low-resolution and noisy environments.

[0050] Step 1021b: Use the right eye image to calculate the disparity through a stereo matching algorithm, and combine it with the binocular camera intrinsic parameter matrix to convert the image coordinate system to the camera coordinate system.

[0051] In some examples, the coordinates of the tip and end of the puncture instrument and the target point in the camera coordinate system are recorded as 、 、 and .

[0052] In the specific implementation, according to the parameters of the binocular camera, the pixel coordinates in the image coordinate system can be Convert to three-dimensional coordinates in the camera coordinate system , the steps of coordinate system conversion may include:

[0053] a. Calculate parallax:

[0054] Calculate the given pixel through the stereo matching algorithm The disparity between the left and right images , which is the horizontal offset between the pixel position in the left image and the corresponding pixel position in the right image.

[0055]

[0056] Where, 、 They represent the horizontal coordinates of the pixel position in the left image and the corresponding pixel position in the right image respectively.

[0057] b. Calculate depth:

[0058] By the focal length of the binocular camera , baseline distance and parallax , the depth of the object can be calculated ,depth is the actual distance from the object to the binocular camera.

[0059]

[0060] Where, represents the focal length of the camera, Indicates the horizontal distance between the two cameras of the binocular camera (baseline distance), and is a constant determined by binocular camera calibration.

[0061] c. Convert from image coordinate system to camera coordinate system:

[0062] Using the intrinsic parameter matrix of the binocular camera , which can convert pixel coordinates into three-dimensional coordinates in the camera coordinate system .

[0063]

[0064] Where, Used to describe the internal parameters of the binocular camera, including focal length and principal point (optical center) offset:

[0065]

[0066] Where, f x and f y Respectively represent the focal length x 、 y The unit of direction is pixel; c x 、 c y They represent the horizontal and vertical coordinates of the optical center in the image respectively.

[0067] By using the left-eye image to identify the coordinates of the puncture instrument and the target point, and using the right-eye image to perform depth calculation, the coordinates of the puncture instrument and the target point can be converted from the image coordinate system to the camera coordinate system, providing accurate three-dimensional positioning information for the visual servo tracking process to improve puncture accuracy.

[0068] After acquiring binocular images of the target surgical area captured by the binocular camera, the process also includes normalizing the left image and resampling the normalized left image to adjust the resolution of the left image to the target resolution. This provides appropriate target image quality for subsequent processing, facilitating accurate positioning results.

[0069] In some examples, the target resolution is 512×512. Since the parameters of different models of binocular cameras may vary, the resolution of the left eye image is adjusted to the target resolution of 512×512 before inputting it into the key point detection model for coordinate extraction. This can provide suitable image quality for subsequent processing.

[0070] In step 1022, a first eigenvector is obtained based on the coordinates of the top and the end of the puncture instrument, and a second eigenvector is obtained based on the coordinates of the target point in the camera coordinate system. The first eigenvector is used to characterize the positioning information of the puncture instrument, and the second eigenvector is used to characterize the positioning information of the target point.

[0071] In this embodiment, the three-dimensional coordinates of the puncture instrument and the target point in the camera coordinate system are used to obtain the feature vectors of the puncture instrument and the target point, which can accurately locate the puncture instrument and the target point in three-dimensional space, provide accurate positioning information for the visual servoing process, and thus improve the puncture accuracy.

[0072] In some embodiments, the Jacobian matrix is ​​defined as follows: the position of the end-of-arm (effector) is denoted as , construct the Jacobian matrix , the Jacobian matrix Defined as the characteristic vector change of the ablation needle in the camera coordinate system The change in the position of the end of the manipulator (actuator) in the manipulator coordinate system The differential of , that is, used to describe the differential of the first change to the second change, Furthermore, the above-mentioned percutaneous puncture surgical navigation method further includes:

[0073] a. Move the end of the robotic arm to its initial position.

[0074] In some examples, taking a six-degree-of-freedom robotic arm as an example, the initial position is expressed as .

[0075] b. Move each position component of the end of the robotic arm at the initial position once to obtain the offset pose after the end of the robotic arm moves each position component, and record the first change after each movement. The first change is the change of the first eigenvector in the camera coordinate system.

[0076] Specifically, at the initial position Each position component of moves once, and 6 linearly independent offset poses are obtained , represents the position of the ablation needle in the robotic arm coordinate system, I represents the unit matrix, Represents a six-row and six-column identity matrix, recording the effect of each movement .

[0077] c. Based on the offset posture and the initial position, calculate the second change, which is the position change of the end of the manipulator in the manipulator coordinate system.

[0078] d. Calculate the differential of the first variation with respect to the second variation to obtain the initial Jacobian matrix.

[0079] Specifically, through and Obtain the initial Jacobian matrix .

[0080] The above steps initialize the robotic arm and the Jacobian matrix. During percutaneous puncture surgery, visual servoing is started based on the initial Jacobian matrix. The Jacobian matrix and the error between the puncture instrument and the target point are used to calculate the movement speed of the robotic arm. The Jacobian matrix is ​​updated in real time, and the robotic arm drives the puncture instrument to always accurately track the target point.

[0081] In some embodiments, based on the error between the puncture instrument and the target point obtained by positioning, the movement of the robotic arm is controlled to maintain the same distance between the puncture instrument and the target point during the percutaneous puncture procedure, thereby dynamically compensating for the displacement of the target point caused by respiratory motion, including:

[0082] Step 103a: Calculate the error between the first eigenvector and the second eigenvector.

[0083] Construct the error between the first and second eigenvectors:

[0084] ,

[0085] Where, The eigenvector representing the target point is the second eigenvector, ; represents the ablation needle feature vector, i.e. the first feature vector, .

[0086] Step 103b: If the error does not meet the preset conditions, the movement velocity of the manipulator is calculated using the Jacobian matrix and the error. The Jacobian matrix describes the differential of a first variation with respect to a second variation. The first variation is the variation of the first eigenvector in the camera coordinate system, and the second variation is the variation of the position of the manipulator end in the manipulator coordinate system.

[0087] When the error meets the preset conditions, the current visual servo state is maintained and the robotic arm maintains the current state. When the error is not met, the robotic arm will move, achieving the navigation effect of automatically following the movement of the target during the operation.

[0088] In some examples, the movement speed of the robot arm (end) is calculated using the following formula:

[0089] ,

[0090] Where, represents the pseudo-inverse of the Jacobian matrix, represents the control gain, Indicates the movement speed of the robotic arm (end effector).

[0091] Since each degree of freedom of the six-degree-of-freedom robot can move, the calculated It is a value containing 6 degrees of freedom. It is a 6-dimensional vector. Each component represents the speed on one degree of freedom. When the speed runs for a unit time (for example, 1 second), the position will change accordingly.

[0092] Step 103c: Control the movement of the robotic arm according to the calculated movement speed of the robotic arm, and update the Jacobian matrix.

[0093] In some examples, the Broyden method is used to update the Jacobian matrix, which is as follows:

[0094]

[0095] formula, Represents the change in the characteristic vector of the ablation needle in the camera coordinate system, represents the current (before update) Jacobian matrix, represents the updated Jacobian matrix, is the difference between the actual motion speed instructions executed in adjacent cycles, T represents transposition, γ Represents the damping factor, which is used to enhance numerical stability so that the divisor will not be 0 during the calculation process. Since the larger the damping factor, the smaller the result, J The slower the update, the more stable the robot arm's motion will be from the perspective of the robot arm's motion, which can achieve stable tracking during visual servoing.

[0096] During the percutaneous puncture procedure, steps 103a to 103c are repeated until the norm of the error is When the preset conditions are met, the system is judged to be converged, where , unit: mm.

[0097] When the system converges, visual servoing is maintained so that when the target point moves with the ups and downs of the patient's body (such as the chest), the puncture instrument can always maintain the same distance from the target point, which can reduce the puncture error caused by chest displacement caused by respiratory movement to a certain extent.

[0098] In some examples, the visual servoing process can be as follows: Figure 3 shown.

[0099] In this embodiment, the Jacobian matrix and the error between the puncture instrument and the target point are used for visual servoing. The robotic arm drives the puncture instrument to always accurately track the changes of the target point, so that the puncture instrument always maintains the same distance from the target point. To a certain extent, the puncture error caused by chest displacement caused by respiratory movement can be reduced, thereby improving the surgical accuracy of percutaneous puncture.

[0100] In a specific implementation, the visual servo process of controlling the movement of the robotic arm based on the error between the puncture instrument and the target point obtained by positioning can be implemented by gradient descent or reinforcement learning methods, or by other methods, which are not specifically limited in this embodiment.

[0101] Example 2:

[0102] Another embodiment of the present application relates to a percutaneous puncture surgical navigation device. The implementation details of the percutaneous puncture surgical navigation device of this embodiment are described in detail below. The schematic diagram of the percutaneous puncture surgical navigation device of this embodiment can be as follows: Figure 4 As shown, it includes an image acquisition module 201 , a positioning module 202 and a motion control module 203 .

[0103] The image acquisition module 201 is used to acquire a binocular image of the target surgical area captured by a binocular camera. Specifically, the binocular image includes a left image and a right image. Before capturing the binocular image of the target surgical area, the binocular camera is initialized. During percutaneous puncture surgery, the binocular camera captures the binocular image of the target surgical area in real time to facilitate surgical navigation.

[0104] The positioning module 202 is used to locate the puncture instrument and the target point using binocular images. The puncture instrument is installed at the end of the robotic arm, and the target point is used to mark the target position of the puncture instrument; specifically, the puncture instrument is installed at the end of the robotic arm, and the target point is used to mark the target position of the puncture instrument.

[0105] The motion control module 203 is used to control the movement of the robotic arm according to the error between the puncture instrument and the target point obtained by positioning, so as to maintain the same distance between the puncture instrument and the target point during the percutaneous puncture surgery.

[0106] In some examples, the puncture instrument includes a puncture needle, the puncture needle includes an ablation needle, the percutaneous puncture procedure includes a percutaneous ablation procedure, the robotic arm can be a six-degree-of-freedom robotic arm, and the ablation needle is mounted on the end effector of the robotic arm. The target point can be marked by attaching a marker point to the target surgical area (the space to be punctured) to mark the position where the puncture needle needs to reach. The marker point can include multiple independent markers, which can be any form of marker, for example, Figure 2 As shown, two circular markers are used as marking points and are pasted on the target surgical area on the patient. The hollow parts of the two rings are used to expose the top and end of the puncture needle to avoid interference with the identification of the target point when the top and end of the puncture needle are blocked by the markers.

[0107] The percutaneous puncture surgical navigation device provided in this embodiment uses a binocular camera to capture a binocular image of the target surgical area, uses the binocular image to locate the puncture instrument and the target point in real time, and controls the movement of the robotic arm based on the error between the located puncture instrument and the target point to maintain a constant distance between the puncture instrument and the target point. This surgical navigation solution is significantly less expensive than the conventional infrared optical positioning system for surgical navigation, and is simple to operate. It can achieve visual servo navigation during percutaneous puncture surgery without the need for prior calibration, ensuring that the puncture instrument always maintains the same distance from the target point during the percutaneous puncture procedure. This can, to a certain extent, reduce puncture errors caused by chest displacement due to respiratory movement, compensate for body surface movement caused by breathing in real time, improve the alignment accuracy of the needle entry point, and thereby improve the surgical accuracy of percutaneous puncture.

[0108] In some embodiments, binocular images are used to locate the puncture instrument and the target target, including: using the binocular image to obtain the coordinates of the top and end of the puncture instrument and the coordinates of the target target in the camera coordinate system; obtaining a first feature vector based on the coordinates of the top and end of the puncture instrument in the camera coordinate system, and obtaining a second feature vector based on the coordinates of the target target in the camera coordinate system, the first feature vector is used to characterize the positioning information of the puncture instrument, and the second feature vector is used to characterize the positioning information of the target target.

[0109] Taking the puncture instrument including the ablation needle as an example, the ablation needle has a specific geometric shape and size, and is a slender needle-like object. The ablation needle can be regarded as a rigid three-dimensional object. The ablation needle is fixed at the base of the end effector of the robotic arm. The ablation needle model can be simplified as a slender cylinder with a length of L ,radius r The point where the ablation needle is connected to the robotic arm (the base of the end effector) is defined as the tip. The tip of the ablation needle is defined as the end , and then the ablation needle model can be expressed as:

[0110] Ablation needle feature vector: ;

[0111] , represents the coordinate of the end of the ablation needle in the camera coordinate system;

[0112] , represents the coordinate of the tip of the ablation needle in the camera coordinate system;

[0113] Satisfy constraints .

[0114] The target points include two markers for marking the top target position and the end target position of the ablation needle. The positions of these two markers (target points) are respectively recorded as and , Indicates the target position that the ablation needle tip should reach. Indicates the target position that the ablation needle tip (end) should reach, satisfying the constraints .

[0115] In a specific implementation, using binocular images to obtain the coordinates of the top and end of the puncture instrument and the coordinates of the target target in the camera coordinate system can further include: inputting the left eye image into a pre-trained key point detection model to obtain the coordinates of the top and end of the puncture instrument in the target surgical area, as well as the coordinates of the target target; using the right eye image, calculating the disparity through a stereo matching algorithm, and combining it with the binocular camera intrinsic parameter matrix to convert from the image coordinate system to the camera coordinate system.

[0116] In some examples, the key point detection model can use the HRNet model to extract the end of the ablation needle from the left image ( ) and top ( ), target and The image coordinates of 、 、 The HRNet model can accurately extract the positions of key points in the left eye image and maintain good stability and accuracy in low resolution and noisy environments. The coordinates of the top and end of the puncture instrument and the target point in the camera coordinate system are recorded as 、 、 and .

[0117] In the specific implementation, according to the parameters of the binocular camera, the pixel coordinates in the image coordinate system can be Convert to three-dimensional coordinates in the camera coordinate system , coordinate system transformations can include:

[0118] a. Calculate parallax:

[0119] Calculate the given pixel through the stereo matching algorithm The disparity between the left and right images , which is the horizontal offset between the pixel position in the left image and the corresponding pixel position in the right image.

[0120]

[0121] Where, 、 They represent the horizontal coordinates of the pixel position in the left image and the corresponding pixel position in the right image respectively.

[0122] b. Calculate depth:

[0123] By the focal length of the binocular camera , baseline distance and parallax , the depth of the object can be calculated ,depth is the actual distance from the object to the binocular camera.

[0124]

[0125] Where, represents the focal length of the camera, Indicates the horizontal distance between the two cameras of the binocular camera (baseline distance), and is a constant determined by binocular camera calibration.

[0126] c. Convert from image coordinate system to camera coordinate system:

[0127] Using the intrinsic parameter matrix of the binocular camera , which can convert pixel coordinates into three-dimensional coordinates in the camera coordinate system .

[0128]

[0129] Where, Used to describe the internal parameters of the binocular camera, including focal length and principal point (optical center) offset:

[0130]

[0131] Where, f x and f y Respectively represent the focal length x 、 y The unit of direction is pixel; c x 、 c y They represent the horizontal and vertical coordinates of the optical center in the image respectively.

[0132] By using the left-eye image to identify the coordinates of the puncture instrument and the target point, and using the right-eye image to perform depth calculation, the coordinates of the puncture instrument and the target point can be converted from the image coordinate system to the camera coordinate system, providing accurate three-dimensional positioning information for the visual servo tracking process to improve puncture accuracy.

[0133] After acquiring binocular images of the target surgical area captured by the binocular camera, the process also includes normalizing the left image and resampling the normalized left image to adjust the resolution of the left image to the target resolution. This provides appropriate target image quality for subsequent processing, facilitating accurate positioning results.

[0134] In some examples, the target resolution is 512×512. Since the parameters of different models of binocular cameras may vary, the resolution of the left eye image is adjusted to the target resolution of 512×512 before inputting it into the key point detection model for coordinate extraction. This can provide suitable image quality for subsequent processing.

[0135] In this embodiment, the three-dimensional coordinates of the puncture instrument and the target point in the camera coordinate system are used to obtain the feature vectors of the puncture instrument and the target point, which can accurately locate the puncture instrument and the target point in three-dimensional space, provide accurate positioning information for the visual servoing process, and thus improve the puncture accuracy.

[0136] In some embodiments, the Jacobian matrix is ​​defined as follows: the position of the end-of-arm (effector) is denoted as , construct the Jacobian matrix , the Jacobian matrix Defined as the characteristic vector change of the ablation needle in the camera coordinate system The change in the position of the end of the manipulator (actuator) in the manipulator coordinate system The differential of , that is, used to describe the differential of the first change to the second change, Furthermore, the motion control module is also used to:

[0137] a. Move the end of the robot arm to the initial position. Taking a six-degree-of-freedom robot arm as an example, the initial position is expressed as .

[0138] b. Move each position component of the end of the robotic arm at the initial position once to obtain the offset pose after the end of the robotic arm moves each position component, and record the first change after each movement. The first change is the change of the first eigenvector in the camera coordinate system.

[0139] Specifically, at the initial position Each position component of moves once, and 6 linearly independent offset poses are obtained , represents the position of the ablation needle in the robotic arm coordinate system, I represents the unit matrix, Represents a six-row and six-column identity matrix, recording the effect of each movement .

[0140] c. Based on the offset posture and the initial position, calculate the second change, which is the position change of the end of the manipulator in the manipulator coordinate system.

[0141] d. Calculate the differential of the first variation with respect to the second variation to obtain the initial Jacobian matrix. Specifically, and Obtain the initial Jacobian matrix .

[0142] During percutaneous puncture surgery, visual servoing is started based on the initial Jacobian matrix. The movement speed of the robotic arm is calculated using the Jacobian matrix and the error between the puncture instrument and the target point. The Jacobian matrix is ​​updated in real time, and the robotic arm drives the puncture instrument to always accurately track the target point.

[0143] In some embodiments, the movement of the robotic arm is controlled based on the error between the puncture instrument and the target point obtained by positioning so that the same distance is maintained between the puncture instrument and the target point during the percutaneous puncture surgery, including: calculating the error between the first eigenvector and the second eigenvector; when the error does not meet the preset conditions, using the Jacobian matrix and the error to calculate the movement speed of the robotic arm; controlling the movement of the robotic arm based on the calculated movement speed of the robotic arm, and updating the Jacobian matrix.

[0144] If the error meets the preset conditions, the current visual servoing is maintained. If it does not, the robotic arm will move, achieving a navigation effect that automatically follows the target during surgery. The Jacobian matrix is ​​used to describe the differential of a first variable with respect to a second variable. The first variable is the change in the first eigenvector in the camera coordinate system, and the second variable is the change in the position of the robotic arm end in the robotic arm coordinate system.

[0145] Construct the error between the first and second eigenvectors:

[0146] ,

[0147] Where, The eigenvector representing the target point is the second eigenvector, ; represents the ablation needle feature vector, i.e. the first feature vector, .

[0148] In some examples, the movement speed of the robot arm (end) is calculated using the following formula:

[0149] ,

[0150] Where, represents the pseudo-inverse of the Jacobian matrix, represents the control gain, Indicates the movement speed of the robotic arm (end effector).

[0151] Since each degree of freedom of the six-degree-of-freedom robot can move, the calculated It is a value containing 6 degrees of freedom. It is a 6-dimensional vector. Each component represents the speed on one degree of freedom. When the speed runs for a unit time (for example, 1 second), the position will change accordingly.

[0152] In some examples, the Broyden method is used to update the Jacobian matrix, which is as follows:

[0153]

[0154] formula, Represents the change in the characteristic vector of the ablation needle in the camera coordinate system, represents the current (before update) Jacobian matrix, represents the updated Jacobian matrix, is the difference between the actual motion speed instructions executed in adjacent cycles, T represents transposition, γ Represents the damping factor, which is used to enhance numerical stability so that the divisor will not be 0 during the calculation process. Since the larger the damping factor, the smaller the result, J The slower the update, the more stable the robot arm's motion will be from the perspective of the robot arm's motion, which can achieve stable tracking during visual servoing.

[0155] During percutaneous puncture surgery, the error is calculated cyclically and the error is judged to meet the preset conditions, and then the movement of the robotic arm is controlled until the norm of the error is When the preset conditions are met, the system is judged to be converged, where , in mm. When the system converges, visual servoing is maintained. This allows the puncture instrument to maintain the same distance from the target point as it moves with the patient's body (e.g., the thorax). This can, to a certain extent, reduce puncture errors caused by thoracic displacement due to respiratory motion.

[0156] In this embodiment, the Jacobian matrix and the error between the puncture instrument and the target point are used for visual servoing. The robotic arm drives the puncture instrument to always accurately track the changes of the target point, so that the puncture instrument always maintains the same distance from the target point. To a certain extent, it can reduce the puncture error caused by chest displacement caused by respiratory movement, compensate for the surface movement caused by breathing in real time, improve the alignment accuracy of the needle insertion point, and thus improve the surgical accuracy of percutaneous puncture.

[0157] In a specific implementation, the visual servo process of controlling the movement of the robotic arm based on the error between the puncture instrument and the target point obtained by positioning can be implemented by gradient descent or reinforcement learning methods, or by other methods, which are not specifically limited in this embodiment.

[0158] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problem proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0159] Example 3:

[0160] Another embodiment of the present application relates to an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods of the above embodiments.

[0161] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0162] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0163] Example 4:

[0164] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0165] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program. The program is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps in the methods described in the various embodiments of this application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0166] Embodiment 5:

[0167] Another embodiment of the present application relates to a percutaneous puncture surgical navigation system, such as Figure 2 As shown, including:

[0168] A robotic arm 11, with a puncture instrument 12 installed at the end;

[0169] A binocular camera 13 for capturing a binocular image of the target surgical area; and

[0170] The electronic device of the above embodiment.

[0171] In some examples, the robotic arm 11 is a six-degree-of-freedom robotic arm, and a puncture instrument 12 is mounted on a base of an end effector of the robotic arm 11 . The puncture instrument 12 is a puncture needle, such as an ablation needle.

[0172] The percutaneous surgical navigation system provided in this embodiment utilizes a binocular camera to capture a binocular image of the target surgical area, uses the binocular image to locate the puncture instrument and the target point in real time, and controls the movement of the robotic arm based on the error between the located puncture instrument and the target point to maintain a constant distance between the puncture instrument and the target point. The surgical navigation cost of this surgical robot is significantly lower than that of the conventional infrared optical positioning system. Furthermore, the system is simple to operate and can implement visual servo navigation during percutaneous puncture surgery without requiring prior calibration. This ensures that the puncture instrument maintains the same distance from the target point throughout the procedure, reducing puncture errors caused by chest displacement due to respiratory movement to a certain extent and improving the surgical accuracy of percutaneous puncture.

[0173] The calibration operation of traditional infrared optical positioning systems is complicated. For example, the calibration ball can easily interfere with needle insertion, and the calibration process is time-consuming, which increases the cost of surgical preparation. However, this embodiment only requires the internal parameter calibration of the binocular camera before the camera leaves the factory or before the operation. It can bypass the high cost process of infrared optical calibration by combining the puncture instrument feature modeling (such as the geometric model of the ablation needle) with the target marker design (for example, using a circular marker to mark the target to avoid blocking the end of the puncture instrument), and achieve a surgical navigation effect that is ready for use with the equipment.

[0174] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A percutaneous puncture surgery navigation method, characterized in that: include: Acquire a binocular image of a target surgical area captured by a binocular camera; The binocular image is used to calculate in real time the coordinates of the top and end of the puncture instrument and the coordinates of the target target in the camera coordinate system to locate the puncture instrument and the target target. The puncture instrument is mounted on the end of the robotic arm, and the target target is used to mark the target position of the puncture instrument. A first feature vector is obtained based on the coordinates of the top and end of the puncture instrument, and a second feature vector is obtained based on the coordinates of the target target in the camera coordinate system. The first feature vector is used to represent the positioning information of the puncture instrument, and the second feature vector is used to represent the positioning information of the target target. calculating an error between the first eigenvector and the second eigenvector; When the error does not satisfy a preset condition, calculating the movement speed of the robotic arm using a Jacobian matrix and the error, wherein the Jacobian matrix is ​​used to describe the differential of a first change amount with respect to a second change amount, the first change amount being the change amount of a first eigenvector in a camera coordinate system, and the second change amount being the change amount of a position of the end of the robotic arm in a robotic arm coordinate system; The movement of the robotic arm is controlled according to the calculated movement speed of the robotic arm, and the Jacobian matrix is ​​updated to maintain the same distance between the puncture instrument and the target point during the percutaneous puncture procedure, thereby achieving dynamic compensation for the target point displacement caused by respiratory movement.

2. The percutaneous puncture surgery navigation method according to claim 1, characterized in that: The coordinates of the top and the end of the puncture instrument and the coordinates of the target point in the camera coordinate system are obtained by parallax calculation using the binocular image.

3. The percutaneous puncture surgery navigation method according to claim 2, characterized in that: The binocular image includes a left image and a right image; the coordinates of the top and the end of the puncture instrument and the coordinates of the target point in the camera coordinate system are obtained by parallax calculation using the binocular image, including: Inputting the left eye image into a pre-trained key point detection model to obtain the coordinates of the top and end of the puncture instrument in the target surgical area, as well as the coordinates of the target point; The right eye image is used to calculate the disparity through a stereo matching algorithm, and the image coordinate system is converted to the camera coordinate system in combination with the binocular camera intrinsic parameter matrix.

4. The percutaneous puncture surgery navigation method according to claim 1, characterized in that: Also includes: Move the end of the robotic arm to the initial position; Move each position component of the end of the manipulator at the initial position once, obtain an offset posture representing the end of the manipulator after each position component moves, and record a first change after each movement; Calculating a second change based on the offset posture and the initial position; The differential of the first variation with respect to the second variation is calculated to obtain an initial Jacobian matrix.

5. The percutaneous puncture surgery navigation method according to claim 1, characterized in that: The binocular image includes a left-eye image and a right-eye image; after acquiring the binocular image of the target surgical area captured by the binocular camera, the method further includes: performing normalization processing on the left-eye image; The normalized left-eye image is resampled to adjust the resolution of the left-eye image to the target resolution.

6. A percutaneous puncture surgical navigation device, characterized in that: include: An image acquisition module, used to acquire a binocular image of the target surgical area captured by a binocular camera; a positioning module, configured to calculate in real time using the binocular image the coordinates of the top and end of the puncture instrument and the coordinates of the target target in the camera coordinate system, so as to locate the puncture instrument and the target target, wherein the puncture instrument is mounted at the end of the robotic arm, and the target target is used to mark the target position of the puncture instrument; a first feature vector is obtained based on the coordinates of the top and end of the puncture instrument, and a second feature vector is obtained based on the coordinates of the target target in the camera coordinate system, wherein the first feature vector is used to represent the positioning information of the puncture instrument, and the second feature vector is used to represent the positioning information of the target target; A visual servo control module is used to calculate the error between the first eigenvector and the second eigenvector; when the error does not meet the preset conditions, the movement speed of the robotic arm is calculated using the Jacobian matrix and the error, the Jacobian matrix is ​​used to describe the differential of the first change quantity with respect to the second change quantity, the first change quantity is the change quantity of the first eigenvector in the camera coordinate system, and the second change quantity is the position change quantity of the end of the robotic arm in the robotic arm coordinate system; the movement of the robotic arm is controlled according to the calculated movement speed of the robotic arm, and the Jacobian matrix is ​​updated to ensure that the same distance is maintained between the puncture instrument and the target point during the percutaneous puncture surgery, thereby realizing dynamic compensation for the target point displacement caused by respiratory movement.

7. The percutaneous puncture surgery navigation device according to claim 6, characterized in that: The positioning module is used for: The coordinates of the top and the end of the puncture instrument and the coordinates of the target point in the camera coordinate system are obtained by parallax calculation using the binocular image.

8. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the percutaneous puncture surgery navigation method according to any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the percutaneous puncture surgery navigation method according to any one of claims 1 to 5 is implemented.

10. A percutaneous puncture surgical navigation system, characterized in that: include: A robotic arm with a puncture instrument installed at the end; a binocular camera for capturing binocular images of the target surgical area; as well as The electronic device according to claim 8.

Images