Percutaneous puncture operation navigation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the field of surgical robots, and discloses a percutaneous puncture surgical navigation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a binocular image of a target surgical area captured by a binocular camera; the binocular image is used for positioning a puncture instrument and a target spot, the puncture instrument is installed at the tail end of a mechanical arm, and the target spot is used for marking the target position of the puncture instrument; according to the error, obtained through positioning, between the puncture instrument and the target spot, the mechanical arm is controlled to move, so that the same distance is kept between the puncture instrument and the target spot in the percutaneous puncture operation process. The percutaneous puncture surgical navigation method solves the problems that in the prior art, calibration needs to be conducted in advance in a surgical navigation mode through an infrared optical positioning system, the surgical navigation cost is high, and operation is complex. At least the effect of visual servo navigation in the percutaneous puncture operation process can be achieved without calibration.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of surgical robots, and particularly to a percutaneous puncture surgery navigation method, device, electronic device and storage medium. Background Art

[0002] At present, the application of surgical robots in the field of percutaneous puncture surgery is gradually expanding. In actual operation, doctors generally use imaging devices such as CT and ultrasound to formulate the needle insertion trajectory to achieve image navigation of percutaneous puncture surgery. However, this process requires coordinate transformation of the target point, surgical instrument, and image navigation system, and the links are relatively complex. Moreover, in order to meet the accuracy requirements of surgical navigation, it is usually necessary to calibrate the infrared optical positioning system in advance, which makes the surgical navigation method costly and complex to operate. In addition, the optical calibration balls used in the infrared optical positioning system will also interfere with the needle insertion operation. In addition, related technical solutions usually adopt an offline planning scheme, which is difficult to compensate for the body surface movement caused by respiration in real time, thereby affecting the alignment accuracy of the needle insertion point and limiting its application in surgeries with high requirements for the needle insertion path. Summary of the Invention

[0003] The purpose of the present invention is to provide at least a percutaneous puncture surgery navigation method, device, electronic device and storage medium, which can at least solve the problem that the method of using an infrared optical positioning system for surgical navigation in the prior art requires prior calibration, and the cost of surgical navigation is high and the operation is complex, and can at least achieve the effect of realizing visual servo navigation during percutaneous puncture surgery without calibration.

[0004] To solve the above technical problems, at least one embodiment of the present application provides a percutaneous puncture surgery navigation method, including: obtaining binocular images of the target surgical area captured by a binocular camera; using the binocular images to real-time locate a puncture instrument and a target point, the puncture instrument is installed at the end of a robotic arm, and the target point is used to mark the target position of the puncture instrument; according to the error between the located puncture instrument and the target point, controlling the movement of the robotic arm so that the distance between the puncture instrument and the target point remains the same during percutaneous puncture surgery, and realizing dynamic compensation for the displacement of the target point caused by respiratory movement.

[0005] At least one embodiment of the present application further provides a percutaneous puncture surgery navigation device, including: an image acquisition module for acquiring binocular images of a target surgical area captured by a binocular camera; a positioning module for using the binocular images to real-time locate a puncture instrument and a target point, the puncture instrument being installed at the end of a robotic arm, and the target point being used to mark the target position of the puncture instrument; a visual servo control module for controlling the movement of the robotic arm according to the error between the located puncture instrument and the target point, so that the distance between the puncture instrument and the target point remains the same during the percutaneous puncture surgery process, and dynamic compensation for the displacement of the target point caused by respiratory movement is achieved.

[0006] At least one embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the percutaneous puncture surgery navigation method described above.

[0007] At least one embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the percutaneous puncture surgery navigation method described above is implemented.

[0008] At least one embodiment of the present application provides a percutaneous puncture surgery navigation system, including: a robotic arm with a puncture instrument installed at its end; a binocular camera for capturing binocular images of a target surgical area; and the electronic device described above.

[0009] The percutaneous puncture surgery navigation method provided by the embodiment of the present application uses a binocular camera to capture binocular images of a target surgical area, uses the binocular images to real-time locate a puncture instrument and a target point, and controls the movement of the robotic arm according to the error between the located puncture instrument and the target point, so that the distance between the puncture instrument and the target point remains unchanged. The cost of this surgical navigation solution is significantly lower than that of the existing infrared optical positioning system for surgical navigation, and the operation is simple. Without prior calibration, visual servo navigation during the percutaneous puncture surgery process can be achieved, so that the distance between the puncture instrument and the target point always remains the same during the percutaneous puncture surgery process. To a certain extent, it can reduce the puncture error caused by the chest displacement caused by respiratory movement, compensate for the body surface movement caused by respiration in real time, improve the alignment accuracy of the needle insertion point, and thus improve the surgical accuracy of percutaneous puncture.

[0010] In some alternative embodiments, real-time positioning of the puncture device and the target using the binocular images includes: obtaining the coordinates of the tip and the end of the puncture device and the coordinates of the target in the camera coordinate system through parallax calculation using the binocular images; obtaining a first feature vector based on the coordinates of the tip and the end of the puncture device, and obtaining a second feature vector based on the coordinates of the target in the camera coordinate system, where the first feature vector is used to characterize the positioning information of the puncture device, and the second feature vector is used to characterize the positioning information of the target. By obtaining the three-dimensional coordinates of the puncture device and the target in the camera coordinate system and getting the feature vectors of the puncture device and the target, the puncture device and the target can be accurately positioned in the three-dimensional space, providing accurate positioning information for the visual servo process to improve the puncture accuracy.

[0011] In some alternative embodiments, the binocular images include a left-eye image and a right-eye image; obtaining the coordinates of the tip and the end of the puncture device and the coordinates of the target in the camera coordinate system through parallax calculation using the binocular images includes: inputting the left-eye image into a pre-trained key-point detection model to obtain the coordinates of the tip and the end of the puncture device and the coordinates of the target in the target surgical area; using the right-eye image, calculating the parallax through a stereo matching algorithm, and combining with the internal parameter matrix of the binocular camera to convert the image coordinate system to the camera coordinate system. By identifying the coordinates of the puncture device and the target using the left-eye image and performing depth calculation using the right-eye image, the coordinates of the puncture device and the target can be converted from the image coordinate system to the camera coordinate system, providing accurate three-dimensional positioning information for the visual servo tracking process to improve the puncture accuracy.

[0012] In some alternative embodiments, according to the error between the puncture device and the target obtained by positioning, controlling the movement of the robotic arm to keep the same distance between the puncture device and the target during the percutaneous puncture surgery process, realizing dynamic compensation for the target displacement caused by respiratory movement, includes: calculating the error between the first feature vector and the second feature vector; in the case where the error does not meet the preset condition, calculating the movement speed of the robotic arm using the Jacobian matrix and the error, where the Jacobian matrix is used to describe the differential of the first change amount with respect to the second change amount, the first change amount is the change amount of the first feature vector in the camera coordinate system, and the second change amount is the position change amount of the end of the robotic arm in the robotic arm coordinate system; controlling the movement of the robotic arm according to the calculated movement speed of the robotic arm and updating the Jacobian matrix. Using the Jacobian matrix and the error between the puncture device and the target for visual servo, the robotic arm drives the puncture device to always accurately track the change of the target, keeping the same distance between the puncture device and the target all the time, which can reduce the puncture error caused by the thoracic displacement due to respiratory movement to a certain extent and improve the surgical precision of percutaneous puncture.

[0013] In some alternative embodiments, the percutaneous puncture surgical navigation method further includes: moving the end of the robotic arm to the initial position; moving each position component of the end of the robotic arm at the initial position once respectively to obtain the offset pose indicating the end of the robotic arm after each position component moves respectively, and recording the first change amount after each movement; calculating the second change amount based on the offset pose and the initial position; calculating the differential of the first change amount with respect to the second change amount to obtain the initial Jacobian matrix. The initialization of the robotic arm and the initialization of the Jacobian matrix are realized. Thus, during the percutaneous puncture surgery process, visual servoing starts based on the initial Jacobian matrix. The movement speed of the robotic arm is calculated by using the Jacobian matrix and the error between the puncture instrument and the target. The Jacobian matrix is updated in real time, and the robotic arm drives the puncture instrument to always accurately track the target.

[0014] In some alternative embodiments, the binocular images include a left-eye image and a right-eye image. After obtaining the binocular images of the target surgical area captured by the binocular camera, it further includes: performing normalization processing on the left-eye image; resampling the normalized left-eye image to adjust the resolution of the left-eye image to the target resolution. It can provide appropriate target image quality for subsequent processing and is beneficial to obtaining accurate positioning results.

[0015] In some alternative embodiments, the positioning module is used to: calculate the coordinates of the tip and the end of the puncture instrument and the coordinates of the target in the camera coordinate system by using the binocular images through parallax calculation; obtain a first feature vector based on the coordinates of the tip and the end of the puncture instrument, and obtain a second feature vector based on the coordinates of the target in the camera coordinate system. The first feature vector is used to characterize the positioning information of the puncture instrument, and the second feature vector is used to characterize the positioning information of the target. The visual servo control module is used to: calculate the error between the first feature vector and the second feature vector; in the case where the error does not meet the preset condition, calculate the movement speed of the robotic arm by using the Jacobian matrix and the error. The Jacobian matrix is used to describe the differential of the first change amount with respect to the second change amount. The first change amount is the change amount of the first feature vector in the camera coordinate system, and the second change amount is the position change amount of the end of the robotic arm in the robotic arm coordinate system; control the movement of the robotic arm according to the calculated movement speed of the robotic arm and update the Jacobian matrix. Visual servoing is performed by using the Jacobian matrix and the error between the puncture instrument and the target. The robotic arm drives the puncture instrument to always accurately track the change of the target, so that the distance between the puncture instrument and the target is always kept the same. To a certain extent, it can reduce the puncture error caused by the thoracic displacement due to respiratory movement, compensate for the body surface movement caused by breathing in real time, improve the alignment accuracy of the needle insertion point, and further improve the surgical accuracy of percutaneous puncture. Brief Description of the Drawings

[0016] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplary illustrations do not constitute a limitation on the embodiments.

[0017] Figure 1 is a flowchart of the percutaneous puncture surgery navigation method provided by the embodiments of the present application; Figure 2 is a schematic diagram of pasting the target point provided by the embodiments of the present application.

[0018] Figure 3 is a schematic diagram of the visual servo process provided by the embodiments of the present application; Figure 4 is a schematic diagram of the percutaneous puncture surgery navigation device provided by the embodiments of the present application. Detailed Description of the Embodiments

[0019] To facilitate the understanding of the embodiments of the present application, the relevant content about percutaneous puncture surgery navigation is introduced here first.

[0020] Image-guided percutaneous puncture surgery is an important means for the diagnosis and treatment of tumors. Compared with open surgery, percutaneous puncture surgery has the advantages of less trauma, less pain, fewer complications, faster recovery, and lower cost. It is widely used in the diagnosis and treatment of important organs in the chest and abdomen. However, it still faces the following problems: 1. Greatly affected by the patient's respiratory movement: The patient's breathing will cause the chest and upper abdomen to rise and fall, making it difficult to accurately puncture the set skin puncture point, thereby affecting the accuracy of the puncture operation. Clinically, patients are often required to adopt techniques such as breath-holding and respiratory gating to artificially reduce the interference of body surface movement, which is not friendly to some patients who cannot control their breathing.

[0021] 2. The doctor's hand tremors will reduce the puncture accuracy and even cause complications such as bleeding; 3. Percutaneous puncture surgery highly depends on the doctor's experience: Clinically, the biopsy diagnosis rate of tumors with a diameter of 5 mm to 10 mm is only 52%. For tumors and nodules with a diameter less than 5 mm, manual puncture is difficult to ensure the success rate of the surgery, and the puncture accuracy and effect highly depend on the doctor's experience.

[0022] At present, the application of surgical robots in the field of percutaneous puncture surgery is gradually expanding. In actual operation, doctors generally use imaging devices such as CT and ultrasound to formulate the needle insertion trajectory to achieve image navigation of percutaneous puncture surgery. Then, this process requires coordinate transformation of the target point, surgical instrument, and image navigation system, and the process is relatively complex. Moreover, in order to meet the accuracy requirements of surgical navigation, it is usually necessary to pre-calibrate the infrared optical positioning system in advance, which makes the surgical navigation method costly and complex to operate. In addition, the optical calibration balls used in the infrared optical positioning system will also interfere with the needle insertion operation. In addition, related technical solutions usually adopt an offline planning scheme, which is difficult to compensate for the body surface movement caused by respiration in real time, thereby affecting the alignment accuracy of the needle insertion point and restricting its application in surgeries with high requirements for the needle insertion path.

[0023] Therefore, there is an urgent need in this field for a surgical navigation solution for percutaneous puncture surgery that is low-cost, easy to operate, and can cope with respiratory movement interference, to solve the problem that the method of using an infrared optical positioning system for surgical navigation requires pre-calibration in advance, and the cost of surgical navigation is high and the operation is complex, and at least achieve the effect of visual servo navigation during percutaneous puncture surgery without calibration.

[0024] To solve the above technical problems, the present invention proposes a percutaneous puncture surgery navigation method. The following specifically describes the implementation details of the percutaneous puncture surgery navigation method of this embodiment. The following content is only for facilitating the understanding of the implementation details provided by the present invention.

[0025] Embodiment 1: The percutaneous puncture surgery navigation method of this embodiment can be applied to an electronic device with communication, computing, and data storage capabilities, and its specific process can be as Figure 1 shown, including: Step 101, obtain binocular images of the target surgical area captured by a binocular camera.

[0026] Specifically, the binocular images include a left-eye image and a right-eye image. Before capturing the binocular images of the target surgical area, the binocular camera is initialized. During the percutaneous puncture surgery process, the binocular camera captures the binocular images of the target surgical area in real time to achieve surgical navigation.

[0027] Step 102, use the binocular images to locate the puncture instrument and the target point in real time.

[0028] Specifically, the puncture instrument is installed at the end of the robotic arm, and the target point is used to mark the target position of the puncture instrument.

[0029] In some examples, the puncture device includes a puncture needle, the puncture needle includes an ablation needle, the percutaneous puncture surgery includes a percutaneous ablation surgery, the robotic arm can be a six-degree-of-freedom robotic arm, and the ablation needle is installed on the end effector of the robotic arm.

[0030] In some examples, the target target point can be marked by pasting marking points in the target surgical area (the space where puncture is required), and the marking points can include several independent markers, which can be markers in any form. For example, as Figure 2 shown, two circular markers are used as the marking points and pasted on the target surgical area of the patient. The hollow parts of the two rings are used to expose the top and end of the puncture needle, so as to avoid interference with the identification of the target target point when the top and end of the puncture needle are blocked by the markers. Step 103: According to the error between the puncture device and the target target point obtained by positioning, control the movement of the robotic arm so that the distance between the puncture device and the target target point remains the same during the percutaneous puncture surgery, and realize the dynamic compensation for the displacement of the target point caused by respiratory movement.

[0031] The percutaneous puncture surgery navigation method provided in this embodiment uses a binocular camera to capture binocular images of the target surgical area, uses the binocular images to locate the puncture device and the target target point in real time, and controls the movement of the robotic arm according to the error between the puncture device and the target target point obtained by positioning, so that the distance between the puncture device and the target target point remains unchanged. The cost of this surgical navigation scheme is significantly lower than that of the existing infrared optical positioning system for surgical navigation, and the operation is simple. Without prior calibration, visual servo navigation during the percutaneous puncture surgery can be realized, so that the distance between the puncture device and the target target point always remains the same during the percutaneous puncture surgery. To a certain extent, it can reduce the puncture error caused by the chest displacement caused by respiratory movement, compensate for the body surface movement caused by breathing in real time, improve the alignment accuracy of the needle insertion point, and thus improve the surgical accuracy of percutaneous puncture.

[0032] In some embodiments, using the binocular images to locate the puncture device and the target target point includes steps 1021 to 1022: Step 1021: Use the binocular images to calculate the coordinates of the top and end of the puncture device and the coordinates of the target target point in the camera coordinate system through parallax.

[0033] Taking the puncture device including an ablation needle as an example, the ablation needle has a specific geometric shape and size, is a slender needle-like object, the ablation needle can be regarded as a rigid three-dimensional object, the ablation needle is fixed to the base of the end effector of the robotic arm, and the ablation needle model can be simplified into a slender cylinder, and the length of the cylinder is L , radius rCan be ignored. The part where the ablation needle is connected to the robotic arm (the base of the end effector) is defined as the tip. , the tip of the ablation needle is defined as the end. , and thus the ablation needle model can be expressed as: Ablation needle feature vector: ; , representing the coordinates of the end of the ablation needle in the camera coordinate system; , representing the coordinates of the tip of the ablation needle in the camera coordinate system; Satisfying the constraint condition .

[0034] The target includes two markers for marking the tip target position and the end target position of the ablation needle. The positions of these two markers (targets) are respectively denoted as and , representing the target position that the tip of the ablation needle should reach, representing the target position that the tip (end) of the ablation needle should reach, satisfying the constraint condition .

[0035] In a specific implementation, step 1021 may further include: Step 1021a, input the left-eye image into a pre-trained key point detection model to obtain the coordinates of the tip and the end of the puncture instrument in the target surgical area, as well as the coordinates of the target.

[0036] In some examples, the key point detection model can adopt the HRNet model to separately extract the end ( ) and the tip ( ) of the ablation needle, the target and from the left-eye image, and the image coordinates are respectively denoted as , , . Using the HRNet model can accurately extract the positions of the key points in the left-eye image and still maintain good stability and accuracy in a low-resolution and noisy environment.

[0037] Step 1021b, using the right-eye image, calculate the disparity through a stereo matching algorithm, and combine the binocular camera internal parameter matrix to convert the image coordinate system to the camera coordinate system.

[0038] In some examples, the coordinates of the tip and the end of the puncture instrument and the target in the camera coordinate system are respectively denoted as , , and .

[0039] In a specific implementation, according to the parameters of the binocular camera, the pixel coordinates in the image coordinate system can be converted into three-dimensional coordinates in the camera coordinate system , and the steps of coordinate conversion may include: a. Calculate the disparity: Calculate the disparity between a given pixel between the left-eye image and the right-eye image , that is, the horizontal offset between the pixel position in the left-eye image and the corresponding pixel position in the right-eye image.

[0040]

[0041] In the formula, , respectively represent the abscissas of the pixel position in the left-eye image and the corresponding pixel position in the right-eye image.

[0042] b. Calculate the depth: Based on the focal length of the binocular camera, the baseline distance and the disparity , the depth of the object can be calculated , and the depth

[0043]

[0044] is the actual distance from the object to the binocular camera. In the formula, represents the focal length of the camera, represents the horizontal distance (baseline distance) between the two cameras of the binocular camera,

[0045] c. Convert from the image coordinate system to the camera coordinate system: Using the internal parameter matrix of the binocular camera, the pixel coordinates can be converted into three-dimensional coordinates

[0046]

[0047] In the formula, is used to describe the internal parameters of the binocular camera, including the focal length and the offset of the principal point (optical center):

[0048] In the formula, f x and fy respectively represent the representation in the x and y directions, and the unit is pixel; c x and c y respectively represent the horizontal and vertical coordinates of the optical center in the image.

[0049] By using the left-eye image to identify the coordinates of the puncture instrument and the target, and using the right-eye image for depth calculation, the coordinates of the puncture instrument and the target can be transformed from the image coordinate system to the camera coordinate system, providing accurate three-dimensional positioning information for the visual servo tracking process to improve the puncture accuracy.

[0050] After obtaining the binocular images of the target surgical area captured by the binocular camera, it further includes: normalizing the left-eye image; and resampling the normalized left-eye image to adjust the resolution of the left-eye image to the target resolution. It can provide appropriate target image quality for subsequent processing and is conducive to obtaining accurate positioning results.

[0051] In some examples, the target resolution is 512×512. Since the parameters of different models of binocular cameras may be different, before inputting the left-eye image into the key-point detection model for coordinate extraction, adjusting the resolution of the left-eye image to the target resolution of 512×512 can provide appropriate image quality for subsequent processing.

[0052] Step 1022: Obtain the first feature vector based on the coordinates of the tip and the end of the puncture instrument, and obtain the second feature vector based on the coordinates of the target in the camera coordinate system. The first feature vector is used to represent the positioning information of the puncture instrument, and the second feature vector is used to represent the positioning information of the target.

[0053] In this embodiment, by using the three-dimensional coordinates of the puncture instrument and the target in the camera coordinate system to obtain the feature vectors of the puncture instrument and the target, the puncture instrument and the target can be accurately positioned in the three-dimensional space, providing accurate positioning information for the visual servo process to improve the puncture accuracy.

[0054] In some embodiments, define the Jacobian matrix: Denote the position of the end (actuator) of the robotic arm as , construct the Jacobian matrix , and define the Jacobian matrix as the change amount of the feature vector of the ablation needle in the camera coordinate system which is the differential of the change amount of the position of the end (actuator) of the robotic arm in the robotic arm coordinate system, that is, used to describe the differential of the first change amount with respect to the second change amount, . Further, the above percutaneous puncture surgery navigation method further includes: a. Move the end of the robotic arm to the initial position.

[0055] In some examples, taking a six - degree - of - freedom robotic arm as an example, the initial position is represented as .

[0056] b. Move each position component of the end of the robotic arm at the initial position once respectively, to obtain the offset poses representing the end of the robotic arm after each position component moves, and record the first change amount after each movement. The first change amount is the change amount of the first eigenvector in the camera coordinate system.

[0057] Specifically, at the initial position make one movement for each position component, to obtain 6 linearly independent offset poses , represents the position of the ablation needle in the robotic arm coordinate system, I represents the identity matrix, represents the six - row and six - column identity matrix, record the after each movement.

[0058] c. Based on the offset poses and the initial position, calculate the second change amount. The second change amount is the position change amount of the end of the robotic arm in the robotic arm coordinate system.

[0059] d. Calculate the differential of the first change amount with respect to the second change amount to obtain the initial Jacobian matrix.

[0060] Specifically, through and obtain the initial Jacobian matrix .

[0061] The above steps achieve the initialization of the robotic arm and the initialization of the Jacobian matrix. Thus, during the percutaneous puncture surgery process, start visual servo based on the initial Jacobian matrix, calculate the movement speed of the robotic arm using the error between the Jacobian matrix, the puncture instrument and the target, and update the Jacobian matrix in real - time. The robotic arm drives the puncture instrument to always accurately track the target.

[0062] In some embodiments, according to the error between the puncture instrument and the target obtained by positioning, control the movement of the robotic arm so that the distance between the puncture instrument and the target remains the same during the percutaneous puncture surgery process, to achieve dynamic compensation for the displacement of the target caused by respiratory movement, including: Step 103a, calculate the error between the first eigenvector and the second eigenvector.

[0063] Construct the error between the first eigenvector and the second eigenvector: , wherein, Represents the feature vector of the target target, i.e., the second feature vector, ; Represents the ablation needle feature vector, i.e., the first feature vector, .

[0064] Step 103b, when the error does not meet the preset condition, use the Jacobian matrix and the error to calculate the movement speed of the robotic arm. The Jacobian matrix is used to describe the differential of the first change quantity with respect to the second change quantity. The first change quantity is the change quantity of the first feature vector in the camera coordinate system, and the second change quantity is the position change quantity of the end of the robotic arm in the robotic arm coordinate system.

[0065] When the error meets the preset condition, the current visual servo state is maintained, and the robotic arm maintains its current state. When the error is not met, the robotic arm will move to achieve the navigation effect of automatically following the target target during the operation.

[0066] In some examples, the movement speed of the robotic arm (end) is calculated using the following formula: , In the formula, Represents the pseudo-inverse of the Jacobian matrix, Represents the control gain, Represents the movement speed of the robotic arm (end effector).

[0067] Since each degree of freedom of the six-degree-of-freedom robotic arm can move, the calculated is a value containing 6 degrees of freedom, which is a 6D vector. Each component represents the speed in one degree of freedom. After running for a unit time (e.g., 1 second) at this speed, the position will change accordingly.

[0068] Step 103c, control the movement of the robotic arm according to the calculated movement speed of the robotic arm, and update the Jacobian matrix.

[0069] In some examples, the Broyden method is used to update the Jacobian matrix, and the formula is as follows:

[0070] In the formula, Represents the change quantity of the feature vector of the ablation needle in the camera coordinate system, Represents the current (before update) Jacobian matrix, Represents the updated Jacobian matrix, Is the difference between the actual movement speed commands executed in adjacent cycles, T represents the transpose, γ Represents the damping factor, which is used to enhance numerical stability so that the situation of the dividend being 0 does not occur during the calculation process. Since the larger the damping factor, the smaller the result obtained,J The slower the update, the more stable the robot's motion will be from the perspective of the robot's motion, enabling stable tracking during visual servoing.

[0071] During the percutaneous puncture procedure, steps 103a to 103c are repeated until the norm of the error is When the preset conditions are met, the system is judged to be convergent, where , unit: mm.

[0072] When the system converges, visual servoing is maintained so that when the target point moves with the ups and downs of the patient's body (such as the chest), the puncture instrument can always maintain the same distance from the target point, which can reduce the puncture error caused by chest displacement caused by respiratory movement to a certain extent.

[0073] In some examples, the visual servoing process can be as follows: Figure 3 shown.

[0074] In this embodiment, the Jacobian matrix and the error between the puncture instrument and the target point are used for visual servoing. The robotic arm drives the puncture instrument to always accurately track the changes of the target point, so that the puncture instrument always maintains the same distance from the target point. To a certain extent, the puncture error caused by chest displacement caused by respiratory movement can be reduced, thereby improving the surgical accuracy of percutaneous puncture.

[0075] In a specific implementation, the visual servo process of controlling the movement of the robotic arm according to the error between the puncture instrument and the target point obtained by positioning can be implemented by gradient descent or reinforcement learning methods, or by other methods, which are not specifically limited in this embodiment.

[0076] Embodiment 2: Another embodiment of the present application relates to a percutaneous puncture surgical navigation device. The implementation details of the percutaneous puncture surgical navigation device of this embodiment are specifically described below. The schematic diagram of the percutaneous puncture surgical navigation device of this embodiment can be as follows: Figure 4 As shown, it includes an image acquisition module 201, a positioning module 202 and a motion control module 203.

[0077] The image acquisition module 201 is used to acquire a binocular image of the target surgical area captured by the binocular camera; specifically, the binocular image includes a left image and a right image. Before capturing the binocular image of the target surgical area, the binocular camera is initialized. During the percutaneous puncture surgery, the binocular camera captures the binocular image of the target surgical area in real time to achieve surgical navigation.

[0078] The positioning module 202 is used to locate the puncture instrument and the target point using binocular images. The puncture instrument is installed at the end of the robotic arm, and the target point is used to mark the target position of the puncture instrument. Specifically, the puncture instrument is installed at the end of the robotic arm, and the target point is used to mark the target position of the puncture instrument.

[0079] The motion control module 203 is used to control the motion of the robotic arm according to the error between the puncture instrument and the target point obtained by positioning, so as to keep the same distance between the puncture instrument and the target point during the percutaneous puncture operation.

[0080] In some examples, the puncture instrument includes a puncture needle, the puncture needle includes an ablation needle, the percutaneous puncture operation includes a percutaneous ablation operation, the robotic arm can be a six-degree-of-freedom robotic arm, and the ablation needle is installed at the end effector of the robotic arm. The target point can be marked by pasting marker points in the target surgical area (the space that needs to be punctured) to mark the position that the puncture needle needs to reach. The marker points can include several independent markers, and specifically can be any form of markers. For example, as Figure 2 shown, two circular markers are used as marker points and pasted on the target surgical area of the patient. The hollow parts of the two circles are used to expose the top and end of the puncture needle, so as to avoid interference with the identification of the target point when the top and end of the puncture needle are blocked by the markers.

[0081] The percutaneous puncture surgical navigation device provided in this embodiment uses a binocular camera to capture binocular images of the target surgical area, uses the binocular images to locate the puncture instrument and the target point in real time, and controls the motion of the robotic arm according to the error between the puncture instrument and the target point obtained by positioning, so that the distance between the puncture instrument and the target point remains unchanged. The cost of this surgical navigation solution is significantly lower than that of the existing infrared optical positioning system for surgical navigation, and the operation is simple. Without prior calibration, visual servo navigation during the percutaneous puncture operation can be achieved, so that the puncture instrument and the target point always maintain the same distance during the percutaneous puncture operation. To a certain extent, it can reduce the puncture error caused by the thoracic displacement caused by respiratory movement, compensate for the body surface movement caused by breathing in real time, improve the alignment accuracy of the needle entry point, and thus improve the surgical accuracy of percutaneous puncture.

[0082] In some embodiments, using binocular images to locate the puncture instrument and the target point includes: obtaining the coordinates of the top and end of the puncture instrument and the coordinates of the target point in the camera coordinate system using binocular images; obtaining a first feature vector based on the coordinates of the top and end of the puncture instrument in the camera coordinate system, and obtaining a second feature vector based on the coordinates of the target point in the camera coordinate system. The first feature vector is used to represent the positioning information of the puncture instrument, and the second feature vector is used to represent the positioning information of the target point.

[0083] Taking a puncture instrument including an ablation needle as an example, the ablation needle has a specific geometric shape and size, and is an elongated needle-shaped object. The ablation needle can be regarded as a rigid three-dimensional object. The ablation needle is fixed to the base of the end effector of the robotic arm. The ablation needle model can be simplified into an elongated cylinder, and the length of this cylinder is L , the radius r can be ignored. The part where the ablation needle is connected to the robotic arm (the base of the end effector) is defined as the tip , and the tip part of the ablation needle is defined as the end . Furthermore, the ablation needle model can be expressed as: Ablation needle feature vector: ; , representing the coordinates of the end of the ablation needle in the camera coordinate system; , representing the coordinates of the tip of the ablation needle in the camera coordinate system; Satisfying the constraint condition .

[0084] The target target includes two markers for marking the tip target position and the end target position of the ablation needle. The positions of these two markers (target targets) are respectively denoted as and , representing the target position that the tip of the ablation needle should reach, representing the target position that the tip (end) of the ablation needle should reach, satisfying the constraint condition .

[0085] In a specific implementation, obtaining the coordinates of the tip and end of the puncture instrument and the coordinates of the target target in the camera coordinate system using binocular images can further include: inputting the left-eye image into a pre-trained key point detection model to obtain the coordinates of the tip and end of the puncture instrument and the coordinates of the target target in the target surgical area; using the right-eye image, calculating the disparity through a stereo matching algorithm, and combining the binocular camera internal parameter matrix to convert from the image coordinate system to the camera coordinate system.

[0086] In some examples, the key point detection model can adopt the HRNet model to respectively extract the end ( ) and the tip ( ) of the ablation needle from the left-eye image, the target targets and The image coordinates of are respectively denoted as , , The HRNet model can accurately extract the positions of key points in the left eye image and maintain good stability and accuracy in low resolution and noisy environments. The coordinates of the top and end of the puncture instrument and the target point in the camera coordinate system are recorded as , , and .

[0087] In the specific implementation, according to the parameters of the binocular camera, the pixel coordinates in the image coordinate system can be Convert to three-dimensional coordinates in the camera coordinate system , coordinate system transformations can include: a. Calculate parallax: Calculate the given pixel through the stereo matching algorithm The disparity between the left and right images , which is the horizontal offset between the pixel position in the left image and the corresponding pixel position in the right image.

[0088]

[0089] In the formula, , They represent the horizontal coordinates of the pixel position in the left image and the corresponding pixel position in the right image respectively.

[0090] b. Calculate depth: By the focal length of the binocular camera , baseline distance and parallax , the depth of the object can be calculated ,depth is the actual distance from the object to the stereo camera.

[0091]

[0092] In the formula, represents the focal length of the camera, Indicates the horizontal distance between the two cameras of the binocular camera (baseline distance), and is a constant determined by binocular camera calibration.

[0093] c. Convert from image coordinate system to camera coordinate system: Using the intrinsic parameter matrix of the binocular camera , which can transform pixel coordinates into three-dimensional coordinates in the camera coordinate system .

[0094]

[0095] In the formula, is used to describe the internal parameters of the binocular camera, including the focal length and the offset of the principal point (optical center):

[0096] In the formula, f x and f y respectively represent the representations of the focal length in the x , y directions, with the unit of pixel; c x , c y respectively represent the horizontal and vertical coordinates of the optical center in the image.

[0097] By using the left-eye image to recognize the coordinates of the puncture instrument and the target, and using the right-eye image for depth calculation, the coordinates of the puncture instrument and the target can be converted from the image coordinate system to the camera coordinate system, providing accurate three-dimensional positioning information for the visual servo tracking process to improve the puncture accuracy.

[0098] After obtaining the binocular images of the target surgical area captured by the binocular camera, it further includes: performing normalization processing on the left-eye image; and resampling the normalized left-eye image to adjust the resolution of the left-eye image to the target resolution. This can provide appropriate target image quality for subsequent processing and is beneficial to obtaining accurate positioning results.

[0099] In some examples, the target resolution is 512×512. Since the parameters of different models of binocular cameras may be different, before inputting the left-eye image into the key point detection model for coordinate extraction, adjusting the resolution of the left-eye image to the target resolution of 512×512 can provide appropriate image quality for subsequent processing.

[0100] In this embodiment, by using the three-dimensional coordinates of the puncture instrument and the target in the camera coordinate system to obtain the feature vectors of the puncture instrument and the target, the puncture instrument and the target can be accurately positioned in the three-dimensional space, providing accurate positioning information for the visual servo process to improve the puncture accuracy.

[0101] In some embodiments, define the Jacobian matrix: The position of the end of the robotic arm (actuator) is denoted as , construct the Jacobian matrix , and define the Jacobian matrix as the change amount of the feature vector of the ablation needle in the camera coordinate system with respect to the change amount of the position of the end of the robotic arm (actuator) in the robotic arm coordinate system, that is, used to describe the differential of the first change amount with respect to the second change amount, . Further, the motion control module is further configured to: a. Move the end of the robotic arm to the initial position. Taking a six-degree-of-freedom robotic arm as an example, the initial position is expressed as .

[0102] b. Move each position component of the end of the robotic arm at the initial position once respectively, to obtain the offset poses representing the end of the robotic arm after each position component moves, and record the first change amount after each movement. The first change amount is the change amount of the first eigenvector in the camera coordinate system.

[0103] Specifically, at the initial position each position component makes one movement, to obtain 6 linearly independent offset poses , representing the position of the ablation needle in the robotic arm coordinate system, I represents the identity matrix, representing the six-row and six-column identity matrix, record after each movement.

[0104] c. Based on the offset poses and the initial position, calculate the second change amount. The second change amount is the position change amount of the end of the robotic arm in the robotic arm coordinate system.

[0105] d. Calculate the differential of the first change amount with respect to the second change amount, to obtain the initial Jacobian matrix. Specifically, through and the initial Jacobian matrix is obtained.

[0106] During the percutaneous puncture surgery, start visual servo based on the initial Jacobian matrix, calculate the motion speed of the robotic arm by using the Jacobian matrix and the error between the puncture instrument and the target, update the Jacobian matrix in real time, and the robotic arm drives the puncture instrument to always accurately track the target.

[0107] In some embodiments, according to the error between the puncture instrument and the target obtained by positioning, control the motion of the robotic arm so that the distance between the puncture instrument and the target remains the same during the percutaneous puncture surgery, including: calculating the error between the first eigenvector and the second eigenvector; in the case that the error does not meet the preset condition, calculate the motion speed of the robotic arm by using the Jacobian matrix and the error; control the motion of the robotic arm according to the calculated motion speed of the robotic arm, and update the Jacobian matrix.

[0108] When the error meets the preset conditions, the current visual servo is maintained. When the error is not met, the robotic arm will move to achieve the navigation effect of automatically following the target during the surgical process. The Jacobian matrix is used to describe the differential of the first change quantity with respect to the second change quantity. The first change quantity is the change quantity of the first eigenvector in the camera coordinate system, and the second change quantity is the position change quantity of the end of the robotic arm in the robotic arm coordinate system.

[0109] Construct the error between the first eigenvector and the second eigenvector: , In the formula, represents the eigenvector of the target, that is, the second eigenvector, ; represents the eigenvector of the ablation needle, that is, the first eigenvector, .

[0110] In some examples, the movement speed of the robotic arm (end) is calculated by the following formula: , In the formula, represents the pseudo-inverse of the Jacobian matrix, represents the control gain, represents the movement speed of the robotic arm (end effector).

[0111] Since each degree of freedom of the six-degree-of-freedom robotic arm can move, the calculated is a value containing 6 degrees of freedom, which is a 6D vector. Each component represents the speed in one degree of freedom. After running for a unit time (such as 1 second) at this speed, the position will change accordingly.

[0112] In some examples, the Broyden method is used to update the Jacobian matrix, and the formula is as follows:

[0113] In the formula, represents the change quantity of the eigenvector of the ablation needle in the camera coordinate system, represents the current (before update) Jacobian matrix, represents the updated Jacobian matrix, is the difference between the actual movement speed commands executed in adjacent cycles. T represents the transpose, γ represents the damping factor, which is used to enhance numerical stability so that the situation of division by zero does not occur during the calculation process. Since the larger the damping factor, the smaller the result obtained, J the slower the update. From the perspective of the movement of the robotic arm, the movement of the robotic arm will also be more stable, and stable tracking during the visual servo process can be achieved.

[0114] During percutaneous puncture surgery, the error is calculated cyclically and judged whether the error meets the preset conditions, and then the movement of the robot arm is controlled until the norm of the error When the preset conditions are met, the system is judged to be convergent, where , unit: mm. When the system converges, visual servoing is maintained, so that when the target point moves with the ups and downs of the patient's body (such as the thorax), the puncture instrument can always maintain the same distance from the target point, which can reduce the puncture error caused by the displacement of the thorax caused by respiratory movement to a certain extent.

[0115] In this embodiment, the Jacobian matrix and the error between the puncture instrument and the target point are used for visual servoing. The robotic arm drives the puncture instrument to always accurately track the changes of the target point, so that the puncture instrument always maintains the same distance from the target point. To a certain extent, the puncture error caused by chest displacement caused by respiratory movement can be reduced, and the surface movement caused by breathing can be compensated in real time, which improves the alignment accuracy of the needle insertion point, thereby improving the surgical accuracy of percutaneous puncture.

[0116] In a specific implementation, the visual servo process of controlling the movement of the robotic arm according to the error between the puncture instrument and the target point obtained by positioning can be implemented by gradient descent or reinforcement learning methods, or by other methods, which are not specifically limited in this embodiment.

[0117] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0118] Embodiment three: Another embodiment of the present application relates to an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods in the above-mentioned embodiments.

[0119] Among them, the memory and the processor are connected in a bus manner. The bus can include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art, so they will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices over a transmission medium. The data processed by the processor is transmitted over a wireless medium via an antenna. Further, the antenna also receives data and transmits the data to the processor.

[0120] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store the data used by the processor when executing operations.

[0121] Embodiment 4: Another embodiment of the present application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above method embodiments are implemented.

[0122] That is, those skilled in the art can understand that all or part of the steps in implementing the above method embodiments can be completed by instructing relevant hardware through a program. The program is stored in a storage medium, including several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM for short), random access memories (RAM for short), magnetic disks, or optical discs, etc., which can store program codes.

[0123] Embodiment 5: Another embodiment of the present application relates to a percutaneous puncture surgery navigation system, as Figure 2 shown, including: A robotic arm 11 with a puncture instrument 12 mounted at its end; A binocular camera 13 for capturing binocular images of the target surgical area; and The electronic device of the above embodiment.

[0124] In some examples, the robotic arm 11 is a six-degree-of-freedom robotic arm, and the puncture instrument 12 is mounted on the base of the end effector of the robotic arm 11. The puncture instrument 12 is a puncture needle, such as an ablation needle.

[0125] The percutaneous puncture surgical navigation system provided by this embodiment uses a binocular camera to capture binocular images of the target surgical area, uses the binocular images to real-time locate the puncture instrument and the target point, and controls the movement of the robotic arm according to the error between the located puncture instrument and the target point, so that the distance between the puncture instrument and the target point remains unchanged. The surgical navigation cost of this surgical robot is significantly lower than that of the existing infrared optical positioning system for surgical navigation, and the operation is simple. Without prior calibration, visual servo navigation during the percutaneous puncture surgery can be achieved, so that the distance between the puncture instrument and the target point is always the same during the percutaneous puncture surgery, which can reduce the puncture error caused by the thoracic displacement caused by respiratory movement to a certain extent and improve the surgical accuracy of percutaneous puncture.

[0126] The calibration operation of the traditional infrared optical positioning system is complex. For example, the calibration ball easily interferes with the needle insertion and the calibration process is time-consuming, which increases the surgical preparation cost. However, this embodiment only requires the internal parameter calibration of the binocular camera at the factory or before the operation, and can combine the feature modeling of the puncture instrument (such as the geometric model of the ablation needle) with the target marker design (for example, using a circular marker to mark the target to avoid blocking the end of the puncture instrument), bypassing the high-cost process of infrared optical calibration and achieving the surgical navigation effect of the device being ready to use.

[0127] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A percutaneous puncture surgical navigation method, characterized in that, Including: Obtaining binocular images of a target surgical area captured by a binocular camera; Real-time positioning of a puncture instrument and a target point using the binocular images, where the puncture instrument is installed at the end of a robotic arm, and the target point is used to mark the target position of the puncture instrument; According to the error between the puncture instrument and the target point obtained by the positioning, controlling the movement of the robotic arm so that the distance between the puncture instrument and the target point remains the same during the percutaneous puncture surgery, realizing dynamic compensation for the displacement of the target point caused by respiratory movement.

2. The percutaneous puncture surgery navigation method according to claim 1, wherein Real-time positioning of a puncture instrument and a target point using the binocular images includes: Obtaining the coordinates of the tip and the end of the puncture instrument and the coordinates of the target point in the camera coordinate system by parallax calculation using the binocular images; Obtaining a first feature vector based on the coordinates of the tip and the end of the puncture instrument, and obtaining a second feature vector based on the coordinates of the target point in the camera coordinate system. The first feature vector is used to represent the positioning information of the puncture instrument, and the second feature vector is used to represent the positioning information of the target point.

3. The percutaneous puncture surgery navigation method according to claim 2, wherein The binocular images include a left-eye image and a right-eye image; obtaining the coordinates of the tip and the end of the puncture instrument and the coordinates of the target point in the camera coordinate system by parallax calculation using the binocular images includes: Inputting the left-eye image into a pre-trained key-point detection model to obtain the coordinates of the tip and the end of the puncture instrument and the coordinates of the target point in the target surgical area; Using the right-eye image, calculating the parallax through a stereo matching algorithm, and combining the internal parameter matrix of the binocular camera to convert the image coordinate system to the camera coordinate system.

4. The percutaneous puncture surgery navigation method according to claim 2, wherein According to the error between the puncture instrument and the target point obtained by the positioning, controlling the movement of the robotic arm so that the distance between the puncture instrument and the target point remains the same during the percutaneous puncture surgery, realizing dynamic compensation for the displacement of the target point caused by respiratory movement, includes: Calculating the error between the first feature vector and the second feature vector; When the error does not meet the preset condition, calculating the movement speed of the robotic arm using the Jacobian matrix and the error. The Jacobian matrix is used to describe the differential of the first change quantity with respect to the second change quantity. The first change quantity is the change quantity of the first feature vector in the camera coordinate system, and the second change quantity is the position change quantity of the end of the robotic arm in the robotic arm coordinate system; Controlling the movement of the robotic arm according to the calculated movement speed of the robotic arm and updating the Jacobian matrix.

5. The percutaneous puncture surgery navigation method according to claim 4, wherein Also including: Moving the end of the robotic arm to the initial position; Moving each position component of the end of the robotic arm at the initial position once to obtain the offset pose representing the end of the robotic arm after each position component moves, and recording the first change quantity after each movement; Calculating the second change quantity based on the offset pose and the initial position; Calculating the differential of the first change quantity with respect to the second change quantity to obtain the initial Jacobian matrix.

6. The percutaneous puncture surgery navigation method according to claim 1, wherein The binocular images include a left-eye image and a right-eye image; after obtaining the binocular images of the target surgical area captured by the binocular camera, it also includes: Performing normalization processing on the left-eye image; Resample the normalized left-eye image to adjust the resolution of the left-eye image to the target resolution.

7. A percutaneous puncture surgical navigation device, characterized in that, Including: An image acquisition module for acquiring binocular images of the target surgical area captured by a binocular camera; A positioning module for real-time positioning of a puncture instrument and a target point using the binocular images, the puncture instrument being mounted at the end of a robotic arm, and the target point being used to mark the target position of the puncture instrument; A visual servo control module for controlling the movement of the robotic arm according to the error between the puncture instrument and the target point obtained by positioning, so as to keep the same distance between the puncture instrument and the target point during the percutaneous puncture surgery process, and achieve dynamic compensation for the displacement of the target point caused by respiratory movement.

8. The percutaneous puncture surgical navigation device according to claim 7, wherein, The positioning module is used for: Obtaining the coordinates of the tip and the end of the puncture instrument and the coordinates of the target point in the camera coordinate system by parallax calculation using the binocular images; obtaining a first feature vector based on the coordinates of the tip and the end of the puncture instrument, and obtaining a second feature vector based on the coordinates of the target point in the camera coordinate system, the first feature vector being used to characterize the positioning information of the puncture instrument, and the second feature vector being used to characterize the positioning information of the target point; The visual servo control module is used for: Calculating the error between the first feature vector and the second feature vector; In the case where the error does not meet the preset condition, calculating the movement speed of the robotic arm using the Jacobian matrix and the error, the Jacobian matrix being used to describe the differential of the first change amount with respect to the second change amount, the first change amount being the change amount of the first feature vector in the camera coordinate system, and the second change amount being the position change amount of the end of the robotic arm in the robotic arm coordinate system; Controlling the movement of the robotic arm according to the calculated movement speed of the robotic arm and updating the Jacobian matrix.

9. An electronic device, characterized in that, Including: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the percutaneous puncture surgery navigation method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the percutaneous puncture surgery navigation method according to any one of claims 1 to 6.

11. A percutaneous puncture surgical navigation system, characterized in that, Including: A robotic arm with a puncture instrument mounted at its end; A binocular camera for capturing binocular images of the target surgical area; And The electronic device according to claim 9.

Images