Surgical instrument and cannula position alignment method and system

Abstract

The invention belongs to the field of medical treatment, and provides a surgical instrument and cannula alignment method and system.The microscope of the surgical instrument is provided with a camera for shooting an instrument needle point and a cannula, firstly, a cannula area, a needle point area and the objective lens height when the definition index of the two areas is the highest are detected in an image shot by the camera; and determining the depth difference between the needle point and the sleeve according to the two height differences. Then, extracting the coordinates of the central point of the casing hole in the casing area; and recognizing the coordinates of the needle point in the needle point area. And finally, based on the conversion relation between the image coordinate system and the instrument movement coordinate system, translating the needle point until the positions of the needle point and the central point of the cannula hole coincide in the image, and controlling the distance of the movement depth difference of the needle point towards the cannula. According to the invention, the robot can clamp the instrument to automatically align the center of the cannula, the complexity of instrument operation is effectively reduced, additional markers and special equipment are not needed, and alignment can be completed by directly using an operating microscope.

Description

Technical Field

[0001] The present invention belongs to the medical field, and particularly relates to a method and system for aligning the positions of a surgical instrument and a cannula. Background Art

[0002] Due to the high vulnerability of the operation object (intraocular tissue) in ophthalmic surgery, extremely high requirements are imposed on the surgical precision and stability. Intraocular surgical operations often reach the limit of human hand-eye coordination and perception ability, and highly depend on the technical level and operation experience of doctors. In recent years, ophthalmic surgical robots have become important auxiliary tools for improving surgical safety and reducing the operation burden of doctors because of their high-precision and high-stability motion control capabilities.

[0003] During the intraocular surgery process, doctors usually need to first fix the scleral cannula on the scleral area on the surface of the eyeball, and then introduce surgical instruments through the cannula to complete various intraocular operations. Although the application of ophthalmic surgical robots has effectively improved the safety and precision of intraocular operations, new challenges have also emerged. For example, since the robotic system does not have the intuitive judgment ability of the human eye, the instrument needs to be more precisely aligned with the cannula entrance fixed on the sclera in order to smoothly enter the eye. This alignment process is complex, prolongs the surgical preparation time, and reduces the overall efficiency of the surgical process.

[0004] To solve the above problems, some scleral cannula positioning and alignment methods based on external sensor systems have been proposed in the prior art. For example, a surgical instrument integrated with two micro cameras and an inertial measurement unit (IMU) is used to estimate the spatial pose of the cannula by tracking two ArUco markers fixed on both sides of the scleral cannula, so as to provide an alignment target for the robot. However, this method requires additional marker deployment during the surgery process and relies on the sensor system of the instrument itself, increasing the complexity of the instrument.

[0005] In addition, a monocular vision system can also be used. The YOLO network is used to detect the scleral cannula in the microscopic image, and the detected local image area is input into the SC6D pose estimation algorithm to estimate the spatial pose information of the cannula. This method avoids the use of special markers, but still requires an independent visual perception module to be configured outside the surgical system, increasing the system complexity and deployment cost.

[0006] In summary, although the prior art has improved the alignment efficiency of the robot with the scleral cannula to a certain extent, there are generally problems such as high dependence on external sensors or markers, high system integration difficulty, and high economic cost. There is an urgent need for a technical solution that does not require additional external devices, can efficiently estimate the cannula pose, and guide the robot to autonomously align. Summary of the Invention

[0007] The present invention provides a method and system for aligning a surgical instrument with a cannula to solve the problems of high dependence on external sensors or markers, high system integration difficulty, and high economic cost in the prior art. To solve the above technical problems, the embodiments of the present invention disclose the following technical solutions: One aspect of the present invention provides a method for aligning a surgical instrument with a cannula. A camera with a shooting range covering the tip of the instrument and the cannula is installed on the microscope of the surgical instrument. The method includes: Detect the cannula area and the tip area in the image captured by the camera in real time; Adopt an autofocus algorithm to respectively obtain the objective lens height when the clarity index of the cannula area and the tip area is the highest, and determine the depth difference between the tip and the cannula according to the difference between the two heights; Extract the cannula hole area within the cannula area and determine the coordinates of the center point of the cannula hole in the image coordinate system; Identify the tip point within the tip area and determine the coordinates of the tip point in the image coordinate system; Based on the conversion relationship between the image coordinate system and the motion coordinate system of the surgical instrument, combined with visual servo technology, translate the tip until the position of the tip point coincides with the position of the center point of the cannula hole in the image, and then control the tip to move towards the cannula by the distance of the depth difference.

[0008] Optionally, the detecting the cannula area and the tip area in the image captured by the camera in real time includes: Obtain the image captured by the camera in real time; Use a preset target detection network to judge whether the cannula and the tip appear in the image. If so, output the cannula bounding box and the tip bounding box, and determine the cannula area in the image based on the cannula bounding box, and determine the tip area in the image based on the tip bounding box.

[0009] Optionally, after performing the step of obtaining the image captured by the camera in real time, the method further includes: Preprocess the image, and the preprocessing at least includes normalization and noise removal.

[0010] Optionally, the adopting an autofocus algorithm to respectively obtain the objective lens height when the clarity index of the cannula area and the tip area is the highest, and determining the depth difference between the tip and the cannula according to the difference between the two heights includes: Within a preset objective lens height adjustment range, adopt an autofocus algorithm to adjust the objective lens height in equal steps; Collect the images captured by the camera when the objective lens is at each height; For each image corresponding to each height, calculate the clarity index of the cannula area and the tip area respectively; Record the objective lens height corresponding to the highest clarity index of the cannula region as ; record the objective lens height corresponding to the highest clarity index of the needle tip region as ; Calculate the depth difference between the needle tip and the cannula using the following formula : .

[0011] Optionally, extracting the cannula hole region within the cannula region and determining the coordinates of the center point of the cannula hole in the image coordinate system includes: Using a preset semantic segmentation algorithm to extract the cannula hole region within the cannula region; Calculate the coordinates of the center point of the cannula hole according to the following formula : where is the coordinate of the i-th pixel point within the cannula hole region in the image coordinate system; N is the total number of pixel points within the cannula hole region.

[0012] Optionally, identifying the needle tip point within the needle tip region and determining the coordinates of the needle tip point in the image coordinate system includes: Using a preset deep learning model to identify the pixel point representing the needle tip point within the needle tip region; Taking the coordinate of the pixel point in the image coordinate system as the coordinate of the needle tip point .

[0013] Optionally, determining the conversion relationship between the image coordinate system and the surgical instrument motion coordinate system in the following way: Controlling the surgical instrument to move along multiple preset trajectory points; At each trajectory point, obtaining the pose coordinate of the surgical instrument in the motion coordinate system and the image coordinate of the needle tip point in the image coordinate system; Establishing a conversion model between the image coordinate system and the surgical instrument motion coordinate system according to the pose coordinate and the image coordinate corresponding to each trajectory point.

[0014] Optionally, based on the conversion relationship between the image coordinate system and the surgical instrument motion coordinate system, combining visual servo technology, translating the needle tip until the positions of the needle tip point and the center point of the cannula hole in the image coincide, and then controlling the needle tip to move towards the cannula by the distance of the depth difference, includes: Real-time monitoring of the coordinates of the needle tip point and the center point of the cannula hole in the image; Based on the conversion relationship between the image coordinate system and the surgical instrument motion coordinate system, control the tip of the needle to move along a plane parallel to the image plane under the condition of constant depth, so that the coordinates of the tip point of the needle and the center point of the cannula hole gradually approach and coincide; Control the tip of the needle to move towards the cannula along the depth direction by a distance of the depth difference, and the depth direction is perpendicular to the image plane.

[0015] Optionally, the method further includes: Calculate the coordinates of the tip point of the needle according to the following formula and the coordinates of the center point of the cannula hole the offset between : When the offset is less than the preset deviation threshold, it is determined that the coordinates of the tip point of the needle coincide with the coordinates of the center point of the cannula hole.

[0016] Another aspect of the present invention discloses a surgical instrument and cannula alignment system. A camera with a shooting range covering the tip of the instrument and the cannula is installed on the microscope of the surgical instrument, and the system is applied to the surgical instrument and cannula alignment method disclosed in the foregoing aspect.

[0017] A surgical instrument and cannula alignment method and system disclosed by the present invention. A camera with a shooting range covering the tip of the instrument and the cannula is installed on the microscope of the surgical instrument. First, in the image captured by the camera in real time, detect the cannula area and the tip area of the needle, and respectively obtain the objective lens height when the clarity index of the cannula area and the tip area is the highest. Determine the depth difference between the tip of the needle and the cannula according to the difference between the two heights. Then, extract the cannula hole area within the cannula area and determine the coordinates of the center point of the cannula hole in the image coordinate system; identify the tip point within the tip area of the needle and determine the coordinates of the tip point in the image coordinate system. Finally, based on the conversion relationship between the image coordinate system and the surgical instrument motion coordinate system, translate the tip of the needle until the position of the tip point of the needle coincides with the position of the center point of the cannula hole in the image, and then control the tip of the needle to move towards the cannula by a distance of the depth difference. The present invention can realize the autonomous alignment of the robot-held instrument to the center of the cannula, effectively reduce the operation complexity of the instrument entering the eye, has no need for additional markers and special equipment, and can directly complete the alignment operation using the surgical microscope, which helps to simplify the surgical process, reduce the equipment cost, and improve the surgical efficiency.

[0018] The summary of the invention is provided to introduce a selection of concepts in a simplified form, which will be further described in the detailed implementation manners below. The summary of the invention is not intended to identify the key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. Brief Description of the Drawings

[0019] The above and other objects, features, and advantages of the present disclosure will become more apparent by describing the exemplary embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, wherein, in the exemplary embodiments of the present disclosure, the same reference numerals generally represent the same components.

[0020] Figure 1 It is a schematic flowchart of a method for aligning a surgical instrument with a cannula disclosed in an embodiment of the present invention; Figure 2 It is for implementing Figure 1 a schematic flowchart of step S100 in; Figure 3 It is for implementing Figure 1 a schematic flowchart of step S200 in; Figure 4 It is a schematic flowchart of a method for determining the conversion relationship between an image coordinate system and a surgical instrument motion coordinate system disclosed in an embodiment of the present invention; Figure 5 It is for implementing Figure 1 a schematic flowchart of step S500 in. Detailed Description of the Embodiments

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0022] The term "including" and its variations used herein mean open inclusion, i.e., "including but not limited to". Unless otherwise specified, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "an exemplary embodiment" and "an embodiment" mean "at least one exemplary embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. There may be other explicit and implicit definitions hereinafter.

[0023] Figure 1 It is a schematic flowchart of a method for aligning a surgical instrument with a cannula disclosed in an embodiment of the present invention. A camera with a shooting range covering the tip of the instrument and the cannula is installed on the microscope of the surgical instrument. The microscope is a necessary device of the surgical system. The present invention only needs to install a camera on the microscope and does not need to add other devices outside the original devices of the surgical system.

[0024] In the embodiment of the present invention, a high-resolution camera is integrated on an ophthalmic surgical microscope for real-time acquisition of images of the surgical area. The acquired images are consistent with the subjective field of view of the microscope, providing original visual information support for subsequent functions such as target detection, focusing control, image-robot coordinate transformation, and visual servo.

[0025] As Figure 1 shown, the method includes the following steps: Step 100: Detect the cannula region and the needle tip region in the images captured by the camera in real time.

[0026] This step realizes the visual recognition of key targets (scleral cannula and needle tip) in the ophthalmic surgical scene, which is the basis for subsequent image analysis, spatial positioning, and control.

[0027] In an embodiment disclosed by the present invention, as Figure 2 shown, step S100 can be implemented by the following sub-steps: Step 101: Obtain the images captured by the camera in real time.

[0028] The images are obtained in real time through the camera installed on the surgical microscope.

[0029] Step 102: Use a preset target detection network to determine whether a cannula and a needle tip appear in the image.

[0030] The images captured in real time are input into a pre-trained deep learning target detection network, such as the YOLOv11 network, which can perform high-precision detection of the cannula and the needle tip. In the embodiment disclosed by the present invention, other target detection networks can also be used to complete the recognition of the cannula and the needle tip. The model needs to be fine-tuned and trained in advance on a dataset containing annotated images of the cannula and the needle tip to enhance its robustness to situations such as reflection, blur, and occlusion in clinical images. Through network inference, a set of detection results including class labels, bounding box positions (upper left coordinates and width and height), and target confidence levels is output. If there are target objects of the cannula or needle tip category with a confidence level higher than a set threshold (such as 0.5) in the detection results, it is regarded as a successful target detection, and step 103 is continued.

[0031] Step 103: Output the cannula bounding box and the needle tip bounding box, and determine the cannula region in the image based on the cannula bounding box, and, determine the needle tip region in the image based on the needle tip bounding box.

[0032] According to the bounding box information output by the target detection network, the bounding boxes corresponding to the cannula and the needle tip are selected, and the two regions in the image are cropped out and used as the input for subsequent modules respectively. Specifically, the image region corresponding to the cannula bounding box is extracted as the cannula region, and at the same time, the image region corresponding to the needle tip bounding box is extracted as the needle tip region.

[0033] If no cannula and needle tip are detected in the image (such as when the target is missing or the confidence level is lower than the threshold), an exception handling mechanism can be triggered. For example, the user can be prompted to adjust the microscope field of view, etc., to ensure the stability and reliability of the entire recognition and operation process.

[0034] In an embodiment disclosed by the present invention, after obtaining the real-time captured image of the camera, in order to improve the accuracy and robustness of subsequent target detection and image analysis, necessary preprocessing operations need to be performed on the image. The preprocessing link can significantly enhance the quality of the image, making the image more suitable for input into the deep learning network model for feature extraction and recognition.

[0035] First, perform normalization processing on the image. The purpose of normalization is to convert the pixel values of the image into a unified numerical range to eliminate the data scale differences caused by factors such as different image sources, exposure times, and light intensities, thereby accelerating the model convergence speed and improving the inference stability.

[0036] Secondly, perform noise removal processing on the image. In the actual surgical environment, the images captured by the camera may be affected by various factors such as light reflection, microscope lens stains, and image transmission interference, resulting in high-frequency noise, random spots, or edge burrs in the image. These noises will interfere with the feature extraction process of the target detection algorithm and reduce the detection accuracy. Therefore, image denoising algorithms such as Gaussian filtering and median filtering can be used to process the original image.

[0037] Step 200: Adopt an autofocus algorithm to respectively obtain the objective lens heights when the clarity indexes of the cannula area and the needle tip area are the highest, and determine the depth difference between the needle tip and the cannula according to the difference between the two heights.

[0038] Through the autofocus algorithm, respectively align the cannula area and the needle tip area in the image, and based on the image clarity evaluation index, obtain their corresponding optimal focusing heights, so as to calculate the depth difference between the two. This provides depth information for subsequent precise alignment.

[0039] In an embodiment disclosed by the present invention, as Figure 3 shown, the following sub-steps can be adopted to implement step S200: Step S201: Within the preset objective lens height adjustment range, adopt an autofocus algorithm to adjust the objective lens height in equal steps.

[0040] According to the focal range of the surgical microscope, a reasonable objective lens height adjustment range (e.g. ±1.0 mm) is set, and the range is divided into several height levels with equal spacing (e.g. 5 μm per step). The automatic focusing process will gradually adjust the vertical height of the objective lens within this range, and each height change is accurately controlled by a stepper motor or the microscope's built-in Z-axis control device.

[0041] Step S202: collecting images taken by the camera when the objective lens is at each height.

[0042] With each fine adjustment of the objective lens height, the image at the current height is collected in real time, and the Z-axis height value of the current objective lens is recorded.

[0043] Step S203: for each image corresponding to the height, the clarity indexes of the cannula area and the needle tip area are calculated respectively.

[0044] The cannula area and needle tip area are extracted from the YOLOv11 target detection results, and the clarity index of the two areas at each height is calculated using image clarity evaluation algorithms (such as Laplace variance, Tenengrad, gradient energy, etc.). This index is used to quantify the clarity of the image, and the larger the value, the clearer it is.

[0045] Step S204: Record the height of the objective lens corresponding to the highest clarity index of the casing area as , the height of the objective lens corresponding to the highest clarity index of the needle tip area is recorded as .

[0046] By comparing the clarity index curves at each height, the corresponding objective lens heights when the clarity indexes of the cannula area and the needle tip area reach the maximum value are found, and recorded as the optimal focusing heights of the cannula. Optimal focusing height with needle tip The two heights represent their relative Z-axis (depth) positions in three-dimensional space.

[0047] Step S205: Calculate the depth difference between the needle tip and the cannula using the following formula: : .

[0048] The depth difference is an important displacement parameter for the subsequent precise insertion of the needle tip into the cannula hole along the Z direction. After the image and robot coordinate systems are established, it can be directly used to guide the surgical robot to perform precise movement in the depth direction.

[0049] Step 300: extracting the casing hole region within the casing region, and determining the coordinates of the casing hole center point in the image coordinate system.

[0050] In an embodiment disclosed by the present invention, the following sub-steps may be adopted to implement step S300: (1) Extract the sleeve hole area within the sleeve area by using a preset semantic segmentation algorithm.

[0051] Further segment the sleeve hole area from the sleeve area detected by the YOLOv11 network. The input is the cropped sleeve area image, and the output is a binary mask image of the same size as the image, where the pixels belonging to the sleeve hole area are assigned a value of 1, and the remaining areas are assigned a value of 0.

[0052] (2) By obtaining the set of coordinate points of all pixels with a value of 1 in the binary image generated in step (1), statistically calculate the coordinates of these pixels in the image, and then calculate the average value of the coordinates of all pixels respectively to obtain the average coordinate of the sleeve hole area.

[0053] Calculate the coordinates of the center point of the sleeve hole according to the following formula : where, is the coordinate of the i-th pixel point in the sleeve hole area in the image coordinate system; N is the total number of pixel points in the sleeve hole area.

[0054] Step 400: Identify the tip point within the tip area and determine the coordinates of the tip point in the image coordinate system.

[0055] In an embodiment disclosed by the present invention, the following method may be adopted to implement step S400: Use a preset deep learning model to identify the pixel point representing the tip point within the tip area.

[0056] The input of the model is the image area within the tip detection bounding box, and the output is a key point. Take this point as the tip point, and take the coordinates of this pixel point in the image coordinate system as the coordinates of the tip point .

[0057] Step 500: Based on the conversion relationship between the image coordinate system and the surgical instrument motion coordinate system, combined with visual servo technology, translate the tip until the position of the tip point coincides with the position of the sleeve hole center point in the image, and then control the tip to move the distance of the depth difference towards the sleeve.

[0058] In an embodiment disclosed by the present invention, to achieve an accurate conversion relationship between the image coordinate system and the surgical instrument motion coordinate system (i.e., the robot coordinate system), a data fitting method based on multiple registration points is adopted. As Figure 4 shown, the following method is adopted to determine the conversion relationship between the image coordinate system and the surgical instrument motion coordinate system: Step 051: Control the surgical instrument to move along multiple preset trajectory points.

[0059] Control the surgical needle clamped by the robot to move precisely along multiple preset trajectory points, which should be distributed at representative spatial positions within the field of view. After each movement to a trajectory point, ensure stable movement without jitter and pause for image acquisition and position recording.

[0060] Step 052: At each trajectory point, obtain the pose coordinates of the surgical instrument in the motion coordinate system and the image coordinates of the needle tip point in the image coordinate system.

[0061] When the robot stays stably at a certain trajectory point position, record the three-dimensional pose information of this point in the robot coordinate system, which can be obtained through the feedback of the robot controller. At the same time, through the image processing module, extract the image coordinates of the needle tip point from the current camera image. For each trajectory point, record a pair of data, namely the robot pose coordinates and the image coordinates.

[0062] Step 053: According to the pose coordinates and image coordinates corresponding to each trajectory point, establish a conversion model between the image coordinate system and the motion coordinate system of the surgical instrument.

[0063] Perform registration modeling on all the collected data pairs, that is, establish the mapping relationship between coordinate systems according to multiple three-dimensional points and their two-dimensional projection points in the image, and methods such as least squares fitting can be used.

[0064] In an embodiment disclosed in the present invention, as Figure 5 shown, the following sub-steps can be used to implement step S500: Step S501: Real-time monitor the coordinates of the needle tip point and the center point of the cannula hole in the image.

[0065] Step S502: Based on the conversion relationship between the image coordinate system and the motion coordinate system of the surgical instrument, control the needle tip to move along a plane parallel to the image plane under the condition of constant depth, so that the coordinates of the needle tip point and the center point of the cannula hole gradually approach and coincide.

[0066] This step is the first stage of the visual servo process: lateral alignment.

[0067] The movement of the needle tip in the image plane corresponds to the movement of the robot end effector in the X-Y direction (parallel to the microscope image plane) in the motion coordinate system of the surgical instrument. Using the conversion model between the image and the robot coordinate system, map the offset in the image to the movement instruction of the robot.

[0068] In an embodiment disclosed in the present invention, Calculate the needle tip point coordinates according to the following formula and the coordinates of the center point of the cannula hole The offset between : At offset When it is less than a preset deviation threshold, it is determined that the coordinates of the needle tip point coincide with the coordinates of the center point of the cannula hole.

[0069] The robot control module continuously adjusts the needle tip position accordingly, acquires new images in real time to update the needle tip position, and repeats this process until the offset meets the requirements and precise alignment is achieved.

[0070] Step S503: Control the needle tip to move toward the cannula along the depth direction by a distance equal to the depth difference, wherein the depth direction is perpendicular to the image plane.

[0071] This step is the second stage of the visual servoing process: longitudinal alignment.

[0072] When the needle tip has coincided with the center point of the cannula hole on the image plane, that is, the lateral alignment is completed, the longitudinal alignment operation is performed. At this time, the needle tip is adjusted according to the depth difference obtained by automatic focusing in the previous step. Move down along the normal direction of the image plane (i.e. the microscope imaging direction or the robot Z axis). During the downward movement, the image is continuously collected in real time and the coordinates of the needle tip and the hole center are monitored to ensure that the needle tip remains aligned during movement. This dynamic closed-loop control ensures that there will be no deviation during the insertion process, improving insertion accuracy and safety.

[0073] An embodiment of the present invention further discloses a surgical instrument and cannula alignment system, wherein a camera with a shooting range covering the instrument needle tip and cannula is installed on a microscope of the surgical instrument, and the system is applied to the surgical instrument and cannula alignment method disclosed in the aforementioned embodiment.

[0074] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for aligning a surgical instrument with a cannula, characterized in that, A camera whose shooting range covers the tip of the instrument and the cannula is installed on the microscope of the surgical instrument. The method includes: Detect the cannula area and the tip area in the image captured by the camera in real time; Adopt an autofocus algorithm to respectively obtain the objective lens height when the clarity index of the cannula area and the tip area is the highest, and determine the depth difference between the tip and the cannula according to the difference between the two heights; Extract the cannula hole area within the cannula area, and determine the coordinates of the center point of the cannula hole in the image coordinate system; Identify the tip point within the tip area, and determine the coordinates of the tip point in the image coordinate system; Based on the conversion relationship between the image coordinate system and the movement coordinate system of the surgical instrument, combined with visual servo technology, translate the tip until the position of the tip point coincides with the center point of the cannula hole in the image, and then control the tip to move towards the cannula by the distance of the depth difference.

2. The alignment method according to claim 1, wherein The detecting the cannula area and the tip area in the image captured by the camera in real time includes: Obtain the image captured by the camera in real time; Use a preset object detection network to judge whether the cannula and the tip appear in the image; If so, output the bounding box of the cannula and the bounding box of the tip, and determine the cannula area in the image based on the bounding box of the cannula, and determine the tip area in the image based on the bounding box of the tip.

3. The alignment method according to claim 2, characterized in that, After performing the step of obtaining the image captured by the camera in real time, the method further includes: Preprocess the image, and the preprocessing at least includes normalization and noise removal.

4. The alignment method according to claim 1, wherein The adopting an autofocus algorithm to respectively obtain the objective lens height when the clarity index of the cannula area and the tip area is the highest, and determine the depth difference between the tip and the cannula according to the difference between the two heights includes: Within a preset objective lens height adjustment range, use an autofocus algorithm to adjust the objective lens height in equal steps; Collect the images captured by the camera when the objective lens is at each height; For each image corresponding to each height, calculate the clarity indexes of the cannula area and the tip area respectively; Record the objective lens height corresponding to the highest clarity index of the sleeve area as and record the objective lens height corresponding to the highest clarity index of the tip area as ; The depth difference between the tip and the cannula is calculated using the following formula :[[]]END]] 。 5. The alignment method according to claim 1, characterized in that, The extracting the cannula hole area within the cannula area, and determining the coordinates of the center point of the cannula hole in the image coordinate system includes: Use a preset semantic segmentation algorithm to extract the cannula hole area within the cannula area; Calculate the coordinates of the center point of the casing hole according to the following formula : Among them, is the coordinate of the i-th pixel point in the casing hole area in the image coordinate system; N is the total number of pixel points in the casing hole area.

6. The alignment method according to claim 1, wherein The identifying the tip point within the tip area, and determining the coordinates of the tip point in the image coordinate system includes: Use a preset deep learning model to identify the pixel point representing the tip point within the tip area; Use the coordinates of the pixel point in the image coordinate system as the coordinates of the tip point .

7. The alignment method according to claim 1, wherein Determine the conversion relationship between the image coordinate system and the movement coordinate system of the surgical instrument in the following way: Control the surgical instrument to move along multiple preset trajectory points; At each trajectory point, obtain the pose coordinates of the surgical instrument in the movement coordinate system, and the image coordinates of the tip point in the image coordinate system; According to the pose coordinates and image coordinates corresponding to each trajectory point, establish a conversion model between the image coordinate system and the movement coordinate system of the surgical instrument.

8. The alignment method according to claim 1, characterized in that, The based on the conversion relationship between the image coordinate system and the movement coordinate system of the surgical instrument, combined with visual servo technology, translate the tip until the position of the tip point coincides with the center point of the cannula hole in the image, and then control the tip to move towards the cannula by the distance of the depth difference includes: Real-time monitor the coordinates of the tip point and the center point of the cannula hole in the image; Based on the conversion relationship between the image coordinate system and the moving coordinate system of the surgical instrument, control the needle tip to move along a plane parallel to the image plane under the condition of constant depth, so that the coordinates of the needle tip point and the center point of the cannula hole gradually approach and coincide until they are the same; Control the needle tip to move towards the cannula along the depth direction by a distance of the depth difference, and the depth direction is perpendicular to the image plane.

9. The alignment method according to claim 8, wherein The method further includes: Calculate the coordinates of the tip point according to the following formula and the coordinates of the center point of the cannula hole The offset between is When the offset is less than a preset deviation threshold, it is determined that the coordinates of the tip point coincide with the coordinates of the center point of the cannula hole.

10. A surgical instrument and cannula alignment system, characterized in that, A camera with a shooting range covering the needle tip and the cannula is installed on the microscope of the surgical instrument, and the system is applied to the method for aligning the surgical instrument and the cannula according to any one of claims 1 to 9.

Images