Surgical suture needle navigation method and system based on magnetic sensing and AI vision

Abstract

The invention relates to a surgical suture needle navigation method based on magnetic sensing and AI vision. The surgical suture needle navigation method comprises the following steps: acquiring a movement track of a suture needle by using a magnetic sensor integrated on the suture needle; the spatial position of the suture needle at the current moment is obtained through the movement track, and the spatial position is projected to an endoscope image at the current moment through the external reference and the internal reference of the endoscope at the current moment to obtain the predicted position of the suture needle on the image at the current moment; according to the predicted position and a suture needle position in an image at a previous moment acquired by an endoscope, determining a search range of a suture needle in the image at the current moment, determining a suture needle area in the search range and obtaining a matching degree, and back-projecting pixel coordinates of the suture needle area into a three-dimensional space and obtaining a suture needle path; according to the sewing needle path and the matching degree of the positions on the sewing needle path, the actual track of the sewing needle is obtained, and the sewing needle is adjusted based on the actual track and the planned navigation path.

Description

Technical Field

[0001] The present invention relates to the field of medicine, and specifically to a surgical suture needle navigation method and system based on magnetic sensing and AI vision. Background Art

[0002] Accurately guiding the suture needle to the target position and performing puncture and suture along a predetermined path can significantly improve the success rate of the operation, reduce the damage to surrounding healthy tissues, shorten the operation time, and reduce the risk of postoperative complications. However, traditional suture needle positioning methods mainly rely on vision to guide the suture needle, but this method becomes very difficult in minimally invasive surgeries with limited vision. Common auxiliary positioning means, such as preoperative image planning combined with the identification of intraoperative anatomical markers, are easily affected by patient position changes, tissue deformation, and individual differences, resulting in deviations between the actual operation and the preoperative plan.

[0003] Magnetic sensing technology utilizes a generated known magnetic field to obtain the position and orientation information of the instrument in three-dimensional space in real time through a micro-sensor integrated on the suture needle, providing comprehensive spatial perception for doctors. In addition, magnetic sensing has a certain degree of penetrability and does not require direct optical visibility. However, magnetic sensing is susceptible to the influence of metal objects and electromagnetic interference in the operating room environment, which may lead to a decrease in positioning accuracy; and its accuracy may decrease as the distance between the sensor and the magnetic field generator increases. AI vision technology uses computer vision algorithms to analyze surgical images or video streams to achieve the identification, positioning, and tracking of specific targets such as suture needles. In the field of surgical navigation, AI vision uses a surgical endoscope or an external camera to obtain images of the surgical field, and through a trained deep learning model, it can accurately identify and track the position and posture of the suture needle in the image. However, AI vision is easily affected by factors such as occlusion of the surgical field and changes in lighting conditions. Using magnetic sensing or AI vision alone is difficult to fully meet the requirements of high-precision surgical suture needle navigation. Summary of the Invention

[0004] In order to improve the accuracy of surgical suture needle navigation, in the first aspect of the present invention, a surgical suture needle navigation method based on magnetic sensing and AI vision is provided. The method includes the following steps:

[0005] Using a magnetic sensor integrated on the suture needle to obtain the movement trajectory of the suture needle; obtaining the spatial position of the suture needle at the current moment through the movement trajectory, and using the external and internal parameters of the endoscope at the current moment to project the spatial position onto the endoscope image at the current moment to obtain the predicted position of the suture needle on the image at the current moment;

[0006] Determine the search range of the suture needle in the current image based on the predicted position and the position of the suture needle in the image acquired by the endoscope at the previous moment. Determine the suture needle area within the search range and obtain the matching degree. Back-project the pixel coordinates of the suture needle area into the three-dimensional space to obtain the suture needle path.

[0007] Obtain the actual trajectory of the suture needle based on the suture needle path and the matching degree of the positions on the suture needle path, and adjust the suture needle based on the actual trajectory and the planned navigation path.

[0008] Preferably, the determining the search range of the suture needle in the current image according to the predicted position and the position of the suture needle in the image acquired by the endoscope at the previous moment is specifically as follows:

[0009] Determine the feature point selection range according to the position of the suture needle in the image at the previous moment;

[0010] Select feature points from within the feature point selection range in the image of the previous moment, calculate the average value of the motion vectors of the feature points, and determine the estimated position of the suture needle according to the average value of the motion vectors;

[0011] Use the predicted position and the estimated position to determine the search range of the suture needle in the current image.

[0012] Preferably, the using the predicted position and the estimated position to determine the search range of the suture needle in the current image is specifically as follows:

[0013] Obtain the minimum bounding rectangle containing the predicted position and the estimated position, and add a safety margin based on the minimum bounding rectangle to obtain the search range of the suture needle in the current image.

[0014] Preferably, the determining the suture needle area within the search range and obtaining the matching degree is specifically as follows:

[0015] Use the template matching algorithm to slide the suture needle template within the search area of the current image. For each position within the search area, calculate the similarity score between the template and the image block covered by the template, and use the maximum value of all similarity scores among all suture needle templates as the matching degree.

[0016] Preferably, the obtaining the actual trajectory of the suture needle according to the suture needle path and the matching degree of the positions on the suture needle path is specifically as follows:

[0017] Calculate the weights of the suture needle path and the motion trajectory based on the matching degree;

[0018] Use the weights to weight the current position of the suture needle in the suture needle path and the motion trajectory to obtain the actual position of the suture needle at the current moment;

[0019] Add the actual position at the current moment to the actual trajectory of the previous moment to obtain the actual trajectory of the suture needle.

[0020] In a second aspect of the present invention, a surgical suture needle navigation system based on magnetic sensing and AI vision is provided. The system includes the following modules:

[0021] A magnetic sensing module, configured to obtain the movement trajectory of the suture needle by using a magnetic sensor integrated on the suture needle; obtain the spatial position of the suture needle at the current moment through the movement trajectory, and project the spatial position onto the endoscopic image at the current moment by using the external and internal parameters of the endoscope at the current moment to obtain the predicted position of the suture needle on the image at the current moment;

[0022] An AI vision module, configured to determine the search range of the suture needle in the current moment image according to the predicted position and the position of the suture needle in the image collected by the endoscope at the previous moment, determine the suture needle area within the search range and obtain the matching degree, and back-project the pixel coordinates of the suture needle area into the three-dimensional space to obtain the suture needle path;

[0023] A navigation module, configured to obtain the actual trajectory of the suture needle according to the suture needle path and the matching degree of the positions on the suture needle path, and adjust the suture needle based on the actual trajectory and the planned navigation path.

[0024] Preferably, the step of determining the search range of the suture needle in the current moment image according to the predicted position and the position of the suture needle in the image collected by the endoscope at the previous moment is specifically as follows:

[0025] Determine the feature point selection range according to the position of the suture needle in the image at the previous moment;

[0026] Select feature points from within the feature point selection range in the image at the previous moment, calculate the average value of the motion vectors of the feature points, and determine the estimated position of the suture needle according to the average value of the motion vectors;

[0027] Determine the search range of the suture needle in the current moment image by using the predicted position and the estimated position.

[0028] Preferably, the step of determining the search range of the suture needle in the current moment image by using the predicted position and the estimated position is specifically as follows:

[0029] Obtain the minimum bounding rectangle containing the predicted position and the estimated position, and add a safety margin on the basis of the minimum bounding rectangle to obtain the search range of the suture needle in the current moment image.

[0030] Preferably, the step of determining the suture needle area within the search range and obtaining the matching degree is specifically as follows:

[0031] Use the template matching algorithm to slide the suture needle template within the search area of the current moment image. For each position within the search area, calculate the similarity score between the template and the image patch covered by the template, and take the maximum value of all similarity scores among all suture needle templates as the matching degree.

[0032] Preferably, obtaining the actual trajectory of the suture needle according to the suture needle path and the matching degree of the positions on the suture needle path is specifically as follows:

[0033] Calculate the weights of the suture needle path and the motion trajectory based on the matching degree;

[0034] Use the weights to weight the current moment position of the suture needle in the suture needle path and the motion trajectory to obtain the actual position of the suture needle at the current moment;

[0035] Add the actual position at the current moment to the actual trajectory of the previous moment to obtain the actual trajectory of the suture needle.

[0036] In the third aspect of the present invention, a surgical suture needle navigation device based on magnetic sensing and AI vision is provided. The navigation device includes a suture needle integrated with a magnetic sensor, an endoscope, and a computing device. The computing device is used for:

[0037] Obtain the motion trajectory of the suture needle from the magnetic sensor integrated on the suture needle; obtain the spatial position of the suture needle at the current moment through the motion trajectory, and project the spatial position onto the endoscope image at the current moment by using the external parameters and internal parameters of the endoscope at the current moment to obtain the predicted position of the suture needle on the image at the current moment;

[0038] Determine the search range of the suture needle in the current moment image according to the predicted position and the position of the suture needle in the image collected by the endoscope at the previous moment, determine the suture needle area within the search range and obtain the matching degree, and back-project the pixel coordinates of the suture needle area into the three-dimensional space to obtain the suture needle path;

[0039] Obtain the actual trajectory of the suture needle according to the suture needle path and the matching degree of the positions on the suture needle path, and adjust the suture needle based on the actual trajectory and the planned navigation path.

[0040] The present invention simultaneously uses two methods of magnetic sensing and vision to navigate the surgical suture needle, and uses magnetic sensing to provide a preliminary 3D position estimate, providing prior information for AI vision to search for the suture needle in a 2D image, thereby improving the efficiency and robustness of visual recognition; conversely, the result of visual recognition can be used to correct or optimize the possible drift or error of magnetic sensing. Moreover, by narrowing the search range, limited computing resources can be more concentrated on performing more refined analysis and matching on the candidate area, thereby improving the accuracy of matching while ensuring efficiency. Description of the Drawings

[0041] Figure 1 It is a flowchart of the first embodiment;

[0042] Figure 2 It is a schematic diagram for mapping the suture needle at the current moment on the movement trajectory to the image;

[0043] Figure 3 It is a schematic diagram of the minimum bounding rectangle and the search area;

[0044] Figure 4 It is a schematic diagram of the movement trajectory and the suture needle path;

[0045] Figure 5 It is a schematic diagram of the actual trajectory;

[0046] Figure 6 It is a schematic structural diagram of the second embodiment. Detailed implementation manners

[0047] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner for easy understanding.

[0048] It can be understood that the "embodiments" mentioned throughout the specification mean that specific features, structures or characteristics related to the embodiments are included in at least one embodiment of this application. Therefore, the various embodiments mentioned throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It can be understood that in the various embodiments of this application, the magnitudes of the sequence numbers of the various processes do not mean the order of execution is prior or subsequent, and the execution order of the various processes should be determined according to their functions and internal logics, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0049] In the present invention, unless otherwise specified, the same or similar parts between the various embodiments can be referred to each other. In the various embodiments of the present invention, as well as in each implementation manner / implementation method / realization method in each embodiment, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments, as well as between each implementation manner / implementation method / realization method in each embodiment, are consistent and can be mutually referred to, and the technical features in different embodiments, as well as in each implementation manner / implementation method / realization method in each embodiment, can be combined to form new embodiments, implementation manners, implementation methods, or realization methods according to their internal logical relationships. The implementation manners of this application described below do not constitute a limitation to the protection scope of this application.

[0050] Figure 1 The first embodiment of the present invention is shown, Figure 1 which is a surgical suture needle navigation method based on magnetic sensing and AI vision, and includes the following steps:

[0051] S1. Use the magnetic sensor integrated on the suture needle to obtain the movement trajectory of the suture needle; obtain the spatial position of the suture needle at the current moment through the movement trajectory, and project the spatial position onto the endoscopic image at the current moment by using the external parameters and internal parameters of the endoscope at the current moment to obtain the predicted position of the suture needle on the image at the current moment;

[0052] An external magnetic field emitter is adopted to generate a known magnetic field in the surgical area, and one or more magnetic sensors such as coils are integrated on the suture needle. When the suture needle moves in the magnetic field, the sensor will sense the change of the magnetic field and generate corresponding signals. By analyzing the intensity and phase difference of these signals, the position and attitude of the suture needle in the three-dimensional space can be calculated. A position and attitude will be generated at each moment, and the movement trajectory is composed of a series of position and attitude data points that change with time. Find the data point corresponding to the current timestamp from the stored trajectory data to obtain the spatial position of the suture needle at the current moment. In one embodiment, the spatial position includes the position and attitude of the suture needle. The position described below can be the position of a specific point or a region. For example, a certain feature point on the suture needle is used as a key point, and the position refers to the position of this key point; the position can also be a region, specifically, the region where the suture needle is located; for the sake of simplicity of expression, the position is uniformly used for expression below.

[0053] In the video of the suture needle moving taken by the endoscope, the endoscope is also constantly moving, which leads to the continuous change of the external parameters of the endoscope. In one embodiment, the position of the endoscope in the world coordinate system is obtained through an additional tracking method, and the additional tracking method includes but is not limited to electromagnetic tracking, optical tracking, etc. In one embodiment, a magnetic sensor is integrated on the endoscope, that is, the electromagnetic tracking method is adopted for tracking. After obtaining the external parameters of the endoscope, the spatial position is projected onto the endoscopic image at the current moment by using the internal parameters and external parameters. Among them, the internal parameters are related to the properties of the endoscope itself and will not change during movement. In a more specific embodiment, obtain the three-dimensional position of the suture needle in the world coordinate system at the current moment, obtain the external parameters of the endoscope relative to the world coordinate system at the current moment, convert the world coordinates of the suture needle to the endoscope camera coordinate system, obtain the internal parameters of the endoscope, that is, the camera matrix, and project the three-dimensional points in the camera coordinate system onto the image coordinate system. In one embodiment, if the spatial position also includes the suture needle attitude information, then the above projection is performed on multiple points on it, and then the complete shape of the suture needle is represented on the image, such as Figure 2As shown. The predicted position is also the position of the suture needle obtained by projecting the spatial position onto the image at the current moment. However, electromagnetic measurement is easily affected by the surrounding environment, and the spatial position is not an accurate position. Moreover, due to distortion of the endoscope camera lens, etc., the predicted position is not an accurate position either.

[0054] S2. Determine the search range of the suture needle in the image at the current moment according to the predicted position and the position of the suture needle in the image collected by the endoscope at the previous moment. Determine the suture needle area within the search range and obtain the matching degree. Back-project the pixel coordinates of the suture needle area into the three-dimensional space to obtain the suture needle path.

[0055] The predicted position is the approximate position of the suture needle in the image at the current moment. Although it is not precise enough, it can give the possible position of the suture needle in the image at the current moment. In one embodiment, obtain the predicted position at the current moment and the position of the suture needle detected at the previous moment. Taking the position of the suture needle in the image at the previous moment as the center, obtain the radius according to the distance from the center to the predicted position. In one embodiment, increase the radius, for example, the radius becomes 1.5 times the original. Obtain a circle according to the center and the radius, and obtain the minimum circumscribed rectangle of the circle. Take the minimum circumscribed rectangle as the search range.

[0056] In yet another embodiment, a feature point selection range is determined based on the position of the suture needle in the image at the previous moment; feature points are selected from within the feature point selection range in the image at the previous moment, the average value of the motion vectors of the feature points is calculated, and the estimated position of the suture needle is determined based on the average value of the motion vectors; the search range of the suture needle in the image at the current moment is determined using the predicted position and the estimated position. In the endoscopic image at the previous time point, based on the position of the suture needle that has been found, a region is delimited, and feature points that can be used for tracking are searched within this region, which can improve the efficiency and relevance of feature point selection and avoid searching for feature points in irrelevant regions of the image. Within the feature point selection range in the image at the previous moment, a corner detection algorithm, such as Harris corner detection, Shi-Tomasi corner detection, etc., is used to select some prominent corners as feature points. Using an optical flow estimation algorithm, such as the Lucas-Kanade optical flow method, these feature points are tracked to the image at the current moment, and the optical flow method calculates a motion vector for each feature point, representing the displacement of the point between two frames. The average value of the motion vectors of all successfully tracked feature points is calculated, and the average motion vector approximately represents the overall motion trend of the suture needle in the image. The position of the suture needle at the previous moment, such as the center point or the position of the entire suture needle in the image, is added to the average motion vector to obtain the estimated position of the suture needle at the current moment. The predicted position comes from the magnetic sensor in the suture needle, and the estimated position comes from the estimation of the suture needle in the image at the previous moment. In the image at the current moment, the suture needle may appear around both of these positions, and only the search range needs to be determined based on these two positions, which not only reduces the amount of calculation but also helps to reduce the possibility of misidentification. In yet another embodiment, the determining the search range of the suture needle in the image at the current moment using the predicted position and the estimated position specifically includes: obtaining the minimum bounding rectangle containing the predicted position and the estimated position, and adding a safety margin, such as increasing it to 1.5 times the minimum bounding rectangle, to obtain the search range of the suture needle in the image at the current moment. Figure 3 The minimum bounding rectangle and the search area are shown. The predicted position obtained by projecting the magnetic sensor and the estimated position obtained by feature tracking are acquired, and the minimum and maximum values of the abscissas of these two points, as well as the minimum and maximum values of the ordinates, are found to obtain the minimum bounding rectangle containing these two points. A certain number of pixel values are added in the four directions of this rectangle. For example, Δx is added in the horizontal direction and Δy is added in the vertical direction, etc., which can prevent other areas of the suture needle from being outside the search area.

[0057] In one embodiment, the determining the suture needle area within the search range and obtaining the matching degree specifically includes:

[0058] Use the template matching algorithm to slide the suture needle template within the search area of the current moment image. For each position within the search area, calculate the similarity score between the template and the image patch covered by the template, and take the maximum value of all similarity scores among all suture needle templates as the matching degree.

[0059] After obtaining the search area in the image of the current moment, crop the suture needle area that has been successfully recognized at the previous moment as the template, or use a typical appearance template of the suture needle prepared in advance. Slide the template within the search range of the current moment image, and calculate the similarity between the template and the corresponding image area within the search window at each position. The calculation methods of the similarity include but are not limited to normalized cross-correlation, sum of squared differences matching, etc. Find the position with the highest similarity within the search range, and the corresponding image area at this position is the suture needle area at the current moment. The calculated similarity is directly used as a measure of the matching degree.

[0060] Optionally, to determine the suture needle area within the search range and obtain the matching degree, a target recognition algorithm such as the YOLO series can also be used, and the confidence score is used as the matching degree.

[0061] In S1, the external parameters and internal parameters of the endoscopic camera have been obtained. Use the external parameters and internal parameters to back-project the pixel coordinates of the suture needle area into the three-dimensional space to obtain the suture needle path. Specifically, use the inverse matrix of the internal parameter matrix to convert the pixel coordinates to the coordinates in the normalized image coordinate system, convert the normalized coordinates to the camera coordinate system, and use the external parameter matrix to convert the three-dimensional points in the camera coordinate system to the coordinates in the world coordinate system. The coordinates at all moments constitute the suture needle path. Figure 4 The schematic diagram of the motion trajectory and the suture needle path is shown.

[0062] S3. Obtain the actual suture needle trajectory based on the suture needle path and the matching degree of the positions on the suture needle path, and adjust the suture needle based on the actual trajectory and the planned navigation path.

[0063] When there is line of sight occlusion, etc., the matching degree is low. The matching degree reflects the credibility of obtaining the suture needle position through vision. In one embodiment, the method of obtaining the actual suture needle trajectory based on the suture needle path and the matching degree of the positions on the suture needle path is specifically as follows:

[0064] Calculate the weights of the suture needle path and the motion trajectory based on the matching degree;

[0065] Use the weights to weight the current moment position of the suture needle in the suture needle path and the motion trajectory to obtain the actual position of the suture needle at the current moment;

[0066] Add the actual position at the current moment to the actual trajectory of the previous moment to obtain the actual suture needle trajectory.

[0067] Specifically, for points at the same moment on the suture path and the motion trajectory, the matching degree is used as the weight of the suture path. The weight of the suture path is calculated based on the matching degree, for example, through function transformation, etc., and 1 - the matching degree is used as the weight of the motion trajectory. After obtaining the weights, the weights are used to weight the positions at the current moment in the suture path and the motion trajectory to obtain the actual position at the current moment. Then, the actual position at the current moment is added to the actual trajectory formed before the current moment to obtain the actual suture trajectory, as Figure 5 shown Figure 5 In the figure, the red line is the obtained actual trajectory. The suture path comes from the image and the motion trajectory comes from the magnetic sensor. The image information can provide an intuitive visual position, but may be affected by factors such as occlusion and light changes, resulting in a low matching degree. The magnetic sensor can provide continuous motion information, but there may be cumulative errors or it may be sensitive to metal objects. By weighting, more reliance can be placed on the dominant information according to the reliability at the current moment, thereby improving the accuracy of the overall position estimation. For example, when the image matching degree is high, more trust is placed on the path provided by the image; when the matching degree is low, more trust is placed on the motion trajectory provided by the magnetic sensor. The actual suture trajectory at the current moment is compared with the planned navigation path in real time, and the calculated deviation is analyzed to determine whether the deviation is within an acceptable range and the trend of the deviation change. The planned navigation path and the actual trajectory are superimposed and displayed on the endoscopic image or a separate display, for example, using different colors or line styles to distinguish the two, and the deviation area is highlighted; or arrows, indicator lines or other marks are superimposed on the image to indicate how the surgeon should adjust the direction and position of the suture to return to or stay on the planned path. For example, the arrow may point to the direction that needs to be moved, and the length of the arrow may represent the distance that needs to be moved.

[0068] Figure 6 shows the structural diagram of the second embodiment of the present invention, as Figure 6 shown in the surgical suture navigation system based on magnetic sensing and AI vision, the system includes the following modules:

[0069] A magnetic sensing module 201, configured to obtain the motion trajectory of the suture by using a magnetic sensor integrated on the suture; obtain the spatial position of the suture at the current moment through the motion trajectory, and project the spatial position onto the endoscopic image at the current moment by using the external parameters and internal parameters of the endoscopic at the current moment to obtain the predicted position of the suture on the image at the current moment;

[0070] An AI vision module 202, configured to determine the search range of the suture in the image at the current moment according to the predicted position and the position of the suture in the image collected by the endoscope at the previous moment, determine the suture area within the search range and obtain the matching degree, and back-project the pixel coordinates of the suture area into the three-dimensional space to obtain the suture path;

[0071] The navigation module 203 is configured to obtain the actual suture trajectory based on the suture path and the matching degree of positions on the suture path, and adjust the suture based on the actual trajectory and the planned navigation path.

[0072] In an optional embodiment, determining the search range of the suture in the current moment image according to the predicted position and the suture position in the image acquired by the endoscope at the previous moment specifically includes:

[0073] Determining the feature point selection range according to the suture position in the image at the previous moment;

[0074] Selecting feature points from within the feature point selection range in the image of the previous moment, calculating the average value of the motion vectors of the feature points, and determining the predicted position of the suture according to the average value of the motion vectors;

[0075] Determining the search range of the suture in the current moment image by using the predicted position and the predicted position.

[0076] In an optional embodiment, determining the search range of the suture in the current moment image by using the predicted position and the predicted position specifically includes:

[0077] Obtaining the minimum bounding rectangle containing the predicted position and the predicted position, and adding a safety margin to the minimum bounding rectangle to obtain the search range of the suture in the current moment image.

[0078] In an optional embodiment, determining the suture area within the search range and obtaining the matching degree specifically includes:

[0079] Using the template matching algorithm to slide the suture template within the search area of the current moment image. For each position within the search area, calculating the similarity score between the template and the image block covered by the template, and taking the maximum value of all similarity scores among all suture templates as the matching degree.

[0080] In an optional embodiment, obtaining the actual suture trajectory according to the suture path and the matching degree of positions on the suture path specifically includes:

[0081] Calculating the weights of the suture path and the motion trajectory based on the matching degree;

[0082] Using the weights to weight the current moment position of the suture in the suture path and the motion trajectory to obtain the actual position of the suture at the current moment;

[0083] Adding the actual position at the current moment to the actual trajectory of the previous moment to obtain the actual suture trajectory.

[0084] In the third embodiment of the present invention, a surgical suture needle navigation device based on magnetic sensing and AI vision is provided. The navigation device includes a suture needle integrated with a magnetic sensor, an endoscope, and a computing device. The computing device is configured to:

[0085] Obtain the movement trajectory of the suture needle from the magnetic sensor integrated on the suture needle; obtain the spatial position of the suture needle at the current moment through the movement trajectory, and project the spatial position onto the endoscope image at the current moment by using the external and internal parameters of the endoscope at the current moment to obtain the predicted position of the suture needle on the image at the current moment;

[0086] Determine the search range of the suture needle in the current moment image according to the predicted position and the position of the suture needle in the image of the previous moment collected by the endoscope, determine the suture needle area within the search range and obtain the matching degree, back-project the pixel coordinates of the suture needle area into the three-dimensional space to obtain the suture needle path;

[0087] Obtain the actual trajectory of the suture needle according to the suture needle path and the matching degree of the positions on the suture needle path, and adjust the suture needle based on the actual trajectory and the planned navigation path.

[0088] The above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices, as Figure 6 shown. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center including one or more available media integrated. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0089] The steps of the methods or algorithms described in the embodiments of this application may be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units may be stored in a RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium may be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Optionally, the storage medium may also be integrated into the processor. The processor and the storage medium may be provided in an ASIC.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in a process Figure 1 one process or multiple processes and / or blocks Figure 1 or steps for the functions specified in multiple blocks.

[0091] Although this application has been described in conjunction with specific features and their embodiments, it is obvious that various modifications and combinations can be made without departing from the spirit and scope of this application. Accordingly, this specification and the drawings are merely exemplary illustrations of the application defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Obviously, those skilled in the art can make various changes and modifications to this application without departing from the scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these changes and modifications.

Claims

1. A surgical needle navigation method based on magnetic sensing and AI vision, characterized in that: The method comprises the following steps: The motion trajectory of the suture needle is obtained by using the magnetic sensor integrated on the suture needle; the spatial position of the suture needle at the current moment is obtained through the motion trajectory, and the spatial position is projected onto the endoscopic image at the current moment using the external and internal parameters of the endoscope at the current moment to obtain the predicted position of the suture needle on the image at the current moment; Determine the search range of the suture needle in the image at the current moment according to the predicted position and the position of the suture needle in the image at the previous moment collected by the endoscope, determine the suture needle area within the search range and obtain the matching degree, back-project the pixel coordinates of the suture needle area into the three-dimensional space and obtain the suture needle path; The actual trajectory of the suture needle is obtained according to the matching degree of the suture needle path and the position on the suture needle path, and the suture needle is adjusted based on the actual trajectory and the planned navigation path.

2. The method according to claim 1, characterized in that The search range of the suture needle in the image at the current moment is determined according to the predicted position and the position of the suture needle in the image at the previous moment acquired by the endoscope, specifically: Determine the feature point selection range according to the position of the sewing needle in the image at the previous moment; Select feature points from the feature point selection range in the image at the previous moment, calculate the average value of motion vectors of the feature points, and determine the estimated position of the sewing needle according to the average value of the motion vectors; The predicted position and the estimated position are used to determine the search range of the suture needle in the image at the current moment.

3. The method according to claim 2, characterized in that The method of using the predicted position and the estimated position to determine the search range of the sewing needle in the image at the current moment is specifically as follows: A minimum enclosing rectangle including the predicted position and the estimated position is obtained, and a safety margin is added to the minimum enclosing rectangle to obtain a search range of the suture needle in the image at the current moment.

4. The method according to claim 1, characterized in that Determining the sewing needle area within the search range and obtaining the matching degree is specifically as follows: The template matching algorithm is used to slide the sewing needle template in the search area of ​​the current image. For each position in the search area, the similarity score between the template and the image block covered by the template is calculated, and the maximum value of all similarity scores among all sewing needle templates is taken as the matching degree.

5. The method according to claim 1, characterized in that The actual trajectory of the sewing needle is obtained according to the matching degree of the sewing needle path and the position on the sewing needle path, specifically: Calculating the weights of the needle path and the motion trajectory based on the matching degree; The weight is used to weight the current position of the suture needle in the suture needle path and the motion trajectory to obtain the actual position of the suture needle at the current moment; The actual position at the current moment is added to the actual trajectory at the previous moment to obtain the actual trajectory of the sewing needle.

6. A surgical needle navigation system based on magnetic sensing and AI vision, characterized in that: The system includes the following modules: A magnetic sensing module is used to obtain the motion trajectory of the suture needle using a magnetic sensor integrated on the suture needle; obtain the spatial position of the suture needle at the current moment through the motion trajectory, and project the spatial position onto the endoscopic image at the current moment using the external and internal parameters of the endoscope at the current moment to obtain the predicted position of the suture needle on the image at the current moment; An AI vision module is used to determine a search range for a suture needle in an image at a current moment according to the predicted position and the position of the suture needle in an image at a previous moment acquired by an endoscope, determine a suture needle area within the search range and obtain a matching degree, and back-project pixel coordinates of the suture needle area into a three-dimensional space to obtain a suture needle path; The navigation module is used to obtain the actual trajectory of the suture needle according to the matching degree of the suture needle path and the position on the suture needle path, and adjust the suture needle based on the actual trajectory and the planned navigation path.

7. The system according to claim 6, characterized in that The search range of the suture needle in the image at the current moment is determined according to the predicted position and the position of the suture needle in the image at the previous moment acquired by the endoscope, specifically: Determine the feature point selection range according to the position of the sewing needle in the image at the previous moment; Select feature points from the feature point selection range in the image at the previous moment, calculate the average value of motion vectors of the feature points, and determine the estimated position of the sewing needle according to the average value of the motion vectors; The predicted position and the estimated position are used to determine the search range of the suture needle in the image at the current moment.

8. The system according to claim 6, characterized in that Determining the sewing needle area within the search range and obtaining the matching degree is specifically as follows: The template matching algorithm is used to slide the sewing needle template in the search area of ​​the current image. For each position in the search area, the similarity score between the template and the image block covered by the template is calculated, and the maximum value of all similarity scores among all sewing needle templates is taken as the matching degree.

9. The system according to claim 6, characterized in that The actual trajectory of the sewing needle is obtained according to the matching degree of the sewing needle path and the position on the sewing needle path, specifically: Calculating the weights of the needle path and the motion trajectory based on the matching degree; The weight is used to weight the current position of the suture needle in the suture needle path and the motion trajectory to obtain the actual position of the suture needle at the current moment; The actual position at the current moment is added to the actual trajectory at the previous moment to obtain the actual trajectory of the sewing needle.

10. A surgical needle navigation device based on magnetic sensing and AI vision, the navigation device comprising a needle integrated with a magnetic sensor, an endoscope, and a computing device, characterized in that: The computing device is used to: The motion trajectory of the suture needle is obtained from the magnetic sensor integrated on the suture needle; the spatial position of the suture needle at the current moment is obtained through the motion trajectory, and the spatial position is projected onto the endoscopic image at the current moment using the external and internal parameters of the endoscope at the current moment to obtain the predicted position of the suture needle on the image at the current moment; Determine the search range of the suture needle in the image at the current moment according to the predicted position and the position of the suture needle in the image at the previous moment collected by the endoscope, determine the suture needle area within the search range and obtain the matching degree, back-project the pixel coordinates of the suture needle area into the three-dimensional space and obtain the suture needle path; The actual trajectory of the suture needle is obtained according to the matching degree of the suture needle path and the position on the suture needle path, and the suture needle is adjusted based on the actual trajectory and the planned navigation path.

Images