Surgical instrument pose analysis method and system

Abstract

The invention provides a surgical instrument pose analysis method and system, and relates to the field of pose analysis, and the method comprises the steps: selecting a plurality of feature points on a surgical instrument as a plurality of infrared reflective mark points, and carrying out the coding processing of the infrared reflective mark points; carrying out synchronous processing on the infrared camera and the color camera and carrying out space calibration; capturing information of the surgical instrument and generating an infrared image and a color image; and constructing a target prediction network and a loss function based on the infrared image and the color image, and solving the pose by adopting a perspective n-point PnP algorithm based on a random sample consensus RANSAC framework to obtain the refined pose of the surgical instrument. According to the method, a strategy combining global shape information and local mark point features is matched with a perspective n-point PnP algorithm based on a random sample consensus RANSAC framework, the instrument shape mask information is used as additional geometric constraint and verification information to correct errors, and high-precision and high-robustness surgical instrument real-time pose tracking is achieved.

Description

Technical Field

[0001] This application relates to the technical field of pose analysis, and specifically relates to a method and system for analyzing the pose of surgical instruments. Background Art

[0002] Currently, when attempting to accurately track the position and orientation of surgical instruments through visual information, some severe practical difficulties are faced. In the dynamic and complex real operating room environment, the stability and accuracy of traditional image analysis methods are often significantly affected. The constantly changing lighting conditions, chaotic background, and the diversity of the shapes of the instruments themselves all pose challenges to the reliability of these methods.

[0003] In the related art, although the recognition technology based on deep learning has brought significant progress, key challenges still exist in specific application scenarios. A core problem lies in how to accurately identify both the relatively large overall surgical instrument and the extremely small positioning marker points attached to it within a unified and efficient framework. These two types of targets have extremely different scales and visual features, resulting in many existing models being difficult to balance the detection accuracy and efficiency of both, often requiring complex system designs or making compromises in performance.

[0004] When performing precise pose calculations using these identified marker points, the existing mainstream methods rely heavily on the information of a few discrete points. This approach lacks the utilization of the overall geometric shape of the instrument itself, resulting in a sharp decline in the accuracy and stability of pose estimation if some of the marker points are occluded, reflect unclearly, or have detection errors. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, this application provides a method and system for analyzing the pose of surgical instruments, which solves the problem that the accuracy and stability of pose estimation of surgical instruments are not ideal.

[0006] To achieve the above objectives, this application is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a surgical instrument pose analysis method, which includes: selecting multiple feature points on the surgical instrument as multiple infrared reflective marker points, and performing encoding processing on the infrared reflective marker points based on a preset encoding rule; synchronizing an infrared camera and a color camera in a preset image acquisition unit so that both can capture the scene at the same time point and perform spatial calibration; respectively capturing information of the surgical instrument through the infrared camera and the color camera to generate an infrared image and a color image; wherein, the infrared image includes the infrared reflective marker points attached to the surgical instrument, and the color image includes instrument shape mask information; constructing a target prediction network and a loss function based on the infrared image and the color image, where the target prediction network is used for infrared optical marker point coordinate detection and instrument mask prediction to identify the overall shape of the instrument and the attached tiny positioning marker points, and fuse global and local information to achieve real-time three-dimensional pose tracking of the surgical instrument in a complex environment; based on the target prediction network, using a perspective n-point PnP algorithm based on the random sample consensus (RANSAC) framework, taking the instrument shape mask information as additional geometric constraints and verification information to correct errors, generating and verifying candidate poses, and solving for the pose to obtain the refined pose of the surgical instrument.

[0008] In a second aspect, an embodiment of the present application provides a surgical instrument pose analysis system, which includes an encoding module, a camera processing module, an image generation module, a network construction module, and a pose solving module; specifically, the encoding module is used to select multiple feature points on the surgical instrument as multiple infrared reflective marker points and perform encoding processing on the infrared reflective marker points based on a preset encoding rule; the camera processing module is used to synchronize an infrared camera and a color camera in a preset image acquisition unit so that both can capture the scene at the same time point and perform spatial calibration; the image generation module is used to respectively capture information of the surgical instrument through the infrared camera and the color camera to generate an infrared image and a color image; wherein, the infrared image includes the infrared reflective marker points attached to the surgical instrument, and the color image includes instrument shape mask information; the network construction module is used to construct a target prediction network and a loss function based on the infrared image and the color image, where the target prediction network is used for infrared optical marker point coordinate detection and instrument mask prediction to identify the overall shape of the instrument and the attached tiny positioning marker points, and fuse global and local information to achieve real-time three-dimensional pose tracking of the surgical instrument in a complex environment; the pose solving module is used to, based on the target prediction network, use a perspective n-point PnP algorithm based on the random sample consensus (RANSAC) framework, take the instrument shape mask information as additional geometric constraints and verification information to correct errors, generate and verify candidate poses, and solve for the pose to obtain the refined pose of the surgical instrument.

[0009] In a third aspect, an embodiment of the present application provides an electronic device, which includes: a processor, a memory, and a program stored on the memory and executable on the processor. When the program is executed by the processor, it implements a surgical instrument pose analysis method in the foregoing first aspect.

[0010] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instructions are stored. When the program or instructions are executed by a processor, they implement a surgical instrument pose analysis method in the foregoing first aspect.

[0011] The present application provides a surgical instrument pose analysis method and system. Compared with the prior art, it has the following beneficial effects:

[0012] The present application collects infrared images and color images through a dual-camera configuration of an infrared camera and a color camera. The infrared images include infrared reflective marker points attached to the surgical instrument, and the color images include instrument shape mask information to represent the overall shape of the surgical instrument. By designing a target prediction network based on infrared images and color images for infrared optical marker point coordinate detection and instrument mask prediction, it can synchronously and accurately identify the overall shape of the surgical instrument and the key infrared positioning marker points. This strategy of combining global shape information with local marker point features, combined with the Perspective-n-Point (PnP) algorithm based on the Random Sample Consensus (RANSAC) framework, uses the instrument shape mask information as additional geometric constraints and verification information to correct errors, which can improve the accuracy and stability of pose tracking under various interference factors including occlusion, illumination changes, and detection errors, obtain a reliable refined pose of the surgical instrument, and can achieve high-precision and high-robustness real-time pose tracking of the surgical instrument. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0014] Figure 1 is a schematic flowchart of a surgical instrument pose analysis method provided by an embodiment of the present application;

[0015] Figure 2 is a schematic structural diagram of a target prediction network provided by an embodiment of the present application;

[0016] Figure 3 is a schematic structural diagram of a small decoupled head provided by an embodiment of the present application;

[0017] Figure 4It is a schematic structural diagram of a large decoupling head provided by an embodiment of the present application;

[0018] Figure 5 It is a schematic structural diagram of a surgical instrument pose analysis system provided by an embodiment of the present application;

[0019] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or sequence between these entities or operations. Moreover, the terms "include", "comprise", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus including a series of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "including a..." does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the element.

[0022] By providing a surgical instrument pose analysis method and system, the embodiments of the present application solve the problem that the accuracy and stability of surgical instrument pose estimation are not ideal.

[0023] The technical solutions in the embodiments of the present application for solving the above technical problems are generally as follows:

[0024] Currently, when attempting to accurately track the position and orientation of surgical instruments through visual information, some severe practical difficulties are faced. In a dynamic and complex real operating room environment, the stability and accuracy of traditional image analysis methods are often significantly affected. Changing lighting conditions, chaotic backgrounds, and the diversity of the instrument's own morphology all pose challenges to the reliability of these methods.

[0025] In the related art, although the recognition technology based on deep learning has brought remarkable progress, it still faces key challenges in specific application scenarios. A core problem lies in how to accurately identify both the overall surgical instrument with a relatively large size and the extremely small positioning marker points attached to it within a unified and efficient framework. These two types of targets have significant differences in scale and visual features, resulting in many existing models being difficult to balance the detection accuracy and efficiency of both, often requiring complex system designs or making compromises in performance.

[0026] When using these identified marker points for precise pose calculation, the existing mainstream methods rely heavily on the information of a few discrete points. This approach lacks the utilization of the overall geometric shape of the instrument itself, leading to a sharp decline in the accuracy and stability of pose estimation if the marker points are partially occluded, reflect unclearly, or have detection errors.

[0027] There is generally a lack of an effective mechanism to fuse the global morphological information of the instrument to assist in judging, verifying, or correcting the pose deduced solely from local point information, thus limiting its reliability in the real surgical environment. In addition, some alternative technical routes may also need to address additional challenges such as insufficient image registration accuracy, non-unique solution results (singularity problems) in specific cases, and computational efficiency that is difficult to meet the real-time surgical requirements. These factors together limit the potential of existing visual tracking technologies in surgical assistance applications that require high precision and high reliability.

[0028] To better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the accompanying drawings of the specification and specific implementation manners.

[0029] First, a method for analyzing the pose of a surgical instrument provided by an embodiment of the present application will be introduced below.

[0030] A schematic flowchart of a method for analyzing the pose of a surgical instrument provided by an embodiment of the present application is as Figure 1 shown, and this method for analyzing the pose of a surgical instrument may include the following steps S110 - S150.

[0031] S110. Select multiple feature points on the surgical instrument as multiple infrared reflective marker points, and perform encoding processing on the infrared reflective marker points based on a preset encoding rule.

[0032] S120. Synchronize the infrared camera and the color camera in the preset image acquisition unit so that both can capture the scene at the same time point and perform spatial calibration.

[0033] S130. Capture the information of the surgical instrument through an infrared camera and a color camera respectively to generate an infrared image and a color image. Among them, the infrared image includes infrared reflective marker points attached to the surgical instrument, and the color image includes instrument shape mask information.

[0034] S140. Construct a target prediction network and a loss function based on the infrared image and the color image. The target prediction network is used for infrared optical marker point coordinate detection and instrument mask prediction to identify the overall shape of the instrument and the attached tiny positioning marker points, and fuse the global and local information to achieve real-time three-dimensional pose tracking of the surgical instrument in a complex environment. Among them, the tiny positioning marker points correspond to the aforementioned infrared optical marker points.

[0035] S150. Based on the target prediction network, adopt the Perspective-n-Point (PnP) algorithm based on the Random Sample Consensus (RANSAC) framework, use the instrument shape mask information as additional geometric constraints and verification information to correct errors, generate and verify candidate poses, and solve the pose to obtain the refined pose of the surgical instrument.

[0036] The above is the specific implementation manner of a surgical instrument pose analysis method provided by this application. It can be understood that this application collects an infrared image and a color image through a dual-camera configuration of an infrared camera and a color camera. The infrared image includes infrared reflective marker points attached to the surgical instrument, and the color image includes instrument shape mask information to characterize the overall shape of the surgical instrument. By designing a target prediction network based on the infrared image and the color image for infrared optical marker point coordinate detection and instrument mask prediction, it can synchronously and accurately identify the overall shape of the surgical instrument and the key infrared positioning marker points. This strategy of combining global shape information with local marker point features, combined with the Perspective-n-Point (PnP) algorithm based on the Random Sample Consensus (RANSAC) framework, uses the instrument shape mask information as additional geometric constraints and verification information to correct errors.

[0037] Based on this, this application designs an innovative multi-task learning network architecture that can efficiently and synchronously and accurately identify two types of targets with huge scale differences within a unified framework: the overall shape of the surgical instrument (outputting an accurate pixel-level segmentation mask) and tiny infrared positioning marker points (capturing fine local features). Compared with traditional image processing methods or deep learning models that only focus on a single task, the present invention has achieved remarkable results in ensuring the accuracy and processing efficiency of the detection of these two types of targets at the same time. This application can improve the accuracy and stability of pose tracking under various interference factors including occlusion, illumination change, and detection error, obtain a reliable refined pose of the surgical instrument, and can achieve high-precision and high-robustness real-time pose tracking of the surgical instrument.

[0038] In one example, the spatial calibration includes the internal calibration of the infrared camera and the color camera and the external calibration between the two, so as to determine the precise rotation and translation relationship of one with respect to the other, and associate the information from both to the same three-dimensional space or a common reference coordinate system.

[0039] It should be noted that the hardware requirements of this application include an image acquisition unit and an infrared light marker point coding design. The image acquisition unit adopts a dual-camera configuration, and the dual-camera configuration includes an infrared camera and a color camera; the marker point information mainly comes from the infrared camera, and the instrument shape (mask) information mainly comes from the color camera; among them, the infrared camera is used to capture the infrared reflective marker points attached to the surgical instrument; through infrared illumination and filtering, an image with clear marker points and minimal background interference can be generated, providing the two-dimensional coordinates of the marker points in the infrared camera image coordinate system; these two-dimensional coordinates are the core data for performing the initial PnP pose calculation and subsequent point-based geometric constraints; the color camera is responsible for capturing the appearance and shape of the surgical instrument under visible light, obtaining a color image containing rich texture and contour information to provide image data for generating the instrument segmentation mask; the network relied on in this application needs to use the color image to accurately identify and segment the pixel-level contour of the surgical instrument, and this mask is the key for subsequent pose verification and refinement using shape information.

[0040] It should also be noted that the PnP (Perspective-n-Point) pose refers to solving the rotation matrix and translation vector of the camera coordinate system relative to the world coordinate system through the known three-dimensional space point coordinates of n points and their projection coordinates in the two-dimensional image, so as to determine the position and pose of the camera.

[0041] In some embodiments, the number of infrared reflective marker points is five and forms a set A0, and the elements in the set A0 are the coding results of the instrument marker points:

[0042] A0 = {A1, A2, A3, A4, A5};

[0043] The preset coding rules corresponding to the five infrared reflective marker points during the coding process satisfy: point A1 is located at a predefined reference position on the instrument; points A2, A3 are co-distributed with point A1, and points A1, A2, and A3 form a non-special triangle; the distance d12 from point A1 to point A2, the distance d13 from point A1 to point A3, and the distance d23 from point A2 to point A3 are all not equal to avoid an isosceles or equilateral structure; the position of point A4 relative to points A1, A2, and A3 is set so that the four-point combination formed by points A1, A2, A3, and A4 does not have symmetry in space;

[0044] In addition, the spatial position of point A5 significantly deviates from the reference plane approximately formed by the remaining four points, and A5 has an obvious non-zero displacement in the direction perpendicular to the corresponding reference plane. The distances D 14 , D 24 , D 34 from point A4 to point A1, point A2, and point A3, together with the distances between other points, constitute a unique geometric shape. It should be noted that the auxiliary positioning point coding design of the solution of this application has universality and can adapt to most single surgical instruments on the market.

[0045] In some embodiments, please refer to Figure 2 , the target prediction network includes: a Backbone backbone network, a Neck intermediate layer, and a Head output layer; among them, the Backbone backbone network includes a CBS module, a Res2Net module, and a first SPP module; the CBS module includes a convolutional layer CNN, a batch normalization layer BN, and an activation function layer SiLU.

[0046] In the Res2Net module, the input feature map first passes through a 1×1 convolution, and then the feature map will be divided into s subsets, so that the number of channels of each subset of the same size is reduced to 1 / s; each independent feature subset will be convolved with a corresponding 3×3 convolution kernel, denoted as f i , and the subset is denoted as x i , i ∈ {1, 2,..., s}; the feature subset x i is added to the output of f i-1 , and then input into f i , and the output is denoted as o i ; among them, the feature map includes an infrared image and a color image; the processing process of the Res2Net module satisfies the expression:

[0047]

[0048] The first SPP module includes multiple parallel max-pooling operations. The pooling kernel size and stride will be adaptively calculated according to the size of the input feature map. The feature vectors output by all parallel max-pooling layers are concatenated along the feature dimension to form a final fixed-length output vector.

[0049] In the embodiments of this application, it can be understood that in order to achieve an excellent balance between computing resources and accuracy improvement, this application designs a bottom-up Backbone backbone network for hierarchical feature extraction. This backbone network is formed by connecting the CBS module and the Res2 Block module in series and splicing them 3 times, and then connecting the first SPP module, which is the first spatial pyramid pooling module.

[0050] In some embodiments, please refer to Figure 2, the Neck middle layer is used for:

[0051] S210. Receive a fixed-length output vector from the Backbone backbone network, and obtain three temporary outputs after a series of upsampling and downsampling operations.

[0052] S220. Conduct the three temporary outputs to a preset second SPP module and obtain three feature maps of different scales. The sizes of the three feature maps are 13×13, 26×26, and 52×52 respectively.

[0053] S230. Perform another upsampling operation on the feature map with a size of 13×13 to bring the features back to the low-level network, so as to obtain a large number of gradients and semantic representations. It should be noted that the feature map with a size of 13×13 contains rich semantic information.

[0054] S240. Through the preset neck module DFPN, splice the output of the second SPP module and the feature map before the prediction network layer again to obtain three feature outputs of different scales. Among them, the number of channels of the three feature outputs at different scales are 1024, 512, and 256 respectively. The resolutions of the three increase in turn, the receptive fields decrease in turn, and they are respectively connected to the preset large decoupling head, medium decoupling head, and small decoupling head in the Head output layer.

[0055] In the embodiments of the present application, it can be understood that when the number of network layers deepens, the problem of information loss will become very serious, and it will even have an adverse impact on the feature information decoding process. To alleviate this situation, the DFPN designed in this solution splices the output of the second SPP module (SPP network layer) and the feature map before the prediction network layer again.

[0056] It should be noted that a learnable weight is assigned to each input feature map of each layer of the Backbone backbone network. For the Neck middle layer, a neck module DFPN is designed in the present application. The neck module DFPN can fully retain the global information of the surgical instrument and the local feature information of the infrared reflection point in the backbone network, greatly improving the instrument positioning accuracy. The core idea of the decoupling head designed in the solution of the present application is to perform pixel-level prediction and classification by introducing an additional branch network. The target prediction network proposed in the present application based on infrared images and color images can realize local detection of infrared reflection markers and multi-task detection of the whole surgical instrument.

[0057] In some embodiments, please refer to Figure 2 , Figure 3 and Figure 4 , the Head output layer is used for:

[0058] S310. Incorporate the large decoupling head, medium decoupling head, and small decoupling head into the overall framework of the target prediction network, separate the classification and regression tasks, perform dimensionality reduction operations through 1×1 convolution blocks, and implement the classification and regression tasks by two network branches respectively.

[0059] S320. Output the prediction results through 1×1 convolution operations. The network branch of the classification part obtains the categories to which different objects in the image belong, and the regression branch obtains the target position and confidence score through two parallel sub-branches.

[0060] S330. The shared features are input into the mask path after dimensionality reduction by the initial 1x1 convolution, processed by the CBL and upsampling layers, and a pixel-level mask prediction is output by a 1x1 convolution layer, generating a single-channel mask representation with twice the resolution of the original feature map for each detection instance.

[0061] In the embodiments of the present application, the large decoupling head and the small decoupling head capture the instrument shape mask information and the infrared reflective marker points of small targets respectively. Please refer to Figure 2 , the Head output layer includes Decoupled Head-Large, Decoupled Head-Medium, and Decoupled Head-Small; Decoupled Head-Large, Decoupled Head-Medium, and Decoupled Head-Small are the large decoupling head, medium decoupling head, and small decoupling head respectively. For the Decoupled Head-Large branch, in addition to the separation of the original classification and regression tasks, the present application adds a parallel mask prediction branch to each decoupling head.

[0062] In one example, the loss function is used for the structural optimization of the target prediction network, and the overall loss of the target prediction network satisfies the expression:

[0063] Loss total =Loss reg +Loss obj +Loss cls +Loss mask

[0064] Wherein, Loss total is the overall loss, Loss reg is the candidate box regression loss function, the confidence loss Loss obj is the confidence loss, Loss cls is the classification loss function, Loss mask is the mask loss function;

[0065] Loss reg satisfies the expression:

[0066]

[0067] IoU(A,V) represents the intersection-over-union ratio between the predicted value and the true value, A represents the predicted bounding box, and B is the corresponding true value; ρ 2 (A center ,B center ) represents the Euclidean distance between the predicted value and the true value, A center and B center Denote the predicted value and the true value respectively, c is the diagonal length of the minimum bounding box, βv is the penalty term; β represents the trade-off index, v is used to measure the uniformity of the aspect ratio; w and h are the width and height of the predicted bounding box, and the corresponding w gt and h gt is the width and height of the true bounding box;

[0068] Confidence loss Loss obj And classification loss function Loss cls This is achieved through binary cross entropy loss with logistic regression, and the calculation of the two satisfies the expression:

[0069]

[0070] Where σ=1 / (1+e -x ) is the sigmoid activation function, n is the number of samples; p0 is the confidence score of the predicted target position, p gt is the overlap ratio between the predicted bounding box and the corresponding true value; c p and c gt are the category probabilities of the predicted bounding box and the true bounding box respectively;

[0071] Loss mask It is obtained by calculating the pixel-level difference between the predicted mask and the true mask, and the binary cross entropy loss algorithm is used to calculate the predicted mask and the corresponding true binary mask of each detection instance; Loss mask The calculation satisfies the expression:

[0072]

[0073] Among them, N pos is the number of positive samples, ROI i is the i-th instance region, p is the pixel index within the region, m gt is the true mask, m pred is the prediction mask.

[0074] In some embodiments, based on the target prediction network, the perspective n-point (PnP) algorithm based on the random sample consensus (RANSAC) framework is adopted. The instrument shape mask information is used as additional geometric constraints and verification information to correct errors, generate and verify candidate poses, and solve for the pose to obtain the refined pose of the surgical instrument. That is, the aforementioned S150 may specifically include the following steps:

[0075] S410. Identify and eliminate the interference of abnormal data points through iterative sampling and model verification. In each RANSAC iteration, randomly extract a minimum subset from all observed corresponding points that can solve the basic PnP problem, and calculate a candidate pose.

[0076] S420. Use the candidate pose to project all three-dimensional marker points onto the image plane, calculate the reprojection error to identify inliers, and form a preliminary support set for this pose; use the candidate pose to project the entire known three-dimensional model of the instrument onto the two-dimensional image to generate an expected instrument contour mask.

[0077] S430. Calculate the mask intersection over union (Mask IoU) between the instrument contour mask and the segmentation mask of the surgical instrument actually output by the target prediction network to achieve geometric consistency comparison at the pixel level and determine the optimal inlier set.

[0078] S440. After determining the optimal inlier set, determine the final refined pose P* by minimizing a preset joint objective function.

[0079] In the embodiments of the present application, it can be understood that since the actually detected 2D marker points p i may contain noise or incorrect matches (outliers), directly using all detected points for PnP solution may lead to unreliable results. Based on this, the present application embeds the PnP solution into the RANSAC framework and enhances it using the segmentation mask; the core logic of this method is to identify and eliminate the interference of abnormal data points through iterative sampling and model verification.

[0080] It should be noted that the perspective n-point (PnP) algorithm based on the random sample consensus (RANSAC) framework can be denoted as the PnP-RANSAC algorithm. The present application innovatively integrates the pixel-level segmentation mask of the surgical instrument output by the network as a strong geometric constraint into the PnP-RANSAC pose solution process based on local marker points. By performing pixel-level shape matching measurement between the projected three-dimensional model contour and the actual segmentation mask generated by the network when verifying candidate poses, the global shape information is effectively utilized to screen and optimize the solution results. This method significantly enhances the resistance of pose estimation to marker point noise, occlusion, and detection errors, and obtains stability and accuracy far beyond methods that only rely on sparse marker points.

[0081] It should also be noted that the mask-based alignment can accurately evaluate the degree of shape matching between the projection model and the actual detection results, and is used to assist in screening the optimal hypothesis. Especially when multiple candidate poses have a similar number of inlier marker points, the pose with the highest degree of coincidence with the segmentation mask is preferentially selected; the final pose refinement step not only minimizes the reprojection error of the marker points, but also can selectively perform joint optimization to achieve the best alignment between the projected 3D model contour and the detected segmentation mask.

[0082] In one example, the joint objective function satisfies the expression:

[0083]

[0084] where P* represents the optimal pose, and argmin P represents finding the pose P that minimizes the objective function, λ reproj and λ mask are weight factors, I * is the index set of inliers found by RANSAC, p i is the detected 2D marker point coordinates, X i is the corresponding known 3D marker point coordinates, K is the camera internal parameter, proj() is the projection function, M proj is the projection mask generated according to the pose P and the known 3D model, and M net is the segmentation mask output by the neural network.

[0085] In some embodiments, the present application provides a surgical instrument pose analysis system 500, as Figure 5 shown, the surgical instrument pose analysis system 500 may include the following modules:

[0086] An encoding module 510, configured to select multiple feature points on the surgical instrument as multiple infrared reflective marker points, and perform encoding processing on the infrared reflective marker points based on a preset encoding rule;

[0087] A camera processing module 520, configured to synchronize the infrared camera and the color camera in a preset image acquisition unit so that both can capture the scene at the same time point and perform spatial calibration;

[0088] An image generation module 530, configured to capture information of the surgical instrument through the infrared camera and the color camera respectively and generate an infrared image and a color image; wherein, the infrared image includes infrared reflective marker points attached to the surgical instrument, and the color image includes instrument shape mask information;

[0089] The network construction module 540 is used to construct a target prediction network and a loss function based on infrared images and color images. The target prediction network is used for infrared optical marker point coordinate detection and instrument mask prediction to identify the overall shape of the instrument and the attached tiny positioning marker points, and fuse global and local information to achieve real-time three-dimensional pose tracking of the surgical instrument in a complex environment;

[0090] The pose solution module 550 is used to, based on the target prediction network, adopt the perspective-n-point (PnP) algorithm based on the random sample consensus (RANSAC) framework, use the instrument shape mask information as additional geometric constraints and verification information to correct errors, generate and verify candidate poses, and solve the pose to obtain the refined pose of the surgical instrument.

[0091] According to the embodiments of the present application, any multiple of the encoding module 510, the camera processing module 520, the image generation module 530, the network construction module 540, and the pose solution module 550 can be combined and implemented in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0092] In some embodiments, the pose solution module 550 can specifically be used for:

[0093] Identify and eliminate the interference of abnormal data points through iterative sampling and model verification. In each RANSAC iteration, randomly extract a minimum subset from all observed corresponding points that can solve the basic PnP problem, and calculate a candidate pose;

[0094] Use the candidate pose to project all three-dimensional marker points onto the image plane, calculate the reprojection error to identify inliers, and form a preliminary support set for this pose; use the candidate pose to project the entire known three-dimensional model of the instrument onto the two-dimensional image to generate an expected instrument contour mask;

[0095] Calculate the mask intersection over union (Mask IoU) between the instrument contour mask and the segmentation mask of the surgical instrument actually output by the target prediction network to achieve geometric consistency comparison at the pixel level and determine the optimal inlier set;

[0096] After determining the optimal inlier set, determine the final refined pose P* by minimizing a preset joint objective function.

[0097] Figure 5 Each module in the shown system has the functions to implement each step in the foregoing method for analyzing the pose of a surgical instrument and can achieve its corresponding technical effects. For the sake of concise description, it will not be elaborated here.

[0098] In some embodiments, the present application provides an electronic device, and a schematic structural diagram of the electronic device is as shown in Figure 6 the following figure.

[0099] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0100] Specifically, the above-mentioned processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0101] The memory 620 may include a mass storage for data or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In a suitable case, the memory 620 may include a removable or non-removable (or fixed) medium. In a suitable case, the memory 620 may be internal or external to the integrated gateway disaster recovery device. In a specific embodiment, the memory 620 is a non-volatile solid state memory.

[0102] The memory 620 may include a read only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, the memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the surgical instrument pose analysis methods in the above embodiments.

[0103] The processor 610 reads and executes the computer program instructions stored in the memory 620 to implement any of the surgical instrument pose analysis methods in the above embodiments.

[0104] In one example, the electronic device may further include a communication interface 630 and a bus 600. Among them, as shown in Figure 6 the following figure, the processor 610, the memory 620, and the communication interface 630 are connected through the bus 600 and complete communication with each other.

[0105] The communication interface 630 is mainly used to implement communication between each module, device, unit, and / or device in the embodiments of the present application.

[0106] The bus 600 includes hardware, software, or both, and couples the components of the online data flow metering device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 600 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0107] In addition, in combination with the surgical instrument pose analysis method in the above embodiments, the embodiments of the present application can be implemented by providing a computer storage medium. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the surgical instrument pose analysis methods in the above embodiments is implemented.

[0108] It should be clear that the present application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present application.

[0109] The functional blocks shown in the above structural block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or a communication link. A "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical discs, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0110] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. That is to say, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0111] As described above with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block in the flowchart and / or block diagram, and the combination of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more blocks of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It is also understood that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can also be implemented by dedicated hardware that performs the specified functions or actions, or by a combination of dedicated hardware and computer instructions.

[0112] In summary, compared with the prior art, this application has the following beneficial effects:

[0113] 1. This application collects infrared images and color images through a dual-camera configuration of an infrared camera and a color camera. The infrared images include infrared reflective marker points attached to the surgical instrument, and the color images include instrument shape mask information to characterize the overall shape of the surgical instrument. By designing a target prediction network based on infrared images and color images for infrared optical marker point coordinate detection and instrument mask prediction, it can synchronously and accurately identify the overall shape of the surgical instrument and the key infrared positioning marker points. This strategy of combining global shape information with local marker point features, combined with the Perspective-n-Point (PnP) algorithm based on the Random Sample Consensus (RANSAC) framework, uses the instrument shape mask information as additional geometric constraints and verification information to correct errors.

[0114] 2. The present application designs an innovative multi-task learning network architecture, which can efficiently and synchronously and accurately identify two types of targets with a huge scale difference within a unified framework: the overall shape of the surgical instrument and the tiny infrared positioning marker points. Compared with traditional image processing methods or deep learning models that only focus on a single task, the present application has achieved remarkable results in ensuring the accuracy and processing efficiency of detecting these two types of targets at the same time; it can also improve the accuracy and stability of pose tracking under various interference factors including occlusion, illumination change, and detection error, and obtain a reliable refined pose of the surgical instrument.

[0115] 3. The present application provides a high-performance real-time tracking solution to empower downstream applications: the overall solution is designed in an end-to-end manner, tightly coupling advanced multi-task perception capabilities (synchronous detection of precise masks and marker points) with a strongly robust pose solving strategy. Therefore, compared with the general deficiencies of the prior art in terms of accuracy, stability, or efficiency, the present invention can provide real-time six-degree-of-freedom (6DoF) pose tracking of surgical instruments with higher performance, higher accuracy, and stronger stability.

[0116] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for analyzing the pose of a surgical instrument, characterized in that, Including: Select multiple feature points on the surgical instrument as multiple infrared reflective marker points, and perform coding processing on the infrared reflective marker points based on a preset coding rule; Synchronize the infrared camera and the color camera in the preset image acquisition unit so that both can capture the scene at the same time point and perform spatial calibration; Capture the information of the surgical instrument through the infrared camera and the color camera respectively and generate an infrared image and a color image; wherein, the infrared image includes the infrared reflective marker points attached to the surgical instrument, and the color image includes the instrument shape mask information; Construct a target prediction network and a loss function based on the infrared image and the color image. The target prediction network is used for infrared optical marker point coordinate detection and instrument mask prediction to identify the overall shape of the instrument and the attached micro-positioning marker points, and fuse the global and local information to realize the real-time three-dimensional pose tracking of the surgical instrument in a complex environment; Based on the target prediction network, adopt the Perspective-n-Point (PnP) algorithm based on the Random Sample Consensus (RANSAC) framework, use the instrument shape mask information as additional geometric constraints and verification information to correct errors, generate and verify candidate poses, and solve the pose to obtain the refined pose of the surgical instrument.

2. The method for analyzing the pose of a surgical instrument according to claim 1, wherein The spatial calibration includes the internal parameter calibration of the infrared camera and the color camera and the external parameter calibration between the two to determine the accurate rotation and translation relationship of one with respect to the other, and associate the information from both to the same three-dimensional space or a common reference coordinate system; The number of the infrared reflective marker points is five and forms a set A0. The elements in the set A0 are the coding results of the instrument marker points: A0 = {A1, A2, A3, A4, A5}; The preset coding rule corresponding to the five infrared reflective marker points during the coding process satisfies: Point A1 is located at a predefined reference position on the instrument; Points A2 and A3 are distributed together with Point A1, and Points A1, A2, and A3 form a non-special triangle; The distance d12 from Point A1 to Point A2, the distance d13 from Point A1 to Point A3, and the distance d23 from Point A2 to Point A3 are all unequal to avoid isosceles or equilateral structures; The position of Point A4 relative to Points A1, A2, and A3 is set so that the four-point combination formed by Points A1, A2, A3, and A4 does not have symmetry in space; The spatial position of Point A5 is significantly deviated from the reference plane approximately formed by the remaining four points, and A5 has an obvious non-zero displacement in the direction perpendicular to the corresponding reference plane.

3. The method for analyzing the pose of a surgical instrument according to claim 1, characterized in that, The target prediction network includes: a Backbone backbone network, a Neck intermediate layer, and a Head output layer; Wherein, the Backbone backbone network includes a CBS module, a Res2Net module, and a first SPP module; The CBS module includes a Convolutional Neural Network (CNN) layer, a Batch Normalization (BN) layer, and a SiLU activation function layer; In the Res2Net module, the input feature map first undergoes a 1×1 convolution. Subsequently, the feature map will be divided into s subsets, such that the number of channels in each subset of the same size is reduced to 1 / s. Each independent feature subset will be convolved with a corresponding 3×3 convolution kernel, denoted as f i , and the subset is denoted as x i , i ∈ {1, 2, …, s}; the feature subset x i is added to the output of f i-1 , and then input into f i , and the output is denoted as o i ; wherein, the feature map includes the infrared image and the color image; the processing process of the Res2Net module satisfies the expression: The first SPP module includes multiple parallel max - pooling operations. The pooling kernel size and stride are adaptively calculated according to the size of the input feature map. The feature vectors output by all parallel max - pooling layers are concatenated along the feature dimension to form a final fixed - length output vector.

4. The method for analyzing the pose of a surgical instrument according to claim 3, characterized in that, The Neck intermediate layer is used for: Receiving the fixed - length output vector from the Backbone backbone network, and obtaining three temporary outputs after a series of up - sampling and down - sampling operations; Conducting the three temporary outputs to a preset second SPP module and obtaining three feature maps of different scales, and the sizes of the three feature maps are 13×13, 26×26, and 52×52 respectively; Performing another up - sampling operation on the feature map with a size of 13×13 to bring the features back to the lower - level network, so as to obtain a large number of gradients and semantic representations; Through a preset neck module DFPN, the output of the second SPP module and the feature map before the prediction network layer are concatenated again to obtain three feature outputs of different scales; Among them, the number of channels of the three feature outputs at different scales are 1024, 512, and 256 respectively. The resolution of the three increases in turn, the receptive field decreases in turn, and they are respectively connected to the large decoupling head, medium decoupling head, and small decoupling head preset in the Head output layer.

5. The method for analyzing the pose of a surgical instrument according to claim 4, wherein The Head output layer is used for: Integrating the large decoupling head, the medium decoupling head, and the small decoupling head into the overall framework of the target prediction network, separating the classification and regression tasks, performing a dimensionality reduction operation through a 1×1 convolution block, and implementing the classification and regression tasks by two network branches respectively; Outputting the prediction result through a 1×1 convolution operation. The network branch of the classification part obtains the categories to which different objects in the image belong, and the regression branch obtains the target position and confidence score through two parallel sub - branches; The shared features are input into the mask path after dimensionality reduction by an initial 1x1 convolution, and are processed by a CBL and an up - sampling layer, and a pixel - level mask prediction is output by a 1x1 convolution layer, generating a single - channel mask representation with a resolution twice that of the original feature map for each detection instance; Among them, the large decoupling head and the small decoupling head capture the instrument shape mask information and the infrared reflective marker points of small targets respectively.

6. The method for analyzing the pose of a surgical instrument according to claim 1, wherein The loss function is used for the structural optimization of the target prediction network. The overall loss of the target prediction network satisfies the expression: Loss total = Loss reg + Loss obj + Loss cls + Loss mask Among them, Loss total is the overall loss, Loss reg is the regression loss function for candidate boxes, and the confidence loss Loss obj is the confidence loss, Loss cls is the classification loss function, and Loss mask is the mask loss function; The Loss reg satisfies the expression: IoU(A,B) represents the intersection over union between the predicted value and the ground truth, where A represents the predicted bounding box and B is the corresponding ground truth; ρ 2 (A center ,B center ) represents the Euclidean distance between the predicted value and the ground truth, c is the diagonal length of the smallest bounding box, and βv is the penalty term; β represents the trade-off metric, and v is used to measure the uniformity of the aspect ratio; w and h are the width and height of the predicted bounding box, and the corresponding w gt and h gt are the width and height of the ground truth bounding box; Confidence Loss obj and the classification loss function Loss cls are implemented by binary cross-entropy loss with logistic regression, and the calculations of both satisfy the expression: where, σ = 1 / (1 + e -x ) is the sigmoid activation function, n is the number of samples; p0 is the confidence score of the predicted target position, p gt is the overlap rate between the predicted bounding box and the corresponding ground truth; c p and c gt are the class probabilities of the predicted bounding box and the ground truth bounding box, respectively; Loss mask It is obtained by calculating the pixel-level difference between the predicted mask and the ground truth mask. The binary cross-entropy loss algorithm is used to calculate the predicted mask and the corresponding ground truth binary mask for each detection instance; Loss mask The calculation of Loss satisfies the expression: Among them, N pos is the number of positive samples, ROI i is the i-th instance region, p is the pixel index within the region, m gt is the ground truth mask, m pred is the predicted mask.

7. The method for analyzing the pose of a surgical instrument according to claim 1, wherein Based on the target prediction network, using the Perspective - n - Point (PnP) algorithm based on the Random Sample Consensus (RANSAC) framework, taking the instrument shape mask information as additional geometric constraints and verification information to correct errors, generating and verifying candidate poses, and solving the pose to obtain the refined pose of the surgical instrument, including: Identifying and excluding the interference of abnormal data points through iterative sampling and model verification. In each RANSAC iteration, a minimum subset that can solve the basic PnP problem is randomly selected from all observed corresponding points, and a candidate pose is calculated; Project all three-dimensional fiducial points onto the image plane using the candidate pose, calculate the reprojection error to identify inliers, and form a preliminary support set for this pose; project the entire known three-dimensional model of the instrument onto the two-dimensional image using the candidate pose to generate an expected instrument contour mask; Calculate the Mask Intersection over Union (Mask IoU) between the instrument contour mask and the segmentation mask of the surgical instrument actually output by the target prediction network to achieve geometric consistency comparison at the pixel level and determine the optimal inlier set; After determining the optimal inlier set, determine the final refined pose P* by minimizing a preset joint objective function; wherein, the joint objective function satisfies the expression: where P* represents the optimal pose, and argmin P denotes finding the pose P that minimizes the objective function, λ reproj and λ mask are weight factors, I * is the index set of inliers found by RANSAC, p i are the coordinates of the detected 2D marker points, X i are the corresponding known 3D marker point coordinates, K is the camera internal parameter, proj() is the projection function, M proj is the projection mask generated according to the pose P and the known 3D model, M net is the segmentation mask output by the neural network.

8. A surgical instrument pose analysis system, characterized in that, including: An encoding module for selecting multiple feature points on the surgical instrument as multiple infrared reflective fiducial points and performing encoding processing on the infrared reflective fiducial points based on a preset encoding rule; A camera processing module for synchronizing the infrared camera and the color camera in a preset image acquisition unit so that both can capture the scene at the same time point and perform spatial calibration; An image generation module for capturing information of the surgical instrument through the infrared camera and the color camera respectively and generating an infrared image and a color image; wherein, the infrared image includes infrared reflective fiducial points attached to the surgical instrument, and the color image includes instrument shape mask information; A network construction module for constructing a target prediction network and a loss function based on the infrared image and the color image, where the target prediction network is used for infrared optical fiducial point coordinate detection and instrument mask prediction to identify the overall shape of the instrument and the attached micro-positioning fiducial points, and fuse global and local information to achieve real-time three-dimensional pose tracking of the surgical instrument in a complex environment; A pose solving module for, based on the target prediction network, using the perspective-n-point (PnP) algorithm based on the random sample consensus (RANSAC) framework, using the instrument shape mask information as additional geometric constraints and verification information to correct errors, generate and verify candidate poses, and solve for the refined pose of the surgical instrument.

9. An electronic device, characterized in that, including: A processor, a memory, and a program stored on the memory and executable on the processor, where the program, when executed by the processor, implements a surgical instrument pose analysis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A program or instruction is stored on the computer-readable storage medium, and the program or instruction, when executed by the processor, implements a surgical instrument pose analysis method according to any one of claims 1 to 7.

Images