Surgical instrument positioning method and device, electronic equipment and storage medium

Abstract

The invention provides a positioning method and device of a surgical instrument, electronic equipment and a storage medium, and the positioning method comprises the steps: obtaining an image frame of the surgical instrument in a surgical process in real time based on an industrial camera, and carrying out the image preprocessing and contour extraction processing of the image frame, determining a plurality of reflective feature points of the tip of the surgical instrument; performing pixel ray normalization processing on the pixel coordinate of each reflective feature point, and reconstructing a three-dimensional coordinate of the tip of the surgical instrument in combination with geometric constraint and ray tracing of a reflective plane; wherein the reflecting planes are symmetrically arranged, and the reflecting planes are arranged between the industrial camera and the surgical instrument; and storing the reconstructed three-dimensional coordinates and generating a control instruction of the surgical instrument. The non-mark and non-contact real-time positioning method for the surgical instrument is realized, interference of additional marks on the surgery is eliminated, and the positioning accuracy of the surgical instrument is improved.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and in particular to a method, device, electronic device and storage medium for positioning surgical instruments. Background Technology

[0002] Traditional surgical instrument positioning techniques heavily rely on optical markers (such as reflective spheres or LED arrays) attached to the instruments themselves. These markers significantly reduce operating space in the confined environment of ophthalmic surgery, increasing the risk of accidental collisions between instruments and intraocular tissues. Biological fluids generated during surgery (such as blood and vitreous fluid) easily adhere to the marker surface, causing optical signal attenuation or even complete failure, forcing frequent interruptions during surgery to clean the markers, thus leading to inaccurate positioning. Furthermore, customized marker clamps are difficult to adapt to different sizes of minimally invasive instruments, significantly increasing clinical procurement and maintenance costs. Therefore, improving the accuracy of surgical instrument positioning has become a significant technical challenge. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method, device, electronic device and storage medium for positioning surgical instruments, which realizes a markerless and non-contact real-time positioning method for surgical instruments, eliminates the interference of additional markers on the operation and improves the accuracy of surgical instrument positioning.

[0004] This application provides a method for positioning a surgical instrument, the method comprising: Based on real-time acquisition of image frames of surgical instruments during the surgical process using an industrial camera, image preprocessing and contour extraction are performed on the image frames to determine multiple reflective feature points at the tip of the surgical instrument. The pixel coordinates of each reflective feature point are normalized using pixel rays, and the three-dimensional coordinates of the tip of the surgical instrument are reconstructed by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and positioned between the industrial camera and the surgical instrument. The reconstructed three-dimensional coordinates are stored and used to generate control commands for the surgical instruments.

[0005] In one possible implementation, the image preprocessing and contour extraction processing of the image frame to determine multiple reflective feature points of the tip of the surgical instrument includes: The image frame is subjected to HSV color space conversion, dual-interval threshold segmentation, contour extraction, and centroid calculation to determine multiple reflective feature points of the tip.

[0006] In one possible implementation, the process of performing HSV color space conversion, dual-interval thresholding, contour extraction, and centroid calculation on the image frame to determine multiple reflective feature points at the tip of the surgical instrument includes: The image frame is converted to HSV color space, and the red region is extracted from the color space converted image frame based on a preset dual-interval threshold to generate a binary mask image. The binary mask image is subjected to multiple morphological dilation processes, and the dilated binary mask image is subjected to contour extraction processing with connected components to determine multiple contours. From the multiple contours, a predetermined number of contours with larger contour areas are selected, and the centroid of the predetermined number of contours with larger contour areas is calculated to determine two centroid points, which are then used as the reflective feature points.

[0007] In one possible implementation, after performing multiple morphological dilation processes on the binary mask image, performing contour extraction processing with connected components on the dilated binary mask image, and determining multiple contours, the localization method further includes: If the centroids determined after centroid calculation for a preset number of contours with large contour areas do not exceed the preset number of centroids, then the next image frame will be processed.

[0008] In one possible implementation, the step of normalizing the pixel coordinates of each reflective feature point using pixel rays, and reconstructing the three-dimensional coordinates of the tip of the surgical instrument by combining preset geometric constraints of the reflective plane and ray tracing, includes: For each of the reflective feature points, distortion removal, coordinate transformation, and unit vector transformation are performed to determine the ray direction vector of each of the reflective feature points. The intersection point of each ray direction vector with the reflection plane is determined, and the virtual binocular path calculation is performed on the ray direction vector corresponding to each intersection point by transforming the reflection matrix to generate the reflected light corresponding to each intersection point; The nearest intersection point of the two reflected light rays is determined based on the least squares method, and the coordinates of the nearest intersection point are determined as the three-dimensional coordinates of the tip of the surgical instrument.

[0009] In one possible implementation, the transformation reflection matrix is ​​determined through the following steps: If the coordinates of the intersection point are greater than a preset value, the reflecting surface of the intersection point is a left-plane reflection; if the coordinates of the intersection point are less than the preset value, the reflecting surface of the intersection point is a right-plane reflection; wherein, the preset value is 0; Construct a corresponding transformation reflection matrix based on the reflecting surfaces at the intersection points.

[0010] In one possible implementation, the included angle of the reflecting plane is 20°.

[0011] This application embodiment also provides a positioning device for surgical instruments, the positioning device comprising: The feature detection module is used to acquire image frames of surgical instruments in real time during the operation based on an industrial camera, perform image preprocessing and contour extraction on the image frames, and determine multiple reflective feature points of the tip of the surgical instrument. The three-dimensional reconstruction module is used to perform pixel ray normalization processing on the pixel coordinates of each of the reflective feature points, and reconstruct the three-dimensional coordinates of the tip of the surgical instrument by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and is set between the industrial camera and the surgical instrument. The control planning module is used to store the reconstructed three-dimensional coordinates and generate control commands for the surgical instruments.

[0012] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the surgical instrument positioning method described above are performed.

[0013] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the surgical instrument positioning method described above.

[0014] This application provides a method, apparatus, electronic device, and storage medium for locating surgical instruments. The locating method includes: acquiring image frames of the surgical instrument during surgery in real time using an industrial camera; performing image preprocessing and contour extraction on the image frames to determine multiple reflective feature points at the tip of the surgical instrument; performing pixel ray normalization on the pixel coordinates of each reflective feature point, and reconstructing the three-dimensional coordinates of the tip of the surgical instrument by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and positioned between the industrial camera and the surgical instrument; storing the reconstructed three-dimensional coordinates and generating control commands for the surgical instrument. This achieves a markerless, non-contact real-time surgical instrument locating method, eliminating the interference of additional markers on the surgery and improving the accuracy of surgical instrument locating.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for positioning a surgical instrument as provided in an embodiment of this application; Figure 2 A schematic diagram of the reflective plane provided in an embodiment of this application; Figure 3 A schematic diagram of the structure of a positioning device for a surgical instrument provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0019] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of 3D reconstruction technology.

[0020] Research has revealed that traditional surgical instrument positioning techniques heavily rely on optical markers (such as reflective spheres or LED arrays) attached to the instruments themselves. These markers significantly reduce operating space in the confined environment of ophthalmic surgery, increasing the risk of accidental collisions between instruments and intraocular tissues. Biological fluids generated during surgery (such as blood and vitreous fluid) easily adhere to the marker surface, causing optical signal attenuation or even complete failure, forcing frequent interruptions during surgery to clean the markers, thus leading to inaccurate positioning. Furthermore, customized marker clamps are difficult to adapt to different sizes of minimally invasive instruments, significantly increasing clinical procurement and maintenance costs. Therefore, improving the accuracy of surgical instrument positioning has become a significant technical challenge.

[0021] Based on this, the embodiments of this application provide a method for locating surgical instruments, which realizes a markerless, non-contact real-time positioning method for surgical instruments, eliminates the interference of additional markers on the operation, and improves the accuracy of surgical instrument positioning.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for positioning a surgical instrument as provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the method for positioning surgical instruments includes: S101: Based on the industrial camera, image frames of surgical instruments during the operation are acquired in real time. Image preprocessing and contour extraction are performed on the image frames to determine multiple reflective feature points of the tip of the surgical instrument.

[0023] In this step, an industrial camera is used to acquire image frames of the surgical instruments during the surgical process in real time. The image frames are then preprocessed and contour extracted to determine multiple reflective feature points at the tip of the surgical instruments.

[0024] Here, the camera parameters of the calibrated industrial camera are pre-loaded, including the intrinsic parameter matrix and distortion coefficients.

[0025] In one possible implementation, the image preprocessing and contour extraction processing of the image frame to determine multiple reflective feature points of the tip of the surgical instrument includes: The image frame is subjected to HSV color space conversion, dual-interval threshold segmentation, contour extraction, and centroid calculation to determine multiple reflective feature points of the tip.

[0026] Here, the current image frame is processed by HSV color space conversion, dual-interval threshold segmentation, contour extraction, and centroid calculation to determine multiple reflective feature points at the tip.

[0027] In one possible implementation, the process of performing HSV color space conversion, dual-interval thresholding, contour extraction, and centroid calculation on the image frame to determine multiple reflective feature points at the tip of the surgical instrument includes: A: Perform HSV color space conversion on the image frame, and extract the red region from the color space converted image frame based on a preset dual-interval threshold to generate a binary mask image.

[0028] Here, the image frame is converted to HSV color space, and the red region is extracted from the color space converted image frame according to the preset dual-interval threshold to generate a binary mask image.

[0029] The preset dual-interval thresholds are the low red interval 0-10 and the high red interval 145-180 for red region extraction.

[0030] Here, HSV color space thresholding (find_reflected_contours) is used: each channel of the HSV color space represents hue, saturation, and value, which can intuitively express the brightness, hue, and vividness of a color. The HSV color space can be described using a cone space model. At the apex of the cone, V=0, H and S are undefined, representing black; at the center of the cone's top surface, V=max, S=0, H is undefined, representing white. When S=1 and V=1, any color represented by H is called a pure color; when S=0, the saturation is 0, the color is the lightest, described as gray, and the brightness of gray is determined by V, at which point H is meaningless; when V=0, the color is the darkest, described as black, at which point both H and S are meaningless, and regardless of their values, it is always black. Hue is a fundamental property of color, indicating that different colors can be used to describe and identify a certain color. For example, green in the HSV space ranges from H=35 to 77, while it is difficult to describe with an expression in the RGB space. Therefore, the HSV color space is often used for color identification and color comparison.

[0031] B: Perform multiple morphological dilation processes on the binary mask image, and then perform contour extraction processing with connected components on the dilated binary mask image to determine multiple contours.

[0032] Here, the binary mask image undergoes multiple morphological dilation processes to bridge minor breaks and enhance the connectivity of reflective areas. The dilated binary mask image is then subjected to contour extraction processing with connected components to determine multiple contours.

[0033] C: Select a preset number of contours with larger contour areas from the multiple contours, and perform centroid calculation on the preset number of contours with larger contour areas to determine two centroid points, and use the centroid points as the reflective feature points.

[0034] Here, a preset number of contours with larger contour areas are selected from multiple contours, and the centroid of the preset number of contours with larger contour areas is calculated to determine two centroid points, which are then used as reflective feature points.

[0035] Specifically, the top 10 contours with larger contour areas can be retained, and the centroid of these top 10 contours can be calculated to determine ten centroid points. Anomaly removal processing is then performed on these ten centroid points. From the centroid points after anomaly removal processing, the centroid points corresponding to the top three contours with larger contour areas are selected. The first centroid point among the centroid points corresponding to the top three contours with larger contour areas is used as the first reflective feature point, the second centroid point among the centroid points corresponding to the top three contours with larger contour areas is used as the second reflective feature point, and the second centroid point among the centroid points corresponding to the top three contours with larger contour areas is used as a redundancy check point.

[0036] After performing multiple morphological dilation processes on the binary mask image, extracting contours with connected components from the dilated binary mask image, and determining multiple contours, the positioning method further includes: if the centroids determined after centroid calculation of a preset number of contours with larger areas do not exceed the preset number of centroids, then the next image frame is processed.

[0037] Here, if the number of centroids determined after centroid calculation for a preset number of contours with large contour areas is less than 3, then the next image frame will be processed.

[0038] S102: Perform pixel ray normalization on the pixel coordinates of each of the reflective feature points, and reconstruct the three-dimensional coordinates of the tip of the surgical instrument by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and is positioned between the industrial camera and the surgical instrument.

[0039] In this step, the pixel coordinates of each reflective feature point are normalized using pixel rays, and the three-dimensional coordinates of the tip of the surgical instrument are reconstructed by combining the geometric constraints of the reflective plane and ray tracing.

[0040] The reflective planes are symmetrically arranged and positioned between the industrial camera and the surgical instrument.

[0041] For further details, please refer to Figure 2 , Figure 2This is a schematic diagram of the reflective plane provided in an embodiment of this application. Figure 2 As shown, the reflective plane is obtained by symmetrically setting two mirrors with the same angle, and the included angle of the reflective plane can be adjusted. The included angle of the reflective plane can be 20°, the thickness of the mirror is 2mm, and the height of the mirror from the plane is 10mm.

[0042] The included angle of the reflecting plane can be 20°-30°.

[0043] In one possible implementation, the step of normalizing the pixel coordinates of each reflective feature point using pixel rays, and reconstructing the three-dimensional coordinates of the tip of the surgical instrument by combining preset geometric constraints of the reflective plane and ray tracing, includes: a: Perform distortion removal, coordinate transformation, and unit vector transformation on each of the reflective feature points to determine the ray direction vector of each of the reflective feature points.

[0044] Here, the coordinates of each reflective feature point after distortion are obtained by performing distortion removal processing on each reflective feature point according to the distortion coefficient corresponding to the industrial camera. Then, the coordinates of each reflective feature point after distortion are transformed into world coordinates and into unit vectors to determine the ray direction vector of the reflective feature point.

[0045] The number of reflective feature points is two.

[0046] b: Determine the intersection point of each ray direction vector with the reflection plane, and perform virtual binocular path calculation on the ray direction vector corresponding to each intersection point by transforming the reflection matrix to generate the reflected light corresponding to each intersection point.

[0047] Here, the intersection point of each ray direction vector and the reflection plane is determined. By transforming the reflection matrix, a virtual binocular path calculation is performed on the ray direction vector corresponding to each intersection point to generate the reflected light corresponding to each intersection point.

[0048] Specifically, the intersection point of the ray and the reflecting plane is calculated based on the dot product relationship between the ray direction vector of the reflective feature point and the normal vector of the reflecting plane.

[0049] It's worth noting that traditional binocular vision systems rely on two physical cameras to acquire images from different perspectives, but this approach suffers from problems such as complex equipment, synchronization difficulties, and high costs. This application constructs a "virtual binocular vision system" by using two symmetrically arranged reflective planes (mirrors) and images captured by a real camera to simulate the perspectives of two virtual cameras through light reflection. The purpose of the reflection transformation matrix is ​​to accurately model the direction of the reflected light rays and simulate the perspectives of the two virtual cameras. The system uses a Householder reflection matrix (Householder Transformation). The Householder matrix is ​​an orthogonal transformation matrix that can mirror a vector about a plane. In this application, it is used to reflect the original light direction vector to the virtual camera perspective, thereby generating a "virtual light path."

[0050] c: Based on the least squares method, the nearest intersection point of the two reflected light rays is determined, and the coordinates of the nearest intersection point are determined as the three-dimensional coordinates of the tip of the surgical instrument.

[0051] Here, the nearest intersection point of the two reflected light rays is determined by the least squares method, and the coordinates of the nearest intersection point are used as the three-dimensional coordinates of the tip of the surgical instrument.

[0052] In one possible implementation, the transformation reflection matrix is ​​determined through the following steps: If the coordinates of the intersection point are greater than a preset value, the reflecting surface of the intersection point is a left-plane reflection; if the coordinates of the intersection point are less than the preset value, the reflecting surface of the intersection point is a right-plane reflection; wherein, the preset value is 0; a corresponding transformation reflection matrix is ​​constructed based on the reflecting surface of the intersection point.

[0053] Here, if the coordinates of the intersection point are greater than 0, the reflecting surface of the intersection point is the left plane reflection; if the coordinates of the intersection point are less than 0, the reflecting surface of the intersection point is the right plane reflection. During 3D reconstruction, it is necessary to find two rays from different viewpoints (i.e., left reflection and right reflection). This judgment logic ensures that the two rays come from different reflection paths, avoiding mismatches.

[0054] S103: Store the reconstructed three-dimensional coordinates and generate control instructions for the surgical instruments.

[0055] Here, the reconstructed 3D coordinates are stored and used to generate control commands for surgical instruments or to generate planned paths.

[0056] In a specific embodiment, pre-calibrated camera parameters (intrinsic parameter matrix, distortion coefficients) and the reflection plane normal vector are loaded, and image frames of the surgical instrument during the surgical process are acquired in real time using an industrial camera. The image frames are converted to HSV space and subjected to red-domain double-threshold segmentation. After morphological dilation, the centroids of the top three contours are extracted as reflective feature points. For each reflective feature point, distortion correction, coordinate transformation, and unit vector transformation are performed to determine the ray direction vector of each reflective feature point. The intersection point of each ray direction vector with the reflection plane is determined. By transforming the reflection matrix, virtual binocular path calculation is performed on the ray direction vector corresponding to each intersection point to generate the reflected light ray corresponding to each intersection point. The nearest intersection point of the two reflected light rays is determined by the least squares method, and the coordinates of the nearest intersection point are determined as the three-dimensional coordinates of the tip of the surgical instrument.

[0057] In this application, a virtual binocular eye path is constructed using a symmetrical reflection plane to replace a physical camera array, reducing cost and complexity. Adaptive color recognition is achieved through a combined computational process of HSV dual-threshold segmentation, contour extraction, and centroid calculation, enhancing anti-interference capabilities. Furthermore, a markerless, non-contact real-time surgical instrument positioning method is implemented, eliminating the interference of additional markers on the surgery and improving the accuracy of surgical instrument positioning.

[0058] The surgical instrument positioning method provided in this application is particularly suitable for sub-millimeter-level high-precision positioning of needle tips in ophthalmic surgery. It constructs a virtual binocular eye path by setting symmetrical reflective planes, automatically identifies instrument tip feature points using image processing algorithms, and reconstructs three-dimensional coordinates using ray tracing and planar reflection principles. Calibration parameters are dynamically loaded using a YAML configuration file to ultimately generate surgical instrument path planning data. This application solves the problems of marker interference and complex multi-view image synchronization in traditional optical positioning systems, significantly improving surgical safety and operational efficiency.

[0059] This application provides a method for locating a surgical instrument. The method includes: acquiring image frames of the surgical instrument during surgery in real time using an industrial camera; performing image preprocessing and contour extraction on the image frames to determine multiple reflective feature points at the tip of the surgical instrument; performing pixel ray normalization on the pixel coordinates of each reflective feature point, and reconstructing the three-dimensional coordinates of the tip of the surgical instrument by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and positioned between the industrial camera and the surgical instrument; storing the reconstructed three-dimensional coordinates and generating control commands for the surgical instrument. This method achieves a markerless, non-contact real-time positioning method for surgical instruments, eliminating the interference of additional markers on the surgery and improving the accuracy of surgical instrument positioning.

[0060] Please seeFigure 3 , Figure 3 This is a schematic diagram of the structure of a positioning device for a surgical instrument provided in an embodiment of this application. Figure 3 As shown, the positioning device 300 for the surgical instrument includes: The feature detection module 310 is used to acquire image frames of surgical instruments during the operation in real time based on an industrial camera, perform image preprocessing and contour extraction on the image frames, and determine multiple reflective feature points of the tip of the surgical instrument. The 3D reconstruction module 320 is used to perform pixel ray normalization processing on the pixel coordinates of each of the reflective feature points, and reconstruct the 3D coordinates of the tip of the surgical instrument by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and is set between the industrial camera and the surgical instrument. The control planning module 330 is used to store the reconstructed three-dimensional coordinates and generate control commands for the surgical instruments.

[0061] Furthermore, the feature detection module 310 is used to perform image preprocessing and contour extraction on the image frame to determine multiple reflective feature points of the tip of the surgical instrument: The image frame is subjected to HSV color space conversion, dual-interval threshold segmentation, contour extraction, and centroid calculation to determine multiple reflective feature points of the tip.

[0062] Furthermore, the feature detection module 310 is used to perform HSV color space conversion, dual-interval threshold segmentation, contour extraction, and centroid calculation on the image frame to determine multiple reflective feature points of the tip of the surgical instrument: The image frame is converted to HSV color space, and the red region is extracted from the color space converted image frame based on a preset dual-interval threshold to generate a binary mask image. The binary mask image is subjected to multiple morphological dilation processes, and the dilated binary mask image is subjected to contour extraction processing with connected components to determine multiple contours. From the multiple contours, a predetermined number of contours with larger contour areas are selected, and the centroid of the predetermined number of contours with larger contour areas is calculated to determine two centroid points, which are then used as the reflective feature points.

[0063] Furthermore, the feature detection module 310 is also used for: If the centroids determined after centroid calculation for a preset number of contours with large contour areas do not exceed the preset number of centroids, then the next image frame will be processed.

[0064] Furthermore, the 3D reconstruction module 320 performs pixel ray normalization processing on the pixel coordinates of each of the reflective feature points, and, in conjunction with the preset geometric constraints of the reflection plane and ray tracing, reconstructs the 3D coordinates of the tip of the surgical instrument: For each of the reflective feature points, distortion removal, coordinate transformation, and unit vector transformation are performed to determine the ray direction vector of each of the reflective feature points. The intersection point of each ray direction vector with the reflection plane is determined, and the virtual binocular path calculation is performed on the ray direction vector corresponding to each intersection point by transforming the reflection matrix to generate the reflected light corresponding to each intersection point; The nearest intersection point of the two reflected light rays is determined based on the least squares method, and the coordinates of the nearest intersection point are determined as the three-dimensional coordinates of the tip of the surgical instrument.

[0065] Furthermore, the 3D reconstruction module 320 determines the transformation reflection matrix through the following steps: If the coordinates of the intersection point are greater than a preset value, the reflecting surface of the intersection point is a left-plane reflection; if the coordinates of the intersection point are less than the preset value, the reflecting surface of the intersection point is a right-plane reflection; wherein, the preset value is 0; Construct a corresponding transformation reflection matrix based on the reflecting surfaces at the intersection points.

[0066] This application provides a surgical instrument positioning device, comprising: a feature detection module for acquiring image frames of the surgical instrument during surgery in real time using an industrial camera, performing image preprocessing and contour extraction on the image frames to determine multiple reflective feature points of the tip of the surgical instrument; a three-dimensional reconstruction module for performing pixel ray normalization on the pixel coordinates of each reflective feature point, and reconstructing the three-dimensional coordinates of the tip of the surgical instrument by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and positioned between the industrial camera and the surgical instrument; and a control planning module for storing the reconstructed three-dimensional coordinates and generating control commands for the surgical instrument. This achieves a markerless, non-contact real-time surgical instrument positioning method, eliminating the interference of additional markers on the surgery and improving the accuracy of surgical instrument positioning.

[0067] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0068] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the surgical instrument positioning method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0069] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the surgical instrument positioning method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0070] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0072] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0074] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for positioning surgical instruments, characterized in that, The positioning method includes: Based on real-time acquisition of image frames of surgical instruments during the surgical process using an industrial camera, image preprocessing and contour extraction are performed on the image frames to determine multiple reflective feature points at the tip of the surgical instrument. The pixel coordinates of each reflective feature point are normalized using pixel rays, and the three-dimensional coordinates of the tip of the surgical instrument are reconstructed by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and positioned between the industrial camera and the surgical instrument. The reconstructed three-dimensional coordinates are stored and used to generate control commands for the surgical instruments.

2. The positioning method according to claim 1, characterized in that, The image preprocessing and contour extraction process performed on the image frame to determine multiple reflective feature points at the tip of the surgical instrument includes: The image frame is subjected to HSV color space conversion, dual-interval threshold segmentation, contour extraction, and centroid calculation to determine multiple reflective feature points of the tip.

3. The positioning method according to claim 2, characterized in that, The process of performing HSV color space conversion, dual-interval threshold segmentation, contour extraction, and centroid calculation on the image frame determines multiple reflective feature points at the tip of the surgical instrument, including: The image frame is converted to HSV color space, and the red region is extracted from the color space converted image frame based on a preset dual-interval threshold to generate a binary mask image. The binary mask image is subjected to multiple morphological dilation processes, and the dilated binary mask image is subjected to contour extraction processing with connected components to determine multiple contours. From the multiple contours, a predetermined number of contours with larger contour areas are selected, and the centroid of the predetermined number of contours with larger contour areas is calculated to determine two centroid points, which are then used as the reflective feature points.

4. The positioning method according to claim 3, characterized in that, After performing multiple morphological dilation processes on the binary mask image, extracting contours with connected components from the dilated binary mask image, and determining multiple contours, the localization method further includes: If the centroids determined after centroid calculation for a preset number of contours with large contour areas do not exceed the preset number of centroids, then the next image frame will be processed.

5. The positioning method according to claim 1, characterized in that, The process of normalizing the pixel coordinates of each reflective feature point using pixel rays, and then reconstructing the three-dimensional coordinates of the tip of the surgical instrument by combining the preset geometric constraints of the reflective plane and ray tracing, includes: For each of the reflective feature points, distortion removal, coordinate transformation, and unit vector transformation are performed to determine the ray direction vector of each of the reflective feature points. The intersection point of each ray direction vector with the reflection plane is determined, and the virtual binocular path calculation is performed on the ray direction vector corresponding to each intersection point by transforming the reflection matrix to generate the reflected light corresponding to each intersection point; The nearest intersection point of the two reflected light rays is determined based on the least squares method, and the coordinates of the nearest intersection point are determined as the three-dimensional coordinates of the tip of the surgical instrument.

6. The positioning method according to claim 5, characterized in that, The transformation reflection matrix is ​​determined through the following steps: If the coordinates of the intersection point are greater than a preset value, the reflecting surface of the intersection point is a left-plane reflection; if the coordinates of the intersection point are less than the preset value, the reflecting surface of the intersection point is a right-plane reflection; wherein, the preset value is 0; Construct a corresponding transformation reflection matrix based on the reflecting surfaces at the intersection points.

7. The positioning method according to claim 1, characterized in that, The included angle of the reflecting plane is 20°.

8. A positioning device for a surgical instrument, characterized in that, The positioning device includes: The feature detection module is used to acquire image frames of surgical instruments in real time during the operation based on an industrial camera, perform image preprocessing and contour extraction on the image frames, and determine multiple reflective feature points of the tip of the surgical instrument. The three-dimensional reconstruction module is used to perform pixel ray normalization processing on the pixel coordinates of each of the reflective feature points, and reconstruct the three-dimensional coordinates of the tip of the surgical instrument by combining the geometric constraints of the reflective plane and ray tracing; wherein the reflective plane is symmetrically arranged and is set between the industrial camera and the surgical instrument. The control planning module is used to store the reconstructed three-dimensional coordinates and generate control commands for the surgical instruments.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the positioning method of the surgical instrument as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the positioning method for surgical instruments as described in any one of claims 1 to 7.

Images