A surgical navigation system and method of determining a position of a point of interest in a coordinate system

Abstract

This invention provides a surgical navigation system and a method for determining the position of a point of interest (POI) in a coordinate system. A system and method for operating a surgical navigation system to determine the position of a POI in the 3D coordinate system (tool coordinate system) of a surgical tool are provided. This involves using a calibrated imaging device to capture an image of a portion of the surface of the surgical tool, the calibrated imaging device having calibration data that enables a processor of the surgical navigation system to map 2D positions in the image onto projection lines in the 3D coordinate system (image coordinate system) of the imaging device. The system is operated to determine a mapping between the image coordinate system and the tool coordinate system. The processor is then operated to determine the position of the POI in the tool coordinate system based on the image and the mapping between the image coordinate system and the tool coordinate system.

Description

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 121,413, filed December 4, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The described implementation relates to the field of medical devices, and in particular to the field of surgical navigation systems. Background Technology

[0003] In a typical surgical navigation system, a stereo pose tracking subsystem dynamically tracks the pose of an optical marker rigidly attached to the tool being navigated. The position of the tracked tool tip in the 3D coordinate system of the optical marker is obtained using a measurement method known as "tool tip calibration." In a typical calibration system, tool tip calibration is accomplished by instructing the user to press the tool tip against a surface feature (such as a bore, groove, or pit) of a calibration device simultaneously tracked by the pose tracking subsystem. This surface feature has a position and shape known to the navigation system (e.g., US 7,166,114, US 2003 / 0040879), which is then used to calculate the tip position. Alternatively, the user is instructed to move the tool around a tool tip position fixed by a surface feature, and the system calculates the tool tip position by estimating that fixed position from the measured movement. These calibration methods disrupt surgical workflows, can introduce inaccuracies due to user error, and require additional preparation steps, such as sterilization of the calibration device.

[0004] Furthermore, the calibration device is designed for specific tip shapes and specific points of interest (POIs) on or within the tip, thus limiting the range of tool tips that can be precisely calibrated. Summary of the Invention

[0005] The various embodiments described herein generally relate to systems and methods for locating points of interest (POIs) on, inside, or near surgical instruments, with the aim of providing navigational guidance for manipulating the instrument during surgery. Example surgical navigation systems for locating points of interest on surgical instruments include:

[0006] processor;

[0007] Surgical tools, wherein the surgical tools have points of interest (POIs) in the 3D coordinate system (tool coordinate system) of the surgical tools;

[0008] A calibration imaging device is configured to capture an image of a portion of the surface of a surgical tool, the calibration imaging device having calibration data that enables the processor of the surgical navigation system to map 2D positions in the image to light projection lines in the 3D coordinate system (image coordinate system) of the imaging device.

[0009] The processor is configured as follows:

[0010] Determine the mapping between the image coordinate system and the tool coordinate system; and

[0011] The position of the POI in the tool coordinate system is determined based on the image and the mapping between the tool coordinate system and the image coordinate system.

[0012] In any embodiment, in order to determine the position of the POI in the tool coordinate system, the processor may be further configured to:

[0013] Identify the POI projection position on the image based on the appearance of the portion of the tool surface in the image;

[0014] Calculate the 2D coordinates of the POI projection position on the image;

[0015] Calculate the POI projection line in the image coordinate system corresponding to the POI projection position based on the calibration data; and

[0016] The position of the POI in the tool coordinate system is determined based on at least the POI projection lines and the mapping between the image coordinate system and the tool coordinate system.

[0017] In any embodiment, the processor may be further configured to additionally determine the position of the POI in the tool coordinate system based on selecting the position on the POI projection line by intersecting the projection line with at least one line or at least one surface in the tool coordinate system on which the POI is known to be located.

[0018] In any embodiment, the at least one line or at least one surface may include a tool tip rotation axis, and the processor may be further configured to select the position on the POI projection line by calculating the intersection position between the POI projection line and the tool tip rotation axis.

[0019] In any embodiment, the surface of the surgical tool may include a patient contact portion, and the processor may be further configured to obtain shape parameters of the patient contact portion.

[0020] In any implementation, determining the position of the POI in the tool coordinate system may include:

[0021] A second calibration imaging device captures a second image of a portion of the surface of the surgical tool from a different perspective than the first image. The second calibration imaging device has second calibration data that enables the processor to map 2D positions in the second image onto projection lines in a second image coordinate system, which is a 3D coordinate system of the second imaging device.

[0022] The processor may be further configured to:

[0023] Determine the mapping between the second image coordinate system and the tool coordinate system;

[0024] Based on the appearance of the portion of the tool surface in the second image, identify the second POI projection position on the second image, and calculate the 2D coordinates of the second POI projection position on the second image;

[0025] Based on the second calibration data, calculate the second POI projection line in the second image coordinate system corresponding to the second POI projection position, and

[0026] The position of the POI in the tool coordinate system is determined by calculating the intersection position between the POI projection line and the second POI projection line.

[0027] In any embodiment, the processor may be configured to determine a mapping between the image coordinate system and the tool coordinate system based on the image of the portion of the surface of the surgical tool.

[0028] In any embodiment, the imaging device may further include a plurality of images configured to capture said portion of the surface of the surgical tool, each of the plurality of images being captured from a different viewpoint; and the processor may be further configured to, for each of the plurality of images, to:

[0029] The POI projection position on the image is identified based on the appearance of a portion of the tool surface in the plurality of images;

[0030] Calculate the 2D coordinates of the POI projection position on the image;

[0031] Based on the calibration data, calculate the POI projection line in the image coordinate system corresponding to the POI projection position on the image, and

[0032] The position of the POI in the tool coordinate system is determined by calculating the intersection position between each of the POI projection lines.

[0033] In any embodiment, the calibration imaging device may be configured to capture a plurality of images of said portion of the surface of the surgical tool, each of said plurality of images having a different mapping between the image coordinate system and the tool coordinate system, and the processor may be further configured to:

[0034] Based on the mapping of each image, the tool surface projection contour search region in each of the plurality of images is calculated, and each tool surface projection contour search region has multiple pixel values;

[0035] Align the tool surface projection contour search region in the plurality of images; and

[0036] The POI projection position is calculated based on the pixel values ​​in the combined alignment tool surface projection contour search area, such that the pixel position corresponding to the aligned edge is distinguishable from the pixel value corresponding to the non-aligned edge.

[0037] In any embodiment, the processor may be further configured to:

[0038] The first and second positions in the tool coordinate system are selected based on the spatial region defined by the surgical tool;

[0039] Map the first and second positions of the tool coordinate system to the image;

[0040] The image is resampled in a rectangular grid to form a resampled image, the rectangular grid having rows and columns, wherein the first and second positions are located on the same row or column;

[0041] Image processing is used to locate the 2D coordinates of the resampled POI projection position in the image coordinate system;

[0042] Map the 2D coordinates of the resampled POI projection location to the image; and

[0043] Calculate the 2D coordinates of the POI projection position on the image.

[0044] In any embodiment, the processor may be further configured to:

[0045] The location of the POI is stored in a computer-readable storage medium, and

[0046] During a plurality of time periods at regular intervals, at each of the plurality of time periods at regular intervals, the processor is configured to:

[0047] Operate the surgical navigation system to determine the mapping between the image coordinate system and the tool coordinate system; and

[0048] The processor is operated to determine whether to update the stored POI location based on the mapping between the image coordinate system and the tool coordinate system.

[0049] An example method for operating a surgical navigation system to determine the position of a point of interest (POI) in the 3D coordinate system of a surgical tool (tool coordinate system) includes:

[0050] A calibration imaging device is used to capture an image of a portion of the surface of a surgical tool, the calibration imaging device having calibration that enables the processor of the surgical navigation system to map 2D positions in the image onto light projection lines in the 3D coordinate system (image coordinate system) of the imaging device.

[0051] Operate the surgical navigation system to determine the mapping between the image coordinate system and the tool coordinate system; and

[0052] The processor of the surgical navigation system operates to determine the position of the POI in the tool coordinate system based on the image and the mapping between the image coordinate system and the tool coordinate system.

[0053] In any embodiment, operating the processor of the surgical navigation system to determine the location of the POI may include operating the processor to:

[0054] The POI projection position on the image is identified based on the appearance of the portion of the tool surface in the image;

[0055] Calculate the 2D coordinates of the POI projection position on the image;

[0056] Calculate the POI projection line in the image coordinate system corresponding to the POI projection position based on the calibration data; and

[0057] The position of the POI in the tool coordinate system is determined based on at least the POI projection lines and the mapping between the image coordinate system and the tool coordinate system.

[0058] In any embodiment, operating the processor to determine the position of the POI in the tool coordinate system may additionally be based on selecting the position on the POI projection line by intersecting the projection line with at least one line or at least one surface in the tool coordinate system on which the POI is known to be located.

[0059] In any embodiment, the at least one line or at least one surface may include a tool tip rotation axis, and selecting the position on the POI projection line may include calculating the intersection position between the POI projection line and the tool tip rotation axis.

[0060] In any embodiment, the surface of the surgical tool may include a patient contact portion, and the method may further include operating the processor to obtain shape parameters of the patient contact portion.

[0061] In any implementation, determining the position of the POI in the tool coordinate system may include:

[0062] A second image of a portion of the surface of the surgical tool is captured from a different perspective than the first image using a second calibration imaging device. The second calibration imaging device has second calibration data that enables the processor to map 2D positions in the second image to a second image coordinate system, which is the 3D coordinate system of the second imaging device.

[0063] The surgical navigation system is operated to determine the mapping between the second image coordinate system and the tool coordinate system;

[0064] The processor of the surgical navigation system operates to:

[0065] The second POI projection position on the second image is identified based on the appearance of the portion of the tool surface in the second image;

[0066] Calculate the 2D coordinates of the second POI projection position on the second image;

[0067] Based on the second calibration data, calculate the second POI projection line in the second image coordinate system corresponding to the second POI projection position.

[0068] Operating the processor to determine the position of the POI in the tool coordinate system may further include calculating the intersection position between the POI projection line and the second POI projection line.

[0069] In any embodiment, the mapping between the image coordinate system and the tool coordinate system may be determined based on the image of the portion of the surface of the surgical tool.

[0070] In any embodiment, the method may further include using a calibrated imaging device to capture multiple images of said portion of the surface of the surgical tool, each of said multiple images being taken from a different perspective; and

[0071] For each of the plurality of images, the processor of the surgical navigation system is operated to:

[0072] The POI projection position on the image is identified based on the appearance of the portion of the tool surface in the image;

[0073] Calculate the 2D coordinates of the POI projection position on the image;

[0074] Based on the calibration data, calculate the POI projection line in the image coordinate system corresponding to the POI projection position on the image; and

[0075] Operating the processor to determine the position of the POI in the tool coordinate system may further include calculating the intersection positions between each of the POI projection lines.

[0076] In any embodiment, the method may further include using a calibrated imaging device to capture a plurality of images of the portion of the surface of the surgical tool, each of the plurality of images having a different mapping between the image coordinate system and the tool coordinate system, and wherein operating the processor to calculate the 2D coordinates of the POI projection position in the images may further include operating the processor to:

[0077] Based on the mapping of each image, the tool surface projection contour search region in each of the plurality of images is calculated, and each tool surface projection contour search region has multiple pixel values;

[0078] Align the tool surface projection contour search region in the plurality of images; and

[0079] The POI projection position is calculated based on the pixel values ​​in the combined alignment tool surface projection contour search area, such that the pixel position corresponding to the aligned edge is distinguishable from the pixel value corresponding to the non-aligned edge.

[0080] In any embodiment, calculating the 2D coordinates of the POI projection location on the image may further include operating the processor to:

[0081] The first and second positions in the tool coordinate system are selected based on the spatial region defined by the surgical tool;

[0082] Map the first and second positions of the tool coordinate system to the image;

[0083] The image is resampled in a rectangular grid to form a resampled image, the rectangular grid having rows and columns, wherein the first and second positions are located on the same row or column;

[0084] Image processing is used to locate the 2D coordinates of the resampled POI projection position in the image coordinate system;

[0085] Map the 2D coordinates of the resampled POI projection location to the image.

[0086] In any embodiment, the method may further include storing the location of the POI in a computer-readable storage medium and at each of the plurality of times at a time interval:

[0087] Operate the surgical navigation system to determine the mapping between the image coordinate system and the tool coordinate system; and

[0088] The processor is operated to determine whether to update the stored POI location based on the mapping between the image coordinate system and the tool coordinate system.

[0089] These and other aspects and features of the various implementations will be described in more detail below. Attached Figure Description

[0090] Several embodiments will now be described in detail with reference to the accompanying drawings, wherein:

[0091] Figure 1 This is an example illustration of a surgical navigation system for locating points of interest on a surgical instrument, according to at least one embodiment;

[0092] Figure 2 It is used according to at least one implementation method Figure 1 An example illustration of images captured by a surgical navigation system;

[0093] Figure 3 It is used according to at least one implementation method Figure 1 Another example of an image captured by a surgical navigation system is illustrated;

[0094] Figure 4 This is a flowchart of an instance method for locating points of interest on a surgical tool.

[0095] The accompanying drawings described below are provided for the purpose of illustrating, and not limiting, various aspects and features of the embodiments described herein. For the sake of simplicity and clarity, the elements shown in the drawings are not necessarily drawn to scale. For clarity, the dimensions of some elements may be exaggerated relative to others. It should be understood that, for the sake of simplicity and clarity, reference numerals in the drawings may be repeated where deemed appropriate to indicate corresponding or similar elements or steps. Detailed Implementation

[0096] It should be understood that many specific details have been set forth in order to provide a thorough understanding of the exemplary embodiments described herein. However, those skilled in the art will understand that the embodiments described herein can be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Furthermore, this description and accompanying drawings should not be construed as limiting the scope of the embodiments described herein in any way, but rather as describing only the implementation of the various embodiments described herein.

[0097] It should be noted that degree terms such as “basically,” “about,” and “approximately” are used in this document to indicate a reasonable amount of deviation from the modified term such that the final result is not significantly altered. These degree terms should be interpreted as including the deviation from the modified term if such deviation does not negate the meaning of the modified term.

[0098] Furthermore, as used herein, the term “and / or” is intended to mean inclusive-or. That is, for example, “X and / or Y” is intended to mean X or Y or both. As a further example, “X, Y and / or Z” is intended to mean X or Y or Z or any combination thereof.

[0099] It should be noted that the term "coupled" as used in this article means that two components can be directly coupled to each other, or they can be coupled to each other through one or more intermediate components.

[0100] In at least one embodiment, aspects of the method described herein, such as those referenced below, are... Figure 4 The described method 1000 can be implemented in hardware or software, or a combination of both. These implementations can be implemented in a computer program that executes on a programmable computer, each computer including at least one processor, a data storage system (including volatile or non-volatile memory or other data storage elements or combinations thereof), and at least one communication component. For example, but not limited to, the programmable computer (hereinafter referred to as the data processor) can be a server, network device, embedded device, computer expansion module, personal computer, laptop computer, personal data assistant, cellular phone, smartphone device, tablet computer, wireless device, or any other computing device capable of being configured to perform the methods described herein.

[0101] In at least one embodiment, the communication component may be a network communication interface. In embodiments in which combined elements are used, the communication component may be a software communication interface, such as those for inter-process communication (IPC). In yet another embodiment, there may be a combination of communication components implemented as hardware, software, or a combination thereof.

[0102] Program code can be applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

[0103] Each program can be implemented in a high-order process or object-oriented programming and / or scripting language, or both, to communicate with the computer system. However, if desired, the program can be implemented in assembly language or machine language. In any case, the language can be a compiled or interpreted language. Each such computer program can be stored on a general-purpose or special-purpose programmable computer-readable storage medium or device (e.g., ROM, disk, optical disk) for configuring and operating the computer to perform the program described herein when the storage medium or device is read by the computer. Implementations of the system can also be considered as being implemented as a non-transitory computer-readable storage medium configured with a computer program, wherein such a storage medium causes the computer to operate in a specific and predetermined manner to perform the functions described herein.

[0104] Now for reference Figure 1 The illustration shows an example of a system 100 for determining the location of a point of interest (POI) in a 3D coordinate system of a surgical instrument, according to at least one embodiment. System 100 includes a surgical instrument 200, a calibration imaging device 300, and a processor 400. By using image processing, the use of the calibration imaging device 300 eliminates the need for a calibration device in the surgical navigation system.

[0105] Surgical tool 200 can be any tool used during any type of surgery. In the example shown, surgical tool 200 is a dental handpiece. Surgical tool 200 has a body 210 with a motor 212 for driving a rotating chuck inside the head of the handpiece. In at least one embodiment, the tool can be another type of power tool, such as a screwdriver or vibratory saw, or a passive tool, such as a probe or indicator with a tapered, round, or spherical tip.

[0106] The surgical navigation system 100 can be used to calibrate one or more points of interest (POIs) of the user on or within the surgical instrument 200 (typically at or near the patient contact surface of the instrument). One or more POIs of the surgical instrument 200 can vary depending on the intended use of the instrument 200. For example, the surgical instrument 200 can have replaceable rotary drills with a fixed axis of rotation 224 in the coordinate system of the instrument 200. Since this axis of rotation is fixed relative to the optical tracker 240 for all rotary drills, it can be individually calibrated before inserting any drill into the instrument chuck, such as by using a special trackable drill. Calibration of the axis of rotation of any drill before insertion into the chuck can be performed in any manner. The results of the axis calibration can be stored in the permanent memory of the processor 400 for retrieval on demand.

[0107] Once the drill bit is inserted into the chuck, the point of interest (POI) will typically be the position of the drill bit tip along a pre-calibrated axis of rotation. In at least one embodiment, the tip of tool 200 may be fixed to tool 200 (i.e., non-replaceable), but the tip's position in the tool coordinate system may have changed, for example, due to accidental impact during or between uses due to tip bending. In at least one embodiment, tool 200 may have a repositionable tip that is movable relative to the tool 200's coordinate system along a confined curve or within a confined region. For example, a replaceable tip may be screwed onto the tool body 200 using a helical thread, such that its POI varies along a substantially circular path depending on the tip's orientation and the magnitude of the tightening force at the start of the tightening process.

[0108] In the illustrated example, tool 200 has a chuck 220 for receiving replaceable drill bit parts. The replaceable drill bit part can be any drill bit used during surgery. For example, in the illustrated example, the drill bit part is drill bit 222. The drill bit part is inserted into the chuck 220, which can be used to rotate the drill bit part about its drill axis 224. Drill bit 222 has a patient contact portion 226, which may also be referred to as a tool tip. In at least one embodiment, drill bit 222 may be fixed or repositioned along the drill path and / or non-removable or removable.

[0109] The calibration imaging device 300 can be any device capable of capturing an image of at least a portion of the surgical instrument 200. For example, in Figure 1In the example shown, the calibration imaging device 300 is a stereo camera with a first sensor 310 and a second sensor 320. The first sensor 310 and the second sensor 320 can simultaneously project light reflected from the surgical field (i.e., the spatial region where the point of interest is located), allowing images to be captured from two perspectives. Associated with the imaging device 300 is a coordinate system 302, a 3D coordinate system used to specify the position relative to the body of the imaging device 300. The calibration imaging device 300 includes calibration data that enables the mapping of light measured at 2D pixel locations in the first sensor 310 and / or the second sensor 320 onto projection lines in the image coordinate system 302.

[0110] In other words, the processor can identify the POI projection location in an image based on the appearance of a portion of the tool surface in the image. Identifying the POI projection location in the image can be based on implicit or explicit assumptions about the position of POI 230 in the tool coordinate system 202. For example, the POI location can be identified by, but is not limited to, marking the POI location with red dots against a blue background, wherein the processor 400 is configured to identify color patterns in the image, geometric features of the projection of the surface of the surgical tool 200 near the POI (such as parallel edges of the cylindrical portion projection), and / or can be trained to enable the processor 400 to identify POIs using instance images showing a portion of the tool surface annotated with POI locations.

[0111] Using multiple images of the tool from multiple different perspectives can make recognition more reliable. For example, using a stereo camera that provides two different perspectives can help distinguish tool and background pixels and reduce measurement noise. Figure 1 As shown. In at least one embodiment, the calibration imaging device 300 may be single-field-of-view and may have a single lens and a single image sensor. As will be further described below, the reliability of distinguishing between tool and background pixels can also be improved by using a series of captured images in which the tool moves relative to a fixed background.

[0112] The surgical tool 200 has a 3D coordinate tool coordinate system 202 that moves with the tool 200. The tool coordinate system 202 can be determined using a calibration imaging device 300 or by any other means. For example, in the illustrated example, the tool 200 has a trackable target 240 rigidly coupled to the tool 200, such that the trackable target 240 shares the tool coordinate system 202. In the illustrated example, the trackable target 240 is an optical marker with three optical targets 242. The optical targets 242 can be tracked using the calibration imaging device 300, another imaging device, or both, to track the tool coordinate system 202.

[0113] Processor 400 communicates with the image data stream generated by imaging device 300. Processor 400 may be a single component or may be implemented as several connected computing elements. Processor 400 can process image data from imaging device 300 and can use stored calibration data to calculate the mapping between tool coordinate system 202 and image coordinate system 302. Imaging device 300 can be used for pose tracking using stereo and single-field-of-view motion tracking methods. In such a case, the image data stream calibrating imaging device 300 can be used for both mapping between tool coordinate system 202 and image coordinate system 302 and for calibrating the POI of the tool, as described herein.

[0114] The surgical navigation system 100 can be used to determine the 2D position of POI 230 in the tool coordinate system based on identifying a spatial region of the tool 200 on or near a known POI. POI 230 may be above or within the surface of the tool 200, above or within the tool tip of the replaceable drill bit 222 of the tool 200, and may be positioned along a curve. For example, in Figure 1 In the example shown, POI 230 is the tool tip of drill bit 222 and the spatial region includes drill axis 224, such that drill axis 224 defines a straight segment, which is a simple curve, on which POI 230 must lie. In another example, in Figure 3 In the example shown, the POI (center 630) is located at the center of the spherical tip surface of drill bit 222. In at least one embodiment, drill bit 222 can be repositioned relative to tool 200. For example, one or more lockable joints can be used to reposition drill bit 222.

[0115] In at least one embodiment, the range of possible locations where the tool tip can be found can be defined by a path or surface that can be mathematically described in the tool coordinate system 202. As described above, the processor 400 can also identify possible locations based on the appearance of a portion of the tool surface in the image. For example, the POI may be located at or near the tip of a flexible shank. The length, shape, and material properties of the shank can limit the location of the POI to a surface in the tool coordinate system 202, whose descriptor can be pre-obtained and stored, as well as the area used to limit the POI identification in 2D. This information can then be used to locate the POI in 3D.

[0116] To locate the 2D coordinates of POI 230 in an image, the pixel position corresponding to POI 230 can be calculated based on the appearance of the tool surface portion in the image. The determination of the pixel position corresponding to POI 230 can vary depending on the type of drill bit used in the tool 200. For example, when the tool has a rotatable, replaceable drill bit, the drill axis 224 can be used to locate the pixel position corresponding to the 2D coordinates of POI 230 in the image captured by the imaging device 300. (Reference) Figure 2 A first position 512 and a second position 514 can be selected in the tool coordinate system 202 based on the spatial region defined by the tool 200. In the example shown, the first position 512 is the position where the drill bit 222 leaves the chuck 220, while the second position 514 is just beyond the longest possible drill bit that its tip needs to be detected. The first position 512 and the second position 514 can be mapped from the tool coordinate system 202 to an image.

[0117] The image can be resampled within a rectangular grid to form a resampled POI search image 500. The rectangular grid has rows and columns, where the first position 512 and the second position 514 are located in the same row or column. Figure 2 In the example shown, the projection of the first position 512 is at the top center of the resampled POI search image 500, and the second position 514 is at the bottom center of that region. The width of the region of the resampled POI search image 500 can be determined based on the size appearance range of the drill bit to be calibrated, which can be based on the width of the largest diameter drill bit used with tool 200 and the closest distance at which the drill bit will be presented to the imaging device 300 during calibration. For example, the width of the region can be a few pixels wider than the appearance of the maximum width of the drill bit used with tool 200 at the shortest distance from the camera in which the drill bit will be focused. Processor 400 can process the image to locate the 2D coordinates of the resampled POI projection position in image coordinate system 302 and can map the 2D coordinates of the resampled POI projection position to the image.

[0118] Image processing methods can be used to locate the coordinates of POI 230 in the resampled POI search image 500 by mapping between coordinates aligned with axis 224 and image coordinates, or alternatively, by locating the coordinates of POI 230 along the column (or row) of the drill bit axis 224 in the original image. In other words, images can be resampled as needed without forming an explicitly axis-aligned image. For example, image processing methods can include, but are not limited to, applying a convolutional neural network trained using a large number of drill bit images, manually labeling POIs in these images, and / or edge detection methods that rely on the approximate symmetry of axis 224 about the projected appearance of the drill bit and variations in the orientation of edges near POI 230.

[0119] Once POI 230 is located in the resampled POI search image 500, the 2D coordinates can be mapped back to the complete image and provided to the processor 400 to calculate one or more 3D projection lines between the image coordinate system 302 and the tool coordinate system 202. Based on the calibration data of the imaging device 300, one or more 3D projection lines can be calculated in the image coordinate system 302 from one or more locations on the tool 200 from the imaging device 300. (Reference) Figure 1 In the example shown, the first sensor 310 projects the first projection line 312 onto the POI 230 and the second projection line 314 onto the tracker 240, while the second sensor 320 projects the third projection line 322 onto the POI 230 and the fourth projection line 324 onto the tracker 240.

[0120] The surgical navigation system 100 can use the first projection line 312 and the third projection line 322 to locate the 3D coordinates of POI 230 in the tool coordinate system 202. Figure 1 In the illustrated example, the second projection line 314 and the fourth projection line 324, optionally together with similar projection lines of other optical targets, can be used to locate the positions of three or more optical targets 242 to calculate the pose of the tracker 240, thereby allowing the surgical navigation system 100 to map between the tool coordinate system 202 and the image coordinate system 302. Once the coordinate mapping between the tool coordinate system 202 and the image coordinate system 302 is calculated, the projection lines of the imaging device 300 can then be mapped into either of the two coordinate systems and intersect in their respective coordinate systems to determine the position of the point of interest (POI) 230.

[0121] In at least one embodiment, tracking can be performed using different perspectives projected from multiple imaging devices at different locations. For example, individually calibrating an imaging device can be used to determine the coordinate mapping between the image coordinate system 302 and the tool coordinate system 202 by tracking the imaging device 300 in its coordinate system 302, while the imaging device 300 can be used to locate the point of interest (POI) 230 on the tool 200. In at least one embodiment, tracking of the tool 200 can be performed using non-optical methods. For example, an electromagnetic tracking system can be used to track the tool 200.

[0122] The position of POI 230 in tool coordinate system 202 can be calculated based on the first projection line 312, the third projection line 322, and / or a known spatial region in tool coordinate system 202 on which POI 230 lies. The spatial region in tool coordinate system 202 may include the line or surface along which POI 230 resides in tool coordinate system 202. For example, in Figure 1In the example shown, the spatial region may include drill axis 224, which contains point of interest (POI) 230, because axis 224 extends through the tool tip and POI 230 is located at the tool tip. The position of the POI in 3D can be calculated using the intersection of one or more projection lines and the tool tip axis 224. Figure 1 In the example shown, axis 224 can intersect with one or both of the first projection line 312 and the third projection line 322 to locate the projection position.

[0123] In at least one embodiment, two intersecting lines can be used to locate POI 230. For example, a first projection line 312 and a third projection line 322 can be used to intersect at POI 230 without using drill axis 224. In at least one embodiment, a single projection line, such as the first projection line 312, can intersect drill axis 224 to locate POI 230. Figure 1 In the example shown, due to the positioning of the first sensor 310 and the second sensor 320, the first projection line 312 and the third projection line 322 are projected along different viewing angles.

[0124] In at least one embodiment, a calibration imaging device 300 can be operated to capture multiple images of a portion of the surface of a surgical tool, each image being captured from a different perspective to enable the calculation of a Point of Interest (POI) 230 via the intersection of multiple projection lines in the tool coordinate system. For each of the multiple images, a processor 400 can be operated to identify the POI projection position on the image based on the appearance of the portion of the tool surface in the image; calculate the 2D coordinates of the POI projection position on each image; calculate the POI projection line in the image coordinate system 302 corresponding to the POI projection position on the image; and map the projection line to the tool coordinate system 202. The position of the POI 230 in the tool coordinate system 202 can then be calculated by locating the intersection points between two or more projection lines from the multiple images.

[0125] In at least one embodiment, different views of the tool in multiple images can be obtained by moving the tool 200 to different positions and / or orientations so that the imaging device 300 captures different perspectives of the tool 200. For example, the tool 200 can be rotated and moved while the imaging device 300 captures multiple images from each of different viewing directions. In at least one embodiment, different perspectives of multiple images can be obtained by using multiple imaging devices.

[0126] In at least one embodiment, different perspectives of multiple images can be obtained by using a second calibration imaging device to capture a second image of a portion of the surface of a surgical tool from a different perspective than the first image. The second calibration imaging device may have second calibration data that enables the processor 400 to map 2D positions in the images captured by the second imaging device to a second image coordinate system, which serves as the 3D coordinate system of the second imaging device.

[0127] In at least one embodiment, multiple images can be obtained, in which the projection of the profile edge of drill 222 appears against multiple different backgrounds. For example, such multiple images can be obtained from stereoscopic or sequentially different video frames, or both. Using a mapping between the tool coordinate system and the image coordinate system, the regions in which the POIs are searched can be aligned, for example, by maximizing registration through the similarity of their boundaries and / or their contents, and the aligned search regions are resampled such that the edges 228 of the surface profile appear at similar locations in all resampled regions. Multiple aligned search regions can then be used to highlight or suppress pixels, making drill 222 more clearly identifiable. For example, the POI projection location can be identified based on the pixel values ​​in the combined aligned search regions, making pixel locations corresponding to aligned edges easier to distinguish from pixel locations corresponding to unaligned edges. For example, pixel values ​​at the same location can be averaged across multiple aligned regions. Static edges in the aligned images can then remain detectable, while moving edges (i.e., background) can be blurred by averaging with non-edge pixels mapped to the same location in other images. Other examples may include, but are not limited to, using median or total pixel values ​​to detect edges in each image by first setting binary pixel values ​​(edge ​​or non-edge) using an edge detection algorithm and then counting edges that pass through each pixel or its vicinity.

[0128] In at least one embodiment, one or more calibration constraints may be applied to processor 400 to limit when calibration occurs. These constraints may be added to the algorithm as a quality control measure to improve its reliability and accuracy. For example, in at least one embodiment, system 100 may include computer-readable storage and the location of POI 230 may be stored in the computer-readable storage. During a plurality of time periods at regular intervals, at each of these time periods, system 100 may be operated to determine the mapping between image coordinate system 302 and tool coordinate system 202. At each of these time periods at regular intervals, processor 400 may be operated to determine whether the stored POI location should be updated based on the mapping between image coordinate system 302 and tool coordinate system 202. In other words, the location of the POI at time t may be stored in computer-readable storage. Later, processor 400 may be operated to evaluate the mapping between tool coordinate system 202 and image coordinate system 302 to determine whether the stored POI location should be recalculated and updated.

[0129] Processor 400 can determine whether to update the stored POI by using one or more conditions. For example, processor 400 can calculate a distance value corresponding to the distance between imaging device 300 and tool 200, and an angle value between a line in tool coordinate system 202 (such as chuck rotation axis 224) and a general line of sight between a position on the tool (such as 512) and the imaging device. Processor 400 can then compare the distance value to a range of distance values ​​and the angle value to a range of angle values. When the distance value is within the range of distance values ​​and the angle value is within the range of angle values, processor 400 can be operated to determine the POI position and update the stored POI position. When the distance value is not within the range of distance values ​​or the angle value is not within the range of angle values, the stored POI position is not updated. For example, if tool 200 is too far or too close to imaging device 300, tool 200 may be out of focus. Therefore, system 100 can only perform calibration when tool 200 is within a certain angle and distance range, making tool 200 easier to focus and thus improving the reliability of calibration. In another example, to ensure that the POI positioning algorithm is applied only when the spatial area containing POI 230 is visible in the tool coordinate system, the algorithm can only be activated by the processor 400 when the drill axis 224 is calculated to be approximately perpendicular to the line of sight. That is, in this example, calibration can only be performed when the tool tip is viewed from the side, such as... Figure 2 shown.

[0130] In at least one embodiment, the processor 400 can be operated to calculate the orientation of the patient contact portion 226 at each of a plurality of time intervals. When the orientation of the patient contact portion 226 is substantially opposite to the direction of gravity acting on the surgical tool 200, the processor 400 can be operated to determine the POI position and update the stored POI position; when the orientation of the patient contact portion 226 is not substantially opposite to the direction of gravity acting on the surgical tool 200, the stored POI position is not updated. For example, the drill bit 222 can be slightly moved along axis 224. When drilling, the force acting on the drill bit 222 presses it into the chuck 220. When not drilling and pointing downwards, the drill bit 222 can be slightly moved out of the chuck 220. Therefore, when the tool tip is pointing upwards, gravity can act on the drill bit 222 to pull it into the chuck 220 to a position similar to when the drill bit 222 is pressed against the tissue, thus providing more reliable calibration. These quality control measures can make calibration more reliable and accurate.

[0131] Additional quality control measures may include, but are not limited to, estimating the strength and symmetry of image edges when locating tips in images, and / or using statistical analysis of tip locations calculated from multiple frames in the video stream to remove outliers and average noise and jitter.

[0132] In at least one embodiment, the processor 400 can be operated to obtain shape parameters of the patient contact portion 226. For example, a descriptor of the outer contour 228 of the drill bit 222 can be obtained. The outer contour 228 can be used to provide a geometric description of the shape of the cutting volume of the drill bit 222 that will be removed when the drill bit 222 is rotated by the chuck 220. In other words, the descriptor of the outer contour can be used to provide additional information to the user during the procedure.

[0133] In at least one embodiment, the position of the POI at the fixed tool tip can be stored in the permanent memory of the processor 400 to allow the processor 400 to recalibrate as needed. For example, some tools, such as indicators or scalpels, have a fixed tip at the end of a slender handle or blade that is typically flexible during surgery, causing the POI 230 to change between surgeries. The POI 230 can be recalibrated at the start of each surgery to compensate for the movement of the POI 230 between surgeries. The previous calibration can be used to estimate where the POI needs to be searched, to verify that the position of the POI has not changed, and if it has changed, to update the position. In other words, a POI position calculated once can be used as the basis for the estimated position of a POI position calculated later.

[0134] refer to Figure 3The drill bit 222 has a substantially spherical tip with a known radius attached to the shank, with a point of interest (POI) 630 located at the center of the sphere. During operation, based on an earlier measurement of the POI, the center of the sphere is projected onto image 600 at pixel 610. The projection profile 620 of the spherical tip on image 600 can be calculated using imaging device 300 and image calibration data. By searching near the projection center 610 to locate the actual projection center 630, image 600 can be processed to determine the coordinates of the center of the nearby circular edge. This process can then be repeated from different perspectives, such as by using two sensors of imaging device 300 or by rotating the tool tip in front of imaging device 300. The projection lines of two or more sphere centers can then be intersected to locate the sphere center (POI) in 3D.

[0135] refer to Figure 4 The flowchart illustrates an example method 1000 for operating a surgical navigation system to determine the position of a point of interest (POI) in the tool coordinate system 202.

[0136] In step 1100, the calibration imaging device 300 is used to capture an image of a portion of the surface of the surgical instrument. As previously described, the calibration imaging device has calibration data that enables the processor 400 of the surgical navigation system 100 to map 2D positions in the image to projection lines in the 3D coordinate system 302 of the imaging device.

[0137] In step 1200, the surgical navigation system 100 is operated to determine the mapping between the image coordinate system 302 and the tool coordinate system 202. This mapping may include any parameters that allow the processor 400 to calculate the position between each coordinate system. For example, parameters may include position, distance, and / or viewpoint.

[0138] In step 1300, the processor 400 of the surgical navigation system 100 determines the position of the POI in the tool coordinate system based on the image and the mapping between the image coordinate system and the tool coordinate system.

[0139] Optionally, the processor may be operated to: (i) identify the POI projection position on the image based on the appearance of the portion of the tool surface in the image; (ii) calculate the 2D coordinates of the POI projection position on the image; (iii) calculate the POI projection line in the image coordinate system corresponding to the POI projection position based on the calibration data; and (iv) determine the position of the POI 230 in the tool coordinate system 202 based on at least the POI projection line and the mapping between the image coordinate system 302 and the tool coordinate system 202.

[0140] Optionally, the surgical navigation system 100 can be operated to determine the position of the POI in the tool coordinate system 202 based on selecting the position on the POI projection line by intersecting the projection line with at least one line or at least one surface in the tool coordinate system 202 on which the POI 230 is known to be located. In at least one embodiment, the POI 230 may be invisible on the surface of the tool 200. For example, in Figure 3 In the example shown, POI 230 is shown at the center of the sphere at 630, which would be invisible in the image captured by imaging device 300.

[0141] This document describes various embodiments by way of example only. Various modifications and variations may be made to these exemplary embodiments without departing from the spirit and scope of the invention as defined only by the appended claims.

Claims

1. A method for operating a surgical navigation system to determine the position of a point of interest (POI) in the 3D coordinate system of a surgical tool (tool coordinate system), the method comprising: a) Using a calibration imaging device to capture an image of a portion of the surface of a surgical tool, the calibration imaging device having calibration data that enables the processor of the surgical navigation system to map 2D positions in the image to light projection lines in the 3D coordinate system (image coordinate system) of the imaging device; b) Operate the surgical navigation system to determine the mapping between the image coordinate system and the tool coordinate system; c) The processor of the surgical navigation system is operated to determine the position of the POI in the tool coordinate system based on the image and the mapping between the image coordinate system and the tool coordinate system; Specifically, operating the processor of the surgical navigation system to determine the location of the POI includes operating the processor to: i) Identify the POI projection position on the image based on the appearance of the portion of the tool surface in the image; ii) Calculate the 2D coordinates of the POI projection position on the image; iii) Calculate the POI projection line in the image coordinate system corresponding to the POI projection position based on the calibration data; and iv) Determine the position of the POI in the tool coordinate system based on at least the POI projection lines and the mapping between the image coordinate system and the tool coordinate system.

2. The method of claim 1, wherein, The processor is operated to determine the position of the POI in the tool coordinate system based additionally on the following: the position of the POI on the projection line is selected based on the intersection of the projection line with at least one line or at least one surface in the tool coordinate system on which the POI is known to be located.

3. The method according to claim 2, wherein, The at least one line or at least one surface includes a tool tip rotation axis, and selecting the position on the POI projection line includes calculating the intersection position between the POI projection line and the tool tip rotation axis.

4. The method according to claim 1, wherein, Determining the position of the POI in the tool coordinate system includes: A second image of a portion of the surface of the surgical tool is captured from a different perspective than the first image using a second calibration imaging device. The second calibration imaging device has second calibration data that enables the processor to map 2D positions in the second image onto projection lines in a second image coordinate system, which is the 3D coordinate system of the second calibration imaging device. The surgical navigation system is operated to determine the mapping between the second image coordinate system and the tool coordinate system; The processor of the surgical navigation system operates to: i) Identify the second POI projection position on the second image based on the appearance of the portion of the tool surface in the second image; ii) Calculate the 2D coordinates of the second POI projection position on the second image; iii) Based on the second calibration data, calculate the second POI projection line in the second image coordinate system corresponding to the second POI projection position. Operating the processor to determine the position of the POI in the tool coordinate system further includes calculating the intersection position between the POI projection line and the second POI projection line.

5. The method of claim 1, further comprising using a calibrated imaging device to capture a plurality of images of the portion of the surface of the surgical tool, each of the plurality of images being captured from a different viewpoint; as well as For each of the plurality of images, the processor of the surgical navigation system is operated to: i) Identify the POI projection position on the image based on the appearance of the portion of the tool surface in the image; ii) Calculate the 2D coordinates of the POI projection position on the image; iii) Based on the calibration data, calculate the POI projection line in the image coordinate system corresponding to the POI projection position on the image; as well as Operating the processor to determine the position of the POI in the tool coordinate system further includes calculating the intersection position between each of the POI projection lines.

6. The method according to any one of claims 1-5, further comprising using a calibrated imaging device to capture a plurality of images of said portion of the surface of the surgical tool, each of said plurality of images having a different mapping between the image coordinate system and the tool coordinate system, and wherein operating the processor to calculate the 2D coordinates of the POI projection position in the images further comprises operating the processor to: Based on the mapping of each image, the tool surface projection contour search region in each of the plurality of images is calculated, and each tool surface projection contour search region has multiple pixel values; Align the tool surface projection contour search region in the plurality of images; as well as The POI projection position is calculated based on the pixel values ​​in the combined alignment tool surface projection contour search area, such that the pixel position corresponding to the aligned edge is distinguishable from the pixel value corresponding to the unaligned edge.

7. The method according to any one of claims 1-5, wherein, Calculating the 2D coordinates of the POI projection location on the image further includes operating the processor to: The first and second positions in the tool coordinate system are selected based on the spatial region defined by the surgical tool; Map the first and second positions of the tool coordinate system to the image; The image is resampled in a rectangular grid to form a resampled image, the rectangular grid having rows and columns, wherein the first position and the second position are located in the same row or column; Image processing is used to locate the 2D coordinates of the resampled POI projection position in the image coordinate system; Map the 2D coordinates of the resampled POI projection location to the image.

8. The method according to any one of claims 1-5, further comprising storing the location of the POI in a computer-readable storage medium and, at each of the plurality of times at a time interval: Operate the surgical navigation system to determine the mapping between the image coordinate system and the tool coordinate system; and The processor is operated to determine whether to update the stored POI location based on the mapping between the image coordinate system and the tool coordinate system.

9. The method according to any one of claims 1-5, wherein, The surface of the surgical tool includes a patient contact portion, and the method further includes operating the processor to obtain shape parameters of the patient contact portion.

10. The method according to any one of claims 1-5, wherein, The mapping between the image coordinate system and the tool coordinate system is determined based on the image of a portion of the surface of the surgical tool.

11. A surgical navigation system, comprising: processor; Surgical tools, wherein the surgical tools have points of interest (POIs) in the 3D coordinate system (tool coordinate system) of the surgical tools; A calibration imaging device is configured to capture an image of a portion of the surface of a surgical tool, the calibration imaging device having calibration data that enables the processor of the surgical navigation system to map 2D positions in the image to light projection lines in the 3D coordinate system (image coordinate system) of the calibration imaging device. The processor is configured as follows: i) Determine the mapping between the image coordinate system and the tool coordinate system; as well as ii) Determine the position of the POI in the tool coordinate system based on the image and the mapping between the image coordinate system and the tool coordinate system; In order to determine the position of the POI in the tool coordinate system, the processor is further configured to: a. Identify the POI projection position on the image based on the appearance of the portion of the tool surface in the image; b. Calculate the 2D coordinates of the POI projection position on the image; c. Calculate the POI projection line in the image coordinate system corresponding to the POI projection position based on the calibration data; and d. The position of the POI in the tool coordinate system is determined based on at least the POI projection lines and the mapping between the image coordinate system and the tool coordinate system.

12. The system according to claim 11, wherein, The processor is further configured to additionally determine the position of the POI in the tool coordinate system based on the following: selecting the position on the POI projection line based on the intersection of the projection line with at least one line or at least one surface in the tool coordinate system on which the POI is known to be located.

13. The system according to claim 12, wherein, The at least one line or at least one surface includes a tool tip rotation axis, and the processor is further configured to select the position on the POI projection line by calculating the intersection position between the POI projection line and the tool tip rotation axis.

14. The system according to claim 11, wherein, Determining the position of the POI in the tool coordinate system includes: A second calibration imaging device is configured to capture a second image of a portion of the surface of the surgical tool from a different perspective than the first image. The second calibration imaging device has second calibration data that enables the processor to map 2D positions in the second image onto projection lines in a second image coordinate system, which is a 3D coordinate system of the second calibration imaging device. The processor is further configured as follows: i) Determine the mapping between the second image coordinate system and the tool coordinate system; ii) Identify the second POI projection position on the second image based on the appearance of the portion of the tool surface in the second image; iii) Calculate the 2D coordinates of the second POI projection position on the second image; iv) Based on the second calibration data, calculate the second POI projection line in the second image coordinate system corresponding to the second POI projection position, and v) The position of the POI in the tool coordinate system is determined by calculating the intersection position between the POI projection line and the second POI projection line.

15. The system according to claim 11, wherein, The imaging device further includes a plurality of images configured to capture the portion of the surface of the surgical tool, each of the plurality of images being captured from a different perspective; as well as The processor is further configured to, for each of the plurality of images, to: i) Identify the POI projection position on the image based on the appearance of the portion of the tool surface in the image; ii) Calculate the 2D coordinates of the POI projection position on the image; iii) Based on the calibration data, calculate the POI projection line in the image coordinate system corresponding to the POI projection position on the image, and iv) The position of the POI in the tool coordinate system is determined by calculating the intersection position between each of the POI projection lines.

16. The system according to any one of claims 11-15, wherein, The calibration imaging device is configured to capture multiple images of said portion of the surface of the surgical tool, each of the multiple images having a different mapping between the image coordinate system and the tool coordinate system, and The processor is further configured as follows: i) Based on the mapping of each image, calculate the tool surface projection contour search region in each of the plurality of images, wherein each tool surface projection contour search region has multiple pixel values; ii) Align the tool surface projection contour search region in the plurality of images; and iii) Calculate the POI projection position based on the pixel values ​​in the combined alignment tool surface projection contour search area, such that the pixel position corresponding to the aligned edge is distinguishable from the pixel value corresponding to the unaligned edge.

17. The system according to any one of claims 11-15, wherein, The processor is further configured to: i) Select a first position and a second position in the tool coordinate system based on the spatial region defined by the surgical tool; ii) Map the first and second positions of the tool coordinate system onto the image; iii) Resample the image in a rectangular grid to form a resampled image, the rectangular grid having rows and columns, wherein the first position and the second position are located in the same row or column; iv) Use image processing to locate the 2D coordinates of the resampled POI projection position in the image coordinate system; v) Map the 2D coordinates of the resampled POI projection location to the image; as well as vi) Calculate the 2D coordinates of the POI projection position on the image.

18. The system according to any one of claims 11-15, wherein, The processor is further configured to: The location of the POI is stored in a computer-readable storage medium, and During a plurality of time periods at regular intervals, at each of the plurality of time periods at regular intervals, the processor is configured to: i) Operate the surgical navigation system to determine the mapping between the image coordinate system and the tool coordinate system; as well as ii) Operate the processor to determine whether to update the stored POI location based on the mapping between the image coordinate system and the tool coordinate system.

19. The system according to any one of claims 11-15, wherein, The surface of the surgical tool includes a patient contact portion, and the processor is further configured to obtain shape parameters of the patient contact portion.

20. The system according to any one of claims 11-15, wherein, The processor is configured to determine a mapping between the image coordinate system and the tool coordinate system based on the image of a portion of the surface of the surgical tool.

Images