Techniques for determining a marker arrangement of marker positions that define a tracker
By combining camera and CT/MRI imaging modes, and utilizing predetermined geometric relationships and image processing techniques, the problem of inaccurate marker placement and recognition in surgical tracking systems has been solved. This enables precise positioning of trackers with deformation or large manufacturing tolerances, thereby improving the accuracy of surgical navigation.
Patent Information
- Application Number
- CN202310211773.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-01
- Filing Date
- 2023-02-27
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-02-27
AI Technical Summary
In existing technologies, surgical tracking systems struggle to accurately determine marker placement when faced with deformable, customizable, or manufacturing-tolerant trackers, leading to inaccurate position recognition.
By combining two imaging modes (such as camera and CT/MRI), the relative position and arrangement of markers are determined by receiving image data from different imaging modes and utilizing predetermined geometric relationships and image processing techniques.
This improves the recognition accuracy of tracker markings that are deformed or have large manufacturing tolerances, ensuring the accuracy and reliability of surgical navigation.
Smart Images

Figure CN116687563B_ABST
Abstract
Description
Technical Field
[0001] This disclosure generally relates to tracking, for example, in a surgical navigation environment. More specifically, a technique for marker arrangement to determine the location of markers defining a tracker is proposed. This technique can be implemented as a method, computer program product, device, or surgical tracking system. Background Technology
[0002] Surgical tracking systems are configured to track surgical subjects, such as surgical instruments or patients in the operating room. Common tracking technologies include trackers with optical markers and cameras capable of detecting light emitted or reflected by the markers. Based on the detected light, the tracking system determines information related to one or both of the tracker's position and orientation, thereby determining information relevant to the surgical subject being tracked.
[0003] In order to determine the location or orientation of a tracked surgical object using image data generated by a camera, the tracking system needs to know the arrangement of markers. For example, the tracking system may need to know the marker arrangement to accurately locate the tracker or distinguish a particular tracker from other trackers. The marker arrangement can be evaluated by the tracking system, for example, in the form of the position coordinates of the markers in a given coordinate system.
[0004] However, in some cases, the marking arrangement may not be known accurately or sufficiently. Such scenarios include trackers that have been intentionally or unintentionally deformed before or during surgical intervention, customizable trackers that may have been configured in a patient-specific manner, or disposable trackers that have been manufactured with large manufacturing tolerances. Summary of the Invention
[0005] A technology is needed that allows the marker arrangement, which defines the location of the markers on the tracker, to be correctly determined.
[0006] According to one aspect, a method is provided for determining the location of markers defining a tracker by marker arrangement. The markers can be detected by a first imaging mode and a second imaging mode different from the first imaging mode. The method includes receiving first image data of the markers captured using the first imaging mode and receiving second image data of the markers captured using the second imaging mode. The method further includes determining the marker arrangement based on the first and second image data.
[0007] The arrangement of markers can define the positions of the markers relative to each other. Alternatively, the arrangement of markers can define the positions of the markers in a common coordinate system. As an example, determining the arrangement of markers can include determining the coordinates of the positions of two or more or all the markers in the coordinate system. As another example, determining the arrangement of markers can include determining the relative positions of the markers (e.g., in the form of relative distances and / or orientations).
[0008] At least one of the first image data and the second image data can represent at least one of an image, multiple images, video, point cloud, and polygon mesh. At least one of the first image data and the second image data can include information related to at least one of the viewpoint, viewing distance, and time in which the corresponding image data was captured.
[0009] At least one of the markers may have a first marker portion detectable by a first imaging mode and a second marker portion spaced apart from the first marker portion and detectable by a second imaging mode. The first marker portion may be arranged relative to the second marker portion in a predetermined geometric relationship. The marker arrangement may be further determined based on the predetermined geometric relationship.
[0010] The first marking portion may include a material configured to reflect or emit electromagnetic radiation (e.g., at least one of infrared, visible, and ultraviolet light) that can be sensed by a camera. The second marking portion may include a material detectable in a computed tomography (CT) scan or a magnetic resonance imaging (MRI) scanner.
[0011] At least one of the markers may have a third marker portion detectable by both the first and second imaging modes. At least one of the markers may have only a third marker portion or only a first and a second marker portion. As an example, the third marker portion may include one or more materials detectable by both the first and second imaging modes. The third marker portion may have a spatial extension that does not significantly exceed (e.g., exceed 50%) the spatial resolution of one or both of the first and second imaging modes.
[0012] One of the first and second imaging modes may include a CT scanner and an MRI scanner. The other of the first and second imaging modes may include at least one camera (e.g., a stereo camera). The camera may be configured to detect electromagnetic radiation (e.g., at least one of infrared, visible, and ultraviolet light).
[0013] The tracker's markers can be arranged in a common plane. Determining the marker arrangement can include virtually arranging the marker positions in a virtual plane determined based on at least one of first image data and second image data. The virtual plane can be determined based on a preliminary marker arrangement and before determining the actual or final marker arrangement by considering the virtual plane. The virtual plane can be determined based on at least one of the first image data and second image data.
[0014] The method may include registering at least one of first image data and second image data, or a third image data, with a tracker using a determined marker arrangement. The method may also include registering a CT or MRI scan from one of a first imaging mode and a second imaging mode (e.g., first image data acquired after the first and second image data, or a third image data) with a tracker using a determined marker arrangement. The method may include tracking (and optionally, visualizing) at least one of the position and orientation of a surgical object with the tracker relative to the registered image data.
[0015] The tracker may have a deformable base supporting the markers, wherein, for the undeformed shape of the base, the markers are arranged in a predetermined (e.g., planar) arrangement. For the deformed shape of the base, the marker arrangement may be further determined based on the predetermined arrangement. The deformable base may be at least one of bendable, stretchable, and compressible. The base may include a mask or spinal mask configured to fit the surface of a patient's face or back.
[0016] The method may include a first arrangement for determining a first position of a marker based on first image data, and a second arrangement for determining a second position of a marker based on second image data. The marker arrangement may be determined based on a combination of the first and second arrangements.
[0017] The marker arrangement can be determined based on the average position of the markers between the first and second arrangements. Alternatively, the marker arrangement can be determined based on a weighted average position of the markers between the first and second arrangements. For example, the position of a specific marker in the first and second arrangements can be processed to calculate (optionally, a weighted) average position.
[0018] In some cases, the average value is weighted toward either the first or second arrangement. The average value may be weighted more toward the imaging mode with a larger spatial resolution (e.g., the larger spatial resolution of the marker). Alternatively, or additionally, the average value may be weighted based on one or more of (i) and (ii), (i) the time series of the first and second image data that have been acquired, and (ii) at least one of the viewpoint and viewing distance of at least one of the first and second imaging modes. The average value may be weighted more toward the most recently captured image data. The first imaging mode (or alternatively, the second imaging mode) may include a camera, and when capturing the first image data, the average value may be weighted more toward the first imaging mode as the viewing distance of the camera decreases. The greater the time difference between capturing the first and second image data, the more the average value may be weighted toward the more recently captured image data.
[0019] The method may include determining invalid locations of a specific marker in at least one of a first arrangement or a second arrangement. The method may include ignoring invalid locations of a specific marker in at least one of the first or second arrangements when determining the location of the specific marker in the marker arrangement. If the location of a specific marker cannot be determined from the corresponding image data, the location may be determined as invalid. The location may be determined as invalid if the distance between the location of a specific marker and the expected location and / or the location of one or more other markers (e.g., in the same marker arrangement or in other marker arrangements) exceeds a threshold.
[0020] The method may include generating an error signal when a quality criterion between the positions of the markers in a first arrangement and the positions of the markers in a second arrangement satisfies an error condition. The quality criterion may include the distance between the positions of the markers in the first and second marker arrangements. The quality criterion may alternatively or additionally include the presence of determinable positions of the first and second marker arrangements.
[0021] The method may further include attempting to match a first position of the first arrangement with a second position of the second arrangement. Any matching algorithm can be used in this regard. The method may also include ignoring at least one second position in the first position that lacks a match. Alternatively, the method may include ignoring at least one first position in the second position that lacks a match. After excluding (e.g., “filtering out”) one or more ignored positions, a determination of the marker arrangement based on the first image data and the second image data can then be performed.
[0022] Matching (e.g., aligning) the first and second positions may include using point-set algorithms, such as algorithms for minimizing the difference between the first and second positions, such as the Iterative Closest Point (ICP) algorithm. A so-called “redundant” first or second position may be considered as lacking an alignment with the second or first position to be registered in at least one of the following cases: (i) the “redundant” first or second position lacks a matching second or first position within a threshold distance, and (ii) the “redundant” first or second position does not meet the alignment criteria of the point-set algorithm used.
[0023] The method may further include determining combined image data based on first image data and second image data. The marker arrangement may be determined based on the combined image data. The combined image data may include an imaging device that uses one of a first imaging mode and a second imaging mode to track the other of the first and second imaging modes. The combined image data may include identifying markers in the first and second image data and aligning the identified markers in the first and second image data.
[0024] According to a second aspect, a computer program product is provided. The computer program product includes instructions that, when executed on at least one processor, cause the at least one processor to perform any of the methods described herein. The computer program product may be stored on a non-volatile storage medium, such as a hard disk drive, optical disk drive, flash drive, cloud computing device, or download server.
[0025] According to a third aspect, an apparatus is provided for determining the location of markers defining a tracker. The markers can be detected by a first imaging mode and a second imaging mode different from the first imaging mode. The apparatus is configured to receive first image data of the markers captured using the first imaging mode, and to receive second image data of the markers captured using the second imaging mode. The apparatus is also configured to determine the marker arrangement based on the first and second image data.
[0026] The device can be configured to perform any of the method aspects and method steps as described herein.
[0027] According to a fourth aspect, a surgical tracking system is provided, comprising the means and tracker described herein, wherein the tracker is attached to a surgical object or includes an interface configured to be attached to a surgical object (e.g., a patient or surgical instrument). The interface may include at least one of an adhesive, a clamp, a screw, and a magnet. Attached Figure Description
[0028] Other details, advantages, and aspects of this disclosure will become apparent from the following embodiments, taken in conjunction with the accompanying drawings, in which:
[0029] Figure 1 A surgical tracking and navigation system including a device according to the present disclosure is shown;
[0030] Figure 2A A cross-section of a first example of a passive marker with a first marking portion and a second marking portion is shown;
[0031] Figure 2B A cross-section of a second example of a passive marker, including a third marker portion, is shown;
[0032] Figure 2C A cross-section of a third example of a passive marker is shown, wherein the first marker portion is spatially offset relative to the second marker portion;
[0033] Figure 2D A perspective view showing a first example of a printed tracker;
[0034] Figure 2E A perspective view of a second example of a printed tracker is shown;
[0035] Figure 2F An example of an active marking having an LED electrically connected to the circuit via solder joints is shown;
[0036] Figure 3 A flowchart is shown showing a method for determining the location of markers that define the tracker;
[0037] Figure 4A An example of a tracker with four markers is shown, which are arranged in a predetermined marker layout.
[0038] Figure 4B This shows a representation of the first image data marked by the tracker;
[0039] Figure 4C An example of the second image data marked by the tracker is shown;
[0040] Figure 4D The first marker arrangement determined based on the first image data is shown;
[0041] Figure 4E The second marker arrangement determined based on the second image data is shown;
[0042] Figure 4F The combination of the first and second mark arrangements is shown;
[0043] Figure 5A The diagram shows the first position of the first marker arrangement and the second position of the second marker arrangement;
[0044] Figure 5B The marked position is shown as the average position between the first and second positions;
[0045] Figure 5C The marked positions are shown, which are determined as the weighted average position between the first and second positions;
[0046] Figure 5D This shows the different marker positions resulting from different weighting standards;
[0047] Figure 6A Examples of a first marker arrangement and a second marker arrangement are shown, wherein the first position of a specific marker at the first position is invalid;
[0048] Figure 6B It shows the basis Figure 6A The marking arrangement determined by the first and second marking arrangements shown;
[0049] Figure 6CExamples of a first marker arrangement and a second marker arrangement are shown, where the first position of a particular marker is invalid;
[0050] Figure 6D It shows the basis Figure 6C The marking arrangement determined by the first and second marking arrangements shown;
[0051] Figure 7 Examples of the first and second marker arrangements are shown, where the location distance threshold is exceeded;
[0052] Figure 8A An example of a first arrangement of markers and unmarked objects determined based on first image data is shown;
[0053] Figure 8B An example of a second arrangement without markers determined based on the second image data is shown;
[0054] Figure 8C It shows Figure 8A The first arrangement shown is Figure 8B The second arrangement shown is aligned;
[0055] Figure 8D It shows Figure 8C The first and second positions are shown, where redundant first positions are ignored;
[0056] Figure 9A An example of a tracker with four markers is shown, arranged in a predetermined marker layout, similar to... Figure 4A ;
[0057] Figure 9B It shows that the capture has Figure 9A The first image data of the marker shown;
[0058] Figure 9C It shows that the capture has Figure 9A The second image data marked as shown;
[0059] Figure 9D The image data shown includes a combination of first image data and second image data;
[0060] Figure 9E It shows that according to Figure 9D The arrangement of markers determined by the combined image data shown;
[0061] Figure 10A An example of a tracker with four markers is shown, where the markers are arranged in a common plane;
[0062] Figure 10B The results determined by any of the methods described herein are shown. Figure 10AThe marking arrangement of the tracker is shown;
[0063] Figure 10C It shows Figure 10B A side view of the markings arrangement;
[0064] Figure 10D The defined marker arrangement is shown, wherein the marker arrangement is located on a virtual plane;
[0065] Figure 11A The undeformed tracker substrate is shown, on which eight marks are formed;
[0066] Figure 11B It shows in Figure 11A The predetermined marking arrangement when the base is undeformed;
[0067] Figure 11C This illustrates the deformed shape of the substrate due to its adaptation to the patient's skin surface;
[0068] Figure 11D It shows the capture based on Figure 11C The marker arrangement is determined by the first image data and the captured second image data of the markers supported by the deformable substrate shown; and
[0069] Figure 11E A predetermined marking arrangement is shown in combination with a deformed, predetermined marking arrangement. Detailed Implementation
[0070] Figure 1 A surgical tracking and navigation system 10 is shown, including a marking arrangement determination device 12 according to the present disclosure. Device 12 is implemented as a local computer system in the operating room. Alternatively, device 12 may be provided at least in part by a remote server or cloud computing resources.
[0071] The surgical tracking and navigation system 10 also includes a tracking system 14 with cameras 16. Optionally, the tracking system 14 may include multiple cameras 16, for example, in the form of individually configured cameras or stereo cameras. The tracking system 14 may also include a processing component (not shown). Such a processing component may be shared with the marker placement determination device 12 (e.g., in the form of a local computer system). In some variations, the marker placement determination device 12 (e.g., in the form of an integrated software product) is integrated into the tracking system 16.
[0072] At least one of the tracking system 14 and the marker arrangement determination device 12 may include an output device 18 or be communicatively connected to the output device 18. Figure 1The exemplary output device 18 shown includes a display (e.g., a computer monitor or tablet computer). Optionally or additionally, the output device may include at least one of a speaker, virtual reality glasses, glasses with a heads-up display, and a haptic feedback device.
[0073] The surgical tracking and navigation system 10 also includes a tracker 20 as a component of the tracking system 14. The tracker 20 is typically configured to be attached to a surgical object, such as a patient 26. In this implementation, the tracker 20 typically supports multiple markers 22. Each marker 22 can be an electromagnetically passive or active element. For example, at least one of the markers 22 can be a passive element configured to reflect electromagnetic radiation, such as a flat reflective surface, a sphere, or a reflective printed surface. Additionally or alternatively, at least one of the markers 22 can be an active element capable of emitting electromagnetic radiation, such as a light-emitting diode (LED) or an incandescent light bulb (any of which can be coupled to an optical fiber). The camera system 14 is capable of detecting electromagnetic radiation reflected or emitted by the markers 22.
[0074] Tracker 20 is placed on patient 26 (e.g., as Figure 1 The patient's back (as shown) or the surface of any surgical instrument or other surgical object. For this purpose, the tracker 20 includes an interface configured to attach to the surgical object. The interface may include at least one of adhesive, clamp, screw, and magnet. Figure 1 The tracker 20 shown includes an adhesive interface that allows the tracker to be adhesively attached to the skin of the patient's back. However, the tracker 20 can be attached to any other surface of the patient 26, such as the face or limbs. Furthermore, the tracker 20 can be positioned on or attached to any other surgical object, such as surgical instruments, surgical frames, tracking system components, or imaging devices. Figure 1 The tracker 20 shown is a single device. Alternatively, the tracker 20 may include separate tracking portions capable of being individually attached to the surgical object, each tracking portion supporting at least one marker. These separate tracking portions would then collectively form the tracker 20.
[0075] Marker arrangement 23 arranges marker 22 according to the (e.g., relative) position of the defined marker 22. Figure 1The tracker 20 shown has a flexible base 24 supporting the markers 22. The base 24 can be at least one of compressible and stretchable. Therefore, the base 24 adapts to the skin surface of the patient 26 and thus presents a deformable state. As a result, the tracking system 14 may not know in advance the marker arrangement 23 in the deformed state of the base 24 before or during surgery. Determining the deformed state of the marker arrangement 23 is necessary for the tracking system 14 to track the tracker 20 (e.g., to reliably identify the marker arrangement 23 in image data captured by the camera 16) and for surgical navigation.
[0076] like Figure 1 The flexible substrate 24 shown is one of many possible reasons why the marking arrangement 23 may not have been known beforehand. In other cases, the tracker 20 may have a rigid substrate, but the substrate can be bent (e.g., intentionally by the user or unintentionally due to wear or impact). In further cases, the tracker 20 may have been manufactured with large tolerances (e.g., using additive manufacturing techniques such as 3D printing). In the case of a customizable tracker 20 with multiple movable, attachable, or detachable tracking parts, the marking arrangement 23 can be arranged in a customized manner by a surgeon prior to surgery.
[0077] The marker arrangement 23 can be determined solely based on first image data captured using a first imaging mode such as camera system 14. However, determining the marker arrangement 23 based solely on the first image data may not be accurate enough for surgical navigation purposes (e.g., in cases of insufficient spatial resolution or obstacles). Thus, a more accurate technique for determining the marker arrangement 23 by marker arrangement determining device 12 has been proposed.
[0078] Image data can be captured from tracker 20 using two different imaging modes. For example, in the case of a CT scan of patient 26, CT scanner 15 is used as the second imaging mode to capture second image data of patient 26 with tracker 20, while camera system 14, as the first imaging mode, captures first image data of tracker 20. The marker 22 of tracker 20 can be identified in the first and second image data by marker arrangement determining device 12, which, for example, allows the second image data to be registered with tracker 20 (and thus patient 26) in the coordinate system of tracking system 14.
[0079] In order to enable the identification of marker 22 in both imaging modes, marker 22 is configured to be detectable by both imaging modes. Various possibilities exist in this regard.
[0080] Figure 2AA cross-section of a first example of a passive marker 22 is shown, wherein a first marker portion 25 can be detected by a first imaging mode (here: camera 16), and a second marker portion 27 can be detected by a second imaging mode (here: CT scanner 15). Figure 2A In the example shown, the first marking portion 25 is a shell, and the second marking portion 27 is a sphere concentrically arranged relative to the shell of the first marking portion. Therefore, the positions determined for the marking 22 in the first image data and the second image data (e.g., in terms of their respective centers) are at least substantially the same.
[0081] Figure 2B A cross-section of a second example of a passive marker 22 is shown, which includes a third marker portion 21 comprising material detectable by both imaging modes (here: detectable by camera 16 and CT scanner 15). Therefore, the location of marker 22 determined in the first and second image data is at least substantially the same.
[0082] Figure 2C A cross-section of a third example of a passive marker 22 is shown, wherein the first marker portion 25 is spatially offset relative to the second marker portion 27. Figure 2C In the example shown, the first marking portion 27 is a sphere, which is screwed onto the spiral-shaped second marking portion 27. Figure 2A Conversely, the center position 29 of the first marking portion 25 does not coincide with the center position 31 of the second marking portion 27. However, the geometric relationship (e.g., in terms of vector or spatial translation between positions 29 and 31) can be predetermined, meaning it is known a priori before the surgery. Therefore, the position of the second marking portion 27 can be determined using the position of the first marking portion 25 and the predetermined geometric relationship (and vice versa).
[0083] Figure 2D A perspective view of a first example of a printed tracker 20 is shown. The tracker 20 has printed markings 22 detectable by a first imaging mode and a second imaging mode. For example, the markings 22 can be printed with ink comprising infrared-reflective pigments detectable by the camera system 16 and metallic particles visible in CT scans.
[0084] Figure 2E A perspective view of a second example of a printed tracker 20 is shown. The tracker 20 has a marker 22 having a first marker portion 25 detectable by a first imaging mode and a second marker portion 27 detectable by a second imaging system. The second marker portion 27 may be printed or include an attachment element (e.g., a screw, nail, or rivet).
[0085] Figure 2FAn example of an active marker 22 is shown, which has an LED 33 electrically connected to a circuit 35 via at least one (here: two) solder points 37. The marker 22 includes a first marker portion 25 in the form of the LED 33, as the LED can be detected by a first imaging mode in the form of a camera 16. The marker 22 also includes two second marker portions 27 in the form of solder points 37, which can be detected by a second imaging mode in the form of a CT scanner 15. The geometric relationship between the first marker portion 25 and the two second marker portions 27 can be predetermined.
[0086] from Figure 2E and 2F As can be seen in the example shown, each mark 22 does not necessarily include only one mark portion 25, 27. Instead, a mark may include more than one first mark portion 25 and / or more than one second mark portion 27.
[0087] Figure 3 A flowchart 100 is shown for a method of determining a marker arrangement 23 that defines the position of markers 22 on tracker 20. Figure 1 The marking arrangement determination device 12 is configured to perform any of the method steps described herein. For this purpose, device 12 may include at least one processor, wherein a computer program product executed by the at least one processor causes the at least one processor to perform any of the method steps described herein.
[0088] The method includes receiving first image data of the marker 22 captured using a first imaging mode in step 102. For this purpose, the marker 22 is detectable by the first imaging mode. As described above, the first imaging mode can be a camera 16. Therefore, the camera 16 can be used both to track the tracker 20 with the marker 22 and to image the marker arrangement 23 during surgery before actual tracking. Alternatively, different cameras or camera systems can be used to capture the first image data for calibration purposes (i.e., accurately determining the marker arrangement 23) and to track the tracker 20. The camera or camera system used to capture the first image data can have high spatial resolution but does not necessarily need to meet the additional requirements for tracking (e.g., high frame rate).
[0089] The first image data may represent one or more images or one or more videos of marker 22. At least one of the images or videos may be captured from different perspectives. For example, the first image data may be captured by a stereo camera or by a single camera that moves relative to tracker 20.
[0090] The method includes receiving, in step 104, second image data of the tag captured using a second imaging mode. The second imaging mode differs from the first imaging mode. The tag 22 can also be detected by the second imaging mode. Figure 1In this example, the first imaging mode includes a CT scanner 15. The second imaging mode may alternatively include a magnetic resonance imaging (MRI) device. Steps 102 and 104 can be performed in any order and may also be performed in parallel.
[0091] The method further includes determining the marker arrangement based on the first image data and the second image data in step 106. During the determination of the marker arrangement 23 based on the first image data and the second image data, the position of the marker 22 is determined in at least one of the first image data, the second image data, or a combination of image data derived from the first and second image data. The position (e.g., coordinates) of a specific marker 22 can be determined in any image data using image processing techniques, which may be based on image points with the highest intensity (e.g., pixels), image points with high contrast or large intensity variations, or image points at the geometric center of a plurality of such image points.
[0092] Determining the marker position may include spatially offsetting the marker position relative to the image points determined above. The spatial offset may be determined based on the geometric relationship between the first marker portion 25 and the second marker portion 27 (see...). Figure 2C , 2E (and 2F). When one of the first and second modes is used for subsequent tracking, the position determined in the image data using the other imaging mode can be spatially offset toward the marked portion of the tracking imaging mode.
[0093] For example, Figure 2F The first image data of the marker 22 shown can be captured by camera 16, and the second image data can be captured by CT scanner 15. Image points of the second marker portion 27 (i.e., solder joint 37) can be located within the second image data and subsequently spatially offset using known (i.e., predetermined) geometric relationships to align with the position of the first marker portion 25 (e.g., LED 33). Therefore, the position of the first marker portion 25 (i.e., LED 33) can be located within both the first and second image data. Using the position of the first marker portion 25 to determine the marker arrangement can better represent the position of LED 33, thus providing greater accuracy for subsequent tracking.
[0094] Similarly, geometric relationships can be used to offset the position of the first marker portion 25 (e.g., LED 33) determined based on image points in the first image data (captured by camera 16) to align with the image points of the second marker portion 27 (i.e., solder joint 37). Therefore, the position of the second marker portion 27 can lie within both the first and second image data. Using the position of the second marker portion 27 to determine the marker arrangement can more accurately represent the position of the solder joint 27, thereby more accurately registering the second image data with the tracker 20.
[0095] The determination of the marker arrangement 23 can be performed according to either the first or second method, which will now be described in more detail. Of course, other methods can also be used. According to the first method, the method further includes determining a first arrangement of defining a first position of the marker 22 based on first image data, and a second arrangement of determining a second position of the marker 22 based on second image data. In this case, the marker arrangement 23 is determined based on a combination of the first and second arrangements. Reference will now be made to... Figures 4A to 4E Explain the method.
[0096] Figure 4A An example of a tracker 20 with four markers 22 is shown, which are arranged in a given marker arrangement 23 for use during surgery. The tracker 20 may have any other number of markers 22, such as two, three, five or more. Figure 4A The markers 22 are intentionally arranged at random locations for better representation (e.g., generated by the deformed marker base 24, see...). Figure 1 Unknown marker arrangement 23.
[0097] Figure 4B A representation of the first image data 28 is shown. The first image data 28 can be derived from one or more images of the marker 22 (e.g., captured by camera 16). The first image data 28 may include data indicating the capture intensity or contrast of the marker 22, such as... Figure 4B As shown. Thus, the marker 22 represented in the first image data 28 can have spatial expansion.
[0098] Figure 4C A representation of the second image data 30 (e.g., captured by CT scanner 15) is shown. The second image data 30 can be defined similarly to the first image data 28. The second image data 30 can be derived from at least one of an X-ray projection through marker 22, a cross-sectional image slice of marker 22, and a volume drawing of marker 22.
[0099] Figure 4D A first marker arrangement 32, determined based on first image data 28, is shown. The first marker arrangement 32 (or any marker arrangement described herein) can be defined in a way that directly or indirectly defines the position of the markers 22 relative to each other. For example, the first marker arrangement 32 can define the marker positions in a common coordinate system. Alternatively or additionally, the first marker arrangement 32 can define the position of the markers 22 relative to other markers 22. Figure 4D In the example shown, the first marker arrangement 32 defines the position of marker 22 in a common coordinate system. If marker 22 represented in the first image data 28 has a spatial extension, image processing techniques can be applied to derive the specific marker coordinates, as explained above.
[0100] Figure 4E A second marker arrangement 34, determined based on second image data 30, is shown. The second marker arrangement 34 can be defined in a manner similar to or different from the first marker arrangement 32. Figure 4E In the example shown, the second marker arrangement 34 defines the position of marker 22 relative to the other markers 22.
[0101] Figure 4F The diagram illustrates a combination of a first marker arrangement 32 (in the form of a square icon) and a second marker arrangement 34 (in the form of a triangle) in a common coordinate system. Although the positions of each marker 22 in the first marker arrangement 32 and the second marker arrangement 34 are similar, their positions are not identical because they were determined from different image data (and may have different spatial resolutions, different angles, etc.). [The last sentence appears to be incomplete and possibly refers to a different topic.] Figure 3 In step 106, the mark arrangement 32 of interest is determined in a different manner based on the first mark arrangement 32 and the second mark arrangement 34, as will be described below.
[0102] Figure 5A The diagram illustrates a first position 32A or a first position 32A within a first marker arrangement 32 of an exemplary marker 22, and a second position 34A or a second position 34A within a second marker arrangement 34B. While square and triangular shapes allow for differentiation between the first marker arrangement 32 and the second marker arrangement 34, the actual position (e.g., actual coordinates) is represented by a circle at the center of each of the two shapes.
[0103] Figure 5B It shows in Figure 3 The position 23A of the first mark in step 106 of the mark arrangement 23 is taken as the average position between the first position 32A and the second position 34A. The determined (average) position 23A of the first mark can be determined by defining a connecting line 36 between the first position 32A and the second position 34A and positioning the determined position at a distance equal to that between the first position 32A and the second position 34A on the connecting line 36.
[0104] Figure 5C It shows in Figure 3 The position 23A of the first mark in the mark arrangement 23 determined in step 106 is used as the weighted average position between the first position 32A and the second position 34A. Therefore, one of the first position 32A and the second position 34A is assigned a larger weight, so that the position with the larger weight contributes more to the weighted average. Thus, the determined (here, the weighted average) position is closer to the position with the larger weight. Figure 5C In the example shown, the second position 34A is assigned a larger weight. Therefore, the position 23A of the first marker is closer to the second position 34A than the first position 32A.
[0105] The weights assigned to the first position 32A and the second position 34A can be fixed, such as 60:40, 3:1, 75:25, or any other ratio. A larger weight can be assigned to the first marker arrangement 32 or the second marker arrangement 34 (i.e., the first imaging mode or the second imaging mode) that has greater spatial resolution or more reliable imaging quality. For example, the second marker arrangement 34 determined based on image data from the CT scanner 15 can be less dependent on the viewing angle and therefore more reliable than the first marker arrangement 32 determined, for example, based on image data from the camera 16. In this case, the second marker arrangement 34 can be assigned a larger weight than the first marker arrangement 32.
[0106] The weights assigned to the first position 32A and the second position 34A can depend on one or more weighting criteria. Therefore, the position 23A of the first marker can also depend on one or more weighting criteria. Figure 5D The diagram illustrates different positions 23A of the first markers in the marker arrangement 23 due to different weighting criteria. One or more weighting criteria may be based on at least one of viewing angle and line-of-sight. For example, due to the limited spatial resolution of any imaging mode, first image data 28 captured at a greater line-of-sight may result in a first marker arrangement 32 with lower precision. Figure 5D In the example shown, the first imaging mode includes camera 16, and the second imaging mode includes CT scanner 15. Therefore, the accuracy of the first position 32A can depend on the viewing angle and distance of the tracker 20 relative to camera 16. At short viewing distances and / or optimal viewing angles (e.g., marker 22 is arranged at least substantially in a plane perpendicular to the optical axis of camera 16), the first image data can produce a first position 32A with high accuracy. In this case, the first position 32A can be assigned a greater weight than the second position 34A. In this case, Figure 5D The position 23A of the first marker shown is arranged close to the first position 32A, for example, the leftmost first position 23A. For the first image data 28 captured at a larger viewing distance and / or a less than ideal viewing angle, a smaller weight can be assigned to the first position 32A, thereby producing Figure 5D The first marker position 23A of the center shown, or even the first position 32A, can be assigned a smaller weight to produce Figure 5D The first marker position 23A on the right is shown. Therefore, at least one of the first image data 28 and the second image data 30 may include at least one of the viewing angle and viewing distance information evaluated by the determining device 12.
[0107] One or more weighting criteria may additionally or alternatively be based on at least one of the time series and time offset between capturing the first image data 28 and the second image data 30. During the time between capturing the first image data 28 and the second image data 30, the marker 22 may move relative to each other (e.g., due to...). Figure 1 The movement of the patient 26 shown with the deformable tracker 20 (due to operation of the tracker 20 or unintentional bending due to temperature changes) is illustrated. In this case, the most recently captured image data more accurately represents the actual marker arrangement 23 of interest. Therefore, the first marker arrangement 32 or the second marker arrangement 34 determined based on the most recently captured image data 28, 30 can be assigned a larger weight. The weights assigned to the first position 32A or the second position 34A determined based on the most recently captured image data 28, 30 can be scaled over time (e.g., linearly or logarithmically).
[0108] The position 23A of the first mark 22 of tracker 22 can be determined, for example, by combining the first position 32A and the second position 34A in the manner described above. The positions of the other marks 22 of tracker 20 can be determined in the same manner. As a result, the position of each mark 22 of tracker 20 in the mark arrangement 23 can be determined. Therefore, the entire mark arrangement 23 defining the positions of all marks 22 of tracker 20 can be determined. As a result, tracker arrangement 23 is calibrated based on the first image data and the second image data.
[0109] The method proposed herein may include determining invalid locations of a specific marker 22 of the tracker 20 in one of the first arrangement 32 or the second arrangement 34. If the location cannot be determined at all (e.g., due to insufficient information about the specific marker 22 in the first image data 28 or the second image data 30) or if an irregular location is identified, the location of the specific marker 22 may be determined as invalid in one of the arrangements 32, 34. If the location is outside a distance threshold relative to a desired location or relative to other markers 22 (e.g., due to patient movement), the location may be identified as irregular.
[0110] Figure 6A Examples of a first marker arrangement 32 and a second marker arrangement 34 are shown, wherein a first position 32B of a specific marker 22 in the first marker arrangement 32 is invalid because it cannot be determined (e.g., because the specific marker 22 was not captured in the first image data 28, or the first image data 28 does not contain enough information to identify the specific marker 22). The fact that the first position 32B cannot be determined is... Figure 6A The square with dashed lines is used to represent this. Similarly, as shown... Figure 6A As shown, the second position 34B of a specific mark 22 can be determined in the second mark arrangement 34.
[0111] Figure 6B It shows in Figure 3 In step 106, based on Figure 6A The first mark arrangement 32 and the second mark arrangement 34 shown define the mark arrangement 23 (represented by a crosshair). Figure 6B In the example shown, the marker arrangement 23 is determined based on the average position (see...). Figures 5B to 5D However, any other method for determining the marker arrangement 32 described herein may be used instead. While the position of most markers 22 is determined to be the average position between the first marker arrangement 32 and the second marker arrangement 34 (as indicated by the cross icon arranged between the center of the square icon and the center of the triangle icon), the position of a particular marker 22 is placed at the center of a second position 34B in the second marker arrangement 34. That is, instead of determining the average position of a particular marker 22, since corresponding position information is lacking in the first arrangement 32, the second position 34B in the second marker arrangement 34 is selected as marker position 23B in the marker arrangement 23.
[0112] Figure 6C An example of a first marker arrangement 32 and a second marker arrangement 34 is shown, wherein a first position 32B of a particular marker 22 is invalid because it is located at a position exceeding a distance threshold from other first positions in the first marker arrangement 32. Alternatively or additionally, other criteria for determining invalid positions may be used, such as distance from an expected position or distance between the positions of the dedicated marker 22 in the first marker arrangement 32 and the second marker arrangement 34.
[0113] Figure 6D It shows the basis Figure 6C The marking arrangement 23 is determined by the first marking arrangement 32 and the second marking arrangement 34 shown. (And...) Figure 6B The example shown is similar, with the position of a specific mark 22 being arranged at the center of the corresponding second position 34B of the second mark arrangement 34, thereby ignoring the invalid first position 32B of the first mark arrangement 32.
[0114] The method proposed herein may include generating an error signal when a quality criterion between the positions of marker 22 in the first arrangement 32 and the second arrangement 34 meets an error condition. The error condition may include identifying invalid positions. The error signal may notify the user of the invalid position of the dedicated marker 22, indicating that... Figure 3 The position of the dedicated marker 22 is less accurate than the (e.g., average) position of other markers 22 in the marker arrangement 23 determined in step 106. Alternatively or additionally, the error signal may request the user to obtain new image data or reposition the tracker 20.
[0115] In other cases, the error condition may include the inability to determine both the first position of the first mark arrangement 32 and the second position of the second mark arrangement 34 for the dedicated mark 22. In this case, the position of the dedicated mark 22 of the mark arrangement 23 cannot be determined.
[0116] In other cases, the error condition may include a position distance threshold, wherein for marker 22, the distance between the first position of the first marker arrangement 32 and the second position of the second marker arrangement 34 does not exceed the position distance threshold. Figure 7 An example of a first marker arrangement 32 and a second marker arrangement 34 is shown, where a positional distance threshold is exceeded. Most positions of the first marker arrangement 32 and the second marker arrangement 34 substantially overlap, but for one of the markers 22, a first position 32B of the first marker arrangement 32 and a second position 34B of the second marker arrangement 34 are spaced apart at a relatively large positional distance 38. Figure 7 In the example shown, the location distance of 38 exceeds the location distance threshold, and an error signal will be generated.
[0117] It should be noted that error conditions may include identifying invalid locations, but error conditions can be detected even when invalid locations are not identified. Figure 7 In the example shown, the first position 32B and the second position 34B can be located sufficiently close to the expected position, thus both being valid positions. However, when viewed relative to each other, the first position 32B and the second position 34B are positioned at a distance 38 beyond the position distance threshold. Therefore, generating an error signal can depend on identifying invalid positions, but it can also be independent of identifying invalid positions.
[0118] The aforementioned invalid locations relate to image points in the image data that actually originate from marker 22 of tracker 20. Therefore, the method proposed in this paper can still attempt, for example, to determine the marker location by omitting an invalid location from one of the first and second locations and using the non-invalid location of the other, such as... Figure 6B and 6D As shown.
[0119] In some implementations, the first image data and / or the second image data may include image points of an unmarked surface that could be mistakenly identified as a first or second location. For example, camera system 14 may capture first image data of patient 26 using tracker 20, where reflected light spots from surrounding devices (e.g., operating table or surgical instruments) cause optical signals similar to the optical signals of marker 22. Due to the similarity, the first location can be determined based on the unmarked surface in the form of reflected light spots. As understood herein, such unmarked surfaces may also include markers from a tracker different from tracker 20.
[0120] Figure 8A An example of a first arrangement 32 of markers 22 and unmarked objects, determined based on first image data captured by camera system 14, is shown. In this example, tracker 20 (not shown) has five markers 22, but the first marker arrangement 32 includes a total of nine first locations because four reflected light spots (or other unmarked objects) are incorrectly identified as first locations.
[0121] Since the second image data is captured by an imaging mode different from the first image data, unmarked locations can be avoided (or at least less so) in the second image data. For example, the CT scanner 15, as the second imaging mode, typically generates image data based on material density rather than light signals (such as reflection). As a result, surfaces that cause reflected light spots are less likely to be confused with the marker 22 in the second image data generated by the CT scanner 15. The second locations determined based on the second image data include fewer or no unmarked locations, so suitable algorithms can be used to filter out the unmarked first locations. Figure 8B An example of a second arrangement 34 determined based on second image data from a CT scanner 15 is shown. Figure 8B In the example shown, the unmarked second location was not determined. As a result, all determined second locations are associated with mark 22 of tracker 20.
[0122] The initial steps in preparing for actual filtering include matching (e.g., aligning) a first position of the first marker arrangement 32 with a second position of the second arrangement 34. To do this, at least one of the first marker arrangement 32 and the second marker arrangement 34 may be rearranged relative to the other marker arrangement 32, 34 (e.g., rotated, translated, and scaled at least once), for example, until the difference between the first marker arrangement 32 and the second marker arrangement 34 is minimized. Minimizing the difference may include at least one of minimizing the distance between the first and second positions, maximizing the number of overlapping first and second positions, and minimizing the rescaling of the first arrangement relative to the second arrangement. Matching the first and second positions may include using a point set algorithm, such as an algorithm for minimizing the difference between the first and second positions, such as the Iterative Closest Point (ICP) algorithm.
[0123] Figure 8C It shows Figure 8A The first arrangement shown is Figure 8B The final alignment of the second arrangement shown. Note that... Figures 8A to 8C The first marker arrangement 32 and the second marker arrangement 34 shown are arranged in a plane to more clearly convey alignment. It should be understood that alignment can also be performed using three-dimensional marker arrangements 32, 34.
[0124] like Figure 8CAs can be seen, the five first positions can be matched or aligned with their corresponding second positions. However, the four "redundant" first positions 41 lack corresponding second positions to be aligned with because the "redundant" first positions 41 are the result of a misinterpretation of the first image data (and this misinterpretation did not occur when determining the second positions).
[0125] Therefore, "redundant" first positions 41 can be ignored (i.e., filtered out) from the first marking arrangement 32. As a result, the risk of the first arrangement having first positions that are not marked is reduced. Figure 8D It shows Figure 8C The first and second positions are shown, where the "redundant" first position 41 is ignored.
[0126] exist Figures 8A to 8D In the example shown, only the first location of the first marking arrangement 32 is ignored. Alternatively, one or more second locations of the second arrangement 34 may be ignored (e.g., to eliminate unmarked areas identified based on implants captured in CT scans). Further alternatively, one or more first locations and one or more second locations may be ignored to eliminate unmarked areas from both marking arrangements 32, 34.
[0127] Subsequently, any “redundant” first and second positions may be ignored in other steps described herein, for example, when determining the marking arrangement 23 based on the first marking arrangement 32 and the second marking arrangement 34. Ignoring “redundant” positions may include at least one of the scaling factors for ignoring, removing, and zeroing out redundant positions.
[0128] As described above, determine as follows Figures 4A to 4E The illustrated marker arrangement 23 includes determining a first marker arrangement 32 based on first image data 28 and determining a second marker arrangement 34 based on second image data 30. Another method includes determining combined image data based on the first and second image data, wherein the marker arrangement 23 is subsequently determined based on the combined image data.
[0129] Figure 9A An example of tracker 20 is shown, which has a marker arrangement 23 defining the position of marker 22, similar to Figure 4A Tracker 20 in the middle. Figure 9B The first image data 28 of the captured marker 22 is shown. Figure 9C The second image data 30 of the captured marker 22 is shown. Figures 9A to 9C Basically corresponds to Figures 4A to 4C .
[0130] Figure 9DA combined image data 40 is shown, comprising first image data 28 and second image data 30. Combining the first and second image data 28 may include identifying corresponding markers 22 in each image data set 28, 30, and minimizing the sum of distances between relevant pairs of identified markers 22. The markers 22 may be identified based on at least one of contrast, pixel intensity, a preliminary estimate of the distance between markers 22, and a preliminary estimate of the marker arrangement. Image processing algorithms may be used for this purpose.
[0131] Alternatively or additionally, the first image data 28 and the second image data 30 may be combined based on a known spatial relationship between the first and second imaging devices (e.g., camera system 14 and CT scanner 15 or MRT device) associated with the first and second imaging modes. The spatial relationship can be predetermined or acquired by using one of the first and second imaging modes to track the other. For example, the first imaging mode may include camera system 14, and the second imaging mode may include CT scanner 15, wherein CT scanner 15 includes a tracker (e.g., a frame attached to CT scanner 15) that can be tracked by camera system 14.
[0132] Figure 9E It shows that according to Figure 9D The marker arrangement 23 is determined by the combined image data shown. Therefore, in the intermediate steps, it is not necessary to use... Figures 4A to 4F The method shown determines a dedicated first and second marker arrangement based on image data.
[0133] Figure 10A An example of tracker 20 is shown, which can be used with any of the techniques presented herein. Tracker 20 includes (e.g., rigidly arranged) four markers 22 in a common plane. Figure 10B The results determined by any of the methods described herein are shown. Figure 10A The marker arrangement 23 of the tracker 20 shown.
[0134] Figure 10C It shows Figure 10B The side view of the marking arrangement 23 in the image. As can be seen (exaggerated for visibility), due to the limited precision of the first image data 28 and the second image data 30, with Figure 10A The actual marker 22 of the tracker 23 shown is different, and the marker positions in the arrangement 23 are not precisely located in the common plane.
[0135] therefore, Figure 3 Step 106 may further include determining a virtual plane 42 based on at least one of the first image data 28 and the second image data 30. For example, the closest marker arrangement 23 (or such) may be determined. Figure 4D and 4E The virtual plane 42 is the location of the mark 22 (either of the first mark arrangement 32 and the second mark arrangement 34 shown). Therefore, in Figure 3 The marker arrangement 23 initially determined in step 106 can be regarded as a preliminary entity that will be further improved.
[0136] Determining the virtual plane 42 may include determining the shortest distance relative to the positions of the marker arrangement 23 (or either the first marker arrangement 32 or the second marker arrangement 34), for example by minimizing the sum or square of the distances between the positions relative to the virtual plane 42. Alternatively or additionally, a connection vector between the positions and the normal vector may be determined, the normal vector having the minimum scalar product between the connection vector and the normal vector, wherein the virtual plane 42 is perpendicular to the determined normal vector arrangement.
[0137] Then, the improved marker arrangement 23 can further include virtually arranging the markers in "preliminary" positions (e.g., in the virtual plane 42) in the virtual plane 42. Figure 10C (As shown). For example, by... Figure 10C The positions shown are projected onto plane 42, and these positions can be arranged in the virtual plane 42. Alternatively, the positions can be rotated onto plane 42 about a center point located at the geometric center of the determined positions. Figure 10D An improved marker arrangement 23 is shown, wherein the marker arrangement 23 is located on a virtual plane 42.
[0138] exist Figures 10A to 10D In the example shown, a virtual plane 42 is determined based on the initial position of the marker arrangement 23, and then the improved position is determined by arranging the initial position on the virtual plane 42. Alternatively, the virtual plane 42 can be determined directly based on at least one of the first image data 28 and the second image data 30, without needing to determine the initial position of the marker arrangement 23.
[0139] When in Figure 3 When determining the marker arrangement 23 in step 106, knowledge of the predetermined (e.g., planar) arrangement of the markers 22 on the undeformed tracker base 24 can also be used. Figure 11A A second example of a tracker 20 with a deformable base 24 supporting multiple markers 22 is shown, but the following observations also apply. Figure 1 An exemplary tracker 20 or any similar tracker. The undeformed base 24 can be flat, such as... Figure 11A As shown. Alternatively, the undeformed base 24 may not be flat but pre-deformed, for example, when the base 24 conforms to the standard contours of a body part (e.g., a standard facial shape).
[0140] Figure 11B A predetermined marker arrangement 44 is shown, (in the form of a crosshair icon) defining predetermined positions 46 of marker 22. The predetermined marker arrangement 44 may further define (e.g., pairs) predetermined distances 48 between predetermined positions 46.
[0141] Figure 11C The base 24 is shown in a deformed shape as it adapts to or conforms to the patient’s skin surface (in this case, the area of the nose and forehead). Figure 11D It shows that based on the Figure 11C The preliminary marker arrangement 23, determined by the first and second image data captured from the marker 22 supported by the deformable substrate 24 shown (see also...) Figure 3 Step 106 in the process.
[0142] The initially determined mark arrangement 23 can be improved based on prior geometric knowledge about the predetermined mark arrangement 44, see [link to relevant documentation]. Figure 11B To this end, the predetermined marking arrangement 44 can be virtually deformed (e.g., by bending, stretching, and compressing at least one of these) to fit the initial marking arrangement 23. For this purpose, the predetermined marking arrangement 44 can be deformed randomly or in a predetermined pattern, while aligning the positions of the markings on the predetermined marking arrangement 44 with the positions of the markings on the initial marking arrangement 23. The predetermined marking arrangement can be deformed while optimizing target parameters (e.g., minimizing stretching between predetermined positions 46 and minimizing distortion of the predetermined marking arrangement 44). The deformed predetermined marking arrangement can then be used to identify the markings 22 of the initial marking arrangement 23.
[0143] Figure 11E The combination of a preliminary marking arrangement 23 and a predetermined marking arrangement 44 for deformation is shown. The predetermined marking arrangement 44 makes it easier to identify each marking 22 and resolves ambiguities between two closely arranged positions of marking 22. Alternatively or additionally, in the case of a flexible but non-stretchable substrate 24, prior geometric knowledge about the predetermined marking arrangement 44 can provide the maximum distance between two marking positions. Prior geometric knowledge can be used in... Figure 3 In step 106, an improved version of the initial marking arrangement 23 is determined.
[0144] The method may also include using, for example Figure 3In step 106, the marker arrangement 23 determines at least one of the first image data 28 and the second image data 30 to register with the tracker 20. For example, the second image data 30 may be a CT scan registered with the tracker 20 using the marker arrangement 23. After registration, the tracking system 14 can use the knowledge about the marker arrangement 23 to track the tracker 20. Based on the obtained tracking information, it can be displayed on the display 18 (see...) Figure 1 The registered CT scan can be oriented on the display 18 for navigation or other purposes. In addition, the registered CT scan can be visualized on the display 18 relative to surgical instruments (such as screwdrivers, drills, or files) equipped with trackers 20.
[0145] The technique proposed in this paper for determining the marker arrangement 23 allows for determination based on first image data 28 and second image data 30, and therefore on redundant information sources. As a result, the accuracy of the determined marker arrangement 23 is improved or calibrated. Therefore, tracking, registration, and navigation are less prone to positional or orientation errors. In other words, the accuracy of subsequent procedures using the marker arrangement 23 is improved, such as tracking, tracker calibration and verification of tracker type or size, image registration, tracker registration, outputting navigation commands to surgeons or surgical robots, etc.
[0146] The features described with respect to the exemplary embodiments shown in the accompanying drawings can be readily combined to produce different embodiments. Therefore, it will be apparent that this disclosure can be modified in various ways. Such modifications should not be considered as a departure from the scope defined by the appended claims.
Claims
1. A method for determining a marker arrangement, the marker arrangement defining the positions of markers on a tracker, wherein, The marker can be detected by a first imaging mode and a second imaging mode different from the first imaging mode, the method comprising: Receive first image data of the marker captured using the first imaging mode; Receive second image data of the marker captured using the second imaging mode; The marker arrangement is determined based on the first image data and the second image data; Based on the first image data, a first arrangement defining the first position of the marker is determined; and Based on the second image data, a second arrangement is determined that defines the second location of the mark. The marker arrangement is determined based on the weighted average position of the markers between the first arrangement and the second arrangement, wherein the average value is weighted over either the first arrangement or the second arrangement, and The average value is weighted based on (i) the time series of the first image data and the second image data already acquired and (ii) at least one of the viewpoint and viewing distance of at least one of the first imaging mode and the second imaging mode.
2. The method according to claim 1, wherein The arrangement of the markers defines at least one of (i) the position of the markers relative to each other and (ii) the position of the markers in a common coordinate system.
3. The method according to claim 1, wherein At least one of the marks has a first mark portion detectable by the first imaging mode and a second mark portion spaced apart from the first mark portion and detectable by the second imaging mode, wherein, The first marking portion is arranged relative to the second marking portion in a predetermined geometric relationship.
4. The method according to claim 3, wherein The marking arrangement is determined based on the predetermined geometric relationship.
5. The method according to claim 1, wherein At least one of the markers has a third marker portion that can be detected by both the first imaging mode and the second imaging mode.
6. The method according to claim 1, wherein One of the first imaging mode and the second imaging mode includes one of a computed tomography scanner and a magnetic resonance imaging scanner; and The other of the first imaging mode and the second imaging mode includes at least one camera.
7. The method according to claim 1, wherein The markers of the tracker are arranged in a common plane, and wherein, Determining the marker arrangement includes virtually arranging the positions of the markers in a virtual plane based on at least one of the first image data and the second image data.
8. The method according to claim 1, further comprising: At least one of the first image data, the second image data, and the third image data is registered with the tracker using the determined marker arrangement.
9. The method according to claim 1, wherein The tracker has a deformable base that supports the marker, wherein, The markings are arranged in a predetermined arrangement for the undeformed shape of the substrate, and wherein the marking arrangement for the deformed shape of the substrate is further determined based on the predetermined arrangement.
10. The method of claim 1, further comprising: Determine the invalid position of a specific mark in at least one of the first arrangement and the second arrangement; and When determining the position of the specific mark in the mark arrangement, invalid positions of the specific mark in at least one of the first arrangement and the second arrangement are ignored.
11. The method according to claim 1, further comprising: An error signal is generated when the quality standard between the position of the marker in the first arrangement and the position of the marker in the second arrangement meets the error condition.
12. The method according to claim 1, further comprising: An attempt is made to match the first position of the first arrangement with the second position of the second arrangement; and Ignore at least one of the following: (i) at least one of the first positions that is missing a matching second position and (ii) at least one of the second positions that is missing a matching first position.
13. A non-transitory computer-readable storage medium comprising instructions that, when executed on at least one processor, cause the at least one server to: Receive first image data of the tracker's markers captured using a first imaging mode; Receive second image data of the tracker's marker captured using a second imaging mode; A marker arrangement is determined based on the first image data and the second image data, the marker arrangement defining the position of the markers on the tracker, wherein the markers can be detected by the first imaging mode and a second imaging mode different from the first imaging mode; A first arrangement defining the first position of the mark is determined based on the first image data; and Based on the second image data, a second arrangement is determined that defines the second location of the mark. The marker arrangement is determined based on the weighted average position of the markers between the first arrangement and the second arrangement, wherein the average value is weighted over either the first arrangement or the second arrangement, and The average value is weighted based on (i) the time series of the first image data and the second image data already acquired and (ii) at least one of the viewpoint and viewing distance of at least one of the first imaging mode and the second imaging mode.
14. A system for determining the arrangement of markers, comprising: A means for determining a marker arrangement that defines the position of markers on a tracker, wherein the markers are detectable by a first imaging mode and a second imaging mode different from the first imaging mode, the means being configured to: Receive first image data of the marker captured using the first imaging mode; Receive second image data of the marker captured using the second imaging mode; The marker arrangement is determined based on the first image data and the second image data; A first arrangement defining the first position of the mark is determined based on the first image data; and Based on the second image data, a second arrangement is determined that defines the second location of the mark. The marker arrangement is determined based on the weighted average position of the markers between the first arrangement and the second arrangement, wherein the average value is weighted over either the first arrangement or the second arrangement, and The average value is weighted based on (i) the time series of the first image data and the second image data already acquired and (ii) at least one of the viewpoint and viewing distance of at least one of the first imaging mode and the second imaging mode.
15. The system of claim 14, further comprising the tracker, wherein, The tracker is attached to the surgical object or includes an interface configured to be attached to the surgical object.
Citation Information
Patent Citations
Intraoperative CT coordinate and navigation camera registration device
CN113925609A
A three-dimensional vision system
EP3556312A1
