Method and system for tool tracking

Abstract

A method for determining an orientation of a tool for performing a medical intervention on a subject is provided. The method comprises obtaining image data of a surface of a subject or a surface above a subject, a tool adjacent to the surface and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined position relative to the image sensor. A representation of the surface and a representation of the tool and the tool shadow are segmented from the image data and used to determine an elevation angle of the tool. An orientation of the tool is determined based on the segmented representation of the tool, the segmented representation of the tool shadow and the elevation angle.

Description

Technical Field

[0001] This invention relates to the field of interventional tool tracking, and more particularly to the field of image-based interventional tool tracking. Background Technology

[0002] Many interventional procedures require precise insertion of instruments through the surface of an object, such as the skin. For example, in interventional or diagnostic cases, locating and identifying vascular and / or tumor pathways with the correct needle trajectory is crucial.

[0003] Exemplary use cases include: needle access for regional anesthesia; insertion of a catheter into a vein of a subject; and ablation catheters requiring tumor access. However, interpreting subsurface image data (such as ultrasound images) in the use cases described above is challenging and requires highly trained and skilled professionals to accurately interpret the images, as well as significant mental effort to plan the needle trajectory, often requiring several trial-and-error iterations of skin entry until the correct trajectory is achieved.

[0004] Ultrasound is a popular medical imaging method and is used in many interventional and diagnostic applications. Ultrasound systems have evolved from Philips' high-end EPIQ (trademark) machines to portable, low-cost solutions. There is a trend toward developing mobile ultrasound imaging solutions that allow handheld ultrasound devices to be connected to mobile devices such as smartphones.

[0005] Currently, the only solution for tracking interventional instruments in an ultrasound system relies solely on tracking the tip of the instrument. However, tracking the tip of the instrument is only possible after it has been inserted into the patient's skin and when the tip is within the acoustic field of view of the ultrasound probe. The clinician must then mentally plan the trajectory of the instrument based on the tracked tip.

[0006] For accurate trajectory planning, the tool must be coplanar with the ultrasound imaging probe. Out-of-plane needle trajectories are difficult to predict because 3D information is not visible. Furthermore, in the case of ablation catheter tracking, the uncertainty in the tip's position will be even greater due to the physical limitations of the catheter tip, meaning tracking accuracy will be reduced. Because of these limitations, multiple skin incisions are often required to identify the correct tool trajectory, which can lead to patient discomfort and slower recovery.

[0007] Therefore, a means is needed to accurately track the interventional instrument before insertion. Summary of the Invention

[0008] This invention is defined by the claims.

[0009] According to this disclosure, a method is provided for determining the orientation of an instrument for performing a medical intervention on a subject, the method comprising:

[0010] An image sensor is used to acquire image data, which is an image of the surface of an object or a surface above the object, a tool adjacent to the surface, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined location relative to the image sensor;

[0011] Obtain a representation of the surface;

[0012] Segmenting the representation of the tool from the image data;

[0013] Segmenting the representation of the tool shadow from the image data;

[0014] The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, wherein the elevation angle is the angle between the surface and the tool;

[0015] The orientation of the tool relative to the surface is determined based on the segmented representation of the tool, the segmented representation of the tool's shadow, and the elevation angle.

[0016] The method provides a means of tracking the orientation of a tool based on the shadow cast by the tool on or over the surface of the object.

[0017] By calculating the orientation of the tool based on the shadow cast by the tool on the surface, the orientation of the tool can be known more accurately before the tool crosses the surface, thereby improving the accuracy of the user's selection of the tool's entry point and tool turning.

[0018] Furthermore, by determining the orientation of the tool based on the tool and its shadow, rather than solely on the tip of the tool, the proportion of tools used to determine the orientation of the tool is increased, thereby improving the accuracy of orientation determination.

[0019] In an embodiment, the method further includes: identifying surface contact points based on the image data, based on the representation of the tool and the representation of the tool shadow, the surface contact points being the positions on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact points.

[0020] In this way, the accuracy of the determined tool orientation can be improved by taking into account the surface contact points.

[0021] In an embodiment, obtaining a representation of the surface includes:

[0022] Motion data is obtained through motion sensors. The motion data represents the motion of the image sensor during the acquisition of the image data;

[0023] A 3D surface map of the object's surface is generated based on a combination of the image data and the motion data; and

[0024] The representation of the surface is segmented from the 3D surface map.

[0025] In this way, when determining the orientation of the tool, the topography of the surface that affects both the representation of the surface and the representation of the tool's shadow can be taken into account, thereby improving the accuracy of the determined orientation.

[0026] In an embodiment, the method further includes: obtaining computed tomography data of the surface of the object, the surface having one or more radiopaque markers fixed thereon, and wherein the representation of segmenting the surface is based on a combination of the computed tomography data, the image data, and the motion data.

[0027] In this way, the accuracy of the surface segmentation can be increased.

[0028] In an embodiment, obtaining the representation of the surface includes generating a 2D planar approximation of the surface.

[0029] In an embodiment, determining the elevation angle of the tool includes:

[0030] Calculate the shadow angle between the representation of the tool and the representation of the tool shadow; and

[0031] The elevation angle is determined by adjusting the shadow angle based on the position of the at least one light source relative to the image sensor.

[0032] In an embodiment, determining the elevation angle includes applying a machine learning algorithm to the segmented representation of the tool and the segmented representation of the tool shadow.

[0033] In this way, the elevation angle can be accurately determined in a computationally efficient manner using reduced input data.

[0034] In one embodiment, the method further includes: calculating a projection trajectory of the tool based on the determined orientation of the tool, the projection trajectory representing a predicted path of the tool after the tool has passed through the surface of the object.

[0035] In this way, the path of the tool after insertion can be predicted before the tool crosses the surface, which means that the orientation can be refined to properly position the tool without requiring unnecessary surface penetration.

[0036] In one embodiment, the method further includes generating a real-time visualization of the orientation of the tool relative to the representation of the surface.

[0037] In this way, the user can be provided with an accurate and real-time representation of the tool's orientation, so as to guide the user to the desired orientation.

[0038] In an embodiment, the light generated by the at least one light source is coded light, each light source having a unique coded light signature, thereby enabling the tool to project one or more unique modulated shadows, wherein the method further includes:

[0039] For each unique modulation shadow in the one or more unique modulation shadows, the unique encoded signature is derived based on the unique modulation shadow; and

[0040] Each unique modulated shadow in the one or more unique modulated shadows is paired with a light source based on the derived unique encoded signature; and wherein...

[0041] The representation of the tool shadow is based on one or more paired unique modulated shadows.

[0042] In this way, the tool shadows can be distinguished from each other, which means that the determined orientation can be determined with greater accuracy.

[0043] According to this disclosure, a computer program including a computer program code module is also provided, wherein when the computer program is run on a computer, the computer program code module is adapted to perform the following steps:

[0044] An image sensor is used to acquire image data, which is an image of the surface of an object or a surface above the object, an instrument adjacent to the surface for performing a medical intervention on the object, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined location relative to the image sensor;

[0045] Obtain a representation of the surface;

[0046] Segmenting the representation of the tool from the image data;

[0047] Segmenting the representation of the tool shadow from the image data;

[0048] The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, wherein the elevation angle is the angle between the surface and the tool;

[0049] The orientation of the tool relative to the surface is determined based on the segmented representation of the tool, the segmented representation of the tool's shadow, and the elevation angle.

[0050] In an embodiment, when the computer program is run on a computer, the computer program is adapted to perform the following additional steps: based on the image data, identifying surface contact points based on the representation of the tool and the representation of the tool shadow, the surface contact points being the positions on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact points.

[0051] According to this disclosure, a computer-readable storage medium including instructions, which, when executed by a computer, cause the computer to perform the following steps:

[0052] An image sensor is used to acquire image data, which is an image of the surface of an object or a surface above the object, an instrument adjacent to the surface for performing a medical intervention on the object, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined location relative to the image sensor;

[0053] Obtain a representation of the surface;

[0054] Segmenting the representation of the tool from the image data;

[0055] Segmenting the representation of the tool shadow from the image data;

[0056] The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, wherein the elevation angle is the angle between the surface and the tool;

[0057] The orientation of the tool relative to the surface is determined based on the segmented representation of the tool, the segmented representation of the tool's shadow, and the elevation angle.

[0058] The computer-readable storage medium also includes instructions that, when executed by a computer, cause the computer to perform additional steps to identify surface contact points within the image data based on the representation of the tool and the representation of the tool's shadow, the surface contact points being the locations on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact points.

[0059] According to this disclosure, a processing system for determining the orientation of an instrument for performing a medical intervention on a subject is also provided, the processing system comprising:

[0060] An input unit is configured to receive image data from an image sensor, the image data being an image of a surface of an object or a surface above the object, a tool adjacent to the surface, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined position relative to the image sensor; and

[0061] A processor, coupled to the input section, performs the following operations:

[0062] Obtain a representation of the surface;

[0063] Segmenting the representation of the tool from the image data;

[0064] Segmenting the representation of the tool shadow from the image data;

[0065] The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool's shadow, wherein the elevation angle is the angle between the surface and the tool; and

[0066] The orientation of the tool relative to the surface is determined based on the representation of the segmented tool and the representation of the segmented tool shadow.

[0067] According to this disclosure, a system for determining the orientation of an instrument is also provided, the system comprising:

[0068] The processing system as defined herein (as described in claim 15);

[0069] A tool for performing a medical intervention on an object, the tool being adapted to pass through the surface of the object;

[0070] An image sensor adapted to acquire the image data; and

[0071] At least one light source adapted to illuminate the tool, the light source being positioned at a predetermined location relative to the image sensor.

[0072] In an embodiment, the processing system is further adapted to identify surface contact points based on the image data, based on the representation of the tool and the representation of the tool shadow, the surface contact points being the positions on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact points.

[0073] In one embodiment, the system further includes: a motion sensor coupled to the image sensor, the motion sensor being adapted to acquire motion data representing the motion of the image sensor during the acquisition of the image data, and wherein, when the representation of the surface is acquired, the processing system is further adapted to:

[0074] A 3D surface map of the object's surface is generated based on a combination of the image data and the motion data; and

[0075] The representation of the surface is segmented from the 3D surface map.

[0076] In one embodiment, the system further includes an ultrasonic probe adapted to acquire ultrasonic data from an imaging region beneath the surface of the object, wherein the image sensor is coupled to the ultrasonic probe, and wherein the processing system is further adapted to:

[0077] Ultrasonic data is obtained from the imaging area, the ultrasonic data including the ultrasonic representation of the tool after the tool has passed through the surface;

[0078] The location of the tool is tracked based on the ultrasound data; and

[0079] The orientation of the tool is updated based on the location of the tracked tool.

[0080] In an embodiment, the at least one light source is adapted to generate coded light, each light source having a unique coded light signature, thereby enabling the tool to project one or more unique modulated shadows, and wherein the processing system is further adapted to:

[0081] For each unique modulation shadow in the one or more unique modulation shadows, the unique encoded signature is derived based on the unique modulation shadow; and

[0082] Each unique modulated shadow in the one or more unique modulated shadows is paired with a light source based on the derived unique encoded signature; and wherein...

[0083] The representation of the tool shadow is based on one or more paired unique modulated shadows.

[0084] These and other aspects of this disclosure will be apparent from and will be illustrated with reference to the embodiments described below. Attached Figure Description

[0085] To better understand the aspects and embodiments of this disclosure and to more clearly illustrate how they can be implemented, reference will now be made to the accompanying drawings by way of example only, wherein:

[0086] Figure 1 The method described in this disclosure is illustrated;

[0087] Figure 2 A representation of an ultrasound imaging system according to this disclosure is shown;

[0088] Figure 3A The representation of the field of view of the image sensor is shown;

[0089] Figure 3B It shows including Figure 2 The system's 3D coordinate system representation; and

[0090] Figure 4 It shows including Figure 2 Another representation of the 3D coordinate system of the system. Detailed Implementation

[0091] The detailed description and specific examples in this disclosure are intended for illustrative purposes only and not to limit the scope of the claims, as they indicate exemplary embodiments of the apparatuses, systems, and methods. These and other features, aspects, and advantages of the apparatuses, systems, and methods of this disclosure will become better understood from the following description, claims, and drawings. The drawings are merely schematic and not drawn to scale. The same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0092] This disclosure provides a method for determining the orientation of an tool used to perform a medical intervention on a subject. The method includes: acquiring an image of a surface of the subject or a surface above the subject, an image of a tool adjacent to the surface, and image data of a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined location relative to the image sensor.

[0093] A representation of the surface is obtained, and representations of the tool and its shadow are segmented from the image data and used to determine the tool's elevation angle. The tool's orientation is determined based on the segmented representation of the tool, the segmented representation of the tool's shadow, and the elevation angle.

[0094] Figure 1 A method 100 for determining the orientation of tools used to perform medical interventions on a subject is shown.

[0095] The method begins with step 110, in which image data of an image of a surface of an object or a surface above an object is acquired via an image sensor. The image data also represents a tool adjacent to the surface and tool shadows on the surface, the tool shadows being caused by light incident on the tool, the light being generated by a light source positioned at a predetermined location relative to the image sensor.

[0096] In practice, such as during a user's medical treatment, an tool is positioned within the image sensor's field of view over a surface, such as the object's skin or a covering on the object's skin, at an initial orientation and location chosen by the user. Light generated by a light source positioned at a known location relative to the image sensor is incident on the tool and the surface, causing the tool to cast a shadow on the surface. The surface can be the object's skin or a covering on the object's skin, such as clothing or medical dressings. In other words, any surface on which the tool's shadow falls, within the image sensor's view, can be used to determine the tool's orientation.

[0097] In step 120, a representation of the surface is obtained. This can be achieved in several ways. For example, the image sensor may be part of a Simultaneous Localization and Mapping (SLAM) unit, which may additionally include a motion sensor adapted to acquire motion signals representing the motion of the image sensor during image data acquisition. In this case, the representation of the skin surface within the image sensor's field of view, i.e., the segmented surface, can be represented by keypoints identified in the image data and tracked within a 3D coordinate system generated by the SLAM unit. In other words, a 3D surface map of the object's surface can be generated based on a combination of image data and motion data. The representation of the surface can then be segmented from the 3D surface map. Furthermore, any suitable image processing technique can be used to generate a 3D surface map based solely on the image data.

[0098] If generating a 3D surface map is not possible, a representation of the surface can be obtained by generating a 2D planar approximation of the surface based on image data.

[0099] If desired, a combination of 3D and 2D surface representations can be used.

[0100] In step 130, the representation of the tool is segmented from the image data, and in step 140, the representation of the tool shadow is segmented from the image data. Segmentation can be performed using any suitable segmentation method. For example, a color space transformation algorithm can be used to analyze the image data to separate the tool shadow from the surface representation. However, other segmentation techniques capable of providing the desired representation can be used for this purpose. Although shown as separation steps for clarity, the steps of segmenting the representations of the surface, tool, and tool shadow can be performed simultaneously or nearly simultaneously.

[0101] In step 150, the elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool shadow. The elevation angle is the angle between the surface and the tool.

[0102] In step 160, the orientation of the tool relative to the surface (and / or its surface representation) is determined based on the segmented representation of the tool, the segmented representation of the tool shadow, and the elevation angle.

[0103] The method may further include: calculating a projected trajectory of the tool based on the determined orientation of the tool, the projected trajectory representing a predicted path of the tool after it has crossed the surface of the object. In other words, after the orientation of the tool relative to the surface (and / or its surface representation) has been determined, the projected trajectory of the tool after insertion into the surface can be calculated. The projected trajectory may include deformation of the tool after insertion into the surface, for example, based on an anatomical model of the area being imaged or additional image data representing structures beneath the surface, such as ultrasound image data.

[0104] The following text is about Figure 3A and Figure 3B An exemplary implementation of the method described above is further illustrated below.

[0105] Figure 2 A schematic representation of an imaging system 200 suitable for implementing the methods described herein is shown. The system includes an image sensor 210 adapted to acquire image data and a tool 220 for performing medical interventions on a subject, for example, by insertion into the skin 230 of the subject.

[0106] Image sensor 210 is adapted to acquire image data of surface 240. The image sensor can be any suitable image sensor, such as a visible spectrum camera, 3D camera, time-of-flight camera, LiDAR camera, or infrared camera.

[0107] The tool can be any tool suitable for performing medical interventions on an object, such as needles, catheters, tubes, etc. The system also includes at least one light source 250 adapted to illuminate the tool 220. The light source can be an LED. It can provide directional or diffused light, as long as it provides sufficient shadows from the tool on the surface to allow for meaningful segmentation of the shadows.

[0108] When light generated by light source 250 is incident on tool 220, tool shadow 260 is cast onto the skin 230 of the object, and more specifically, onto the surface 240 seen by the image sensor. The determination of tool orientation based on tool shadow is described below with reference to Figure 3.

[0109] The position of the light source relative to the image sensor is known. For example, the light source can be directly coupled to the image sensor or integrated with the image sensor. Alternatively, the light source can be separated from the image sensor, in which case the system can undergo a calibration phase before the method of the present invention begins, wherein the position of the light source relative to the image sensor is determined by any suitable means. The light source can be a single light source or include multiple light sources. Multiple light sources can be positioned at the same general location relative to the image sensor or positioned at different locations relative to the image sensor, wherein the position of each different light source relative to the image sensor is known. One or more light sources can be modulated, or activated in a given activation mode, to cause a change in the position of the tool shadow. The change in position, combined with the known position of the light source, provides a means of calculating a single orientation of the tool based on multiple different datasets, thereby increasing the accuracy of the determined tool orientation. In addition, the modulation of the light source can increase the accuracy of segmenting the representation of the tool shadow from image data.

[0110] At least one light source 250 can be adapted to generate coded light, each light source having a unique coded light signature, thereby enabling one or more unique modulated shadows to be projected by the tool. In this case, for each of the one or more unique modulated shadows, a unique coded signature from the unique modulated shadow can be derived, and a light source can be paired based on the derived unique coded signature. The segmentation of the representation of the tool shadow can then be based on the paired one or more unique modulated shadows.

[0111] Additionally, system 200 includes a processing unit 270 adapted to perform the methods described herein. The processing unit can be any suitable processing unit, such as a processing unit within a computer, laptop, smart device, or any other processing system. The processing unit can be wired or wirelessly connected to the image sensor. The processing unit can be part of one or more of a personal computer, workstation, laptop computer, desktop computer, or other non-mobile device. The processing unit can be part of a handheld or mobile device, such as a mobile phone or tablet computer, or other such device. The processing unit can have the input and / or output devices necessary for receiving or outputting various data processed using the methods described herein.

[0112] System 200 may further include a motion sensor unit 280 adapted to acquire motion data representing the motion of the image sensor, wherein the motion data is acquired together with the image data. The motion sensor unit may include or be any suitable motion sensor, such as an accelerometer or a gyroscope. Where the system does include a motion sensor unit, the processing unit may generate a 3D surface map and / or a 2D surface map of the object's surface based on a combination of image data and motion signals.

[0113] In the example, the processing unit 270 or imaging system 200 can be integrated into a smart and / or mobile device, such as a smartphone, tablet, or laptop, which includes an image sensor, a light source, a motion sensor unit, and a processor for simultaneous localization and mapping (SLAM) of surfaces, tools, and tool shadows. SLAM can be implemented in either the smart and / or mobile device or a separate processing unit. When the imaging system is integrated into the smart and / or mobile device, the device processor can perform SLAM processing. Alternatively, dedicated SLAM hardware on the device can also implement SLAM processing. Furthermore, separate dedicated SLAM hardware can be used to implement SLAM processing.

[0114] In other words, the image sensor unit, motion sensor unit, and processor unit can form part of an integrated unit, such as an integrated unit of a non-mobile or mobile computer or device, as previously mentioned, for example. Preferably, they are integrated into a mobile device such as a smartphone or SLAM unit. In the example where the image sensor, inertial measurement unit, and processor unit form part of an integrated unit, the integrated unit may include any other suitable components for the operation of the integrated unit, such as a battery and / or a Wi-Fi communication unit.

[0115] The imaging described above can be or includes an ultrasound imaging system console that can be used in conjunction with an ultrasound probe. The ultrasound probe can be any ultrasound probe suitable for acquiring ultrasound data of an object. For example, the ultrasound probe can have a 1D ultrasound transducer array, a 2D ultrasound transducer array, or a 3D matrix ultrasound array, and can be part of a static or portable ultrasound system.

[0116] In the example described above, the image sensor and motion sensor can be coupled to the ultrasound probe. When the imaging system is integrated into a mobile device such as a smartphone, the mobile device can be coupled to the ultrasound probe. When SLAM functionality is implemented within the probe, dedicated SLAM hardware can be integrated into the probe to perform the SLAM processing described above. The coupling can be releasable and / or reconnectable.

[0117] The imaging system or console for such a system may also include a real-time visualization for showing the user the orientation of a tool relative to a surface. The display may be part of any computer, mobile, or non-mobile device mentioned above herein. For example, the display may be adapted to display a graphical representation of the tool at an orientation determined by a graphical representation relative to a surface. The real-time visualization of the tool's orientation may be continuously updated based on incoming image data from an image sensor. The display may include conventional display units, such as monitors. Alternatively or additionally, the display may include a headset worn by the user, such as an augmented reality headset. In examples of displays including augmented reality headsets, the headset may be adapted to display a graphical representation of the tool's orientation and trajectory, such that the graphical representation is aligned with the tool in the user's vision. In some examples, the processing unit is configured to generate data for displaying a region of interest of an object and a graphical representation of the surface adjacent to that region of interest, as well as a projected trajectory of the tool within the region of interest, wherein the projected trajectory is the trajectory that the tool can follow when inserted into the region of interest through the surface and based on the tool's orientation relative to the surface. In some examples, the tool is also graphically represented, but this is not mandatory.

[0118] Figure 3A A representation 300 of the field of view 310 of the image sensor is shown. Within the field of view of the image sensor is a representation of surface 320, wherein the surface is described by a plane having a first axis x and a second axis y orthogonal to the x-axis, a representation of tool 330, and a representation of tool shading 340. The surface can be represented as a 3D surface map, thereby enabling more accurate determination of the orientation of the tool or 2D plane, thus achieving a reduced computational load, as described above.

[0119] Figure 3B Representation 350 shows representations of surface 320, tool 335, and tool 330 projected onto the xy plane, as well as a representation of tool shading 340 in a 3D coordinate system defined by three orthogonal axes (x, y, and z).

[0120] As mentioned above, image-based segmentation can be applied to image data to identify representations of tools and tool shadows within the image data.

[0121] In the first example, the first angle α between the calculation tool 330 and the second axis (i.e., the y-axis) can be calculated. t The orientation of tool 335 is approximated by a second angle β between the representation of tool 330 and the representation of tool shadow 340. The location of the surface of the tool in contact with the skin can be determined as the junction of the segmented representation of the tool and the segmented representation of the tool shadow. Camera perspective projection can affect this approximation and can be taken into account by performing camera calibration.

[0122] The representation of tools and tool shadows can be parameterized in pixel space as follows:

[0123] x t =a t y t +b t (1)

[0124] as well as

[0125] x s =a s y s +b s (2)

[0126] Where: x t and x s These are the x-coordinates representing the segmented tool and the tool's shadow, respectively; y-coordinates represent the x-coordinates of the segmented tool and the tool's shadow. t and y s These are the y-coordinates of the segmented tool and the tool shadow representation, respectively; and a and b are constants used to describe the relationship between the x and y coordinates of the tool and the tool shadow representation.

[0127] Rearrange equation (1), first angle α t It can be calculated as follows:

[0128]

[0129] Similarly, rearranging equation (2), the tool shadow represents the angle α between the y-axis and the y-axis. s It can be calculated as follows:

[0130]

[0131] Then, the second angle β can be calculated as:

[0132] β=atan(a s -a t (5)

[0133] By setting x t =x s and y t =y s The junction between the tool representation and the tool shadow representation is calculated, and this junction is considered as the surface contact point, which leads to the following relationship:

[0134]

[0135] as well as

[0136] x t =at y t +b t (1)

[0137] The coordinates given in equations (6) and (1) above are in the pixel domain, that is, they are coordinates within the image data obtained by the image sensor. Therefore, the coordinates given in equations (6) and (1) above require additional processing in order to obtain the true coordinates of the tool and tool shadow in the 3D coordinate system.

[0138] A surface (such as skin) can be approximated as a 2D plane, or a 3D surface map of the surface can be generated using the SLAM hardware described above. This can be achieved by using vector V t The vector V is used to calculate the 3D coordinates of the surface contact point by intersecting with a surface (such as the skin surface). t The distance from the image sensor origin C0 to the tool entry point P is defined as in equations (6) and (1). t The vector. The 3D coordinates of the surface contact point can be calculated as follows:

[0139] X 3Dt =x t -x o (7)

[0140] as well as

[0141] Y 3Dt =y t -y o (8)

[0142] Where: X 3Dt and Y 3Dt These are the x and y coordinates of the surface contact point in the 3D coordinate system, respectively; and x o and y o These are pixel coordinates belonging to the optical axis and can be exported via camera calibration.

[0143] For example, camera calibration can be used to derive certain camera parameters, such as focal length f, which can be determined by the distance from the image sensor to the optical center of the lens. Other camera parameters, such as distortion parameters, can also be considered.

[0144] The surface can be approximated as a fixed plane, orthogonal to the optical axis of the camera, at a fixed distance D from the tip of the tool. This results in the 3D z-coordinate of the tip of the tool, Z... 3Dt =D. Then, the perspective camera model using the following equation can be used to define the x and y 3D coordinates of the surface contact points.

[0145]

[0146]

[0147] Next, calculate the elevation angle β1 between the tool and the surface. Figure 4 A schematic representation 400 is shown, illustrating the representation of surface 320, tool 335, and tool 330 on the projected xy plane, as well as the representation of tool shading 340 in a 3D coordinate system defined by three orthogonal axes (x, y, and z). Additionally, Figure 4 An image sensor 240 and a light source 260 positioned relative to tool 335 are shown.

[0148] As described above, the light source 260 is positioned at a known location relative to the image sensor 240, which causes the shadow of the tool 340 to be projected onto the surface. The angle β between the representation of the tool and the representation of its shadow, i.e., the shadow angle, can be used as an approximation of the elevation angle β1 between the tool and the surface. Figure 4 In, α l The angle between the indicator light source 260 and the center of the surface at the front of the ultrasonic probe. In α t When the surface contact point is very small and located close to the center of the probe's front edge, a scaling term can be applied to angle β to achieve angle β1. Scaling term a 缩放 It can be defined as follows:

[0149] a 缩放 =tan(α1)

[0150] For α l =45°, a 缩放 =1.0, and the relationship between elevation angle and viewpoint is 1.0. In other words, elevation angle can be determined based on the angle between the representation of the tool and tool shadow and the known position of the light source relative to the image sensor.

[0151] The above calculations produce a set of 3D coordinates representing the position of the tool tip or distal end of the tool, and two angles representing the orientation of the needle in the camera's 3D coordinate system. This virtual object can then be visualized by combining it with a 3D surface map of the surface (if available) and / or ultrasonic data obtained via an ultrasonic probe.

[0152] The tool trajectory can then be determined based on the derived tool orientation. For example, when displayed to a user, the tool trajectory can be represented on ultrasound data, which may include ultrasound volumes or 2D ultrasound slices obtained from an ultrasound probe. For example, when considering a rigid tool, the tool trajectory can be determined by extrapolating a straight line from the tool entry point below the surface (e.g., the skin) based on the determined orientation. Alternatively, in the case of a non-rigid tool, the trajectory can be determined by estimating the path of the tool from the tool entry point based on the resistance of the object's anatomy below the surface (e.g., the skin).

[0153] The tool's orientation and trajectory can be updated in real time as the user moves the tool, providing a means to examine the tool's trajectory before it enters the object's skin. The tool's trajectory can be displayed along with ultrasound data using any suitable graphical method.

[0154] In an example where the system includes an ultrasound probe, ultrasound data can be used to track the tool's position after it has passed through a surface. In other words, ultrasound data can be used to track the tool's position beneath the surface once it has been inserted into the object's skin. The tool position tracked based on ultrasound data can be used to correct and update the tool's real-time orientation and trajectory, as determined from image data, as described above.

[0155] In addition to the geometric methods described above, tool orientation can be determined based on 3D models of surfaces, tools, light sources, and image sensors. The model can take the position of the tool shadow representation relative to the tool representation as input and provide the tool orientation as output.

[0156] Furthermore, the orientation of a tool can be determined using machine learning algorithms. A machine learning algorithm is any self-trained algorithm that processes input data to produce or predict output data. Here, the input data may include the angle between the tool and a representation of the tool's shadow in image data, and the output data includes the tool's orientation.

[0157] Suitable machine learning algorithms used in this invention will be apparent to those skilled in the art. Examples of suitable machine learning algorithms include decision tree algorithms and artificial neural networks. Other machine learning algorithms, such as logistic regression, support vector machines, or Naive Bayes models, are suitable alternatives.

[0158] The structure of artificial neural networks (or simply neural networks) is inspired by the human brain. A neural network consists of layers, each containing multiple neurons. Each neuron performs a mathematical operation. Specifically, each neuron can include different weighted combinations of a single type of transformation (e.g., transformations of the same type, sigmoid transformations, etc., but with different weights). In processing the input data, the mathematical operation of each neuron is performed on the input data to produce a numerical output, and the outputs of each layer in the neural network are sequentially fed into the next layer. The final layer provides the output.

[0159] Methods for training machine learning algorithms are well-known. Typically, such methods involve obtaining a training dataset, which includes training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and its corresponding training output data entry is used to modify the machine learning algorithm. This process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is often referred to as a supervised learning technique.

[0160] For example, in machine learning algorithms formed by neural networks, the mathematical operations (weights) of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, backpropagation, and other algorithms.

[0161] The training input data entries correspond to example angles between the representations of the tool and its shadow. The training output data entries correspond to the orientation of the tool.

[0162] The subject may undergo preoperative or cone-beam computed tomography (CT) scans. In this case, additional 3D information about the subject's surface can be obtained by adding metallic references or radiopaque markers to the skin surface. These markers are visible in the CT scan and detectable in the image data captured by the image sensor. Therefore, surface segmentation can also be based on computed tomography data and image data, thereby increasing the accuracy of surface segmentation.

[0163] By studying the accompanying drawings, description, and appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality.

[0164] A single processor or other unit can perform the functions of several items as described in the claims.

[0165] Although specific measures are described in different dependent claims, this does not imply that combinations of these measures cannot be used advantageously.

[0166] Computer programs can be stored / distributed on suitable media, such as optical storage media or solid-state media provided together with or as part of other hardware, but computer programs can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0167] If the term “suitable” is used in the claims or specification, it should be noted that the term “suitable” is intended to be equivalent to the term “configured as”.

[0168] Any reference numerals in the claims should not be construed as limiting the scope.

[0169] List of non-limiting embodiments

[0170] Example 1: A method (100) for determining the orientation of an instrument for performing a medical intervention on a subject, the method comprising:

[0171] An image sensor is used to acquire (110) image data, which is an image of the surface of an object or a surface above the object, a tool adjacent to the surface, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined position relative to the image sensor;

[0172] Obtain a representation of the surface described in (120);

[0173] Segment (130) the representation of the tool from the image data;

[0174] Segment (140) the representation of the tool shadow from the image data;

[0175] The elevation angle of the tool is determined (150) based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, the elevation angle being the angle between the surface and the tool;

[0176] The orientation of the tool relative to the surface is determined (160) based on the representation of the segmented tool, the representation of the segmented tool shadow, and the elevation angle.

[0177] Example 2, the method (100) defined in Example 1, further includes: identifying a surface contact point based on the image data based on the representation of the tool and the representation of the tool shadow, the surface contact point being the position on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact point.

[0178] Example 3. A method (100) as defined in any one of Examples 1 to 2, wherein obtaining the representation of the surface comprises:

[0179] Motion data is obtained through a motion sensor, the motion data representing the motion of the image sensor during the acquisition of the image data;

[0180] A 3D surface map of the object's surface is generated based on a combination of the image data and the motion data; and

[0181] The representation of the surface is segmented from the 3D surface map.

[0182] Example 4, the method (100) as defined in Example 3, wherein the method further includes: obtaining computed tomography data of the surface of the object, the surface having one or more radiopaque markers fixed thereon, and wherein the representation of the surface segmentation is based on a combination of the computed tomography data, the image data and the motion data.

[0183] Example 5: A method (100) defined according to any one of Examples 1 to 2, wherein obtaining the representation of the surface includes generating a 2D planar approximation of the surface.

[0184] Example 6: A method (100) defined according to any one of Examples 1 to 5, wherein determining the elevation angle of the tool includes:

[0185] Calculate the shadow angle between the representation of the tool and the representation of the tool shadow; and

[0186] The elevation angle is determined by adjusting the shadow angle based on the position of the at least one light source relative to the image sensor.

[0187] Example 7: A method (100) defined according to any one of Examples 1 to 6, wherein determining the elevation angle includes: applying a machine learning algorithm to the segmented representation of the tool and the segmented representation of the tool shadow.

[0188] Example 8: A method (100) as defined in any one of Examples 1 to 7, wherein the method further comprises: calculating a projection trajectory of the tool based on the determined orientation of the tool, the projection trajectory representing a predicted path of the tool after the tool has passed through the surface of the object.

[0189] Example 9: The method (100) according to any one of Examples 1 to 8, wherein the method further includes: generating a real-time visualization of the orientation of the tool relative to the representation of the surface.

[0190] Example 10: A method (100) according to any one of Examples 1 to 9, wherein the light generated by the at least one light source is coded light, each light source having a unique coded light signature, thereby causing one or more unique modulated shadows to be projected by the tool, wherein the method further includes:

[0191] For each unique modulation shadow in the one or more unique modulation shadows, the unique encoded signature is derived based on the unique modulation shadow; and

[0192] Each unique modulated shadow in the one or more unique modulated shadows is paired with a light source based on the derived unique encoded signature; and wherein...

[0193] The representation of the tool shadow is based on one or more paired unique modulated shadows.

[0194] Example 11: A computer program including computer program code modules, wherein when the computer program is run on a computer, the computer program code modules are adapted to perform the following steps:

[0195] An image sensor is used to acquire image data, which is an image of the surface of an object or a surface above the object, an instrument adjacent to the surface for performing a medical intervention on the object, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined location relative to the image sensor;

[0196] Obtain a representation of the surface;

[0197] Segmenting the representation of the tool from the image data;

[0198] Segmenting the representation of the tool shadow from the image data;

[0199] The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, wherein the elevation angle is the angle between the surface and the tool;

[0200] The orientation of the tool relative to the surface is determined based on the segmented representation of the tool, the segmented representation of the tool's shadow, and the elevation angle.

[0201] Example 12: A computer program according to the definition in Example 11, wherein, when the computer program is run on a computer, the computer program is adapted to perform the following additional steps: based on the representation of the tool and the representation of the tool shadow, identifying a surface contact point based on the image data, the surface contact point being the position on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact point.

[0202] Example 13: A computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform the following steps:

[0203] An image sensor is used to acquire image data, which is an image of the surface of an object or a surface above the object, an instrument adjacent to the surface for performing a medical intervention on the object, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined location relative to the image sensor;

[0204] Obtain a representation of the surface;

[0205] Segmenting the representation of the tool from the image data;

[0206] Segmenting the representation of the tool shadow from the image data;

[0207] The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, wherein the elevation angle is the angle between the surface and the tool;

[0208] The orientation of the tool relative to the surface is determined based on the segmented representation of the tool, the segmented representation of the tool's shadow, and the elevation angle.

[0209] Example 14: A computer-readable storage medium as defined in Example 13, wherein the computer-readable storage medium further includes instructions that, when executed by a computer, cause the computer to perform additional steps of identifying surface contact points within the image data based on the representation of the tool and the representation of the tool's shadow, the surface contact point being a position on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact point.

[0210] Example 15: A processing system (270) for determining the orientation of an instrument for performing a medical intervention on a subject, the processing system comprising:

[0211] An input unit is configured to receive image data from an image sensor, the image data being an image of a surface of an object or a surface above the object, a tool adjacent to the surface, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined position relative to the image sensor; and

[0212] A processor, coupled to the input section, performs the following operations:

[0213] Obtain a representation of the surface;

[0214] Segmenting the representation of the tool from the image data;

[0215] Segmenting the representation of the tool shadow from the image data;

[0216] The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool's shadow, wherein the elevation angle is the angle between the surface and the tool; and

[0217] The orientation of the tool relative to the surface is determined based on the representation of the segmented tool and the representation of the segmented tool shadow.

[0218] Example 16: A system (200) for determining the orientation of a tool, the system comprising:

[0219] The processing system (270) as defined in Embodiment 15;

[0220] Tool (220) for performing medical intervention on an object, said tool being adapted to pass through said surface of said object;

[0221] Image sensor (210), adapted to acquire the image data; and

[0222] At least one light source (250) adapted to illuminate the tool, the light source being positioned at a predetermined location relative to the image sensor.

[0223] Example 17: A system (200) according to Example 16, wherein the processing system is further adapted to identify surface contact points based on the image data based on the representation of the tool and the representation of the tool shadow, the surface contact points being the positions on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact points.

[0224] Example 18: A system (200) as defined in any one of Examples 16 to 17, wherein the system further comprises: a motion sensor (280) coupled to the image sensor, the motion sensor being adapted to acquire motion data representing the motion of the image sensor during the acquisition of the image data, and wherein, when the representation of the surface is acquired, the processing system is further adapted to:

[0225] A 3D surface map of the object's surface is generated based on a combination of the image data and the motion data; and

[0226] The representation of the surface is segmented from the 3D surface map.

[0227] Example 19. A system (200) according to any one of Examples 16 to 18, wherein the system further includes an ultrasonic probe adapted to acquire ultrasonic data from an imaging region below the surface of the object, wherein the image sensor is coupled to the ultrasonic probe, and wherein the processing system is further adapted to:

[0228] Ultrasonic data is obtained from the imaging area, the ultrasonic data including the ultrasonic representation of the tool after the tool has passed through the surface;

[0229] The location of the tool is tracked based on the ultrasound data; and

[0230] The orientation of the tool is updated based on the location of the tracked tool.

[0231] Example 20: A system (200) as defined in any one of Examples 16 to 19, wherein the at least one light source is adapted to generate coded light, each light source having a unique coded light signature, thereby enabling one or more unique modulated shadows to be projected by the tool, and wherein the processing system is further adapted to:

[0232] For each unique modulation shadow in the one or more unique modulation shadows, the unique encoded signature is derived based on the unique modulation shadow; and

[0233] Each unique modulated shadow in the one or more unique modulated shadows is paired with a light source based on the derived unique encoded signature; and wherein...

[0234] The representation of the tool shadow is based on one or more paired unique modulated shadows.

Claims

1. A method (100) for determining the orientation of an instrument for performing a medical intervention on a subject, the method comprising: An image sensor is used to acquire (110) image data, which is an image of the surface of an object or a surface above the object, a tool adjacent to the surface, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined position relative to the image sensor; Obtain a representation of the surface described in (120); Segment (130) the representation of the tool from the image data; Segment (140) the representation of the tool shadow from the image data; The elevation angle of the tool is determined (150) based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, the elevation angle being the angle between the surface and the tool; The orientation of the tool relative to the surface is determined (160) based on the representation of the segmented tool, the representation of the segmented tool shadow, and the elevation angle.

2. The method (100) of claim 1, further comprising: Based on the representation of the tool and the representation of the tool shadow, surface contact points are identified based on the image data. The surface contact points are the positions on the surface where the tool contacts the surface, and the elevation angle of the tool is determined based on the surface contact points.

3. The method (100) according to any one of claims 1 to 2, wherein Obtaining the representation of the surface includes: Motion data is obtained through a motion sensor, the motion data representing the motion of the image sensor during the acquisition of the image data; A 3D surface map of the object's surface is generated based on a combination of the image data and the motion data; and The representation of the surface is segmented from the 3D surface map.

4. The method (100) of claim 3, wherein The method further includes: obtaining computed tomography data of the surface of the object, the surface having one or more radiopaque markers fixed thereon, and wherein the representation of segmenting the surface is based on a combination of the computed tomography data, the image data, and the motion data.

5. The method (100) according to any one of claims 1 to 2, wherein Obtaining the representation of the surface includes generating a 2D planar approximation of the surface.

6. The method (100) according to any one of claims 1 to 5, wherein Determining the elevation angle of the tool includes: Calculate the shadow angle between the representation of the tool and the representation of the tool's shadow; and The elevation angle is determined by adjusting the shadow angle based on the position of the at least one light source relative to the image sensor.

7. The method (100) according to any one of claims 1 to 6, wherein Determining the elevation angle includes applying a machine learning algorithm to the segmented representation of the tool and the segmented representation of the tool's shadow.

8. The method (100) according to any one of claims 1 to 7, wherein, The method further includes: calculating a projected trajectory of the tool based on the determined orientation of the tool, the projected trajectory representing a predicted path of the tool after the tool has passed through the surface of the object.

9. The method (100) according to any one of claims 1 to 8, wherein The method further includes generating a real-time visualization of the orientation of the tool with respect to the representation of the surface.

10. The method (100) according to any one of claims 1 to 9, wherein The light generated by the at least one light source is coded light, each light source having a unique coded light signature, thereby enabling the tool to project one or more unique modulated shadows, wherein the method further includes: For each unique modulation shadow in the one or more unique modulation shadows, the unique encoded signature is derived based on the unique modulation shadow; and Each unique modulated shadow in the one or more unique modulated shadows is paired with a light source based on the derived unique encoded signature; and wherein... The representation of the tool shadow is based on one or more paired unique modulated shadows.

11. A computer program including computer program code modules, wherein when the computer program is run on a computer, the computer program is adapted to perform the following steps: An image sensor is used to acquire image data, which is an image of the surface of an object or a surface above the object, an instrument adjacent to the surface for performing a medical intervention on the object, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined location relative to the image sensor; Obtain a representation of the surface; Segmenting the representation of the tool from the image data; Segmenting the representation of the tool shadow from the image data; The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, wherein the elevation angle is the angle between the surface and the tool; The orientation of the tool relative to the surface is determined based on the segmented representation of the tool, the segmented representation of the tool's shadow, and the elevation angle.

12. A processing system (270) for determining the orientation of an instrument for performing a medical intervention on a subject, the processing system comprising: An input unit is configured to receive image data from an image sensor, the image data being an image of a surface of an object or a surface above the object, a tool adjacent to the surface, and a tool shadow on the surface, the tool shadow being caused by light incident on the tool, the light being generated by at least one light source positioned at a predetermined position relative to the image sensor; as well as A processor, coupled to the input section, performs the following operations: Obtain a representation of the surface; Segmenting the representation of the tool from the image data; Segmenting the representation of the tool shadow from the image data; The elevation angle of the tool is determined based on the representation of the surface, the representation of the tool, and the representation of the tool shadow, wherein the elevation angle is the angle between the surface and the tool; and The orientation of the tool relative to the surface is determined based on the representation of the segmented tool and the representation of the segmented tool shadow.

13. A system (200) for determining the orientation of an tool, the system comprising: The processing system (270) according to claim 12; Tool (220) for performing medical intervention on an object, said tool being adapted to pass through said surface of said object; An image sensor (210) adapted to acquire the image data; as well as At least one light source (250) adapted to illuminate the tool, the light source being positioned at a predetermined location relative to the image sensor.

14. The system (200) of claim 13, wherein, The processing system is also adapted to identify surface contact points based on the image data, based on the representation of the tool and the representation of the tool shadow, the surface contact points being the positions on the surface where the tool contacts the surface, and wherein determining the elevation angle of the tool is also based on the surface contact points.

15. The system (200) according to any one of claims 13 to 14, wherein, The system further includes: a motion sensor (280) coupled to the image sensor, the motion sensor being adapted to acquire motion data representing the motion of the image sensor during the acquisition of the image data, and wherein, when the representation of the surface is acquired, the processing system is further adapted to: A 3D surface map of the object's surface is generated based on a combination of the image data and the motion data; and The representation of the surface is segmented from the 3D surface map.

16. The system (200) according to any one of claims 13 to 15, wherein, The system further includes an ultrasonic probe adapted to acquire ultrasonic data from an imaging region beneath the surface of the object, wherein the image sensor is coupled to the ultrasonic probe, and wherein the processing system is further adapted to: Acquire ultrasonic data from the imaging region, the ultrasonic data including the ultrasonic representation of the tool after the tool has passed through the surface; The location of the tool is tracked based on the ultrasound data; and The orientation of the tool is updated based on the location of the tracked tool.

Images