Positioning and orienting a handheld modality

Abstract

A handheld modality (4) is to be positioned with less effort and / or radiation exposure. For this purpose, an object model is created from 3D data of an object (3). A surface model of the object (3) is also created. Both models are registered relative to each other. Depending on a target area (1) in the object model, a starting position and orientation of the handheld modality (4) are calculated. A target image of the object (3) is also calculated based on the object model, the target area, and the starting position and orientation. Furthermore, a real ultrasound image is acquired from the starting position and orientation. Based on any deviation between the ultrasound image and the target image, the handheld modality (4) is repositioned and / or reoriented.

Description

[0001] The present invention relates to a method for positioning and orienting a hand-held modality. Furthermore, the present invention relates to a device with a hand-held modality that is to be positioned in a suitable manner.

[0002] Microwave and radiofrequency ablation is a treatment option for early stages of liver cancer. An ablation needle must be precisely positioned within the tumor to be removed. A preinterventional 3D imaging study (CT or MRI) is usually available to demonstrate the tumor's location. Optimal needle insertion and placement are essential for the procedure. Imaging techniques can be used to assist the physician in this process.

[0003] There are various methods to control needle positioning. For example, ultrasound is used as a real-time imaging system to visualize the needle during insertion. In some cases, the pre-interventional MR or CT datasets are registered with the live ultrasound signal to assist in tumor localization. Registration can be rigid or elastic. Visualization can be achieved through side-by-side images or overlaid images. Robust registration typically requires a tracking device that continuously monitors the position and orientation of the ultrasound probe and adjusts the visualization of the CT / MR dataset accordingly. Registration is achieved by identifying landmarks (distinctive features in the image, e.g.Vessels) in both ultrasound and CT / MR are initiated either by user interaction or by automatic image analysis. Typically, at least two image planes from different directions are required to locate such landmarks in 3D. Alternatively, a 3D sweep (series of images from at least two different directions) can be performed using ultrasound, and the multiple image stacks can be aligned either by manual user interaction or a suitable algorithm. In general, image registration is cumbersome due to organ deformation caused by changes in patient positioning and respiratory artifacts, and requires trained and experienced operators to perform these procedures.

[0004] In more advanced practices, CT or MRI scanners are used to continuously monitor needle positioning. The advantage of this approach is that a complete 3D dataset is available, facilitating needle positioning. Disadvantages include the radiation exposure of CT (computed tomography) to both the patient and the operator, and the limited accessibility and high cost of MRI (magnetic resonance imaging).

[0005] US 2013 / 0035583 A1 discloses a method for generating a synthetic image in which images from different medical imaging devices are registered with each other. The synthetic image is generated by mapping the images onto one another. The synthetic image can be used for guidance during a diagnosis or a robot-controlled surgical procedure.

[0006] Publication WO 2006 / 036458 A1 describes a method in which, in one step, data associated with the position or orientation of a simulated scanner relative to an object is acquired. In another step, image data is calculated based on the acquired data, essentially in real time. In a further step, a simulated ultrasound image is rendered from the image data on a graphic display. The simulated ultrasound image represents the interior or simulated interior of an object in the ultrasound scanning plane.

[0007] The object of the present invention is to provide accurate imaging for intervention as an alternative to CT and MR.

[0008] According to the invention, this object is achieved by a device and a method according to the independent claims. Advantageous developments of the invention emerge from the subclaims.

[0009] According to the present invention, a method for positioning and orienting a hand-held modality is therefore provided. Throughout this document, the term "ultrasound device" is used as a representative for other hand-held modalities (e.g., SPECT device; single photon emission computed tomography). The term "hand-held" means that at least part of the modality (e.g., probe) can be guided by hand. The ultrasound device is intended to be used as an alternative to stressful CT procedures and expensive MR procedures. However, in order to achieve high image quality, particularly for the positioning of (medical) instruments, it is necessary to position and orient the ultrasound device very precisely. In addition to the position, the orientation of the ultrasound device is also of crucial importance.

[0010] One step of the method involves loading 3D data from an object. Loading the 3D data can be part of an acquisition process. However, the 3D data (e.g., MRI or CT data) can also be provided from a storage medium or via an interface. In this case, they are loaded or stored in the RAM of a computing device, for example. The object can be a human or animal patient, but also a technical object.

[0011] In a further step, an object model of the object is automatically generated from the 3D data. This object model is typically a 3D model that represents the object and its internal structure. The object model can therefore, for example, contain appropriately segmented spatial regions. In the case of a patient, the object model is a patient model that preferably represents important internal organs, such as the liver, blood vessels, rib cage, etc. The object model can be automatically generated from the 3D data using a suitable application.

[0012] In the method according to the invention, a target area is also specified in the object model. This target area can, for example, be a lesion that is to be observed or treated. This target area is marked in the object model, for example, through user interaction. If necessary, the target area can also be determined automatically or specified through data input.

[0013] Furthermore, surface data of the object is acquired. Since the ultrasound device must be placed on the surface of the object, it is essential to at least partially capture the surface of the object. This capture can be achieved, for example, optically or, if necessary, using other electromagnetic techniques, such as those present on the surface. The capture techniques should be tailored to the surface properties of the object.

[0014] A surface model of the object is automatically generated from the surface data. A surface model of the object or patient is thus automatically generated independently of the object model. While the object model is automatically generated from complex, preferably pre-interventionally acquired 3D data (e.g., MRI or CT), a current surface model can be easily obtained from optically acquired data.

[0015] If necessary, an initial surface model is also captured pre-interventionally during the acquisition of the object model. This creates a direct link between the object and surface models and can be compared with the new or current surface model during the ultrasound-guided intervention. This has the particular advantage that the two surface models can contain a larger section of the patient than the object model, thus facilitating registration.

[0016] In a next step, the surface model is registered with the object model. The object model also represents a surface structure, which is compared or registered with that of the surface model. This makes it possible to estimate the internal structure of the object in terms of position and orientation based on a current surface model. Under certain circumstances, the aforementioned initial surface model can be used for registration.

[0017] Furthermore, a starting position and orientation of the ultrasound device are automatically calculated on the object model based on the target area. The ultrasound device, and in particular its probe, must be placed on the object. For this purpose, the starting position and orientation of the ultrasound device are virtually calculated based on the desired target area on the object model. Since the surface model is registered with the object model, an updated object model is essentially available, on whose surface the ultrasound device must be virtually placed and aligned. This provides an initial estimate of the position and orientation of the ultrasound device in order to acquire suitable images of the target area.

[0018] In a further step, a target image of a partial area of ​​the object is automatically calculated based on the object model, the target area, and the starting position and orientation of the ultrasound system. From the starting position and orientation of the ultrasound system, a target image (e.g. a virtual ultrasound image) is estimated that represents a partial area of ​​the object. For this purpose, a slice representation (the target image) is generally calculated from the 3D data set that corresponds to the starting position and orientation. It is then generally up to the user to adjust the real ultrasound image so that it matches the calculated target image. The user can therefore compare the calculated MR / CT contrast image for a corresponding slice from the 3D or voxel data set with the real ultrasound image.The mentioned sub-area of ​​the object preferably includes the aforementioned target area (ROI: Region of Interest). Since the acquisition geometry of the ultrasound system is usually known, the target image can be estimated based on the determined starting position and orientation of the ultrasound system using the object model. In this document, the term "virtual ultrasound image" is also used as a substitute for the term "target image." In addition to the virtual ultrasound image (simulated US contrast image), the calculated target image can also be a CT contrast image or MR contrast image.

[0019] Furthermore, in the method according to the invention, a real ultrasound image of the partial area of ​​the object is acquired using the ultrasound device in the starting position and starting orientation. For this purpose, the real ultrasound device is moved to the real starting position and starting orientation, which correspond to the virtual starting position and starting orientation, which were automatically calculated. If appropriate, the virtual data correspond to real data; the data can be identical or can be obtained by coordinate transformation.

[0020] In a subsequent step, the (real) ultrasound system is repositioned and / or reoriented based on a deviation between the target image and the real ultrasound image. This repositioning or reorientation can be performed automatically, semi-automatically, or manually. The deviation between the target image and the real ultrasound image can be determined, for example, using a landmark. If, for example, a vascular branch is located in a different image area in the target image than in the real ultrasound image, a corresponding deviation exists, which can be corrected (fully or largely) by reorienting the real ultrasound system.Using the target image, which is obtained based on the object model or, preferably, the 3D data acquired pre-interventionally, the real ultrasound device can be optimally positioned and oriented, as the object model is updated with the easily acquired surface model. This allows an ultrasound device to be easily (optimally) positioned and oriented without additional radiation exposure.

[0021] This can help users identify landmarks, lesions / tumors, and needles with greater certainty, accelerating the workflow. The process utilizes information about the body's interior (3D MR / CT data), information about the body surface (patient avatar based on sensor data, such as from a stereoscopic camera), and information about the actual position / orientation of the ultrasound probe to determine the appropriate target position / orientation of the ultrasound probe for visualizing landmarks, lesions, and needles (devices).

[0022] In one embodiment, the object is a patient, and during the automatic generation of the object model, the patient's organs are automatically segmented. The object model is thus a patient model in which organs such as the liver, blood vessels, or the like are modeled separately. The modeling is based on the 3D data. If necessary, the patient model can be supplemented with a deformation model to account for deformations resulting from patient positioning and respiratory artifacts, for example.

[0023] According to another embodiment, the target image is a computed ultrasound image, MRI image, or CT image. Thus, the virtual image can be created in a manner that is optimal for the respective situation. In another embodiment, in addition to the 3D data of the object, initial surface data of the object, which are acquired during acquisition of the 3D data, are also loaded, an initial surface model is generated from the initial surface data, and the initial surface model, which is implicitly registered with the object model, is used to register the surface model with the object model. This has the advantage that an initial surface model of the object can be acquired pre-interventionally in the same situation as the 3D data or the object model of the object. The initial surface model is thus automatically registered with the object model and can thus be used for indirect registration of the object model with the current surface model.

[0024] In another embodiment, a 3D camera is used to capture the object's surface data, which also captures the position and orientation of the handheld modality. Thus, the 3D camera has dual functionality.

[0025] Furthermore, in one embodiment, the 3D camera can be used to capture the position and orientation of an instrument, which is optionally guided on the handheld modality. This provides the 3D camera with an additional functionality, namely monitoring the instrument. The following procedures are exemplary: The needle (i.e., the instrument) is advanced through a guide rail attached to the ultrasound head, or the operator guides the ultrasound head with one hand and a needle holder with the other.

[0026] In a further embodiment, a path of an instrument into the partial area of ​​the object is planned based on the object model, and the path is used to automatically calculate the target image (e.g. virtual ultrasound image). The instrument can be a treatment instrument such as an ablation needle, for example. The path of an ablation needle or the like can be planned using a treatment model, for example. In doing so, obscuring structures such as ribs can be taken into account. For example, the path past the ribs must be planned. The planned path can be used to calculate the virtual ultrasound image. The path should be clearly visible in the ultrasound image (in-plane). Consequently, the planned path should be taken into account for optimizing the position and orientation of the ultrasound device.

[0027] Planning the path may involve repositioning the object. For example, in a first planning step, an initial path design is planned that turns out not to be optimal for the treatment. Therefore, further planning may include the step of repositioning the object or patient. This means changing the pose (position and orientation) of the object. This typically not only changes the spatial relationship of the object's substructures to external devices (treatment instruments, ultrasound equipment, and the like), but may also alter deformations of the object's / patient's internal structures or organs.

[0028] In a further embodiment, it can be provided that the repositioning of the object is taken into account by updating the object model using a deformation model. As already indicated above, the object model can be supplemented by a deformation model in order to account for current deformations (for example, of a patient's organs) for the path planning of an instrument or the positioning and orientation of an ultrasound device. In particular, for example, a surface deformation in the 3D object model registered to the surface model can be extrapolated. In a further embodiment, an acquisition geometry of the ultrasound device can be taken into account during the automatic calculation of the virtual ultrasound image.In addition to the position and orientation of the ultrasound device, the acquisition geometry can also include, for example, the aperture angle for imaging. For example, if a previously known ultrasound device is used, it is advantageous to adapt the acquisition geometry for the virtual ultrasound image to the real ultrasound device. This can increase the similarity between virtual and real ultrasound images. The acquisition geometry can also include the selected plane of ultrasound image acquisition in relation to an instrument (e.g., cannula or needle).

[0029] In a specific embodiment, the automatic calculation of the virtual ultrasound image involves optimization with respect to different ultrasound image planes. For example, so-called in-plane ultrasound images or off-plane (out-of-plane) images can be acquired. With the in-plane technique, for example, the cannula or needle runs in the sound plane. With the off-plane technique, the cannula or needle intersects the sound plane. In the latter case, for example, the transducer and needle are at an angle of, for example, 90 degrees to each other. A user can also participate in the optimization of the virtual ultrasound image with respect to the respective plane and define the plane using a suitable interface.

[0030] In a further embodiment, user interaction can be considered for the repositioning and / or reorientation of the ultrasound device. For example, it is advantageous to provide an interface for so-called fine-tuning that allows a user to at least partially reposition or reorient the ultrasound device. Thus, for example, the repositioning or reorientation for fine-tuning can also be performed manually or semi-automatically.

[0031] In another embodiment, the deviation between the virtual ultrasound image and the real ultrasound image is determined using only a single landmark in both ultrasound images for repositioning and / or reorienting the ultrasound device. For example, a single vessel branch is used as a landmark to detect a deviation between the real and virtual ultrasound images and register the two images accordingly. With a refined registration, multiple landmarks can then be observed.

[0032] Thus, an initial rough repositioning can be carried out using a single landmark, while further fine positioning is carried out using several landmarks.

[0033] In the course of optimizing the position or orientation of the ultrasound device, one embodiment may provide for the steps of acquiring a real ultrasound image, on the one hand, and repositioning and / or reorienting the ultrasound device, on the other, to be repeated alternately several times to minimize the difference between the virtual and real ultrasound image. In an optimization loop, an ultrasound image can thus be taken after each repositioning or reorientation in order to compare it again with the virtual ultrasound image.

[0034] In a particularly preferred embodiment, a path for an instrument is mechanically defined by positioning and orienting the ultrasound device. For example, the ultrasound device has a guide element (e.g., guide rail) on its housing. The positioning and orientation of the ultrasound device also positions and orients the path for the instrument. This allows the path for the instrument to be easily defined simultaneously with the ultrasound device. Alternatively or additionally, as mentioned above, the position and orientation of the instrument / needle can also be monitored and controlled using a 3D camera or stereoscopic camera, which may also capture the surface data of the object.

[0035] The above-mentioned object is also achieved according to the invention by a device with - a hand-held modality, - a first acquisition device for loading (if necessary, acquiring) 3D data (MR, CT) of an object and for specifying a target area (e.g. lesion) in the object model, - a second detection device for detecting surface data of the object, - a computing device designed to * automatic creation of an object model of the object from the 3D data, * automatic generation of a surface model of the object from the surface data, * Registering the surface model to the object model, * Automatic calculation of a starting position and starting orientation of the hand-guided modality on the object model based on the target area, and * Automatic calculation of a target image of a region of the object (ROI) based on the object model, the target area and the starting position and orientation of the hand-guided modality, whereby - with the hand-held modality, a real ultrasound image of the part of the object can be acquired using the ultrasound device in the starting position and orientation, and - the device comprises a display device designed to visualize a deviation between the target image and the real ultrasound image.

[0036] Based on the discrepancy between the target image and the actual ultrasound image, the handheld modality (automatic / manual) can be repositioned and / or reoriented. Positioning and orienting the handheld modality means that at least one part of the handheld modality (e.g., ultrasound probe) is positioned or oriented.

[0037] The ultrasound device is an ultrasound-based imaging device. It may include an ultrasound probe, which is typically placed on the surface of the object or patient for ultrasound images.

[0038] The first acquisition device is designed at least to load 3D data of the object. For example, this 3D data can be provided from a storage unit. In particular, pre-interventional data can be made available for the device in this way. If necessary, the first acquisition device can also have an MR or CT scanner, with which the 3D data is directly acquired and provided. Furthermore, the first acquisition device is also designed to specify the target area in the object model. This can be done by loading target data. Alternatively, the target area can be specified by recording user interactions via a corresponding interface and converting target area coordinates. For example, a user can mark a lesion in a three-dimensional object model, thereby defining the target area.

[0039] The second detection device for detecting the surface data can, for example, comprise a stereoscopic camera with which the surface of the object is optically detected.

[0040] The computing device for generating the object model and the surface model, for registering both models, and for calculating the starting position and orientation of the ultrasound device and for calculating the target image, may comprise a computer or processor with memory elements. However, the computing device may also, under certain circumstances, perform control tasks for controlling the ultrasound device and the acquisition devices.

[0041] The positioning device of the device can comprise positioning means for automatic, semi-automatic, or manual positioning and / or orientation. This allows the ultrasound device, or a part thereof, to be positioned and aligned with respect to the target area.

[0042] The same advantages and possible variations apply to the device according to the invention as to the method according to the invention described above, and vice versa. The method features can therefore be viewed as corresponding functional features of the device.

[0043] In one embodiment, the ultrasound device has a needle guide for guiding an ablation needle or a cannula. The needle guide can be designed as a mechanical guide that allows, for example, a degree of freedom of movement for the ablation needle or cannula. If necessary, the corresponding instrument (e.g., ablation needle or cannula) is automatically moved by means of a propulsion in the needle guide. Preferably, the needle guide is firmly attached to the ultrasound device and in particular to a housing of the ultrasound device. This has the advantage that the needle guide is also automatically positioned and aligned with the alignment or positioning of the ultrasound device.

[0044] In a further embodiment, the device can further comprise a planning device for planning a path of an instrument into the partial area of ​​the object based on the object model, as well as a navigation device for navigating the instrument along the planned path depending on a real ultrasound image acquired after repositioning and / or reorienting the ultrasound device. The planning device and / or the navigation device can be based on a computer. The planning device can have a corresponding HMI interface. The HMI interface can be used, for example, to display the desired and actual positions of the ultrasound probe and / or the instrument in order to provide the user with information on changing the actual position.The navigation device may have corresponding control elements to control a separate drive device for the ultrasound device or the ultrasound device itself.

[0045] For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and / or a request to enter user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0046] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0047] The present invention will now be explained in more detail with reference to the accompanying drawings, in which: Fig. 1 is a schematic view of an embodiment of a device according to the invention; Fig. 2 a schematic flow diagram of an embodiment of a method according to the invention and Fig. 3 a concrete design of a process detail in a schematic sequence.

[0048] The exemplary embodiments described in more detail below represent preferred embodiments of the present invention.

[0049] The Fig. The device shown in Figure 1 can be understood as a system to assist a user in positioning an ultrasound probe so that the registration process and any needle positioning can be facilitated. For example, the device is to be used for a system with which a liver tumor 1 in a liver 2 of a patient 3 can be treated. The patient 3 here represents the object for the ultrasound examination or treatment. In the present example, the ultrasound device 4 has an ultrasound probe 5, an evaluation and control unit 6 and optionally a positioning device 7. Preferably, the ultrasound probe 5 is hand-held and, if necessary, the position and orientation are monitored by means of a second detection device (e.g., a stereoscopic camera).The ultrasound probe 5 can be positioned and oriented by the evaluation and control unit 6 via the optional positioning device 7, in particular on the surface of the object or patient 3.

[0050] The ultrasound system can be a 2D ultrasound system, for example, capable of delivering real-time b-mode images. The ultrasound probe 5 can have a guide 8, with which, for example, an ablation needle 9 can be guided from the outside through the chest and liver 2 of the patient 3. Care should be taken, for example, that the path of the ablation needle 9 does not intersect with a rib 10.

[0051] In the present case, the liver tumor 1, or a portion thereof, would be the target area to be reached by the ablation needle 9 and imaged by the ultrasound device 4. It is particularly advantageous if the ultrasound device 4 is capable of real-time imaging of the target area.

[0052] The device of Fig. 1 also has a first acquisition device 11 for loading 3D data of the patient 3. For example, a pre-interventionally acquired 3D dataset of the patient 3 can be used for the device. The 3D dataset can originate, for example, from an MR or CT system. In particular, the dataset should represent the target area. The acquisition device 11 can have an interface to the Internet, a CD or DVD drive, or another data interface. If appropriate, the first acquisition device 11 is also part of an MR or CT system or the like. In the present example, the 3D dataset can originate from an MRI examination of a liver, wherein segmented lesions are intended for ablation.

[0053] The Fig. The device illustrated in Figure 1 also has a second acquisition device 12 for acquiring surface data of the object or patient 3. In the present example, this can be a stereoscopic camera with which the surface of the patient 3 can be scanned three-dimensionally. Thus, an avatar of the patient surface (surface model) can be created, and the internal position of the liver can be estimated together with the 3D data. Optionally, the stereoscopic camera can already acquire a surface model of the patient during the pre-interventional image acquisition and make it available together with the object model via acquisition device 11. If necessary, the surface model acquired during the pre-interventional image acquisition can be used to calculate the registration with the interventional surface model and the virtual surface model.Optionally, visible markers can be placed on the patient's body during pre-interventional image acquisition. If necessary, the markers can be reused during the intervention.

[0054] The ultrasound probe 5 can also have one or more markers 13, by means of which the second detection device 12 can detect the position and / or orientation of the ultrasound probe 5. Thus, the second detection device 12 can provide not only surface data of the patient 3, but also position and orientation data of the ultrasound probe 5 for controlling the device.

[0055] Furthermore, the Fig. The device shown in Figure 1 comprises a computing device 14. The computing device 14 can, for example, be used to automatically generate the patient model from the 3D data. It can also be used to generate the surface model of the patient 3 from the data of the second acquisition device 12. Furthermore, it can also be used to estimate a suitable position and orientation for positioning the ultrasound device 4 or its ultrasound probe 5 such that the target region (ROI) is visible. For this purpose, the computing device 14 can, for example, automatically calculate a starting position and starting orientation of the ultrasound device 4 on the object model or surface model so that the target region is visible. The starting position and starting orientation thus represent a rough pose of the ultrasound device 4.

[0056] The computing device 14 can also be used to automatically calculate a virtual ultrasound image of the partial area of ​​the object depending on the object model, the target area, and the (starting) position and (starting) orientation of the ultrasound device 4. For example, virtual b-mode ultrasound images are simulated based on the 3D data sets, taking into account the anatomy of the patient 3. From the virtual ultrasound images or target images, the one in which the target area is most clearly visible can be selected.

[0057] While a virtual ultrasound image is calculated using the computing device 14, a real ultrasound image of a partial area of ​​the patient 3 can be obtained using the ultrasound device 4. A deviation between the virtual ultrasound image or target image and the real ultrasound image can be used to reposition or reorient the ultrasound device 4. This deviation between the two ultrasound images can in turn be determined by the computing device 14. To do so, the computing device 14 uses, for example, a single or multiple landmarks in the two ultrasound images. An optimal position and orientation of the ultrasound device 4 can be achieved when the deviation is minimal or the two ultrasound images match.

[0058] The device may also include a feedback device that helps the user place the ultrasound probe. For example, this feedback device may be a monitor (in Fig. 1 not shown), which shows the planned and current position of the ultrasound probe 5 and, if necessary, also provides instructions on where it should be moved. Alternatively, a laser guidance system can be used to indicate an optimal location and orientation. Furthermore, a monitor 15 (the same or a different monitor) can be provided that displays the real-time ultrasound images and the virtual ultrasound images, thus giving the user the opportunity to adapt the real ultrasound image until it matches the virtual image (target image). Optionally, the views can be displayed side by side or overlaid with the 3D data.

[0059] The Fig. The device for positioning and orienting the ultrasound device 4 shown in Figure 1 can optionally also be used to position and / or orient an instrument, such as the ablation needle 9. This means that the second detection device 12, the computing device 14, and the feedback device can be used for this purpose.

[0060] An example system can be based on a 3D MR / CT imaging system for pre-interventional imaging, an optical tracking system, and an ultrasound device for real-time (“live”) imaging. Fig. 2 schematically illustrates a preferred application workflow. In a pre-interventional stage PI, 3D data (e.g., MR or CT data) of the region of interest are acquired in a step S1. This region of interest covers, for example, the abdominal body region including the liver. Optionally, a surface model can also be acquired or markers can be placed on the body surface. Furthermore, a patient model is created from the 3D dataset in step S1. For this purpose, the patient surface, the chest, the objects of interest such as the liver surface, the liver vessels and the bile ducts, as well as the target lesion(s) are segmented using known automatic and user-guided segmentation methods.

[0061] In a step S2, the target area or lesion is marked, for example, by a user 16 in the 3D object model. Optionally, the treatment can also be modeled. Using special visualization and interaction tools, the user 16 can, for example, plan the best path for placing the treatment devices (needles, cannulas, etc.) in relation to the target lesion, risk structures, and obscuring structures (such as the chest). As an additional function, the repositioning of patient 3 for better access to the treatment site could be modeled using special body deformation modeling and deformation techniques.

[0062] The extraction of the patient surface is highlighted separately here again as step S3. In the simplest case, the patient surface corresponds to the patient surface segmented in step S1. However, any deformations according to step S2 are also taken into account.

[0063] In an optical tracking stage OT, a navigation system for surface-based registration is initialized in step S4. In particular, multi-stage registration may be required to roughly align the patient and treatment model with the ultrasound imaging system used for image-based live treatment guidance. Patient 3 is positioned in the planning step as previously described.

[0064] According to step S5, the navigation system (second detection device 12) “scans” the surface of the patient 3 and creates a live surface model (avatar) of the patient’s body including the markers placed on the body.

[0065] In step S6, the body surface of the live image and the body surface of the pre-interventional patient model obtained in step S3 are compared or registered with each other. If necessary, the patient model is deformed using special deformation models. These models take the body surface captured by the navigation system (surface model) and the patient model from step S1 as input and further transfer the surface deformations to deeper body parts, e.g., using 3D extrapolation and optical flow methods.

[0066] In a further step S7, the target lesion (target area) is calculated with respect to the avatar (surface model of the navigation system) based on the registration of the two surfaces to each other.

[0067] In an ultrasound stage US, a real-time ultrasound imaging system is preferably initialized according to step S8. In a subsequent step S9, the ultrasound probe 5 is registered in the coordinate system of the navigation system. Thus, both the ultrasound system and the optical system have the same coordinates.

[0068] In a subsequent step S10, an optimal probe position and orientation are calculated with respect to the target area (lesion). A slice representation corresponding to the optimal probe position and orientation is calculated from the 3D dataset. The representation can either be based on the type of 3D dataset (CT or MR) or, optionally, a virtual ultrasound image can be generated from it. The treatment path, the pre-interventional 3D dataset, the patient model including the position of the lesion, the acquisition geometry of the ultrasound system to be simulated, and a model of the optimal imaging geometry of (oblique) ultrasound imaging planes such as in-plane and off-plane b-mode ultrasound images can be used as input variables for the calculation. The virtual ultrasound image can be displayed to the user 16.

[0069] In a further step S11, the probe position of the real ultrasound probe is optically tracked or detected, for example, with the second detection device 12.

[0070] In addition, in step S12, the actual real-time ultrasound image is made available to the user 16. The user can position the ultrasound probe 5 more precisely if necessary, as described below in connection with Fig. 3. For example, during ultrasound image acquisition, the user is asked to acquire ultrasound images as planned in the virtual ultrasound imaging session. The navigation system guides the user to a corresponding probe position and orientation (start position and start orientation). To do this, the navigation system determines (see Fig. 2 and Fig. 3) in step S7, the target area in relation to the avatar (surface model). Subsequently, the optimal starting position and starting orientation are determined for this in step S10. This starting position and starting orientation can be made available to the user 16. They can also be made available to the real ultrasound device, whose probe position and orientation are tracked in step S11. Furthermore, the starting position and starting orientation are used in step S13 to generate a virtual ultrasound image based on the pre-interventional 3D data set. In a further step S14, the virtual ultrasound image and the real ultrasound image can be displayed.

[0071] In a post-processing step, the virtual ultrasound image and the real or live ultrasound image can be compared. This can be done by the user 16 or automatically. If necessary, the user 16 or a corresponding algorithm will align the two ultrasound images according to step S15. Based on this alignment, the registration between avatar and target area is updated according to step S16. With this updated registration, the ultrasound device 4 can be repositioned or reoriented. Thus, after the starting position (first optimal position), a second optimal position and orientation with respect to the target area can be calculated according to step S17. This second or refined position and orientation can be used for steps S11 and S13 and made available to the user 16.Thus, the registration between the live data stream and the patient or treatment model can be further refined through user interaction. This refinement is demonstrated, for example, in the example shown in . Fig. 3 shown loop, which can be executed multiple times.

[0072] In a further optional step, the navigation system can be used to provide live guidance on how to guide an instrument (needle, cannula, etc.). Specifically, the navigation system can provide a user with live guidance regarding the position of the treatment device in relation to the patient or treatment model and the live ultrasound image stream.

[0073] This would make the following workflow possible, for example: 1. Loading a 3D imaging study with the target lesion delineated in the system. 2. Positioning of the patient on the operating table and creation of the avatar by the system. 3. The system estimates the registration of the 3D dataset with the avatar and determines the target position / orientation of the ultrasound probe for visualizing the landmarks. 4. The user places the probe and identifies the first landmark. 5. The system adjusts the registration based on the actual measurement and suggests additional positions / orientations to visualize additional landmarks / features / lesions. 6. After registration is complete, the system suggests the position / orientation for needle positioning and displays the ideal trajectory of the needle in the live image and the registered image. 7. Needle insertion is either assisted by a physical needle holder attached to the ultrasound probe or can be guided by the navigation / tracking system. 8. When the needle is optimally positioned in the first ultrasound pose, the system suggests a second position / orientation to confirm the needle positioning in 3D.

Claims

[1] Method for positioning and orienting a hand-held modality (4) by - Loading 3D data of an object (3), - automatic generation of an object model of the object (3) from the 3D data, - Specifying a target area in the object model, - Collecting surface data of the object, - automatically generating a surface model of the object (3) from the surface data, - Registering the surface model to the object model, - automatic calculation of a starting position and starting orientation of the hand-guided modality (4) on the object model depending on the target area, - automatically calculating a target image of a partial area of ​​the object (3) based on the object model, the target area and the starting position and starting orientation of the hand-guided modality (4), - capturing a real ultrasound image of the partial area of ​​the object (3) by means of the hand-held modality (4) in the starting position and starting orientation and - repositioning and / or reorienting (automatically / manually) the hand-held modality (4) based on a deviation between a virtual ultrasound image and the real ultrasound image, wherein the target image is the virtual ultrasound image. [2] Method according to claim 1, wherein the object (3) is a patient, and during the automatic generation of the object model, organs of the patient are automatically segmented. [3] Method according to one of the preceding claims, wherein the target image is a calculated ultrasound image, MR image or CT image. [4] Method according to one of the preceding claims, wherein in addition to the 3D data of the object (3), first surface data of the object obtained during acquisition of the 3D data are also loaded, a first surface model is generated from the first surface data, and the first surface model, which is implicitly registered with the object model, is used to register the surface model to the object model. [5] Method according to one of the preceding claims, wherein a 3D camera is used to capture the surface data of the object, which also captures a position and orientation of the hand-held modality. [6] The method of claim 5, wherein the 3D camera detects a position and orientation of an instrument optionally guided on the hand-held modality. [7] Method according to one of the preceding claims, wherein a path of an instrument (9) into the partial area of ​​the object (3) is planned on the basis of the object model, and the path is used for the automatic calculation of the target image. [8] The method of claim 7, wherein planning the path comprises repositioning the object (3). [9] Method according to claim 8, wherein the repositioning of the object (3) is taken into account by updating the object model by means of a deformation model. [10] Method according to one of the preceding claims, wherein an acquisition geometry of the hand-guided modality (4) is taken into account in the automatic calculation of the target image. [11] The method of claim 10, wherein the automatic calculation of the target image includes optimization with respect to different ultrasound image planes. [12] Method according to one of the preceding claims, wherein a user interaction is taken into account for the repositioning and / or reorientation of the hand-guided modality (3). [13] Method according to one of the preceding claims, wherein the deviation between the virtual ultrasound image and the real ultrasound image for repositioning and / or reorienting the hand-held modality (4) is determined using only a single landmark in both ultrasound images. [14] Method according to claim 13, wherein a further repositioning and / or reorientation of the hand-guided modality (4) is carried out with at least one further landmark in both ultrasound images. [15] Method according to one of the preceding claims, wherein the steps of acquiring a real ultrasound image on the one hand and repositioning and / or reorienting the hand-held modality on the other hand are alternately repeated several times in order to minimize a difference between the virtual and real ultrasound image. [16] Method according to one of the preceding claims, wherein the positioning and orientation of the hand-guided modality (4) mechanically defines a path for an instrument. [17] Method according to one of claims 4 to 16, wherein the loading of the 3D data of the object (3) is preceded by a capture of this 3D data by a modality and a simultaneous capture of the first surface data of the object. [18] Device with - a hand-held modality (4), - a first detection device (11) for loading 3D data of an object (3) and for specifying a target area in an object model, - a second detection device (12) for detecting surface data of the object (3), - a computing device (14) designed to * automatic generation of the object model of the object (3) from the 3D data, * automatic generation of a surface model of the object (3) from the surface data, * Registering the surface model to the object model, * automatically calculating a starting position and starting orientation of the hand-guided modality (4) on the object model based on the target area, and * automatically calculating a target image of a partial area of ​​the object (3) based on the object model, the target area and the starting position and starting orientation of the hand-guided modality (4), wherein - with the hand-guided modality (4) a real ultrasound image of the partial area of ​​the object (3) can be acquired by means of the hand-guided modality (4) in the starting position and starting orientation and - the device comprises a display device designed to visualize a deviation between the target image and the real ultrasound image. [19] The apparatus of claim 18, wherein the handheld modality is an ultrasound device or a single-photon emission computed tomography device. [20] Device according to claim 18 or 19, wherein the hand-guided modality (4) comprises a needle guide (8) for guiding an instrument, in particular an ablation needle (9) or a cannula. [21] Apparatus according to claim 20, comprising a planning device for planning a path of the instrument into the partial area of ​​the object (3) based on the object model, and a navigation device for navigating the instrument along the planned path depending on a real ultrasound image recorded after repositioning and / or reorienting the hand-held modality (4).

Images