Computer-implemented method, data processing system, user assistance system and computer program for assisting placement of medical device

Abstract

The invention relates to a computer-implemented method, a data processing system, a user assistance system and a computer program for assisting the placement of a medical device. Wherein a predetermined target path (3) of the medical device (1) is received. Subsequently, an assistance visualization (11) for assisting the user (4) in placing the medical device (1) on the object (2) is generated as a function of the target path (3), and the assistance visualization is shown by means of an augmented reality device (10). Wherein the auxiliary visualization (11) comprises a target line (12) according to a target position (13) of the medical device (1) and a target orientation of the medical device (1). Furthermore, the auxiliary visualization comprises a tolerance element (15) which visualizes a predetermined positional tolerance of the target position (13) and / or a predetermined orientation tolerance of the target orientation.

Description

Technical Field

[0001] This invention relates to a computer-implemented method for assisting in the placement of a medical device on an object, wherein a predetermined target path of the medical device is received. Furthermore, this invention relates to a data processing system configured to execute computer-implemented methods, user assistance systems, and computer program products. Background Technology

[0002] For medical interventions, even before the intervention begins, the correct initial position and orientation of the medical device on the subject can be critical. For example, when the subject is a patient and the medical device is a needle (which is to be injected into the patient's body), the initial position and orientation of the needle can be important. This is because the target path of the medical device may be inaccessible or the initial orientation may be irreversible, and corrections after the intervention begins may increase the risk of patient injury or the risk of intervention failure and / or delay.

[0003] Prior art related to this technical field is disclosed, for example, in US 2017 0 186 157 A1 and US 2022 0 409290 A1.

[0004] To optimize placement, particularly initial placement, medical interventions can be supported, for example, by an imaging system capable of generating images of the object. The target path can be estimated manually or calculated using a data processing system based on at least one image generated by the imaging system. With correct initial position and orientation, corrections to placement or orientation during the medical intervention can be avoided. Furthermore, correct initial placement and orientation can prevent or at least minimize the need for verification images of the object during the medical intervention. Better initial placement of medical devices can significantly improve surgical safety because sensitive areas of the patient that should not be affected by the medical intervention can be avoided. Additionally, if X-ray-based imaging is used, the X-ray dose applied to the patient and surrounding staff can be reduced, and the duration of the medical intervention can be shortened.

[0005] In addition, medical devices can also be, for example, medical scalpels or other cutting devices intended for cutting a portion of a patient's skin. Correct initial placement and orientation are just as important as described above to reduce the risk of injury from misplacement.

[0006] For example, a similar situation may arise when placing an ultrasound device on an object. Again, in this case, the correct initial location and orientation may be related to the duration and success of the medical intervention, even if both factors may be reversible.

[0007] Laser guidance can be used, for example, to assist in the placement of intervention needles on the patient's body according to a target path. A lasing beam may be projected onto the patient. Furthermore, a laser fan-shaped beam consisting of at least two laser regional beams can be projected from two directions, and the initial position and projection of the lasing beam on the patient are marked by the intersecting lines of at least two lasing beams, and the initial orientation is additionally marked. Such a projection system requires fixed infrastructure, i.e., at least two laser generation units, for a precise workflow. The disadvantage of this infrastructure is the increased complexity of the system, leading to high costs and potential usability bottlenecks. Due to the need for sufficient floor space and investment in the infrastructure, the potential usability bottleneck may stem from a limited number of projection systems. Summary of the Invention

[0008] The purpose of this invention is to help users place medical devices on objects while reducing the complexity of the system used compared to laser-guided methods.

[0009] This objective is achieved by means of the subject matter of the invention. Other implementations and preferred embodiments are the subject of this specification.

[0010] The present invention is based on the concept of supporting the precise placement of a medical device by using an extended reality device for visualizing a target line and a tolerance element for estimating the deviation of the medical device from the target line based on a predetermined target path.

[0011] According to one aspect of the invention, a computer-implemented method is provided for assisting in placing a medical device on an object (particularly on the outer surface of the object). A predetermined target path for the medical device is received. An auxiliary visualization for assisting a user in placing the medical device on the object is generated based on the target path, and this auxiliary visualization is displayed via an extended reality device. The auxiliary visualization includes visualization of a target line, particularly a target line, based on the target position and target orientation of the medical device. Furthermore, the auxiliary visualization includes the visualization of tolerance elements, particularly tolerance elements that visualize predetermined positional tolerances of the target position and / or predetermined orientation tolerances of the target orientation.

[0012] Unless otherwise stated, all steps of the computer-implemented method can be performed by a data processing system comprising at least one data processing device. In particular, at least one data processing device is configured or adapted to perform the steps of the computer-implemented method. For this purpose, at least one data processing device may, for example, store a computer program containing instructions that, when executed by at least one data processing device, cause the at least one data processing device to perform the computer-implemented method. The expressions "data processing system" and "at least one data processing device" are used interchangeably herein and hereinafter. This also applies to the corresponding expressions derived therefrom.

[0013] In cases where at least one data processing device comprises two or more data processing devices, certain steps performed by at least one data processing device can also be understood as causing different data processing devices to perform different steps or different parts of steps. In particular, it is not required that each data processing device fully perform these steps. In other words, the execution of these steps can be distributed across two or more data processing devices.

[0014] In particular, extended reality devices can be considered as part of a data processing system. In other words, displaying auxiliary images via an extended reality device is considered a computer-implemented step.

[0015] Determining the target path is not necessarily part of a computer-implemented method, but it may be in some implementations. For example, the target path can be determined based on at least one image of an object or a portion of an object generated by an imaging system.

[0016] From each implementation of the computer-implemented method, a corresponding implementation of a method for assisting in placing a medical device on an object is obtained by including steps such as generating at least one image of the object or a portion thereof by an imaging system, the method not being purely computer-implemented.

[0017] The placement of a medical device may include, but is not necessarily limited to, a manual process. In other words, the medical device may be positioned and / or guided along an object by a user's hand. The placement of a medical device is not necessarily part of a computer-implemented method, but in some implementations it may be, particularly the automated portion of the placement and / or the control of the components required for placement.

[0018] In particular, in some embodiments, the medical device may include an intervention needle, and the computer-implemented method is a computer-implemented method for assisting in the placement of the intervention needle (e.g., placement on a patient, particularly on the patient's skin). In other embodiments, the medical device may include a scalpel, other medical cutting or puncture device, or imaging device, such as an ultrasound probe, which may require direct contact with the patient's skin. It should be emphasized that the computer-implemented method or the method according to the invention does not include any intervention performed using a medical device.

[0019] The predetermined target path can be a path starting from the surface of an object and leading to a point of interest inside the object. For example, the point of interest might be a part of the object that requires treatment or from which a sample should be taken. In the case of a patient, the point of interest might be a point within the patient's organ or other part. The target path can be estimated or calculated manually, for example, by a data processing system, and can be based on imaging data of the object that was generated prior to the medical invention. The imaging data can be generated by an imaging system. The target path can consider other areas within the object that should not be affected by the medical invention. In this case, the target path can avoid these areas, and the data processing system can find alternative paths.

[0020] Alternatively, the target path can be determined, for example, manually or semi-manually by personnel supported by a data processing system.

[0021] In particular, the user can be the operator of the medical intervention, especially a doctor or physician assistant. Specifically, they can be a single person being assisted by visual aids and operating a medical device. Additionally, other individuals can also receive assistance through visual aids, for example, acting as an auditor or as support for a single person.

[0022] The auxiliary visualization can be generated by a data processing system, and the shape of the auxiliary visualization can be the result of calculations based on the target path. In some embodiments, the auxiliary visualization can be the result of calculations additionally based on the type of medical device, the shape and / or orientation and / or location of the object, and the type of medical intervention.

[0023] Extended reality devices can include or consist of augmented reality devices. In particular, augmented reality devices can be configured to combine portions of the real-world environment with computer-generated content. Specifically, a user may simultaneously see real objects and three-dimensional or two-dimensional computer-generated content. In this document, computer-generated content includes at least assisted visualization.

[0024] Extended reality devices may also include virtual reality devices or may consist of virtual reality devices. Virtual reality devices can be configured to simultaneously display images of real objects, medical devices, and assistive visualizations.

[0025] Computer-generated content can be images, geometric shapes, text, or other visual effects. In particular, the augmented reality device can be a wearable device that may at least partially include transparent areas, similar to a pair of glasses. The wearable device can be configured to display the computer-generated content at a location relative to a real object and allow the user to see the real object through the transparent areas. The computer-generated content may be displayed on a different layer than the real object. Specifically, the position of the augmented reality device relative to the real object can be moved. Even during or after the position movement, the combined arrangement of the computer-generated content and the real object may remain unchanged.

[0026] Extended reality devices may also include, for example, a 3D image projection device for projecting 3D images or a holographic display for projecting holograms, which can be configured to display computer-generated content at a location associated with a real object and allow both the user and an additional user to see the real object and the computer-generated content simultaneously. This may be possible without requiring the user or additional user to use or wear a device such as a pair of glasses.

[0027] Alternatively, extended reality devices may also include display devices or, for example, computer screens. The computer screen can display images of real objects and computer-generated content. In this case, the image of the real object can be generated by a camera system and combined with the computer-generated content.

[0028] In other embodiments, the augmented reality device may include an augmented reality headset, augmented reality glasses, a smartphone, a tablet device, or a camera with a display. In particular, the augmented reality device may include, for example, a head-mounted display.

[0029] Assistive visualization can guide users to change the position and orientation of a medical device to align it with the visualization. Specifically, the medical device may include a pointed tip on a first side, and its shape may extend toward a second side opposite the first side, for example, along a straight line or substantially along a straight line. In this case, the assistive visualization can guide the user to place the medical device parallel and consistent with a target line. Specifically, the target line may point to a target location on the object or patient, particularly on the patient's skin, or begin at a target location on the object or patient. The assistive visualization can highlight the target location along the target line to simplify the placement of the medical object. The target line may extend parallel to the target orientation, and its length may be limited to the corresponding length of the medical object. To increase visibility, the target line may be longer than the corresponding length of the medical object.

[0030] Tolerance elements can visualize positional and orientation tolerances, which are defined prior to medical intervention and relate to corresponding deviations from the target line. Positional and orientation tolerances can, for example, depend on the type of medical device, the type of medical intervention, the object, and / or user settings. Tolerance elements can guide the user to adjust the position and orientation of the medical device to fit the target line, particularly when the user keeps the tip of the medical device in a constant position. Tolerance elements can also change shape or pattern based on the user's adjustment movements. This effect can be considered a reward and leads to a reduction in adjustment time.

[0031] The target line can be determined based on the target path, for example, through linear interpolation of the target path. For instance, since the target path may extend inside the object, the target line may be a continuation of the target path outside the object. In particular, the target line can be a straight extension of the target path. Optionally, the target line and the target path can also depend on the shape of the medical device. For example, a curved medical device may follow a different target path to reach the point of interest compared to a straight medical device.

[0032] Tolerance values ​​can be determined by the user or predetermined in other ways. Tolerance elements can be generated from tolerance values ​​and, for example, target paths.

[0033] The advantages of this invention lie in its intensive and comprehensive assistance to the user, thereby shortening the duration of medical interventions. Assisted visualization can improve the accuracy of medical device placement because the object and the assisted visualization are simultaneously visible to the user, and continuous feedback on the user's behavior is possible. Furthermore, the computer-implemented method eliminates the need for fixed or permanent installation of the device in medical buildings or facilities, such as laser-guided systems within hospital imaging systems. Extended reality devices offer the opportunity for use alongside objects, such as patients. The implementation workload is significantly reduced compared to fixed installations.

[0034] According to several implementations, the tolerance element includes an arc of a first circle, particularly a visualization of the arc of the first circle, the first circle indicating positional tolerance and / or orientation tolerance, and the target line passing through the center of the first circle.

[0035] In particular, the arc of the first circle can also be a complete circle. In other words, the angle of the arc of the first circle can be in the range of [0º, 360º]. Positional tolerances and / or orientation tolerances can be visualized through the radius of the first circle. In particular, the larger the positional tolerances and / or orientation tolerances, the larger the radius of the first circle.

[0036] In some implementations, the shape or diagram of the arc of the first circle can be changed according to the position and orientation of the medical device. In particular, the shape and diagram of the arc of the first circle may change if the position and orientation of the medical device are outside or inside the first circle. For example, when the position and orientation of the medical device are outside the first circle, the arc of the first circle can be represented in red, and when the position and orientation of the medical device are inside the first circle, the arc of the first circle can be represented in green.

[0037] The center of the first circle can be located at the target location or at another location on the target line. The first circle can lie in a plane and the target line can be, for example, perpendicular to that plane.

[0038] The tolerance element may include additional arcs of the first circle, specifically a visualization of additional arcs of the first circle. A gap may exist between this arc and the additional arc. In some implementations, this may also be extended to more than two arcs of the first circle.

[0039] The advantage of this implementation is that users can receive indications of orientation and position tolerances and therefore find the correct position of the medical device more quickly. The combination of the arc of the first circle and the target line can provide a comprehensive description of the target position and orientation of the medical device.

[0040] According to several implementations, the tolerance element includes an arc of a second circle, particularly a visualization of the arc of the second circle, which indicates a predetermined additional first position tolerance of the target position and / or a predetermined additional first orientation tolerance of the target orientation, wherein the second circle is concentric with the first circle and the radius of the second circle is greater than the radius of the first circle.

[0041] In particular, the arc of the second circle can also be a complete circle. In other words, the angle of the arc of the second circle can be in the range of [0º, 360º]. Furthermore, the first positional tolerance can be greater than the positional tolerance, and the second orientation tolerance can be greater than the orientation tolerance. For example, the second circle and the first circle can lie in the same plane.

[0042] The advantage of this implementation is that the user receives progressive feedback on the deviation of the medical device from the target position and orientation. The arc of the first circle can be understood as an enhanced position compared to the arc of the second circle. To help the user find the correct position of the medical device, the user receives qualitative feedback on the position and orientation of the medical device and can estimate the distance that needs to be overcome to place the medical device on the object.

[0043] According to several implementations, the area of ​​the ring between the first circle and the second circle is displayed differently compared to the area inside the first circle and compared to the area outside the second circle.

[0044] A ring is defined as the region between the second circle and the first circle and can be shaped as a ring around the center of the first circle. In particular, the ring can be a two-dimensional ring.

[0045] For example, the rings can be displayed in different colors or with different background patterns, particularly shadow patterns, compared to the area within the first circle and the area outside the second circle. Furthermore, the area within the first circle can be displayed with a signal color to indicate to the user that the medical device is approaching the target line.

[0046] The advantage of this implementation is that users can easily distinguish the position and orientation of the medical device relative to the target line. The tolerance element can be understood as a shooting target, while the area within the first circle can be analogous to the bullseye. This illustration allows even inexperienced users to operate the computer-implemented method without explanation.

[0047] According to several implementations, the tolerance element includes an arc of an additional circle, particularly a visualization of the arc of the additional circle, which indicates a predetermined additional second positional tolerance for the target position and / or a predetermined additional second orientation tolerance for the target orientation. The center of the additional circle is shifted relative to the center of the first circle along the target line.

[0048] In other words, the arc of another circle appears on a different plane than the plane of the arc of the first circle or the plane of the arc of the second circle. In particular, the other plane may be parallel to the plane of the first circle.

[0049] In particular, the arc of the other circle can also be a complete circle. In other words, the angle of the arc of the other circle can be in the range of [0º, 360º].

[0050] The advantage of this implementation is that users can receive positional and orientation feedback, for example, regarding the first and second sides of the medical device. This enhances the user's 3D orientation and accelerates the placement of the medical device on the object.

[0051] According to several implementations, the tolerance element includes a cone, particularly a visualization of a cone, wherein the vertex of the cone is located at the target position, the target line is located on the axis of the cone, and the angle of the cone depends on the orientation tolerance, or the tolerance element includes a truncated cone of the cone.

[0052] In other words, a cone can represent a three-dimensional shape, the vertex of the cone can represent the position of the tip of a medical device, and the angle of the cone can represent the deviation of the target line.

[0053] The advantage of this implementation is that the cone shows the deviation of the target position and orientation in a three-dimensional manner. This helps the user place the medical device within the cone. If the tolerance element includes a truncated cone and the length of the truncated cone is less than the length of the medical device, then the user can see a second side of the medical device.

[0054] According to several implementations, the tolerance element includes an additional cone, particularly a visualization of an additional cone, wherein the apex of the additional cone is located at the target position, the axis of the additional cone is the same as the axis of the cone, and the angle of the additional cone is greater than the angle of the cone and depends on a predetermined additional third orientation tolerance of the target orientation.

[0055] In other words, a cone can be shown within another cone. This cone and the other cone can be shown in different colors and / or different patterns.

[0056] The advantage of this approach is that it provides users with progressive 3D feedback and accelerates the placement of medical devices on objects.

[0057] Depending on several implementations, the medical device includes a medical intervention needle.

[0058] Medical intervention needles can be used for medical interventions and, for example, for intradermal injection into a patient. In particular, the medical intervention needle can be used for percutaneous interventions, wherein the medical intervention needle can be, for example, a rigid needle.

[0059] The advantage of this approach is that it enhances the placement of the medical intervention needle on the patient's skin and minimizes the risk of injury. This is particularly relevant for rigid needles, where correction of the needle's orientation may be limited or impossible.

[0060] Depending on the implementation method, the target path may include a straight line.

[0061] In particular, the target path may consist of straight lines. For example, a medical device may be a rigid medical device or a rigid needle, and the target path may only allow straight lines from the target location to a point of interest within the object.

[0062] The advantage of this implementation is that the computer-based method can enhance the placement of rigid medical devices on objects.

[0063] According to several implementation methods, a target path is determined based on at least one predetermined image of an object or a portion of an object generated by an imaging system.

[0064] In particular, at least one image of the object may show points of interest within the object and additionally show other areas within the object that should be protected from medical intervention.

[0065] An imaging system can be configured to generate an image of an object or a portion thereof, particularly, for example, an image of the object's interior. The target path can be determined, for example, by a data processing system and can be based on at least one image.

[0066] The advantage of this approach is that the target path is available at the start of the medical intervention. In particular, calculating the target path based on at least one image of the object can reduce the risk of patient injury.

[0067] Depending on the implementation, imaging systems include medical imaging systems, such as computed tomography (CT) imaging systems.

[0068] In particular, the imaging system may be the same as a medical imaging system or a computed tomography (CT) imaging system. In other embodiments, the imaging system may include a cone beam CT system, a mammography system, a dental X-ray system, a fluorescence system, an angiography system, a C-arm system, or a conventional X-ray machine.

[0069] The advantage of this approach is that the computer-based method can be implemented using a medical imaging system that can generate high-resolution images of the patient or at least a portion thereof, thereby improving the accuracy of medical device placement.

[0070] According to another aspect of the present invention, a data processing system is provided. This data processing system is configured to perform a computer-implemented method according to the present invention.

[0071] In this disclosure, the terms "data processing system" and "at least one data processing system" are used interchangeably. A data processing system can be particularly understood as a data processing system containing processing circuitry. Therefore, a data processing system can, in particular, process data to perform computational operations. This can also include operations performing index accesses on data structures, such as lookup tables (LUTs), and data processing procedures implemented in hardware.

[0072] In particular, the data processing system may include one or more computers, one or more microcontrollers and / or one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more system-on-a-chip (SoCs). The data processing system may also include one or more processors, such as one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, especially one or more digital signal processors (DSPs). The data processing system may also include a physical or virtual cluster of computers or other units.

[0073] In various embodiments, the data processing system includes one or more hardware and / or software interfaces and / or one or more storage units.

[0074] The storage cell can be implemented as a volatile data memory, such as dynamic random access memory (DRAM) or static random access memory (SRAM), or as a non-volatile data memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or flash EEPROM, ferroelectric random access memory (FRAM), magnetoresistive random access memory (MRAM) or phase-change random access memory (PCRAM).

[0075] According to another aspect of the invention, a user assistance system is provided for assisting in placing a medical device on an object. The user assistance system includes a data processing system and an augmented reality device.

[0076] According to several implementations, the user assistance system includes an imaging system configured to generate at least one image of an object or a portion thereof, and the user assistance system is configured to calculate a target path based on the at least one image.

[0077] Further implementations of the user assistance system according to the invention are directly derived from various embodiments of the computer-implemented method according to the invention, and vice versa. In particular, the various features, corresponding explanations, and advantages associated with the various implementations of the computer-implemented method according to the invention can be similarly transferred to the corresponding implementations of the user assistance system according to the invention. In particular, the user assistance system according to the invention is designed or programmed to perform the computer-implemented method according to the invention. In particular, the user assistance system according to the invention performs the computer-implemented method according to the invention.

[0078] According to another aspect of the present invention, a computer program product is provided. The computer program product includes instructions that, when executed by a data processing system, cause the data processing system to perform a computer-implemented method according to the present invention.

[0079] For example, the instructions can be provided as program code. The program code can be provided, for example, as binary code or source code of an assembler and / or a programming language (such as C) and / or a program script (such as Python).

[0080] The computer program product may be a computer program including instructions or a computer-readable storage medium storing the computer program.

[0081] Further features and combinations of features of the invention can be obtained from the accompanying drawings, their description, and the claims. In particular, other implementations of the invention may not necessarily include all the features of one of the claims. Other embodiments of the invention may include features or combinations of features not recited in the claims. Attached Figure Description

[0082] The present invention will now be explained in detail with reference to specific exemplary implementations and corresponding schematic diagrams. In the accompanying drawings, elements that are identical or functionally identical can be represented by the same reference numerals. Descriptions of identical or functionally identical elements need not be repeated in different drawings.

[0083] In the attached diagram,

[0084] Figure 1 This schematically illustrates various aspects of exemplary implementations of a computer-based method for assisting in the placement of a medical device according to the present invention; and

[0085] Figure 2 A schematic flowchart illustrating another exemplary implementation of a computer-based method for assisting in the placement of a medical device according to the present invention; and

[0086] Figure 3 The illustration schematically depicts various aspects of the auxiliary visualization of another exemplary implementation of the computer-implemented method for assisting in the placement of a medical device according to the present invention; and

[0087] Figure 4 The illustration schematically depicts various aspects of the auxiliary visualization of another exemplary implementation of the computer-implemented method for assisting in the placement of a medical device according to the present invention; and

[0088] Figure 5 The illustration schematically depicts various aspects of the auxiliary visualization of another exemplary implementation of the computer-implemented method for assisting in the placement of a medical device according to the present invention; and

[0089] Figure 6 The illustration schematically shows various aspects of an auxiliary visualization of another exemplary implementation of a computer-based method for assisting in the placement of a medical device according to the present invention. Detailed Implementation

[0090] Figure 1This illustration schematically depicts an exemplary implementation of a computer-based method according to the present invention for assisting in placing a medical device 1 onto an object 2. A predetermined target path 3 of the medical device 1 is received. Subsequently, an auxiliary visualization 11 for assisting a user 4 in placing the medical device 1 onto the object 2 is generated based on the target path 3 and displayed by an extended reality device 10. The auxiliary visualization 11 includes a target line 12 based on a target position 13 and a target orientation of the medical device 1. Furthermore, the auxiliary visualization 11 includes a tolerance element 15 that visualizes a predetermined positional tolerance of the target position 13 and / or a predetermined orientation tolerance of the target orientation.

[0091] User 4 may be, for example, a doctor or a physician's assistant. User 4 can hold the medical device 1 (which may be, for example, an intervention needle) with one hand and can manually change the position and orientation of the medical device 1. The position of the medical device 1 can be defined, in particular, by the position of its tip, which may be located on a first side of the medical device 1. The first side may be referred to as the application side, which is intended for use on a patient. The application side may, for example, include the tip of an intervention needle, the tip or blade of a scalpel, or the active side of an ultrasound probe. The orientation of the medical device 1 may be defined, for example, as a direction along a straight line that begins at this position and ends at a second side of the medical device 1 opposite to the first side.

[0092] The augmented reality device 10 may include, for example, a wearable device, particularly an AR headset, which can be configured to display an auxiliary visualization 11 on a first layer. The second layer may be transparent and allow the user 4 to align the auxiliary visualization 11 with the medical device 1 and the object 2. Specifically, the augmented reality device 10 can follow the user 4's head movements and / or gaze, enabling the user 4 to locate the object 2 and the medical device 1 while simultaneously seeing the auxiliary visualization 11.

[0093] The extended reality device 10 may also include, for example, a 3D image projection device for projecting 3D images or a holographic display for projecting holograms. The 3D image projection device or holographic display may be configured to display auxiliary visualization 11 at a location associated with the medical device 1 and the object 2.

[0094] Alternatively, the extended reality device 10 may include a display device or, for example, a computer screen. The computer screen may be configured to display the object 2 and the auxiliary visualization 11. The image of the medical device 1 and / or the image of the object 2 may be generated by a camera system and displayed in conjunction with the auxiliary visualization 11.

[0095] For example, the auxiliary visualization 11 may include computer-generated content. Using the extended reality device 10, the user 4 can see and compare the position of the medical device 1 with the target position 13. Furthermore, the user 4 can see and compare the orientation of the medical device 1 with the target line 12. Additionally, the user 4 can determine the deviation between the position and the target position, and the deviation between the orientation and the target orientation, by comparing with the tolerance element 15.

[0096] The target path 3 can be manually estimated or calculated, for example, based on at least one image 7 of the object 2, which can be generated by the imaging system 20. In particular, at least one image 7 can show a point of interest within the object 2. This point of interest may require treatment or may be related to sample acquisition. The target path 3 can be calculated based on the location of the point of interest and the location of other parts of the object 2 that should be avoided or protected. The target path 3 can include a straight line within the object 2, originating at a target location 13. The target line 12 can include another straight line, which, for example, can extend the target path 3 to the outer portion of the object 2 and can also originate from the target location 13. In particular, the target location 13 can be the unique intersection of the target path 3 and the target line 12.

[0097] Figure 2 A schematic flowchart illustrating an exemplary implementation of the computer-implemented method is provided. The computer-implemented method may include manual estimation or calculation of the target path 3, which may depend on at least one image 7 generated by the imaging system 20, such as a CT system. In some embodiments, the generation of at least one image 7 may, for example, be part of the computer-implemented method. The calculation of the target path 3 may be performed by the data processing system 21. The data processing system 21 may generate an auxiliary visualization 11 including the target line 12 and tolerance elements 15.

[0098] Alternatively, the auxiliary visualization 11 can be generated by another data processing system. The data processing system 21 can also determine the target location 13 and the target orientation. The tolerance element 15 can depend on the location tolerance and / or orientation tolerance. The location tolerance and orientation tolerance can, for example, be configured by the user 4 to optimize the auxiliary visualization 11 for the required application. The required application might be a medical intervention. A first medical intervention may require high precision, and therefore the initial setting of the location tolerance and orientation tolerance can be a first value. A second medical intervention may not have the same requirements, and therefore a second setting of the location tolerance and orientation tolerance can be a second value, which may be greater than the first value.

[0099] The auxiliary visualization 11 may include target lines 12 and tolerance elements 15 and can be displayed by the augmented reality device 10. In addition, the augmented reality device 10 may also display target position 13.

[0100] Figure 3and Figure 4 The illustration schematically depicts the placement of medical device 1 according to another exemplary implementation of the computer-implemented method. Two figures show a combined image of the assistive visualization 11 and medical device 1. This combined image can be displayed on the extended reality device 10 to assist user 4 in placing medical device 1 on object 2. Figure 3 The first orientation of the medical device 1 is shown. The first orientation may be an orientation position where the medical device 1 is not aligned with the target line 12 and / or the target position 13. Figure 4 The second orientation of the medical device 1 is shown. The second orientation may be an orientation in which the medical device 1 is aligned with the target line 12 and the target position 13.

[0101] Target location 13 can be a portion of the surface of object 2, which can be a location on the patient's skin. Target line 12 can originate from target location 13. Target path 3 can also originate from target location 13. Target line 12 can be an extension of target path 3, which can extend within object 2, while target line 12 extends outside object 2.

[0102] Tolerance element 15 may include an arc of a first circle 5a. The center 6a of the first circle 5a may be located on the target line 12. The arc of the first circle 5a may enclose a first angle. The value of the first angle may be greater than zero and may be a maximum of 360º. For the first angle 360º, the arc of the first circle 5a may be the same as the first circle 5a. The arc of the first circle 5a, and in particular the radius of the first circle 5a, may depend on the positional tolerance of the target location and / or the orientation tolerance of the target orientation. Therefore, the user 4 may be able to limit the deviation. For example, if the user can see the medical device 1 within the arc of the first circle 5a, meaning that the main direction of the medical device 1 from the first side to the second side passes through the arc of the first circle 5a, then the deviation may be less than the positional tolerance and / or the orientation tolerance. On the other hand, if the user can see the medical device 1 outside the arc of the first circle 5a, meaning that the main direction extends outside the arc of the first circle 5a, then the deviation may be greater than the positional tolerance and / or the orientation tolerance. Tolerance element 15 may include additional arcs of the first circle 5a.

[0103] Furthermore, the tolerance element 15 may include an arc of a second circle 5b, which is concentric with the first circle 5a and extends in the same plane as the first circle 5a. The radius of the second circle 5b may be larger than the radius of the first circle 5a. The radius of the second circle 5b may be determined based on a predetermined additional first positional tolerance and a predetermined additional first orientation tolerance.

[0104] The tolerance element 15 may also include a ring 8 between the first circle 5a and the second circle 5b, which may be displayed differently from the region 9a inside the first circle 5a and from the region 9b outside the second circle 5b. Therefore, the user 4 can distinguish and determine the deviation from the target line by identifying the medical device 1 within one or the other region 9a, 9b or the ring 8. The ring may be illustrated differently from the region 9a inside the first circle 5a and the region 9b outside the second circle 5b in terms of color, line shadow, shading, or brightness.

[0105] The tolerance element 15 may also include an arc of an additional circle 5c. The center 6b of this additional circle 5c may be located on the target line 12 and may be displaced a certain distance from the center 6a of the first circle 5a, which is the same as the center 6a of the second circle 5b. The radius of the additional circle 5c may be determined based on a predetermined additional second positional tolerance and a predetermined additional second orientation tolerance.

[0106] Because of the three-dimensional view, user 4 will be able to locate the position and orientation of medical device 1 and compare that position and orientation with target line 12. This distance can be the same as or less than the length of medical device 1 in the main direction.

[0107] Figure 5 The illustrations depict computer-implemented methods, particularly alternative implementations of the assisted visualization 11. (For...) Figure 4 All the explained features can be applied to Figure 5 The embodiment shown.

[0108] Figure 6 The diagram schematically illustrates another implementation of the computer-implemented method. Tolerance element 15 may include a cone 30, wherein the apex 31 of the cone 30 may be positioned in the same location as the target position 13, or the position of the apex 31 of the cone 30 may depend on a positional tolerance. The axis 32 of the cone 30 may lie on the target line 12, and the angle 33 of the cone 30 may depend on an orientation tolerance. Tolerance element 15 may also include a further cone 40, wherein the apex 41 of the further cone 40 may lie in the same location, or the position of the apex 41 of the further cone 40 may depend on a predetermined additional third positional tolerance. The axis 42 of the further cone 40 may lie on the target line 12, and the angle 43 of the further cone 40 may depend on a predetermined additional third orientation tolerance.

[0109] The volume defined by cone 30 and the additional cone 40 may be displayed differently compared to the volume inside cone 30 and the volume outside additional cone 40.

[0110] In several embodiments, the present invention supports precise positioning of the intervention needle by using extended reality glasses integrated into dedicated CT intervention software.

[0111] In several embodiments, the radiologist, who positions the intervention needle within the patient, observes the patient through AR glasses. A pre-planned target path, designed in the CT intervention software, is projected as a line into the radiologist's 3D AR view, similar to projecting a laser onto the patient using a laser-guided system. The user can align the intervention needle in the AR view with a virtual needle that may extend parallel to the target path to precisely position the needle, matching the target path (potentially an entry point) at the target location and two angles along the target path.

[0112] In several embodiments, the target path can be visualized in different ways than a simple line, which may make aligning the intervention needle with the target path easier. For example, the target disc can be visualized every few centimeters along the target line.

[0113] Needle positioning without a guide system is often inaccurate, especially for bi-angled needles. This is also time-consuming, as it requires manual distance measurement on both the software and the patient's skin.

[0114] Both navigation and laser-guided systems offer fast and accurate workflows for needle positioning. However, the availability of these systems can be limited, primarily due to the high cost of dedicated hardware (cameras, laser projectors, etc.) and additional software. Furthermore, navigation systems typically require consumables (reference markers, dedicated needle holders, and caps), which can be expensive. Non-integrated navigation and laser-guided systems often also require additional time and floor space for setup, calibration, and the exchange of planning data between the CT scanner and the system.

Claims

1. A computer-implemented method for assisting in placing a medical device (1) on an object (2), wherein, - The predetermined target path (3) for receiving the medical device (1); - Generate an auxiliary visualization (11) based on the target path (3) to assist the user (4) in placing the medical device (1) on the object (2), and display it through an extended reality device (10); - The auxiliary visualization (11) includes: i) Based on the target position (13) of the medical device (1) and the target line (12) of the target orientation of the medical device (1), and ii) Tolerance element (15) which visualizes a predetermined positional tolerance of the target position (13) and / or a predetermined orientation tolerance of the target orientation.

2. The computer-implemented method according to claim 1, wherein, The tolerance element (15) includes an arc of a first circle (5a) indicating the positional tolerance and / or the orientation tolerance, and the target line (12) passes through the center (6a) of the first circle (5a).

3. The computer-implemented method according to claim 2, wherein, The tolerance element (15) includes an arc of a second circle (5b) indicating a predetermined additional first position tolerance of the target position (13) and / or a predetermined additional first orientation tolerance of the target orientation, wherein the second circle (5b) is concentric with the first circle (5a) and the radius of the second circle (5b) is greater than the radius of the first circle (5a).

4. The computer-implemented method according to claim 3, wherein, The loop (8) between the first circle (5a) and the second circle (5b) is displayed differently from the region (9a) inside the first circle (5a) and from the region (9b) outside the second circle (5b).

5. The computer-implemented method according to any one of claims 2 to 4, wherein, The tolerance element (15) includes an arc of an additional circle (5c) indicating a predetermined additional second position tolerance of the target position (13) and / or a predetermined additional second orientation tolerance of the target orientation, wherein the center (6b) of the additional circle (5c) is displaced along the target line (12) relative to the center (6a) of the first circle (5a).

6. The computer-implemented method according to any one of the preceding claims, wherein, The tolerance element (15) includes a cone (30), wherein the vertex (31) of the cone (30) is located at the target position (13), the target line (12) is located on the axis (32) of the cone (30), and the angle (33) of the cone (30) depends on the orientation tolerance, or the tolerance element (15) includes a truncated cone of the cone (30).

7. The computer-implemented method according to claim 6, wherein, The tolerance element (15) includes an additional cone (40) wherein the vertex (41) of the additional cone (40) is located at the target position (13), the axis (42) of the additional cone (40) is the same as the axis (32) of the cone (30), and the angle (43) of the additional cone (40) is greater than the angle (33) of the cone (30) and depends on a predetermined additional third orientation tolerance of the target orientation, or the tolerance element (15) includes a truncated cone of the additional cone (40).

8. The computer-implemented method according to any one of the preceding claims, wherein, The medical device (1) includes a medical intervention needle.

9. The computer-implemented method according to any one of the preceding claims, wherein, The target path (3) includes a straight line.

10. The computer-implemented method according to any one of the preceding claims, wherein, The target path (3) is determined based on at least one predetermined image (7) of the object (2) or a portion thereof generated by the imaging system (20).

11. The computer-implemented method according to claim 10, wherein, The imaging system (20) includes a medical imaging system or a computed tomography imaging system.

12. A data processing system (21) configured to perform a computer-implemented method according to any one of the preceding claims.

13. A user assistance system for assisting in placing a medical device (1) on an object (2), comprising a data processing system (21) according to claim 12 and an extended reality device (10).

14. The user assistance system according to claim 13, wherein, The user assistance system includes an imaging system (20) configured to generate at least one image (7) of the object (2) or a portion thereof, and the user assistance system is configured to determine the target path (3) based on the at least one image (7).

15. A computer program product comprising instructions that, when executed by a data processing system (21), cause the data processing system (21) to perform a computer-implemented method according to any one of claims 1 to 11.

Images