Enhanced superposition apparatus

Abstract

The invention relates to an enhanced overlay device (100), comprising: an image source (10) for providing real-time image data (Di) of a region of interest (roi) in the form of a stream of real-time image pixels; a data server (20) for providing the superimposed data (Do); a synthesizer device (30), where the synthesizer device (30) is a hardware device, where the functionality of the hardware device is implemented in hardware and for providing display data (Dd) in the form of a stream of real-time image pixels by superimposing the provided real-time image data (Di) with the provided superimposition data (Do); wherein the synthesizer device (30) is configured to determine a restricted area (41) in the provided real-time image data (Di), wherein the provided overlay data (Do) comprises restricted area overlay data (Dor) associated with the restricted area (41) of the real-time image data (Di) and free overlay data (Dof) associated with areas of the real-time image data (Di) outside the restricted area (41); wherein the synthesizer device (30) is configured to limit superimposition with the restricted area superimposition data (Dor); and a display device (40) for receiving and displaying the provided display data (Dd).

Description

Technical Field

[0001] The present invention relates to an enhanced overlay device with hardware-based functional safety, in particular for use in a digital operating room, and a method for enhanced overlay with hardware-based functional safety. Background Art

[0002] Enhancement / video synthesis products (also called enhancement overlay devices) generally allow the user to overlay a video stream with any kind of additional data. Therefore, in the context of an operating room with digital support (also called a digital operating room), enhancement overlay devices may be very helpful to medical personnel. Enhancement overlay devices like Buzz Virtual from Brainlab AG overlay additional patient data (called overlay data) onto the main video while delivering documentation needs on the display. This is done by processing the main video with software (especially by using a graphics card), however, hiding important areas of the main video with overlay data may lead to harm to the patient. Common systems therefore provide a software solution with an "overlay off" button to deactivate all overlays so that the display only shows the main video. In the event of a software error, this function may lead to an extended blocking of the main video with the overlay data, or in the worst case a complete switch-off of the displayed information. Such an event at the wrong time of operation may lead to significant harm to the patient. Alternatively, the display is provided with a bypass cable and the user can switch to a second input, but the source must provide an additional output and the user must find the input selector switch quickly enough. Performing such an action during operation is highly undesirable.

[0003] The drawback of current solutions is that software solutions are not very secure. Software solutions essentially bypass or disable the entire device, which results in additional installation / material effort and the need to train users to enable bypass. In several installation scenarios, this is difficult to achieve. For example, in scenarios where there is no backup video output on the image source available for switching in the event of a software failure, or in scenarios where the display is out of reach of the user (ceiling mount, wall mount, technical room), bypassing cannot be achieved.

[0004] An object of the present invention is to provide an enhancement overlay device which safely prevents accidental overlays in a main video.

[0005] The present invention can be used for intraoperative augmentation overlay in digital operating room procedures, for example in combination with a system for augmentation overlay, such as Buzz Virtual from Brainlab AG.

[0006] Various aspects, examples and exemplary steps of the invention and its embodiments are disclosed below.Different exemplary features of the invention may be combined according to the invention where technically advantageous and feasible.

[0007] Exemplary brief description of the present invention

[0008] In the following, a brief description of specific features of the present invention is given, which should not be construed as limiting the present invention to only the features or combinations of features described in this section.

[0009] The present invention relates to an enhanced overlay device, comprising the following components: an image source, which is used to provide real-time image data of an area of ​​interest in the form of a real-time image pixel stream. A data server, which is used to provide overlay data. A synthesizer device, wherein the synthesizer device is a hardware device and is used to provide display data in the form of a real-time image pixel stream by overlaying the provided real-time image data with the provided overlay data. The synthesizer device is used to determine a restricted area in the provided real-time image data, wherein the provided overlay data includes: restricted area overlay data associated with the restricted area of ​​the real-time image data, and free overlay data associated with an area of ​​the real-time image data outside the restricted area. The synthesizer device is used to limit the overlay performed using the restricted area overlay data. The enhanced overlay device also includes a display device for receiving and displaying the provided display data.

[0010] General Description of the Invention

[0011] In this section, a description of general features of the invention is given, for example, by reference to possible embodiments of the invention.

[0012] The invention is defined by the subject matter of the independent claims. Additional characteristics of the invention are presented in the dependent claims.

[0013] In a first aspect of the present invention, an enhanced overlay device includes an image source for providing real-time image data of an area of ​​interest in the form of a real-time image pixel stream. The enhanced overlay device includes a data server for providing overlay data. The enhanced overlay device includes a synthesizer device. The synthesizer device is a hardware device, wherein the function of the hardware device is implemented in hardware, and is used to provide display data in the form of a real-time image pixel stream by overlaying the provided real-time image data with the provided overlay data. The synthesizer device is used to determine a restricted area in the provided real-time image data. The provided overlay data includes: restricted area overlay data associated with a restricted area of ​​the real-time image data, and free overlay data associated with an area outside the restricted area of ​​the real-time image data. The synthesizer device is used to limit the overlay performed using the restricted area overlay data. The enhanced overlay device includes a display device, which is used to receive and display the provided display data.

[0014] In other words, the synthesizer device is a hardware circuit that safely prevents accidental superimposition in a restricted area (also referred to as a special area) of the real-time image displayed on the display device. Thus, patient safety is increased, in particular by functional safety measures.

[0015] Known approaches use software to protect restricted areas, which is not considered a functional safety measure because any misbehaving software may hide significant portions of live video related to live video data, or resolving the behavior involves deactivation and / or restart of the entire software system.

[0016] The term "enhanced overlay device" as used herein relates to a device for overlaying a main video with overlay data. The main video (also referred to as display data) comprises a video (particularly a real-time video) of a patient on which a user (particularly a medical staff) is performing a medical workflow (particularly a surgical procedure). Thus, the main video comprises important information of the user which the enhanced overlay device overlays with additional information (overlay data).

[0017] The term "real-time image data" as used herein relates to image data, in particular image data of a region of interest of a patient for which a medical workflow is performed. The real-time image data is preferably provided by an image source, in particular a camera, such as an endoscope or a microscope, in the form of a real-time image pixel stream. During the medical workflow, in particular the real-time image data is displayed to a user by a display device. The term "real-time" as used herein relates to a time delay (also referred to as latency) of less than 100us, wherein the time delay is defined by a time window between the time when the image source provides the real-time image pixel stream and the time when the display displays the corresponding real-time image pixel stream.

[0018] The term "overlay data" as used herein relates to any kind of auxiliary data that should help a user to perform a medical workflow.

[0019] The term "region of interest" as used herein relates to an area of ​​a patient where a user is treated during a medical workflow.

[0020] The term "display device" as used herein relates to a display that is, for example, provided in an operating room and used by a user to visualize parts of a patient, such as provided by a digital endoscope or a digital microscope. For example, the display device is a smart display. Preferably, the display device is an ultra-high definition display device.

[0021] The term "restricted area" as used herein relates to a portion of the real-time image data that is particularly important for the user. In particular, the restricted area relates to a portion of the real-time image data, which portion of the real-time image data shows a portion of a patient treated by the user, such as an organ or a blood vessel. The restricted area is also referred to as a special area or a critical area. In other words, the region of interest superimposed by the real-time image includes the portion of the patient treated by the user (usually the central portion of the image of the real-time image data). The region of interest preferably superimposes a larger area than the portion of the patient treated by the user, thereby leaving extra space for auxiliary data (in particular at the borders of the region of interest).

[0022] The term "compositor device" as used herein relates to a compositor component (which is also referred to as a mixer component) implemented in hardware. In other words, the compositor device is used to mix incoming real-time image data and overlay data to provide corresponding display data shown on a display, showing real-time video of the real-time image data and auxiliary data of the overlay data overlaying the real-time video. In other words, the compositor device ingests incoming real-time pixels (real-time image data) of a video stream and synthesizes output pixels (display data) from the input (real-time image data) and the overlay (overlay data).

[0023] The term "hardware device" as used herein refers to a device for performing its functions using a series of logic blocks. A logic block includes a circuit of multiple logic gates. In contrast, a software device, which may include a logic block, has its main functions performed by a central processing unit CPU to some extent. Examples of hardware devices include field programmable gate arrays FPGAs and application-specific integrated circuits ASICs. The CPU in a software device executes the program in turn. A hardware device does not execute a program in the software sense, but rather executes its functions defined by the circuits of the logic blocks of the hardware device, thereby allowing a parallel system architecture to have a much higher speed than software. In addition, without running any software, hardware devices are not prone to software errors and are therefore more reliable than software applications.

[0024] The augmented overlay device is preferably part of a digital operating room. The digital operating room comprises a plurality of devices which should help a user, in particular a surgeon, to perform a medical workflow on a patient. The user uses, for example, a digital endoscope which provides real-time image data. The real-time image data is displayed on a display in the operating room so that the user can watch his movements on the patient on the display (in particular in a magnified manner). The real-time image data is further processed by a synthesizer device to superimpose the real-time image data with overlay data (also referred to as auxiliary data) so as to provide the user with additional information within the real-time image data on the display.

[0025] If the software generates an overlay on a restricted area of ​​the real-time image data, or in other words a real-time video based on the real-time image data, then the software method processes each pixel of the real-time image pixel stream identically. A hardware-based synthesizer device is used to restrict the overlay of pixel areas of the real-time image pixel stream, thereby providing functional safety.

[0026] The hardware approach of the synthesizer device allows processing of overlay data on a real-time pixel stream and therefore does not use any frame buffer or similar representation of frames of real-time image data in memory that software would need to use. Therefore, the hardware synthesizer device is less prone to software errors. This provides overlay of real-time image data with improved reliability.

[0027] This is achieved by the following structure: real-time video is fed to the mixer as a real-time stream of pixels. The mixer parses the pixel stream and synchronizes to it in terms of picture start and line position start in the pixel stream. Using this synchronization mechanism, the mixer pre-fetches a line of overlay data from the slower main DRAM to the internal RAM before it is needed. This allows overlay calculations to be performed in real time based on the received real-time video pixels and the corresponding overlay pixel information from the pre-fetch buffer.

[0028] Furthermore, since the hardware synthesizer device acts directly on the real-time image pixel stream, the overlay can be provided in real time, particularly under 100us, thereby providing an overlay of real-time image data with improved latency (particularly in ultra-HD video scenes). In other words, the display data may also be referred to as real-time display data, since the overlay is performed directly on the real-time image data, the display data has a latency of less than 100us to the real-time image pixel stream.

[0029] This allows for hardware-based real-time overlay of real-time image data with improved safety.

[0030] In a preferred embodiment, each pixel of the real-time image pixel stream of the real-time image data comprises an xy coordinate. Using the predetermined restricted xy coordinates, a restricted area of ​​the real-time image pixel stream of the real-time image data is determined.

[0031] In other words, the synthesizer device is provided with the xy coordinates of the region of interest and the xy coordinates of the region of interest relative to the restricted region. Since each pixel of the real-time image pixel stream includes the xy coordinates of the region of interest, the synthesizer device can determine on a pixel-by-pixel basis whether the pixel is part of the restricted region and, if necessary, apply restrictions.

[0032] In a preferred embodiment, the restricted area comprises a shape.The size of the shape and the position of the shape in the region of interest are at least partially defined by xy coordinates.

[0033] Preferably, the shape comprises a rectangle, a square, a circle and / or an ellipse. Further preferably, the shape of the restricted area is predetermined, in particular, as required or fixed in hardware implementation. Alternatively, the shape of the restricted area is selectable (in particular, selected by a user).

[0034] Therefore, the synthesizer device is provided with predetermined dimensions of the shape of the confined area, in particular the type and size of the shape, and the position of the shape in the real-time image data. The position of the shape preferably includes at least one anchor point, in particular in xy coordinates. For example, the position of the shape includes the center point of a circular shape, wherein the dimensions of the shape indicate that the shape is circular and has a diameter of 1000 pixels.

[0035] Thus, a user can easily define a restricted area which can be interpreted by the compositor device using the xy coordinates of the pixels of the real-time image pixel stream.

[0036] In a preferred embodiment, a mask layer is used to determine restricted areas of the real-time image pixel stream of the real-time image data.

[0037] Preferably, the mask layer is implemented as a random access memory, RAM or RAM area. Each storage cell of the RAM or RAM area comprises a restriction of pixels (or groups of pixels for lower granularity) of the real-time image stream defining the restricted area, or in other words a restriction level.

[0038] In a preferred embodiment, the compositor device comprises a blending unit for providing the display data in the form of a real-time image pixel stream by superimposing the provided real-time image data with the provided overlay data. The blending unit is for not reading the restricted area overlay data.

[0039] In other words, the blending unit checks each pixel of the overlay data to see if it is part of a restricted area. Therefore, each part of the overlay data relating to a restricted area is not processed further, or in other words is not read. Therefore, only the overlay data not relating to a restricted area is stored in the read access memory RAM of the augmented overlay device to be blended with the real-time image pixel stream by the blending unit. Therefore, fewer memory transactions have to be performed on the RAM (or virtual RAM) of the augmented overlay device when determining display data.

[0040] In a preferred embodiment, the compositor device comprises a blending unit for providing the display data in the form of a real-time image pixel stream by superimposing the provided real-time image data with the provided overlay data. The compositor device is arranged to discard the restricted area overlay data.

[0041] In other words, the mixing unit checks each pixel of the overlay data after the overlay data has been read into a memory (in particular a RAM) of the enhanced overlay device, whether it is part of a restricted area. Thus, each part of the overlay data relating to the restricted area is simply discarded, or in other words deleted from the memory. Thus, the mixing unit mixes only overlay data not relating to the restricted area with the real-time image pixel stream.

[0042] In a preferred embodiment, the compositor device comprises a blending unit for providing display data in the form of a real-time image pixel stream by superimposing the provided real-time image data with the provided overlay data. The compositor device is for determining a transparency / alpha value of the restricted area overlay data.

[0043] Preferably, the synthesizer device is used to limit the transparency / alpha value of the restricted area overlay data to a predetermined minimum value. Further preferably, the minimum value is less than 50% transparency. Further preferably, the minimum value is less than 20% transparency.

[0044] Preferably, the transparency value is provided by a user (particularly via an interactive menu displayed on the display device).

[0045] In a preferred embodiment, the synthesizer device is used to overlay data for the restricted area only when the additional security procedure is successfully performed.

[0046] Preferably, the term "taking into account" the superimposed data as used herein relates to reading or not discarding the superimposed data as described above.

[0047] As used herein, the term "security process" relates to additional security measures implemented to allow overlaying of restricted areas with overlay numbers without restriction or overlaying of restricted areas with overlay data outside the aforementioned transparency levels.

[0048] In some scenarios, full overlay of real-time image data may be desired. For example, augmented reality navigation functions or augmented reality 3D functions inherently require overlaying of corresponding parts of the patient in the real-time image stream. Therefore, additional safety measures referred to as safety procedures are used to protect the ability to overlay restricted areas with overlay data outside of the above restrictions.

[0049] In a preferred embodiment, the security process includes a predetermined specific user interaction performed by the user.

[0050] In other words, the restricted area overlay data is only taken into account after permission through user interaction. Preferably, the augmented overlay device comprises a hardware switch that can be activated by user interaction and activates / deactivates the consideration of the restricted area overlay data. The hardware switch can be used by the user directly at the augmented overlay device and / or by the user on a user interface displayed in a non-restricted area of ​​the real-time image data.

[0051] In a preferred embodiment, the security process includes a complex enabling sequence.

[0052] This means that it is not enough to flip a bit by software to enable or disable overlay, but a safer way to avoid unwanted activation is also needed. This preferably includes having a "magic" byte sequence to write to a register (e.g., 4711) or having a changing pattern (e.g., first time 4711, second time 4712...) or expecting a correct calculation (e.g., read a number from a register, multiply it by 3 and write it back to the register, and only if this is correct, activation occurs).

[0053] Therefore, accidental unrestricted superposition of real-time image data can be prevented.

[0054] In a preferred embodiment, the complex enabling sequence includes a challenge response from a data server providing the overlay data.

[0055] In other words, a challenge response from the software to the hardware of the synthesizer device is necessary. This is similar to challenge response authentication: the software requests a challenge "string" from the secure overlay device. The software calculates a response with a security "certificate", specifically only available in the authorized software part, and the legitimacy of the request is checked in the distributed software part. This response is then sent to the augmented overlay device, where it is checked for a correct response, and only in case of a correct response the restricted area overlay data is considered.

[0056] In a preferred embodiment, the security process includes a request for overlay that enables the synthesizer device to consider the restricted area overlay data for a predetermined limited time and automatically disable the overlay if the security process is not passed before the limited time expires.

[0057] In other words, the unrestricted area overlay data is considered only for a limited time. Therefore, the request for the overlay must be repeated regularly.

[0058] The value of the limited time is predetermined in the hardware of the synthesizer device, or the user may input the value of the limited time through a user interface displayed in the non-restricted area of ​​the real-time image data.

[0059] Preferably, the user may input a request for superimposition on a user interface displayed in a non-restricted area of ​​the real-time image data.

[0060] In a preferred embodiment, the image source comprises a medical imaging device.

[0061] Preferably, the image source is an endoscope or a microscope (particularly a digital endoscope or a digital microscope).

[0062] In a preferred embodiment, the overlay data includes sensor data, patient monitoring data, 3D visualization data, overlay menu data, and / or video conferencing data.

[0063] The superimposed data includes, for example, patient monitoring data indicating the patient's vital signs. The superimposed data includes, for example, 3D visualization data indicating a 3D menu or 3D extension of 2D data of the real-time image data. The superimposed data includes, for example, patient safety data indicating navigation data or marking data of a surgical navigation system used in a medical workflow. The superimposed data includes, for example, superimposed menu data indicating an interactive menu and user input of the superimposed data. The superimposed data includes, for example, video conferencing data indicating a video feed of a second user communicating with a user performing a medical workflow. The superimposed data includes, for example, sensor fusion data indicating preoperative and / or intraoperative sensor images (such as computed tomography images or ultrasound images). The superimposed data includes, for example, remote illustration data intended to transmit video data to another doctor, particularly for obtaining a second opinion. The other doctor can mark an area in the video, which is then displayed as an enhancement on the main doctor's video screen on the real-time video.

[0064] In a preferred embodiment, the synthesizer device is at least partially implemented in an ASIC or FPGA.

[0065] Preferably, the synthesizer device is fully implemented in an ASIC or FPGA.Preferably, the functionality of the synthesizer device is described by a hardware description language, HDL, such as VHDL or Verilog.

[0066] In a second aspect of the present invention, a method for enhancing superposition includes the following steps. Real-time image data of an area of ​​interest is provided by an image source in the form of a real-time image pixel stream. Superposition data is provided by a data server. Display data is provided in the form of a real-time image pixel stream by a synthesizer device as a hardware device by superimposing the provided real-time image data with the provided superposition data, wherein the function of the hardware device is implemented in hardware, including determining a restricted area in the provided real-time image data by the synthesizer device. The provided superposition data includes: restricted area superposition data associated with the restricted area of ​​the real-time image data, and free superposition data associated with an area outside the restricted area of ​​the real-time image data. The method also includes the following steps: the synthesizer device limits the superposition performed using the restricted area superposition data. The provided display data is displayed by a display device.

[0067] For example, the present invention does not involve or specifically include or cover invasive steps that would represent substantial physical interference with the body that would require professional medical expertise and would require substantial health risks even when performed with the required professional care and expertise. For example, the present invention does not include steps for positioning a medical implant so that it is fastened to an anatomical structure, or steps for fastening a medical implant to an anatomical structure, or steps for preparing an anatomical structure so that a medical implant is fastened thereto. More specifically, the present invention does not involve or specifically include or cover any surgical or therapeutic activity. Instead, the present invention is applicable to a digital operating room, where an enhanced overlay device with hardware-based functional safety overlays additional patient data onto a real-time video stream while delivering documentation needs. For this reason alone, performing the present invention does not require or imply surgical or therapeutic activities, particularly no surgical or therapeutic steps.

[0068] The invention also relates to the use of an augmented overlay device or any of its embodiments in a digital operating room, in particular for overlaying additional patient data onto a real-time video stream while delivering documentation requirements.

[0069] definition

[0070] In this section, definitions of specific terms used in the present disclosure are provided, which also form a part of the present disclosure.

[0071] Markers

[0072] The function of the marker to be detected by a marker detection device (e.g. a camera or ultrasound receiver or an analysis device, such as a CT or MRI device) makes it possible to determine the spatial position of the marker (i.e. its spatial position and / or alignment). The detection device is, for example, part of a navigation system. The marker may be an active marker. An active marker may, for example, emit electromagnetic radiation and / or waves that may be in the infrared, visible and / or ultraviolet spectral range. However, the marker may also be passive, i.e. may, for example, reflect electromagnetic radiation in the infrared, visible and / or ultraviolet spectral range, or may block x-ray radiation. To this end, the marker may be provided with a surface having corresponding reflection properties, or may be made of metal to block x-ray radiation. The marker may also reflect and / or emit electromagnetic radiation and / or waves in the radiofrequency range or in the ultrasonic range. The marker preferably has a spherical and / or ellipsoidal shape, and may therefore be referred to as a marker sphere; however, the marker may also present an angular shape, such as a cubic shape.

[0073] Marking equipment

[0074] The marking device may be, for example, a reference star or an indicator or a single marker or a plurality of (separate) markers, which are then preferably in a predetermined spatial relationship. The marking device comprises one, two, three or more markers, wherein two or more such markers are in a predetermined spatial relationship. The predetermined spatial relationship is, for example, known to the navigation system and, for example, stored in a computer of the navigation system.

[0075] In another embodiment, the marking device includes an optical pattern, for example, on a two-dimensional surface. The optical pattern may include a plurality of geometric shapes, such as circles, rectangles, and / or triangles. The optical pattern may be identified in an image captured by a camera, and the position of the marking device relative to the camera may be determined based on the size of the pattern in the image, the orientation of the pattern in the image, and the distortion of the pattern in the image. This allows relative positions in up to three rotational dimensions and up to three translational dimensions to be determined from a single two-dimensional image.

[0076] The position of the marking device may be determined, for example, by a medical navigation system. If the marking device is attached to an object, such as a bone or a medical instrument, the position of the object may be determined based on the position of the marking device and the relative position between the marking device and the object. Determining this relative position is also known as registering the marking device and the object. The marking device or object may be tracked, which means that the position of the marking device or object is determined two or more times over time.

[0077] Marker Holder

[0078] A marker holder is understood to mean an attachment device for a single marker, which is used to attach the marker to a holding element of an instrument, a part of a body and / or a reference star, wherein the marker holder can be attached so that the marker holder is stationary and advantageously enables the marker to be detached. The marker holder can be, for example, rod-shaped and / or cylindrical. A fastening device for the marker device, such as a latch mechanism, can be arranged at the end of the marker holder facing the marker and helps to place the marker device on the marker holder with a force fit and / or a positive fit.

[0079] Pointer

[0080] A pointer is a stick that includes one or more (advantageously, two) markers fastened thereto and that can be used to measure individual coordinates on a part of the body, such as spatial coordinates (i.e., three-dimensional coordinates), wherein a user guides the pointer (e.g., a part of the pointer having a defined and advantageously fixed position relative to at least one marker attached to the pointer) to a position corresponding to the coordinates, such that the position of the pointer can be determined by detecting the markers on the pointer using a surgical navigation system. The relative position between the markers of the pointer and the part of the pointer used to measure the coordinates (e.g., the tip of the pointer) is, for example, known. The surgical navigation system then enables the position (of the three-dimensional coordinates) to be assigned to a predetermined body structure, wherein the assignment can be performed automatically or by user intervention.

[0081] Reference star

[0082] A "reference star" refers to a device having a plurality of markers (advantageously three markers) attached thereto, wherein the markers are (e.g., detachably) attached to the reference star such that they are stationary, thus providing a known (and advantageously fixed) position of the markers relative to each other. The position of the markers relative to each other may be individually different for each reference star used within the framework of the surgical navigation method, so as to enable the surgical navigation system to identify the corresponding reference star based on the position of its markers relative to each other. Thus, the object (e.g., an instrument and / or a part of the body) to which the reference star is attached may also be identified and / or distinguished accordingly. In the surgical navigation method, the reference star is used to attach a plurality of markers to an object (e.g., a bone or a medical device) so as to be able to detect the position of the object (i.e., its spatial position and / or alignment). Such a reference star is characterized, for example, by a way of attaching to the object (e.g., a clamp and / or a wire), and / or by a holding element to ensure a distance between the marker and the object (e.g., in order to facilitate visibility of the marker for a marker detection device), and / or by a marker holder that is mechanically connected to the holding element and to which the marker can be attached.

[0083] Navigation system

[0084] The present invention also relates to a navigation system for computer-assisted surgery. The navigation system preferably includes the aforementioned computer for processing data provided according to the computer-implemented method described in any one of the embodiments described herein. The navigation system preferably includes a detection device for detecting the position of detection points representing the main point and the auxiliary point, so as to generate a detection signal and provide the generated detection signal to the computer, so that the computer can determine the absolute main point data and the absolute auxiliary point data based on the received detection signal. The detection point is, for example, a point on the surface of the anatomical structure, for example, detected by an indicator. In this way, the absolute point data can be provided to the computer. The navigation system also preferably includes a user interface for receiving the calculation results (for example, the position of the main plane, the position of the auxiliary plane and / or the position of the standard plane) from the computer. The user interface provides the received data as information to the user. Examples of user interfaces include display devices such as monitors or speakers. The user interface can use any kind of indication signal (for example, a visual signal, an audio signal and / or a vibration signal). An example of a display device is an augmented reality device (also called augmented reality glasses), which can be used as so-called "goggles" for navigation. A specific example of such augmented reality glasses is Google Glass (trademark of Google Inc.). Augmented reality devices can be used to input information into a navigation system's computer through user interaction and to display information output by the computer.

[0085] Surgical navigation system

[0086] A navigation system, such as a surgical navigation system, is understood to mean a system which may comprise: at least one marking device; a transmitter which emits electromagnetic waves and / or radiation and / or ultrasound; a receiver which receives electromagnetic waves and / or radiation and / or ultrasound; and an electronic data processing device connected to the receiver and / or the transmitter, wherein the data processing device (e.g. a computer) comprises, for example, a processor (CPU) and a working memory, and advantageously comprises an indication device for emitting indication signals (e.g. a visual indication device, such as a monitor and / or an audio indication device, such as a loudspeaker and / or a tactile indication device, such as a vibrator) and a permanent data memory, wherein the data processing device processes the navigation data forwarded to it by the receiver and may advantageously output guidance information to a user via the indication device. The navigation data may be stored in the permanent data memory and, for example, compared with data previously stored in said memory.

[0087] Landmark

[0088] Landmarks are defining elements of an anatomical body part that are always the same or are repeated with high similarity in the same anatomical body part of multiple patients. Typical landmarks are, for example, the tip of the epicondyle of the femur or the transverse and / or dorsal processes of the vertebrae. Points (primary or auxiliary) can represent such landmarks. Landmarks located on a characteristic anatomical structure of a body part (for example, on its surface) can also represent the structure. Landmarks can represent an anatomical structure as a whole or only one point or part thereof. Landmarks can also be located, for example, on an anatomical structure, such as a protruding structure. An example of such an anatomical structure is the posterior part of the iliac. Another example of a landmark is a landmark defined by the edge of the acetabulum, such as a landmark defined by the center of the edge. In another example, a landmark represents the bottom or deepest point of the acetabulum, which is derived from multiple detection points. Therefore, a landmark can, for example, represent multiple detection points. As described above, a landmark can represent an anatomical feature defined based on a characteristic structure of a body part. In addition, a landmark can also represent an anatomical property defined by the relative motion of two body parts, such as the rotation center of the femur when moving relative to the acetabulum.

[0089] Imaging geometry

[0090] The information about the imaging geometry preferably comprises information allowing the calculation of the analysis image (x-ray image), given the known relative position between the imaging geometry analysis device and the analysis object (anatomical body part) to be analyzed by x-ray radiation, if the analysis object to be analyzed is known, where "known" means that the spatial geometry (size and shape) of the analysis object is known. This means that, for example, three-dimensional "spatially resolved" information about the interaction between the analysis object (anatomical body part) and the analysis radiation (x-ray radiation) is known, where "interaction" means, for example, that the analysis radiation is blocked by the analysis object or partially or completely allowed to pass. The position and specific orientation of the imaging geometry are defined, for example, by the position of the x-ray device, for example, by the position of the x-ray source and the x-ray detector and / or, for example, by the position of the multiple (multi-path) x-ray beams that pass through the analysis object and are detected by the x-ray detector. The imaging geometry describes, for example, the position (i.e., the position, in particular the orientation) and the shape (e.g., a cone exhibiting a specific tilt angle) of the multiple (multi-path) x-ray beams. The position can be represented, for example, by the position of an x-ray beam passing through the center of the multiple beams or by the position of a geometric object (such as a truncated cone) representing multiple (multi-path) x-ray beams. Information about the above interactions is preferably known in three dimensions, for example from three-dimensional CT, and describes the interactions of points and / or regions of the analysis object (for example, all points and / or regions of the analysis object) in a spatially resolved manner. For example, knowledge of the imaging geometry allows the position of a radiation source (for example, an x-ray source) to be calculated relative to an image plane (for example, the plane of an x-ray detector). Regarding the connection between a three-dimensional analysis object and a two-dimensional analysis image defined by the imaging geometry, reference is made, for example, to the following publications:

[0091] 1. “An Efficient and Accurate Camera Calibration Technique for 3DMachine Vision,” Roger Y. Tsai, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Miami Beach, Florida, 1986, pp. 364-374

[0092] 2. “A Versatile Camera Calibration Technique for High-Accuracy 3DMachine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses,” Roger Y. Tsai, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, August 1987, pp. 323-344.

[0093] 3. “Fluoroscopic X-ray Image Processing and Registration for Computer-Aided Orthopedic Surgery”, Ziv Yaniv

[0094] 4. EP08156293.6

[0095] 5. US61 / 054,187

[0096] shape representative

[0097] A shape representative represents a characteristic aspect of the shape of an anatomical structure. Examples of shape representatives include straight lines, planes, and geometric figures. Geometric figures can be one-dimensional (e.g., axes or arcs), two-dimensional (e.g., polygons and circles), or three-dimensional (e.g., cubes, cylinders, and spheres). The relative positions between shape representatives can be described in a reference system (e.g., by coordinates or vectors), or can be described by geometric variables (e.g., lengths, angles, areas, volumes, and proportions). Characteristic aspects represented by shape representatives are, for example, symmetry properties, such as represented by a symmetry plane. Another example of a characteristic aspect is the direction of extension of an anatomical structure, which is represented, for example, by a longitudinal axis. Another example of a characteristic aspect is the cross-sectional shape of an anatomical structure, which is represented, for example, by an ellipse. Another example of a characteristic aspect is the surface shape of a portion of an anatomical structure, which is represented, for example, by a plane or a hemisphere. For example, the characteristic aspect constitutes an abstraction of an actual shape or an abstraction of an attribute of an actual shape (e.g., its symmetry attribute or longitudinal extension). For example, a shape representative represents such an abstraction.

[0098] Analytical equipment

[0099] The movement of the treatment body part is, for example, due to the movement referred to as "significant movement" hereinafter. In this regard, reference is also made to EP 2189943 A1 and EP 2189940 A1, which are also disclosed as US 2010 / 0125195 A1 and US2010 / 0160836 A1, respectively, which discuss these important movements in detail. In order to determine the position of the treatment body part, an analysis device such as an x-ray device, a CT device or an MRT device is used to generate an analysis image (such as an x-ray image or an MRT image) of the body. For example, the analysis device is constructed to perform a medical imaging method. The analysis device, for example, uses a medical imaging method, and is, for example, a device for analyzing the patient's body, for example, by using waves and / or radiation and / or energy beams, such as electromagnetic waves and / or radiation, ultrasound and / or particle beams. The analysis device is, for example, a device for generating an image (for example, a two-dimensional or three-dimensional image) of the patient's body (and, for example, the internal structure and / or anatomical part of the patient's body) by analyzing the body. The analysis device is, for example, used for medical diagnosis, such as radiology. However, it may be difficult to identify the treatment body part within the analysis image. For example, an indicator body part that is associated with a change in position of the treated body part and, for example, a movement of the treated body part may be more easily identified. Thus, tracking the indicator body part allows tracking the movement of the treated body part based on a known correlation between a change in position (e.g., movement) of the indicator body part and a change in position (e.g., movement) of the treated body part. As an alternative or in addition to tracking the indicator body part, a marker detection device may be used to track a marker device (which may be used as an indicator and is therefore referred to as a "marker indicator"). The position of the marker indicator has a known (predetermined) correlation (e.g., a fixed relative position relative to the indicator structure) with the position of an indicator structure (such as the chest wall, e.g., true ribs or false ribs, or a diaphragm or intestinal wall, etc.), which indicator structures change their position, for example, due to vital movements.

[0100] Imaging methods

[0101] In the medical field, imaging methods (also referred to as imaging modalities and / or medical imaging modalities) are used to generate image data (e.g., two-dimensional or three-dimensional image data) of anatomical structures of the human body (such as soft tissue, bones, organs, etc.). The term "medical imaging method" is understood to mean an (advantageously, device-based) imaging method (e.g., so-called medical imaging modalities and / or radiological imaging methods), such as, for example, computed tomography (CT) and cone beam computed tomography (CBCT, such as volume CBCT), x-ray tomography, magnetic resonance tomography (MRT or MRI), conventional x-ray, ultrasound imaging and / or ultrasonography, and positron emission tomography. For example, the medical imaging method is performed by an analysis device. Examples of medical imaging modes applied by the medical imaging method include: X-ray radiography, nuclear magnetic resonance imaging, medical sonography or ultrasound, endoscopy, elastic imaging, tactile imaging, thermography, medical photography, and nuclear medicine functional imaging techniques, such as positron emission tomography (PET) and single photon emission computed tomography (SPECT), as mentioned in Wikipedia.

[0102] The image data thus generated are also referred to as "medical imaging data". The analysis device is used, for example, to generate image data in an imaging method based on an apparatus. Imaging methods are used, for example, for medical diagnosis to analyze an anatomical body in order to generate an image described by the image data. Imaging methods are also used, for example, to detect pathological changes in the human body. However, some changes in the anatomical structure, such as pathological changes in the structure (tissue), may not be detectable and, for example, may not be visible in the image generated by the imaging method. A tumor represents an example of a change in the anatomical structure. If a tumor grows, it can be said that it represents an expanded anatomical structure. This expanded anatomical structure may not be detectable; for example, only a portion of the expanded anatomical structure may be detectable. Primary / high-grade brain tumors are, for example, typically visible on MRI scans when a contrast agent is used to infiltrate the tumor. MRI scans represent an example of an imaging method. In the case of an MRI scan of such a brain tumor, the signal enhancement in the MRI image (due to the contrast agent infiltrating the tumor) is considered to represent a solid tumor mass. Therefore, the tumor is detectable and, for example, discernible in the image generated by the imaging method. In addition to these tumors, known as "enhancing" tumors, approximately 10% of brain tumors are believed to be indistinguishable on scans and are not visible, for example, to a user viewing an image generated by an imaging method.

[0103] Medical Workflow

[0104] A medical workflow includes multiple workflow steps performed during a medical treatment and / or medical diagnosis. The workflow steps are typically, but not necessarily, performed in a predetermined order. Each workflow step, for example, means a specific task, which can be a single action or a group of actions. Examples of workflow steps are capturing medical images, positioning a patient, attaching a marker, performing a resection, moving a joint, placing an implant, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] In the following, the present invention is described with reference to the accompanying drawings, which give a background explanation and show specific embodiments of the present invention. However, the scope of the present invention is not limited to the specific features disclosed in the context of the accompanying drawings, in which:

[0106] FIG. 1a shows an enhanced overlay device with activated overlay in a restricted area;

[0107] FIG1 b shows an enhanced overlay device with deactivated overlay in a confined area;

[0108] FIG2 is a schematic diagram of a synthesizer device; and

[0109] FIG. 3 is a schematic diagram of a method for enhancing superposition. DETAILED DESCRIPTION

[0110] FIG. 1a shows an augmented overlay device 100, which includes an image source 10, a data server 20, a synthesizer device 30, and a display 40. In addition, FIG. 1a shows that a user 50 (in this case a surgeon) is performing a medical workflow (in this case a surgery) on a patient 70. The scenario shown in FIG. 1a relates to a digital operating room, wherein the user 50 is supported by the augmented overlay device 100. The image source 10 includes a camera that records a region of interest ROI of the patient 70. The region of interest ROI is a portion of the patient 70 treated by the user 50. In this case, the user 50 treats an organ 71 of the patient 70. The image source 10 records the region of interest ROI and provides real-time image data Di to the synthesizer device 30. The data server 20 is used to provide overlay data Do to the synthesizer device 30. The synthesizer device 30 is an ASIC / FPGA for determining display data Dd using the provided real-time image data Di and overlay data Do. The real-time image data Di includes a real-time image pixel stream, wherein the overlay data Do includes information that should be displayed on the display 40 together with the real-time image data Di. Therefore, the display data Dd is basically the real-time image data Di on which the superimposed data Do is superimposed.

[0111] The real-time image data Di includes a real-time video captured by the image source 10 with a superimposed region of interest roi. The real-time video shows the organ being treated by the user 50. The area around the organ is of key interest to the user 50 and is therefore defined as a restricted area 41. The superimposition of the real-time image data Di should be limited in the restricted area 41 so that the user 50 is not restricted in his medical workflow.

[0112] The superimposition data Do may be divided into limited area superimposition data Dor and free superimposition data Dof.

[0113] The restricted area overlay data Dor is related to the restricted area. In this case, the restricted area overlay data Dor includes indicators of blood vessels passing through the organ 71. Due to the general structure of the organ 71, the restricted area overlay data Dor may be generally known to the data server 20, or may be provided externally by a sensor (such as an X-ray device).

[0114] The free overlay data Dof is associated with the real-time image data Di outside the restricted area 41. As shown, the free overlay data Dof includes sensor data, in this case ultrasound data, patient monitoring data, menu data and / or video conferencing data. The menu data indicates a user interface for the user 50 that helps the user 50 control the synthesizer device 30. The user 50 includes an input device 60 that provides control data 61 to the data server 20 to adjust the overlay data Do.

[0115] The synthesizer device 30 comprises an FPGA or an ASIC which performs the determination of the display data Dd by mixing the real-time image data Di with the superimposed data Do. In this case, the synthesizer device 30 also comprises an external hardware switch 31 which can be activated by the user 50. In FIG. 1a , the switch is activated, allowing the synthesizer device 30 to mix the real-time image data Di with the free superimposed data Dof and the restricted area superimposed data Dor. Thus, the indicator of the blood vessel is superimposed with the real-time image data Di in the restricted area 41.

[0116] The user 50 has activated the superimposition of the restricted area 41, in particular via a user interface on the display 40. However, due to an unexpected emergency situation of the patient 70, the user 50 wants to immediately prevent superimposition in the restricted area 41. Therefore, the user 50 may deactivate the external switch 31 of the synthesis device 30, preventing any superimposition of the restricted area 41 (which is indicated in FIG. 1 b ).

[0117] Therefore, the synthesizer device 30 still uses the free superimposition data Dof to superimpose the real-time image data Di, however, the restricted area superimposition data Dor is no longer considered, so that the restricted area 41 is not superimposed.

[0118] Since the compositor device 30 is implemented in hardware, the overlay process is not prone to any software errors and allows direct manipulation of the real-time image pixel stream, allowing improved latency with latency below 100 us.

[0119] 2 is a schematic diagram of a synthesizer device 30. The synthesizer device 30 comprises a mixing unit 32 for receiving real-time image data Di. The synthesizer device 30 further comprises a superposition unit 33 for receiving superposition data Do. In addition, the superposition unit 33 is used to receive switch data Ds (particularly provided by the hardware switch 31). The switch data Ds indicates whether to allow a restricted area 41 in the real-time image data Di to be superimposed by the superposition data Do.

[0120] Therefore, if the switch data Ds indicates that all superpositions are free, the superposition unit 33 provides the free superposition data Dof and the restricted area superposition data Dor of the superposition data Do to the mixing unit 32. The mixing unit 32 therefore uses the free superposition data Dof and the restricted area superposition data Dor to perform superposition on each pixel of the real-time image data Di. Otherwise, the superposition unit 33 only provides the free superposition data Dof to the mixing unit 32, and the mixing unit 32 only uses the free superposition data Dof to superimpose the real-time image data Di, thereby inherently causing the restricted area 41 to not be superimposed.

[0121] FIG. 3 is a schematic diagram of a method for enhancing superposition.

[0122] In a first step S10, the image source 10 provides real-time image data Di of a region of interest roi in the form of a real-time image pixel stream. In a second step S20, the data server 20 provides overlay data Do. In a third step S30, the synthesizer device 30 as a hardware device provides display data Dd in the form of a real-time image pixel stream by overlaying the provided real-time image data Di with the provided overlay data Do, the step comprising: the synthesizer device 30 determines a restricted area 41 in the provided real-time image data Di, and the synthesizer device 30 limits the overlaying using the restricted area overlay data Dor, wherein the provided overlay data Do comprises: restricted area overlay data Dor associated with the restricted area 41 of the real-time image data Di, and free overlay data Dof associated with an area of ​​the real-time image data Di outside the restricted area 41. In a fourth step S40, the display device 40 receives and displays the provided display data Dd.

Claims

1. An enhanced superposition device (100), comprising: An image source (10) for providing real-time image data (Di) of a region of interest (roi) in the form of a real-time image pixel stream; A data server (20) for providing overlay data (Do); A synthesizer device (30), wherein the synthesizer device (30) is a hardware device, wherein the function of the hardware device is implemented in hardware and is used to provide display data (Dd) in the form of a real-time image pixel stream by superimposing provided real-time image data (Di) using provided superimposition data (Do); wherein the synthesizer device (30) is used to determine a restricted area (41) in the provided real-time image data (Di), wherein the provided superimposition data (Do) includes: restricted area superimposition data (Dor) associated with the restricted area (41) of the real-time image data (Di), and free superimposition data (Dof) associated with an area of ​​the real-time image data (Di) outside the restricted area (41); wherein the synthesizer device (30) is used to limit superimposition performed using the restricted area superimposition data (Dor); The display device (40) is used to receive and display the provided display data (Dd).

2. The enhanced superposition device according to claim 1, in, Each pixel of the real-time image pixel stream of the real-time image data (Di) comprises xy coordinates; wherein the restricted area (41) of the real-time image pixel stream of the real-time image data (Di) is determined using predetermined restricted xy coordinates.

3. The enhanced superposition device according to claim 2, in, The restricted area (41) comprises a shape, wherein a size of the shape and a position of the shape in the region of interest are at least partially defined by xy coordinates.

4. The enhanced superposition device according to claim 1, in, A mask layer is used to determine a restricted area (41) of a real-time image pixel stream of real-time image data (Di).

5. An enhanced superposition device according to any one of the preceding claims, in, The synthesizer device (30) comprises a mixing unit (32) for providing the display data (Dd) in the form of a real-time image pixel stream by superimposing the provided real-time image data (Di) with the provided superimposition data (Do); The mixing unit (32) is used for not reading the restricted area overlay data (Dor).

6. The enhanced superposition device according to any one of claims 1 to 4, in, The synthesizer device (30) comprises a mixing unit (32) for providing display data (Dd) in the form of a real-time image pixel stream by superimposing the provided real-time image data (Di) with the provided superimposition data (Do); The synthesizer device (30) is used to discard the restricted area overlay data (Dor).

7. The enhanced superposition device according to any one of claims 1 to 4, in, The synthesizer device (30) comprises a mixing unit (32) for providing the display data (Dd) in the form of a real-time image pixel stream by superimposing the provided real-time image data (Di) with the provided superimposition data (Do); The synthesizer device (30) is used to determine the transparency value / alpha value of the restricted area overlay data (Dor).

8. An enhanced superposition device according to any one of the preceding claims, in, The synthesizer device (30) is used to consider the restricted area overlay data (Dor) only when the additional safety process is successfully performed.

9. The enhanced superposition device according to claim 8, in, The security process includes a predetermined specific user interaction performed by a user.

10. The enhanced superposition device according to any one of claims 7 to 9, in, The safety process includes a complex enabling sequence.

11. The enhanced superposition device according to claim 10, in, The complex enabling sequence includes a challenge response from a data server (20) providing the overlay data (Do).

12. The enhanced superposition device according to any one of claims 8 to 11, in, The security procedure includes a request for overlay, which enables the synthesizer device to consider the restricted area overlay data (Dor) for a predetermined limited time and automatically disable overlay if the security procedure is not passed before the limited time expires.

13. An enhanced superposition device according to any one of the preceding claims, in, The image source comprises a medical imaging device.

14. An enhanced superposition device according to any one of the preceding claims, in, The overlay data includes sensor data, patient monitoring data, 3D visualization data, overlay menu data and / or video conferencing data.

15. An enhanced superposition device according to any one of the preceding claims, in, The synthesizer device (30) is at least partially implemented in an ASIC or an FPGA.

16. A method for enhancing superposition, comprising the steps of: Real-time image data (Di) of a region of interest (roi) is provided (S10) by an image source (10) in the form of a real-time image pixel stream; The data server (20) provides (S20) overlay data (Do); A synthesizer device (30) as a hardware device provides (S30) display data (Dd) in the form of a real-time image pixel stream by superimposing the provided real-time image data (Di) using the provided superimposition data (Do), wherein the function of the hardware device is implemented in hardware; the step comprises: determining, by the synthesizer device (30), a restricted area (41) in the provided real-time image data (Di), wherein the provided superimposition data (Do) comprises restricted area superimposition data (Dor) associated with the restricted area (41) of the real-time image data (Di) and free superimposition data (Dof) associated with an area of ​​the real-time image data (Di) outside the restricted area (41); and, by the synthesizer device (30), limiting superimposition performed using the restricted area superimposition data (Dor); The provided display data (Dd) is displayed (S40) by a display device (40).

Images